Logged-In As
ACCOUNT
Not Logged In
Measure startup time The NetBSD Project
Status: Closed Time to complete: 96 hrs Mentors: Radoslaw Kujawa, Aleksej Saushev, Julian Coleman, Julian Fagir Tags: system, service

NetBSD doesn't have many services enabled by default, though most people add some more services they need for a system (depending on the usage).
Your task is about researching some of the most used services and the daemons that are inside NetBSD already, and measure the startup time depending on which and how many services are enabled.


You should write your results either in tabular form or in a paper at least two pages long, then also concluding your results.

Uploaded Work
File name/URL File size Date submitted
boottime.pdf 48.5 KB December 17 2012 19:40 UTC
boottime.tex 2.0 KB December 17 2012 21:24 UTC
boottime.pdf 70.2 KB December 18 2012 19:52 UTC
boottime.tex 4.9 KB December 18 2012 19:52 UTC
boottime.pdf 70.2 KB December 18 2012 20:10 UTC
boottime.tex 4.9 KB December 18 2012 20:10 UTC
Comments
mgoszcz2 on November 29 2012 22:55 UTC Task Claimed

I would like to work on this task.

Radoslaw Kujawa on November 29 2012 23:57 UTC Task Assigned

This task has been assigned to mgoszcz2. You have 96 hours to complete this task, good luck!

Aleksej Saushev on December 1 2012 11:33 UTC Progress?

Hi!


This task is not easy one and subject to some controversy.


Please, try to reach us on #netbsd-code to discuss progress.


If you're misdirected or stuck, we need to get you on track as soon as possible.

mgoszcz2 on December 3 2012 19:55 UTC Appology

I'm sorry but I over the last few days I didn't had internet acess for long enough to work on the task. Also my attempts to get netBsd working are failing so it's proably better if you reopen the task :(

mgoszcz2 on December 3 2012 19:55 UTC Claim Removed

The claim on this task has been removed, someone else can claim it now.

Ehsan A on December 11 2012 01:09 UTC Task Claimed

I would like to work on this task.

Ehsan A on December 11 2012 01:09 UTC Claim Removed

The claim on this task has been removed, someone else can claim it now.

Michał Zieliński on December 14 2012 19:46 UTC Task Claimed

I would like to work on this task.

Radoslaw Kujawa on December 14 2012 19:49 UTC Task Assigned

This task has been assigned to Michał Zieliński. You have 96 hours to complete this task, good luck!

Aleksej Saushev on December 17 2012 08:59 UTC Progress?

Hi!


You're working on the task for 3 days, by this time you should have elaborated at least your approach to the problem and get some results. Could you submit your preliminary for review?

Michał Zieliński on December 17 2012 12:07 UTC Progress

I have measured startup times on virtual machine. They were very similar, regardless of number of enabled services. 


Today I'm measuring startup on real machine.

Michał Zieliński on December 17 2012 12:25 UTC Approach

I am measuring starup time by simple script which ssh's into target machine, reboots it and waits for reboot to complete. I regard startup as complete when init invokes /etc/rc.local.


I would upload results, but I left them in my home on computer which is turned off.

Michał Zieliński on December 17 2012 20:31 UTC Ready for review

The work on this task is ready to be reviewed.

Aleksej Saushev on December 17 2012 23:24 UTC Review

Hi!


The report as you provide it cannot be taken seriously. It lacks description of method (including illustrations how to set experiment up and how to run it), thus it is hard to reproduce. It lacks description of how much time is spent on each stage. Also, measuring time on virtual hardware makes little sense, unless you gather a lot of samples since dispersion is very high in these cases. If VM is the only "hardware" available to you, how many times have you run your measurements? What is the distribution?


Your report lacks the most important analysis of data acquired. What fraction of time is spent in BIOS? loader? kernel? rc? How much time is spent in sh? Can we draw any conclusions from the report? Does startup mechanism need to be improved or is it impractical?

Aleksej Saushev on December 17 2012 23:24 UTC Task Needs More Work

One of the mentors has sent this task back for more work. Talk to the mentor(s) assigned to this task to satisfy the requirements needed to complete this task, submit your work again and mark the task as complete once you re-submit your work.

Aleksej Saushev on December 17 2012 23:25 UTC Deadline extended

The deadline of the task has been extended with 2 days and 0 hours.

Michał Zieliński on December 18 2012 08:34 UTC Response

I wonder if you are commenting the same PDF file as I have uploaded. I have explicitly written that I used virtual machine and real PC. I have also mentioned that I repeated tests 10 times. I have also written about my approach, but I felt that there is no reason to include 20 line bash script and inform how to connect ethernet cable to computer.


As far as I know NetBSD doesn't have any distributions. At least netbsd.org page doesn't offer anything more than one image for each architecture, format and version.


The second paragraph makes more sense to me - I will get more detailed data and analysis. I will also expand sentence about BIOS, DHCP and kernel taking majority of time, to conclude that rc is good enough.

Radoslaw Kujawa on December 18 2012 11:07 UTC Re: Response

Michal, it's good that you've understood the second paragraph, however about the first paragraph:


Aleksej expressed valid concerns regarding your PDF. The problem is that you didn't use statistical methods (or at least didn't describe it). Do you know anything about statistics? Maybe it's time to learn, as this knowledge is very useful in a task where you have to measure performance of something. The words "samples", "dispersion" and "distribution" were used in statistical sense. Until you understand what they mean then first paragraph won't make much sense to you.


Just measuring average boot time is not enough.

Michał Zieliński on December 18 2012 11:39 UTC Ok

Oh, I needed to switch from IT mode to physics mode. 


I hope that standard deviation from average will be enough.

Aleksej Saushev on December 18 2012 13:23 UTC Re: Response

Hi!


There's fundamental misunderstanding on your side. The inclusion of 20-line bash script and description of test setup is the most important part here. It is the basis of your approach to measurement, it cannot be unimportant


For instance, you could have chosen the most straightforward approach with sitting with stop-watch and waiting for messages to appear on console. Though it sounds stupid (and it is, there're better approaches), it is _perfectly_valid_. Only it requires separate experiments, including ones to measure your latency.


I strongly suggest that you don't throw away unprocessed measurements. Standard deviation makes sense for normal distribution, and it isn't clear yet whether yours is. You don't need excellent knowledge of statistics to draw a picture with dots surrounding some mean value, but you do need to understand some basics.


 


In general, your report should give to another person clear understanding what one should do to repeat your experiments, get similar measurements, and make conclusions similar to what you make.

Michał Zieliński on December 18 2012 19:52 UTC Ready for review

The work on this task is ready to be reviewed.

Radoslaw Kujawa on December 18 2012 20:13 UTC Re: Ready for review

Thanks, good work! I'm closing this task.

Radoslaw Kujawa on December 18 2012 20:14 UTC Task Closed

Congratulations, this task has been completed successfully.

Michał Zieliński on December 18 2012 20:17 UTC Mistake

I have just noticed that I did "*0.5" instead of "**0.5" when calculating std deviation. 


http://user.zielm.com/gci/