Logged-In As
ACCOUNT
Not Logged In
Describe how to backup NetBSD The NetBSD Project
Status: Closed Time to complete: 144 hrs Mentors: blymn, Julian Coleman, Julian Fagir Tags: howto, system, research

Though NetBSD is much like other Unixes in this respect, backup is still something you should consider specially for every operating system.

Which tools are available in the base distribution for backupping, like dump(8) and restore(8)?
Which one suits better, pax(1), dump(8) or even just rsync or other special backup solutions? What are their use cases?
What is a full, a differential, an incremental backup? What is the estimated space usage of them, depending on the backups?
How would you restore your system after a crash, which steps have to be taken to get a working system again?


After reading the resulting article, the reader should be able to decide for a backup scheme and solution and implement it without further research.

Our preferred format for the submission is some form of plain text, as that is most portable and can be read anywhere.  If you want to use a word processor, please export the file as text.  You could also add something like markdown or latex to add structure (these are both plain text based), but that is not a requirement.

Uploaded Work
File name/URL File size Date submitted
netbsd-backup.odt 22.6 KB December 04 2012 12:40 UTC
netbsd-backup1.odt 22.6 KB December 04 2012 12:43 UTC
netbsd-backup1.odt 23.4 KB December 04 2012 12:45 UTC
index.md 6.6 KB January 12 2013 18:06 UTC
index.md 7.0 KB January 12 2013 18:21 UTC
index.md 11.7 KB January 12 2013 21:12 UTC
other.md 15.7 KB January 13 2013 11:14 UTC
other.md 15.9 KB January 13 2013 11:38 UTC
BackupNetBSD.md 15.9 KB January 13 2013 11:43 UTC
BackupNetBSD2.md 14.8 KB January 13 2013 12:42 UTC
BackupNetBSD3.md 15.1 KB January 13 2013 12:50 UTC
BackupNetBSD4.md 15.4 KB January 13 2013 13:01 UTC
BackupNetBSD5.md 20.0 KB January 13 2013 14:45 UTC
BackupNetBSD6.md 21.8 KB January 13 2013 16:46 UTC
BackupNetBSD7.md 24.6 KB January 13 2013 19:42 UTC
BackupNetBSD8.md 30.4 KB January 14 2013 12:02 UTC
BackupNetBSD9.md 31.9 KB January 14 2013 16:33 UTC
Comments
Filip Mostafa on November 29 2012 17:31 UTC Task Claimed

I would like to work on this task.

Aleksej Saushev on November 29 2012 17:52 UTC Task Assigned

This task has been assigned to Filip Mostafa. You have 144 hours to complete this task, good luck!

Aleksej Saushev on December 1 2012 11:27 UTC Progress?

Hi!


You're working on this task for two days. Please, get in touch with us on #netbsd-code so that we could discuss your progress.


If you have problems or questions, we need to get them resolved as soon as possible.

Filip Mostafa on December 4 2012 12:45 UTC :)

The lastone , the biggest as size is the final. Have a nice day! :)

Filip Mostafa on December 4 2012 12:45 UTC Ready for review

The work on this task is ready to be reviewed.

Julian Fagir on December 4 2012 14:23 UTC Looks nice

Ok, this looks nice. It is not exactly what I expected (I expected a generic howto), but it's still nice.


Just a few questions:



  • SFU is ServerForYou? Which operating system is it running?

  • What is Atari, is it the server?

  • Could you give a direct souce for the nixCraft shell script?

Julian Fagir on December 4 2012 14:23 UTC Task Needs More Work

One of the mentors has sent this task back for more work. Talk to the mentor(s) assigned to this task to satisfy the requirements needed to complete this task, submit your work again and mark the task as complete once you re-submit your work.

Filip Mostafa on December 4 2012 15:12 UTC server?

Would you mind giving me the irc server?

Julian Fagir on December 4 2012 15:22 UTC #netbsd-code on freenode

You can reach us on #netbsd-code on freenode. You'll find instructions how to get there here: http://www.komkon2.de/nbwiki/

Radoslaw Kujawa on December 4 2012 16:35 UTC Copypasta

Please don't just copy and paste things from other projects. This "work" can not be accepted, as it seems that you did not do any real research. 

Aleksej Saushev on December 4 2012 16:44 UTC Review

First of all, your work should be more or less self-contained. There should not be simple references other than to official or semi-official documentation. It should be possible to read the text offline.


Second, if you find problems, you should report them in a way that allows reproduction at least.


Third, given that you're working with critical tools (system backup and restore _is_ critical), you should have definitly contacted us when you have found problems.


Fourth, your instructions are hard to reproduce. It is only possible to guess what SFU means when one has prior knowledge or experience with MS Windows XP (prior to that it was called Interix, now it is called differently too, and I don't remember how). One has also to have experience of working with MS server technologies, since SFU was either hard to get or just didn't work on low-end systems. This program is about open-source projects, and you should have followed open-source solutions first of all.


Fifth, it is not clear what your backup script is and what technology it uses. There're at least 3 of them suggested explicitly, and you could have found at least four more open-source solutions.

Filip Mostafa on December 4 2012 18:03 UTC Claim Removed

The claim on this task has been removed, someone else can claim it now.

Puck Meerburg on January 10 2013 18:37 UTC Task Claimed

I would like to work on this task.

Julian Fagir on January 10 2013 18:47 UTC Task Assigned

This task has been assigned to Puck Meerburg. You have 144 hours to complete this task, good luck!

Puck Meerburg on January 12 2013 18:06 UTC Ready for review

The work on this task is ready to be reviewed.

Aleksej Saushev on January 12 2013 18:52 UTC Review

Hi!


1. How long has it been since you used magnetic tape last time? Have you ever used it at all?


I suggest that you remove all references to magnetic tape except, perhaps, explanation why you have to use "-a" and "-f" flags (which seem to deal with overriding obsolete tape defaults).


2. Consider I have chosen differential backups. How do I store them? Do I need to use the same file name? Do I need to use different file name? How do I restore in the case of differential backups?


2a. Same as 2 only applied to incremental backups.


3. You instruct to "newfs". What if I don't want to recreate the whole file system from scratch. Is it possible? If yes, how do I do that?


4. Using backup tools without compression seems to be a waste of space.


4a. And using backup tools without encryption may be dangerously insecure.


5. "pax -r" loses access permissions. If you're restoring your / or /usr, it makes your system unusable.


6. "-e ssh" seems superfluous for rsync, doesn't it? Why not remove it then?


7. I have heard that it is possible to create incremental backups with rsync. Is it right?


8. I have heard that there're other backups solutions: Amanda, Bacula, rdiff-backup, rsnapshot. What about them?


9. I have heard about fss(4). Can it be used for backups?

Aleksej Saushev on January 12 2013 18:52 UTC Task Needs More Work

One of the mentors has sent this task back for more work. Talk to the mentor(s) assigned to this task to satisfy the requirements needed to complete this task, submit your work again and mark the task as complete once you re-submit your work.

Puck Meerburg on January 12 2013 21:12 UTC Ready for review

The work on this task is ready to be reviewed.

Aleksej Saushev on January 12 2013 21:32 UTC Review

This got worse.


The text reads incoherent, it is more like a pile of unconnected thoughts.


I suggest that you rethink the structure to introduce those aspects that are general for several types (e.g. encryption and compression), backup plans, other things, and reuse those parts.


You introduce a concept of backup level, but then revert to "monthly-weekly-daily". What if I need to create full backups weekly? You "monthly-weekly-daily" scheme doesn't generalize to that.


Most issues from previous review are not addressed. I suggest that you go through it again.

Aleksej Saushev on January 12 2013 21:33 UTC Task Needs More Work

One of the mentors has sent this task back for more work. Talk to the mentor(s) assigned to this task to satisfy the requirements needed to complete this task, submit your work again and mark the task as complete once you re-submit your work.

Puck Meerburg on January 13 2013 11:14 UTC Ready for review

The work on this task is ready to be reviewed.

Aleksej Saushev on January 13 2013 12:34 UTC Review (partial)

Hi!


This is nicer. Still...


1. Please, drop all references to magnetic tape unless it is a caution for user not to get bogged down in this obsolete mess!


2. I suggest that you group your examples by backup-restore. That is you write how you backup, and next to it you write how you restore from this backup. It is a bit hard to understand if restore step is separated from backup step. (It is hard to review either.) Ideally, examples should be fully symmetrical: restore examples should restore from backup created by backup state.


 

Aleksej Saushev on January 13 2013 12:36 UTC Task Needs More Work

One of the mentors has sent this task back for more work. Talk to the mentor(s) assigned to this task to satisfy the requirements needed to complete this task, submit your work again and mark the task as complete once you re-submit your work.

Aleksej Saushev on January 13 2013 12:39 UTC Review (partial, part 2)

3. You mention cron setup for rsnapshot but not for other tools. Make it consistent. If you don't have time or don't want to explain it multiple times for readability reasons, write single example and mention how to generalize it.

Puck Meerburg on January 13 2013 12:43 UTC Ready for review

The work on this task is ready to be reviewed.

Aleksej Saushev on January 13 2013 13:26 UTC Review

This is better to read and review.


3. cron issues.


"If you easily forget things".


I suggest that you don't write this. The point of using cron is not about forgetting or not forgetting, it is about automation. It is possible always to remember that you need to run "sleep 36000 && /sbin/dump ...", but it is way better to put the command into crontab and only recall about it when you plan to run something at 4 o'clock "in the morning".


4. Consider I have created /mnt/full.dump (level 0) and /mnt/increment.dump (level 1). How do I restore to the state shortly after increment was created?


4.1. What if some file was removed before increment was created? How does "restore" handle incremental backups?


4.2. Consider some important differential/incremental examples. E.g. backup level sequences 0 1 2 3 4 and 0 1 2 1 2.


5. I suggest that you write parallel examples. If you write example of full backup for dump-restore, write same example for pax. If you mean that you have to maintain timestamp manually for pax, just mention it in text (no need to write backup script). Same about rsnapshot and rdiff-backup.


6. Compression-encryption.


6.1. Introduce optional compression step and mention that encryption is optional (though both are advised).


Since you don't know cryptography, most probably, compression _have_to_ precede encryption.


6.2. Mention which encryption algorythm is used. It is very important.


6.3. Mention how one can encrypt backups in case of rdiff-backup and rsnapshot (e.g., using encrypted file system).

Aleksej Saushev on January 13 2013 13:26 UTC Task Needs More Work

One of the mentors has sent this task back for more work. Talk to the mentor(s) assigned to this task to satisfy the requirements needed to complete this task, submit your work again and mark the task as complete once you re-submit your work.

Puck Meerburg on January 13 2013 14:45 UTC Ready for review

The work on this task is ready to be reviewed.

Puck Meerburg on January 13 2013 14:47 UTC Ready for review

Hey,


I fixed everything you asked, except the compression. This is due to the backup tools having internal compression. At `pax`, I mentioned the flag for compressing. 


 


 

Aleksej Saushev on January 13 2013 15:42 UTC Review

5. Parallel examples (addressed incompletely).


You write:


    dump -0u -f /mnt/backup/monthly.13 /  # This will backup the whole / disk to /mnt/backup/monthly.13
    dump -1u -f /mnt/backup/weekly.13.1 / # This will backup all changes made after the previous example to /mnt/backup/weekly.13.1
    restore -r -f /backup              # Restore a full backup
    restore -r -f /incrementalbackup   # Now restore an incremental backup
and


    pax -w -f /mnt/backup/monthly.13 /                  # This will fully backup the whole / disk to /mnt/backup/monthly.13
    pax -w -T 201301130000 -f /mnt/backup/weekly.13.1 / # This will incrementally/differentially backup the changes made after january 13, 2013 to /mnt/backup/weekly.13.1
[no corresponding examples to restore]


 


I suggest that you devise some more structured backup labels. E.g. "13-3-4" for a 4th backup of level 2 since 3rd backup of level 1 since 13th backup of level 0.


_And_ follow this naming consistently throughout the document.


_And_ give examples in such a way that is uniform. If you did "dump -f /mnt/backup/backup-file" then you should "restore -f /mnt/backup/backup-file". Same for pax and other tools as appropriate.


Of course, you should rather give restoration examples too (e.g. pax).


6. Compression-encryption (addressed incompletely).


6.1. (addressed incompletely) Introduce optional compression step and mention that encryption is optional (though both are advised).


Since you don't know cryptography, most probably, compression _have_to_ precede encryption.


6.1.1. In particular "dump" output can be compressed.


6.2. (not addressed) Mention which encryption algorythm is used. It is very important.


6.3. Perhaps a reference to other documentation on cgd would be nice.


7. When you're talking about backup scheme, make it clear that "1 2 1 2 1 2 ..." is _sample_ scheme used for demonstration only, it is not meant for practical use.


7.1. Since you denote full backup as level 0, it is better if you write it "0 1 2 1 2 ... 0 1 2 1 2 ..." or something like that.


8. Box scheme is good idea, but it is a bit confusing. I think that you should rethink how you mark boxes.


Just an idea, enumerate them in order of creation: 1 (level 0), 2 (level 1), 3 (level 2), 4 (level 1), 5 (level 2), and so on. "Level 2-1" is confusing, I had to read till the end of the document to understand what it means.


9. There's a problem with pax that it doesn't remove files that were removed between backups. I think that you should research using mtree to compensate.


10. rdiff-backup is underdescribed. You should make it clearer what is backup step, what is restore step.


10.1. It isn't clear from your description whether rdiff-backup supports differential and/or incremental backups.


11. It isn't clear from your description how rsnapshot supports differential and/or incremental backups.


12. It isn't clear from your description whether rsync supports differential and/or incremental backups.


13. You should explain whether fss snapshots are persistent or they don't survive reboot.


13.1. Perhaps "/mnt/snapshot" is a better name for mount point, so as to stress that /mnt/snapshot is a snapshot of some file system.


14. When talking about mounting extra file systems for backup, try using mount points inside /mnt. This applies to cgd which you have placed into "/securebackups".

Aleksej Saushev on January 13 2013 15:43 UTC Task Needs More Work

One of the mentors has sent this task back for more work. Talk to the mentor(s) assigned to this task to satisfy the requirements needed to complete this task, submit your work again and mark the task as complete once you re-submit your work.

Puck Meerburg on January 13 2013 16:46 UTC Ready for review

The work on this task is ready to be reviewed.

Aleksej Saushev on January 13 2013 17:34 UTC Review

5. Parallel examples (addressed incompletely).


You write:


    dump -0u -f /mnt/backup/monthly.13 /  # This will backup the whole / disk to /mnt/backup/monthly.13
    dump -1u -f /mnt/backup/weekly.13.1 / # This will backup all changes made after the previous example to /mnt/backup/weekly.13.1


    restore -r -f /mnt/backup/backup.bak              # Restore a full backup
    restore -r -f /mnt/backup/incrementalbackup.bak   # Now restore an incremental backup


I suggest that you devise some more structured backup labels. E.g. "13-3-4" for a 4th backup of level 2 since 3rd backup of level 1 since 13th backup of level 0.


_And_ follow this naming consistently throughout the document.


_And_ give examples in such a way that is uniform. If you did "dump -f /mnt/backup/backup-file" then you should "restore -f /mnt/backup/backup-file". Same for pax and other tools as appropriate.


6. Compression-encryption (addressed incompletely).


6.1.2. "pax" example lacks compression: "pax -w /data | openssl bf -salt | dd of=/backupofdata.pax"


6.3. (not addressed) Perhaps a reference to other documentation on cgd would be nice.


6.4. Blowfish is too weak for modern realities with its less than 512-bit long keys, S-boxes, unclear analysis. Its author recommends using another cipher instead. It isn't good to recommend this cipher. We should propose something better.


9. (not addressed) There's a problem with pax that it doesn't remove files that were removed between backups. I think that you should research using mtree to compensate.


10. (not addressed) rdiff-backup is underdescribed. You should make it clearer what is backup step, what is restore step.


10.1. (addressed incompletely) It isn't clear from your description whether rdiff-backup supports differential and/or incremental backups.


Try giving examples how to organize differential/incremental backups. Preferrably, similar to examples you give for dump and pax (see issue #5 too).


11. (not addressed) It isn't clear from your description how rsnapshot supports differential and/or incremental backups.


See issues no. 5 and 10.1.


12.  (not addressed) It isn't clear from your description whether rsync supports differential and/or incremental backups.


See issues no. 5 and 10.1 and 11.


13.  (not addressed) You should explain whether fss snapshots are persistent or they don't survive reboot.


In particular, you shold tell how to remove old snapshot correctly. You write:


To stop the snapshot, run:

    # cd /
    # fssconfig -u fss0


But what happens to /tmp/snapshotdata?? (BTW, should that file or directory be created before invoking fssconfig?)


13.2. Using /tmp is bad idea for snapshots, it is meant for temporary data. It resides on another file system as well, thus /tmp/snapshotdata is bad idea.


13.3. Inconsistent naming.


You write:


    # mount /dev/fss0 /mnt/snapshot      # Mount the file system

`/mnt` is now a static version of the `/` file system.


This reads inconsistent.


14. When talking about mounting extra file systems for backup, try using mount points inside /mnt. This applies to cgd which you have placed into "/securebackups".


15. pax generates standard "ustar" archives, thus the natural suffix for generated files is ".tar" rather than ".pax".


16. Mention somewhere why we don't discuss tar tool, by the way: it is way too non-standard (and declared obsolete by POSIX/SUS), in particular, syntax varies too much across implementations and you may have problems extracting archives created by GNU tar or Schilly tar.

Aleksej Saushev on January 13 2013 17:43 UTC Task Needs More Work

One of the mentors has sent this task back for more work. Talk to the mentor(s) assigned to this task to satisfy the requirements needed to complete this task, submit your work again and mark the task as complete once you re-submit your work.

Puck Meerburg on January 13 2013 19:43 UTC Ready for review

The work on this task is ready to be reviewed.

Puck Meerburg on January 13 2013 19:45 UTC Ready for review

Hey,


 


I fixed everything, I just have some doubts at the 'underdescribed' rdiff-backup, it is meant to be easy, so I have much less to tell about. 

Aleksej Saushev on January 13 2013 21:52 UTC Review

5.1. (partially addressed) Backup labelling.
I propose that you explain it a bit better, e.g. stress that your level 0 backup is for "monthly.12", and so on.

6. Compression-encryption (addressed incompletely).

6.1.2. "pax" example lacks compression but written as if it does: "pax -w /data | openssl aes-256-cbc -salt | dd of=/mnt/backup/monthly.13.12.gz.enc"

10. rdiff-backup is underdescribed.

10.1. (addressed incompletely) It isn't clear from your description whether rdiff-backup supports differential and/or incremental backups.
Try giving examples how to organize differential/incremental backups. Preferrably, similar to examples you give for dump and pax (see issue #5 too).
In particular, it is not clear how it stores everything under the same name and restores to arbitrary date.

11. (partially addressed) It isn't clear from your description how rsnapshot supports differential and/or incremental backups.
It looks as if it introduces hardlinks into working space, is that true? This may be dangerous.

12.  (not addressed) It isn't clear from your description whether rsync supports differential and/or incremental backups.

13. (partially addressed) You should explain whether fss snapshots are persistent or they don't survive reboot.

16. (partially addressed) tar tool.

You write:

We use `pax` instead of `tar`, because `tar` is less defined

This is wrong. tar is not less defined, it just has less standard features that are useful for backup, has more variations between different versions and more quirks. E.g., you may have problems extracting archives created by GNU tar or Schilly tar. It is declared obsolete by POSIX/SUS in addition.

17. dd vs cat.
You use this pattern frequently: dd if=file | stage2 | stage3 ...
Please, replace it with: "stage2 < file | stage3 ...".
Or use cat instead of dd, at least.

18. mtree specs.
After introduction of mtree specs, you should rearrange your examples to take it into consideration.
If you mean that you should create mtree spec at the backup time, then you should mention it along with the main example.

19. dump -u
If you provide example of backing up separate subdirectories, then this flag is ignored, as you can easily see in your own example.
Perhaps, you should mention it or focus on backing up the whole file systems.

20. Text organization.
In general, it is fine, although it might have improvements.

20.1. Explaining command syntax
I think, that you can safely drop syntax sections. Consider this change.
Since you provide examples, a user can easily refer to documentation. Just reference it to shorten path (reference man page, if there's no man page, then direct to HTML user manual, etc). This can significantly improve your document since by compacting it.

20.2. More parallels, more parallels.
I suggest that you try hard to make all your examples consistent.
If you write example for differential weekly updates for dump, so you should for rsync.
This way you can show similarities and difference in different approaches.

20.3. Stress use cases.
I see that you explain two different use cases for restoring: extracting selected file and reverting to some state in past. Make these use cases clear, distinct, and univeral.

21. Amanda, Bacula.
You have dropped any mention of these tools. This is fundamentally wrong.
Even if you don't expand on their usage, you should definitly mention them.

Aleksej Saushev on January 13 2013 21:52 UTC Task Needs More Work

One of the mentors has sent this task back for more work. Talk to the mentor(s) assigned to this task to satisfy the requirements needed to complete this task, submit your work again and mark the task as complete once you re-submit your work.

Puck Meerburg on January 14 2013 12:02 UTC Ready for review

The work on this task is ready to be reviewed.

Puck Meerburg on January 14 2013 12:06 UTC Ready for review

Hey,


I made the changes you requested (and some more). I hope that this document now meets the requirements for this task.


 


 


:Puck

Aleksej Saushev on January 14 2013 13:56 UTC Review

5.1. (can be improved) Backup labelling.
I propose that you explain it a bit better, e.g. stress that your level 0 backup is for "monthly.12", and so on.

6. Compression-encryption (addressed incompletely).

6.1.2. "pax" example lacks compression but written as if it does: "pax -w /data | openssl aes-256-cbc -salt | dd of=/mnt/backup/monthly.13.12.gz.enc"


6.1.3. Your "pax" examples usually lack compression in general.


6.1.4. ".pax" suffix returned or missed. pax generates "ustar" archives, the natural suffix for them is ".tar".

12.  (not addressed) It isn't clear from your description whether rsync supports differential and/or incremental backups.

13. (can be improved) You should explain whether fss snapshots are persistent or they don't survive reboot. If time remains, try to stress it somehow.

18. mtree specs.
After introduction of mtree specs, you should rearrange your examples to take it into consideration.
If you mean that you should create mtree spec at the backup time, then you should mention it along with the main example.

20. Text organization.
In general, it is fine, although it might have improvements.

20.1. Explaining command syntax
I think, that you can safely drop syntax sections. Consider this change.
Since you provide examples, a user can easily refer to documentation. Just reference it to shorten path (reference man page, if there's no man page, then direct to HTML user manual, etc). This can significantly improve your document since by compacting it.

20.2. More parallels, more parallels.
I suggest that you try hard to make all your examples consistent.
If you write example for differential weekly updates for dump, so you should for rsync.
This way you can show similarities and difference in different approaches.

20.3. Stress use cases.
I see that you explain two different use cases for restoring: extracting selected file and reverting to some state in past. Make these use cases clear, distinct, and univeral.


22. Incremental dump


You write:


`dump` is a built in tool for backing up data based on dump levels, which means it can't make actual incremental backups, only differential backups.


This is not true, you can implement incremental dumps by using this backup plan: 0 1 2 3 4 5 6 7 8 9. Sure, it has the limitation of only 9 increments, but it is usually enough for practical purpose.

Aleksej Saushev on January 14 2013 13:57 UTC Task Needs More Work

One of the mentors has sent this task back for more work. Talk to the mentor(s) assigned to this task to satisfy the requirements needed to complete this task, submit your work again and mark the task as complete once you re-submit your work.

Puck Meerburg on January 14 2013 16:33 UTC Ready for review

The work on this task is ready to be reviewed.

Julian Fagir on January 16 2013 00:22 UTC Task Closed

Congratulations, this task has been completed successfully.