Backups are important.
RAID is not a backup!!!
ZFS provides redundancy features much better than traditional RAID, however like RAID, it is not a backup solution in itself. ZFS does however provide features that make it easy to backup a ZFS dataset. I have been periodically backing up the LFS system I am building to an external hard drive, and thought I should share how those backups are being done.
Current LFS backups.
Current zpools.
I have two zpools on my LFS build host system: livecd gentoo # zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
gentooScratch 3.72G 3.34G 391M - 65% 89% 1.00x ONLINE -
rootPool 119G 21.2G 97.8G - 9% 17% 1.00x ONLINE -
The gentooScratch is a small file backed pool on the gentoo boot thumb drive that contains build-host persistence changes needed across reboots. This includes home directory, and scripts and source needed to perform the LFS bootstrap steps.
The rootPool is a pool backed by a single partition on one of the installed M.2 NVME drive. It contains all the LFS base source downloads, and the LFS target partition $LFS.
Preparing for backup.
Zfs provides features that make for easy backups. This starts with using snapshots. A snapshot records the exact state of the filesystem at the time the snapshot was taken. livecd gentoo # zfs snapshot -r gentooScratch@`date +backup_%Y%m%d_%H%M`
livecd gentoo # zfs snapshot -r rootPool@`date +backup_%Y%m%d_%H%M`
The recursive option on the snapshot command (-r) for the entire pool will create snapshots for every file system in the pool.
I have also mounted an external USB hard drive formatted as ext4 under /mnt/backups and created a new date based directory '2019-01-10' under the backup directory.
Performing the backup.
livecd gentoo # zfs send -R gentooScratch/persistentHomeGentoo@backup_20200111_0116 > /mnt/backups/2019-01-10/persistentHomeGentoo.zfslivecdgentoo #zfs send -R gentooScratch/persistentPortageDistfiles@backup_20200111_0116 > /mnt/backups/2019-01-10/persistentPortageDistfiles.zfslivecdgentoo #zfs send -R gentooScratch/persistentPortageTmp@backup_20200111_0116 > /mnt/backups/2019-01-10/persistentPortageTmp.zfslivecdgentoo #zfs send -R gentooScratch/scripts@backup_20200111_0116 > /mnt/backups/2019-01-10/scripts.zfslivecdgentoo #zfs send -R gentooScratch/sources@backup_20200111_0116 > /mnt/backups/2019-01-10/sources.zfslivecdgentoo #zfs send -R rootPool/root_fs@backup_20200111_0116 > /mnt/backups/2019-01-10/rootPool-root_fs_recursive.zfslivecdgentoo #cd /mnt/backups/2019-01-10/livecd2019-01-10 # ls -lh * -rw-r--r-- 1 root root 3.3G Jan 11 01:24 persistentHomeGentoo.zfs -rw-r--r-- 1 root root 31M Jan 11 01:25 persistentPortageDistfiles.zfs -rw-r--r-- 1 root root 86K Jan 11 01:26 persistentPortageTmp.zfs -rw-r--r-- 1 root root 21G Jan 11 01:32 rootPool-root_fs_recursive.zfs -rw-r--r-- 1 root root 659K Jan 11 01:27 scripts.zfs -rw-r--r-- 1 root root 74M Jan 11 01:28 sources.zfslivecd2019-01-10 #
The 'zfs send' command creates a stream of data that can be used to reconstruct the file system up to the snapshot that was sent. The '-R' option is a recursive option that will include all descendant data sets.
As as an example the 'zfs send -R gentooScratch/sources@backup_20200111_0116 > /mnt/backups/2019-01-10/sources.zfs' command performs a full backup from the creation of the sources file system, and all descendant filesystems, up to the snapshot I made in the preparation step above which got labeled as 'backup_20200111_0116'. This backup should contain the following files systems and mount points:
ZFS File System Mount Point
gentooScratch/sources /gentooScratch/sources
gentooScratch/sources/gentoo /gentooScratch/sources/gentoo
gentooScratch/sources/gentoo/cpuid2cpuflags /gentooScratch/sources/gentoo/cpuid2cpuflags
gentooScratch/sources/lfs_book /gentooScratch/sources/lfs_book
And can be seen in the `ls -lh` command performed in the destination backup directory:
-rw-r--r-- 1 root root 74M Jan 11 01:28 sources.zfsI go on and use the xz compression command so that my backups take less space on my backup drive.
livecd2019-01-10 # xz -v9eT 0 *.zfs persistentHomeGentoo.zfs (1/6) 100 % 2308.9 MiB / 3305.7 MiB = 0.698 21 MiB/s 2:34 persistentPortageDistfiles.zfs (2/6) 100 % 28.0 MiB / 30.0 MiB = 0.933 2.7 MiB/s 0:10 persistentPortageTmp.zfs (3/6) 100 % 7632 B / 85.0 KiB = 0.088 rootPool-root_fs_recursive.zfs (4/6) 100 % 14.6 GiB / 20.5 GiB = 0.709 24 MiB/s 14:48 scripts.zfs (5/6) 100 % 59.6 KiB / 658.3 KiB = 0.090 sources.zfs (6/6) 100 % 5623.1 KiB / 73.9 MiB = 0.074 2.7 MiB/s 0:27livecd2019-01-10 #
A backup is not a backup if it can't be restored!!!
Testing that the backup can be restored.
Preparing a restore location.
The first step needed to test that a backup can be restored is a place to restore the backup to. Normally a backup would be restored to it's original location. However for this we are- Testing the backup
- Original system the backup came from is still functioning with original data.
livecdgentoo #zpool create -o ashift=12 -o comment="Backups Testing Area" -o altroot="/backups" tempTestDataPool /dev/sdglivecdgentoo #zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT gentooScratch 3.72G 3.35G 377M - 65% 90% 1.00x ONLINE - rootPool 119G 21.3G 97.7G - 9% 17% 1.00x ONLINE - tempTestDataPool 2.72T 288K 2.72T - 0% 0% 1.00x ONLINE /backupslivecdgentoo #zpool status pool: gentooScratch state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM gentooScratch ONLINE 0 0 0 /mnt/cdrom/scratch/LFSBootstrap/disk1.img ONLINE 0 0 0 errors: No known data errors pool: rootPool state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM rootPool ONLINE 0 0 0 nvme0n1p6 ONLINE 0 0 0 errors: No known data errors pool: tempTestDataPool state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tempTestDataPool ONLINE 0 0 0 sdg ONLINE 0 0 0 errors: No known data errorslivecdgentoo #
The 'altroot' option is important, since my restore test location is on the same box, and I don't want the restored filesystem to attempt to mount over the original source location. With the altroot value, all mount points of filesystems in this zpool will be mounted relative to that altroot location, instead of the original location.
Now create file system locations that the backups can be restored to:
livecdgentoo #zfs create -o mountpoint=/backupPool tempTestDataPool/backupslivecdgentoo #zfs create tempTestDataPool/backups/scratchlivecdgentoo #zfs create tempTestDataPool/backups/scratch/homelivecdgentoo #zfs create tempTestDataPool/backups/scratch/distlivecdgentoo #zfs create tempTestDataPool/backups/scratch/tmplivecdgentoo #zfs create tempTestDataPool/backups/scratch/scriptslivecdgentoo #zfs create tempTestDataPool/backups/scratch/srclivecdgentoo #zfs create tempTestDataPool/backups/rootlivecdgentoo #zfs create tempTestDataPool/backups/root/all_root_fs_backuplivecdmount -agentoo #zfslivecd[Note no file systems under the specified mount point of /backupPool]gentoo #ls /backupPool/livecdgentoo #ls /backups/ backupPool tempTestDataPool [Note the actual filesystems get mounted under the '/backups/' altroot]livecdgentoo #
Performing a restore.
Now I will use the 'zfs receive' command to restore the data.livecdgentoo #ls /mnt/backups/2019-01-10/ persistentHomeGentoo.zfs.xz persistentPortageTmp.zfs.xz scripts.zfs.xz persistentPortageDistfiles.zfs.xz rootPool-root_fs_recursive.zfs.xz sources.zfs.xzlivecdgentoo #xzcat /mnt/backups/2019-01-10/persistentHomeGentoo.zfs.xz | zfs receive -F tempTestDataPool/backups/scratch/homelivecdgentoo #xzcat /mnt/backups/2019-01-10/persistentPortageTmp.zfs.xz | zfs receive -F tempTestDataPool/backups/scratch/tmplivecdgentoo #xzcat /mnt/backups/2019-01-10/scripts.zfs.xz | zfs receive -F tempTestDataPool/backups/scratch/scriptslivecdgentoo # xzcat /mnt/backups/2019-01-10/persistentPortageDistfiles.zfs.xz | zfs receive -F tempTestDataPool/backups/scratch/distlivecdgentoo #xzcat /mnt/backups/2019-01-10/rootPool-root_fs_recursive.zfs.xz | zfs receive -F tempTestDataPool/backups/root/all_root_fs_backuplivecdgentoo #xzcat /mnt/backups/2019-01-10/sources.zfs.xz | zfs receive -F tempTestDataPool/backups/scratch/srclivecdgentoo # ls /backups backupPool home tempTestDataPool tmp usr/backupPool/root/all_root_fs_backup/ esp home opt sources tmp usrlivecdgentoo # ls /backupslivecdgentoo #
For each of the 5 file systems I backed up above, I xzcat (to decompress the backup file) and pipe the data stream into the 'zfs receive' command. The '-F' option forces the restore, overwriting any existing data with the version from the backup. I also specify the zpool and zfs filesystem location I want to restore each backup to. I did a quick ls to see the restored locations, and the recieve commands completed without any errors, so I can be decently confident that the backup was good.
Validating the restore.
Even though I am confident that the backup was good, I will go through the procedure of validating the files to ensure that my backup method is working.The first backup to verify is the persistentHomeGentoo backup. Performing my zfs list, looking for home I see the following:
livecd gentoo # zfs list | grep -i home
gentooScratch/persistentHomeGentoo 3.17G 257M 714M /home/gentoo
rootPool/root_fs/home 3.95G 94.0G 96K /rootPool/root_fs/home
rootPool/root_fs/home/gentooExtra 3.95G 94.0G 3.95G /home/gentoo/extraSpace
tempTestDataPool/backups/root/all_root_fs_backup/home 3.95G 2.61T 96K /backups/backupPool/root/all_root_fs_backup/home
tempTestDataPool/backups/root/all_root_fs_backup/home/gentooExtra 3.95G 2.61T 3.95G /backups/home/gentoo/extraSpace
tempTestDataPool/backups/scratch/home 3.24G 2.61T 737M /backups/home/gentoo
livecd gentoo #
Performing a diff between '/home/gentoo' and '/backups/home/gentoo' will also validate the 'rootPool/root_fs/home/gentooExtra' and 'tempTestDataPool/backups/root/all_root_fs_backup/home/gentooExtra'. Another problem I will find for the home directory comparison will be any changes since the backup, so what I really want to do is compare it against the snapshot the backup was made from. Running the following diff command will show me any differences between the backup, and the snapshot:
livecdgentoo #diff -rq /home/gentoo/.zfs/snapshot/backup_20200111_0116/ /backups/home/gentoo diff: /home/gentoo/.zfs/snapshot/backup_20200111_0116/.kde4/cache-livecd: No such file or directory diff: /backups/home/gentoo/.kde4/cache-livecd: No such file or directory diff: /home/gentoo/.zfs/snapshot/backup_20200111_0116/.kde4/socket-livecd: No such file or directory diff: /backups/home/gentoo/.kde4/socket-livecd: No such file or directory diff: /home/gentoo/.zfs/snapshot/backup_20200111_0116/.kde4/tmp-livecd: No such file or directory diff: /backups/home/gentoo/.kde4/tmp-livecd: No such file or directory diff: /home/gentoo/.zfs/snapshot/backup_20200111_0116/.mozilla/firefox/7040su8h.default-release/lock: No such file or directory diff: /backups/home/gentoo/.mozilla/firefox/7040su8h.default-release/lock: No such file or directory diff: /home/gentoo/.zfs/snapshot/backup_20200111_0116/.mozilla/firefox/hmud4qrk.default-1558998297270/lock: No such file or directory diff: /backups/home/gentoo/.mozilla/firefox/hmud4qrk.default-1558998297270/lock: No such file or directory livecd gentoo #
There appears to be some differences, however looking closer, these are 'diff' errors, not actual differences. For each file, it complains about 'No such file or directory' for both locations. Examining the missing files/directories, it turns out that the directories/files are symbolic links to elsewhere in the system for temporary files that have since been cleaned up. I would consider this to be a validated backup.
The next backup to validate is the persistentPortageTmp, using the command:
livecd gentoo # diff -qr /tmp/portage/.zfs/snapshot/backup_20200111_0116/ /backups/tmp/portage/livecdgentoo #
This shows no difference, so another validated backup.
The next backup to validate is the scripts backup. I have not made any changes to the scripts folder since the backup, so no need to compare against a specific snapshot
This shows no difference, so another validated backup.livecd gentoo # diff -qr /gentooScratch/scripts/ /backups/backupPool/scratch/scripts/livecdgentoo #
The next backup to validate is the persistentPortageDistfiles. Once again I have made no changes, so I can just compare the directories instead of a snapshot.
This shows no difference, so another validated backup.livecd gentoo # diff -qr /usr/portage/distfiles/ /backups/usr/portage/distfiles/livecdgentoo #
The next backup to validate is the sources backup. I have not made any changes to the sources folder since the backup, so no need to compare against a specific snapshot
This shows no difference, so another validated backup.livecd gentoo # diff -qr /gentooScratch/sources/ /backups/backupPool/scratch/src/livecdgentoo #
The final backup to validate is the full recursive $LFS root location using
livecd gentoo # diff -qr /rootPool/root_fs/ /backups/backupPool/root/all_root_fs_backup/
diff: /rootPool/root_fs/sources/bison/bison/build-aux/move-if-change: No such file or directory
diff: /backups/backupPool/root/all_root_fs_backup/sources/bison/bison/build-aux/move-if-change: No such file or directory
diff: /rootPool/root_fs/sources/bison/bison/data/m4sugar/foreach.m4: No such file or directory
diff: /backups/backupPool/root/all_root_fs_backup/sources/bison/bison/data/m4sugar/foreach.m4: No such file or directory
diff: /rootPool/root_fs/sources/bison/bison/data/m4sugar/m4sugar.m4: No such file or directory
diff: /backups/backupPool/root/all_root_fs_backup/sources/bison/bison/data/m4sugar/m4sugar.m4: No such file or directory
diff: /rootPool/root_fs/sources/bison/bison/m4/m4.m4: No such file or directory
diff: /backups/backupPool/root/all_root_fs_backup/sources/bison/bison/m4/m4.m4: No such file or directory
diff: /rootPool/root_fs/sources/kmod/kmod/testsuite/rootfs-pristine/test-loaded/sys/module/btusb/drivers/usb:btusb: No such file or directory
diff: /backups/backupPool/root/all_root_fs_backup/sources/kmod/kmod/testsuite/rootfs-pristine/test-loaded/sys/module/btusb/drivers/usb:btusb: No such file or directory
diff: /rootPool/root_fs/sources/m4/m4/COPYING: No such file or directory
diff: /backups/backupPool/root/all_root_fs_backup/sources/m4/m4/COPYING: No such file or directory
diff: /rootPool/root_fs/sources/m4/m4/INSTALL: No such file or directory
diff: /backups/backupPool/root/all_root_fs_backup/sources/m4/m4/INSTALL: No such file or directory
diff: /rootPool/root_fs/sources/m4/m4/build-aux/compile: No such file or directory
diff: /backups/backupPool/root/all_root_fs_backup/sources/m4/m4/build-aux/compile: No such file or directory
diff: /rootPool/root_fs/sources/m4/m4/build-aux/config.guess: No such file or directory
diff: /backups/backupPool/root/all_root_fs_backup/sources/m4/m4/build-aux/config.guess: No such file or directory
diff: /rootPool/root_fs/sources/m4/m4/build-aux/config.sub: No such file or directory
diff: /backups/backupPool/root/all_root_fs_backup/sources/m4/m4/build-aux/config.sub: No such file or directory
diff: /rootPool/root_fs/sources/m4/m4/build-aux/depcomp: No such file or directory
diff: /backups/backupPool/root/all_root_fs_backup/sources/m4/m4/build-aux/depcomp: No such file or directory
diff: /rootPool/root_fs/sources/m4/m4/build-aux/install-sh: No such file or directory
diff: /backups/backupPool/root/all_root_fs_backup/sources/m4/m4/build-aux/install-sh: No such file or directory
diff: /rootPool/root_fs/sources/m4/m4/build-aux/mdate-sh: No such file or directory
diff: /backups/backupPool/root/all_root_fs_backup/sources/m4/m4/build-aux/mdate-sh: No such file or directory
diff: /rootPool/root_fs/sources/m4/m4/build-aux/texinfo.tex: No such file or directory
diff: /backups/backupPool/root/all_root_fs_backup/sources/m4/m4/build-aux/texinfo.tex: No such file or directory
diff: /rootPool/root_fs/sources/texinfo/texinfo/js/build-aux/texinfo.tex: No such file or directory
diff: /backups/backupPool/root/all_root_fs_backup/sources/texinfo/texinfo/js/build-aux/texinfo.tex: No such file or directory
livecd gentoo #Once again, similar to the gentoo's host home direcorty we have some diff errors, but no differences. I took a look at some of the 'missing' files, and once again they are symbolic links, some of which are absolute and expecting the 'chroot' environment of using '$LFS'. Since every 'missing' file is the same between the two datasets, this backup is also 'valid'.
No comments:
Post a Comment