Full Disk Encryption (with ZFS root) for FreeBSD 9.x

To follow on from my post about full disk encryption (well almost), this is how to do the same but with a ZFS filesystem.  Like the other post, your /boot folder (which contains your kernel and modules) will not be encrypted, but the rest of your filesystem will be.

One disadvantage of this method is that you have to enter a passphrase for EACH disk in your ZFS system each boot.  Encryption inside ZFS will appear at some point – but until then this will suffice !

Boot from any FreeBSD 9 install medium (except bootonly), and choose Live CD at the install menu.

For the purposes of this article, I will assume that you’re using 4 disks (da0, da1, da2, da3), a 10GB /boot (this will be mirrored on each of the 4 disks), and the remaining space as a raidz1 (roughly similar to RAID5) ZFS filesystem.  The contents will be encrypted with 256-bit AES-XTS encryption with a 4 kb random data partial key and a secondary passphrase (required to type on each boot).

If your CPU supports the AESNI flag, the crypto(4) framework will utilise this too.

First we need to remove any existing GPT or MBR partition tables on each of the disks (ignore any ‘invalid argument’ messages):

gpart destroy -F da0
gpart destroy -F da1
gpart destroy -F da2
gpart destroy -F da3

Now we need to create a new GPT partition table on each disk:

gpart create -s gpt da0
gpart create -s gpt da1
gpart create -s gpt da2
gpart create -s gpt da3

We will now create a 64kb boot partition (this contains the boot loader only, so is safe and required to be unencrypted):

gpart add -s 128 -t freebsd-boot da0
gpart add -s 128 -t freebsd-boot da1
gpart add -s 128 -t freebsd-boot da2
gpart add -s 128 -t freebsd-boot da3

Next, we will create the /boot partition – you can adjust the sizes here if you need, but i’d suggest not shrinking it too much or you’ll get into problems when doing OS upgrades later… Note: this is mirrored not striped across the disks for maximum resilience – so will use 10GB on each disk for 10GB total usable space.

gpart add -s 10G -t freebsd-zfs da0
gpart add -s 10G -t freebsd-zfs da1
gpart add -s 10G -t freebsd-zfs da2
gpart add -s 10G -t freebsd-zfs da3

Finally, we will assign the remaining space on each disk to the root ZFS partition.  This will be encrypted before we build ZFS on top of it.

gpart add -t freebsd-zfs da0
gpart add -t freebsd-zfs da1
gpart add -t freebsd-zfs da2
gpart add -t freebsd-zfs da3

Now that we’ve created daXp1 (bootloader), daXp2 (/boot partition), daXp3 (root partition) – we need to write the boot loader code to each disk:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da2
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da3

Ok, next we will build a ramdisk to mount on /boot/zfs – this helps us mount things temporarily…

mdconfig -a -t malloc -s 128m -u 2
newfs -O2 /dev/md2
mount /dev/md2 /boot/zfs

Now we will load the modules required for ZFS and encryption:

kldload opensolaris
kldload zfs
kldload geom_eli

Next, we build a ZFS mirror for the /boot partition and mount it temporarily (to house the encryption key) – ignore any mention of unable to mount:

zpool create bootdir mirror /dev/da0p2 /dev/da1p2 /dev/da2p2 /dev/da3p2
zpool set bootfs=bootdir bootdir
mkdir /boot/zfs/bootdir
zfs set mountpoint=/boot/zfs/bootdir bootdir
zfs mount bootdir

Now we generate a random 4kb encryption key that will form (along with passphrase) the encryption key for the disk:

dd if=/dev/random of=/boot/zfs/bootdir/encryption.key bs=4096 count=1

We have everything we need to start encrypting the disks now… Enter your passphrase twice for each init phase and once again for each attach phase below:

geli init -b -B /boot/zfs/bootdir/da0p3.eli -e AES-XTS -K /boot/zfs/bootdir/encryption.key -l 256 -s 4096 /dev/da0p3
geli init -b -B /boot/zfs/bootdir/da1p3.eli -e AES-XTS -K /boot/zfs/bootdir/encryption.key -l 256 -s 4096 /dev/da1p3
geli init -b -B /boot/zfs/bootdir/da2p3.eli -e AES-XTS -K /boot/zfs/bootdir/encryption.key -l 256 -s 4096 /dev/da2p3
geli init -b -B /boot/zfs/bootdir/da3p3.eli -e AES-XTS -K /boot/zfs/bootdir/encryption.key -l 256 -s 4096 /dev/da3p3
geli attach -k /boot/zfs/bootdir/encryption.key /dev/da0p3
geli attach -k /boot/zfs/bootdir/encryption.key /dev/da1p3
geli attach -k /boot/zfs/bootdir/encryption.key /dev/da2p3
geli attach -k /boot/zfs/bootdir/encryption.key /dev/da3p3

Now that we have encrypted and mounted the partitions, we can build a ZFS root filesystem on top of it like so:

zpool create zroot raidz1 /dev/da0p3.eli /dev/da1p3.eli /dev/da2p3.eli /dev/da3p3.eli
zfs set mountpoint=/boot/zfs/zroot zroot
zfs mount zroot
zfs unmount bootdir
mkdir /boot/zfs/zroot/bootdir
zfs set mountpoint=/boot/zfs/zroot/bootdir bootdir
zfs mount bootdir

Note we unmounted the old boot mirror and re-mounted it within the root filesystem.  This will be used later to copy the kernel and modules into.

Ok, now we create all our ZFS mounts with various options as follows:

zfs set checksum=fletcher4 zroot
zfs create -o compression=on -o exec=on -o setuid=off zroot/tmp
chmod 1777 /boot/zfs/zroot/tmp
zfs create zroot/usr
zfs create zroot/usr/home
cd /boot/zfs/zroot; ln -s /usr/home home
zfs create -o compression=lzjb -o setuid=off zroot/usr/ports
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/distfiles
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/packages
zfs create zroot/var
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/crash
zfs create -o exec=off -o setuid=off zroot/var/db
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/db/pkg
zfs create -o exec=off -o setuid=off zroot/var/empty
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log
zfs create -o compression=gzip -o exec=off -o setuid=off zroot/var/mail
zfs create -o exec=off -o setuid=off zroot/var/run
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/tmp
chmod 1777 /boot/zfs/zroot/var/tmp

Now we’re ready to install FreeBSD onto the new ZFS partitions.  We’re going to install the base, all sources and a generic kernel – this takes some time so please be patient…

cd /boot/zfs/zroot
unxz -c /usr/freebsd-dist/base.txz | tar xpf –
unxz -c /usr/freebsd-dist/kernel.txz | tar xpf –
unxz -c /usr/freebsd-dist/src.txz | tar xpf –

Now we can set /var/empty to readonly:

zfs set readonly=on zroot/var/empty

And now we’re ready to chroot into the installed system to setup the configuration:

chroot /boot/zfs/zroot

Now that the base system and kernel are installed, we can move our /boot folder to it’s final place on the ZFS unencrypted mirror and do a little housekeeping:

cd /
mv boot bootdir/
ln -fs bootdir/boot
mv bootdir/encryption.key bootdir/boot/
mv bootdir/*.eli bootdir/boot/

We need to setup an initial /etc/rc.conf which will mount all ZFS filesystems on boot:

echo ‘zfs_enable=”YES”‘ > /etc/rc.conf
touch /etc/fstab

And an initial /boot/loader.conf that will load ZFS, encryption and settings for encrypted disks on boot:

echo ‘vfs.zfs.prefetch_disable=”1″‘ > /boot/loader.conf
echo ‘vfs.root.mountfrom=”zfs:zroot”‘ >> /boot/loader.conf
echo ‘zfs_load=”YES”‘ >> /boot/loader.conf
echo ‘aesni_load=”YES”‘ >> /boot/loader.conf
echo ‘geom_eli_load=”YES”‘ >> /boot/loader.conf
echo ‘geli_da0p3_keyfile0_load=”YES”‘ >> /boot/loader.conf
echo ‘geli_da0p3_keyfile0_type=”da0p3:geli_keyfile0″‘ >> /boot/loader.conf
echo ‘geli_da0p3_keyfile0_name=”/boot/encryption.key”‘ >> /boot/loader.conf
echo ‘geli_da1p3_keyfile0_load=”YES”‘ >> /boot/loader.conf
echo ‘geli_da1p3_keyfile0_type=”da1p3:geli_keyfile0″‘ >> /boot/loader.conf
echo ‘geli_da1p3_keyfile0_name=”/boot/encryption.key”‘ >> /boot/loader.conf
echo ‘geli_da2p3_keyfile0_load=”YES”‘ >> /boot/loader.conf
echo ‘geli_da2p3_keyfile0_type=”da2p3:geli_keyfile0″‘ >> /boot/loader.conf
echo ‘geli_da2p3_keyfile0_name=”/boot/encryption.key”‘ >> /boot/loader.conf
echo ‘geli_da3p3_keyfile0_load=”YES”‘ >> /boot/loader.conf
echo ‘geli_da3p3_keyfile0_type=”da3p3:geli_keyfile0″‘ >> /boot/loader.conf
echo ‘geli_da3p3_keyfile0_name=”/boot/encryption.key”‘ >> /boot/loader.conf

The above settings tell the OS which encryption keyfile to use for each disk partition.

Now you can set your root password:

passwd root

And configure your timezone:

tzsetup

And setup a dummy /etc/mail/aliases file to prevent sendmail warnings:

cd /etc/mail
make aliases

Now you can configure any additional settings you require (such as adding new users, configuring networking or setting sshd to run on boot) – when you’re done, we need to exit the chroot:

exit

Now, we need to make sure the bootloader can read our ZFS pool cache (or it wont mount our ZFS disks on boot):

cd /boot/zfs
cp /boot/zfs/zpool.cache /boot/zfs/zroot/boot/zfs/zpool.cache

Finally, we need to unmount all the ZFS filesystems and configure their final mountpoints…

zfs unmount -a
zfs set mountpoint=legacy zroot
zfs set mountpoint=/tmp zroot/tmp
zfs set mountpoint=/usr zroot/usr
zfs set mountpoint=/var zroot/var
zfs set mountpoint=/bootdir bootdir

Now we can ‘reboot’ and remove the media while the computer reboots.  Do this as soon as you can.

The computer should reboot into a ZFS-based filesystem, booted from a software RAID array on fully protected disks with all but /boot partition encrypted.  Note: it will ask you to enter a passphrase for each disk parition used above (4 times) – you should take care to enter the correct passwords as it will treat any passwords missed as a failed disk (you get 3 attempts at each password)

Once it’s booted, you can login and run sysinstall to configure other options like networking and startup programs (like SSH!)

The only point to note is that when you do an OS upgrade, during the “mergemaster” stage, it will complain that /boot is a symlink not a directory.  Simply tell it to ignore/do nothing and it will install the files as normal.

Enjoy!

106 thoughts on “Full Disk Encryption (with ZFS root) for FreeBSD 9.x

  1. dan Post author

    The only thing I was concerned with was doing a ‘receive’ on the main device, but that article seems to suggest it works fine. In which case, you can restore using zfs receive.
    (remember that when you do a zfs send from the encrypted zfs, the dumpfile it generates is not encrypted! unless the destination you are sending it to is encrypted of course)

    For you the only difference from that article is that you’re not wrapping ssh around your commands as the drive is local.

    Reply
  2. BSDGuy

    If this works its perfect for using it in a DR scenario (server failure). I know you can’t do a single file restore using a zfs send file but thats not a problem for me. Having a full restore plan is in case of a server failure 😉

    I did wonder about the encryption side of it for the dumpfile and came to the same conclusion as you in my head! Files not encrypted but the contents will be if restored to encrypted drives/partitions.

    SSH wouldn’t come into it at all for me. All I would be doing (on the target server) is mounting an external USB drive with the zfs send dumpfile on it and then doing a zfs receive to restore the root system (and the bootdir).

    I will definitely give this a try on another weekend. I like the fact that you can also use a smaller drive on the target server (so long as theres enough space to hold the data in the dumpfile).

    If it all works…very cool 😉

    Reply
    1. dan Post author

      As you’re building a ZFS pool on top of encrypted disks, you should follow the following guide to setup a swapfile rather than swap device. As it’s located on the ZFS pool, it is covered by the same encryption.

      Reply
  3. Martin

    Hi, I’m missing the point of encrypting your data on daXp3 partitions since the decryption key /boot/encryption.key is located on every daXp2 partition which is 1. unencrypted, and 2. on the same drive.

    A better idea in my opinion is to decrypt the root partition by manually entering the password and place the keyfiles to the encrypted partitions to root partition. These keyfiles can be picked up automatically during boot by /etc/rc.d/geli by setting geli_da0p3_keyfile0_flags=”-p -k /root/encryption.key” in /etc/rc.conf .

    Reply
    1. dan Post author

      The keyfiles become part of your passphrase (you still need to enter part of it) – it’s a way of introducing some binary data into the passphrase.
      The combination of the keyfiles and your typed passphrase unlock the encryption key and decrypt the drive contents.

      Reply
    2. dan Post author

      The more ideal way would be for the keyfiles to be stored on USB medium so none of the passphrase is located on the disk – but there’s no way to do this on FreeBSD that i’m aware of yet.
      The method in this article is as secure as having just a passphrase, but also allows binary data within the passphrase so makes brute forcing require a larger dictionary.

      Reply
  4. Martin

    As long as you passphrase is “complex” (whatever it means) it is completely OK for it to consist only of alphanumeric character without any binary data. Key derivation function (GELI used PBDKF2) takes care of creating a suitable binary key for the encryption algorithm.

    That said, using /boot/encryption.key placed on an unencrypted partition doesn’t add any security at all in this case. Anyway, thank you for the write up!

    Reply
  5. BSDGuy

    Dan said: The more ideal way would be for the keyfiles to be stored on USB medium so none of the passphrase is located on the disk – but there’s no way to do this on FreeBSD that i’m aware of yet.

    I thought you could have /boot on an unencrypted USB drive with your encryption keys? So to boot the machine and “unlock” or decrypt the disks you need the USB drive inserted in the machine so that this can happen. Afterwards I thought you could unmount the USB key and take it with you for maximum security (that way you know no one can have access to your keys AND you know no one can modify files on the /boot).

    Is this possible?

    Reply
    1. dan Post author

      it is possible. you can create a bootable USB drive which then mounts an encrypted system.
      The downside is that you would have to manually duplicate any changes to /boot onto the penstick (if/when you made them)

      I have done this before and it worked fine.
      If there is sufficient interest, I don’t mind doing a blog post about how to do it?

      Reply
  6. BSDGuy

    I would be VERY interested in an article about this!

    I am interested to find out:

    1) How to setup an encrypted ZFS root system with /boot on a USB drive with the encryption keys (that can be removed after booting up)

    2) Knowing when I can unmount and remove the USB key with /boot

    3) When/how to duplicate changes to /boot when/if you make them

    4) Making sure you have a backup USB key incase the first one fails

    5) What (if anything) needs to be done when upgrading the OS with a setup like this (I assume you need the USB drive plugged in and mounted before upgrading FreeBSD?)

    6) Knowing when you need the USB key inserted

    I look forward to another great article on your blog 😉

    Reply
  7. BSDGuy

    The other thing I have been wondering, is it possible to put all of the above commands of this guide into a script and run it while in the Live CD mode?

    Typing all these commands everytime when setting up an encrypted ZFS root is prone to error (fat fingers!) and takes forever! 😉

    Can you do some or all of these commands via a script off of a USB drive maybe?

    Reply
    1. dan Post author

      you could put them in a script on the usb install media sure… the only thing you’ll come across is different disk names (da0,ad4,mfid0 etc) and potentially different sizes (or raid types)…
      If all your installs are identical then no reason not to put the script on it though.
      Just mount the usb drive on any working freebsd machine and copy the script to it 🙂

      Reply
  8. BSDGuy

    I’ve tried copying the steps I used in your guide into a script file but for some reason I get errors when running the script. Is there anything special one needs to do when putting so many steps into a single script? I’m booting FreeBSD off a USB image and the script is on the same USB drive.

    Reply
  9. dan Post author

    You should be able to put as many instructions into a script as you want.
    Hard to comment on any errors without knowing what they are however.

    A common one is that paths may be different and you may need to specify the full path to the binaries in the script (e.g. /sbin/zpool and /sbin/zfs)

    Reply
  10. BSDGuy

    Since I was having no joy with my script I ended up trying to use this script as a template:

    https://calomel.org/zfs_freebsd_root_install.html

    I’ve been trying to adapt your steps to this script (which uses gnop to allign 4k drives) but there is no geli used in this example which makes it really tricky to adapt.

    What I’m ultimately trying to do, and I’m having a tough time finding the answers to, is to setup a server as in your guide on this page but to allign the disks for 4k drives…and put /boot on a USB key…and automate it all in a script (phew!).

    I’ve spent countless hours looking for the right guide for this but there doesn’t seem to be one. Theres many guides available that do one or the other but none that do ALL the items I’m trying to achieve. I’ve tried my best to adapt various guides into one new one unsuccessfully. Any ideas? 😉 I think I’m really pushing my FreeBSD skills here!

    Reply
  11. BSDGuy

    I have put together a small example of how to create a /boot on a USB key using your guide for the encrypted zroot. Would you mind having a look at it? I haven’t tried the steps yet but just wanted to make sure I was on the right track!

    Reply
  12. bobaroo

    I’m really interested in an article too that explains how to put /boot or the zfs pool bootdir on a bootable usb stick with the encryption keys.

    Reply
    1. dan Post author

      I’m just without internet at the moment due to a home move – but once i’m all settled, i’ll do an article on zfs with usb boot for encryption.

      Reply
  13. sukosevato

    Very nice and well written tutorial. Worked perfectly (when I did it for the 3rd time and didn’t make a type in the boot loader or something similar :p). Thank you for writing it.

    I’d love to see a similar tutorial for having your /boot on a USB stick as well.

    Reply
  14. whisper

    Hi Dan – I was looking for your article on encrypted ZFS root with a bootable USB bootdir. Do you have the link for this? Great site!

    Reply
  15. David Blewett

    After doing some research, it seems that the AES-XTS mode is not recommended for larger drives (I’ve read anything from over 500GB to over 1.5TB; google for “nist seagate comments xts”). I would recommend AES-CBC if you are using modern sized drives (>1TB).

    Wouldn’t it also be better to create random key files for each disk, instead of re-using the same chunk ( -K /boot/zfs/bootdir/encryption.key ). I was thinking of using the gpart rawuuid to save a key for each disk.

    Reply
  16. Oliver Briscbois

    I followed your guide. So far so good.

    At first boot, after booting and entering passwrd… hard drive still churning after 14 hours.

    Should first boot take so long?

    Reply
    1. dan Post author

      No, definitely not!
      It should take a few seconds at most before continuing to boot as normal.
      You might want to check your disk for errors with a 3rd party tool.

      Reply
  17. Stephen Warren

    Can’t you mount the boot partition directly on /boot rather than /bootdir? That would eliminate the need for the /boot -> /bootdir symlink…

    Reply
    1. dan Post author

      Unfortunately not… you boot from that partition when booting, and the OS expects the path to be “/boot/loader” within it.
      If you mounted it directly at /boot then the path would be “/loader” and the OS would not boot.
      So to get around this, we mount it in a different folder and then it contains a ‘boot’ folder – but this requires that we symlink it for normal operation after mounting the decrypted ZFS root.

      Reply
  18. keen

    Hi Dan

    I managed to get my test server to boot using a ZFS USB stick for the bootdir and this seems to be working great! My zroot is now fully encrypted.

    Couple questions:

    1) When removing the USB key I am running “zpool export bootdir” (and then unplugging the USB stick), is this the correct way of doing it?

    2) I would like to clone or create a second identical bootable USB stick as a backup, what is the best way to do this?

    3) When upgrading FreeBSD, is anything different with having an external USB key with /boot on it?

    Thanks!!

    Reply
  19. Beastie

    Hi Dan

    Can you change:
    zfs set mountpoint=legacy zroot

    to

    zfs set mountpoint=/ zroot

    Will this work?

    I am having issues with legacy mode as I can’t do a ZFS send/recv of my root ZFS pool to my backup pool and have it mounted. Have you come across this issue?

    Is it better to have a root ZFS pool set with NON legacy mode? Can this work?

    Can’t wait to see some new guides! 😉

    Reply
    1. dan Post author

      I’ve never tried setting it to “/” instead of legacy, I believe it has to be set to legacy to work as a root filesystem.
      I’ve never had a problem doing a “zfs send” and receive of a legacy root though.

      Reply
  20. Beastie

    You can boot with a / instead of legacy…I tried it and it works.

    I wanted to use / instead of legacy so that I could use ZFS to mount eeverything.

    FYI: Doing a zfs send/recv of your root pool to a backup pool mounts the backup pool OVER the live pool! The workaround is to mount your backup pool with an alternative mountpoint. This works great but was painful when I didn’t know this!

    Any new guides planned for 2014? 😉

    Reply
  21. NewKid

    Hi Dan

    In one of the comments you mentioned that labels don’t work with geli. I read somewhere that GPT labels (using gpart) DO work with geli…is this correct?

    I don’t think glabels work with geli though.

    Is there anyway you can label your disks with FreeBSD/GPT/ZFS/geli??

    Reply
    1. dan Post author

      I’m not sure if they work to boot from, as it would depend at which stage geom_label would come along and generate the necessary /dev/ entries.
      I’m not aware of any way to label them 🙁

      Reply
  22. ejr

    I followed this guide and the FreeBSD install I’m ending up with has an empty /bootdir folder. That seems odd to me. Somehow I’m still able to boot which would require this folder and the encryption key inside of it. I wonder where I went wrong…

    Reply
    1. dan Post author

      either the original mounting of bootdir or the later set mountpoint of it i’d imagine.
      With FreeBSD 10, you can install using encryption and ZFS from the installer now too.

      Reply
  23. Xylex Rayne

    Hello,
    I am very experienced ZFS enthusiast and I found this guide super helpful and I built a few NAS systems based on these instructions. I have been running my system on this and have written myself a nice GUI for managing the encrypted volume within KDE.

    I would like to point and issue out that I noticed.
    When you create the gpt partitions, the main zpool partitions are flagged freebsd-zfs. I believe that the ZFS metadata is NOT being encrypted in this case.
    I had to move a large 12 Disk volume from a build to another existing system, that also had zfs and geom_eli. Upon adding those drives to that environment, it appears that ZFS attempted to make the encrypted partitions part of the volume. Once the disks were decrypted, not all of the disks were readable and the pool appeared to be corrupted.

    Reply
    1. dan Post author

      So long as we always work on the “.eli” suffix devices, the metadata should also be encrypted at the end of *that* device (which is the size of the actual gpt partition minus the geom_eli metadata at the end of the partition).
      At the end of the non-.eli suffix device (i.e. the gpt partition) will be the geom_eli metadata only.

      The freebsd-zfs gpt partition type is purely descriptive (except for the boot pool, which must be freebsd-zfs as the boot loader looks for the first freebsd-zfs partition on the disk to boot from)

      zfs shouldn’t attempt to use the partitions when not decrypted, as they shouldn’t contain any metadata.
      The only hint at metadata for the partitions is in the zpool.cache file – but it doesn’t contain any sensitive data (just pool names and device names really) and must remain unencrypted.
      zfs should still be smart enough to know there’s no zpool metadata on the raw partition though!

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *