Full Disk Encryption (with ZFS root) for FreeBSD 9.x

To follow on from my post about full disk encryption (well almost), this is how to do the same but with a ZFS filesystem.  Like the other post, your /boot folder (which contains your kernel and modules) will not be encrypted, but the rest of your filesystem will be.

One disadvantage of this method is that you have to enter a passphrase for EACH disk in your ZFS system each boot.  Encryption inside ZFS will appear at some point – but until then this will suffice !

Boot from any FreeBSD 9 install medium (except bootonly), and choose Live CD at the install menu.

For the purposes of this article, I will assume that you’re using 4 disks (da0, da1, da2, da3), a 10GB /boot (this will be mirrored on each of the 4 disks), and the remaining space as a raidz1 (roughly similar to RAID5) ZFS filesystem.  The contents will be encrypted with 256-bit AES-XTS encryption with a 4 kb random data partial key and a secondary passphrase (required to type on each boot).

If your CPU supports the AESNI flag, the crypto(4) framework will utilise this too.

First we need to remove any existing GPT or MBR partition tables on each of the disks (ignore any ‘invalid argument’ messages):

gpart destroy -F da0
gpart destroy -F da1
gpart destroy -F da2
gpart destroy -F da3

Now we need to create a new GPT partition table on each disk:

gpart create -s gpt da0
gpart create -s gpt da1
gpart create -s gpt da2
gpart create -s gpt da3

We will now create a 64kb boot partition (this contains the boot loader only, so is safe and required to be unencrypted):

gpart add -s 128 -t freebsd-boot da0
gpart add -s 128 -t freebsd-boot da1
gpart add -s 128 -t freebsd-boot da2
gpart add -s 128 -t freebsd-boot da3

Next, we will create the /boot partition – you can adjust the sizes here if you need, but i’d suggest not shrinking it too much or you’ll get into problems when doing OS upgrades later… Note: this is mirrored not striped across the disks for maximum resilience – so will use 10GB on each disk for 10GB total usable space.

gpart add -s 10G -t freebsd-zfs da0
gpart add -s 10G -t freebsd-zfs da1
gpart add -s 10G -t freebsd-zfs da2
gpart add -s 10G -t freebsd-zfs da3

Finally, we will assign the remaining space on each disk to the root ZFS partition.  This will be encrypted before we build ZFS on top of it.

gpart add -t freebsd-zfs da0
gpart add -t freebsd-zfs da1
gpart add -t freebsd-zfs da2
gpart add -t freebsd-zfs da3

Now that we’ve created daXp1 (bootloader), daXp2 (/boot partition), daXp3 (root partition) – we need to write the boot loader code to each disk:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da2
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da3

Ok, next we will build a ramdisk to mount on /boot/zfs – this helps us mount things temporarily…

mdconfig -a -t malloc -s 128m -u 2
newfs -O2 /dev/md2
mount /dev/md2 /boot/zfs

Now we will load the modules required for ZFS and encryption:

kldload opensolaris
kldload zfs
kldload geom_eli

Next, we build a ZFS mirror for the /boot partition and mount it temporarily (to house the encryption key) – ignore any mention of unable to mount:

zpool create bootdir mirror /dev/da0p2 /dev/da1p2 /dev/da2p2 /dev/da3p2
zpool set bootfs=bootdir bootdir
mkdir /boot/zfs/bootdir
zfs set mountpoint=/boot/zfs/bootdir bootdir
zfs mount bootdir

Now we generate a random 4kb encryption key that will form (along with passphrase) the encryption key for the disk:

dd if=/dev/random of=/boot/zfs/bootdir/encryption.key bs=4096 count=1

We have everything we need to start encrypting the disks now… Enter your passphrase twice for each init phase and once again for each attach phase below:

geli init -b -B /boot/zfs/bootdir/da0p3.eli -e AES-XTS -K /boot/zfs/bootdir/encryption.key -l 256 -s 4096 /dev/da0p3
geli init -b -B /boot/zfs/bootdir/da1p3.eli -e AES-XTS -K /boot/zfs/bootdir/encryption.key -l 256 -s 4096 /dev/da1p3
geli init -b -B /boot/zfs/bootdir/da2p3.eli -e AES-XTS -K /boot/zfs/bootdir/encryption.key -l 256 -s 4096 /dev/da2p3
geli init -b -B /boot/zfs/bootdir/da3p3.eli -e AES-XTS -K /boot/zfs/bootdir/encryption.key -l 256 -s 4096 /dev/da3p3
geli attach -k /boot/zfs/bootdir/encryption.key /dev/da0p3
geli attach -k /boot/zfs/bootdir/encryption.key /dev/da1p3
geli attach -k /boot/zfs/bootdir/encryption.key /dev/da2p3
geli attach -k /boot/zfs/bootdir/encryption.key /dev/da3p3

Now that we have encrypted and mounted the partitions, we can build a ZFS root filesystem on top of it like so:

zpool create zroot raidz1 /dev/da0p3.eli /dev/da1p3.eli /dev/da2p3.eli /dev/da3p3.eli
zfs set mountpoint=/boot/zfs/zroot zroot
zfs mount zroot
zfs unmount bootdir
mkdir /boot/zfs/zroot/bootdir
zfs set mountpoint=/boot/zfs/zroot/bootdir bootdir
zfs mount bootdir

Note we unmounted the old boot mirror and re-mounted it within the root filesystem.  This will be used later to copy the kernel and modules into.

Ok, now we create all our ZFS mounts with various options as follows:

zfs set checksum=fletcher4 zroot
zfs create -o compression=on -o exec=on -o setuid=off zroot/tmp
chmod 1777 /boot/zfs/zroot/tmp
zfs create zroot/usr
zfs create zroot/usr/home
cd /boot/zfs/zroot; ln -s /usr/home home
zfs create -o compression=lzjb -o setuid=off zroot/usr/ports
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/distfiles
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/packages
zfs create zroot/var
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/crash
zfs create -o exec=off -o setuid=off zroot/var/db
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/db/pkg
zfs create -o exec=off -o setuid=off zroot/var/empty
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log
zfs create -o compression=gzip -o exec=off -o setuid=off zroot/var/mail
zfs create -o exec=off -o setuid=off zroot/var/run
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/tmp
chmod 1777 /boot/zfs/zroot/var/tmp

Now we’re ready to install FreeBSD onto the new ZFS partitions.  We’re going to install the base, all sources and a generic kernel – this takes some time so please be patient…

cd /boot/zfs/zroot
unxz -c /usr/freebsd-dist/base.txz | tar xpf –
unxz -c /usr/freebsd-dist/kernel.txz | tar xpf –
unxz -c /usr/freebsd-dist/src.txz | tar xpf –

Now we can set /var/empty to readonly:

zfs set readonly=on zroot/var/empty

And now we’re ready to chroot into the installed system to setup the configuration:

chroot /boot/zfs/zroot

Now that the base system and kernel are installed, we can move our /boot folder to it’s final place on the ZFS unencrypted mirror and do a little housekeeping:

cd /
mv boot bootdir/
ln -fs bootdir/boot
mv bootdir/encryption.key bootdir/boot/
mv bootdir/*.eli bootdir/boot/

We need to setup an initial /etc/rc.conf which will mount all ZFS filesystems on boot:

echo ‘zfs_enable=”YES”‘ > /etc/rc.conf
touch /etc/fstab

And an initial /boot/loader.conf that will load ZFS, encryption and settings for encrypted disks on boot:

echo ‘vfs.zfs.prefetch_disable=”1″‘ > /boot/loader.conf
echo ‘vfs.root.mountfrom=”zfs:zroot”‘ >> /boot/loader.conf
echo ‘zfs_load=”YES”‘ >> /boot/loader.conf
echo ‘aesni_load=”YES”‘ >> /boot/loader.conf
echo ‘geom_eli_load=”YES”‘ >> /boot/loader.conf
echo ‘geli_da0p3_keyfile0_load=”YES”‘ >> /boot/loader.conf
echo ‘geli_da0p3_keyfile0_type=”da0p3:geli_keyfile0″‘ >> /boot/loader.conf
echo ‘geli_da0p3_keyfile0_name=”/boot/encryption.key”‘ >> /boot/loader.conf
echo ‘geli_da1p3_keyfile0_load=”YES”‘ >> /boot/loader.conf
echo ‘geli_da1p3_keyfile0_type=”da1p3:geli_keyfile0″‘ >> /boot/loader.conf
echo ‘geli_da1p3_keyfile0_name=”/boot/encryption.key”‘ >> /boot/loader.conf
echo ‘geli_da2p3_keyfile0_load=”YES”‘ >> /boot/loader.conf
echo ‘geli_da2p3_keyfile0_type=”da2p3:geli_keyfile0″‘ >> /boot/loader.conf
echo ‘geli_da2p3_keyfile0_name=”/boot/encryption.key”‘ >> /boot/loader.conf
echo ‘geli_da3p3_keyfile0_load=”YES”‘ >> /boot/loader.conf
echo ‘geli_da3p3_keyfile0_type=”da3p3:geli_keyfile0″‘ >> /boot/loader.conf
echo ‘geli_da3p3_keyfile0_name=”/boot/encryption.key”‘ >> /boot/loader.conf

The above settings tell the OS which encryption keyfile to use for each disk partition.

Now you can set your root password:

passwd root

And configure your timezone:


And setup a dummy /etc/mail/aliases file to prevent sendmail warnings:

cd /etc/mail
make aliases

Now you can configure any additional settings you require (such as adding new users, configuring networking or setting sshd to run on boot) – when you’re done, we need to exit the chroot:


Now, we need to make sure the bootloader can read our ZFS pool cache (or it wont mount our ZFS disks on boot):

cd /boot/zfs
cp /boot/zfs/zpool.cache /boot/zfs/zroot/boot/zfs/zpool.cache

Finally, we need to unmount all the ZFS filesystems and configure their final mountpoints…

zfs unmount -a
zfs set mountpoint=legacy zroot
zfs set mountpoint=/tmp zroot/tmp
zfs set mountpoint=/usr zroot/usr
zfs set mountpoint=/var zroot/var
zfs set mountpoint=/bootdir bootdir

Now we can ‘reboot’ and remove the media while the computer reboots.  Do this as soon as you can.

The computer should reboot into a ZFS-based filesystem, booted from a software RAID array on fully protected disks with all but /boot partition encrypted.  Note: it will ask you to enter a passphrase for each disk parition used above (4 times) – you should take care to enter the correct passwords as it will treat any passwords missed as a failed disk (you get 3 attempts at each password)

Once it’s booted, you can login and run sysinstall to configure other options like networking and startup programs (like SSH!)

The only point to note is that when you do an OS upgrade, during the “mergemaster” stage, it will complain that /boot is a symlink not a directory.  Simply tell it to ignore/do nothing and it will install the files as normal.


106 thoughts on “Full Disk Encryption (with ZFS root) for FreeBSD 9.x

  1. Alex

    Hi, how could i apply this example to a netbook with just one hard drive? I would like to have full disk encryption with zfs and separate partitions: /root,/usr, /tmp, var.

    1. dan Post author

      If you have a laptop with a single disk, it will almost certainly be “ada0” instead of “da0” (and forget da1, da2, da3).
      Then change the line ‘zpool create bootdir …’ to be ‘zpool create bootdir /dev/ada0’ (single disk)
      Then change the line ‘zpool create zroot …’ to be ‘zpool create zroot /dev/ada0p3.eli’ (single encrypted partition)
      The rest of the instructions should still apply as per the post.

  2. dan Post author

    As this guide is designed for a raidz1 system, swap is not convenient (it’s not protected and can cause a kernel panic if a disk fails and a chunk of swap disappears) – the best way for swap on this setup is to use the swapfile on the ZFS partition… however, if you’re doing a single-disk ZFS install you can create a swap partition… See my other guide on how to do encrypted swap partitions using a one-time encryption key each boot 🙂

  3. dan Post author

    zpool create bootdir /dev/ada0p2 (not /dev/ada0 – my mistake, sorry) Also in loader.conf you need ada0p4 not ada0p3 as you’ve pushed the encrypted partition from p3 to p4 by adding the swap partition. (otherwise when you boot you wont decrypt the partition and the system wont boot)

  4. Alex

    Hey Dan, good catch on the partion. I’ve further than i was before. I’m now getting an error at the following line. ‘zfs create –o compression=on –o exec=on –o setuid=off zroot’.

    It gives a cannot create ‘zroot’: missing dataset name.

  5. dan Post author

    None of the ‘zfs create’ lines in the post above have only ‘zroot’ as the last parameter… they’re always in the format of zroot/mountpoint. You cant create a zfs root mountpoint (zroot) as it always exists as it is the pool name. I’m not sure where you got the zfs create line from – but that’s the problem.

  6. Alex

    Thanks for pointing that out Dan. I can’t believe i didn’t see that. I went through the set without anymore error but now when the system boots and ask for my passphrase, it says it’s wrong. I’m pretty sure I’m putting in the right passphrase. The same thing had happen when i tried PCBSD and used their installer to setup the disk encryption.

  7. dan Post author

    If it says the passphrase is wrong, and you’re certain it is correct then it is having trouble reading the file portion of the key… make sure the loader.conf says ada0p4. Also ensure that the path is correct to the keyfile… the ‘bootdir’ partition is read as root during kernel initialisation so /bootdir is really just / during kernel init. This is why in my instructions, the ‘boot’ folder is inside /bootdir and also symlinked /boot to /bootdir/boot and all files inside the boot folder (this way, referencing /boot is always correct).
    Essentially what happens is the bootloader reads the ZFS root from ada0p2 which loads the kernel modules and kernel itself (requires a “/boot” within the partition to boot from), then it mounts root from zfs:zroot (which is on the encrypted ada0p4 partition) via a remount.
    You can, of course, omit the keyfile part of the encryption entirely by removing the “-K keyfile” and “-k keyfile” sections of the geli instructions. In this configuration, it’s no less secure to do this.

  8. Alex

    Ok, I will try it without the -k sections. I’m wondering if it had anything to do with my keymap. I dont remember even setting it up. does freebsd automatically set the correct keymap?

  9. dan Post author

    The default keymap is a US ASCII one I believe… but during kernel boot and during the installer, it’s the same keymap. You can configure an alternative keymap in rc.conf – but that only applies after the kernel has booted anyway.

  10. Magnus

    Nice guide! I’m thinking of doing something similar. What I need is a backup for my laptop so that I can be up and running quickly in case of a disk failure. I’m thinking of doing this by installing FreeBSD (or perhaps PCBSD) onto an encrypted ZFS mirror, where one half of the mirror is the local disk of my laptop, and the other is an external USB harddrive. I believe I will be able to do so by following your guide, adjusting as necessary. If I setup the external drive like you describe, I should be able to boot from it if my main drive dies, assuming my laptop will boot from USB, right?

    I’ve also been thinking if I should put /boot on a SD-card or a pen drive, and skip having it on the main harddrive.

    Do you have any thoughts, warnings, or recommendations?

    1. dan Post author

      Indeed, it should… however USB is slow so it would slow things down.
      You can take regular backups from ZFS to USB though… I may write a guide on how to do this soon as people love backups 😉

      Booting from SD/USB is perfectly fine, and should work without a problem (other than being a bit slower to boot up as the medium is slower) – if you did that, you can encrypt the entire hard drive instead of partitions inside it (work on da0/ada0 directly instead of da0p3/adap3) You can even boot from USB and then remove it once booted – meaning the keyfile part of the key is not stored on the disk itself (a sorta half-way house to a 2-factor solution) but you’d have to remember to copy any changes from /boot on the booted system to /boot on the USB/SD card whenever you made them.

  11. BSDGuy

    Hi Dan

    Great article…really enjoying all the FreeBSD/ZFS/Encryption ones…keep ’em coming please! 😉 Would be great to include an encrypted swap partition on ZFS in this article too.

    I was wondering, where in this article would one be able to do a complete system restore using a dump file? Can you even do this? I have my entire system backed up into a dump file and would like to restore it on an encrypted mirrored ZFS partition…is this possible?

    Thank you!

    1. dan Post author

      Thanks 🙂 I’ve never tried using ‘restore’ to restore onto it, but I would imagine it would work ok.
      You can use zfs send and zfs receive to backup/restore an existing ZFS-based system too.
      If restore works, you would create the zpool, create all the zfs sets you wanted, mount them all and then issue the restore command instead of the unxz steps.
      When I get some free time, i’ll do some tests in a VM and post an article about it if it works.

  12. BSDGuy

    Thanks Dan! My systems on UFS currently and backed up using dump so zfs send/receive won’t work for me just yet!

    Look forward to some more (great) articles.

    I’m going to give this article a try this weekend but trying it with a dump restore and an encrypted swap partition 😉

  13. BSDGuy

    Hi Dan

    I came across this to add a ZFS swap:

    zfs create -V 4G zroot/swap
    zfs set org.freebsd:swap=on zroot/swap
    zfs set checksum=off zroot/swap

    I assume this will be encrypted as the swap will be on the ZFS partition which is on top of an encrypted partition?

  14. dan Post author

    Yep it would be encrypted. You could also do a file based swapfile (see my other posts) but that would be slightly less desirable than the method you have mentioned (as the swapfile would still have checksumming enabled)

  15. BSDGuy

    Hi Dan

    I followed your (excellent) guide today on a test machine. When following all the instructions my drives are ada0 and ada1. Everything went fine until I rebooted. When it prompted me for a passphrase it kept saying wrong key for ad0p3. Why is it referencing my drive as ad0 when it is ada0? If I try to enter my passphrase it says its wrong. I tried doing it all again but when running:

    gpart create -s gpt ad0 — it fails
    gpart create -s gpt ada0 — works fine but then when I reboot it fails

    Any ideas?

  16. dan Post author

    Hmm, I can’t think of any reason why it would say ad0p3 – i’ve only ever seen it call IDE disks ada0p3. if you use a keyfile with the passphrase, then the kernel won’t load the key for “ad0p3” as the name doesnt match (then it wont accept your passphrase), but as to why it calls it ad0p3 i’m at a complete loss!
    You can duplicate the /boot/loader.conf lines for ada0p3 to ad0p3 safely, and see if that lets you continue – but i really don’t know why it would call it that.

  17. BSDGuy

    Thanks Dan!

    Whats the best way of accessing the /boot/loader.conf since I have rebooted into the Live CD? I am battling to access this file since rebooting!

  18. BSDGuy

    I got it working! 😉

    Don’t ask me why but when I boot off the Live CD the drives are ada0 and ada1 but when bootup its looking for ad0p3 and ad1p3 (instead of what I thought would be ada0p3 and ada1p3). Your suggestion did the trick of modifying the /boot/loader.conf. I also added:

    Is 512M enough for a system with 4GB RAM?

    Thanks for all your help! Its great to have an encrypted ZFS root now!!

  19. dan Post author

    I’ve no idea why it would change the names of the drives, but ok!
    The kmem sizes are fine (though I generally don’t set them on a 4GB machine.. but I do usually turn prefetch off without lots of ram)

  20. BSDGuy

    Hi Dan

    If you have a problem booting with an encrypted ZFS configuration (as your article outlines)…how do you access /boot/loader.conf while booting off the LiveCD?

    I made a change to /boot/loader.conf and when I rebooted it failed. How would you rollback a change like this while booting off the LiveCD?

    I tried a few things but just couldn’t get access to the bootdir. Any ideas?

  21. dan Post author

    You would have to mount the unencrypted boot folder…
    “mount /dev/ada0p2 /mnt”
    then you can access /mnt/boot/loader.conf and reboot.
    Remember, you can always override loader.conf details within the boot loader too (by escaping to the loader prompt and using the built-in commands there)

  22. dan Post author

    oh wait, I was thinking of another article LOL! I forgot this has full ZFS.
    It’s not easy to mount without causing issues in the live system later. It’s better to override the settings in the loader until the machine boots and then edit loader.conf directly from there. Which setting do you need to fix?

  23. BSDGuy

    Yeah, I was having difficulties too doing this!

    What do you mean by “override the settings in the loader until the machine boots”?

    I changed:




    and then the machine wouldn’t boot…at all.

    The reason I ask this question is that, when this machine is live, and I make a boo boo in the boot/loader.conf, how can I roll the change back?

    How do you manage this? 😉

  24. dan Post author

    When you get to the boot menu, choose escape to loader, then… “set vm.kmem_size=512M” etc then “boot” when you’re ready to boot. it should override the settings (just for that boot) but long enough that you can edit your loader.conf.

  25. BSDGuy

    Aaah, thats perfect…I shall give that a try. Thats far better than what I was trying. I assume you can set any of the settings in the loader.conf this way? Including paths to the encryption keys etc?

    Two more questions 😉

    1) Have you considered using labels for all the device (hard drive/parition) names? ie: instead of using ada0p2 it would be BootPartition0? I was trying this earlier but didn’t get it right when booting up. It makes reading the drive/partition names much easier!

    2) How does one backup a ZFS pool? I’m not talking about a snap shot but a FULL and complete backup to file that I can copy to an external USB drive and keep it safe somewhere. How do you backup your encrypted ZFS system? I use dump with my current UFS (non encrypted) system but have been wondering how you approach this on the enxrypted ZFS root setup. With a dump backup to file (which I keep on a USB drive) I can do a full restore to another machine quickly and easily. Is this doable with ZFS?

  26. dan Post author

    You should be able to specify any settings there. You can’t use labels for encryption as the kernel only sees the raw device names. The labels come later. To backup an entire ZFS pool, you should look at the ‘zfs send’ command (check out the man page). I will look at doing a zfs send/receive post at some point in the future.

  27. BSDGuy

    Thanks Dan.

    What did you mean by: the labels come later? Is this a new feature coming in the future?

    I will definitely check out the zfs send command then. I (of course) look forward to an article that you write on your web site!

  28. dan Post author

    I mean the part of the kernel which looks for (and creates device nodes for) labels occurs very late in the kernel, after the encryption is actioned. For this reason, you can’t use labels in the encryption. You should never need to refer to any of the partitions individually though. Everything should be via ZFS.

  29. BSDGuy

    Thanks for the explanation. So I guess you have to be VERY careful about moving disks around in a machine with encrypted ZFS as the device names could change which would stop the machine booting up?

    Thats what I like about labels on my current UFS system…I can move the disks around and not worry about the system not being able to boot again.

  30. dan Post author

    I’ve never tried moving them around so couldn’t say for sure. I’d imagine each disk has its own ID though. I’ve never had the need to move disks around in a computer :-/

  31. BSDGuy

    How easy (or difficult) is it to store the encryption key on an external USB drive? Can the USB drive be removed once the server has booted up? Or does the encryption key need to be avaialble at all times while the machine is running?

  32. dan Post author

    The only way to do that would be to boot from USB (containing the encryption key) which would then mount the root filesystem from the encrypted ZFS. Obviously any changes to the /boot folder would need to be kept in sync with the USB stick, so it’s not ideal.
    The encryption key is kept in memory once loaded so is not needed beyond that.
    You can’t mount a USB drive for the purposes of loading the data that I know of – the only method is to boot from the USB drive.

  33. BSDGuy

    Can you clarify (please) what you mean by: Obviously any changes to the /boot folder would need to be kept in sync with the USB stick

    I would have thought that in this scenario the /boot folder was *only* on the USB drive (with the encryption key)? So why would it get out of sync? Is /boot in the encrypted root filesystem too? If so why are there two copies?

    Obviously having your encryption key readily available on an unencrypted partition isn’t ideal as if someone gets the key then they already have half of what they need to break into your encrypted system…

    I like all the options you have with this. If I hosted my server in a datacentre I would have /boot and the key on a USB key and once the server booted up I would remove the USB key and take it with me. Thats about as secure as it gets I think!

  34. dan Post author

    To do it that way, you’re essentially booting from one device (the USB stick), but then mounting another root filesystem (which would also contain a /boot folder – the USB drive is not present once mount_root is called) – the boot folder on the root filesystem would be used if you kldload’d any modules (anything not in loader.conf)

    The USB drive is a temporary root filesystem to load the kernel and modules, after which you mount an entirely different root filesystem (the encrypted one)

    Remember that the file (although called encryption.key) is not a key. It creates a binary segment to your passphrase for decrypting the key on the drive. Without both the file and the passphrase, the data is useless. Some people do away with the file segment entirely and just use a passphrase.

    I like having both as the file contains binary data not just text.

    The contents of the file and your passphrase are concatenated to become the entire passphrase for the encrypted key on the hard drive.

    Using the USB method for a datacentre server is ok, but if you ever need to reboot (or the server crashes or has an issue such as power failure) then you need to attend the datacentre to boot it.

    With the keyfile on the unencrypted drve, you can use remote console to enter the passphrase (assuming the server supports it of course!) and is still almost as secure – and certainly more than enough for anyone other than extremely determined people that wish to gain access.

  35. BSDGuy

    Thanks for that (useful) explanation! I with everyone explained things as clearly as you did.

    Heres another thought. Lets say I setup my server as described here but later on decided that I wanted to add a ZFS log or ZFS cache…can these be encrypted too?

    What are your thoughts on using a hardware accelerator to improve encryption performance? Such as the Soekris VPN 1401:


    My CPU…an Intel Dual Core 2 E8400 doesn’t support AES-NI unfortanately so am looking for ways to improve encryption performance.

  36. dan Post author

    ZFS will use any raw device, so you can encrypt anything you wish and use the device with the “.eli” suffix. Obviously, the log or cache would be slower if encrypted – but if needs must etc.

    I’ve never tried any hardware accelleration cards under FreeBSD so can’t really comment.

    AESNI does make quite a difference to the encryption speeds, and the kernel/geli modules will use it by default so long as the aesni.ko module is loaded.

    I use 256bit AES-XTS geli encryption on my laptop (without AESNI) and can’t say I notice the encryption slowdown. My CPU is a Core i7 1.7GHz (mobile) so not necessarily a huge core speed either.

  37. dan Post author

    So long as FreeBSD has drivers for the crypto framework for the card, then geli will use it to accellerate encryption

  38. BSDGuy

    Its amazing how flexible and powerful ZFS is!

    In my “new” machine I plan on have two 320GB drives mirrored (to be used as the boot/root partitions) and one 500GB drive used for zfs send backups.

    I was thinking…would it be helpful (for a performance boost) to add a few USB drives to the machine to be used as a ZFS log/cache? Maybe two 8GB USB drives mirrored for the log and a single 8GB USB drive the cache? Maybe multiple 8GB drives striped for the cache? The 320GB drives I have are only 5400RPMs so I was wondering if adding some USB drives would make any difference to performace as ZFS log/cache?

  39. dan Post author

    I would avoid USB for log/cache. USB is very slow compared to hard drives (max of around 35MB/sec compared to around 130MB/sec for hard drive) – also most usb sticks have very slow write speeds (even down to around 3MB/sec). I would think it would slow things down instead of speed them up.

    Generally, it’s advised to use a SSD device on your SATA bus if possible (as these can do up to 600MB/sec with virtually no latency)

    I personally don’t use a log/cache drive in any of my machines – i’ve always been satisfied with the performance. To me, more ram is better for frequently used/recently used read caches.

  40. BSDGuy

    I’ve been giving this some thought 😉

    I have two options for setting up the disks in my “new” machine:

    1) 2 x 320GB 5400RPM drives uses for the ZFS root mirrored, 1 x 500GB drive used for zfs send backups for a month and to use the 120 GB SSD drive from my old laptop as a ZFS cache

    2) Buy another 120GB SSD drive and use the two SSD drives for a mirrored ZFS root and maybe use the 2 x 320GB drives as a mirrored ZFS in another pool for backups

    First option is free 😉 Second option I think will be faster but required buying another SSD drive. How would you go about deciding on your disk layout for an encrypted ZFS root?

    Oh theres a third option too! Use 2 x 320GB and 2 x 500GB drives in a striped mirror ZFS root pool.

    I can’t make up my mind which option would be the best approach. I don’t need much storage space currently (only 12GB) but having good performance would be nice!

  41. BSDGuy

    Today I migrated my server to an encrypted ZFS root as per your guide and it went great…thank you!

    One think I would like to do is encrypt a single ZFS drive using the existing encryption key. I know I have to:

    geli init -b -B /boot/zfs/bootdir/ad4p1.eli -e AES-XTS -K /boot/zfs/bootdir/encryption.key -l 256 -s 4096 /dev/ad4p1
    geli attach -k /boot/zfs/bootdir/encryption.key /dev/ad4p1

    But I think the syntax has to be modified here as I won’t be using this drive as a boot drive…just a backup drive for data. Am I on the right track? 😉

  42. dan Post author

    There’s no real difference between a data disk and a boot disk other than the “-b” flag to geli init asks for the passphrase on boot. Without that, you have to call ‘attach’ every time you want to use the disk and will be asked for the passphrase then.
    If the single drive will be permanently attached to the computer, then feel free to configure it in the same way as the boot drive. If it’s a removable disk, then skip the “-b” flag so it wont ask on boot for it if the drive is present.

  43. BSDGuy

    Thanks very much Dan. I followed the same steps for the data drive and it is now encrypted and mounted. Been playing around with zfs send ans scrub. Got much to learn about all this like scheduling these to run automatically every day/week.

    Its looking good 😉

  44. BSDGuy

    How easy is it to restore a system like this using zfs receive from a file? Lets say I have the zfs send file on a USB drive and I lose the entire source (original) server. Can I restore the entire server from the zfs send file using zfs receive? I have backed up both the bootdir and zroot using zfs send to a file option and have been wondering about how to approach doing the restore!

  45. dan Post author

    zfs receive will receive anything you send it… but for the main root (/) i’m not so sure you can receive the root. You could, of course, zfs receive the root backup into another mount and then copy/move its contents into the new root i guess.

  46. BSDGuy

    So if I wanted to restore my encrypted ZFS root onto a new server (without doing a reinstall or moving the hard drives) this could be an issue?

    Thats one thing I loved about dump/restore…I could move my system to any new machine so easily. I would have thought this would have been doable with ZFS too since its so much better than UFS?

    I will try doing a test zfs receive on my (almost identical) spare machine to see if I can do a zfs receive with the encrypted ZFS root.


Leave a Reply

Your email address will not be published. Required fields are marked *