This is how to make freebsd boot from a ZFS volume (whether it be raid0, raid5 or raid6).  There are rumours that a future installer will support ZFS – but this was not available for the 9.0-RELEASE, so we have to do this manually.

If you’re using FreeBSD 8.x then follow the guide at https://www.dan.me.uk/blog/2010/02/08/booting-from-zfs-raid0156-in-freebsd/

First, grab yourself a copy of DVD1 iso or the memory stick image and boot from it.  No other boot image will work – it MUST be the DVD or memory stick image!

Once the installer loads up, choose ‘Live CD’ and login with ‘root’

For my example, i’m going to build a RAID5 array on disks da0 da1 and da2.

First, we need to remove any existing GPT partition info from the disks – ignore the ‘invalid argument’ message if you get it at this stage:

gpart destroy da0
gpart destroy da1
gpart destroy da2

Now we need to initialise the GPT partitions on each disk:

gpart create -s gpt da0
gpart create -s gpt da1
gpart create -s gpt da2

We will now make a boot (64KB) and ZFS (remaining space) partition on each disk in turn:

gpart add -s 128 -t freebsd-boot da0
gpart add -s 128 -t freebsd-boot da1
gpart add -s 128 -t freebsd-boot da2

gpart add -t freebsd-zfs -l disk0 da0
gpart add -t freebsd-zfs -l disk1 da1
gpart add -t freebsd-zfs -l disk2 da2

And now we have to install the protected MBR boot code into all the drives:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da2

Now that we’ve configured the disks, we need to load the ZFS kernel modules from the CD so that we can build ZFS volumes:

kldload opensolaris
kldload zfs

And create a ZFS pool.  If you want a RAID6 volume, choose raidz2 instead of raidz1 here.  If you want a mirror, use mirror or if you want RAID0 (or single disk) just omit the raidz1 completely:

zpool create zroot raidz1 /dev/gpt/disk0 /dev/gpt/disk1 /dev/gpt/disk2
zpool set bootfs=zroot zroot

Ok, now we’ve made our ZFS pool (and it complained about not being able to mount as /zroot) – we need to mount it, then make all our filesystems on it… this is complicated, but here we go:

mdconfig -a -t malloc -s 128m -u 2
newfs -O2 /dev/md2
mount /dev/md2 /boot/zfs
zfs set mountpoint=/boot/zfs/zroot zroot
zfs mount zroot
zfs set checksum=fletcher4 zroot
zfs create -o compression=on -o exec=on -o setuid=off zroot/tmp
chmod 1777 /boot/zfs/zroot/tmp
zfs create zroot/usr
zfs create zroot/usr/home
cd /boot/zfs/zroot; ln -s /usr/home home
zfs create -o compression=lzjb -o setuid=off zroot/usr/ports
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/distfiles
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/packages
zfs create zroot/var
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/crash
zfs create -o exec=off -o setuid=off zroot/var/db
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/db/pkg
zfs create -o exec=off -o setuid=off zroot/var/empty
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log
zfs create -o compression=gzip -o exec=off -o setuid=off zroot/var/mail
zfs create -o exec=off -o setuid=off zroot/var/run
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/tmp
chmod 1777 /boot/zfs/zroot/var/tmp

Now we’re ready to install FreeBSD onto the new ZFS partitions.  We’re going to install the base, all sources and a generic kernel – this takes some time so be patient…

cd /boot/zfs/zroot
unxz -c /usr/freebsd-dist/base.txz | tar xpf –
unxz -c /usr/freebsd-dist/kernel.txz | tar xpf –
unxz -c /usr/freebsd-dist/src.txz | tar xpf –

Now we need to set /var/empty to readonly:

zfs set readonly=on zroot/var/empty

And now we’re ready to configure the installation.  To make things easier, we will chroot into the environment:

chroot /boot/zfs/zroot

We need to setup an initial /etc/rc.conf which will mount all ZFS filesystems:

echo ‘zfs_enable=”YES”‘ > /etc/rc.conf
touch /etc/fstab

And an initial /boot/loader.conf that will load the ZFS modules and set our root mountpoint:

echo ‘vfs.zfs.prefetch_disable=”1″‘ > /boot/loader.conf
echo ‘vfs.root.mountfrom=”zfs:zroot”‘ >> /boot/loader.conf
echo ‘zfs_load=”YES”‘ >> /boot/loader.conf

Now you can set your root password:

passwd root

And configure your timezone:

tzsetup

And setup a dummy aliases file for sendmail to keep it quiet 😉

cd /etc/mail
make aliases

You can do other configuration here, like adding a user etc – but when you’re done we can exit the environment:

exit

Now, we need to export our ZFS configuration (and reimport it) so we can save out the cache file:

cd /boot/zfs
zpool export zroot && zpool import zroot
cp /boot/zfs/zpool.cache /boot/zfs/zroot/boot/zfs/zpool.cache

This is the tricky part, we need to unmount the ZFS partitions and re-assign their mountpoints for the root filesystems:

zfs unmount -a
zfs set mountpoint=legacy zroot
zfs set mountpoint=/tmp zroot/tmp
zfs set mountpoint=/usr zroot/usr
zfs set mountpoint=/var zroot/var

Now we can ‘reboot’ and remove the media while the computer reboots.  Do this as soon as you can.

The computer should reboot into a ZFS-based filesystem, booted from a software RAID array on fully protected disks.

Once it’s booted, you can login and run sysinstall to configure other options like networking and startup programs (like SSH!)

Enjoy!