Monthly Archives: February 2010

Booting from ZFS RAID0/1/5/6 in FreeBSD 8.x

71 Replies

Ok, this is a long post but a useful one.  This is how to make freebsd boot from a ZFS volume (whether it be raid0, raid5 or raid6).  The freebsd installer doesn’t support anything exotic so we have to do this manually.

If you’re using FreeBSD 9.0, then follow the guide at https://www.dan.me.uk/blog/2012/01/22/booting-from-zfs-raid0156-in-freebsd-9-0-release/

First, grab yourself a copy of DVD1 iso or the memory stick image and boot from it.  No other boot image will work – it MUST be the DVD or memory stick image!

Once you’ve booted into the installer and chosen your country and keyboard layouts, go to the Fixit menu and choose either CDROM/DVD or USB depending on the installation media you used.  This will open up a terminal window into a live filesystem booted from the DVD/USB.

For my example, i’m going to build a RAID5 array on disks da0 da1 and da2.

First, we need to remove any existing GPT partition info from the disks – ignore the ‘invalid argument’ message if you get it at this stage:

gpart destroy da0
gpart destroy da1
gpart destroy da2

Now we need to initialise the GPT partitions on each disk:

gpart create -s gpt da0
gpart create -s gpt da1
gpart create -s gpt da2

We will now make a boot (64KB) and ZFS (remaining space) partition on each disk in turn:

gpart add -s 128 -t freebsd-boot da0
gpart add -s 128 -t freebsd-boot da1
gpart add -s 128 -t freebsd-boot da2

gpart add -t freebsd-zfs -l disk0 da0
gpart add -t freebsd-zfs -l disk1 da1
gpart add -t freebsd-zfs -l disk2 da2

And now we have to install the protected MBR boot code into all the drives:

gpart bootcode -b /mnt2/boot/pmbr -p /mnt2/boot/gptzfsboot -i 1 da0
gpart bootcode -b /mnt2/boot/pmbr -p /mnt2/boot/gptzfsboot -i 1 da1
gpart bootcode -b /mnt2/boot/pmbr -p /mnt2/boot/gptzfsboot -i 1 da2

Now that we’ve configured the disks, we need to load the ZFS kernel modules from the CD so that we can build ZFS volumes:

kldload /mnt2/boot/kernel/opensolaris.ko
kldload /mnt2/boot/kernel/zfs.ko

And create a ZFS pool.  If you want a RAID6 volume, choose raidz2 instead of raidz1 here.  If you want a mirror, use mirror or if you want RAID0 (or single disk) just omit the raidz1 completely:

zpool create zroot raidz1 /dev/gpt/disk0 /dev/gpt/disk1 /dev/gpt/disk2
zpool set bootfs=zroot zroot

Ok, now we’ve made our ZFS pool (and it’s currently mounted at /zroot/) – we need to make all our filesystems on it… this is complicated, but here we go:

zfs set checksum=fletcher4 zroot
zfs create -o compression=on -o exec=on -o setuid=off zroot/tmp
chmod 1777 /zroot/tmp
zfs create zroot/usr
zfs create zroot/usr/home
cd /zroot; ln -s /usr/home home
zfs create -o compression=lzjb -o setuid=off zroot/usr/ports
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/distfiles
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/packages
zfs create zroot/var
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/crash
zfs create -o exec=off -o setuid=off zroot/var/db
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/db/pkg
zfs create -o exec=off -o setuid=off zroot/var/empty
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log
zfs create -o compression=gzip -o exec=off -o setuid=off zroot/var/mail
zfs create -o exec=off -o setuid=off zroot/var/run
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/tmp
chmod 1777 /zroot/var/tmp

Now we’re ready to install FreeBSD onto the new ZFS partitions.  We’re going to install the base, manual pages, all sources and a generic kernel – this takes some time so be patient…

cd /dist/8.1-RELEASE/
export DESTDIR=/zroot
for dir in base manpages ; do (cd $dir ; ./install.sh) ; done
cd src ; ./install.sh all
cd ../kernels ; ./install.sh generic
cd /zroot/boot ; cp -Rlp GENERIC/* /zroot/boot/kernel/

Now we need to set /var/empty to readonly:

zfs set readonly=on zroot/var/empty

And now we’re ready to configure the installation.  To make things easier, we will chroot into the environment:

chroot /zroot

We need to setup an initial /etc/rc.conf which will mount all ZFS filesystems:

echo ‘zfs_enable=”YES”‘ > /etc/rc.conf

And an initial /boot/loader.conf that will load the ZFS modules and set our root mountpoint:

echo ‘vfs.zfs.prefetch_disable=”1″‘ > /boot/loader.conf
echo ‘vfs.root.mountfrom=”zfs:zroot”‘ >> /boot/loader.conf
echo ‘zfs_load=”YES”‘ >> /boot/loader.conf

Now you can set your root password:

passwd root

And configure your timezone:

tzsetup

And setup a dummy aliases file for sendmail to keep it quiet 😉

cd /etc/mail
make aliases

You can do other configuration here, like adding a user etc – but when you’re done we can exit the environment:

exit

Now, we need to export our ZFS configuration (and reimport it) so we can save out the cache file:

mkdir /boot/zfs
cd /boot/zfs
zpool export zroot && zpool import zroot
cp /boot/zfs/zpool.cache /zroot/boot/zfs/zpool.cache

We now create an empty /etc/fstab file as follows:

touch /zroot/etc/fstab

This is the tricky part, we need to unmount the ZFS partitions and re-assign their mountpoints for the root filesystems:

export LD_LIBRARY_PATH=/mnt2/lib
zfs unmount -a
zfs set mountpoint=legacy zroot
zfs set mountpoint=/tmp zroot/tmp
zfs set mountpoint=/usr zroot/usr
zfs set mountpoint=/var zroot/var

Now we can exit the fixit shell, remove the media and reboot the computer.  Do this as soon as you can.

The computer should reboot into a ZFS-based filesystem, booted from a software RAID array on fully protected disks.

Once it’s booted, you can login and run sysinstall to configure other options like networking and startup programs (like SSH!)

Enjoy!

PPTP VPN in FreeBSD (for Windows XP/Vista/7 clients)

37 Replies

Here’s a simple guide to setting up a VPN server on FreeBSD so that Windows clients can connect using their built-in VPN clients…

First, make sure your ports collection is up-to-date, then build poptop in /usr/ports/net/poptop:

# cd /usr/ports/net/poptop/
# make
# make install

Next we need to create a config file for poptop… create the file /usr/local/etc/pptpd.conf as follows:

nobsdcomp
proxyarp
pidfile /var/run/pptpd.pid
+chapms-v2
mppe-40
mppe-128
mppe-stateless

Next we need to create a PPP configuration file called /etc/ppp/ppp.conf (overwrite the existing file) as follows (edit the IPs to suit your network requirements):

loop:
set timeout 0
set log phase chat connect lcp ipcp command
set device localhost:pptp
set dial
set login
# Server (local) IP address, Range for Clients, and Netmask
set ifaddr 192.168.31.254 192.168.31.120-192.168.31.128 255.255.255.255
set server /tmp/vpn-in-%d “” 0177

loop-in:
set timeout 0
set log phase lcp ipcp command
allow mode direct

pptp:
load loop
disable pap
disable passwdauth
disable ipv6cp
enable proxy
accept dns
enable MSChapV2
enable mppe
disable deflate pred1
deny deflate pred1
set device !/etc/ppp/secure

Every line above except those ending with a colon(:) should be indented or ppp.conf will not work – the CMS in use on this site wont indent – sorry 🙁

Next we need to create a file called /etc/ppp/secure with the following contents:

#!/bin/sh
exec /usr/sbin/ppp -direct loop-in

And set it to be executable with chmod 0755 /etc/ppp/secure – this script will be run automatically during the VPN setup process.

Now we need to add a login for the VPN (you can have multiple username/passwords in this file) called /etc/ppp/ppp.secret like so:

user1 pass1
user2 pass2
user3 pass3

Now we need to enable proxy ARP in Freebsd.  Add the following line into /etc/sysctl.conf:

net.link.ether.inet.proxyall=1

To activate it without a reboot, type sysctl net.link.ether.inet.proxyall=1

And finally set the VPN server to start on bootup automatically by adding the following into /etc/rc.conf:

pptpd_enable=”YES”

Now startup the VPN server by running:

/usr/local/etc/rc.d/pptpd start

Your VPN server is now ready on your FreeBSD server and you’re ready to configure your Windows clients to connect to it.  I’ll give you an example of how to do it on Windows 7, i’m sure you can find out how to do it on earlier versions of windows…

  1. Go to Start
  2. Open Control Panel
  3. Under Network and Internet, click View network status and tasks
  4. Click Set up a new connection or network at the bottom of the page
  5. Choose Connect to a workplace and click Next
  6. Select No, create a new connection and click Next
  7. Select Use my Internet connection (VPN)
  8. Enter the IP address of your VPN server in Internet address and give it a description below.
  9. Check Don’t connect now; just set it up so I can connect later and click Next
  10. Enter the username and password from your ppp.secret file, leave Domain blank, click Create
  11. Click Close
  12. Click Change adapter settings on the left of your Network and Sharing Center window
  13. Right-click on your new VPN and go to Properties
  14. Go to the Networking tab, click Internet Protocol Version 4 (TCP/IPv4) and click Properties
  15. Click Advanced
  16. Uncheck Use default gateway on remote network (this enables split tunnelling mode which is probably what you will want to use)
  17. Click Ok then Ok then Ok and close the Network Connections window.

Now your VPN is setup, you can connect by clicking on the network icon in your taskbar, clicking your VPN in the list and clicking on Connect.

Windows Vista is similar to Windows 7 to configure.  Windows XP is a little different – but the general setup is identical on all three flavours of Windows.

Split Tunnelling mode is when you join the remote network but not route your entire internet connection via it.  If you want to route your entire connection via the VPN you can skip steps 12-17 above.

RAID5/6 (using ZFS) in FreeBSD 8.x

5 Replies

Ok, FreeBSD still lacks a decent RAID5 implementation within its core system (some people use the geom_raid5 3rd party module that you can find in freenas) – but with ZFS moved into production status in freebsd 8 now we can use this.

ZFS supports various raid levels.  We will use RAID5 in this example – I’ll explain how to use RAID6 later in the article.

Ok, for my example I will use 6 x 2TB hard drives freshly installed in my system (listed as ad10 ad12 ad14 ad16 ad18 ad20 in dmesg) to generate a RAID5 raid set, giving 5 x 2TB of usable space and capable of a single disk failure without loss of data.  Remember, you need a minimum of 3 disks to do RAID5, and you get N-1 capacity (N-2 for RAID6)

First, we need to load ZFS into the system… add the following into your /boot/loader.conf:

vfs.zfs.prefetch_disable=”1″
zfs_load=”YES”

This will cause ZFS to load in the kernel during each boot.  The prefetch_disable is set by default on servers with less than 4GB of ram, but it’s safe to add it anyway.  I’ve found this produces far more stable results in live systems so go with it 😉

Next, add the following into your /etc/rc.conf file:

zfs_enable=”YES”

This will re-mount any ZFS filesystems on every boot, and setup any necessary settings on each boot.

Now, we will add all 6 disks into a raid5 set called ‘datastore’ – run the following as root:

zpool create datastore raidz ad10 ad12 ad14 ad16 ad18 ad20

‘raidz’ is ZFS’s name for RAID5 – to do RAID6 you would use ‘raidz2’ instead.  You can confirm the command was successful with zpool status as follows:

pool: datastore
state: ONLINE
scrub: none
config:

NAME        STATE     READ WRITE CKSUM
datastore   ONLINE       0     0     0
raidz1    ONLINE       0     0     0
ad10    ONLINE       0     0     0
ad12    ONLINE       0     0     0
ad14    ONLINE       0     0     0
ad16    ONLINE       0     0     0
ad18    ONLINE       0     0     0
ad20    ONLINE       0     0     0

errors: No known data errors

This shows the raid set is online and healthy.  When there are problems, it will drop to DEGRADED state.  If you have too many disk failures, it will show FAULTED and the entire array is lost (in our example we would need to lose 2 disks to cause this, or 3 in a RAID6 setup)

Now we will set the pool to auto-recover when a disk is replaced, run the following as root:

zpool set autoreplace=on datastore

This will cause the array to auto-readd when you replace a disk in the same physical location (e.g. if ad16 fails and you replace it with a new disk, it will re-add the disk to the pool)

You will now notice that you have a /datastore folder with the entire storage available to it.  you can confirm this with zfs list as follows:

NAME             USED  AVAIL  REFER  MOUNTPOINT
datastore       2.63T  6.26T  29.9K  /datastore

You now have a working RAID5 (or RAID6) software raid setup in FreeBSD.

Generally to setup RAID6 instead of RAID5 you replace the word raidz with raidz2.  RAID5 allows for a single disk failure without data loss, RAID6 allows for a double disk failure without data loss.

After a disk failure, run zpool status to ensure the state is set to ONLINE for all the disks in the array then run the command zpool scrub datastore to make zfs rebuild the array.  Rebuilding takes time (it rebuilds based on used data so the more full your array the longer the rebuild time!) – once it’s completed the scrub or “resilver” process, your array will return back to ONLINE status and be fully protected against disk failures once again.

As this process can take (literally) hours to complete some people prefer a RAID6 setup to allow for a 2nd disk failure during those few hours.  This is a decision you should make based on the importance of the data you will store on the array!