Category Archives: Hints & Tips

General Hints & Tips

Speed up FreeBSD reboots by 7 seconds

Ok, 7 seconds doesn’t seem like much… but if you’re rebooting a live server then every second of downtime counts – so why not?

This is a simple change that reduces the timer on the initial FreeBSD menu from 10 seconds to 3 seconds.  (I don’t recommend reducing it below 3 seconds, as it may make it more difficult for you to use the menu should you ever need to!)

Ok, so you need to edit the /boot/loader.conf file (or create it if it doesn’t exist) and add the following line:


and that’s all you need to do.  Next reboot, the timer will start at 3 instead of 10 and you have a reboot that takes 7 seconds less than before.

HTTP Public Key Pinning (HPKP)

As we move into an more secure environment, simply have HTTPS isn’t sufficient.  Many cases of forged SSL certificates for man-in-the-middle attacks have appeared recently.

These can be obtained through deception or hacking attemps on SSL CAs.

One method to help combat this is HTTP Public Key Pinning (HPKP) – this is where the webserver can communicate to the web browser an allowed certificate path, with a sufficiently long expiry time to be of use to return visitors.  It’s not perfect or ideal, but it’s better than nothing.

This post does NOT detail the most secure method of HPKP but a compromise in terms of usability vs security.  It will ensure that an unauthorised certificate would need to have been generated by one of your chosen SSL CAs only.  The more secure method of having dual online and offline keys is out of scope for this blog post and a more advanced topic.

The way this works is to publish a header in the HTTPS response detailing the allowed hashes of public keys of webserver keys or keysigner keys that are permitted to be in the certificate chain.  This is in addition to the normal process of validation of the full SSL certificate chain.

BEWARE: misconfiguration of this header can lead people to be unable to view your site for MONTHS so be careful!

First let’s look how to obtain the SHA-256 hash needed for a certificate.  We use OpenSSL to do this based on the .crt file… Let’s assume the certificate is in the cert.crt file…

openssl x509 -pubkey < cert.crt | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary| openssl enc -base64

This will provide a Base64 encoded string which we can use later.

Now we provide the information to the webserver… i’ll detail Apache 2.4 here, but other servers will have similar methods of adding header outputs.  We add the following into the VirtualHost block for the SSL site:

Header always set Public-Key-Pins “pin-sha256=\”HASH\”; pin-sha256=\”HASH\”; max-age=30;”

where the Base64 has generated previously replaces the first ‘HASH’.

HPKP requires a minimum of 2 hashes to be present, including one hash that is NOT present in the certificate chain.  This is to encourage you to ensure you have a backup plan.

In my examples, I use the hash of my SSL public key and the hashes of my chosen primary and secondary SSL certificate authority’s intermediate certificates.

In my case, I use StartSSL’s class 2 intermediate and RapidSSL’s SHA-2 under SHA-2 root intermediate.  To generate their hashes, use the above OpenSSL method on their CA certificates.

This means that I can use my existing key/csr with any SSL CA, or I can generate a new key with my primary or secondary CA.  It also satisfies the requirement for one of the hashes to not exist (my current certificate is not present via my secondary CA)

The max-age above is set to 30 seconds for testing purposes… once you’re completely happy with your choice, this should be raised to a much longer figure.  I use a 30 day period (2592000 seconds)

A good method of testing is to use Qualys SSLlabs tester at – this will show you the pinned list, including any currently in the chain in a different colour.

SSH Fingerprints in DNS

Here’s how to add SSHFP records to DNS.  You need to have DNSSEC signed zones for this to work.

To generate the records, simply use the following command against your host public keys and copy them into your DNS zonefile:

/bin/sh -c ‘for i in /etc/ssh/ssh_host_*; do ssh-keygen -r HOSTNAME -f $i; done’

If you’re using openssh-portable under FreeBSD, change the /etc/ssh/ path above to be /usr/local/etc/ssh/

The HOSTNAME part of the above command line is cosmetic and used purely for the DNS output.

Ignore any ‘unsupported algorithm’ message – this is due to certain keytypes not being supported in SSHFP for now.

Prefer IPv6 over IPv4 in FreeBSD

To tell your FreeBSD installation to prefer IPv6 over IPv4, add the following line to your rc.conf file:


And you can activate it without a reboot by typing:

service ip6addrctl start

This instructs FreeBSD to prefer IPv6 over IPv4 where possible (and falling back to IPv4 when needed)

Using a swapfile in FreeBSD10+

Configuring a swap file in FreeBSD 10 changed to handle some internal issues and race conditions.

Swapfiles are useful if you need to add extra swap space to an existing installation, or if you want to add swap space protected by ZFS pools.


First, create the swapfile on the disk.  In the example below, i’m creating a 4GB swapfile called /swapfile.dat:

dd if=/dev/zero of=/swapfile.dat bs=1m count=4096
chmod 0600 /swapfile.dat

Now we’ve created the swapfile, we need to tell FreeBSD to mount it on startup.

Edit (or create) your /etc/fstab file with your favourite editor, and add the following line:

md99        none        swap        sw,file=/swapfile.dat        0        0

This will instruct the boot sequence to create a pseudo node called /dev/md99 which will be mapped to /swapfile.dat and mounted as swap.

If you wanted multiple swapfiles (perhaps on multiple pools), repeat the above steps and change the md99 to md98 etc. for each swapfile.

A few quick thoughts about FreeBSD 10 installer

The new FreeBSD 10 installer makes a lot of things easier (such as installing ZFS or encrypted filesystems), but there’s a few quick hints I have:


If installing using a UFS fileystem type (the default), I prefer to re-partition so that the swap space is at the start of the disk instead of the end.

This makes it easiest to expand the root filesystem later (especially if you’re installing onto a virtual machine)


If installing using ZFS and using multiple disks the installer will spread your swap partition across all disks – however these will NOT be protected by ZFS.

If you have a disk failure, any swap space on that disk will disappear and your machine will potentially crash and reboot.

I prefer to set the swap space to “0G” which causes the installer not to create a swap partition, and then I configure a swap file instead (i’ll post another blog post about creating swap files in FreeBSD 10+) which is protected by the ZFS subsystem.

This doesn’t apply if you’re installing onto a single drive (or a hardware raid array)


If you want any kind of performance from encrypted ZFS, make sure your CPU supports the ‘AESNI’ (or AES New Instructions) flag – it really makes a huge difference to the speed achieved.

Increase capacity of FreeBSD ZFS array by replacing disks

ZFS arrays do not support what you may know as RAID level expansion.

There are two ways to increase the capacity of a ZFS pool… either add more disks to the pool (e.g. 3 more disks in RAIDZ1), or replace all the existing disks with larger ones… this is the method discussed here today.  These instructions assume that you have followed my installation guide for ZFS.  If you have varied from that guide at all, you may need to vary the instructions below.  I am not responsible for any data loss by following any of these instructions!

NOTE: you can only increase the size of a mirror or raidz1/2/3 pool using this method.

I am replacing the 4 x 3TB disks in my storage array for 4 x 4TB disks.  This is a time consuming process and is risky if using mirror/RAIDZ1 (as you have to degrade the array !) – if you do not have full backups of the contents, do so at your own risk.  (if you’re using RAIDZ2 then you’re just at reduced resilience and a little safer)

First, we want to make sure that the autoexpand option is enabled, this can be run at any time with the following command:

zpool set autoexpand=on zroot

Next, check the status of your ZFS pool to make sure it is healthy… here’s the command and the output from my array:

zpool status

pool: zroot
state: ONLINE
scan: scrub repaired 0 in 6h38m with 0 errors on Thu Nov  8 16:06:21 2012

zroot       ONLINE       0     0     0
raidz1-0  ONLINE       0     0     0
ada0p2  ONLINE       0     0     0
ada1p2  ONLINE       0     0     0
ada2p2  ONLINE       0     0     0
ada3p2  ONLINE       0     0     0

errors: No known data errors

As you can see, my array consists of 4 members (ada0p2 through ada3p2) and is currently healthy.  We’re good to proceed!

First we shutdown the machine, and replace one of the disks… I prefer to start with the last disk and work backwards so i’m going to replaceada3… Once replaced, start the machine up again.

Now we can confirm that the disk is missing (and confirm which one) as follows… (command and output listed):

zpool status

pool: zroot
status: One or more devices has been removed by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using ‘zpool online’ or replace the device with
‘zpool replace’.
scan: scrub repaired 0 in 6h38m with 0 errors on Thu Nov  8 16:06:21 2012

NAME                     STATE     READ WRITE CKSUM
zroot                    DEGRADED     0     0     0
raidz1-0               DEGRADED     0     0     0
ada0p2               ONLINE       0     0     0
ada1p2               ONLINE       0     0     0
ada2p2               ONLINE       0     0     0
5075744959138230672  REMOVED      0     0     0  was /dev/ada3p2

errors: No known data errors

You can see that my ada3p2 device is now missing, and the array is degraded (it will run slower while degraded, but no data loss unless another disk fails during this long process)

Now we need to partition the newly installed ada3 disk so that it is bootable and contains a large ZFS partition for us to use… commands as follows:

gpart create -s gpt ada3
gpart add -s 128 -t freebsd-boot ada3
gpart add -t freebsd-zfs -l disk3 ada3
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada3

The above creates a GPT partition table, adds a small boot loader parition and the remainder of the disk for ZFS.  It then installs the boot loader into the small partition.

We are now ready to re-add the disk into the ZFS pool.  This will trigger an auto-resilver of the disks (a rebuild of the disk)…

zpool replace zroot ada3p2 /dev/ada3p2

This command takes a little while to process, so be patient.  The resilver stage can take a long time (it depends how much data you have on the pool, how many disks are in it and how fast you can read from them!)

You can check on the status of the rebuild with the following command:

zpool status zroot

Here’s an example output so you know what to look for:

  pool: zroot
status: One or more devices is currently being resilvered.  The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Wed Nov 14 18:34:57 2012
16.8G scanned out of 6.59T at 116M/s, 16h30m to go
4.19G resilvered, 0.25% done

NAME                       STATE     READ WRITE CKSUM
zroot                      DEGRADED     0     0     0
raidz1-0                 DEGRADED     0     0     0
ada0p2                 ONLINE       0     0     0
ada1p2                 ONLINE       0     0     0
ada2p2                 ONLINE       0     0     0
replacing-3            REMOVED      0     0     0
5075744959138230672  REMOVED      0     0     0  was /dev/ada3p2
ada3p2               ONLINE       0     0     0  (resilvering)

errors: No known data errors

Once the disk has been fully reconstructed, the array will be healthy again (like at the start), and you can move onto the next disk.  Repeat until all disks have been replaced and resilvered.

You will only see the new space once all the disks have finished resilvering.

I will note again that your array is vulnerable if a mirror or raidz1 configuration while doing this.  If a 2nd disk fails during the resilver of any of the disks and you’re doing a mirror or raidz1 pool, you will LOSE your data.

Full system backups for FreeBSD systems using ZFS

Assuming you have created your ZFS FreeBSD system using the instructions on my site, here is how to do full system backups to an extra attached disk.

You can adjust these instructions if you need to store the backup remotely – but they are out of scope of this post.

First, in case you haven’t already… here is how to format/dev/da1 as a dedicated ZFS backup drive.  You can configure the backup drive however you want (it doesn’t even need to be ZFS-based) but you will also have to adjust these instructions accordingly to restore too.

gpart destroy -F da1
dd if=/dev/zero of=/dev/da1 bs=1m count=128
zpool create zbackup /dev/da1
zfs set mountpoint=/backup zbackup

The above will destroy any existing data on /dev/da1, and create a ZFS filesystem which is mounted at /backup.

Next we snapshot all filesystems under zroot and send them to a gzip’d file on the backup medium.  We then release the snapshot:

zfs snapshot -r zroot@backup
zfs send -Rv zroot@backup | gzip > /backup/full-system-backup.zfs.gz
zfs destroy -r zroot@backup

This is all you need to do on the live system.

If the worst happens, and you need to restore to a new system (or a freshly formatted one)…

Firstly, follow the original instructions up to and including the line “zfs set checksum=fletcher4 zroot”.

Next, we import the backup ZFS drive and mount it – then we use ZFS receive to restore the filesystem and all its dependants:

zpool import -f zbackup
zfs set mountpoint=/boot/zfs/backup zbackup
zfs mount zbackup
gunzip -c /boot/zfs/backup/full-system-backup.zfs.gz | zfs receive -vdF zroot

Now we need to unmount the backup drive, and mount the original root ZFS so we can re-create the cache file (the system will not boot without the correct cache file):

zpool export zbackup
zfs set mountpoint=/boot/zfs/zroot zroot
cd /boot/zfs
zpool export zroot && zpool import zroot
cp /boot/zfs/zpool.cache /boot/zfs/zroot/boot/zfs/zpool.cache
zfs unmount -a
zfs set mountpoint=legacy zroot

This will reboot the system in its original state.  If you want to re-mount your backup medium, it will need to be re-imported and mounted:

zpool import -f zbackup
zfs set mountpoint=/backup zbackup

That’s all there is to it.  A fully working disaster recovery solution.


Fix choppy audio under PulseAudio in Gnome on FreeBSD

If you have “choppy” audio in Gnome on FreeBSD, it’s likely caused by PulseAudio.  My definition of choppy audio is where you get very brief gaps of silence as you play audio – a sort of slow crackle.

As root, edit the file /usr/local/etc/pulse/daemon.conf and go to the bottom of the file.  There are 2 lines currently hashed out, modify them to read:

default-fragments = 8
default-fragment-size-msec = 5

then you need to restart pulseaudio.  As the username you have logged into Gnome with (not root!), type:

/usr/local/bin/pulseaudio –kill

This should sort things for you (or at least, it did for me)

Reducing the effects of a DNS Amplification attack on authorative DNS server with FreeBSD and ipfw

DNS is an ideal transport for amplification attacks (and even moreso with DNSSEC/IPv6 bringing EDNS extensions to DNS).  A forged 60 byte DNS request packet can generate up to a 4096 byte response packet via UDP.

You can tell if you are being used in an attack by a massive increase in outbound DNS responses on your authorative DNS server.  Your DNS will receive a packet and return the data for the query to the IP it believes it came from. There’s no handshaking in UDP so it will just trust the sender address and blindly reply to it.  This is normal operation.

To reduce the effects of this in FreeBSD, we can apply a bandwidth limiter on our responses on a per-IP basis.

First, we need to ensure the dummynet module is loaded:

kldload dummynet

To make it load at boot time, edit your /boot/loader.conf file and add the following line:


Next, edit your ipfw script (or create a new one).  We need to define the pipe as follows:

/sbin/ipfw pipe 1 config mask dst-ip 0xffffffff bw 256Kbit/s

This will limit dns traffic to each IP to 256kbits/sec (about 32KB/sec) when we use it.  Dummynet granularity isn’t all that great so don’t expect it to be accurate – but the fact we are limiting it is enough.  NOTE: your setup may require a different figure depending on your normal DNS load.

Next, we need to apply it to all DNS responses from us:

/sbin/ipfw add 01000 pipe 1 udp from me 53 to any out

You may need to tweak the rule number (01000) to fit into your setup.  It needs to ideally be as high up in your firewall as possible.

That’s all you need to do.  The above will add additional load to your server in terms of CPU – but that’s the price you have to pay.

You can, of course, use the above settings (slightly tweaked) on a firewall in front of any DNS server.

Personally, to reduce load on my nameservers I also add a second pipe for incoming bandwidth on a per IP basis.  For this, you need to use src-ip mask instead of dst-ip and apply it to incoming traffic.