Speed up FreeBSD reboots by 7 seconds

Leave a Reply

Ok, 7 seconds doesn’t seem like much… but if you’re rebooting a live server then every second of downtime counts – so why not?

This is a simple change that reduces the timer on the initial FreeBSD menu from 10 seconds to 3 seconds.  (I don’t recommend reducing it below 3 seconds, as it may make it more difficult for you to use the menu should you ever need to!)

Ok, so you need to edit the /boot/loader.conf file (or create it if it doesn’t exist) and add the following line:

autoboot_delay=”3″

and that’s all you need to do.  Next reboot, the timer will start at 3 instead of 10 and you have a reboot that takes 7 seconds less than before.

Closure of Nominet Registrar Account ‘AUSTIN’

2 Replies

On Friday 22nd January 2016, I formally gave 30 day notice to Nominet for the closure of my registrar account.  This post details the reasons for this.

Nominet contacted me stating that they were going to link my registrar account with that of my employer due to their “connected persons” section of their AUP.  I should note that I had never seen this section before, so it must have been added since I became a registrar.

I contacted them detailing that my personal account was for the express purposes of keeping my private and business life separate… including that I go as far as to have provider independant IP address space, my own Autonomous System, BGP routers and servers – all separate from my employer.  I stated the fact that I had been (save for a short break) a Nominet member for longer than I had been an employee of my current employer too.

In response I received:

I have spoken with our legal team with regards to your information, however, they have advised that we need to link the services in regards to the acceptable use policies (Anti-avoidance and Connected Persons link below).

Obviously my employer would be rather upset if anything I could do in my personal capacity could affect them in any way (it’s a fair expectation of an employee), so I was left with no alternative but to serve notice on my registrar account with the minimum possible term (30 days)

Nominet’s AUP (whenever it was added/updated) contains the following description of connected persons:

A person is ‘connected’ to another person if:

  1. they are the same person, have the same Nominet account, or have connected memberships under the voting rights policy;
  2. they make any declaration that they are connected or when challenged by us they fail to make a declaration in the reasonably required legal form that they are not connected;
  3. they have social, family, ownership or business links (directly or indirectly) which mean that they either:
    1. do not appear to operate truly independently of one another, or
    2. it could reasonably be assumed that they will not operate truly independently of one another; or
  4. the object or effect of their activities is such that we reasonably think that not linking them would compromise one or more of the AUP Principles.

For the purposes of (2) we may decide that two parties are not connected even if one declares that they are, if the other disputes this and our investigations support the denial (so, for example, you cannot just declare that you are linked to another person and then use all their limit up to disrupt their business).

For the purposes of (3) there is a strong presumption that:

  • group companies (or other businesses) are connected to other businesses in the group;
  • a company and its employees/officers/partners are connected to one another;
  • members of the same family group are connect to each other (including in-laws, co-habitees, civil partners, adopted children and others who may not have a relationship by blood but who are part of the family group); and
  • a business owned, run or ultimately controlled by one member of a family is connected to a business owned, run or ultimately controlled by another member of the same family.

The final decision rests with us. When deciding whether to link people under (2) to (4) above, we will take into account the AUP principles.

I’m unhappy with so many things in the above snippet that I don’t even know where to start…

In short, Nominet have lost my annual membership fee (i’m sure they won’t be concerned about this given that they make more profit than their non-profit status allows anyway), but also one of their last registrars who had implemented their entire feature set.

I have recommended to any people with domains on my account to migrate to another registrar ASAP.  For those without a preferred registrar, I have suggested PortFast as they appear to have a similar feature set to that I had offerred.

This follows a disturbing trend within Nominet which I wont go into in this post.  Feel free to use your favourite search engine to lookup recent events at Nominet if you want to read further.

In an age where ICANN are dishing out new TLDs constantly, you would expect Nominet to be trying to strengthen the .uk brand but instead recently it seems to be trying to harm it.

In light of this, I have been considering migrating my main domain name away from the .uk namespace to one of the more level-headed registries.

HTTP Public Key Pinning (HPKP)

Leave a Reply

As we move into an more secure environment, simply have HTTPS isn’t sufficient.  Many cases of forged SSL certificates for man-in-the-middle attacks have appeared recently.

These can be obtained through deception or hacking attemps on SSL CAs.

One method to help combat this is HTTP Public Key Pinning (HPKP) – this is where the webserver can communicate to the web browser an allowed certificate path, with a sufficiently long expiry time to be of use to return visitors.  It’s not perfect or ideal, but it’s better than nothing.

This post does NOT detail the most secure method of HPKP but a compromise in terms of usability vs security.  It will ensure that an unauthorised certificate would need to have been generated by one of your chosen SSL CAs only.  The more secure method of having dual online and offline keys is out of scope for this blog post and a more advanced topic.

The way this works is to publish a header in the HTTPS response detailing the allowed hashes of public keys of webserver keys or keysigner keys that are permitted to be in the certificate chain.  This is in addition to the normal process of validation of the full SSL certificate chain.

BEWARE: misconfiguration of this header can lead people to be unable to view your site for MONTHS so be careful!

First let’s look how to obtain the SHA-256 hash needed for a certificate.  We use OpenSSL to do this based on the .crt file… Let’s assume the certificate is in the cert.crt file…

openssl x509 -pubkey < cert.crt | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary| openssl enc -base64

This will provide a Base64 encoded string which we can use later.

Now we provide the information to the webserver… i’ll detail Apache 2.4 here, but other servers will have similar methods of adding header outputs.  We add the following into the VirtualHost block for the SSL site:

Header always set Public-Key-Pins “pin-sha256=\”HASH\”; pin-sha256=\”HASH\”; max-age=30;”

where the Base64 has generated previously replaces the first ‘HASH’.

HPKP requires a minimum of 2 hashes to be present, including one hash that is NOT present in the certificate chain.  This is to encourage you to ensure you have a backup plan.

In my examples, I use the hash of my SSL public key and the hashes of my chosen primary and secondary SSL certificate authority’s intermediate certificates.

In my case, I use StartSSL’s class 2 intermediate and RapidSSL’s SHA-2 under SHA-2 root intermediate.  To generate their hashes, use the above OpenSSL method on their CA certificates.

This means that I can use my existing key/csr with any SSL CA, or I can generate a new key with my primary or secondary CA.  It also satisfies the requirement for one of the hashes to not exist (my current certificate is not present via my secondary CA)

The max-age above is set to 30 seconds for testing purposes… once you’re completely happy with your choice, this should be raised to a much longer figure.  I use a 30 day period (2592000 seconds)

A good method of testing is to use Qualys SSLlabs tester at https://www.ssllabs.com/ssltest/ – this will show you the pinned list, including any currently in the chain in a different colour.

SSH Fingerprints in DNS

Leave a Reply

Here’s how to add SSHFP records to DNS.  You need to have DNSSEC signed zones for this to work.

To generate the records, simply use the following command against your host public keys and copy them into your DNS zonefile:

/bin/sh -c ‘for i in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -r HOSTNAME -f $i; done’

If you’re using openssh-portable under FreeBSD, change the /etc/ssh/ path above to be /usr/local/etc/ssh/

The HOSTNAME part of the above command line is cosmetic and used purely for the DNS output.

Ignore any ‘unsupported algorithm’ message – this is due to certain keytypes not being supported in SSHFP for now.

Prefer IPv6 over IPv4 in FreeBSD

Leave a Reply

To tell your FreeBSD installation to prefer IPv6 over IPv4, add the following line to your rc.conf file:

ip6addrctl_policy=”ipv6_prefer”

And you can activate it without a reboot by typing:

service ip6addrctl start

This instructs FreeBSD to prefer IPv6 over IPv4 where possible (and falling back to IPv4 when needed)

Activate IPFW without kernel compile in FreeBSD

Leave a Reply

To activate IPFW in a FreeBSD machine without having to recompile a kernel, you can add the following lines to /boot/loader.conf (you may need to create it if it doesn’t exist):

net.inet.ip.fw.default_to_accept=”1″
ipfw_load=”YES”

On the next reboot, your machine will have ipfw loaded and will default to accept.  (if you had run kldload ipfw the default rule would be deny and you’d be locked out of your machine so be warned!)

Using a swapfile in FreeBSD10+

Leave a Reply

Configuring a swap file in FreeBSD 10 changed to handle some internal issues and race conditions.

Swapfiles are useful if you need to add extra swap space to an existing installation, or if you want to add swap space protected by ZFS pools.

 

First, create the swapfile on the disk.  In the example below, i’m creating a 4GB swapfile called /swapfile.dat:

dd if=/dev/zero of=/swapfile.dat bs=1m count=4096
chmod 0600 /swapfile.dat

Now we’ve created the swapfile, we need to tell FreeBSD to mount it on startup.

Edit (or create) your /etc/fstab file with your favourite editor, and add the following line:

md99        none        swap        sw,file=/swapfile.dat        0        0

This will instruct the boot sequence to create a pseudo node called /dev/md99 which will be mapped to /swapfile.dat and mounted as swap.

If you wanted multiple swapfiles (perhaps on multiple pools), repeat the above steps and change the md99 to md98 etc. for each swapfile.

A few quick thoughts about FreeBSD 10 installer

Leave a Reply

The new FreeBSD 10 installer makes a lot of things easier (such as installing ZFS or encrypted filesystems), but there’s a few quick hints I have:

 

If installing using a UFS fileystem type (the default), I prefer to re-partition so that the swap space is at the start of the disk instead of the end.

This makes it easiest to expand the root filesystem later (especially if you’re installing onto a virtual machine)

 

If installing using ZFS and using multiple disks the installer will spread your swap partition across all disks – however these will NOT be protected by ZFS.

If you have a disk failure, any swap space on that disk will disappear and your machine will potentially crash and reboot.

I prefer to set the swap space to “0G” which causes the installer not to create a swap partition, and then I configure a swap file instead (i’ll post another blog post about creating swap files in FreeBSD 10+) which is protected by the ZFS subsystem.

This doesn’t apply if you’re installing onto a single drive (or a hardware raid array)

 

If you want any kind of performance from encrypted ZFS, make sure your CPU supports the ‘AESNI’ (or AES New Instructions) flag – it really makes a huge difference to the speed achieved.

Increase capacity of FreeBSD ZFS array by replacing disks

29 Replies

ZFS arrays do not support what you may know as RAID level expansion.

There are two ways to increase the capacity of a ZFS pool… either add more disks to the pool (e.g. 3 more disks in RAIDZ1), or replace all the existing disks with larger ones… this is the method discussed here today.  These instructions assume that you have followed my installation guide for ZFS.  If you have varied from that guide at all, you may need to vary the instructions below.  I am not responsible for any data loss by following any of these instructions!

NOTE: you can only increase the size of a mirror or raidz1/2/3 pool using this method.

I am replacing the 4 x 3TB disks in my storage array for 4 x 4TB disks.  This is a time consuming process and is risky if using mirror/RAIDZ1 (as you have to degrade the array !) – if you do not have full backups of the contents, do so at your own risk.  (if you’re using RAIDZ2 then you’re just at reduced resilience and a little safer)

First, we want to make sure that the autoexpand option is enabled, this can be run at any time with the following command:

zpool set autoexpand=on zroot

Next, check the status of your ZFS pool to make sure it is healthy… here’s the command and the output from my array:

zpool status

pool: zroot
state: ONLINE
scan: scrub repaired 0 in 6h38m with 0 errors on Thu Nov  8 16:06:21 2012
config:

NAME        STATE     READ WRITE CKSUM
zroot       ONLINE       0     0     0
raidz1-0  ONLINE       0     0     0
ada0p2  ONLINE       0     0     0
ada1p2  ONLINE       0     0     0
ada2p2  ONLINE       0     0     0
ada3p2  ONLINE       0     0     0

errors: No known data errors

As you can see, my array consists of 4 members (ada0p2 through ada3p2) and is currently healthy.  We’re good to proceed!

First we shutdown the machine, and replace one of the disks… I prefer to start with the last disk and work backwards so i’m going to replaceada3… Once replaced, start the machine up again.

Now we can confirm that the disk is missing (and confirm which one) as follows… (command and output listed):

zpool status

pool: zroot
state: DEGRADED
status: One or more devices has been removed by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using ‘zpool online’ or replace the device with
‘zpool replace’.
scan: scrub repaired 0 in 6h38m with 0 errors on Thu Nov  8 16:06:21 2012
config:

NAME                     STATE     READ WRITE CKSUM
zroot                    DEGRADED     0     0     0
raidz1-0               DEGRADED     0     0     0
ada0p2               ONLINE       0     0     0
ada1p2               ONLINE       0     0     0
ada2p2               ONLINE       0     0     0
5075744959138230672  REMOVED      0     0     0  was /dev/ada3p2

errors: No known data errors

You can see that my ada3p2 device is now missing, and the array is degraded (it will run slower while degraded, but no data loss unless another disk fails during this long process)

Now we need to partition the newly installed ada3 disk so that it is bootable and contains a large ZFS partition for us to use… commands as follows:

gpart create -s gpt ada3
gpart add -s 128 -t freebsd-boot ada3
gpart add -t freebsd-zfs -l disk3 ada3
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada3

The above creates a GPT partition table, adds a small boot loader parition and the remainder of the disk for ZFS.  It then installs the boot loader into the small partition.

We are now ready to re-add the disk into the ZFS pool.  This will trigger an auto-resilver of the disks (a rebuild of the disk)…

zpool replace zroot ada3p2 /dev/ada3p2

This command takes a little while to process, so be patient.  The resilver stage can take a long time (it depends how much data you have on the pool, how many disks are in it and how fast you can read from them!)

You can check on the status of the rebuild with the following command:

zpool status zroot

Here’s an example output so you know what to look for:

  pool: zroot
state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Wed Nov 14 18:34:57 2012
16.8G scanned out of 6.59T at 116M/s, 16h30m to go
4.19G resilvered, 0.25% done
config:

NAME                       STATE     READ WRITE CKSUM
zroot                      DEGRADED     0     0     0
raidz1-0                 DEGRADED     0     0     0
ada0p2                 ONLINE       0     0     0
ada1p2                 ONLINE       0     0     0
ada2p2                 ONLINE       0     0     0
replacing-3            REMOVED      0     0     0
5075744959138230672  REMOVED      0     0     0  was /dev/ada3p2
ada3p2               ONLINE       0     0     0  (resilvering)

errors: No known data errors

Once the disk has been fully reconstructed, the array will be healthy again (like at the start), and you can move onto the next disk.  Repeat until all disks have been replaced and resilvered.

You will only see the new space once all the disks have finished resilvering.

I will note again that your array is vulnerable if a mirror or raidz1 configuration while doing this.  If a 2nd disk fails during the resilver of any of the disks and you’re doing a mirror or raidz1 pool, you will LOSE your data.

Full system backups for FreeBSD systems using ZFS

52 Replies

Assuming you have created your ZFS FreeBSD system using the instructions on my site, here is how to do full system backups to an extra attached disk.

You can adjust these instructions if you need to store the backup remotely – but they are out of scope of this post.

First, in case you haven’t already… here is how to format/dev/da1 as a dedicated ZFS backup drive.  You can configure the backup drive however you want (it doesn’t even need to be ZFS-based) but you will also have to adjust these instructions accordingly to restore too.

gpart destroy -F da1
dd if=/dev/zero of=/dev/da1 bs=1m count=128
zpool create zbackup /dev/da1
zfs set mountpoint=/backup zbackup

The above will destroy any existing data on /dev/da1, and create a ZFS filesystem which is mounted at /backup.

Next we snapshot all filesystems under zroot and send them to a gzip’d file on the backup medium.  We then release the snapshot:

zfs snapshot -r zroot@backup
zfs send -Rv zroot@backup | gzip > /backup/full-system-backup.zfs.gz
zfs destroy -r zroot@backup

This is all you need to do on the live system.

If the worst happens, and you need to restore to a new system (or a freshly formatted one)…

Firstly, follow the original instructions up to and including the line “zfs set checksum=fletcher4 zroot”.

Next, we import the backup ZFS drive and mount it – then we use ZFS receive to restore the filesystem and all its dependants:

zpool import -f zbackup
zfs set mountpoint=/boot/zfs/backup zbackup
zfs mount zbackup
gunzip -c /boot/zfs/backup/full-system-backup.zfs.gz | zfs receive -vdF zroot

Now we need to unmount the backup drive, and mount the original root ZFS so we can re-create the cache file (the system will not boot without the correct cache file):

zpool export zbackup
zfs set mountpoint=/boot/zfs/zroot zroot
cd /boot/zfs
zpool export zroot && zpool import zroot
cp /boot/zfs/zpool.cache /boot/zfs/zroot/boot/zfs/zpool.cache
zfs unmount -a
zfs set mountpoint=legacy zroot
reboot

This will reboot the system in its original state.  If you want to re-mount your backup medium, it will need to be re-imported and mounted:

zpool import -f zbackup
zfs set mountpoint=/backup zbackup

That’s all there is to it.  A fully working disaster recovery solution.