Monthly Archives: June 2016

CARP (IPv4 and IPv6) in FreeBSD 10+ (for failover IPs)

Here’s how to configure CARP (common address redundancy protocol) to failover between two machines for both IPv4 and IPv6 IPs.  This requires FreeBSD 10+.

NOTE: Your switch requires you to allow multicast and IGMP.  Most switches do this, but VMware virtual switches generally do not unless you disable all security on them (not advised) – FreeBSD bhyve VMs work fine so long as the host’s upstream switch supports it.

We have a machine that should normally be the master of the pair.  It has non-shared IPs of and 2001:db8::101/64

We have a machine that should normally be the slave of the pair.  It has non-shared IPs of and 2001:db8::102/64

The two machines will share the IPs and 2001:db8::1 – the master will respond to these IPs, and the slave will take over if the master disappears.  This ordinarily happens in under a second.  A few packets are lost during the failover but TCP retransmits etc take care of this and it is usually unnoticable.

Each CARP setup requires a VHID (this determines the MAC address used so it should be unique on the network) and a password to protect announcements.  We’ll use VHID 1 for the IPv4 setup and VHID 2 for the IPv6 setup.  We will use a password of testpass for demonstration purposes.

The only difference between the two machine setups is the advskew value.  This decides the priority of each machine.  The lower the number, the higher the priority.  The master will be the machine with the lowest advskew

First, we need to load the kernel module on each machine.  In /boot/loader.conf add:


Now we need to add the network configuration.  Setup the server as normal with its static IPs, then we can add additional IPs to the network card for the CARP configuration.  In our example, we are using the network interface vtnet0 (a bhyve vm).

Add the following lines (modified for your use) to /etc/rc.conf:

ifconfig_vtnet0_alias0=”inet vhid 1 advskew 100 pass testpass”
ifconfig_vtnet0_alias1=”inet6 2001:db8::1 prefixlen 64 vhid 2 advskew 100 pass testpass”

As mentioned before, setup the slave machine identically except the two above lines would have advskew 200 to ensure they are lower priority.

Reboot the machines and login to check with ifconfig vtnet0 command.

Here’s the output on the master machine:

vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 58:9c:fc:05:c4:0d
inet6 fe80::5a9c:fcff:fe05:c40d%vtnet0 prefixlen 64 scopeid 0x1
inet netmask 0xffffff00 broadcast vhid 1
inet6 2001:db8::1 prefixlen 64 vhid 2
inet6 2001:db8::101 prefixlen 64
inet netmask 0xffffff00 broadcast
media: Ethernet 10Gbase-T <full-duplex>
status: active
carp: MASTER vhid 1 advbase 1 advskew 100
carp: MASTER vhid 2 advbase 1 advskew 100

You can see at the bottom of the output, the carp status shows that the machine is in MASTER state.

Here’s the output on the slave machine:

vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 58:9c:fc:04:48:35
inet6 fe80::5a9c:fcff:fe04:4835%vtnet0 prefixlen 64 scopeid 0x1
inet netmask 0xffffff00 broadcast vhid 1
inet6 2001:db8::1 prefixlen 64 vhid 2
inet6 2001:db8::102 prefixlen 64
inet netmask 0xffffff00 broadcast
media: Ethernet 10Gbase-T <full-duplex>
status: active
carp: BACKUP vhid 1 advbase 1 advskew 200
carp: BACKUP vhid 2 advbase 1 advskew 200

Here you can see the carp status is BACKUP.

If you were to shutdown or reboot the master machine, the slave machine would change to MASTER status.

This is also logged into /var/log/messages:

Jun  5 22:34:51 carp-slave kernel: carp: VHID 1@vtnet0: BACKUP -> MASTER (master down)
Jun  5 22:34:51 carp-slave kernel: carp: VHID 2@vtnet0: BACKUP -> MASTER (master down)

It’s also possible to monitor this and trigger a script upon changes using devd – but that is out of scope for this article.  If there’s enough interest, I might do a further article.

Add DKIM Signing to FreeBSD Servers

In today’s very spammy internet, it becomes increasingly important to verify the source of emails are valid (this is especially true if you want to send emails to gmail or!)

One method is to add DKIM signing to outbound emails.

This will detail how to add DKIM signing to a stock FreeBSD server (e.g. a webserver) using sendmail.

First, we need to install the DKIM package:

pkg install opendkim

Now, we need to edit /usr/local/etc/mail/opendkim.conf and change the following lines:

Canonicalization  relaxed/simple
KeyTable          refile:/usr/local/etc/mail/opendkim.keytable
LogWhy            yes
Selector          myselector
SigningTable      refile:/usr/local/etc/mail/opendkim.signingtable
Socket            local:/var/run/dkim/opendkim.sock
SyslogSuccess     Yes
UserID            mailnull:mailnull

You can change myselector to anything you wish to use.  This will form part of the DNS entry required for signature verification so it should be fairly unique (this allows you to have different DKIM signing keys for different servers/services.  I often call them based on the server name.

And we need to do a few steps to setup things:

touch /usr/local/etc/mail/opendkim.keytable
touch /usr/local/etc/mail/opendkim.signingtable
mkdir -p /var/run/dkim /var/db/dkim
chown mailnull:mailnull /var/run/dkim /var/db/dkim
chmod 0700 /var/run/dkim

This creates some files and folders, and sets the necessary permissions.

Now we need to tell opendkim to start on boot, add to /etc/rc.conf the following:


and now we can start with:

service milter-opendkim start

If all is well, you’ll see two copies of opendkim running in your process list.

Finally, we need to tell sendmail to sign emails that go out.

cd /etc/mail

Now edit /etc/mail/ and add the following line just before the ‘MAILER’ lines:

INPUT_MAIL_FILTER(`opendkim’, `S=local:/var/run/dkim/opendkim.sock’)dnl

Now we can build a new sendmail configuration file and activate it:

cd /etc/mail
service sendmail restart

Now, at this point, nothing has really changed as we haven’t given opendkim any domains to sign.

I wrote a handle script for generating new DKIM configuration… Add the following lines to a script called generate-dkim somewhere in your path:



if [ -z ${1} ]; then
echo “Usage: ${0} <domain>”
exit 1

if [ -f /var/db/dkim/${DOMAIN}/${SELECTOR}.txt ]; then
echo “=> ERROR: found DKIM keys for ${DOMAIN} already”
/bin/cat /var/db/dkim/${DOMAIN}/${SELECTOR}.txt
exit 1

echo “==> Generating DKIM key for ${DOMAIN}…”
/bin/mkdir /var/db/dkim/${DOMAIN}
/usr/sbin/chown mailnull:mailnull /var/db/dkim/${DOMAIN}
/usr/local/sbin/opendkim-genkey -a -b 2048 -d ${DOMAIN} -D /var/db/dkim/${DOMAIN} -s ${SELECTOR}
/usr/sbin/chown mailnull:mailnull /var/db/dkim/${DOMAIN}/*
echo “==> Done”
echo “==> Updating config to enable DKIM signing…”
echo “${SELECTOR}._domainkey.${DOMAIN}      ${DOMAIN}:${SELECTOR}:/var/db/dkim/${DOMAIN}/${SELECTOR}.private” >> /usr/local/etc/mail/opendkim.keytable
echo “*@${DOMAIN}       ${SELECTOR}._domainkey.${DOMAIN}” >> /usr/local/etc/mail/opendkim.signingtable
echo “==> Done”
echo “==> Reloading OpenDKIM configuration…”
/bin/pkill -USR1 -F /var/run/milteropendkim/pid
echo “==> Done”
echo “DKIM DNS entry to add is below:”
/bin/cat /var/db/dkim/${DOMAIN}/${SELECTOR}.txt

If you chose a different ‘selector’ in the earlier step, remember to change the SELECTOR= line above to match or your signing wont work!

And dont forget to make it executable:

chmod a+x generate-dkim

Now, whenever we want to add a new signing domain, for example, we can run


This will generate a new key, add it to the opendkim configuration files, reload opendkim, and output the required DNS entry to you.

If you’ve previously added the domain, it will just show the DNS entry to you without modifying anything.

Now any emails sent out with a sender of * will be DKIM signed using the key generated in the step above.

You MUST add the DNS entry to the domain’s DNS zone for DKIM signing to work.  If the receiving mailserver can’t lookup the DNS entry then it can’t verify your email!

If you ever need to remove a domain from signing, you have to do this manually…

Remove it from /usr/local/etc/mail/opendkim.keytable and /usr/local/etc/mail/opendkim.signingtable (one line in each file), then there’s a folder in /var/db/dkim/ named after the domain which needs to be removed.

For the best chance of delivering to the world’s biggest spammers (gmail/outlook/aol/yahoo/etc), I would recommend SPF, DKIM and DMARC!

NetFlow v9 Exporting from FreeBSD routers/firewalls

Sometimes it’s useful to have netflow reporting from a FreeBSD router or firewall.

For this example, we’ll assume you want to export flows on the igb0 network interface with netflow v9 packets to the flow collector at port 1234 (UDP) – v9 packets are useful as they can contain IPv6 flows.

First, we need to load some kernel modules at boot time – to do this we need to add the following lines to your /boot/loader.conf file:


You can load these now without having to reboot by typing:

kldload netgraph ng_ether ng_socket ng_ksocket ng_tee ng_netflow

Next we need a startup script to configure netflow.  Create a file with the following contents called /usr/local/etc/rc.d/

case “$1” in
/usr/sbin/ngctl -f- <<-SEQ
mkpeer igb0: netflow lower iface0
name igb0:lower netflow
connect igb0: netflow: upper out0
mkpeer netflow: ksocket export9 inet/dgram/udp
msg netflow:export9 connect inet/
echo “Please specify ‘start’ or ‘stop'”

And ensure it runs at boot time with:

chmod a+x /usr/local/etc/rc.d/

and now you can start it:

/usr/local/etc/rc.d/ start

You should start seeing netflow packets arrive at your collector.

FreeBSD doesn’t have ASN information in its kernel, so the netflow packets will only report flows with IPs and port numbers present.

Link Failover in FreeBSD (without requiring switch configuration)

It’s often useful to have simple link failover… for example a server connected to two discrete switches.  In this scenario, LACP is of no use as it can’t span discrete switches.

Link failover in FreeBSD will failover to a second switchport in the event of a link down event of the first.  (it does require the link itself to drop)

The MAC address of the failover pseudo network card will be the MAC address of the first network card added to the card.

First, we need to ensure the link aggregation module (failover is part of this module) is started at boot, so edit /boot/loader.conf and add the following line:


Now we need to configure the port… in this example we will bond igb0 and bge0 together into a failover network, with igb0 being the primary port.  We will assign the IP to the interface.

Add the following to /etc/rc.conf:

ifconfig_lagg0=”laggproto failover laggport igb0 laggport bge0 up”

Now it’s possible to put the IP assignment on the same line as the failover definitions but I think it looks cleaner separate – so I used an alias instead.

You can use a lagg port in the same way as a normal network port (e.g. clone interfaces from it for vlans etc)

On the FreeBSD server, you can check the failover with ‘ifconfig lagg0‘ command:

        laggproto failover lagghash l2,l3,l4
laggport: bge0 flags=0<>
laggport: igb0 flags=5<MASTER,ACTIVE>

You can see both ports are connected, and igb0 is the master port, and active port.

Data received on any non-active port will be discarded, and data from the server outbound will only be sent to active ports.

You can, of course, have more than 2 network ports in a failover configuration – but only one will ever be active.

It can also be useful on a laptop environment so failover between wireless and wired connections with the wired connection being preferred if plugged in, and wireless if not.