Category Archives: HowTo

Add Google Authenticator Style 2FA (Two Factor Auth) to FreeBSD SSH

1 Reply

In the current climate we are all concerned with security, so this article will show you how to enforce Google Authenticator style (there are many other apps other than Google’s!) two-factor authentication on FreeBSD servers SSH.

First, we need to install a package which includes the libraries and tools for the authentication:

pkg install oath-toolkit pam_per_user

You could also install this from ports if you prefer ports-based installations.

Next we will create a new token for a user (we’ll call the user twofa):

head -10 /dev/urandom | md5 | cut -b 1-30
oathtool –totp -v <hex code from above cmd>

in the example above, my random hex code was d1ac1a3d57deeefd6b83c3c340feea.

In the output from the oathtool command, the following line shows the code you need to add to your Google Authenticator style app… for me it was:

Base32 secret: 2GWBUPKX33XP224DYPBUB7XK

You can ignore the rest of the output.  If you wanted to use a QR code to scan in your app, this is the text you need to put in your QR generator:

otpauth://totp/<user>@<host>?secret=<base32 secret>

The “<user>@<host>” part of the text is the descriptive name shown in the app and doesn’t need to be in that format, it’s just a useful reminder!

Next we need to add the reference to map this user to this authenticator code.  Using your favourite editor, edit (or create if it’s the first user) the file: /usr/local/etc/users.oath and add:

HOTP/T30/6 <username> – <hex code>

Note: you use the hex code here, not the base32 secret.  For example:

HOTP/T30/6 twofa – d1ac1a3d57deeefd6b83c3c340feea

The “HOTP/T30/6” tells it that we’re using a TOTP code with 30 second rolling window that generates 6 digit codes (this is what Google Authenticator style apps will return to us)

This file needs to be secure, so make sure you set the permissions correctly:

chmod 0600 /usr/local/etc/users.oath

Next, we need to configure the “per-user” PAM module which allows us to selectively choose which users have to use TOTP auth.  Using your favourite editor, create the file: /usr/local/etc/pam_per_user.map and add:

twofa : sshd-oath
* : sshd-default

If you wanted to enforce 2FA for all users, change the last line from sshd-default to sshd-oath.  Again, we want this file to be secure, so set the permissions:

chmod 0600 /usr/local/etc/pam_per_user.map

So now we have the code added to our authenticator app, and the reference added to our config file… all we need to do is tell SSH to enforce it!

First, make a copy of your existing sshd file as the “default” (fallback, non-2FA) option, and also a template for our new “oath” copy:

cp /etc/pam.d/sshd /etc/pam.d/sshd-default
cp /etc/pam.d/sshd /etc/pam.d/sshd-oath

Using your favourite editor, edit the file: /etc/pam.d/sshd and comment out the following lines:

#auth sufficient pam_opie.so no_warn no_fake_prompts
#auth requisite pam_opieaccess.so no_warn allow_local
#auth sufficient pam_krb5.so no_warn try_first_pass
#auth sufficient pam_ssh.so no_warn try_first_pass
#auth required pam_unix.so no_warn try_first_pass

then add the following line after the last “auth” line:

auth required /usr/local/lib/security/pam_per_user.so.1 /usr/local/etc/pam_per_user.map

This tells PAM authentication for SSHD that we need to require a lookup via the per-user module.  Now edit the file: /etc/pam.d/sshd-oath and comment out the following lines:

#auth sufficient pam_opie.so no_warn no_fake_prompts
#auth requisite pam_opieaccess.so no_warn allow_local

then add the following line after the last “auth” line:

auth required /usr/local/lib/security/pam_oath.so usersfile=/usr/local/etc/users.oath

Finally, we have to set a few settings in SSHD so that it can prompt for the code (this isn’t strictly necessary as these are all the default options anyway, but sometimes it is best to enforce them in case they change in future versions)… edit the file /etc/ssh/sshd_config and add the following at the bottom of the file:

ChallengeResponseAuthentication yes
PasswordAuthentication no
UsePAM yes

This enforces the use of PAM which will enforce the use of the per-user and 2FA modules.  Finally we have to restart the SSH service:

service sshd restart

And if we now try to SSH to the server, we will see the following:

% ssh -l twofa 192.0.2.1
Password for twofa@totp-test.local:
One-time password (OATH) for `twofa’:
Password for twofa@totp-test.local:
One-time password (OATH) for `twofa’:
Last login: Mon Nov 6 19:57:56 2017 from test.local

I deliberately entered an invalid code the first time, so it re-prompted for a code.  The second time I entered a valid code from the authenticator and you can see that it allowed us access.

To add a new user token, simply repeat the instructions for generating a hex code, base32 secret and adding to the users.oath file (also add an entry to the pam_per_user.map file unless you set the default to be sshd-oath).  No reload of SSH is needed to add new users.

To change an existing user’s token, do the same as adding a new user except update the existing entry in the file.

There can only be a single entry for each user, so you can’t add multiple tokens for a user.  However, you could add the same token onto multiple authenticator devices.

Every attempt to authenticate a 2FA user will modify the users.oath file.  For example the above modifed the entry in the file to this:

HOTP/T30/6 twofa – d1ac1a3d57deeefd6b83c3c340feea 0 217227 2017-11-06T19:58:12L

The additional parameters are:

  • 0 – The sequence number (this is unused in TOTP, it’s only used for HOTP)
  • 217227 – The last valid 2FA code entered
  • 2017-11-06T19:58:12L – The datestamp of the last valid code entry

Be very careful setting this up for the first time.  I would advise having console access available in case you make a mistake and can’t authenticate!

This should help make your FreeBSD system a little more secure by adding two-factor authentication to SSH logins.  You can also apply this to any PAM-based authentication mechanism but that’s beyond the scope of this article.

For applications, I recommend the Authy app under Android (and perhaps other mobile devices) as it can be used on multiple devices, and migrated between devices easily too.  They also include a desktop version now.

CARP (IPv4 and IPv6) in FreeBSD 10+ (for failover IPs)

Leave a Reply

Here’s how to configure CARP (common address redundancy protocol) to failover between two machines for both IPv4 and IPv6 IPs.  This requires FreeBSD 10+.

NOTE: Your switch requires you to allow multicast and IGMP.  Most switches do this, but VMware virtual switches generally do not unless you disable all security on them (not advised) – FreeBSD bhyve VMs work fine so long as the host’s upstream switch supports it.

We have a machine that should normally be the master of the pair.  It has non-shared IPs of 192.0.2.101/24 and 2001:db8::101/64

We have a machine that should normally be the slave of the pair.  It has non-shared IPs of 192.0.2.102/24 and 2001:db8::102/64

The two machines will share the IPs 192.0.2.1 and 2001:db8::1 – the master will respond to these IPs, and the slave will take over if the master disappears.  This ordinarily happens in under a second.  A few packets are lost during the failover but TCP retransmits etc take care of this and it is usually unnoticable.

Each CARP setup requires a VHID (this determines the MAC address used so it should be unique on the network) and a password to protect announcements.  We’ll use VHID 1 for the IPv4 setup and VHID 2 for the IPv6 setup.  We will use a password of testpass for demonstration purposes.

The only difference between the two machine setups is the advskew value.  This decides the priority of each machine.  The lower the number, the higher the priority.  The master will be the machine with the lowest advskew

First, we need to load the kernel module on each machine.  In /boot/loader.conf add:

carp_load=”YES”

Now we need to add the network configuration.  Setup the server as normal with its static IPs, then we can add additional IPs to the network card for the CARP configuration.  In our example, we are using the network interface vtnet0 (a bhyve vm).

Add the following lines (modified for your use) to /etc/rc.conf:

ifconfig_vtnet0_alias0=”inet 192.0.2.1/24 vhid 1 advskew 100 pass testpass”
ifconfig_vtnet0_alias1=”inet6 2001:db8::1 prefixlen 64 vhid 2 advskew 100 pass testpass”

As mentioned before, setup the slave machine identically except the two above lines would have advskew 200 to ensure they are lower priority.

Reboot the machines and login to check with ifconfig vtnet0 command.

Here’s the output on the master machine:

vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
ether 58:9c:fc:05:c4:0d
inet6 fe80::5a9c:fcff:fe05:c40d%vtnet0 prefixlen 64 scopeid 0x1
inet 192.0.2.1 netmask 0xffffff00 broadcast 192.0.2.255 vhid 1
inet6 2001:db8::1 prefixlen 64 vhid 2
inet6 2001:db8::101 prefixlen 64
inet 192.0.2.101 netmask 0xffffff00 broadcast 192.0.2.255
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-T <full-duplex>
status: active
carp: MASTER vhid 1 advbase 1 advskew 100
carp: MASTER vhid 2 advbase 1 advskew 100

You can see at the bottom of the output, the carp status shows that the machine is in MASTER state.

Here’s the output on the slave machine:

vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
ether 58:9c:fc:04:48:35
inet6 fe80::5a9c:fcff:fe04:4835%vtnet0 prefixlen 64 scopeid 0x1
inet 192.0.2.1 netmask 0xffffff00 broadcast 192.0.2.255 vhid 1
inet6 2001:db8::1 prefixlen 64 vhid 2
inet6 2001:db8::102 prefixlen 64
inet 192.0.2.102 netmask 0xffffff00 broadcast 192.0.2.255
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-T <full-duplex>
status: active
carp: BACKUP vhid 1 advbase 1 advskew 200
carp: BACKUP vhid 2 advbase 1 advskew 200

Here you can see the carp status is BACKUP.

If you were to shutdown or reboot the master machine, the slave machine would change to MASTER status.

This is also logged into /var/log/messages:

Jun  5 22:34:51 carp-slave kernel: carp: VHID 1@vtnet0: BACKUP -> MASTER (master down)
Jun  5 22:34:51 carp-slave kernel: carp: VHID 2@vtnet0: BACKUP -> MASTER (master down)

It’s also possible to monitor this and trigger a script upon changes using devd – but that is out of scope for this article.  If there’s enough interest, I might do a further article.

Add DKIM Signing to FreeBSD Servers

31 Replies

In today’s very spammy internet, it becomes increasingly important to verify the source of emails are valid (this is especially true if you want to send emails to gmail or outlook.com!)

One method is to add DKIM signing to outbound emails.

This will detail how to add DKIM signing to a stock FreeBSD server (e.g. a webserver) using sendmail.

First, we need to install the DKIM package:

pkg install opendkim

Now, we need to edit /usr/local/etc/mail/opendkim.conf and change the following lines:

Canonicalization  relaxed/simple
KeyTable          refile:/usr/local/etc/mail/opendkim.keytable
LogWhy            yes
Selector          myselector
SigningTable      refile:/usr/local/etc/mail/opendkim.signingtable
Socket            local:/var/run/dkim/opendkim.sock
SyslogSuccess     Yes
UserID            mailnull:mailnull

You can change myselector to anything you wish to use.  This will form part of the DNS entry required for signature verification so it should be fairly unique (this allows you to have different DKIM signing keys for different servers/services.  I often call them based on the server name.

And we need to do a few steps to setup things:

touch /usr/local/etc/mail/opendkim.keytable
touch /usr/local/etc/mail/opendkim.signingtable
mkdir -p /var/run/dkim /var/db/dkim
chown mailnull:mailnull /var/run/dkim /var/db/dkim
chmod 0700 /var/run/dkim

This creates some files and folders, and sets the necessary permissions.

Now we need to tell opendkim to start on boot, add to /etc/rc.conf the following:

milteropendkim_enable=”YES”
milteropendkim_uid=”mailnull”
milteropendkim_cfgfile=”/usr/local/etc/mail/opendkim.conf”

and now we can start with:

service milter-opendkim start

If all is well, you’ll see two copies of opendkim running in your process list.

Finally, we need to tell sendmail to sign emails that go out.

cd /etc/mail
cp freebsd.mc dkimsigned.mc

Now edit /etc/mail/dkimsigned.mc and add the following line just before the ‘MAILER’ lines:

INPUT_MAIL_FILTER(`opendkim’, `S=local:/var/run/dkim/opendkim.sock’)dnl

Now we can build a new sendmail configuration file and activate it:

cd /etc/mail
make dkimsigned.cf
mv sendmail.cf sendmail.cf.pre-dkim
cp dkimsigned.cf sendmail.cf
service sendmail restart

Now, at this point, nothing has really changed as we haven’t given opendkim any domains to sign.

I wrote a handle script for generating new DKIM configuration… Add the following lines to a script called generate-dkim somewhere in your path:

#!/bin/sh

SELECTOR=myselector
DOMAIN=${1}

if [ -z ${1} ]; then
echo “Usage: ${0} <domain>”
exit 1
fi

if [ -f /var/db/dkim/${DOMAIN}/${SELECTOR}.txt ]; then
echo “=> ERROR: found DKIM keys for ${DOMAIN} already”
/bin/cat /var/db/dkim/${DOMAIN}/${SELECTOR}.txt
exit 1
fi

echo “==> Generating DKIM key for ${DOMAIN}…”
/bin/mkdir /var/db/dkim/${DOMAIN}
/usr/sbin/chown mailnull:mailnull /var/db/dkim/${DOMAIN}
/usr/local/sbin/opendkim-genkey -a -b 2048 -d ${DOMAIN} -D /var/db/dkim/${DOMAIN} -s ${SELECTOR}
/usr/sbin/chown mailnull:mailnull /var/db/dkim/${DOMAIN}/*
echo “==> Done”
echo “==> Updating config to enable DKIM signing…”
echo “${SELECTOR}._domainkey.${DOMAIN}      ${DOMAIN}:${SELECTOR}:/var/db/dkim/${DOMAIN}/${SELECTOR}.private” >> /usr/local/etc/mail/opendkim.keytable
echo “*@${DOMAIN}       ${SELECTOR}._domainkey.${DOMAIN}” >> /usr/local/etc/mail/opendkim.signingtable
echo “==> Done”
echo “==> Reloading OpenDKIM configuration…”
/bin/pkill -USR1 -F /var/run/milteropendkim/pid
echo “==> Done”
echo “DKIM DNS entry to add is below:”
/bin/cat /var/db/dkim/${DOMAIN}/${SELECTOR}.txt

If you chose a different ‘selector’ in the earlier step, remember to change the SELECTOR= line above to match or your signing wont work!

And dont forget to make it executable:

chmod a+x generate-dkim

Now, whenever we want to add a new signing domain, for example dan.me.uk, we can run

generate-dkim dan.me.uk

This will generate a new key, add it to the opendkim configuration files, reload opendkim, and output the required DNS entry to you.

If you’ve previously added the domain, it will just show the DNS entry to you without modifying anything.

Now any emails sent out with a sender of *@dan.me.uk will be DKIM signed using the key generated in the step above.

You MUST add the DNS entry to the domain’s DNS zone for DKIM signing to work.  If the receiving mailserver can’t lookup the DNS entry then it can’t verify your email!

If you ever need to remove a domain from signing, you have to do this manually…

Remove it from /usr/local/etc/mail/opendkim.keytable and /usr/local/etc/mail/opendkim.signingtable (one line in each file), then there’s a folder in /var/db/dkim/ named after the domain which needs to be removed.

For the best chance of delivering to the world’s biggest spammers (gmail/outlook/aol/yahoo/etc), I would recommend SPF, DKIM and DMARC!

NetFlow v9 Exporting from FreeBSD routers/firewalls

Leave a Reply

Sometimes it’s useful to have netflow reporting from a FreeBSD router or firewall.

For this example, we’ll assume you want to export flows on the igb0 network interface with netflow v9 packets to the flow collector at 192.0.2.10 port 1234 (UDP) – v9 packets are useful as they can contain IPv6 flows.

First, we need to load some kernel modules at boot time – to do this we need to add the following lines to your /boot/loader.conf file:

netgraph_load=”YES”
ng_ether_load=”YES”
ng_socket_load=”YES”
ng_ksocket_load=”YES”
ng_tee_load=”YES”
ng_netflow_load=”YES”

You can load these now without having to reboot by typing:

kldload netgraph ng_ether ng_socket ng_ksocket ng_tee ng_netflow

Next we need a startup script to configure netflow.  Create a file with the following contents called /usr/local/etc/rc.d/900.netflow.sh:

#!/bin/sh
case “$1” in
‘start’)
/usr/sbin/ngctl -f- <<-SEQ
mkpeer igb0: netflow lower iface0
name igb0:lower netflow
connect igb0: netflow: upper out0
mkpeer netflow: ksocket export9 inet/dgram/udp
msg netflow:export9 connect inet/192.0.2.10:1234
SEQ
;;
‘stop’)
;;
*)
echo “Please specify ‘start’ or ‘stop'”
;;
esac

And ensure it runs at boot time with:

chmod a+x /usr/local/etc/rc.d/900.netflow.sh

and now you can start it:

/usr/local/etc/rc.d/900.netflow.sh start

You should start seeing netflow packets arrive at your collector.

FreeBSD doesn’t have ASN information in its kernel, so the netflow packets will only report flows with IPs and port numbers present.

Link Failover in FreeBSD (without requiring switch configuration)

Leave a Reply

It’s often useful to have simple link failover… for example a server connected to two discrete switches.  In this scenario, LACP is of no use as it can’t span discrete switches.

Link failover in FreeBSD will failover to a second switchport in the event of a link down event of the first.  (it does require the link itself to drop)

The MAC address of the failover pseudo network card will be the MAC address of the first network card added to the card.

First, we need to ensure the link aggregation module (failover is part of this module) is started at boot, so edit /boot/loader.conf and add the following line:

if_lagg_load=”YES”

Now we need to configure the port… in this example we will bond igb0 and bge0 together into a failover network, with igb0 being the primary port.  We will assign the IP 192.0.2.10/24 to the interface.

Add the following to /etc/rc.conf:

cloned_interfaces=”lagg0″
ifconfig_igb0=”up”
ifconfig_bge0=”up”
ifconfig_lagg0=”laggproto failover laggport igb0 laggport bge0 up”
ifconfig_lagg0_alias0=”inet 192.0.2.10/24″

Now it’s possible to put the IP assignment on the same line as the failover definitions but I think it looks cleaner separate – so I used an alias instead.

You can use a lagg port in the same way as a normal network port (e.g. clone interfaces from it for vlans etc)

On the FreeBSD server, you can check the failover with ‘ifconfig lagg0‘ command:

        laggproto failover lagghash l2,l3,l4
laggport: bge0 flags=0<>
laggport: igb0 flags=5<MASTER,ACTIVE>

You can see both ports are connected, and igb0 is the master port, and active port.

Data received on any non-active port will be discarded, and data from the server outbound will only be sent to active ports.

You can, of course, have more than 2 network ports in a failover configuration – but only one will ever be active.

It can also be useful on a laptop environment so failover between wireless and wired connections with the wired connection being preferred if plugged in, and wireless if not.

OpenVPN Setup in FreeBSD (with NAT for IPv4 and IPv6)

Leave a Reply

It seems people always use Linux when it comes to setting up an OpenVPN server for internet access, so here’s how to do it on FreeBSD.

You’ll need at least FreeBSD 10.3, but here’s how to do it.

Our interface will be igb0, our IPv4 public IP will be 192.0.2.10 and our IPv6 public IP will be 2001:DB8::10 – these will both be shared via NAT.

First we need to install some packages…

pkg install bash easy-rsa openvpn

Now we need to load some kernel drivers, add to /boot/loader.conf:

net.inet.ip.fw.default_to_accept=”1″
aesni_load=”YES”
crypto_load=”YES”
if_bridge_load=”YES”
if_tap_load=”YES”
ipfw_load=”YES”
ipfw_nat_load=”YES”

You will need to reboot to activate them, but you can do that at the end of the process.

We need to configure pf to NAT our IPv6 IPs, create /etc/pf.conf with these contents:

v6_wan_if=”igb0″
v6_wan_ip=”2001:DB8::10″

no nat on $v6_wan_if inet6 from $v6_wan_ip to any
nat on $v6_wan_if inet6 from any to any -> $v6_wan_ip

and we need to activate the pf config in /etc/rc.conf by adding:

pf_enable=”YES”
pf_rules=”/etc/pf.conf”
pf_program=”/sbin/pfctl”
pf_flags=””

We’re going to handle IPv4 nat using ipfw instead of pf (this may not be the best of things, but it works well) – we’ll also do some firewalling at the same time.

Create a file called /usr/local/etc/rc.d/000.ipfw.sh with the following contents:

#!/bin/sh

case “$1” in
‘start’)
/sbin/ipfw -f flush

/sbin/ipfw nat 4 config log ip 192.0.2.10

/sbin/ipfw add 00100 allow ipv6-icmp from :: to ff02::/16 // allow DAD
/sbin/ipfw add 00110 allow ipv6-icmp from fe80::/10 to fe80::/10 // allow RS RA NS NA Redirects
/sbin/ipfw add 00120 allow ipv6-icmp from fe80::/10 to ff02::/16 // allow RS RA NS NA Redirects
/sbin/ipfw add 00130 allow ipv6-icmp from any to any icmp6types 1 // allow destination unreachables
/sbin/ipfw add 00140 allow ipv6-icmp from any to any icmp6types 2,135,136 // allow NS/NA/toobig ICMPs

/sbin/ipfw add 00500 nat 4 ip4 from any to any via igb0 // NATv4

/sbin/ipfw add 00700 check-state

/sbin/ipfw add 01000 allow all from any to any via lo0 // allow loopback
/sbin/ipfw add 01010 deny all from any to 127.0.0.0/8
/sbin/ipfw add 01020 deny all from 127.0.0.0/8 to any
/sbin/ipfw add 01030 deny all from any to ::1
/sbin/ipfw add 01040 deny all from ::1 to any

/sbin/ipfw add 02000 allow ipv6-icmp from :: to ff02::/16 // allow DAD
/sbin/ipfw add 02010 allow ipv6-icmp from fe80::/10 to fe80::/10 // allow RS RA NS NA Redirects
/sbin/ipfw add 02020 allow ipv6-icmp from fe80::/10 to ff02::/16 // allow RS RA NS NA Redirects
/sbin/ipfw add 02030 allow ipv6-icmp from any to any icmp6types 1 // allow destination unreachables
/sbin/ipfw add 02040 allow ipv6-icmp from any to any icmp6types 2,135,136 // allow NS/NA/toobig ICMPs

/sbin/ipfw add 03000 allow tcp from any to any established
/sbin/ipfw add 03100 allow all from any to any frag
/sbin/ipfw add 03200 allow tcp from me to any setup
/sbin/ipfw add 03300 allow udp from me to any 53 keep-state
/sbin/ipfw add 03400 allow udp from me to any 123 keep-state

;;
‘stop’)
;;
*)
echo “Please specify ‘start’ or ‘stop'”
;;
esac

and ensure it runs on boot:

chmod a+x /usr/local/etc/rc.d/000.ipfw.sh

Now… onto OpenVPN configuration.

Create a file called /usr/local/etc/openvpn/openvpn.conf with the following contents:

script-security 2
local 192.0.2.10
port 1194
proto udp
dev tun
ca /usr/local/etc/openvpn/keys/ca.crt
cert /usr/local/etc/openvpn/keys/vpn.crt
key /usr/local/etc/openvpn/keys/vpn.key  # This file should be kept secret
crl-verify /usr/local/etc/openvpn/keys/crl.pem
dh /usr/local/etc/openvpn/keys/dh.pem
tun-ipv6
ifconfig-pool-persist /usr/local/etc/openvpn/ipp.txt
server 172.31.255.0 255.255.255.0
server-ipv6 fc00:da::/64
push “redirect-gateway def1 bypass-dhcp”
push “dhcp-option DNS 8.8.8.8”
push “route-ipv6 ::/1”
push “route-ipv6 8000::/1”
client-config-dir /usr/local/etc/openvpn/ccd
keepalive 10 30
comp-lzo
persist-key
persist-tun
status /var/log/openvpn-status.log
log-append /var/log/openvpn.log
verb 4

You’ll notice we’re using fc00:da::/64 as the IPv6 prefix for VPN clients, this is a range reserved for local usage so will not conflict with any globally reachable IP addresses.

We need to configure OpenVPN’s certificate authority now:

easyrsa init-pki
easyrsa build-ca
(enter password, set Common Name to VPN hostname)
easyrsa gen-dh
easyrsa gen-req vpn nopass
(set Common Name to VPN hostname)
easyrsa sign server vpn
(answer ‘yes’ and use password from CA step above)
easyrsa gen-crl
(enter password from CA step above)

Now we need to copy some files into the OpenVPN working directory:

mkdir /usr/local/etc/openvpn/ccd
cd /usr/local/share/easy-rsa/pki
mkdir -p /usr/local/etc/openvpn/keys
cp -p ca.crt crl.pem dh.pem index* serial* private/vpn.key issued/vpn.crt /usr/local/etc/openvpn/keys/

We’re now ready to add some users… repeat the following for each new client you want to create:

easyrsa gen-req client1 nopass
easyrsa sign client client1
cd /usr/local/share/easy-rsa/pki
cp -p private/client1.key issued/client1.crt /usr/local/etc/openvpn/keys/

and also create a file for each called /usr/local/etc/openvpn/ccd/client1 (where client1 is the username) containing:

iroute 10.0.0.0 255.0.0.0
iroute 192.168.0.0 255.255.0.0
iroute 172.16.0.0 255.240.0.0

Now all we need to do is generate the configure file(s) for each client to use.

Create a file called /usr/local/etc/openvpn/client1.ovpn containing:

client
dev tun
remote 192.0.2.10 1194 udp
resolv-retry infinite
nobind
persist-key
persist-tun
tun-ipv6
mute-replay-warnings
<ca>
(contents of /usr/local/etc/openvpn/keys/ca.crt)
</ca>
<key>
(contents of /usr/local/etc/openvpn/keys/client1.key)
</key>
<cert>
(contents of /usr/local/etc/openvpn/keys/client1.crt)
</cert>
route-delay
comp-lzo
verb 3
mute 20

The file needs to contain the contents of three files, including the lines starting “—–”

Copy this .ovpn file to the client device’s config directory – it should also work with the Android OpenVPN client.

Finally, we need to tell OpenVPN to start when FreeBSD starts by adding the following to your /etc/rc.conf file:

openvpn_enable=”YES”

Now, simply reboot and everything should work fine!

If you need to revoke a user (e.g. if they lose their device, or they become compromised), then run the following commands:

easyrsa revoke client1
easyrsa gen-crl
cp /usr/local/share/easy-rsa/pki/crl.pem /usr/local/etc/openvpn/keys/
killall -USR1 openvpn

This will revoke the client from connecting without having to re-generate all other client certificates.

Using FreeBSD as a Hypervisor (using bhyve and vm-bhyve to manage them)

14 Replies

To do this, you must be running at least FreeBSD 10.3-RELEASE – if you’re running an older version of FreeBSD, you should update!  Also, ensure that you have a CPU that supports hardware virtualisation.  Most new server CPUs seem to work just fine.

First, we need to install some packages to make things much easier:

pkg install vm-bhyve grub2-bhyve

Now we need to load some kernel modules on boot to allow us to virtualise and bridge networking… add the following to /boot/loader.conf:

if_bridge_load=”YES”
if_tap_load=”YES”
nmdm_load=”YES”
vmm_load=”YES”

You need to reboot to activate the modules, or you can load them now:

kldload if_bridge if_tap nmdm vmm

And we need to add some configuration to /etc/rc.conf:

vm_enable=”YES”
vm_dir=”zfs:zroot/vms”
vm_list=””
vm_delay=”5″

Now we will setup some storage for the hypervisor – For this, we’ll assume ‘zroot’ zfs pool has enough capacity:

zfs create -o mountpoint=/vms zroot/vms
vm init
cp /usr/local/share/examples/vm-bhyve/* /vms/.templates/

Now we can create a virtual switch (bridge) and bridge it to our network interface.  Change igb0 to your network interface below:

vm switch create public
vm switch add public igb0

Now, fetch an installation ISO for a FreeBSD guest:

vm iso ftp://ftp.uk.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/10.3/FreeBSD-10.3-RELEASE-amd64-disc1.iso

Now edit the default template /vms/.templates/default.conf (actually every .conf file in the folder) and change the disk config:

disk0_name=”disk0″
disk0_dev=”sparse-zvol”

This will make vms use zvols instead of disk image files.  Now we’re all prepared.

To create a test 16GB FreeBSD vm:

vm create -s 16G testvm
vm -f install testvm FreeBSD-10.3-RELEASE-amd64-disc1.iso

Select ‘vt100’ when asked for console type, and install as normal.  Once finished, login to the new vm and power it off (use ‘poweroff’) – control will return to the host.

Here’s some useful commands to manage VMs…

To list all your VMs:

vm list

To start a VM called testvm:

vm start testvm

To stop a VM (gracefully via ACPI) called testvm:

vm stop testvm

To forcibly power off a VM called testvm:

vm poweroff testvm

To press reset on a VM called testvm:

vm reset testvm

To connect to the serial console on a VM called testvm:

vm console testvm

(to disconnect, send ~. to exit the console session)

To permanently destroy a VM called testvm (must be turned off first):

vm destroy testvm

To edit the configuration of a VM called testvm (e.g. to change the MAC address):

vm configure testvm

To increase the size of the disk on a VM called testvm to 32GB:

zfs set volsize=32G zroot/vms/testvm/disk0

To create an Ubuntu guest using a previously downloaded ISO called ubuntu.iso:

vm create -t ubuntu -s 16G testubuntu
vm install testubuntu ubuntu.iso

To make Ubuntu boot unaided each time, edit the configuration file for the VM just created (in /vms/testubuntu/testubuntu.conf) and add:

grub_run_dir=”/grub”
grub_run_file=”grub.cfg”

You can also add the above to the ubuntu template file for it to be automatically added to ubuntu VMs.

Windows guests take a lot more effort as there is no GUI console.  This also means debugging a broken windows install is almost impossible so I would discourage its use for production.  However, this is how to prepare to install Windows VMs:

fetch -o /vms/.config/BHYVE_UEFI.fd http://people.freebsd.org/%7Egrehan/bhyve_uefi/BHYVE_UEFI_20151002.fd
pkg install git p7zip cdrtools-devel
mkdir /vms/work
cd /vms/work
git clone https://github.com/nahanni/bhyve-windows-unattend-xml
fetch https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.96/virtio-win-0.1.96.iso

This fetches a special UEFI loader, a collection of automated unattended Windows installation scripts, and the VirtIO drivers to allow storage to work.

Now copy a Windows installation ISO to /vms/work/win.iso and then:

mkdir tmp
cd tmp
7z x ../win.iso
cp /vms/work/bhyve-windows-unattend-xml/files/win2012r2_AutoUnattend.xml AutoUnattend.xml

This extracts the Windows ISO and copies an automated unattended installation script to it.  There are several AutoUnattend.xml files so choose the one matching the OS you’re installing… we’re doing a Windows 2012R2 install here.

You can edit the AutoUnattend.xml file to set a product key if required (not required for recent Windows OS)

Now we copy the VirtIO drivers into the temporary work directory and build a new ISO image:

mkdir virtio
cd virtio
tar xf /vms/work/virtio-win-0.1.96.iso
cd /vms/work/tmp
mkisofs \
-b boot/etfsboot.com -no-emul-boot -c BOOT.CAT \
-iso-level 4 -J -l -D \
-N -joliet-long \
-relaxed-filenames -v \
-V Custom -udf \
-boot-info-table -eltorito-alt-boot -eltorito-platform 0xEF \
-eltorito-boot efi/microsoft/boot/efisys_noprompt.bin \
-no-emul-boot \
-o /vms/.iso/win2k12r2.iso .

You now have a win2k12r2.iso file ready to build a windows VM:

vm create -t windows -s 64G wintest
vm install wintest win2k12r2.iso

This will start the unattended installation of windows.  It can take up to around 25 minutes.  You can monitor its progress by looking at the logfile:

tail -f /vms/wintest/vm-bhyve.log

You need to wait for it to restart twice (so three “starting bhyve” messages) – you can then get the IP from the server:

vm console wintest

and press “i” to view the IP address – this only seems to work on win2k12r2 and above.  You may need to check your DHCP server logs otherwise.

You can now RDP to the VM using Administrator and Test123 as the password.

Again, you should shutdown the VM and start it using the vm start wintest command.

To make VMs automatically start on boot, you can add them to /etc/rc.conf in the vm_list variable, for example to start wintest and testvm:

vm_list=”wintest testvm”

Enjoy.

Zen Broadband with IPv6 using FreeBSD ppp router

Leave a Reply

A little while ago, i switched broadband providers so that I could enjoy the greater speeds on FTTC (fibre-to-the-cabinet) technology.  My existing ISP charged an insanely high amount for their unlimited package so I went through the hassle of renumbering my home network.

I chose to go with Zen broadband… I’ve used Zen in the past, and always been happy… plus they support IPv6 (of which I’m a big supporter)

So… here’s how to connect to Zen broadband (and possibly others) using FreeBSD as a router (via PPPoE) with the Zen provided DSL modem in passthrough mode.

Zen provide a Zyxel VMG1312-B10D DSL modem.  First you need to change the modem into bridged mode.  I won’t detail that here, there’s way too many documents online that show how to do it.  (alternatively, use another modem in bridge mode… I couldn’t get a Draytek to bridge properly – so perhaps avoid that.)

On the FreeBSD router, you need a dedicated network card for running PPPoE – I added an intel gigabit card to my router, so my PPPoE interface is igb0

First, we need to load some kernel modules, so add to /boot/loader.conf:

netgraph_load=”YES”
ng_ether_load=”YES”
ng_pppoe_load=”YES”
ng_socket_load=”YES”

These will be activated when you reboot.

Now we need to configure PPP.  Replace the entire contents of /etc/ppp/ppp.conf with:

zen:
set speed sync
set mru 1492
set mtu 1492
set ctsrts off

enable echo
set echoperiod 15
enable lqr
set lqrperiod 15

set log phase connect ipcp ipv6cp tun

enable ipcp
disable dns

set device PPPoE:igb0
set redial 10
set server /tmp/pppoe-adsl0 “” 0177

set authname USERNAMEHERE
set authkey PASSWORDHERE

add! default HISADDR
add! default HISADDR6

NOTE: everything except the first “zen:” line is indented – wordpress has a habit of losing the indentation!  Put your Zen-provided username and password in place of the capitalised placeholders above.  Also, change ‘igb0’ to a different interface depending on your router configuration.

Now, we need to set ppp to start on boot, so add to /etc/rc.conf:

ipv6_cpe_wanif=”tun0″
ifconfig_tun0_ipv6=”inet6 -ifdisabled -no_radr accept_rtadv”
# PPPoE configuration
ppp_enable=”YES”
ppp_program=”/usr/sbin/ppp”
ppp_nat=”NO”
ppp_user=”root”
ppp_profile=”zen”
ppp_zen_mode=”ddial”
ppp_zen_nat=”NO”

This will load the ‘zen’ profile in ‘ddial’ mode on boot.  It disables NAT in the PPP daemon.

The above is enough to negotiate IPv4, and to start the IPv6 negotiation – but zen requires that you use SLAAC to obtain an IPv6 IP, so we need to create another file for this.  This will run after connecting.  Create the file /etc/ppp/ppp.linkup with the following contents:

zen:
shell /sbin/ifconfig tun0 inet6 -ifdisabled -no_radr accept_rtadv
shell /sbin/rtsol -a tun0 &

Again, everything except the first “zen:” line is indented.

This enables SLAAC on the tun0 interface (the pseudo-interface that ppp creates) and triggers rtsol to obtain an IPv6 IP.

Zen also issue a /48 IPv6 netblock, which seems to be routed over the link automatically (although others say you need to do proxy DHCPv6 for it to work – I certainly don’t need to)

Link Aggregation/Bonding in FreeBSD using LACP

Leave a Reply

To bond multiple ethernet links together in FreeBSD is fairly simple.  In this example, we’ll use LACP which does require some switch configuration to work.

First, we need to ensure the link aggregation module is started at boot, so edit /boot/loader.conf and add the following line:

if_lagg_load=”YES”

Now we need to configure the port… in this example we will bond igb0 and bge0 together into a two port LACP bundle.  We will assign the IP 192.0.2.10/24 to the interface.

Add the following to /etc/rc.conf:

cloned_interfaces=”lagg0″
ifconfig_igb0=”up”
ifconfig_bge0=”up”
ifconfig_lagg0=”laggproto lacp laggport igb0 laggport bge0 up”
ifconfig_lagg0_alias0=”inet 192.0.2.10/24″

Now it’s possible to put the IP assignment on the same line as the LACP definitions but I think it looks cleaner separate – so I used an alias instead.

You can use a lagg port in the same way as a normal network port (e.g. clone interfaces from it for vlans etc)

Aggregated ports will use the MAC address from the first “laggport” in the bundle for all ports.

In FreeBSD, LACP is always configured in ‘active’ mode and system/port priorities use 0x8000 – these are not configurable.

As a quick example, here’s how you would configure a Cisco 3750/3850 switchport to support a FreeBSD LACP bundle.  We’ll use ports Gi1/0/1 and Gi2/0/1 to create a bundle called Port-channel1

interface Port-channel1
description LACP Bundle 1
switchport mode access
switchport access vlan 1
switchport nonegotiate
spanning-tree portfast
!
interface Gi1/0/1
description LACP Bundle 1 – 1/2
switchport mode access
switchport access vlan 1
switchport nonegotiate
channel-group 1 mode active
spanning-tree portfast
!
interface Gi2/0/1
description LACP Bundle 1 – 2/2
switchport mode access
switchport access vlan 1
switchport nonegotiate
channel-group 1 mode active
spanning-tree portfast

And you can confirm once it’s setup and working (it requires you to have rebooted the FreeBSD server above so that it can negotiate LACP) – you can use the ‘show etherchannel summary‘ command:

Group  Port-channel  Protocol    Ports
——+————-+———–+———————————————–
1      Po1(SU)         LACP      Gi1/0/1(P) Gi2/0/1(P)

You can see Po1 is up using LACP protocol and both ports in the bundle.

On the FreeBSD server, you can check LACP with ‘ifconfig lagg0‘ command:

        laggproto lacp lagghash l2,l3,l4
laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: bge0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>

You can see both ports are connected and working fine.

Speed up FreeBSD reboots by 7 seconds

Leave a Reply

Ok, 7 seconds doesn’t seem like much… but if you’re rebooting a live server then every second of downtime counts – so why not?

This is a simple change that reduces the timer on the initial FreeBSD menu from 10 seconds to 3 seconds.  (I don’t recommend reducing it below 3 seconds, as it may make it more difficult for you to use the menu should you ever need to!)

Ok, so you need to edit the /boot/loader.conf file (or create it if it doesn’t exist) and add the following line:

autoboot_delay=”3″

and that’s all you need to do.  Next reboot, the timer will start at 3 instead of 10 and you have a reboot that takes 7 seconds less than before.