To do this, you must be running at least FreeBSD 10.3-RELEASE – if you’re running an older version of FreeBSD, you should update!  Also, ensure that you have a CPU that supports hardware virtualisation.  Most new server CPUs seem to work just fine.

First, we need to install some packages to make things much easier:

pkg install vm-bhyve grub2-bhyve

Now we need to load some kernel modules on boot to allow us to virtualise and bridge networking… add the following to /boot/loader.conf:

if_bridge_load=”YES”
if_tap_load=”YES”
nmdm_load=”YES”
vmm_load=”YES”

You need to reboot to activate the modules, or you can load them now:

kldload if_bridge if_tap nmdm vmm

And we need to add some configuration to /etc/rc.conf:

vm_enable=”YES”
vm_dir=”zfs:zroot/vms”
vm_list=””
vm_delay=”5″

Now we will setup some storage for the hypervisor – For this, we’ll assume ‘zroot’ zfs pool has enough capacity:

zfs create -o mountpoint=/vms zroot/vms
vm init
cp /usr/local/share/examples/vm-bhyve/* /vms/.templates/

Now we can create a virtual switch (bridge) and bridge it to our network interface.  Change igb0 to your network interface below:

vm switch create public
vm switch add public igb0

Now, fetch an installation ISO for a FreeBSD guest:

vm iso ftp://ftp.uk.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/10.3/FreeBSD-10.3-RELEASE-amd64-disc1.iso

Now edit the default template /vms/.templates/default.conf (actually every .conf file in the folder) and change the disk config:

disk0_name=”disk0″
disk0_dev=”sparse-zvol”

This will make vms use zvols instead of disk image files.  Now we’re all prepared.

To create a test 16GB FreeBSD vm:

vm create -s 16G testvm
vm -f install testvm FreeBSD-10.3-RELEASE-amd64-disc1.iso

Select ‘vt100’ when asked for console type, and install as normal.  Once finished, login to the new vm and power it off (use ‘poweroff’) – control will return to the host.

Here’s some useful commands to manage VMs…

To list all your VMs:

vm list

To start a VM called testvm:

vm start testvm

To stop a VM (gracefully via ACPI) called testvm:

vm stop testvm

To forcibly power off a VM called testvm:

vm poweroff testvm

To press reset on a VM called testvm:

vm reset testvm

To connect to the serial console on a VM called testvm:

vm console testvm

(to disconnect, send ~. to exit the console session)

To permanently destroy a VM called testvm (must be turned off first):

vm destroy testvm

To edit the configuration of a VM called testvm (e.g. to change the MAC address):

vm configure testvm

To increase the size of the disk on a VM called testvm to 32GB:

zfs set volsize=32G zroot/vms/testvm/disk0

To create an Ubuntu guest using a previously downloaded ISO called ubuntu.iso:

vm create -t ubuntu -s 16G testubuntu
vm install testubuntu ubuntu.iso

To make Ubuntu boot unaided each time, edit the configuration file for the VM just created (in /vms/testubuntu/testubuntu.conf) and add:

grub_run_dir=”/grub”
grub_run_file=”grub.cfg”

You can also add the above to the ubuntu template file for it to be automatically added to ubuntu VMs.

Windows guests take a lot more effort as there is no GUI console.  This also means debugging a broken windows install is almost impossible so I would discourage its use for production.  However, this is how to prepare to install Windows VMs:

fetch -o /vms/.config/BHYVE_UEFI.fd http://people.freebsd.org/%7Egrehan/bhyve_uefi/BHYVE_UEFI_20151002.fd
pkg install git p7zip cdrtools-devel
mkdir /vms/work
cd /vms/work
git clone https://github.com/nahanni/bhyve-windows-unattend-xml
fetch https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.96/virtio-win-0.1.96.iso

This fetches a special UEFI loader, a collection of automated unattended Windows installation scripts, and the VirtIO drivers to allow storage to work.

Now copy a Windows installation ISO to /vms/work/win.iso and then:

mkdir tmp
cd tmp
7z x ../win.iso
cp /vms/work/bhyve-windows-unattend-xml/files/win2012r2_AutoUnattend.xml AutoUnattend.xml

This extracts the Windows ISO and copies an automated unattended installation script to it.  There are several AutoUnattend.xml files so choose the one matching the OS you’re installing… we’re doing a Windows 2012R2 install here.

You can edit the AutoUnattend.xml file to set a product key if required (not required for recent Windows OS)

Now we copy the VirtIO drivers into the temporary work directory and build a new ISO image:

mkdir virtio
cd virtio
tar xf /vms/work/virtio-win-0.1.96.iso
cd /vms/work/tmp
mkisofs \
-b boot/etfsboot.com -no-emul-boot -c BOOT.CAT \
-iso-level 4 -J -l -D \
-N -joliet-long \
-relaxed-filenames -v \
-V Custom -udf \
-boot-info-table -eltorito-alt-boot -eltorito-platform 0xEF \
-eltorito-boot efi/microsoft/boot/efisys_noprompt.bin \
-no-emul-boot \
-o /vms/.iso/win2k12r2.iso .

You now have a win2k12r2.iso file ready to build a windows VM:

vm create -t windows -s 64G wintest
vm install wintest win2k12r2.iso

This will start the unattended installation of windows.  It can take up to around 25 minutes.  You can monitor its progress by looking at the logfile:

tail -f /vms/wintest/vm-bhyve.log

You need to wait for it to restart twice (so three “starting bhyve” messages) – you can then get the IP from the server:

vm console wintest

and press “i” to view the IP address – this only seems to work on win2k12r2 and above.  You may need to check your DHCP server logs otherwise.

You can now RDP to the VM using Administrator and Test123 as the password.

Again, you should shutdown the VM and start it using the vm start wintest command.

To make VMs automatically start on boot, you can add them to /etc/rc.conf in the vm_list variable, for example to start wintest and testvm:

vm_list=”wintest testvm”

Enjoy.

To bond multiple ethernet links together in FreeBSD is fairly simple.  In this example, we’ll use LACP which does require some switch configuration to work.

First, we need to ensure the link aggregation module is started at boot, so edit /boot/loader.conf and add the following line:

if_lagg_load=”YES”

Now we need to configure the port… in this example we will bond igb0 and bge0 together into a two port LACP bundle.  We will assign the IP 192.0.2.10/24 to the interface.

Add the following to /etc/rc.conf:

cloned_interfaces=”lagg0″
ifconfig_igb0=”up”
ifconfig_bge0=”up”
ifconfig_lagg0=”laggproto lacp laggport igb0 laggport bge0 up”
ifconfig_lagg0_alias0=”inet 192.0.2.10/24″

Now it’s possible to put the IP assignment on the same line as the LACP definitions but I think it looks cleaner separate – so I used an alias instead.

You can use a lagg port in the same way as a normal network port (e.g. clone interfaces from it for vlans etc)

Aggregated ports will use the MAC address from the first “laggport” in the bundle for all ports.

In FreeBSD, LACP is always configured in ‘active’ mode and system/port priorities use 0x8000 – these are not configurable.

As a quick example, here’s how you would configure a Cisco 3750/3850 switchport to support a FreeBSD LACP bundle.  We’ll use ports Gi1/0/1 and Gi2/0/1 to create a bundle called Port-channel1

interface Port-channel1
description LACP Bundle 1
switchport mode access
switchport access vlan 1
switchport nonegotiate
spanning-tree portfast
!
interface Gi1/0/1
description LACP Bundle 1 – 1/2
switchport mode access
switchport access vlan 1
switchport nonegotiate
channel-group 1 mode active
spanning-tree portfast
!
interface Gi2/0/1
description LACP Bundle 1 – 2/2
switchport mode access
switchport access vlan 1
switchport nonegotiate
channel-group 1 mode active
spanning-tree portfast

And you can confirm once it’s setup and working (it requires you to have rebooted the FreeBSD server above so that it can negotiate LACP) – you can use the ‘show etherchannel summary‘ command:

Group  Port-channel  Protocol    Ports
——+————-+———–+———————————————–
1      Po1(SU)         LACP      Gi1/0/1(P) Gi2/0/1(P)

You can see Po1 is up using LACP protocol and both ports in the bundle.

On the FreeBSD server, you can check LACP with ‘ifconfig lagg0‘ command:

        laggproto lacp lagghash l2,l3,l4
laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: bge0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>

You can see both ports are connected and working fine.

Ok, 7 seconds doesn’t seem like much… but if you’re rebooting a live server then every second of downtime counts – so why not?

This is a simple change that reduces the timer on the initial FreeBSD menu from 10 seconds to 3 seconds.  (I don’t recommend reducing it below 3 seconds, as it may make it more difficult for you to use the menu should you ever need to!)

Ok, so you need to edit the /boot/loader.conf file (or create it if it doesn’t exist) and add the following line:

autoboot_delay=”3″

and that’s all you need to do.  Next reboot, the timer will start at 3 instead of 10 and you have a reboot that takes 7 seconds less than before.