Using FreeBSD as a Hypervisor (using bhyve and vm-bhyve to manage them)

To do this, you must be running at least FreeBSD 10.3-RELEASE – if you’re running an older version of FreeBSD, you should update!  Also, ensure that you have a CPU that supports hardware virtualisation.  Most new server CPUs seem to work just fine.

First, we need to install some packages to make things much easier:

pkg install vm-bhyve grub2-bhyve

Now we need to load some kernel modules on boot to allow us to virtualise and bridge networking… add the following to /boot/loader.conf:

if_bridge_load=”YES”
if_tap_load=”YES”
nmdm_load=”YES”
vmm_load=”YES”

You need to reboot to activate the modules, or you can load them now:

kldload if_bridge if_tap nmdm vmm

And we need to add some configuration to /etc/rc.conf:

vm_enable=”YES”
vm_dir=”zfs:zroot/vms”
vm_list=””
vm_delay=”5″

Now we will setup some storage for the hypervisor – For this, we’ll assume ‘zroot’ zfs pool has enough capacity:

zfs create -o mountpoint=/vms zroot/vms
vm init
cp /usr/local/share/examples/vm-bhyve/* /vms/.templates/

Now we can create a virtual switch (bridge) and bridge it to our network interface.  Change igb0 to your network interface below:

vm switch create public
vm switch add public igb0

Now, fetch an installation ISO for a FreeBSD guest:

vm iso ftp://ftp.uk.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/10.3/FreeBSD-10.3-RELEASE-amd64-disc1.iso

Now edit the default template /vms/.templates/default.conf (actually every .conf file in the folder) and change the disk config:

disk0_name=”disk0″
disk0_dev=”sparse-zvol”

This will make vms use zvols instead of disk image files.  Now we’re all prepared.

To create a test 16GB FreeBSD vm:

vm create -s 16G testvm
vm -f install testvm FreeBSD-10.3-RELEASE-amd64-disc1.iso

Select ‘vt100’ when asked for console type, and install as normal.  Once finished, login to the new vm and power it off (use ‘poweroff’) – control will return to the host.

Here’s some useful commands to manage VMs…

To list all your VMs:

vm list

To start a VM called testvm:

vm start testvm

To stop a VM (gracefully via ACPI) called testvm:

vm stop testvm

To forcibly power off a VM called testvm:

vm poweroff testvm

To press reset on a VM called testvm:

vm reset testvm

To connect to the serial console on a VM called testvm:

vm console testvm

(to disconnect, send ~. to exit the console session)

To permanently destroy a VM called testvm (must be turned off first):

vm destroy testvm

To edit the configuration of a VM called testvm (e.g. to change the MAC address):

vm configure testvm

To increase the size of the disk on a VM called testvm to 32GB:

zfs set volsize=32G zroot/vms/testvm/disk0

To create an Ubuntu guest using a previously downloaded ISO called ubuntu.iso:

vm create -t ubuntu -s 16G testubuntu
vm install testubuntu ubuntu.iso

To make Ubuntu boot unaided each time, edit the configuration file for the VM just created (in /vms/testubuntu/testubuntu.conf) and add:

grub_run_dir=”/grub”
grub_run_file=”grub.cfg”

You can also add the above to the ubuntu template file for it to be automatically added to ubuntu VMs.

Windows guests take a lot more effort as there is no GUI console.  This also means debugging a broken windows install is almost impossible so I would discourage its use for production.  However, this is how to prepare to install Windows VMs:

fetch -o /vms/.config/BHYVE_UEFI.fd http://people.freebsd.org/%7Egrehan/bhyve_uefi/BHYVE_UEFI_20151002.fd
pkg install git p7zip cdrtools-devel
mkdir /vms/work
cd /vms/work
git clone https://github.com/nahanni/bhyve-windows-unattend-xml
fetch https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.96/virtio-win-0.1.96.iso

This fetches a special UEFI loader, a collection of automated unattended Windows installation scripts, and the VirtIO drivers to allow storage to work.

Now copy a Windows installation ISO to /vms/work/win.iso and then:

mkdir tmp
cd tmp
7z x ../win.iso
cp /vms/work/bhyve-windows-unattend-xml/files/win2012r2_AutoUnattend.xml AutoUnattend.xml

This extracts the Windows ISO and copies an automated unattended installation script to it.  There are several AutoUnattend.xml files so choose the one matching the OS you’re installing… we’re doing a Windows 2012R2 install here.

You can edit the AutoUnattend.xml file to set a product key if required (not required for recent Windows OS)

Now we copy the VirtIO drivers into the temporary work directory and build a new ISO image:

mkdir virtio
cd virtio
tar xf /vms/work/virtio-win-0.1.96.iso
cd /vms/work/tmp
mkisofs \
-b boot/etfsboot.com -no-emul-boot -c BOOT.CAT \
-iso-level 4 -J -l -D \
-N -joliet-long \
-relaxed-filenames -v \
-V Custom -udf \
-boot-info-table -eltorito-alt-boot -eltorito-platform 0xEF \
-eltorito-boot efi/microsoft/boot/efisys_noprompt.bin \
-no-emul-boot \
-o /vms/.iso/win2k12r2.iso .

You now have a win2k12r2.iso file ready to build a windows VM:

vm create -t windows -s 64G wintest
vm install wintest win2k12r2.iso

This will start the unattended installation of windows.  It can take up to around 25 minutes.  You can monitor its progress by looking at the logfile:

tail -f /vms/wintest/vm-bhyve.log

You need to wait for it to restart twice (so three “starting bhyve” messages) – you can then get the IP from the server:

vm console wintest

and press “i” to view the IP address – this only seems to work on win2k12r2 and above.  You may need to check your DHCP server logs otherwise.

You can now RDP to the VM using Administrator and Test123 as the password.

Again, you should shutdown the VM and start it using the vm start wintest command.

To make VMs automatically start on boot, you can add them to /etc/rc.conf in the vm_list variable, for example to start wintest and testvm:

vm_list=”wintest testvm”

Enjoy.

11 thoughts on “Using FreeBSD as a Hypervisor (using bhyve and vm-bhyve to manage them)

  1. dan Post author

    Sometimes it can take a little while to RDP to the machine after its first boot, even though you get the IP address via the console. This is due to waiting for the RDP service to start which is competing for CPU time with the .net compilation done on first boot. Just be patient and RDP will respond!

    Reply
  2. dan Post author

    It’s possible to use bhyve without ZFS, but ZFS gives you so many more options (e.g. snapshots, clones etc) – check out ‘man vm’ to see some extra commands not covered in this document.

    Reply
  3. Dave Smith

    Dan, great intro. I have created a FreeBSD and a Debian 8.5 virtuals. I can get into them with
    the console login and password. But if the password is lost, I see no way to get in and reset the
    password. With ZFS, I am not sure I can see the virtual’s files to repair a problem and I don’t seem
    to be able to get a trusted console that does not require a password. Is there a way to do get in to
    help a user with a problem?

    Reply
    1. dan Post author

      You would target it as if it were a physical machine, so shut it down (using vm stop), then start it up (using vm start) and connect to the console straight away (with vm console).
      Do this quickly so you can access the boot loader where you can boot freebsd in single user mode to access the secure console without a password.
      (if the user has configured /etc/ttys to insecure then you would still be prompted for a password though)

      I’m not familiar with Linux, but presumably it has a similar method from the boot loader too.
      With FreeBSD guests, the boot loader only gives you 3 seconds – so it’s all about getting on the console fast!

      With ZFS on the host, you have a device node just like a physical disk device – in my example, you’d have /dev/zvol/zroot/vms//disk0
      You should always try to avoid accessing the guest’s disk from the host though.

      Reply
      1. Churchers

        Hey Dan,

        Thanks for the vm-bhyve write up 🙂

        In recent versions you can use the “loader_timeout” configuration option to change the amount of time bhyveload/grub-bhyve wait before booting the default option. Useful if you need to get to the boot menu.

        If you’re using UEFI graphics then you can use the “graphics_wait” option to force the guest to not boot until you have connected with VNC. I think this was mainly implemented so you could catch the Windows “press any key to boot from cd” option, but also works well for getting to the BSD/Linux loader screen.

        Reply
        1. dan Post author

          Thanks… For any readers out there, UEFI graphics etc is coming in FreeBSD 11 (or some manual compilation of bhyve now in 10.x).

          Reply
  4. Dominique Gagnon

    Thanks for the mini how-to, particularly setting zfs zvols as default storage..

    By the way, the proper way to exit an nmdm session is through “~” + “ctrl-d”. Pressing “~.” sometimes also ejects you from the ssh session on which you were while executing “vm console ..”.

    Reply
    1. dan Post author

      if it’s bridged to your igb0 network card, then you would treat it like any other physical server you wanted to connect.
      if you don’t have DHCP running, then you’d need to static assign an IP.

      Reply
  5. devlaam

    Instructive guide, thank you!
    If you run the installer from some linux distro, they will at some time ask if grub should be installed on the MBR. Is that needed? Or is the host side taking care of this?

    Reply
    1. dan Post author

      A good question, I’m not sure to be honest. I’ve always said yes as I treat them as if they are physical machines, but it may not be needed.

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *