To do this, you must be running at least FreeBSD 10.3-RELEASE – if you’re running an older version of FreeBSD, you should update! Also, ensure that you have a CPU that supports hardware virtualisation. Most new server CPUs seem to work just fine.
First, we need to install some packages to make things much easier:
pkg install vm-bhyve grub2-bhyve
Now we need to load some kernel modules on boot to allow us to virtualise and bridge networking… add the following to /boot/loader.conf:
if_bridge_load=”YES”
if_tap_load=”YES”
nmdm_load=”YES”
vmm_load=”YES”
You need to reboot to activate the modules, or you can load them now:
kldload if_bridge if_tap nmdm vmm
And we need to add some configuration to /etc/rc.conf:
vm_enable=”YES”
vm_dir=”zfs:zroot/vms”
vm_list=””
vm_delay=”5″
Now we will setup some storage for the hypervisor – For this, we’ll assume ‘zroot’ zfs pool has enough capacity:
zfs create -o mountpoint=/vms zroot/vms
vm init
cp /usr/local/share/examples/vm-bhyve/* /vms/.templates/
Now we can create a virtual switch (bridge) and bridge it to our network interface. Change igb0 to your network interface below:
vm switch create public
vm switch add public igb0
Now, fetch an installation ISO for a FreeBSD guest:
vm iso ftp://ftp.uk.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/10.3/FreeBSD-10.3-RELEASE-amd64-disc1.iso
Now edit the default template /vms/.templates/default.conf (actually every .conf file in the folder) and change the disk config:
disk0_name=”disk0″
disk0_dev=”sparse-zvol”
This will make vms use zvols instead of disk image files. Now we’re all prepared.
To create a test 16GB FreeBSD vm:
vm create -s 16G testvm
vm -f install testvm FreeBSD-10.3-RELEASE-amd64-disc1.iso
Select ‘vt100’ when asked for console type, and install as normal. Once finished, login to the new vm and power it off (use ‘poweroff’) – control will return to the host.
Here’s some useful commands to manage VMs…
To list all your VMs:
vm list
To start a VM called testvm:
vm start testvm
To stop a VM (gracefully via ACPI) called testvm:
vm stop testvm
To forcibly power off a VM called testvm:
vm poweroff testvm
To press reset on a VM called testvm:
vm reset testvm
To connect to the serial console on a VM called testvm:
vm console testvm
(to disconnect, send ~. to exit the console session)
To permanently destroy a VM called testvm (must be turned off first):
vm destroy testvm
To edit the configuration of a VM called testvm (e.g. to change the MAC address):
vm configure testvm
To increase the size of the disk on a VM called testvm to 32GB:
zfs set volsize=32G zroot/vms/testvm/disk0
To create an Ubuntu guest using a previously downloaded ISO called ubuntu.iso:
vm create -t ubuntu -s 16G testubuntu
vm install testubuntu ubuntu.iso
To make Ubuntu boot unaided each time, edit the configuration file for the VM just created (in /vms/testubuntu/testubuntu.conf) and add:
grub_run_dir=”/grub”
grub_run_file=”grub.cfg”
You can also add the above to the ubuntu template file for it to be automatically added to ubuntu VMs.
Windows guests take a lot more effort as there is no GUI console. This also means debugging a broken windows install is almost impossible so I would discourage its use for production. However, this is how to prepare to install Windows VMs:
fetch -o /vms/.config/BHYVE_UEFI.fd http://people.freebsd.org/%7Egrehan/bhyve_uefi/BHYVE_UEFI_20151002.fd
pkg install git p7zip cdrtools-devel
mkdir /vms/work
cd /vms/work
git clone https://github.com/nahanni/bhyve-windows-unattend-xml
fetch https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.96/virtio-win-0.1.96.iso
This fetches a special UEFI loader, a collection of automated unattended Windows installation scripts, and the VirtIO drivers to allow storage to work.
Now copy a Windows installation ISO to /vms/work/win.iso and then:
mkdir tmp
cd tmp
7z x ../win.iso
cp /vms/work/bhyve-windows-unattend-xml/files/win2012r2_AutoUnattend.xml AutoUnattend.xml
This extracts the Windows ISO and copies an automated unattended installation script to it. There are several AutoUnattend.xml files so choose the one matching the OS you’re installing… we’re doing a Windows 2012R2 install here.
You can edit the AutoUnattend.xml file to set a product key if required (not required for recent Windows OS)
Now we copy the VirtIO drivers into the temporary work directory and build a new ISO image:
mkdir virtio
cd virtio
tar xf /vms/work/virtio-win-0.1.96.iso
cd /vms/work/tmp
mkisofs \
-b boot/etfsboot.com -no-emul-boot -c BOOT.CAT \
-iso-level 4 -J -l -D \
-N -joliet-long \
-relaxed-filenames -v \
-V Custom -udf \
-boot-info-table -eltorito-alt-boot -eltorito-platform 0xEF \
-eltorito-boot efi/microsoft/boot/efisys_noprompt.bin \
-no-emul-boot \
-o /vms/.iso/win2k12r2.iso .
You now have a win2k12r2.iso file ready to build a windows VM:
vm create -t windows -s 64G wintest
vm install wintest win2k12r2.iso
This will start the unattended installation of windows. It can take up to around 25 minutes. You can monitor its progress by looking at the logfile:
tail -f /vms/wintest/vm-bhyve.log
You need to wait for it to restart twice (so three “starting bhyve” messages) – you can then get the IP from the server:
vm console wintest
and press “i” to view the IP address – this only seems to work on win2k12r2 and above. You may need to check your DHCP server logs otherwise.
You can now RDP to the VM using Administrator and Test123 as the password.
Again, you should shutdown the VM and start it using the vm start wintest command.
To make VMs automatically start on boot, you can add them to /etc/rc.conf in the vm_list variable, for example to start wintest and testvm:
vm_list=”wintest testvm”
Enjoy.
Sometimes it can take a little while to RDP to the machine after its first boot, even though you get the IP address via the console. This is due to waiting for the RDP service to start which is competing for CPU time with the .net compilation done on first boot. Just be patient and RDP will respond!
It’s possible to use bhyve without ZFS, but ZFS gives you so many more options (e.g. snapshots, clones etc) – check out ‘man vm’ to see some extra commands not covered in this document.
Dan, great intro. I have created a FreeBSD and a Debian 8.5 virtuals. I can get into them with
the console login and password. But if the password is lost, I see no way to get in and reset the
password. With ZFS, I am not sure I can see the virtual’s files to repair a problem and I don’t seem
to be able to get a trusted console that does not require a password. Is there a way to do get in to
help a user with a problem?
You would target it as if it were a physical machine, so shut it down (using vm stop), then start it up (using vm start) and connect to the console straight away (with vm console).
Do this quickly so you can access the boot loader where you can boot freebsd in single user mode to access the secure console without a password.
(if the user has configured /etc/ttys to insecure then you would still be prompted for a password though)
I’m not familiar with Linux, but presumably it has a similar method from the boot loader too.
With FreeBSD guests, the boot loader only gives you 3 seconds – so it’s all about getting on the console fast!
With ZFS on the host, you have a device node just like a physical disk device – in my example, you’d have /dev/zvol/zroot/vms//disk0
You should always try to avoid accessing the guest’s disk from the host though.
Hey Dan,
Thanks for the vm-bhyve write up 🙂
In recent versions you can use the “loader_timeout” configuration option to change the amount of time bhyveload/grub-bhyve wait before booting the default option. Useful if you need to get to the boot menu.
If you’re using UEFI graphics then you can use the “graphics_wait” option to force the guest to not boot until you have connected with VNC. I think this was mainly implemented so you could catch the Windows “press any key to boot from cd” option, but also works well for getting to the BSD/Linux loader screen.
Thanks… For any readers out there, UEFI graphics etc is coming in FreeBSD 11 (or some manual compilation of bhyve now in 10.x).
Thanks for the mini how-to, particularly setting zfs zvols as default storage..
By the way, the proper way to exit an nmdm session is through “~” + “ctrl-d”. Pressing “~.” sometimes also ejects you from the ssh session on which you were while executing “vm console ..”.
what should one configure during vm install for the network? I dont have dhcp on my igb0
if it’s bridged to your igb0 network card, then you would treat it like any other physical server you wanted to connect.
if you don’t have DHCP running, then you’d need to static assign an IP.
Instructive guide, thank you!
If you run the installer from some linux distro, they will at some time ask if grub should be installed on the MBR. Is that needed? Or is the host side taking care of this?
A good question, I’m not sure to be honest. I’ve always said yes as I treat them as if they are physical machines, but it may not be needed.
I’m struggling with networking when using vm-bhyve on FreeBSD 11.1-RELEASE. I have two NICs and have configured the first (igb0) on a management network and want to use the second (igb1) for VMs. However, I can’t get any VM to communicate through the virtual switch if I have igb1 added to it. If I take the NIC out of the switch and configure an ipv4 address on it, I can reach other hosts on the wider network so I believe the NIC and physical switch set-up are good. If I remove igb1 from the switch and add igb0, then create VMs on the same management network, they can access the outside world. Feels like I’m missing something basic here or can I only use an interface the is configured with an ip address and default router with my virtual switch?
Weird, I don’t usually apply IPs to the bridge or NIC that’s in use for the VMs – I usually let them talk upstream of the server.
I know that VMs have trouble talking to the host when i’ve used bhyve (but not usually an issue as the VMs shouldn’t need to talk to the HV generally) – never really looked into why though.
Hi Dan,
Don’t know if you are still out there, but I’m looking for some advice on observed behaviour on the HV host.
I’m seeing swap usage for each process running “/bin/sh /usr/local/sbin/vm -tf _run {vm name}”. Is that expected behaviour?