# GPU passthrough for bhyve on FreeBSD 14 21-05-2024 **Table of Contents** - [GPU passthrough for bhyve on FreeBSD 14](#gpu-passthrough-for-bhyve-on-freebsd-14) - [Why even bother](#why-even-bother) - [Lets get started!](#lets-get-started) - [buildkernel](#buildkernel) - [make bhyve](#make-bhyve) - [Bhyve passthrough](#bhyve-passthrough) - [Update for newer releases](#update-for-newer-releases) - [References](#references) We recently purchased a GPU for our University computer club [[1]] for Machine Learning tasks. Being most comfortable on a FreeBSD system I decided that running a VM through bhyve would be most suitable. Especially as I expect that several students will connect to this machine concurrently using harmful software such as Visual Studio Codes remote development over SSH extension. ## Why even bother From experience I know that VSCode launches a gazillion processes and wont kindly terminate them. It also downloads binaries from microsoft servers and runs a little node session remote on the server for each user and doesnt reuse old ones. A bonus is that it doesnt run naively on FreeBSD (You can get this working using linuxulator [[2]] but dont bother). But the problem is that meanwhile Bhyve does support PCI-e passthrough its GPU support is lacking according to people online. [[3]] > bhyve supports passing of host PCI devices to a virtual machine for its exclusive use of them. > Note: VGA / GPU pass-through devices are not currently supported. That last part is not really true, the problem lies within some drivers (NVIDIA) only supporting hypervisors with the KVM signature, so we need to patch Bhyve. Luckily patches are already available [[4]], and the instructions are as simple as following the FreeBSD developers handbook [[5]]. ## Lets get started! First get the sources for FreeBSD, you probably only care about the branch with the patches applied. They reside on github from the Beckhoff github account under the tag phab/corvink/14.0/nvidia-wip. ``` cd /where/you/want/to/build git clone --depth 1 --branch phab/corvink/14.0/nvidia-wip https://github.com/Beckhoff/freebsd-src ``` ### buildkernel Now we can either use the script provided by corvink@ [[6]] or just follow the handbook. ``` cd freebsd-src make -j64 buildkernel make installkernel ``` Now the new kernel is installed, the previous kernel is located at `/boot/kernel.old/kernel` ### make bhyve Now its time to build bhyve with our patches, this will go quick. The provided script automates this for you. (really, use the script [[6]]) ``` make -C include -j64 make -C include install make -C lib/libvmmapi -j64 make -C lib/libvmmapi install make -C sys/modules/vmm -j64 make -C sys/modules/vmm install make -C usr.sbin/bhyve -j64 make -C usr.sbin/bhyve install make -C usr.sbin/bhyvectl -j64 make -C usr.sbin/bhyvectl install make -C usr.sbin/bhyveload -j64 make -C usr.sbin/bhyveload install ``` Really, just use the script. But now you should be ready to reboot into your freshly patches system! For future FreeBSD releases you can simply go in and apply the patch manually if necessary. ### Bhyve passthrough Now to have our pci device to passthrough to bhyve we first figure out what id its assigned. This is basically following the wiki page at [[3]]. ``` $ pciconf -v -l | grep -B 1 -A 3 NVIDIA ppt0@pci0:193:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x2204 subvendor=0x1028 subdevice=0x3880 vendor = 'NVIDIA Corporation' device = 'GA102 [GeForce RTX 3090]' class = display subclass = VGA ppt1@pci0:193:0:1: class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1aef subvendor=0x1028 subdevice=0x3880 vendor = 'NVIDIA Corporation' device = 'GA102 High Definition Audio Controller' class = multimedia subclass = HDA ``` Now note down those numbers and insert them into /boot/loader.conf as following ``` # echo pptdevs="193/0/0 193/0/1" >> /boot/loader.conf ``` Now the host wont steal them when the system is started, now to map them into Bhyve. If you want to passthrough another device, such as an USB hub to the VM you can specify more devices by appending `N` to pptdevs such as `pptdevs2="194/0/0"`. We use the excellent bhyve-vm [[7]] utility to simplify vm management. Simply add the following to the end of your $VM.conf file, they will now show up in pci slots 9:0 and 9:1 on the VM. You might want to make sure that you dont make them to a spot thats already occupied, if so just go higher. ``` passthru0="193/0/0=9:0" passthru1="193/0/1=9:1" ``` And presto! Now you can reboot your FreeBSD host and your VM will have access to the GPU. On a linux VM you can verify that the patches were applied correctly by checking your hypervisor signator by running `dmesg` ``` # dmesg | grep KVM ``` # Update for newer releases The patch still works on 14.1 it just needed some tiny refactoring as `read_config` was renamed to `pci_host_read_config`. A patch file is provided [nvidia.patch](./nvidia.patch). Apply it by running `git apply -v nvidia.patch` after cloning 14.1 by running `git clone --depth 1 --branch releng/14.1 https://github.com/freebsd/freebsd-src` # References [[1]] https://df.lth.se/ [[2]] https://wiki.freebsd.org/Linuxulator [[3]] https://wiki.freebsd.org/bhyve/pci_passthru [[4]] https://github.com/Beckhoff/freebsd-src/tree/phab/corvink/14.0/nvidia-wip [[5]] https://docs.freebsd.org/en/books/developers-handbook/kernelbuild/ [[6]] http://okturing.com/src/19931/body [[7]] https://github.com/churchers/vm-bhyve [1]: https://df.lth.se/ [2]: https://wiki.freebsd.org/Linuxulator [3]: https://wiki.freebsd.org/bhyve/pci_passthru [4]: https://github.com/Beckhoff/freebsd-src/tree/phab/corvink/14.0/nvidia-wip [5]: https://docs.freebsd.org/en/books/developers-handbook/kernelbuild/ [6]: http://okturing.com/src/19931/body [7]: https://github.com/churchers/vm-bhyve