virtio-nvgpu: NVIDIA GPU in a KVM guest without passthrough

The stated target is headless streaming, not model serving. A compositor inside the VM renders, composites and encodes frames on the GPU, and only the compressed bitstream leaves; the VM has no monitor and the host keeps the card. The measured artefact is 618 frames of H.264 that ffmpeg decodes without an error, produced by a Wayland client presenting inside a guest. The repository page is dated 24 September 2026, marked experimental, and carries 108 commits, 193 stars and 2 forks.

VFIO passthrough gives native performance and a complete driver stack in the guest, but it dedicates the whole GPU to one VM. Commenters on Hacker News made the practical version of that point: with normal passthrough the host loses access to the card, so a single-GPU box needs a second GPU or a fallback to integrated graphics to stay usable. virtio-gpu with Venus instead serialises every Vulkan or OpenGL call in the guest, transports it over virtio and replays it host-side, which the README costs out at roughly 2,000 boundary crossings per frame against 5 to 20 for virtio-nvgpu. Games issue 1,000 to 5,000 draw calls per frame; at 60 fps the frame budget is 16.6 ms, and 1 to 3 ms of serialisation is 6 to 18% gone before any GPU work starts.

The other Venus problem is ownership. GPU buffers belong to the host, so a guest compositor cannot see or import them and there is no practical path to a CUdeviceptr in the guest pointing at a Venus-managed buffer, which means no NVENC without a full CPU readback and copy. DRM native context solves this for Intel and AMD by letting the guest run the real Mesa driver and crossing the boundary only on submission. It does not exist for NVIDIA.

The benchmark runs an identical headless Vulkan load on an RTX 3060, driver 595.99.02, guest against the same host bare metal. At a 39 ms frame the guest is 0.4% faster than bare metal, which the README calls noise. At 9.9 ms it is 0.7% faster, at 2.0 ms it is 1.7% slower, at 0.5 ms it is 7.1% slower, and at 0.05 ms it is 40.8% slower; a wake costs about 0.02 ms, and at that point the frame is half of one wake. The claim is therefore narrow: above about 2 ms a frame, a guest is within 2% of bare metal.

The CPU figure explains the mechanism. Unpaced at roughly 100 fps for 12 s, the host took 0.40 s of CPU and the guest 0.37 s. Nothing is forwarded in the render loop because NVIDIA's user-mode driver submits through memory it has mapped, and that memory is the host's. Over 813,691 frames the backend served 13,792 messages, one crossing per 59 frames, nearly all of it device setup.

The confirmed working list is a guest enumerating the card with nvidia-smi reporting real power, memory and the host's deviceUUID; Vulkan rendering with vulkaninfo exiting 0 and pixel-correct offscreen draws; a Wayland client presenting through a guest compositor; NVENC through Vulkan Video, encoding on the client's own device; and imported buffers that are host memory mapped through a shared window. CUDA appears in the targeted scope - device memory allocation, zero-copy CUDA to Vulkan or GL interop with GPU-side pointers, NVENC from CUDA device pointers, NVDEC - but the repository's own list of what is not done says CUDA is forwarded and untested beyond enumeration.

Explicitly out of scope: cudaMallocManaged and full unified virtual memory, scanout, MIG and SR-IOV. UVM and modeset ioctls also have no ABI filtering table. A serving stack that relies on managed memory has no story here, and no guest has been run on anything heavier than vkcube at 720p.

Four guests running the same load on one RTX 3060 produced 25.84, 26.49, 25.57 and 25.79 fps - 103.7 together against 102.9 for a single guest - with p50 frame times of 39.165, 39.164, 39.168 and 39.165 ms. All four rendered correctly at once and all four encoded H.264 simultaneously, each paced at exactly 60 Hz, with no NVENC session limit reached. The README is careful about what this is: four is what was run, not a limit found. Eight has not been tried.

There is no IOMMU boundary between guest GPU work and the host. The card belongs to the host's NVIDIA driver and sits in the host's IOMMU domain; the guest gets the driver's ioctl interface, not the device. What separates guest from host memory is the GPU's own MMU, with page tables RM programs on the guest's behalf, which puts the host NVIDIA driver in the TCB. The guest also authors its own command streams, which is precisely why there is no per-submission cost.

What narrows the surface today is that ioctls the ABI profile does not describe are refused rather than forwarded, with a --permissive-abi flag that turns the refusal off for diagnosis and says so loudly. What does not: RM_ALLOC classes and RM control commands are unfiltered, UVM and modeset have no equivalent table, and the backend holds the host descriptors inside the VMM's own process. The unprivileged per-guest isolate is designed and unbuilt; isolate/ holds the design note and no code.

The README states that VFIO passthrough with an IOMMU is strictly stronger because it constrains the device to the guest's own memory, and that for mutually untrusted tenants VFIO or vGPU is still the answer. That settles the multi-tenant question for anyone running untrusted workloads on shared hardware.

Supported driver versions are explicit, as with nvproxy. Shipped ABI profiles are 535.129.03, 580.178.04 and 595.71.05, matched by range, with anything older than the first refused. The measured card is an RTX 3060 on 595.99.02; an RTX A2000 on 615.71.09 renders but is not benchmarked. Arbitrary NVIDIA driver versions are out of scope, so a host driver outside a shipped range needs a new profile before a guest can use it.

The guest kernel module registers /dev/nvidiactl, /dev/nvidia0 through N and /dev/nvidia-uvm, serialises each ioctl onto a control virtqueue, maps shared-memory regions on mmap with the correct caching attributes, and makes no ABI decisions of its own. The device crate maps guest handles to host device file descriptors and performs the ABI-aware translation, rewriting embedded pointers and file descriptors before issuing against the host's devices. A second virtqueue runs the other way: the host watches each descriptor it has opened and reports when one becomes readable, which is how a guest waiting for the GPU is woken. Without it the guest cannot wait at all and spins on a descriptor the kernel reports as permanently ready.

The repository has four components across three licence zones: driver/ is GPL-2.0 because the guest kernel module touches kernel symbols, device/ and isolate/ are Apache-2.0, and protocol/ is BSD-3-Clause OR GPL-2.0+ so both halves can include the same headers. gen/ holds generated, checked-in, reproducible ABI tables, and the layout follows chromeos/virtio-media. device/ has no VMM in its dependency list - every VMM concern is a trait, so a VMM adopts the device by implementing descriptor chains as Read and Write, an event queue, guest memory mapping and host memory mapping. The isolate will not be a trait: adopting it eventually means inheriting a process model, one sandboxed helper per guest process.

Commenters on Hacker News raised three substantive objections. The first is the benchmark framing: one commenter called the percentage-overhead comparison nonsense on the grounds that it uses percentages to argue overheads do not matter when the system is slow anyway, and said a fair comparison would be this project against virtio. The README does not contest that directly; it says the numbers do not support a comparison with any other hypervisor, because none was run.

The second is overlap with gVisor's nvproxy. The author's answer in the thread is that the architectural design was borrowed from nvproxy and then built out to support graphical workloads, and that it is reusable enough to hot-plug into a microVM, cloud-hypervisor, or possibly Firecracker. The third is security: one commenter noted the lack of discussion about what level of access the card has to the host absent IOMMU-restricted passthrough, and another was sceptical that NVIDIA GPUs can be shared across host and guests without licensed drivers. One commenter raised vGPU as a fourth option, splitting a physical GPU's memory and time budget across KVM guests and not legally available on consumer-grade GPUs; another replied that consumer vGPU support is more or less just drivers and can be patched back in. Questions about live migration between hosts and about GPU or canvas-based bot detection got no answer in the sources.

Three things would change the calculus for an inference host. First, any CUDA measurement beyond enumeration - the whole targeted scope list, including zero-copy CUDA to Vulkan interop and NVENC from CUDA device pointers, currently rests on design intent. Second, the isolate: until that sandboxed per-guest helper exists, the backend holds real device FDs inside the VMM process and RM control commands go through unfiltered. Third, a second card and a second driver version, since RTX 3060 on 595.99.02 is the only benchmarked configuration and everything measured is Vulkan rendering and H.264 encode.