Virtio, vhost, VFIO, and IOMMU-backed assignment

Before you start#

Review DMA and IOMMUs, device lifetimes, block I/O, and KVM. Linux v6.12 is the historical source baseline. A virtio device, vhost backend, VFIO device and IOMMU each require separate configuration and ownership. This lesson does not bind, reset, map or assign a host device.

Learning goals#

Trace virtio feature negotiation and a block completion, explain split and packed queue ownership, distinguish vhost from VFIO, and map guest, I/O and host address spaces without treating their numeric addresses as interchangeable.

Source walkthrough#

Virtio is a family of device interfaces between a driver and a device implementation. A guest virtio driver might use PCI or MMIO transport to communicate with a virtual device implemented by QEMU, a kernel backend, or another supported component. The transport discovers the device, provides configuration and notification mechanisms, and carries feature and status information. The device type defines operations such as network packets or block requests; transport, device type and backend are distinct layers. A device can also be used outside a conventional VM, so this is a representative arrangement rather than a universal topology.

During probe, the driver and device negotiate features. virtio.c implements virtio_dev_probe. This exact v6.12 GPL-2.0-only excerpt omits the preceding table construction and later transport-feature handling:

	if (device_features & (1ULL << VIRTIO_F_VERSION_1))
		dev->features = driver_features & device_features;
	else
		dev->features = driver_features_legacy & device_features;

The device's version feature selects the modern or legacy driver feature table; the bitwise intersection keeps features offered by both sides. The surrounding code preserves transport features, finalizes negotiation, optionally validates the result, verifies feature acceptance and calls the driver probe. Any of those stages can fail and follow the probe error path; it is not enough that a device advertised a bit. Device status tracks stages including driver recognition, features accepted, driver ready and failure. A reset is a lifecycle operation with transport and driver requirements, not a substitute for waiting until outstanding accesses are safe.

Modern virtio uses specified little-endian field conventions, while legacy interfaces have transport and guest-endian distinctions. Read the negotiated version and exact transport/device specification rather than assuming that a host C structure has the correct byte order or alignment. Feature negotiation also affects optional behavior: an event-index or packed-ring algorithm must not be assumed just because the implementation contains code for it.

Queue ownership, publication and wrap#

A virtqueue transfers ownership of buffers between driver and device. In the split-ring layout, a descriptor table describes address, length, flags and optional next descriptor. An available ring publishes head descriptor indices supplied by the driver; a used ring reports completed heads and lengths supplied by the device. A descriptor chain can represent multiple segments, and an indirect descriptor table can reduce pressure on the main descriptor ring when negotiated. Descriptor addresses are device-visible addresses under the applicable transport and DMA mapping rules, not necessarily host virtual pointers.

The descriptor layout is in include/uapi/linux/virtio_ring.h, v6.12; the header comment says it is BSD licensed (surrounding code omitted):

struct vring_desc {
	__virtio64 addr;
	__virtio32 len;
	__virtio16 flags;
	__virtio16 next;
};

Field order and types give each descriptor 16 bytes: addr at offset 0, len at 8, flags at 12 and next at 14. The __virtio types mean the byte order follows the virtio transport and version rules, which this excerpt does not show. A descriptor identifies a buffer; it does not prove the device has finished using it.

The driver initializes descriptors before publishing availability; the device must see the descriptor contents before it sees the updated available index. Corresponding ordering is needed when consuming used entries before reclaiming buffers. Linux's virtio_ring.c contains the barriers and transport-dependent ordering logic. A kick notifies a device that work may be available; a callback or interrupt notifies the driver about completion. Notifications can be suppressed or coalesced according to negotiated mechanisms, so a missing notification is not itself proof that no entry changed. The event-index feature uses event thresholds to reduce notifications; it does not remove the need to check indices and memory ordering.

The notification side is in virtio_commit_rqs, v6.12, SPDX GPL-2.0-only (surrounding code omitted):

static void virtio_commit_rqs(struct blk_mq_hw_ctx *hctx)
{
	struct virtio_blk *vblk = hctx->queue->queuedata;
	struct virtio_blk_vq *vq = &vblk->vqs[hctx->queue_num];
	bool kick;

	spin_lock_irq(&vq->lock);
	kick = virtqueue_kick_prepare(vq->vq);
	spin_unlock_irq(&vq->lock);

	if (kick)
		virtqueue_notify(vq->vq);
}

The driver asks the virtqueue whether a notification is needed while holding the queue lock, then sends it after unlocking. Submission and notification are therefore separate steps in this driver. The excerpt does not show descriptor publication, so it does not establish the ordering between them, and a notification is not proof that the device handled the request.

Packed rings use descriptor availability and used state with wrap counters in a different layout. They are not simply a split ring with a different name: each side must interpret descriptor flags and wrap state correctly. Consider a synthetic split ring of size eight. The driver publishes heads in slots 6, 7 and 0 as its monotonically wrapping index crosses a ring boundary; the storage slot uses the ring size while the index's wrapping arithmetic tracks progress. The device completes entries and advances its used index; the driver must consume exactly the completed heads and reclaim their chains. Slot 0 after wrap is not proof that the queue is empty or that a prior slot 0 buffer has already been reclaimed. This is an illustrative sequence, not captured guest activity.

Virtqueue buffers remain owned by the appropriate side until the protocol reports completion or the device is reset and the driver has followed its teardown contract. On removal or failed setup, callbacks and backend accesses must be quiesced, queues reset or deleted as required, and outstanding requests completed or failed before backing memory and driver state are freed. A notification race can occur between observing an empty queue and enabling callbacks; the driver must use the queue API's recheck pattern rather than simply enabling interrupts once and assuming no completion was missed.

A real virtio-block completion#

In drivers/block/virtio_blk.c, virtblk_done processes completed buffers. This exact v6.12 GPL-2.0-only excerpt omits the lock, request conversion and surrounding loop:

		virtqueue_disable_cb(vq);
		while ((vbr = virtqueue_get_buf(vblk->vqs[qid].vq, &len)) != NULL) {

The callback first disables queue callbacks while it drains completions. It then repeatedly obtains used buffers until none is available. The full function converts each returned request-private object to a block request and normally asks blk-mq to complete it; the surrounding loop re-enables callbacks and repeats if a completion raced with re-enabling. A per-queue spinlock with interrupt-state handling protects that critical section. The request status and later block completion path determine success or error; merely receiving a used buffer is not evidence that the disk write is durable.

For a concrete request, a guest submits a block read whose header, writable data buffer and status buffer are described through a queue. The backend consumes the available chain and eventually returns its head in the used ring. The guest callback retrieves the corresponding virtblk_req, then the block layer handles completion. If queue submission fails for lack of descriptors, the request has not been handed off as a normal successful submission and the caller must follow the driver's retry/error path. If the device disappears while requests are outstanding, teardown must coordinate queue access and complete or fail them safely. Source presence does not establish that your machine has a virtio block device or that this particular request ran.

vhost and VFIO have different jobs#

vhost is a family of backends that can move supported virtqueue processing into the host kernel. The VMM still sets up memory and queues, and supported device/backend features and ownership determine the actual path. In vhost.c, vhost_dev_set_owner associates a backend with an owner context under its locking rules; memory tables, eventfds and IOTLB messages can then participate in access and notification. Eventfd provides a counter-based notification mechanism, not a copy of the packet or disk payload. Host address-space lifetime, memory access validation, IOTLB invalidation and backend shutdown matter while descriptors are in flight.

The vhost_dev_set_owner path makes this lifetime concrete. If the backend already has an owner it returns -EBUSY. Otherwise it attaches the owner's memory-management context with vhost_attach_mm, allocates I/O vectors, and, when configured to use a worker, creates that worker and attaches queues to it. If worker creation fails, the error labels free the vectors and detach the memory context; if vector allocation fails, the context is detached without pretending ownership setup succeeded. The source comments note why worker creation is ordered after the allocation. Queue access, worker lifetime and the VMM's memory context must remain coordinated; an eventfd notification alone does not keep an arbitrary address valid. The IOTLB describes device-visible mappings and permissions, so invalidation and backend response matter when the VMM changes mappings.

A vhost-user backend is a separate userspace protocol and implementation, usually coordinated with a VMM over a socket and shared memory. It is not the same kernel object as a vhost backend. Protocol versions, supported features and backend process lifetime must be pinned independently of the Linux tag when used; this chapter does not assume a particular vhost-user project or version. Both approaches have trust and memory-sharing boundaries, and neither converts a virtqueue descriptor into a safe host pointer without validation.

VFIO instead supports controlled userspace access to devices under kernel and IOMMU constraints. The legacy interface organizes devices into IOMMU groups and containers, with group viability and isolation relevant to assignment. Newer v6.12 interfaces can use device character devices and iommufd where configured; the exact binding, IO address space and compatibility flow differ from legacy group/container ioctls. Consult the VFIO core, iommufd integration and matching documentation for the selected path. This lesson excludes unsafe no-IOMMU operation.

An IOMMU group expresses a minimum practical isolation boundary for devices sharing relevant topology or translation limitations; assigning one function without considering the group can expose other devices. Interrupt remapping, device reset support, firmware, peer-to-peer paths and hot unplug also matter. DMA map operations establish mappings and associated page references/pinning under the selected VFIO/IOMMU interface; unmap and teardown must coordinate in-flight DMA before pages are released. Interrupt eventfds signal events to userspace; reset does not automatically guarantee every device and platform has identical isolation or recovery behavior.

A three-address example#

Suppose guest software has a buffer at guest physical address 0x20000, while the VMM maps its backing at host virtual address 0x7f000000. A virtual or assigned device is given an I/O virtual address 0x40000 that the relevant device translation maps to the intended backing. These numbers are synthetic and deliberately different. GPA is a guest address-space concept, HVA belongs to a host process, and IOVA is interpreted by a device/IOMMU domain. The host physical pages behind the HVA can also be different and can change subject to pinning and mapping contracts. No arithmetic relationship between these addresses is guaranteed without the actual mapping metadata.

If a device writes after its mapping has been removed and pages reused, the result can be corruption or a fault; the driver and userspace owner must stop or synchronize DMA before unmapping and releasing memory. If a hot-unplug arrives during an operation, notification, request completion, queue deletion and object references require coordinated teardown. These are distinct from a guest application closing a file. A virtio backend and a VFIO-assigned device can each require careful lifetime management, but their APIs and trust boundaries are not interchangeable.

Negotiation and teardown as a single lifecycle#

Consider a virtio block device offering a set of features including modern virtio and an optional notification feature. The driver declares only the features it understands. The core intersects the device and driver sets for ordinary device features and preserves the applicable transport features for finalization. If the transport cannot finalize the selected set, or the device does not accept the features, the driver cannot proceed as if negotiation succeeded. A feature bit in a host configuration screen is therefore not enough: the running guest's negotiated state, transport and backend must agree.

After a successful probe, the driver creates queues and posts requests using DMA addresses and buffers whose ownership is tracked until completion. A failed descriptor allocation or mapping requires unwinding whatever was already acquired. Once a descriptor chain has been published, reusing or unmapping its buffers before the device returns ownership is a different and more dangerous error than failing before publication. The specific driver and transport determine how a reset synchronizes with device access. In the block callback, returning a used descriptor identifies a completed queue operation, while status and block-layer completion decide the request result; application-level durability can require additional flush or ordering guarantees.

For assignment, the sequence is similarly stateful but uses different objects. A userspace owner must establish an authorized device and IOMMU context, map the intended address range, configure interrupts and only then let the device perform DMA. An unmap request cannot be treated as proof that a misbehaving or unsynchronized device stopped issuing transactions. Device reset support, interrupt isolation, peer access and group composition are separate platform properties. If the device is unplugged or the owner exits, the kernel and VMM must coordinate file references, mappings, interrupt notification and teardown. The safe response to an unavailable group or unsupported reset is to record that the assignment cannot be established under the stated requirements, not to bypass isolation.

Practice#

Read-only device inventory

On an authorized host or disposable guest, inspect existing device and group names only:

uname -r
for p in /sys/bus/virtio/devices /sys/kernel/iommu_groups; do
  if [ -d "$p" ]; then printf '\n%s\n' "$p"; ls -l "$p"; else printf '%s unavailable\n' "$p"; fi
done
if [ -r /proc/modules ]; then awk '$1 ~ /^(virtio|vhost|vfio)/ {print $1, $2, $3}' /proc/modules; fi

A listed virtio device does not identify its negotiated features or prove a request completed; an IOMMU group listing does not prove the entire platform is safe for assignment. Built-in drivers may not appear in /proc/modules. Missing directories, empty output and permission errors are valid results. Do not unbind a driver, open device-assignment interfaces, map DMA, reset hardware or change permissions.

Source-only alternative: draw the split-ring slot sequence 6, 7, 0 and show when ownership transfers and returns. Then use the supplied distinct GPA, HVA and IOVA values to identify the mapping owner and one teardown race. Add the virtio-block callback's disable/drain/re-enable sequence. Deliver the diagram or actual inventory with the kernel version and an explicit statement that no guest I/O or assignment was run.

Knowledge check#

1. Why does a device-advertised feature not prove the driver can use it?
2. Why must descriptor contents be visible before publishing an available index?
3. What race does the virtio-block callback's drain/re-enable loop address?
4. Is a vhost backend the same object as a VFIO-assigned physical device?
5. Why can a GPA, HVA and IOVA have different numeric values for related storage?

Answers#

1. Probe intersects device and driver features and still requires transport finalization, validation and device acceptance. Unsupported or failed negotiation prevents use.
2. The device must not observe a published head before the corresponding descriptor data is initialized and visible under the queue's ordering rules.
3. Completion can arrive while callbacks are disabled or being re-enabled. The queue API's recheck causes another drain when necessary rather than relying on a possibly missed notification.
4. No. vhost processes supported virtqueues as a backend; VFIO exposes controlled device access with IOMMU and ownership constraints. Their setup and isolation requirements differ.
5. They belong to different address spaces: guest physical, VMM virtual and device-visible I/O virtual. Explicit mappings relate them; numerical equality is not required.

Checkpoint#

Submit the wrapped queue and address-space diagrams plus a request completion and teardown timeline. Ready means ownership, ordering, feature negotiation and vhost/VFIO boundaries are distinct, with no claim of an unrun device test. Retry the pinned probe and completion functions if any buffer is freed before its owner returns it. This is not certification.

Sources#

• Virtio core v6.12
• Virtio ring v6.12
• Virtio block v6.12
• vhost core v6.12
• VFIO core v6.12
• VFIO documentation v6.12
• IOMMUFD documentation v6.12
• Virtio kernel documentation
• OASIS Virtual I/O Device (Virtio) Specification 1.2