KVM, vCPUs, stage-two translation, and VM exits

Before you start#

Review virtual memory and page tables, CPU privilege and I/O, interrupts, and cgroup scheduling. This chapter reads upstream v6.12. The main example is conditional on x86-64 KVM; arm64 uses different architecture contracts. No VM or assigned device is created in the observation lab.

Learning goals#

Trace KVM's device, VM and vCPU file-descriptor interfaces, distinguish guest and host address spaces, follow a memory-slot validation failure and a userspace-visible exit, and identify what a KVM capability query cannot prove.

Source walkthrough#

KVM provides kernel interfaces and architecture-specific virtualization support. QEMU is a separate userspace virtual machine monitor (VMM) that can use KVM as an accelerator, manage guest memory and devices, and handle exits. QEMU without KVM can use software emulation; KVM is not the whole guest machine or a replacement for QEMU's device models. Other VMMs can use KVM. A guest and host share the host kernel's KVM implementation as a security boundary; guest isolation depends on hardware, configuration, kernel and VMM correctness.

The KVM UAPI exposes /dev/kvm and ioctls. A device fd is used for global capability and VM creation operations. A successful KVM_CREATE_VM returns a VM fd representing a struct kvm; a VM can create vCPU fds associated with struct kvm_vcpu and, where supported, device fds. These are file-descriptor references with distinct operations and lifetimes. The VM and its resources remain alive until the last VM-fd reference is released. However, the ioctl contract still requires VM ioctls to come from the process (address space) that created the VM; vCPU ioctls normally come from the thread that created that vCPU, except documented asynchronous operations. Passing an fd with fork() or SCM_RIGHTS is not a supported way to migrate KVM control to an unrelated process.

In virt/kvm/kvm_main.c, kvm_dev_ioctl handles the device-level query. The exact v6.12 GPL-2.0-only excerpt below omits the surrounding switch cases:

	case KVM_GET_API_VERSION:
		if (arg)
			goto out;
		r = KVM_API_VERSION;
		break;

A nonzero argument follows the default error path; a zero argument returns the compiled UAPI version, defined as 12 for this interface. This establishes an API version only. It does not prove that a VM can be created, a particular extension exists, nested virtualization is enabled, or a guest will boot. Use KVM_CHECK_EXTENSION and architecture-specific documentation for individual capabilities, and still handle operation failure.

A conditional x86 VMM sequence#

A minimal x86 VMM commonly opens /dev/kvm, issues KVM_GET_API_VERSION, creates a VM, allocates suitable userspace-backed memory and configures a memory slot with KVM_SET_USER_MEMORY_REGION. It creates a vCPU, obtains KVM_GET_VCPU_MMAP_SIZE, maps the vCPU's shared struct kvm_run, configures initial state using appropriate x86 register and special-register ioctls, and calls KVM_RUN. This is an API outline, not complete executable code or a universal boot protocol. A real boot also needs a chosen CPU mode, guest image, memory layout, interrupt/timer configuration and device model. Architecture-specific ioctls and initial state differ on arm64.

The VM fd and vCPU fd operations belong to different objects. The shared kvm_run mapping is a protocol area between userspace and the vCPU ioctl; its exit_reason and union fields are valid according to the exit returned. The VMM must read the appropriate member for the actual reason and use the documented re-entry protocol. A failed ioctl must be handled before reading results as if they were valid. A vCPU usually runs on a host thread that calls KVM_RUN; the host scheduler can preempt or migrate that thread subject to affinity, cgroups and scheduling policy. A guest's virtual CPU count is not a reservation of equal physical CPUs.

The KVM_RUN case of kvm_vcpu_ioctl in virt/kvm/kvm_main.c, v6.12, SPDX GPL-2.0-only (surrounding dispatch omitted):

	case KVM_RUN: {
		struct pid *oldpid;
		r = -EINVAL;
		if (arg)
			goto out;
		oldpid = rcu_access_pointer(vcpu->pid);
		if (unlikely(oldpid != task_pid(current))) {
			/* The thread running this VCPU changed. */
			struct pid *newpid;

			r = kvm_arch_vcpu_run_pid_change(vcpu);
			if (r)
				break;

			newpid = get_task_pid(current, PIDTYPE_PID);
			rcu_assign_pointer(vcpu->pid, newpid);
			if (oldpid)
				synchronize_rcu();
			put_pid(oldpid);
		}
		vcpu->wants_to_run = !READ_ONCE(vcpu->run->immediate_exit__unsafe);
		r = kvm_arch_vcpu_ioctl_run(vcpu);
		vcpu->wants_to_run = false;

If the calling thread differs from the one recorded for this vCPU, the handler records the new pid after architecture preparation, and waits for an RCU grace period before releasing an old pid when one existed. The flag is read through immediate_exit__unsafe, a kernel-side name; the UAPI header marks the field with HINT_UNSAFE_IN_KVM, so userspace should rely on the documented field semantics. The excerpt does not show what the architecture run function does on exit.

KVM_RUN can return for a signal, an error or a userspace-visible exit. On x86, KVM_EXIT_IO can describe port I/O that the VMM must handle using the supplied direction, size, count and data offset. KVM_EXIT_MMIO can require userspace to emulate a memory-mapped device access, with direction and length from the exit structure. KVM_EXIT_HLT indicates a halt exit when that exit is delivered to userspace; a VMM decides what that means for its machine model. An exit must not be interpreted as proof that the guest completed all work or that another vCPU is stopped. Many hardware exits are handled inside KVM and never appear as a userspace KVM_RUN return.

Memslots, references and invalidation#

A KVM memory slot relates a guest physical address range to userspace-backed memory, subject to the selected API, flags and kernel support. The slot's guest physical base is not itself a host virtual or host physical address. struct kvm_memory_slot holds the kernel's representation; updates are coordinated with VM locks and memory-slot publication and readers use appropriate SRCU-protected access. The VMM must keep its backing memory and VM address-space relationship valid under KVM's contract. The kernel uses memory-management notification and invalidation mechanisms when host mappings change; device assignment and guest-private memory introduce further configuration-specific constraints.

In __kvm_set_memory_region, this exact v6.12 GPL-2.0-only excerpt checks basic alignment and size; preceding flag checks and later validation are omitted:

	if ((mem->memory_size & (PAGE_SIZE - 1)) ||
	    (mem->memory_size != (unsigned long)mem->memory_size))
		return -EINVAL;
	if (mem->guest_phys_addr & (PAGE_SIZE - 1))
		return -EINVAL;

The first condition rejects a memory size not aligned to the host kernel page size or not representable as required by the cast. The next rejects an unaligned guest physical base. Later checks validate the userspace address, range overflow, slot IDs and other flags. The function's comment warns that a pointer to an old slot can be invalidated or destroyed by an update; callers cannot retain it beyond the protection and lifetime contract. A size of zero requests a deletion only for an existing slot, subject to the later checks; it is not an arbitrary way to create empty guest memory.

For a hypothetical host with 4 KiB base pages, a requested guest physical base 0x1003 fails this alignment check with -EINVAL. Correcting it to 0x2000 passes that one condition, but does not establish that the host address, size, overlap, flags or slot index are valid. If validation fails before publication, the VMM must not assume the slot was installed. A concurrent host unmap or slot update can invalidate translations; KVM's MMU-notifier and synchronization paths coordinate with memory management. A successful memory-region ioctl is not permission to free the backing mapping while a running vCPU can still use it.

Four address spaces, two translations#

A guest process uses a guest virtual address (GVA). Its guest page tables translate that to a guest physical address (GPA), according to the guest architecture and current state. KVM's second-stage translation—EPT on supported Intel VMX systems, NPT on supported AMD SVM systems, or stage-two translation on arm64—maps guest physical addresses toward host physical backing under hardware and KVM control. Shadow paging is an alternative technique on supported configurations where KVM constructs mappings representing the combined translation. Actual translation, page sizes and fault paths depend on hardware and configuration.

Use this purely illustrative mapping: guest page tables map GVA 0x400123 to GPA 0x20123; the VMM's memslot associates the GPA page beginning 0x20000 with a region in its host userspace address space beginning at HVA 0x7f000000; host memory management currently backs that HVA page with a host physical page beginning 0x900000. The byte offset 0x123 gives illustrative HVA 0x7f000123 and HPA 0x900123. These numbers are invented, not measured. The HVA is a VMM process address used to describe backing; the guest does not execute using it as its GPA, and the HPA can change with host memory management. KVM's second-stage mappings must be invalidated or updated when backing changes, and TLB invalidation ensures stale cached translations are not used under the relevant MMU protocol.

A guest page fault and a second-stage fault have different causes. The first can reflect guest page-table permissions or missing mappings; the second can reflect unavailable or protected backing, dirty logging, MMIO or another KVM condition. A fault is not automatically delivered to the VMM as an MMIO exit, and an exit reason should not be guessed solely from a guest address. Debugging requires guest page tables, slot mapping, host backing, permissions and the actual exit or fault evidence.

Entry, exit, interrupts and scheduling#

On x86, VMX and SVM expose different hardware mechanisms for entering a guest and reporting exits; KVM abstracts common UAPI while implementing architecture-specific details. On arm64, EL2 and stage-two mechanisms have their own contracts and configuration. An exit from guest hardware to KVM can be handled entirely in the kernel or can require a return to the VMM. In-kernel interrupt-controller and timer support, where enabled and supported, can handle some events without userspace device emulation. Other I/O, MMIO, unsupported instructions, faults or debug events can require different paths.

A vCPU can be kicked to force it to leave guest execution for work such as interrupt delivery or a state change. The kvm_run interface includes immediate_exit where supported, and signals can interrupt a running ioctl; the VMM must handle the documented return and synchronization semantics. A kick, a hardware exit and KVM_EXIT_IO are not synonyms. Timers, interrupt injection, host thread scheduling and guest interrupt masking can all affect observed latency. A userspace exit counter alone misses in-kernel exits and cannot explain all scheduling delays.

Device assignment, VFIO and IOMMU isolation add another trust boundary and require device group, interrupt, DMA and reset analysis. Confidential VM features can protect selected guest state or memory from some host access under supported hardware and a specific threat model, but do not make every device, host kernel, firmware component or shared page trustworthy. Support is version- and configuration-dependent. This lesson does not assign host devices or claim that a confidential VM is available.

VM ownership and an exit/re-entry timeline#

A VMM normally creates the VM and its userspace memory mappings in a consistent host address-space context. KVM records and checks relationships involving the creating process's memory management context, and some ioctls cannot be treated as portable operations on an fd handed to an unrelated process with a different mm. A VM fd's existence also does not pin every arbitrary userspace mapping forever. The VMM must coordinate vCPU threads, slot updates and backing-memory lifetime; KVM's SRCU and MMU-notifier mechanisms address kernel-side readers and invalidation, not a license for userspace to race arbitrary unmapping against its own protocol.

Consider a hypothetical single x86 vCPU executing an output instruction to a port that requires userspace emulation. Hardware leaves guest mode; KVM examines the exit. If it cannot complete the operation internally, KVM_RUN returns with an I/O exit and data describing the operation. The VMM validates the exit fields and handles the relevant emulated device, then calls KVM_RUN again according to the UAPI's completion rules. A signal arriving before re-entry can alter that sequence, and an invalid exit payload or device-model failure is an error for the VMM to handle, not evidence that the guest instruction completed correctly.

Now suppose the guest executes a halt instruction. Depending on in-kernel handling and configuration, the vCPU may block in a kernel-managed path or return a halt exit to userspace. A pending interrupt or timer can later make progress possible; a host scheduler delay is separate from guest interrupt state. With multiple vCPUs, another vCPU can continue while this one is halted. Testing a VMM therefore requires explicit synchronization, guest and host timestamps, and a record of whether the exit was visible in userspace. None of those runtime events is asserted by the read-only API query in this lesson.

Practice#

Read-only environment and API query

On an authorized host or disposable guest, inspect availability without changing permissions or creating a VM:

uname -r
if [ -e /dev/kvm ]; then ls -l /dev/kvm; else echo '/dev/kvm absent'; fi
if [ -r /proc/modules ]; then awk '$1 ~ /^kvm/ {print $1, $2, $3}' /proc/modules; else echo 'module list unavailable'; fi
if [ -r /proc/cpuinfo ]; then awk '/^(flags|Features)[[:space:]]*:/ {print; exit}' /proc/cpuinfo; fi

A CPU flag, loaded module or device node is only a clue. Permissions, firmware settings, host policy, nested virtualization, architecture and kernel configuration determine whether a caller can use KVM. Absence or permission denial is a legitimate finding; do not use sudo, change device ownership or bypass a container's device policy.

If Python 3 is available and you are authorized to open the device, this optional query opens /dev/kvm read-only and issues only KVM_GET_API_VERSION; it does not create a VM or map memory:

python3 - <<'PY'
import fcntl, os
try:
    fd = os.open('/dev/kvm', os.O_RDONLY | os.O_CLOEXEC)
except OSError as e:
    print('KVM device unavailable or denied:', e.errno)
else:
    try:
        print('KVM API version:', fcntl.ioctl(fd, 0xAE00, 0))
    except OSError as e:
        print('KVM API query failed:', e.errno)
    finally:
        os.close(fd)
PY

A successful supported query returns 12 in this UAPI. ENOENT, EACCES, EPERM or an ioctl error means the query did not establish the API result; it does not identify a single cause. The ioctl number shown is for the Linux KVM UAPI on supported architectures; use the matching headers when writing portable software. Record the actual kernel, architecture and query result, not a guessed guest capability.

Source-only alternative: calculate the given GVA/GPA/HVA/HPA mapping and identify which translation each address belongs to. Add the unaligned 0x1003 slot failure and state that an aligned address still needs the remaining validation. For an exit table, distinguish a userspace-visible I/O exit, a halt exit and an exit handled internally by KVM. Deliver these source-derived tables or the actual read-only query result, with no claim that a VM ran.

Knowledge check#

1. Why does KVM API version 12 not prove that a particular virtualization feature is available?
2. What is the difference between a guest physical address and the VMM's host virtual address?
3. What does the memory-slot code do with an unaligned guest physical base?
4. Why can a host mapping change require KVM translation invalidation?
5. Does every hardware VM exit become a KVM_RUN exit visible to QEMU?

Answers#

1. The version identifies the base UAPI. Individual extensions, permissions, architecture support and the ability to create and run a VM require separate checks.
2. GPA is the guest's physical-address view; HVA identifies backing in the VMM process. A slot relates them, and host physical backing is another, potentially changing, address.
3. The quoted validation returns -EINVAL before the slot is accepted. Passing alignment alone does not satisfy the rest of the checks.
4. Cached second-stage or shadow translations can refer to backing that is no longer valid. KVM coordinates invalidation and reader lifetimes with host memory management and TLB protocols.
5. No. KVM handles many exits in the kernel; only some conditions return to userspace with a reason in kvm_run.

Checkpoint#

Submit the four-address mapping, invalid-slot branch and three-exit comparison. Ready means HVA and GPA are distinct, userspace-visible and in-kernel exits are not conflated, and the API query is not treated as a successful guest boot. Retry the source validation and mapping arithmetic if any offset or lifetime is unexplained. This is not certification.

Sources#

• KVM common implementation v6.12 (device ioctl and memory-region validation; line ranges may shift in later revisions)
• KVM x86 implementation v6.12
• KVM UAPI v6.12 (KVM_API_VERSION, struct kvm_run, exit payloads)
• KVM API documentation, v6.12 (fd/ioctl classes and ownership contract; KVM_RUN, memory regions and capabilities)