QEMU, libvirt, and userspace virtualization

Before you start#

Review KVM internals, virtio and VFIO, userspace process boundaries, and OCI containers. QEMU v9.1.0 is pinned to fd1952d814da738ed107e05583b3e02ac11e88ff; libvirt v10.6.0 is pinned separately to a33190e7177ad39d7c5d8b498e4fb44b0635e0d9. These source versions were not tested as a pair. The kernel laboratory from chapter 08 has not been booted here.

Learning goals#

Separate QEMU's virtual machine monitor from libvirt management and kernel KVM, follow QMP request validation and a libvirt startup error, and reason about process, memory, descriptors and asynchronous monitor events without starting a VM.

Source walkthrough#

QEMU is a userspace process that can model a machine, devices and guest memory. With KVM acceleration, host kernel KVM executes supported guest CPU operations while QEMU handles selected exits and device models. Without KVM, QEMU can use its TCG dynamic translation path where supported. These are different execution arrangements with different host and guest architecture constraints; neither is guaranteed available for every binary or configuration. The earlier KVM chapter covers kernel objects and ioctls, while this chapter follows the userspace side.

The first exact excerpt is from QEMU 9.1.0 qapi/qmp-dispatch.c, qmp_dispatch, licensed LGPL-2.1-or-later according to the file notice. The surrounding object conversion and later command lookup are omitted:

    if (!qmp_dispatch_check_obj(dict, allow_oob, &err)) {
        goto out;
    }

The dispatcher validates the request object before selecting and executing a command. On failure it follows the common output path with an error rather than treating malformed input as a valid command. A QMP request is a JSON object with protocol-defined members; a JSON string that merely parses is not necessarily a valid QMP command. The dispatcher also handles command lookup, enabled state, policy and optional out-of-band behavior in other branches. An identifier, when present, helps correlate a response with a request; it is not a process reference or permission grant.

The second exact excerpt is from libvirt 10.6.0 src/qemu/qemu_process.c, qemuProcessStart, also LGPL-2.1-or-later. The surrounding preparation and later launch are omitted:

    if (qemuProcessPrepareDomain(driver, vm, flags) < 0)
        goto stop;

    if (qemuProcessPrepareHost(driver, vm, flags) < 0)
        goto stop;

The first preparation stage is checked before host preparation begins. If either returns an error, control reaches the stop path. There, libvirt chooses stop flags, clears monitor log handling if a monitor exists, calls qemuProcessStop with a failed shutoff reason, and then enters common cleanup. This does not mean all possible preparation errors have identical external effects or that arbitrary host modifications can be undone. The startup function has initialization, launch, state refresh and finish stages, each with its own failure point and acquired resources.

QEMU's process and memory model#

A QEMU machine contains configured virtual devices and address spaces. A guest physical address identifies a location in the guest's physical address space; it is not inherently a host virtual or host physical address. QEMU can back guest RAM with host memory and, for KVM, register memory slots that relate guest physical ranges to host userspace addresses. KVM and host memory management handle the appropriate translations and invalidation. The guest page tables can translate a guest virtual address to a guest physical address, while the host and hardware provide additional translations. A numerical match between two addresses does not make their meanings interchangeable.

A vCPU is represented in userspace by state and typically a host thread in a KVM configuration. That thread invokes KVM operations and is scheduled by the host kernel. Some guest operations run without an exit to QEMU; others cause exits that userspace or in-kernel device support must handle. A vCPU thread can be delayed by host scheduling or blocked on I/O. TCG translates guest instructions in userspace rather than relying on KVM's hardware virtualization path. The actual threading and acceleration options depend on QEMU's version and configuration; one generic diagram cannot specify every machine type.

QEMU models device registers, interrupts and backends. A guest virtio driver communicates with a virtual device using negotiated features and virtqueues. QEMU or an associated backend consumes descriptors according to the selected configuration, performs host-side work and reports completion. The guest descriptor, QEMU memory mapping, host fd and backend request have distinct owners. A backend can fail or disconnect while a request is pending, so completion and cleanup must not assume that all referenced memory or file descriptors remain usable indefinitely.

QEMU uses event loops and asynchronous I/O infrastructure. An AioContext groups event-loop resources and synchronization for relevant operations; IOThreads can host work outside the main loop when configured. The big QEMU lock, often called the BQL, serializes selected global operations, but it is not a guarantee that every device, backend or callback is protected by one lock. A callback can race with teardown unless its own lifetime and synchronization rules prevent it. A worker thread, vCPU thread and monitor connection have different execution contexts and must not be treated as one synchronous call chain.

libvirt as management plane#

libvirt provides an API for describing and managing domains, including QEMU-based virtual machines. A domain XML definition describes requested devices, resources and policy; the active process, monitor connection, logs and persistent definition are separate objects. A client uses a URI and connection context, and system and session deployments can have different authority and available resources. Some installations use a monolithic daemon while others use modular services such as virtqemud; virtlogd can handle logging. Which services exist and how they are activated depends on the installation.

libvirt validates and prepares configuration, coordinates domain jobs, constructs a QEMU command and environment, starts the process, connects to its monitor and updates domain state. A job or domain lock can serialize operations that would conflict, but a client connection can still fail or disappear. A domain marked as running in a management view is a state observation from libvirt, not proof that the guest application is healthy. A process may be alive while a virtual device or guest service is unavailable.

In the pinned qemuProcessStart source, the domain object has private QEMU state including monitor information. The function validates flags, initializes and prepares the domain and host, launches QEMU, refreshes state and finishes startup. On selected failures it enters stop, clears monitor log handling when applicable, calls qemuProcessStop, and then frees incoming migration state in cleanup. Error paths differ depending on how far startup progressed. Security labeling, storage access and external services can fail independently; a failed start should retain the precise error and cleanup outcome.

The stages below follow the order in which the pinned source calls them. The failed-start exit is the path this chapter follows, and no QEMU process was launched here.

QMP and asynchronous messages#

QEMU Machine Protocol, or QMP, is a JSON-based control protocol. A connection begins with a greeting; a client negotiates capabilities before issuing ordinary commands. A request can carry an id that is returned with the corresponding response. Events can arrive independently between a request and its response. A robust client parses complete JSON objects, handles asynchronous events and matches responses by ID instead of assuming that the next object is the answer to the most recent command. A monitor disconnect or EOF can occur while requests remain outstanding.

In qapi/qmp-dispatch.c, QEMU 9.1.0, LGPL-2.1-or-later per its notice (the dispatch code after this helper is omitted):

static QDict *qmp_dispatch_check_obj(QDict *dict, bool allow_oob,
                                     Error **errp)
{
    const char *exec_key = NULL;
    const QDictEntry *ent;
    const char *arg_name;
    const QObject *arg_obj;

    for (ent = qdict_first(dict); ent;
         ent = qdict_next(dict, ent)) {
        arg_name = qdict_entry_key(ent);
        arg_obj = qdict_entry_value(ent);

        if (!strcmp(arg_name, "execute")
            || (!strcmp(arg_name, "exec-oob") && allow_oob)) {
            if (qobject_type(arg_obj) != QTYPE_QSTRING) {
                error_setg(errp, "QMP input member '%s' must be a string",
                           arg_name);
                return NULL;
            }

The helper checks member names before any command lookup: execute must be a string, and a second key of that kind is rejected as a clash. The excerpt does not show command lookup, the arguments checks or out-of-band handling, so a JSON object that passes this step still has to name a valid, enabled command.

Consider these synthetic messages: first a greeting object, then a client capabilities request with ID c1; the server sends an asynchronous event, then a success response with ID c1. The event is not a response to c1 merely because it arrived first. Next, the client sends a query with ID q2; a response carrying q2 can be matched even if another event intervenes. If the socket closes before the response, the client must fail or otherwise resolve the pending request according to its policy. It must not invent a successful result because it previously received a greeting.

The timeline below is a teaching model of that message order. The greeting is not a return, and the event is not matched to c1.

QMP transports use sockets or other configured channels. The monitor connection has a descriptor and protocol state; the QEMU process has its own lifetime. Closing a client socket does not necessarily kill QEMU, and a QEMU exit invalidates the monitor connection even if a stale fd number is later reused. File descriptors passed to a process or over an appropriate channel carry separate references. A client must retain request state until a response, cancellation or connection failure resolves it, and release buffers and references exactly once.

Worked failed startup#

Assume a synthetic domain has a prepared XML definition and libvirt begins a start job. Domain initialization succeeds; domain preparation succeeds; host preparation fails because an authored hypothetical resource is unavailable. The quoted qemuProcessPrepareHost branch goes to stop. The code applies stop flags and invokes the QEMU stop path with a failed reason, then common cleanup returns an error. In this scenario no claim is made that a QEMU process reached guest execution. A management client should report the failed stage and result, not a successful VM start merely because XML parsing succeeded.

In src/qemu/qemu_process.c, libvirt 10.6.0, LGPL-2.1-or-later per its notice (the earlier and launch stages are omitted; this is the stop label that the preparation failures jump to):

 stop:
    stopFlags = 0;
    if (!relabel)
        stopFlags |= VIR_QEMU_PROCESS_STOP_NO_RELABEL;
    if (migrateFrom)
        stopFlags |= VIR_QEMU_PROCESS_STOP_MIGRATED;
    if (priv->mon)
        qemuMonitorSetDomainLog(priv->mon, NULL, NULL, NULL);
    qemuProcessStop(driver, vm, VIR_DOMAIN_SHUTOFF_FAILED, asyncJob, stopFlags);
    goto cleanup;
}

The label selects stop flags, clears the monitor's domain log handler when a monitor exists, and calls qemuProcessStop with a failed shutoff reason before jumping to cleanup. It does not undo changes that a host step made outside libvirt's control, as the scenario above already notes.

Change the case so QEMU launches and sends a QMP greeting, then exits before responding to capabilities negotiation. The manager receives EOF or another transport error and must resolve pending monitor work and perform appropriate process cleanup. A valid greeting proves only that a monitor endpoint spoke at that moment; it does not prove a guest booted or a disk was usable. If the host resource remains referenced elsewhere or was modified outside libvirt's control, a stop call cannot be assumed to restore every external side effect.

The domain job serializes relevant management operations, but external actors and host failures remain possible. A concurrent stop request, monitor event or child exit can change the state while a client is waiting. Keep a reference to the domain object for the duration required by the API and avoid accessing monitor data after teardown. Record whether the observed status came from the management client, libvirt daemon, QEMU process, KVM or the guest. A single timestamp or PID is not a complete object-lifetime proof.

Storage and recovery boundaries#

A virtual disk can use raw or a structured format such as qcow2. A disk image can have a backing chain in which reads depend on another file; moving or deleting a backing file can make the apparent disk unusable. A snapshot captures some defined state or references at a point in time, depending on the mechanism. It is not automatically an independent backup, a consistent application checkpoint or protection from loss of the underlying storage. Recovery requires a tested restore plan and understanding of all dependencies.

A VM can interact with host files, networks, devices and services through configured virtual devices, shared folders or passthrough. Guest isolation does not make those exposed host resources harmless. Device assignment and IOMMU protection have configuration and hardware requirements; no device is assigned in this lesson. A guest can also be unsafe to its own data even when the host boundary works as intended. The historical Linux v6.12 baseline is educational and is not a current security recommendation.

A real performance or failure investigation records QEMU binary and machine type, acceleration mode, guest and host kernel, libvirt version, domain definition and relevant backing storage. A matching source revision helps interpret a trace, but a source excerpt cannot prove which binary or configuration was running. Avoid collecting another user's domain XML, disk paths or monitor traffic. This source-only exercise uses authored data and contacts no libvirt daemon.

Monitor lifetime and recovery evidence#

Suppose a management client has two outstanding QMP commands when the monitor connection fails. The client owns two pending request records, each identified by its own ID. It must complete both with an appropriate connection failure or cancellation according to its implementation; freeing only the most recent record leaks or strands the other waiter. An asynchronous event received before the failure does not complete either request unless the protocol and client explicitly associate it with one. When a new monitor connection is established, old IDs and callbacks should not be silently attached to the new connection's requests. A generation or ownership rule can prevent a late callback from updating a newly created domain object by mistake.

The failure can also race with the QEMU child exit. A process watcher and a monitor reader may both report symptoms of the same failure, but cleanup still needs a single coordinated ownership decision for each resource. A useful test plan would separately simulate an invalid QMP request, EOF before a response and a normal response followed by an event, using only an explicitly disposable environment if a real QEMU process were later authorized. The source-only message model below tests none of those process races; it shows why the client must first distinguish responses from events.

Practice#

This own-process model matches responses by ID while preserving an interleaved event. It does not run QEMU, connect to libvirt or open a monitor socket:

python3 - <<'PY'
messages = [
    {'event': 'SYNTHETIC_EVENT'},
    {'return': {}, 'id': 'c1'},
    {'return': {'version': 'example'}, 'id': 'q2'},
]
pending = {'c1', 'q2'}
for message in messages:
    if 'id' in message and message['id'] in pending:
        pending.remove(message['id'])
        print('matched response:', message['id'])
    else:
        print('independent event:', message.get('event'))
print('pending:', sorted(pending))
PY

The event is printed separately and both responses match their IDs. The program is an authored model, not a QMP implementation or protocol capture. As a counterexample, remove the q2 response: the pending set remains nonempty, and a real client would need to handle EOF or timeout. If Python is unavailable, perform the same matching on paper.

For the source-only alternative, write the supplied greeting/event/response timeline and the libvirt host-preparation failure as a state table. Include the owner of the monitor fd, the pending request and the process, then identify which states are unknown after a greeting and which cleanup path the cited source chooses on host-preparation failure. Do not connect to libvirt or launch a VM.

Knowledge check#

1. Why is a guest physical address not the same object as a QEMU host virtual address?
2. What does the QMP request validation branch do with an invalid request?
3. Why must a client match QMP responses by ID rather than by arrival order?
4. What happens in the cited libvirt branch when host preparation fails?
5. Why is a snapshot not automatically a backup?

Answers#

1. They belong to different address spaces. QEMU and KVM can establish mappings between them, but the addresses have different owners and translation rules.
2. The dispatcher follows its common error output path before ordinary command execution. Merely parsing JSON does not make the request a valid QMP command.
3. Asynchronous events and other responses can arrive between a request and its response. IDs correlate the appropriate response with pending request state.
4. It jumps to the stop path, which selects flags, handles an existing monitor log association and invokes failed-state process cleanup before common cleanup. The exact acquired resources depend on the stage reached.
5. A snapshot can depend on the same storage or backing chain and may not contain a consistent or independent copy. Recovery requires a separate verified restore strategy.

Checkpoint#

Submit the QMP message ledger, guest/host address map and failed-start cleanup timeline. Pass when the event is not mistaken for a response, QEMU is distinguished from KVM and libvirt, and a greeting is not called a guest boot. Revisit the cited branches and retry if a snapshot is treated as an independent backup. This is not certification.

Sources#

• QEMU 9.1.0 QMP dispatcher
• QEMU 9.1.0 system startup
• libvirt 10.6.0 QEMU process lifecycle
• Linux v6.12 KVM core