V4L2, media graphs, cameras, and video buffers

Before you start#

Review DMA and I/O completion and device ownership. This lesson reads Linux v6.12; a camera, media graph, and video device depend on hardware, firmware and configuration. The state transitions are a source-based example, not a recorded camera session.

Learning goals#

Describe how a pipeline and a queue are different, trace a buffer through driver completion and userspace dequeue, and explain why an error buffer still has to be returned to its owner.

Source walkthrough#

Video4Linux2 (V4L2) offers userspace video interfaces. A camera may comprise a sensor, receiver, image signal processor and capture engine. The media-controller graph describes entities, pads and links where the driver exposes one; not every camera exposes the same topology. The graph describes connectivity, while format negotiation and buffer queues establish a particular stream. The size and layout of a captured frame depend on negotiated pixel format, dimensions, plane layout, stride and device constraints, not simply width times height.

The videobuf2 framework (vb2) manages buffer queues used by many video drivers. In videobuf2-core.c, each vb2_buffer has a state and a vb2_queue owns, among other things, a done list, a lock, a wait queue and a count of buffers owned by the driver. Queuing is not copying a finished picture: it transfers a buffer for a future operation. When a driver finishes a buffer, it calls vb2_buffer_done. This exact excerpt is from v6.12; the source attributes copyright to Samsung Electronics and contains a GPL notice. Earlier and later lines are omitted:

	if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
		return;

	if (WARN_ON(state != VB2_BUF_STATE_DONE &&
		    state != VB2_BUF_STATE_ERROR &&
		    state != VB2_BUF_STATE_QUEUED))
		state = VB2_BUF_STATE_ERROR;

The first condition requires the buffer being returned to be ACTIVE: the driver must not report completion of an arbitrary buffer it does not currently own. The next condition restricts the reported completion state. An invalid requested state emits a warning and is converted to ERROR rather than treated as successful data. A later branch calls the memory-finish operation except for the special QUEUED case, acquires done_lock with interrupts saved, adds DONE or ERROR buffers to done_list, records state and decrements owned_by_drv_count. It then wakes waiters for the normal completion cases. The QUEUED case is special and does not put the buffer on the done list at that point; do not treat it as a completed frame.

Walk a buffer through a stream#

Consider buffers 0 and 1 in a hypothetical capture queue. Userspace requests buffers, maps them according to the selected V4L2 memory model, and queues both. When streaming begins and the driver accepts buffer 0, it becomes ACTIVE. While ACTIVE, userspace must not assume it owns stable frame contents. The driver finishes capture and calls vb2_buffer_done(vb, VB2_BUF_STATE_DONE). vb2 moves it to the done list and wakes a waiting consumer. vb2_core_dqbuf obtains a completed buffer, accepts DONE or ERROR, calls the optional finish callback, fills the userspace description, and removes the queued entry. Userspace may process the reported payload and then queue that buffer again.

In include/media/videobuf2-core.h, v6.12, Samsung Electronics copyright with a GPL notice in the file header (surrounding comments omitted), the buffer states are:

enum vb2_buffer_state {
	VB2_BUF_STATE_DEQUEUED,
	VB2_BUF_STATE_IN_REQUEST,
	VB2_BUF_STATE_PREPARING,
	VB2_BUF_STATE_QUEUED,
	VB2_BUF_STATE_ACTIVE,
	VB2_BUF_STATE_DONE,
	VB2_BUF_STATE_ERROR,
};

These seven states are the vocabulary for the walk above. The enum does not show which function performs each transition or which transitions are legal; the functions that assign vb->state do that. DONE and ERROR are the two states returned to userspace through DQBUF, while IN_REQUEST, PREPARING and QUEUED describe stages before the driver owns the buffer.

Now suppose the device detects a corrupted frame. Returning ERROR still completes the ownership transfer: it tells userspace that the buffer was returned but its payload needs error handling. Silently dropping an ACTIVE buffer would leave ownership and waiters unresolved. If q->error is set, vb2_core_qbuf returns -EIO; request-mode constraints can produce -EBADR or -EBUSY, and a request buffer not in DEQUEUED state can produce -EINVAL. These are different failures from “camera image was dark.” The v6.12 queue code gives the exact checks.

In drivers/media/common/videobuf2/videobuf2-core.c, v6.12, Samsung Electronics copyright with a GPL notice (the function header and the normal enqueue path omitted):

	if (q->error) {
		dprintk(q, 1, "fatal error occurred on queue\n");
		return -EIO;
	}

	if (!req && vb->state != VB2_BUF_STATE_IN_REQUEST &&
	    q->requires_requests) {
		dprintk(q, 1, "qbuf requires a request\n");
		return -EBADR;
	}

Both checks return before any state change, so a rejected QBUF leaves the buffer's ownership as it was. The excerpt does not show the other request-related checks or the normal enqueue, so an error here says nothing about the camera's hardware state.

Stream stop and unplug must return or otherwise account for outstanding buffers through the driver/framework contract, stop DMA before backing memory is released, and wake relevant waiters. Exact callback ordering is driver specific; disconnect can race with applications and hardware. A dma-buf-backed video buffer introduces exporter/importer lifetime and synchronization questions from DRM and dma-buf, but V4L2 does not require every device to use that memory model.

Graph configuration and device nodes#

A media-controller graph is a description of a hardware or software pipeline, not an ordered list of user processes. An entity represents a functional block, such as a sensor, receiver, processing unit or video interface. A pad is an endpoint on an entity; source and sink pads describe the direction in which media data flows. A link connects pads. Some links are immutable, while configurable links can be enabled or disabled subject to the device's rules. A graph may branch, and an enabled link does not by itself prove that both endpoints use compatible formats or that the stream has started.

A V4L2 subdevice represents a component with operations such as format, selection, routing or control handling; where exposed to userspace it can have a subdevice node. A video node, commonly /dev/videoN, is an endpoint for streaming or other video operations. One node does not necessarily correspond to one physical camera: a single device can expose several functions, and several entities may contribute to one captured image. media-ctl can describe graphs where the media-controller API is present, while v4l2-ctl can query video node capabilities. They are separate userspace tools, and neither list output establishes that the connected pipeline is configured for the application.

Formats must agree across relevant links. A sensor might emit a raw Bayer format, a receiver pass it through, and an image processor produce a different pixel format for the capture node. Width, height, field order, color encoding, bytes per line and plane sizes all affect interpretation. A requested format can be adjusted or rejected by a driver; userspace must inspect the negotiated result. A raw frame with padding at the end of each line cannot safely be parsed by assuming width * bytes_per_pixel is always the stride. This example is conceptual: actual entities and conversion capabilities are hardware-specific.

Queue setup and ownership across ioctls#

A common streaming sequence uses VIDIOC_REQBUFS to request queue buffers, VIDIOC_QUERYBUF to obtain information needed for a mapping where applicable, VIDIOC_QBUF to enqueue buffers, VIDIOC_STREAMON to start, and VIDIOC_DQBUF to recover completed buffers. After use, VIDIOC_STREAMOFF stops the stream and close releases file-related resources. Depending on memory type and driver, some details differ, including whether QUERYBUF information is used for mapping. A request can return fewer buffers than requested or fail, so an application must use the returned count and check errors rather than assume its original count was granted.

With V4L2 MMAP, the driver/framework allocates or arranges buffers and the application maps the offsets reported for them. Mapping makes storage accessible in the application, but does not give the application permission to modify a buffer while it is queued for capture. With DMABUF, userspace supplies dma-buf file descriptors for import, and attachments, mapping and synchronization follow the participating drivers' contracts. These are memory-sharing choices, not two names for the same address. A descriptor is an fd, not an MMAP offset; a buffer index is a queue slot, not an fd. Multi-planar formats can carry multiple planes and per-plane lengths, offsets and payload sizes; they must be handled according to the negotiated UAPI.

A typical capture example with two MMAP buffers starts with indices 0 and 1 allocated and mapped. The application queues both; vb2 passes them to the driver when the conditions for streaming are met. The driver uses its hardware and completion path to report an ACTIVE buffer as DONE or ERROR. The application dequeues a completed index, reads its reported bytes-used and error flags, processes valid data, and requeues it. The two buffers let capture and processing overlap, but the application still cannot own and modify one while the driver is using it. Too few buffers or a slow consumer can starve capture. The exact effect of starvation is device and driver dependent, not automatically a guaranteed fixed frame drop pattern.

The V4L2 ioctl layer dispatches and checks arguments through v4l2-ioctl.c; vb2 handles queue operations for drivers that use it. vb2_core_qbuf checks queue error state and request-mode constraints. When dequeuing in nonblocking mode, an empty done queue can produce -EAGAIN instead of waiting for a frame; a blocking caller may wait until a completion, error, signal or stream state change. Poll readiness indicates that a corresponding operation may make progress under the interface rules; it does not eliminate the need to handle a subsequent error or race.

Failed start, completion and disconnect#

There is a particularly useful error branch in vb2_start_streaming: previously queued buffers are passed to the driver, then the driver's start_streaming callback is called. This exact excerpt from v6.12 videobuf2-core.c, copyright Samsung Electronics with GPL notice in the source, omits surrounding lines:

	q->start_streaming_called = 1;
	ret = call_qop(q, start_streaming, q,
		       atomic_read(&q->owned_by_drv_count));
	if (!ret)
		return 0;

	q->start_streaming_called = 0;

The core marks that start was called, passes the current driver-owned count to the callback, and returns success only if the callback succeeds. On failure it clears the flag and continues through cleanup. The driver's contract requires returning buffers correctly when its start callback fails. The code warns if owned_by_drv_count remains nonzero and forcefully returns still-ACTIVE buffers with VB2_BUF_STATE_QUEUED; it also warns if the done list is nonempty in this failed-start situation. This fallback is not a license for a driver to leak buffers or report erroneous successful frames. A negative callback result remains a failed start.

Work a concrete branch: buffers 0 and 1 were queued; the framework handed both to the driver; hardware setup then fails, for example because a required pipeline component rejects the selected configuration. If the driver returns both to vb2 as QUEUED and returns an error, ownership returns without inventing completed images. If it forgets one ACTIVE buffer, the warning and fallback code expose and attempt to repair the broken ownership count. Userspace must handle the failed STREAMON rather than attempt to treat that buffer as a successfully captured frame. Retrying without correcting the underlying format, resource or device error may repeat the same failure.

STREAMOFF and disconnect introduce another boundary. A driver must stop or synchronize hardware access, return buffers according to its contract and unblock users as appropriate. A USB camera can disappear while a process still has mappings or an open fd; holding an fd is not proof that the device remains physically present. The file, queue, mapping and hardware lifetimes can differ. The correct error and wakeup details depend on the driver and vb2 operation, so inspect the concrete teardown rather than imposing one universal disconnect sequence. The reason for ordering is nevertheless clear: freeing memory while DMA can still write into it risks use-after-free or data corruption.

Read-only Linux observation lab#

On your own authorized host or a disposable guest, list device nodes and query only existing tools; do not start a stream, change controls, request buffers or reconfigure links:

uname -r
ls -l /dev/video* /dev/media* 2>/dev/null || echo 'No video or media nodes visible'
if command -v v4l2-ctl >/dev/null 2>&1; then v4l2-ctl --list-devices; else echo 'v4l2-ctl unavailable'; fi
if command -v media-ctl >/dev/null 2>&1 && [ -r /dev/media0 ]; then media-ctl -d /dev/media0 -p; else echo 'media graph tool or readable /dev/media0 unavailable'; fi

If v4l2-ctl and a readable node exist, v4l2-ctl -d /dev/video0 --all queries current capabilities and controls; substitute an actual node identified in the listing. It opens the device but does not intentionally start capture. Some drivers may wake hardware on open. Permission denied, no nodes, absent tools, or a media graph that is not exposed are valid findings. Do not change permissions or use sudo to force a result. Record the kernel and v4l2-ctl --version or package version: these utilities are separate from the kernel.

A printed entity/link graph shows the driver's reported topology, not a successful frame. An --all result reports supported and current information from that node, not a guarantee that every graph combination works. Source-only alternative: use the failed-start example above and make a three-column table for start_streaming_called, owned count, and result at callback entry, callback failure and cleanup. Distinguish required driver cleanup from the core's warning fallback. Deliver the graph or source table and state what evidence would still be needed to claim a real camera captured a valid frame.

Practice#

Source-backed table exercise; no camera access. Run this standalone Python model of the two valid final states. It intentionally does not implement the entire vb2 API:

python3 - <<'PY'
for result in ('DONE', 'ERROR'):
    state, done, owned = 'ACTIVE', [], 1
    if state != 'ACTIVE': raise ValueError('not owned by driver')
    state = result
    done.append((0, state)); owned -= 1
    item = done.pop(0)
    print('dequeued', item, 'driver-owned:', owned)
PY

Expected output reports buffer 0 once as DONE and once as ERROR, with zero driver-owned buffers in each independent scenario. For a counterexample, change the initial state to DEQUEUED; the model raises an error rather than creating a completion. Produce a table with owner, state and permitted next action for ACTIVE, DONE, ERROR and dequeued. Note that this program does not model locks, the special QUEUED case, actual memory mappings or device failures. It creates no files and makes no device calls.

Knowledge check#

1. Why does an ERROR completion still matter for buffer lifetime?
2. What does done_lock protect in the illustrated source path?
3. Why does a media graph alone not determine the byte layout of a captured frame?
4. What happens when a driver start callback fails while it still owns ACTIVE buffers?

Answers#

1. It transfers a buffer back through the completion path even though the image must not be interpreted as a successful frame. Failing to return ownership can strand the queue and waiters.
2. The code takes it around updates to the done list, buffer state and related queue bookkeeping, using an interrupt-safe lock form. The precise locking scope is the code, not a blanket claim about every vb2 field.
3. The graph describes connected entities and pads; negotiated formats, plane layouts, strides and driver constraints determine the actual buffer representation.
4. The driver is required to return them properly. The core clears its start flag, warns if ownership remains, and has a fallback that returns ACTIVE buffers as QUEUED; it still propagates the start error and does not manufacture successful frames.

Checkpoint#

Submit the four-state ownership table and identify how ERROR differs from the special QUEUED completion. Ready means you can trace one normal and one error completion without letting userspace and hardware own the same ACTIVE buffer. Revisit vb2_buffer_done and vb2_core_dqbuf if the table cannot account for the buffer. This is practice, not certification.

Sources#

• vb2 core v6.12
• V4L2 device registration v6.12
• V4L2 userspace API
• Media controller model