Remote processors, rpmsg, mailbox, and FPGA

Before you start#

Review device lifecycle, DMA and IOMMUs, and references. Linux v6.12 is the source baseline. A remote processor, firmware image, resource table and message transport are platform-specific; no such device is required or operated here.

Learning goals#

Trace remoteproc boot and shutdown power references, distinguish them from object references, explain firmware failure cleanup, and separate processor management, messages, mailboxes and FPGA configuration.

Source walkthrough#

An SoC may have a processor running firmware independently of the CPU running Linux. Remoteproc offers lifecycle support for supported remote processors: it can load firmware, configure resources, start or attach to a processor, and manage shutdown or recovery. The remote processor may access shared memory and peripherals, so its trust and memory isolation depend on the platform, firmware and IOMMU configuration. A Linux pointer cannot simply be sent as a universally valid remote address.

In remoteproc_core.c, rproc_boot first locks rproc->lock interruptibly and rejects a deleted processor. This v6.12 GPL-2.0-only excerpt is from its powered-up branch; surrounding lines are omitted:

	if (atomic_inc_return(&rproc->power) > 1) {
		ret = 0;
		goto unlock_mutex;
	}

The increment records an additional power user. If the resulting count exceeds one, an existing user already requires the processor, so this path returns success without repeating boot. The shared mutex serializes the lifecycle branch; the atomic counter is not a replacement for the rest of that locking. If the previous count was zero, the first user continues: a detached processor is attached, otherwise firmware is requested and passed to rproc_fw_boot. The acquired firmware is released after the boot attempt. On failure the downref_rproc path decrements the power count, then unlocks. An interrupted mutex acquisition or a deleted object returns before this increment.

Two users and one failure#

Start with a hypothetical stopped device, power count zero, and a valid handle. User A calls rproc_boot; the count becomes one. If firmware loading and boot succeed, A holds one power use. User B calls it next: the count becomes two, and the quoted branch returns without a second firmware load. When A calls rproc_shutdown, the count becomes one and the processor remains needed. B's matching shutdown decrements to zero and calls rproc_stop; after success, the source cleans resources and disables the IOMMU where applicable. If stop fails, it restores the power count and returns the error. These are source-derived branches, not evidence of an actual remote device.

Now instead let the first firmware request fail. The error path decrements the count from one to zero; there is no successful boot for the caller to balance with shutdown. A power reference is also not the remoteproc object's general lifetime reference: the source documentation explicitly distinguishes shutdown's power decrement from releasing the rproc handle. Do not use a stale handle after object removal simply because the power counter was once positive.

Messaging and hardware boundaries#

rpmsg supplies message channels on supported transports; its endpoints and address/driver matching are separate from remoteproc's lifecycle. A remote processor may use virtio-based rpmsg, but the subsystems are not identical and a platform need not instantiate every mechanism. Shared memory ownership, buffer sizes, message validity, access permissions and remote resets matter. Treat a remote peer's message as input requiring validation; a running remote processor does not automatically share Linux process isolation.

A mailbox controller transports notifications or messages according to its controller-specific contract; a notification can indicate that shared memory needs attention rather than contain the entire payload. FPGA manager interfaces configure programmable logic, which is a different lifecycle from starting firmware on a processor. Resource dependencies and teardown order may connect these devices in a system, but no universal direct call chain follows from their coexistence. See rpmsg core and the remoteproc documentation.

Processor states and firmware resources#

A struct rproc represents a remote processor and carries state, a lifecycle mutex, firmware name and power-use count among other fields. In v6.12 the state distinguishes, among other conditions, an offline or running processor, a crashed processor, a detached or attached externally managed processor, and a deleted object. These are lifecycle states, not numeric guarantees about a particular SoC's power rails. The boot path's RPROC_DETACHED branch attaches to an already running external processor instead of unconditionally loading a new image. A platform driver supplies the operations that actually start, stop, attach or otherwise manage its hardware.

A common firmware format is ELF, but support and exact loading behavior depend on the remoteproc implementation and platform. A resource table can describe resources the firmware expects or announces, such as carveout memory, device memory, trace buffers and virtio devices. A carveout is a reserved or allocated memory region described for remote use; the exact mapping between remote device addresses and host physical memory depends on the platform and any IOMMU. An IOMMU can constrain device-visible mappings where configured, but its presence in a design does not automatically confine every remote bus master. Firmware resource information is input that the loader and platform must validate against their supported contracts.

A firmware-provided virtio device can lead to virtqueues with descriptor rings, buffers and notification mechanisms. A ring is shared protocol state, not a Linux userspace queue and not inherently an rpmsg endpoint. Notification may use a mailbox or another platform mechanism to tell the peer to inspect a queue. The exact resource table entries, ring allocation, address translation and interrupt routing require the specific driver and firmware; the diagram in this lesson is a conceptual route, not a universal DSP boot sequence.

Boot, failure unwind, shutdown and crash#

In rproc_boot, a null handle returns -EINVAL; interruptible mutex acquisition can fail; a deleted processor returns -ENODEV. None of those cases grants a power use. A first power user increments the count, and either attaches a detached processor or requests firmware before rproc_fw_boot. The firmware object returned by request_firmware is released after the boot attempt. If request or boot fails, the downref_rproc label decrements the just-acquired power count before unlocking and returning the error. A later user increments the count but skips redundant initialization. These branches appear in the v6.12 core.

In the same remoteproc_core.c, v6.12, GPL-2.0-only (the enclosing branch and the lines before it omitted; the /* ... */ line marks omitted lines 1953 to 1956, which hold the firmware boot call and its release):

		ret = request_firmware(&firmware_p, rproc->firmware, dev);
		if (ret < 0) {
			dev_err(dev, "request_firmware failed: %d\n", ret);
			goto downref_rproc;
		}
		/* ... */
	}

downref_rproc:
	if (ret)
		atomic_dec(&rproc->power);
unlock_mutex:
	mutex_unlock(&rproc->lock);

A failed request_firmware jumps to downref_rproc, which decrements the power count taken earlier in this call before the mutex is released. A failure in the omitted firmware boot reaches the same ret test, so one label rolls back both failure points.

In the worked two-user example, suppose rproc_stop fails during the last shutdown. The count had reached zero, but the shutdown path increments it again and returns the failure; it must not proceed as though all hardware resources were safely stopped and released. A successful last shutdown continues with resource cleanup, device unprepare, IOMMU disable and cached resource-table cleanup in the source. The handle's object reference and the power count are separate: a caller cannot derive handle validity from a count of one. A successful power reference also does not prove the firmware is trustworthy or that the remote subsystem will remain healthy.

In drivers/remoteproc/remoteproc_core.c, v6.12, GPL-2.0-only (the state check before the decrement and the function header omitted):

	/* if the remote proc is still needed, bail out */
	if (!atomic_dec_and_test(&rproc->power))
		goto out;

	ret = rproc_stop(rproc, false);
	if (ret) {
		atomic_inc(&rproc->power);
		goto out;
	}

A power count still above zero returns without stopping the processor. A failed rproc_stop restores the count, so it again reflects a user who still needs the processor. Resource cleanup, IOMMU disable and the cached-table free happen only after a successful stop and are not in this excerpt.

A crash is an asynchronous event relative to Linux clients. The framework and platform can report a crash and, when configured, schedule recovery work; recovery may stop, clean up and restart the remote processor. Whether recovery is enabled, successful or safe for a client depends on the device and deployment. A client with an outstanding message must not assume that an endpoint or remote state survives a crash and restart. Teardown and recovery must synchronize callbacks and shared resources; a client should identify whether an error means no delivery, an unknown delivery state, or a remote restart before retrying a non-idempotent operation.

rpmsg channels and endpoint lifetime#

The rpmsg bus represents communication services as devices or channels. An rpmsg endpoint associates a local address with a receive callback and private context; outgoing messages use source and destination addresses. The remote address is a protocol address, not necessarily a process ID or a host pointer. Some virtio rpmsg transports use shared virtqueues; other rpmsg implementations can have different transports. A named service or dynamically announced channel is not evidence that any arbitrary firmware implements a particular message schema.

The following exact v6.12 excerpt is from rpmsg_send in rpmsg_core.c; its source carries SPDX GPL-2.0 and copyright Texas Instruments, and surrounding lines are omitted:

	if (WARN_ON(!ept))
		return -EINVAL;
	if (!ept->ops->send)
		return -ENXIO;

	return ept->ops->send(ept, data, len);

The first check rejects a null endpoint; the next rejects an endpoint whose operations do not supply send. Only then does the core invoke the transport's send method and return its result. The API documentation in that file says rpmsg_send can block waiting for a transmit buffer and must be called from process context. A caller cannot use that potentially sleeping operation while holding a spinlock or in hard interrupt context. The timeout and transport behavior must be checked for the selected implementation; the return of zero means the send operation succeeded under the transport contract, not that a remote application completed the requested work.

Creation and removal require matching lifetimes. rpmsg_create_ept uses the transport's endpoint-creation operation; rpmsg_destroy_ept dispatches endpoint destruction if an endpoint and operations are present. In virtio_rpmsg_bus.c, the transport's destroy function delegates to __rpmsg_destroy_ept; that implementation removes the endpoint from lookup and coordinates callback access before releasing the endpoint reference. A driver must keep callback private data alive while a callback can still use it, and respect its transport's teardown synchronization. A remote restart can remove a channel while userspace or another kernel component still wants service; hold the correct references, cancel work and treat subsequent sends as failures rather than using a stale pointer.

For a stateful example, a service creates endpoint E with local address A and remote destination B. A send with a valid callback and available transmit buffer is submitted by the transport; the remote service may later reply to A. If the channel is removed before another send, the client must stop new submissions and destroy E under the transport contract. If the send operation is absent, rpmsg_send returns -ENXIO; if E is null, it returns -EINVAL. Neither error can be repaired by assuming B is still reachable. If a reply never arrives, a successful send alone cannot distinguish a remote crash from application-level processing failure.

Mailboxes, FPGA managers, and trust#

A mailbox controller provides a channel for notifications or messages between processors or hardware blocks. The mailbox payload and blocking or interrupt behavior are controller-specific. A doorbell can merely signal that shared memory contains new work; it is not necessarily the data transport, nor is every rpmsg transport a mailbox. Shared buffer visibility and ownership still require the platform's memory-ordering and DMA rules.

FPGA manager, bridge and region are separate abstractions. A manager loads an image; a bridge can isolate a region from the rest of the system while it changes; a region coordinates a manager and associated bridges. In fpga-region.c, fpga_region_program_fpga gets the region, locks its manager, optionally obtains bridges, disables them, loads the image and then enables bridges. Its error labels release acquired bridge references when appropriate, unlock the manager and put the region. A load failure is not evidence that the old design was automatically restored or that a disabled bridge has been safely re-enabled; recovery depends on hardware and the caller's policy. Never test this sequence by programming a shared host device.

A firmware image or bitstream is executable or hardware-configuring input with its own origin and trust boundary. A signature or verification policy, if configured, answers a specific authenticity question; remoteproc or FPGA registration by itself does not prove code is benign or isolated. An IOMMU mapping, firmware authentication and rpmsg message validation protect different boundaries. Use a threat model that includes the remote processor's access to memory and peripherals, not just the Linux-side endpoint.

Read-only Linux observation lab#

On an authorized host or disposable guest, list existing class and bus entries without changing state or requesting firmware:

uname -r
for base in /sys/class/remoteproc /sys/bus/rpmsg/devices /sys/class/fpga_manager; do
  if [ -d "$base" ]; then printf '\n%s\n' "$base"; ls -l "$base"; else printf '%s unavailable\n' "$base"; fi
done
for d in /sys/class/remoteproc/remoteproc*; do [ -d "$d" ] && { printf '%s ' "$d"; cat "$d/name" "$d/state" 2>/dev/null; }; done

Existing remoteproc names and state attributes are the running driver's view; an rpmsg device entry indicates enumeration, not successful application messaging; an FPGA manager entry does not prove a particular image is loaded or trusted. Some attributes or directories can be absent or inaccessible, and ordinary x86-64 QEMU may expose none. Record that as a legitimate result, together with the actual kernel version. Do not write state, select firmware, bind drivers, send messages or program an FPGA.

For source-only evidence, use the boot and send excerpts: write counts 0 → 1 → 0 when the first firmware request fails, and compare null endpoint -EINVAL with missing send operation -ENXIO. The return from a successful transport send does not establish a remote reply. Deliver the observed inventory or these two branch tables, with an explicit statement that a remote processor was not exercised.

Practice#

The following userspace model represents successful first boot, a second user and two shutdowns; it is not a firmware emulator:

python3 - <<'PY'
power = 0
for operation in ('boot A', 'boot B', 'shutdown A', 'shutdown B'):
    if operation.startswith('boot'):
        power += 1
        action = 'first boot' if power == 1 else 'reuse powered processor'
    else:
        power -= 1
        action = 'stop' if power == 0 else 'still needed'
    assert power >= 0
    print(operation, power, action)
PY

Expect counts 1, 2, 1, 0, with stop only on the final shutdown. Counterexample: append another shutdown; the assertion detects the unbalanced synthetic operation. Separately write the failure path 0 → 1 → 0 for failed firmware loading, and annotate that a failed stop restores its decremented count. Deliver both timelines and distinguish power from object references. No device files are opened.

Knowledge check#

1. Why is firmware not requested again when the power increment returns two?
2. What happens to the first power increment when firmware loading fails?
3. Why does a shutdown call not necessarily invalidate the rproc handle?
4. Why is rpmsg_send unsuitable while holding a spinlock?
5. If FPGA image loading fails, what does the region cleanup establish and what remains unknown?

Answers#

1. Another user already holds a power use, so the greater-than-one branch returns after incrementing rather than repeating first-user initialization.
2. The error path decrements power before unlocking, undoing the failed acquisition. The caller must handle the returned error.
3. The power count controls processor usage; the source explicitly says shutdown does not decrement the remoteproc object's general reference count. Object lifetime is a separate responsibility.
4. The API may block waiting for a transmit buffer and is documented for process context; holding a spinlock across a sleeping operation violates the locking context.
5. The error path releases acquired bridge references when applicable, unlocks the manager and puts the region. It does not establish that the previous image or bridge state was restored; that is platform and recovery-policy dependent.

Checkpoint#

Deliver both count timelines plus a stop-failure recovery note. Ready means every successful boot is balanced by a shutdown, failed boot rolls back, and power count is not confused with handle lifetime. Retry by tracing the mutex, increment, error label and shutdown decrement. This is not certification.

Sources#

• remoteproc core v6.12
• rpmsg core v6.12
• Remoteproc documentation
• rpmsg documentation