DMA mappings, IOMMUs, MSI, and device ownership

Before you start#

Read PCI and PCIe and bio and DMA completion. Linux v6.12 is a historical source baseline. DMA mapping behavior depends on device masks, architecture, IOMMU domain, coherency, configuration and platform firmware. The lab only reads published sysfs state; it does not assign devices or change IOMMU settings.

Learning goals#

Distinguish CPU virtual, physical and device-visible addresses; trace mapping and scatterlist error paths; explain ownership transitions, cache synchronization and ordering; and separate DMA translation from MSI interrupt remapping.

Source walkthrough#

A CPU virtual address is interpreted through CPU page tables. Physical memory addresses identify memory in the system's physical address space, but a device does not necessarily use those numbers directly. dma_addr_t is an address returned by the DMA API for a particular device and mapping; it can be an IOVA translated by an IOMMU, a direct mapping, or an address involving platform translation or bounce buffering. A CPU must not dereference a dma_addr_t as if it were a CPU pointer. Conversely, a kernel pointer or userspace virtual address cannot simply be handed to a device as its bus address. See kernel/dma/mapping.c and the DMA API HOWTO.

A device has DMA addressing constraints described by masks and other limits. dma_set_mask_and_coherent requests appropriate streaming and coherent masks and can fail; a driver that requires the mask must check and handle its result. A 32-bit mask means device-visible addresses must fit the permitted range, not that the CPU has only 4 GiB of RAM. An IOMMU can map suitable physical pages into an IOVA range the device can reach; SWIOTLB can use bounce buffers in applicable cases. Neither mechanism guarantees that every mapping succeeds: aperture exhaustion, allocation limits, device constraints and unsupported configurations can still cause failure.

In dma_map_page_attrs, Linux v6.12, GPL-2.0, a missing device DMA mask is rejected:

	if (WARN_ON_ONCE(!dev->dma_mask))
		return DMA_MAPPING_ERROR;

The function validates direction, checks the mask, and then chooses a direct, IOMMU or device DMA-ops path under its conditions. The returned dma_addr_t must be tested with dma_mapping_error where the API requires it. A mapping error is not an address to put into a hardware descriptor. The mapping operation can have cache, bounce-buffer or page-table work beyond a numerical address calculation.

iommu_dma_map_page, Linux v6.12, GPL-2.0-only, derives a physical address and obtains a DMA domain, IOVA allocator and device mask. For an applicable unaligned bounce case it checks:

		if (!is_swiotlb_active(dev)) {
			dev_warn_once(dev, "DMA bounce buffers are inactive, unable to map unaligned transaction.\n");
			return DMA_MAPPING_ERROR;
		}

If that branch requires a bounce buffer but SWIOTLB is not active, the function reports failure instead of exposing an unsuitable mapping. The rest of the function handles allocation, mapping and cleanup under its own constraints. This is a conditional example, not a claim that every unaligned DMA mapping bounces or that every IOMMU uses the same page-table implementation.

Coherent and streaming ownership#

Coherent DMA allocation supplies CPU and device addresses with a coherency contract suitable for shared control structures, subject to platform and API rules. It does not remove the need for ordering. A descriptor ring can be coherent while the device still needs to see descriptor fields in the right order before a doorbell write. dma_wmb can be needed for the relevant shared-memory ordering, and the MMIO accessor/barrier requirements are architecture and device specific. Do not assume a compiler assignment, cache coherence or an arbitrary barrier alone proves that every posted MMIO write reached hardware.

Streaming mappings are commonly used for packet or block data. For DMA_TO_DEVICE, the device reads memory; for DMA_FROM_DEVICE, the device writes it; DMA_BIDIRECTIONAL has the appropriate two-way synchronization requirements and should be chosen only when needed. A conceptual ownership sequence is: prepare memory, map for the right device and direction, give the returned address to hardware, wait for device completion or a documented abort, synchronize for CPU or unmap as required, then allow CPU reuse or free the pages. If the mapping is reused for another device operation, use the API's dma_sync_*_for_device and dma_sync_*_for_cpu rules where applicable. Access while the other side owns a streaming buffer can be invalid even on a platform where a casual test appears to work.

For a receive buffer, the device writes data into memory (DMA_FROM_DEVICE). A completion tells the driver that this operation has finished according to the device protocol; the driver performs required sync/unmap before the CPU consumes the data. For a transmit buffer (DMA_TO_DEVICE), the CPU prepares data before handing it to the device and must not mutate or free it during device ownership. A coherent descriptor ring and a streaming data buffer can coexist in one driver, with separate synchronization obligations. Completion, unmap and freeing are not interchangeable operations.

Scatterlists and a concrete mapping#

A scatterlist describes input memory segments. dma_map_sg can merge adjacent or otherwise compatible entries into fewer device-visible segments. The caller programs the returned number of DMA segments using the DMA addresses and lengths supplied by the API. The matching dma_unmap_sg receives the original number of entries passed to map, not the smaller returned count. In dma_map_sg_attrs, v6.12, a negative internal mapping result is converted to zero, the public failure indication for this API. The exact error cause is not necessarily visible from that zero alone.

In dma_map_sg_attrs, Linux v6.12, GPL-2.0 (the kerneldoc is omitted):

unsigned int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg,
		    int nents, enum dma_data_direction dir, unsigned long attrs)
{
	int ret;

	ret = __dma_map_sg_attrs(dev, sg, nents, dir, attrs);
	if (ret < 0)
		return 0;
	return ret;
}

The wrapper turns a negative internal result into 0, so at this level a failure appears only as a zero count. The excerpt does not show __dma_map_sg_attrs, where that negative value originates, or the merge rules that set the returned segment count, so it does not establish why a particular mapping failed.

Suppose a driver has three original scatterlist entries and a 32-bit DMA mask. The map succeeds and returns two DMA segments because the implementation merged eligible entries. The driver programs two returned DMA address/length pairs, but later calls unmap with original nents = 3 and the same device and direction. The addresses might be IOVAs, direct addresses or bounce-buffer addresses; the count alone does not reveal which. If the map returns zero, the driver must not program the device with CPU addresses or attempt a normal unmap as if mapping succeeded. It must release previously acquired, independently mapped resources according to their own successful-map records. These values are hypothetical and do not imply that all 32-bit devices bounce.

User buffers introduce another lifetime boundary. A userspace pointer can be unmapped or changed while a device request is in flight unless the subsystem establishes the required page pinning, access permissions and ownership. A driver must use the appropriate pinning and mapping APIs for its subsystem and release pins after device access is finished. A page pin is not a DMA mapping, and a DMA mapping is not permission for userspace to change a buffer while the device owns it. Long-term pins and memory accounting have additional rules. A buffer must survive cancellation and teardown until the specific operation's completion contract says it can be reused.

IOMMU domains, groups, and faults#

An IOMMU domain can contain page tables that translate an IOVA issued by a device into permitted physical memory. Mapping installs translation state; unmapping removes it and requires appropriate I/O translation lookaside buffer (IOTLB) invalidation under the implementation's rules. A device fault can indicate an unmapped address, permission violation, stale translation or hardware-specific condition, but its mere existence does not identify the application bug without context. The IOMMU API and driver decide how domain attachment, invalidation and fault reporting work. A driver must not unmap an IOVA and immediately reuse the underlying page while an uncontrolled device can still issue a stale transaction.

Isolation is a property of the whole topology and policy. An IOMMU group groups devices according to the kernel's isolation constraints; peer-to-peer paths, ACS, firmware and devices sharing translation domains can affect boundaries. A group symlink or enabled-looking sysfs directory is not a complete security proof against an untrusted device. Correct DMA masks and mappings limit addresses under the configured domain, but do not authenticate data the device writes within an allowed buffer. Host device passthrough and group reassignment are outside this lesson.

MSI and teardown#

MSI and MSI-X deliver interrupts using device-generated messages rather than a conventional shared INTx line. An interrupt-remapping facility, when present and configured, controls translation or authorization of interrupt messages. That is conceptually separate from IOMMU translation of a device's data DMA addresses, even if platform hardware or kernel subsystems coordinate both. Allocating an MSI-X vector, installing an IRQ handler and enabling a device queue are separate stages with separate failure paths. Vector count and affinity depend on hardware, driver and platform.

For removal, a sound dependency order is: prevent new submissions, stop or quiesce the device's DMA through its hardware protocol, disable or synchronize interrupts, cancel and drain work and completions, unmap streaming DMA mappings, release pins and buffers, then free device state under its reference rules. The exact order can vary with the device, and some hardware failure makes quiescing difficult; the driver must follow its recovery and bus contracts. synchronize_irq waits for relevant interrupt handlers but does not stop device DMA. Freeing a buffer just because no handler is currently running can permit a later device write into reused memory. Likewise, unmapping an IOVA is not a substitute for stopping a broken device and observing the required invalidation and ownership rules.

Mapping failure is part of the normal API#

A DMA mask is established for a particular struct device; an address valid for one device is not automatically valid for another. A successful mask setup states that the platform accepted the requested addressing constraint, not that future mappings are guaranteed. A subsequent map can fail because IOVA space, bounce buffers, page-table memory or another resource is unavailable. The caller must maintain enough state to unwind only maps that actually succeeded. A zero return from dma_map_sg is failure, while a page mapping uses the DMA mapping error check specified for that API. Mixing those error conventions can lead either to a leaked mapping or to programming an invalid address.

Suppose a driver maps buffer A successfully, then buffer B fails. It must not submit a descriptor referencing B. Before releasing A's mapping, it must know whether hardware could already access A; if the device was never notified, ordinary pre-submission unwind can release it under the API. If the device was notified between the two maps, the driver first needs the hardware-specific abort or completion protocol. Then it unmaps A with the original device, direction and size, and finally frees the backing memory when all other users are done. The exact order depends on the driver's protocol; the key invariant is that neither a failed map nor a software cleanup label stops an independently running DMA engine.

Visibility, ordering, and the doorbell#

Coherency and ordering solve different questions. A coherent ring can make descriptor updates visible to CPU and device without the streaming cache-maintenance sequence, but the device must not observe an ownership bit or doorbell before the descriptor fields it depends on are ordered. A driver can write address and length, use the appropriate DMA write barrier, publish the descriptor ownership state and notify the device using the required MMIO accessor, subject to the hardware's specified protocol. Some devices require a readback or additional ordering for posted writes; no single generic code fragment is correct for all hardware. The source and device specification must establish the actual barrier placement.

Streaming buffer ownership is also directional. For a device read (DMA_TO_DEVICE), CPU preparation precedes the map or sync-for-device transition, and the CPU must respect the period of device ownership. For a device write (DMA_FROM_DEVICE), software waits for the hardware completion and performs the required sync-for-CPU or unmap before consuming data. A bidirectional mapping needs both sides' applicable transitions. On a coherent architecture some synchronization operations can be cheap or compile to little work, but portable code must still call the APIs required by its contract.

IOVA invalidation and a device fault#

In an IOMMU-backed DMA domain, the DMA API may allocate an IOVA, install translations to physical pages, and return that IOVA as the device address. Translation caches in an IOMMU or device can retain entries, so unmapping involves the relevant invalidation protocol before those addresses or pages are safely reused under the implementation's rules. A driver does not normally implement an arbitrary IOMMU page-table walk itself. It must, however, make sure the device is no longer issuing requests to the mapping being removed. IOTLB invalidation and hardware quiescence are complementary responsibilities.

Imagine a faulty device issues an IOVA after its mapping has been removed. If the configured IOMMU and topology enforce the domain, a fault may be recorded instead of allowing that access; fault reporting and subsequent device handling are platform-specific. The existence of an IOMMU group does not prove the device is attached to the expected domain, that all peer paths are isolated, or that permitted buffers contain trustworthy data. A device can still corrupt memory it is legitimately mapped to. Interrupt remapping addresses delivery of MSI messages separately from data-buffer DMA translation.

A userspace-backed DMA request adds page lifetime and permission checks. A subsystem can pin pages before mapping them, but pinning consumes resources and has limits, especially for long-lived registrations. Cancellation, process exit and device removal can race with I/O completion; references to the pages and mapping must remain until the applicable operation is stopped and completion handled. Only then can the driver unmap, unpin and permit memory reuse. A virtual address remaining numerically unchanged is not proof that it still maps the same pages.

Practice#

Source-only route:

rg -n -A30 'dma_addr_t dma_map_page_attrs\(' "$KERNEL_SRC/kernel/dma/mapping.c"
rg -n -A18 'unsigned int dma_map_sg_attrs\(' "$KERNEL_SRC/kernel/dma/mapping.c"
rg -n -A42 'dma_addr_t iommu_dma_map_page\(' "$KERNEL_SRC/drivers/iommu/dma-iommu.c"

Verify v6.12. Trace the missing-mask error, direct/IOMMU/ops selection, sg failure return and conditional SWIOTLB error. Write out the three-input/two-output mapping example and identify the original count used for unmap. The code-reading route does not map a real buffer.

Read-only Linux observation:

python3 - <<'PY'
from pathlib import Path
groups = Path('/sys/kernel/iommu_groups')
try:
    gs = sorted(groups.iterdir()) if groups.is_dir() else []
    print('visible IOMMU groups:', len(gs))
except OSError as e: print('groups unavailable:', e.strerror)
pci = Path('/sys/bus/pci/devices')
try:
    ds = sorted(pci.iterdir()) if pci.is_dir() else []
    linked = sum((d / 'iommu_group').exists() for d in ds)
    print('visible PCI functions:', len(ds), 'with group link:', linked)
except OSError as e: print('PCI inventory unavailable:', e.strerror)
PY

Zero groups or devices can mean unsupported hardware, disabled configuration, a restricted container or another environment-specific condition. A visible group link shows a published relationship, not that a device is safely isolated or that any particular DMA operation used an IOMMU. Do not change kernel arguments, domains, drivers or device assignment. Deliver the counts or error, the 3-to-2 scatterlist example and a teardown timeline with a hardware-DMA stop distinct from IRQ synchronization.

Knowledge check#

1. Why is a dma_addr_t not generally a CPU pointer?
2. What does the shown mapping function return when the device has no DMA mask?
3. If three scatterlist entries map to two DMA segments, which counts are used for programming and unmapping?
4. Why can coherent DMA memory still require a write barrier before a doorbell?
5. Why is synchronize_irq insufficient to stop a device from writing a freed buffer?

Answers#

1. It is a device-visible address for a particular mapping and can be translated or bounced; CPU virtual address translation is a different domain.
2. DMA_MAPPING_ERROR; the caller must not program that result into the device.
3. Program the two mapped DMA segments returned; unmap using the original input count of three with the same mapping parameters.
4. Coherency addresses visibility properties, while ordering of descriptor writes and notification remains a separate CPU/device protocol requirement.
5. It synchronizes interrupt handlers, not the hardware DMA engine; new and in-flight DMA must be stopped or otherwise safely handled before unmapping and freeing.

Checkpoint#

Save the mapping and teardown timelines and the read-only group counts or limitation. Pass when you distinguish CPU, physical and DMA addresses; correctly use both scatterlist counts; and explain mapping failure, ordering and device quiescence. Re-read the cited mapping branches and retry if any ownership step is unclear. This is practice, not certification.

Sources#

• DMA mapping dispatch, Linux v6.12.
• IOMMU DMA mappings.
• Direct DMA.
• SWIOTLB bounce buffering.
• DMA API HOWTO.
• PCI interrupt-vector API.