PCI/PCIe discovery, resources, and error handling

Before you start#

Read platform firmware and devices and the driver model. Linux v6.12 is the historical source baseline. PCI topology, firmware resource assignment, drivers, IOMMU and interrupts depend on the actual platform. The lab reads published information only; it does not write configuration space or reset hardware.

Learning goals#

Describe enumeration and BAR resources, follow a specific NVMe PCI probe and error unwind, distinguish CPU MMIO from DMA, and explain why reset, interrupts and hot removal require coordinated lifetime management.

Source walkthrough#

A PCI domain contains buses connected through bridges and endpoints. A PCI function is identified in a given topology by domain:bus:device.function (BDF); this is an address in the current enumeration, not a permanent hardware identity. struct pci_bus models a bus and struct pci_dev models a function. Configuration space contains vendor/device IDs, class information, capabilities and resource declarations. Enumeration scans buses, discovers functions and bridges, assigns or validates resources under firmware and kernel policy, and creates device objects. A driver matches a device through IDs, class or other supported matching rules and can then be probed. See drivers/pci/probe.c and drivers/pci/pci-driver.c.

A base address register (BAR) describes a device resource aperture such as memory-mapped registers. Bridge windows must make downstream resources reachable. The kernel resource describes a bus resource range; a driver must claim and map an appropriate memory BAR before CPU register access using the correct I/O accessors and device protocol. A BAR address, a kernel __iomem virtual address and a dma_addr_t usable by the device are different domains. A device writing host memory uses a DMA mapping, not the CPU's BAR virtual pointer. Reading a published resource file is not the same as safely accessing arbitrary configuration registers or device MMIO.

For a concrete v6.12 driver, the PCI NVMe path has its own ordering; it is not a universal probe recipe. nvme_probe allocates controller state, adds the controller, calls nvme_dev_map to request memory regions and map the BAR, creates PRP pools and an I/O-descriptor mempool, then calls nvme_pci_enable. Its device allocation path selects a DMA mask according to a quirk and configures alignment and segment limits. The particular mask-setting calls in this source are not checked at that point: do not invent a branch that aborts probe on their return. Other drivers can and should handle their own mask setup failures according to their APIs and requirements.

The helper nvme_dev_map, Linux v6.12, GPL-2.0, has this region-acquisition boundary:

	if (pci_request_mem_regions(pdev, "nvme"))
		return -ENODEV;

If the memory regions cannot be claimed, the helper returns an error before mapping them. If the later BAR remap fails, its release label calls pci_release_mem_regions. On success, the caller owns the acquired mapping/resources until the corresponding unmap and release. Merely finding a BAR in enumeration is not permission for two drivers to use it simultaneously.

nvme_pci_enable calls pci_enable_device_mem, enables bus mastering, checks controller state and allocates an initial interrupt vector. Its vector error branch is:

	result = pci_alloc_irq_vectors(pdev, 1, 1, flags);
	if (result < 0)
		goto disable;

The driver requests one vector for this setup stage and follows its disable path if allocation fails. It can later adjust queue and vector setup. queue_request_irq uses pci_request_irq for an actual queue handler on applicable paths; allocating a vector is not the same as registering a handler or receiving an interrupt. NVMe's ordering and error labels are evidence for this driver and version, not an instruction to use its exact ordering for every PCI device.

Resources, mask failure, and unwind#

Imagine a BAR assigned the half-open range [0x80000000, 0x80010000): its aperture is 64 KiB. The driver claims the resource, maps the permitted register range to a CPU __iomem address, and uses device-specific register offsets within that aperture. It must check the actual resource length before assuming offset 0x10000 is valid; that offset is just beyond the example range. The mapping's virtual address is not necessarily numerically equal to the BAR's bus address. A bridge window and platform translation also affect access.

The NVMe helper that maps the register window checks the requested size against the BAR length before it calls ioremap. In drivers/nvme/host/pci.c, Linux v6.12, GPL-2.0 (surrounding code omitted):

static int nvme_remap_bar(struct nvme_dev *dev, unsigned long size)
{
	struct pci_dev *pdev = to_pci_dev(dev->dev);

	if (size <= dev->bar_mapped_size)
		return 0;
	if (size > pci_resource_len(pdev, 0))
		return -ENOMEM;
	if (dev->bar)
		iounmap(dev->bar);
	dev->bar = ioremap(pci_resource_start(pdev, 0), size);
	if (!dev->bar) {
		dev->bar_mapped_size = 0;
		return -ENOMEM;
	}
	dev->bar_mapped_size = size;
	dev->dbs = dev->bar + NVME_REG_DBS;

	return 0;
}

nvme_remap_bar rejects a size larger than pci_resource_len(pdev, 0) and maps the BAR's start address with ioremap. The claim step, pci_request_mem_regions in nvme_dev_map, is not in this excerpt and runs before it. The result is a CPU pointer, not a DMA address the device can use.

For a separate hypothetical driver, suppose dma_set_mask_and_coherent for a 32-bit mask returns an error. That driver must follow its own checked failure path and not enable DMA using an assumed working mask. If it has already registered an IRQ or mapped a BAR, it must unwind only the stages it acquired, in a safe reverse dependency order. This example is not a claim that the cited NVMe source checks that exact mask return. DMA mappings can fail later even after a mask is accepted; the mask describes a device addressing constraint, not a guarantee that every allocation can be mapped.

The cited NVMe nvme_probe has staged labels for failures: a failure after enable can disable the device and release queues/host memory, while later labels destroy the IOD mempool, release PRP pools, unmap the device, uninitialize the controller and put its reference. Which labels are reached depends on the failing stage. An error before a resource is acquired must not free an uninitialized object; an error after a successful acquisition must not leak it. Managed PCI APIs used by another driver can alter the cleanup mechanics, but cannot stop device DMA or synchronize a private worker without the relevant driver protocol.

Interrupts, reset, and topology#

Legacy INTx is a line-based interrupt mechanism with sharing and acknowledgement considerations. MSI and MSI-X use message-signaled interrupts; MSI-X can support multiple independently configured vectors where the device and platform permit them. Drivers request a supported range and must handle fewer vectors or failure according to their design. Multiqueue devices often distribute queues across vectors and CPUs, but one queue per CPU or one interrupt per I/O is not guaranteed. Polling, interrupt moderation and shared vectors change observation. Interrupt remapping, when configured, is separate from translating the device's data DMA addresses.

PCIe Advanced Error Reporting (AER), where supported and enabled, reports and coordinates certain PCIe errors. A driver's pci_error_handlers can participate in error detection, reset and recovery through the PCI error-recovery framework; callbacks and recovery outcomes depend on the driver and topology. Reset, suspend and resume can require saving/restoring configuration, quiescing queues and reinitializing device-specific state. AER is not a guarantee that every device can recover without data loss. See drivers/pci/pci-driver.c and PCI error recovery documentation.

Hot unplug creates a race between removal and I/O completion. A safe driver stops new submissions, tells the device to stop DMA where possible, synchronizes interrupts and work, finishes or fails outstanding requests, then releases mappings, IRQs and private state under its actual device protocol. synchronize_irq waits for interrupt handlers; it does not stop a device from issuing DMA. A pci_dev reference can keep a kernel object alive after physical disappearance, but cannot make the device respond. Reading a BAR after removal may be invalid even while a pointer to pci_dev exists.

IOMMU groups, ACS, and SR-IOV#

An IOMMU group is a kernel grouping of devices that cannot necessarily be isolated from each other at a finer granularity under the platform's topology and rules. Access Control Services (ACS) and PCIe routing affect whether peer-to-peer traffic can bypass expected isolation points; the presence of an IOMMU group symlink alone does not prove a complete security boundary. SR-IOV can expose physical and virtual functions with distinct configuration and driver lifecycles, but their isolation and resource sharing depend on hardware, firmware and kernel configuration. This chapter does not assign devices to VMs or change group settings.

A BDF and driver symlink observed in sysfs can change after hotplug, driver binding or reboot. A vendor/device ID identifies a model or family, not a unique serial number. A BAR resource listing gives assigned ranges and flags; it does not establish current safe MMIO access, DMA translation or interrupt delivery. Read-only inventories are useful for forming a topology hypothesis, but a real driver audit must inspect its probe, removal and error-recovery code.

Enumeration and resource windows#

A root bridge begins a PCI hierarchy, and a PCI-to-PCI bridge can expose a subordinate bus. The bridge's forwarding windows and the endpoint's BAR assignment must both support access to a downstream resource. Firmware may provide initial assignments and the kernel can validate or adjust them under platform policy. Enumeration reads standard configuration information to discover functions and capabilities; not every arbitrary configuration register is safe to read on every device. A BDF can change after topology or firmware changes, so it is useful for locating the current function, not for proving that a device is a particular physical unit across reboots.

A driver match is followed by probe only under the driver core and PCI bus rules. The probe must verify required capabilities and resources before enabling the device's data path. A memory BAR can represent control registers, device memory, or another device-defined aperture; the driver needs the device's programming specification to interpret it. The kernel mapping has I/O semantics, so ordinary C pointer dereferences are not a general replacement for readl/writel or the appropriate accessors. A BAR describes device resources, while the DMA mask and DMA mapping describe which host-memory addresses the device may use.

Why the NVMe sequence is not a template#

For the cited nvme_probe, controller allocation and nvme_add_ctrl precede nvme_dev_map; that helper requests memory regions and maps the BAR. PRP pools and the IOD mempool are then created before nvme_pci_enable. The enable helper calls pci_enable_device_mem, sets bus mastering, checks a controller status register and requests an initial vector. Admin and I/O queue setup and their IRQ requests occur in later device-specific work. The error labels unwind only resources reached on the corresponding path. In particular, the mask calls observed in nvme_pci_alloc_dev are not checked there; the hypothetical checked-mask example illustrates a different driver's valid failure obligation rather than attributing nonexistent behavior to NVMe.

The enable helper is where the controller register is first read through the mapped BAR. In the same file, Linux v6.12, GPL-2.0 (surrounding code omitted):

	if (pci_enable_device_mem(pdev))
		return result;

	pci_set_master(pdev);

	if (readl(dev->bar + NVME_REG_CSTS) == -1) {
		result = -ENODEV;
		goto disable;
	}

pci_enable_device_mem and bus mastering come before the read. A status read of all ones is treated as a missing controller and jumps to disable, so the enable step depends on the BAR mapping from nvme_dev_map succeeding. The vector allocation that follows is the one quoted earlier in the chapter; the excerpt stops before it.

A real driver can choose managed region or mapping APIs, different DMA masks, multiple vector requests, or a different registration order. These choices must still satisfy dependencies: no device DMA into unprepared memory, no handler using uninitialized private state, and no freeing of a mapping while callbacks or hardware can still access it. If requesting an IRQ fails after allocating vectors, the appropriate path must release the vector resources and any prior setup. A successful probe means the driver's initialization reached its success point under its conditions; it does not prove every later device command will complete.

Error recovery and removal are state machines#

AER and PCI error recovery can ask a driver whether it can recover, whether reset is required, and whether it can resume, according to the framework and device. While a channel is offline, reads can fail or return device-defined error values; blindly continuing normal MMIO is unsafe. An NVMe reset or removal must coordinate block queues, controller state, interrupts and DMA mappings. In the cited driver, reset preparation is serialized by the PCI device lock against removal as noted in its source comment; this fact should not be generalized to every asynchronous callback without checking its own synchronization.

Consider an unplug after request R has been submitted. The application fd and pci_dev object can still have references while the physical endpoint disappears. The driver must stop new requests, handle or fail R, stop or account for DMA, synchronize handlers and workers, and then release the resources that no longer have users. A late completion must not access a freed request or ring. If the device is already inaccessible, the recovery path may have to treat operations as failed rather than waiting forever for a hardware acknowledgement. The exact path depends on the bus and driver; the source exercise maps the NVMe labels instead of claiming that every PCI driver performs one identical sequence.

SR-IOV can create virtual functions associated with a physical function, with resources and reset behavior governed by the device and PCI core. IOMMU grouping and ACS influence the isolation granularity, while peer-to-peer routing and shared hardware resources can create additional constraints. A read-only iommu_group symlink is a useful starting point for inventory, not proof that a function is safe to assign to an untrusted guest. This curriculum does not ask you to enable SR-IOV or assign a host device.

Interrupt and DMA teardown are independent#

An allocated MSI or MSI-X vector is a kernel resource; registering a handler associates a callback with an IRQ, and enabling the device's interrupt generation is another device-specific step. On failure or removal, a driver must account for the exact steps reached. A handler can schedule deferred work, so waiting for the handler alone does not necessarily drain that work. Similarly, a device can continue DMA while no interrupt handler is currently running. The correct cleanup order depends on the device, but it must stop or safely contain DMA and synchronize every callback that can access state before freeing that state.

The queue handler is registered by a separate helper after the vector is allocated. In drivers/nvme/host/pci.c, Linux v6.12, GPL-2.0 (surrounding code omitted):

static int queue_request_irq(struct nvme_queue *nvmeq)
{
	struct pci_dev *pdev = to_pci_dev(nvmeq->dev->dev);
	int nr = nvmeq->dev->ctrl.instance;

	if (use_threaded_interrupts) {
		return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq_check,
				nvme_irq, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
	} else {
		return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq,
				NULL, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
	}
}

The helper passes the queue's vector, a handler (a check function is added when threaded interrupts are selected) and the queue as the cookie to pci_request_irq. The excerpt does not show the matching free path, and it does not say when the device starts generating interrupts for this vector.

Consider the hypothetical BAR aperture again. The CPU mapping might remain a valid kernel virtual mapping while the endpoint is physically gone; an access can nevertheless fail or return an error value because the hardware is unavailable. Keeping a pci_dev reference preserves the kernel object under its lifetime rules, not the physical link. In the NVMe source, probe error labels and reset/remove paths are different: do not jump from a failure before queue creation into a cleanup operation that assumes a fully live queue. Record each successful stage and follow the corresponding source label.

Practice#

Source-only route:

rg -n -A22 'static int nvme_dev_map\(' "$KERNEL_SRC/drivers/nvme/host/pci.c"
rg -n -A42 'static int nvme_pci_enable\(' "$KERNEL_SRC/drivers/nvme/host/pci.c"
rg -n -A105 'static int nvme_probe\(' "$KERNEL_SRC/drivers/nvme/host/pci.c"
rg -n -A20 'static int queue_request_irq\(' "$KERNEL_SRC/drivers/nvme/host/pci.c"

Verify v6.12. Mark region claim/remap, vector allocation, actual IRQ request and each visible failure label. Distinguish the NVMe mask calls from the hypothetical checked-mask failure; do not assume the source checks a return it ignores.

Read-only inventory:

if command -v lspci >/dev/null 2>&1; then lspci -nn; lspci -k; else echo 'lspci unavailable'; fi
python3 - <<'PY'
from pathlib import Path
p = Path('/sys/bus/pci/devices')
devs = sorted(p.iterdir()) if p.is_dir() else []
print('visible PCI functions:', len(devs))
if devs:
    d = devs[0]
    print('example BDF:', d.name)
    for link in ('driver', 'iommu_group'):
        try: print(link + ':', (d / link).resolve(strict=True).name)
        except OSError: print(link + ': unavailable')
    try: print('resource lines:', len((d / 'resource').read_text().splitlines()))
    except OSError as e: print('resource unavailable:', e.strerror)
PY

A VM or container can expose no PCI functions, no driver symlink, no group or no lspci; permission and hotplug races are valid limitations. The resource-line count is not a proof that every BAR is assigned or safely accessible. Do not use lspci -xxx, write config, bind/unbind, rescan or reset. Deliver the BAR arithmetic, NVMe acquisition/unwind map and observed or missing inventory.

Knowledge check#

1. What does a BAR resource describe, and why is it not a DMA address?
2. What does the cited nvme_dev_map do when memory-region reservation fails?
3. What is the difference between allocating a vector and registering a handler?
4. Why does synchronize_irq alone not make removal safe for DMA memory?
5. What does an IOMMU group link fail to prove by itself?

Answers#

1. It describes a device resource aperture; CPU mapping and the device-visible addresses for host memory are separate address domains and APIs.
2. It returns -ENODEV before the later BAR remap attempt.
3. Allocation obtains interrupt-vector resources; requesting an IRQ installs a handler for a selected vector under the driver's setup.
4. It waits for relevant handlers, but a device may still issue DMA until the hardware and driver stop or otherwise synchronize it.
5. It does not by itself prove isolation from every peer, correct IOMMU policy, trustworthy firmware or a complete VM-assignment boundary.

Checkpoint#

Save the resource arithmetic, staged NVMe error diagram and read-only inventory. Pass when you can distinguish BAR, CPU mapping and DMA address, and explain a failure before IRQ registration and an unplug race after I/O starts. Re-read the actual source labels if any resource owner is unclear. This is practice, not certification.

Sources#

• PCI enumeration, Linux v6.12.
• PCI driver binding and PM.
• NVMe PCI probe, IRQ, and removal.
• PCI interrupt-vector API.
• PCI error recovery documentation.