CPU privilege, caches, memory, and I/O

Before you start#

Read ELF, linking and the ABI. The practice uses read-only host observations and requires no privileged access. The source examples are v6.12; the host's architecture, configuration and kernel may differ. Later memory, interrupt and DMA chapters develop these models further.

Learning goals#

Distinguish registers, privilege levels, system calls and interrupts; explain a virtual-address translation and the role of a TLB; distinguish cache coherence from ordering; and identify why MMIO and DMA use dedicated kernel interfaces.

Instructions and privilege#

A CPU executes instructions using registers: small, fast architectural storage holding values such as operands, a program counter and a stack pointer. An ABI tells independently compiled code how to pass arguments and preserve selected registers. The CPU also maintains privilege state. On common x86-64 Linux, userspace executes at a less privileged level and kernel code at a privileged level. On arm64 Linux, userspace normally runs at EL0 and the kernel at EL1; virtualization can add another level. These are architecture-specific descriptions, not a universal machine model.

A system call is a deliberate request to enter the kernel using an architecture-defined instruction and ABI. An exception can also arise from an instruction, such as a page fault; an interrupt generally signals an asynchronous event such as a device or timer. The processor and entry code preserve enough context to handle the event and eventually resume or choose other work. These paths have different causes and context restrictions. A device interrupt does not mean the interrupted userspace program explicitly requested a system call.

Source walkthrough#

The x86-64 entry path in v6.12 arch/x86/entry/entry_64.S begins the entry_SYSCALL_64 body with the following excerpt (SPDX GPL-2.0; hints and surrounding instructions omitted):

swapgs
movq	%rsp, PER_CPU_VAR(cpu_tss_rw + TSS_sp2)
SWITCH_TO_KERNEL_CR3 scratch_reg=%rsp

This is x86 assembly, not C. The code swaps the relevant GS base, saves the incoming stack pointer in per-CPU scratch storage, and invokes the kernel page-table switching macro. Configuration and mitigation alternatives affect generated behavior; follow the surrounding comments and macros before assuming a particular machine instruction executes. The following source loads the kernel stack and constructs saved register state. This excerpt is a map of one entry implementation, not a trace from your workstation. Compare arm64 arch/arm64/kernel/entry.S, whose entry machinery uses different registers and exception-level conventions.

For device registers, the generic implementation of readl in v6.12 include/asm-generic/io.h includes (SPDX GPL-2.0-or-later; logging and return omitted):

__io_br();
val = __le32_to_cpu((__le32 __force)__raw_readl(addr));
__io_ar(val);

The before/after hooks express architecture-defined I/O ordering around the raw read; the conversion interprets a little-endian 32-bit register value as CPU order. An architecture may override this generic implementation. __iomem marks a device-I/O address in the function declaration, not a normal userspace pointer. A driver must follow the device and architecture contracts, use the appropriate accessor and mapping, and not assume ordinary C loads provide the required semantics. A posted write can need a device-specific readback or other synchronization; the exact rule is not universal. See device I/O documentation.

Translation and locality#

A process uses virtual addresses. Page tables describe mappings and permissions from virtual pages to physical memory or other backed regions. A TLB caches translations, not the data stored at those addresses. If translation or permission checks cannot satisfy an access, the CPU raises a fault; the kernel may resolve a legitimate demand fault, or report failure to the process. A TLB hit does not prove the data is in a CPU data cache. When mappings change, the kernel must coordinate appropriate translation invalidation; exact mechanisms and page-table formats depend on architecture and configuration.

Caches hold copies of data at a granularity often called a cache line. Nearby accesses can benefit from locality; unrelated variables sharing a line can also cause contention between CPUs. Coherence helps CPUs converge on a consistent view of a location under its rules, but does not by itself supply the ordering needed for a multi-step synchronization protocol. Compiler reordering and CPU ordering both matter. Do not infer a portable synchronization algorithm from one apparently successful experiment on x86.

In a simplified example, a program reads virtual address V on CPU 0. Its page-table mapping grants read permission and maps V to physical page P; a cached TLB translation may avoid a fresh page-table walk. The data cache can still miss and fetch the corresponding line. If another CPU changes the mapping under the relevant memory-management protocol, invalidation is needed before stale translations can be used incorrectly. This is a conceptual sequence, not a measurement and not a claim that every access walks the tables.

Devices and ownership#

Memory-mapped I/O maps device registers into an address space. Reading a register can have side effects; ordinary RAM assumptions about speculative or repeated reads may not hold. DMA allows a device to access a buffer using a device-visible address returned by the DMA API. A CPU virtual pointer, CPU physical address and DMA address are distinct concepts, especially with an IOMMU. Mapping, synchronization and unmapping transfer or coordinate ownership as documented by the API. A device completing a transfer must not write into memory that has been unmapped or freed. Consult the DMA API guide and v6.12 DMA declarations.

In v6.12 include/linux/dma-mapping.h, SPDX GPL-2.0 (the inline wrapper only; the rest of the header omitted):

static inline dma_addr_t dma_map_single_attrs(struct device *dev, void *ptr,
		size_t size, enum dma_data_direction dir, unsigned long attrs)
{
	/* DMA must never operate on areas that might be remapped. */
	if (dev_WARN_ONCE(dev, is_vmalloc_addr(ptr),
			  "rejecting DMA map of vmalloc memory\n"))
		return DMA_MAPPING_ERROR;
	debug_dma_map_single(dev, ptr, size);
	return dma_map_page_attrs(dev, virt_to_page(ptr), offset_in_page(ptr),
			size, dir, attrs);
}

The wrapper takes a kernel virtual pointer, rejects vmalloc memory with a warning, and passes the backing page and in-page offset to dma_map_page_attrs. The value the caller receives is whatever that call returns, a dma_addr_t, not the CPU pointer. The excerpt does not show the device's view of the address, any IOMMU translation, or the later synchronization and unmapping that the ownership rules require.

Faults and device completion#

Suppose a process accesses a valid virtual address whose physical backing has not yet been installed. A page fault enters architecture-specific kernel code. The memory-management path can establish the mapping and retry the instruction if the access is legitimate and resources are available; an invalid address or prohibited access can instead lead to a signal. A fault is therefore not synonymous with a bug, and a fault count is not a count of disk reads. By contrast, a device DMA completion concerns an asynchronous device operation: the driver must determine that the device is finished and follow the DMA synchronization and unmapping contract before the CPU reuses or frees the buffer. Neither the interrupt notification nor an ordinary C pointer alone describes all ownership transitions.

From instructions to a scheduled task#

An instruction is an operation interpreted by a processor architecture; registers hold operands and execution state. The program counter identifies the next instruction under the architecture's execution model, and a stack pointer helps locate a task's current stack. Registers are not a process's whole state: address-space roots, signal state, file references and kernel bookkeeping also matter. A physical core can expose multiple logical CPUs through simultaneous multithreading, and those logical CPUs may share caches and execution resources. “One CPU” in a scheduler listing normally means one logical CPU, not necessarily one separate core.

A context switch can save one task's registers and stack position and restore another's, with architecture-specific address-space handling when needed. Returning to a previously scheduled task resumes its earlier continuation. By contrast, a syscall transition enters privileged kernel code on behalf of the same task and need not schedule another task. An interrupt can arrive while a task is running, execute an interrupt path, and return to it without a task switch; it can also lead to later rescheduling. Counting syscalls, interrupts and context switches therefore measures different events.

Privilege level and user identity are separate axes. On a typical x86-64 Linux system an ordinary application and a program running with effective UID 0 both execute their normal instructions in user mode; neither may directly perform arbitrary privileged instructions merely because of UID. The UID and capabilities affect kernel authorization decisions when requests cross the boundary. The kernel itself runs in privileged mode, but it still validates pointers and enforces applicable policy. On arm64, EL0 and EL1 are a useful ordinary userspace/kernel contrast, with EL2 and firmware arrangements depending on the system. Root is an identity and authority concept, not a CPU ring.

An arm64 entry contrast#

In v6.12 arch/arm64/kernel/entry.S, the kernel_entry macro saves register pairs, including this excerpt (SPDX GPL-2.0-only; conditionals and other registers omitted):

stp	x0, x1, [sp, #16 * 0]

stp stores two registers at an address derived from the stack pointer. The entry macro saves additional registers and constructs architecture-specific state; the one instruction does not establish a complete syscall path or show how the stack was selected. Compare the x86 swapgs and stack-switch excerpt above: neither register naming nor the transition mechanics are interchangeable. A source listing tells us which conditional code exists at v6.12; configuration, CPU features and the particular exception determine which instructions execute. A failed user-memory access can take an exception-fixup or signal path rather than returning from a normal device interrupt.

A worked translation and false-sharing example#

Assume, only for this example, a machine with 4 KiB base pages and 64-byte cache lines. Process A accesses virtual address 0x400038; the offset within a 4 KiB page is 0x038. If its page table maps the containing virtual page to physical page frame F with suitable permission, the access refers to offset 0x038 within F. A TLB can cache that translation, while a data cache separately holds a line containing the data. A permission failure or missing entry can cause a fault even though the address is numerically well formed. Neither the virtual value nor a TLB hit reveals the device DMA address.

Now put independent counters at offsets 0x00 and 0x08 in a 64-byte-aligned shared line. Two threads updating different counters may still exchange cache-line ownership because coherence operates on the line, a phenomenon called false sharing. They need not have a language-level data race on the same counter for the coherence traffic to hurt performance. Padding or separating counters can change the result, but the right line size, alignment, sharing topology and workload must be measured. This is a conceptual example, not a benchmark; other CPUs can use different line sizes or coherence behavior. Coherence does not itself order a multi-location publication protocol, as the later memory-model chapter explains.

ASIDs or PCIDs can tag translations so a CPU need not discard every cached translation on every address-space switch, depending on architecture and configuration. They do not turn a virtual address into a global physical identifier, and they do not remove the need to invalidate stale translations after mapping changes. A page-table walk can be cached at multiple levels; “TLB miss” is not synonymous with a disk read. A present translation can still encounter a data-cache miss, while a valid virtual address can fault because its page is not currently present.

Device handoff and ordering#

For DMA, a driver obtains a device-visible mapping under the DMA API, prepares data and transfers ownership according to the mapping's direction and synchronization rules. The device may then read or write memory asynchronously. Completion must be established through the device's documented mechanism before the CPU reuses or frees a buffer; an interrupt is one possible notification, and it does not automatically validate every field in a descriptor. A timeout or disconnect can leave work in flight, so teardown must stop the device and synchronize completion before unmapping or freeing the buffer.

An IOMMU can translate device-visible addresses and enforce configured isolation, but it does not make a CPU virtual pointer a DMA address or remove the driver's ownership obligations. An MMIO register read can have device side effects, and writes may be posted. The readl example above includes generic ordering hooks; a particular architecture can override it and a particular device can require additional ordering or a readback. CPU memory barriers, MMIO accessor rules and DMA synchronization each solve different parts of the handoff. Inspect the v6.12 DMA mapping declarations and the device's contract before reasoning about a driver.

Practice#

On an ordinary Linux host, inspect the machine without changing settings:

uname -m
getconf PAGESIZE
lscpu | sed -n '1,25p'
ls /sys/devices/system/cpu/cpu0/cache 2>/dev/null
cat /proc/self/maps | head -12

uname reports an architecture label; page size and cache information depend on the machine and exposure. /proc/self/maps is read by cat, so it describes that process, not the shell. Its ranges are virtual mappings, not physical addresses or proof that pages are resident. In a restricted container, cache information or proc entries can be missing. Record the exact error instead of assuming the host lacks a cache. No command above triggers a controlled page fault measurement or reveals the true ordering of a device transaction.

Deliver a sketch with a virtual address, a translation, a cache and a device DMA address, and label which observed command supports each statement. The failure/counterexample is treating an address printed by maps as a device DMA address: the display does not supply such a mapping. No files are created, so there is no cleanup.

Knowledge check#

1. How does a system call differ from an asynchronous device interrupt?

2. What does a TLB cache, and why is a TLB hit not a data-cache hit?

3. Why is a CPU virtual pointer not a DMA address?

4. What do the readl hooks and conversion show, and why is this not a universal implementation?

5. Why does a userspace program with effective UID 0 not automatically execute privileged CPU instructions?

Answers#

1. A system call is a deliberate ABI-defined request by a program; an interrupt signals an event that can arrive independently of the current instruction stream. Both involve architecture-specific entry handling.

2. It caches address translations and associated information, not the requested data line. Data may still need to be fetched through the cache hierarchy.

3. A device uses an address supplied under the DMA mapping contract, potentially translated by an IOMMU. Mapping and ownership rules must be followed separately from CPU addressing.

4. This generic implementation surrounds the raw read with I/O ordering hooks and converts little-endian data. Architectures can override it and devices impose additional ordering requirements.

5. UID and capabilities affect kernel authorization checks. The program normally still executes in user privilege mode and must use allowed interfaces to request privileged operations.

Checkpoint#

The artifact is the labeled address-and-device sketch with host observations. Pass when it separates translation from data caching, distinguishes syscall from interrupt, and names one ownership rule for DMA. If uncertain, reread the linked interfaces and revise the sketch rather than extrapolating from maps. This is not certification.

Sources#

• v6.12 x86 entry code.
• v6.12 arm64 entry code.
• v6.12 generic I/O accessors.
• Device I/O documentation.
• DMA API guide.