Atomics, barriers, and the Linux memory model

Before you start#

Read locks and sleeping, CPU, memory and I/O, and scheduler core. This chapter uses upstream v6.12 Linux Kernel Memory Model (LKMM) sources. Its supplied two-CPU analysis is source-only; the optional host program reports properties of a C11 atomic in its own process and is not a kernel memory-model test. No hardware ordering outcome is claimed from running Python threads.

Learning goals#

Distinguish atomicity, compiler ordering, CPU ordering and object lifetime. Explain a release/acquire publication and its failure modes, including a failed conditional read-modify-write. Analyze the supplied message-passing outcomes with their assumptions and identify why MMIO and DMA need separate ordering rules.

Coherence is not a complete protocol#

A compiler can reorder, combine, eliminate or invent some memory accesses while preserving the semantics it is required to provide for the language and compilation environment. CPUs can also make operations visible to other CPUs in an order different from a reader's intuitive source-code sequence, subject to architecture rules. Cache coherence constrains how CPUs observe writes to a location, but it is not by itself a complete ordering rule for a protocol involving several locations. “The cache is coherent” does not prove that seeing a flag implies seeing an earlier initialization of a different object.

A data race involves concurrent conflicting accesses without the synchronization required by the relevant model. The ISO C userspace memory model and LKMM are related areas of reasoning, not interchangeable specifications. C11 atomics in an ordinary application have C-language semantics and are implemented by its compiler and libraries. Kernel concurrency uses kernel primitives, documented compiler constraints and LKMM rules, including marked accesses and kernel-specific dependencies. An experiment that happens to work on one x86 machine does not prove it is permitted on every architecture or that compiler transformations are harmless.

READ_ONCE and WRITE_ONCE mark accesses with particular compiler and access semantics; they are useful building blocks but do not by themselves form a general release/acquire synchronization pair for multiple variables. Their guarantees have size, alignment and type considerations documented by the kernel. Ordinary volatile is not a general lock, reference count or inter-CPU publication protocol. For a shared structure, first identify the invariant and writers, then decide which lock, atomic or publication mechanism provides the necessary ordering and lifetime.

Source walkthrough#

In v6.12 include/linux/rcupdate.h, the non-constant-null branch of rcu_assign_pointer uses this statement (file SPDX GPL-2.0+; other macro branches and checks omitted):

smp_store_release(&p, RCU_INITIALIZER((typeof(p))_r_a_p__v));

The release store publishes the pointer after the writer's prior initialization under the documented RCU rules. The macro also has a special constant-NULL branch and sparse/type annotations; this isolated line is not a complete implementation. A reader must use the matching RCU dereference and read-side protection appropriate to its context. Publishing a pointer does not by itself serialize later mutations of all fields or grant a reference that remains valid after the read-side protection ends.

In v6.12 include/linux/refcount.h, the public refcount_inc_not_zero wrapper contains this excerpt (SPDX GPL-2.0; declaration and surrounding documentation omitted):

return __refcount_inc_not_zero(r, NULL);

It delegates to the internal increment-if-nonzero operation and returns whether it succeeded. The associated documentation says that this operation does not by itself provide general memory ordering and assumes that the caller has already made the object memory stable, for example with an appropriate RCU protocol. A successful increment can establish a reference only under the object's documented lifetime scheme; it cannot safely resurrect freed memory merely because a pointer value still exists. refcount_t adds saturation and misuse checks intended for reference counts; an arbitrary atomic_t counter is not automatically an equivalent lifetime design.

Atomicity and ordering are separate#

An atomic read-modify-write, or RMW, combines a read and update so competing operations on that atomic location cannot interleave as independent reads and writes. cmpxchg conditionally replaces a value only if it matches the expected value and reports the prior result according to its interface. This atomicity says something about that location; the ordering of other loads and stores depends on the chosen variant and whether the operation succeeds. A lock can provide atomic exclusion around a larger invariant; a single atomic increment cannot automatically make two fields change together.

The v6.12 LKMM ordering guide distinguishes unordered, relaxed, acquire, release and fully ordered operations. A relaxed operation can provide atomicity for its target without supplying the additional ordering needed for a publication protocol. Acquire orders relevant later accesses after the acquire; release orders relevant prior accesses before the release. An unsuffixed value-returning RMW such as the documented cmpxchg generally has full ordering on success under LKMM rules. These descriptions must be read for the exact primitive and operation, not mechanically inferred for every API with a similar name.

A failed conditional RMW is especially important: the ordering guide states that conditional RMW operations are guaranteed to provide their specified ordering only when they succeed. If atomic_cmpxchg_acquire fails, do not assume that failure provided an acquire edge for a subsequent object access. Handle the returned value, retry or use another documented synchronization operation. Likewise, atomic_read or atomic_set is not a full barrier merely because the storage has an atomic type. A counter used for statistics may need different semantics from a reference count that decides when memory can be freed.

smp_mb() provides a full SMP memory barrier under its contract; smp_rmb() and smp_wmb() provide narrower read/read and write/write ordering, respectively. smp_mb__before_atomic() and smp_mb__after_atomic() can augment a suitable unordered atomic RMW, but the ordering guide explains their placement and operation constraints. A barrier is not a lock, does not wait for a device to finish DMA, and cannot repair a use-after-free. Lock acquire/release has its own ordering rules and must not be casually treated as a universal full barrier across unrelated locks. Address, data and control dependencies have specific LKMM meanings; a control dependency is not a general replacement for an acquire, especially for later loads.

CPU shared-memory barriers also differ from device ordering. MMIO accessors, posted-write completion and DMA synchronization have architecture and device-specific contracts. A CPU's smp_wmb alone does not automatically flush a device's caches or provide the ordering required by an IOMMU-backed DMA mapping. Chapter 06 introduced those boundaries; follow the DMA and device-I/O APIs for the actual driver.

A two-CPU message-passing litmus#

The upstream v6.12 MP+poonceonces.litmus describes an initially zero buffer and flag. In source order, CPU 0 writes buf=1 and then flag=1 using marked writes. CPU 1 reads flag and then buf using marked reads. The test's documented result for the outcome flag read = 1, buf read = 0 is Sometimes under LKMM. The outcome is counterintuitive because seeing the flag does not, with only those accesses, establish the cross-location ordering needed to force observation of the earlier data write. This is a model result documented by the upstream test, not a runtime observation made here.

A stronger publication design initializes the object and performs a release publication; a reader that obtains the published value through the corresponding acquire or documented RCU dereference can then rely on the required ordering for the initialized data, subject to the protocol and lifetime assumptions. The upstream RCU message-passing litmus documents the analogous bad outcome—reader obtains the newly published pointer yet reads the pre-initialization value—as Never. It uses rcu_assign_pointer, rcu_dereference, a read-side critical section and marked data accesses. Its particular pointer-dependency and RCU semantics matter; it is not a proof that any arbitrary pair of pointer loads is an acquire.

Write the state explicitly. Initially the old object is reachable and the new object's data is 0. The writer initializes the new object's data to 1, then publishes its address. In the documented RCU test, if the reader's dereference obtains the new address and accesses the data as specified, observing 0 is forbidden by the model. A reader that obtains the old pointer is a different case and may see the old object's state. Freeing either object is another question: publication ordering does not replace a grace period or reference ownership. The following chapter develops that lifetime pipeline.

As a counterexample, replacing the release/dereference protocol with plain marked stores and loads can permit the message-passing outcome noted by the first litmus. READ_ONCE(flag)==1 is not a reference to a particular version of every other field. Reusing the same flag for multiple generations without a generation or lock protocol can introduce further ambiguity even with ordering. A model test has specified initial state, operations and an exists outcome; adding an unrelated operation changes the test and requires a new analysis.

References and successful acquisition#

Suppose a shared pointer names object O with reference count 1, held by a registry. A reader enters the appropriate RCU read-side critical section and obtains O through the documented dereference. While the object memory is stable, it attempts refcount_inc_not_zero. If the increment succeeds and the subsystem's protocol makes that reference valid, the reader can leave the read-side section and later drop its reference. If the count has reached zero, the increment fails and the reader must not use O as an owned object. A writer removes the registry's publication and eventually drops its own reference; freeing still requires the correct combination of reference-count and RCU rules.

This is conditional reasoning, not a general recipe for every cache. With SLAB_TYPESAFE_BY_RCU, the memory location can be reused for a new object while an RCU reader still has an address. Stable storage is not stable identity: a reader may need acquire ordering and revalidation of a key or generation after taking a nonzero reference. Check the exact allocation, initialization and lookup protocol. A stale pointer saved outside protection cannot be rescued safely by incrementing its old-looking counter.

A failed compare is a different event#

Suppose a reader loads a candidate pointer, then tries cmpxchg_acquire to transition a state from READY to CLAIMED. If the comparison succeeds, the specified acquire ordering applies on that success path; the reader must still verify that this state machine and its reference protocol make the pointed-to object valid. If another CPU changes the state first, the compare can fail and return the observed value without granting the assumed acquire ordering. Following the failure with reads of the candidate's initialized fields would require a separate valid ordering and lifetime argument. A retry loop must also handle teardown and the possibility that the object or generation changed; spinning forever on a dead object is not a progress strategy.

In include/linux/atomic/atomic-arch-fallback.h, a generated v6.12 file with SPDX GPL-2.0, the documentation and signature of raw_atomic_cmpxchg_acquire read as follows (the function body is omitted):

/**
 * raw_atomic_cmpxchg_acquire() - atomic compare and exchange with acquire ordering
 * @v: pointer to atomic_t
 * @old: int value to compare with
 * @new: int value to assign
 *
 * If (@v == @old), atomically updates @v to @new with acquire ordering.
 * Otherwise, @v is not modified and relaxed ordering is provided.
 *
 * Safe to use in noinstr code; prefer atomic_cmpxchg_acquire() elsewhere.
 *
 * Return: The original value of @v.
 */
static __always_inline int
raw_atomic_cmpxchg_acquire(atomic_t *v, int old, int new)

The comment says that a failed compare leaves @v unmodified and provides relaxed ordering, so acquire ordering is promised only on success, which is the failure caveat above. The body, not shown, chooses an architecture acquire variant or a relaxed operation followed by an acquire fence. The comment does not make a failed read of a candidate object safe, and it does not establish that object's lifetime.

A memory barrier only constrains ordering for the accesses covered by its contract. It does not make an unmarked compound data structure update indivisible, validate a pointer, or extend an allocation's lifetime. If a protocol needs both publication and a stable reference, prove both properties explicitly. The model's allowed outcomes are about the stated events and relations; they do not predict the frequency of an outcome on a given CPU.

Practice#

Source-only route: produce a table with initial values, CPU 0's two writes, CPU 1's two reads and the outcome asked by the upstream MP+poonceonces test. Mark flag=1, buf=0 as documented Sometimes for that test. In a separate row, record the RCU test's newly published pointer and old data outcome as documented Never, and list the RCU primitives and lifetime assumptions. Do not turn this table into a claim that every release/acquire protocol or every architecture has been tested.

If an existing v6.12 checkout and an already installed compatible herd7 are available, the LKMM tools can run its litmus tests using their documented configuration. Record the exact tool version and command, and distinguish model execution from hardware execution. No installation is required or assumed here. If the tool is missing, the source-only table is the complete exercise; copying the documented expected result is not a claim that you ran herd7.

Own-process observation: with a C11 compiler, create a private directory and run this synthetic program. It reports whether one C11 atomic object is lock-free under the host compiler/runtime; it does not run two threads or test LKMM ordering.

lab_dir=$(mktemp -d) || exit 1
cd "$lab_dir" || exit 1
cat > atom.c <<'C'
#include <stdatomic.h>
#include <stdio.h>
int main(void) {
    atomic_int value = ATOMIC_VAR_INIT(0);
    atomic_store_explicit(&value, 1, memory_order_release);
    int result = atomic_load_explicit(&value, memory_order_acquire);
    printf("value=%d lock_free=%d\n", result, atomic_is_lock_free(&value));
    return result != 1;
}
C
cc -std=c11 -Wall -Wextra -Werror -o atom atom.c || exit 1
./atom

Expect value=1; lock_free depends on the implementation. Both operations execute in one thread, so the result says nothing about a second CPU, kernel barriers or an RCU grace period. If compilation or linking fails, record compiler and diagnostic rather than changing kernel settings. Deliver the source-only outcome table and this observation if performed. Remove only the created files:

rm atom atom.c
cd /tmp
rmdir "$lab_dir"

Knowledge check#

1. Why does coherent caching of flag not by itself order an earlier write to buf for another CPU?
2. What outcome does the upstream marked-access message-passing litmus document as sometimes allowed?
3. Why is a failed atomic_cmpxchg_acquire not a reliable acquire operation under the cited LKMM rules?
4. What prerequisite does refcount_inc_not_zero have before it can safely inspect an object?
5. Why doesn't an smp_wmb alone establish a device's DMA completion?

Answers#

1. Coherence constrains observations of individual locations but does not supply the necessary cross-location publication edge. Compiler and CPU ordering must be provided by the relevant protocol.
2. The reader can see flag=1 while still seeing buf=0 in that test. The result is a model statement for its specified operations, not a measured event on every machine.
3. The documentation guarantees conditional RMW ordering only on success. Failure must be handled without assuming that acquire ordering was obtained.
4. The caller must already make the object memory stable and follow the subsystem's identity and reference protocol. A stale freed pointer cannot be safely rescued by the increment.
5. CPU memory ordering is separate from device completion, DMA ownership and cache synchronization. The relevant DMA and device APIs must establish those properties.

Checkpoint#

Deliver the two litmus outcome table and, if run, the labeled single-thread C11 result. Pass when you explain the release/dereference edge, the failed-RMW caveat, and a separate lifetime requirement. If uncertain, reread the cited ordering guide and the exact upstream litmus before revising the table. This is not certification.

Sources#

• v6.12 RCU publication macro.
• v6.12 refcount interfaces.
• v6.12 LKMM ordering guide.
• v6.12 marked message passing and RCU message passing.
• Reference counts versus atomics.