KASAN, KCSAN, UBSAN, kmemleak, and fault injection

Before you start#

Review object lifetime, allocation, locking, and crash evidence. Linux v6.12 is the source baseline. Sanitizers depend on architecture, compiler, configuration and available memory. No host fault-injection, crash, or sanitizer-triggering experiment is performed here.

Learning goals#

Differentiate memory, race and undefined-behavior detectors; trace a KASAN report and a fault-injection decision; and design a bounded failure-path test without mistaking absence of a report for correctness.

Source walkthrough#

KASAN instruments or checks memory accesses to detect selected classes of invalid access, such as out-of-bounds accesses and use-after-free. Generic KASAN uses shadow memory; software tag-based KASAN uses tags on supported configurations; hardware tag-based KASAN relies on supported arm64 hardware and configuration. These modes have different coverage, precision, compiler and memory overhead. A report identifies an observed access and available allocation/free context, not necessarily the earliest bug that made the access possible. Hardware tag checks can be synchronous or asynchronous under supported settings; asynchronous reporting can provide less precise attribution.

In mm/kasan/report.c, kasan_report saves user-access state, checks whether reporting is suppressed or disabled, and otherwise enters its reporting section. This exact v6.12 SPDX GPL-2.0 excerpt omits the earlier setup and later report printing:

	info.type = KASAN_REPORT_ACCESS;
	info.access_addr = addr;
	info.access_size = size;
	info.is_write = is_write;
	info.ip = ip;

The function classifies this report as an access and copies the reported address, access size, read/write flag and instruction pointer into a kasan_report_info. Subsequent code completes and prints the report and ends reporting; the common exit restores saved user-access state. If reporting is suppressed, it branches to that restore path without printing. The reporting implementation must respect its locking and interrupt context; these fields are evidence for one report, not proof of every allocation's correctness.

Read a synthetic use-after-free#

The following is invented teaching data, not output from a running kernel. Function names, addresses and offsets are fictional:

BUG: KASAN: use-after-free in demo_read+0x10/0x60
Read of size 4 by task demo-worker/410
Allocated by task demo-open/400: demo_alloc
Freed by task demo-close/401: demo_free

The claimed invalid access is a four-byte read in a worker after an object was freed. Allocation and free traces, if genuinely captured and symbolized against the matching build, would identify where the object came from and where its lifetime ended. They would not by themselves show why the worker retained a pointer, whether the pointer was protected by a reference or RCU read-side section, or which thread first violated the ownership contract. The apparent free stack may also reflect allocator reuse and instrumentation limits; inspect the complete actual report and matching source.

A plausible synthetic sequence is: task A allocates object O and queues work containing a pointer to O; task B tears down the owning object and frees O without synchronizing with the queued work; worker C later reads a field of O. A correct repair might hold a reference through work completion, cancel and synchronize the work before freeing, or use a correctly designed RCU lifetime. Which is appropriate depends on the actual object, context and ownership. Merely adding READ_ONCE to the field read does not keep O allocated or prevent the worker from using a freed pointer. Conversely, blindly adding a global lock may deadlock or sleep in an invalid context; prove the lifecycle and lock ordering first.

KCSAN and the memory model#

KCSAN is a sampled dynamic data-race detector. It uses compiler instrumentation and watchpoint-style sampling with delays to expose conflicting accesses. Its reports are evidence that the instrumented execution encountered a potentially conflicting pattern under its rules, not an exhaustive proof about all interleavings. Sampling, watchpoint capacity, filtering, annotations and configuration affect what can be seen. A run without a report cannot prove a race-free design. Consult the KCSAN documentation and the Linux memory model.

Consider another synthetic sequence: one CPU writes a pointer to a newly initialized object while another reads it without a synchronization protocol. A reported conflicting access is a reason to establish publication ordering and lifetime, not automatically to silence the report. READ_ONCE and WRITE_ONCE constrain particular compiler transformations and access properties; they are not universal release/acquire ordering, mutual exclusion, or object reclamation. Depending on the intended invariant, appropriate locks, RCU publication, reference counts, atomics or explicit barriers may be needed. A documented benign race still requires a defensible invariant and appropriate annotation, not indiscriminate suppression.

Lockdep checks lock dependencies and selected locking rules under observed execution; it can reveal an inversion before the exact deadlock occurs. It is not a use-after-free detector, and a clean lockdep run does not establish that an unlocked object's lifetime is safe. PREEMPT_RT can change the semantics and implementation of some locks and execution contexts, so a fix must be checked against the relevant configuration rather than assuming every spinlock_t behaves identically on every build.

Other tools answer different questions#

UBSAN instruments selected undefined-behavior checks, such as supported arithmetic or bounds violations, with coverage and recovery dependent on build options. KMSAN tracks uninitialized memory use through shadow/origin metadata where supported; it has substantial constraints and overhead and is not interchangeable with KASAN. KFENCE samples allocations into guarded regions with lower overhead and catches a subset of heap errors; sampling means many allocations are never protected. kmemleak scans for apparently unreachable allocated objects, subject to scanning limitations and false positives/negatives; an unreachable allocation is a leak candidate, while a reachable but logically leaked object may not be reported.

These tools can affect layout, timing, memory consumption and scheduling, changing whether a race appears. Compiler version, optimization, architecture, kernel configuration and supported instrumentation affect coverage. A KASAN report on one build and a KCSAN report on another may concern the same underlying invariant, but that relationship needs a reproducer and source evidence. Do not declare a report false merely because it disappears with instrumentation disabled; equally, investigate documented instrumentation limitations before treating every report as a confirmed exploitable defect.

Fault injection and cleanup invariants#

Fault injection deliberately makes selected operations fail so error paths can be exercised. failslab targets eligible slab allocations; fail_page_alloc targets eligible page allocations. Filtering by task, stack, allocation flags, probability, interval, space and number of failures controls which paths may fail. These are not promises that exactly one arbitrary allocation in an entire system will fail. Availability and controls depend on configuration and the particular injector; writing fault-injection controls on a shared host could destabilize unrelated work.

In lib/fault-inject.c, should_fail_ex has an optional per-task fail_nth path before its normal attribute checks. This exact v6.12 GPL-2.0-only excerpt omits that earlier path and later stack, interval and probability checks:

	if (attr->task_filter && !fail_task(attr, current))
		return false;

	if (atomic_read(&attr->times) == 0)
		return false;

When the task filter is enabled and the current task does not meet it, this normal path declines to inject a failure. If the configured remaining count is zero, it also declines. The earlier nonzero fail_nth branch has separate handling and must not be ignored when interpreting the result. Later checks include stack filtering, space, interval and a random probability decision; on the failure path the implementation optionally logs and decrements the times counter unless it is configured for unlimited failures. Individual attribute accesses and atomic operations do not imply that every combined configuration update is an atomic transaction.

mm/failslab.c illustrates exclusions: the bootstrap cache and allocations marked __GFP_NOFAIL are not failed through that path, and configurable filters can exclude other allocations. A test that requests failure but receives success may have missed the intended call, hit an exclusion or exhausted its configured count. Record configuration, task and observed return rather than assuming injection occurred.

The return value is where an injected failure becomes visible to a caller. In mm/failslab.c, v6.12, SPDX GPL-2.0 (the earlier exclusion checks and the rest of the file are omitted):

	if (gfpflags & __GFP_NOWARN)
		flags |= FAULT_NOWARN;

	return should_fail_ex(&failslab.attr, s->object_size, flags) ? -ENOMEM : 0;

The -ENOMEM return is the error the caller must unwind from; a 0 return means the allocation proceeds normally. The excerpt does not show which attribute checks inside should_fail_ex allowed the failure, and it does not show that the caller released the resources it already held.

A concrete allocation-unwind case: function F allocates object A, obtains a reference to B, then allocates C. If the third step returns -ENOMEM, a correct error path releases B and frees A exactly once, without publishing a half-initialized object. If work was already queued or hardware was started, cancellation and synchronization may be needed before releasing those resources. The exact cleanup order comes from F's real ownership and callback contract. A test should assert the returned error and observable absence of leaks or dangling callbacks; injecting failure alone does not prove cleanup was correct.

Bounded guest design and recovery#

A meaningful fault-injection run uses a dedicated disposable guest, a known v6.12 configuration and compatible toolchain, a narrow test process and a finite failure count. Prepare a throwaway image and a snapshot or reproducible rebuild, record the debugfs control values before the test, set a task filter and bounded interval/count only within that guest, run one identified test, restore the controls or discard the guest, and inspect both the expected failure and cleanup. A crash or nonresponsive guest must be terminated and recreated from the clean image; never rely on a shared host as the recovery target. Exact injector files and availability must be checked against the guest's configuration and version before writing them.

This is a test design, not a claim that a guest image exists or that failure was injected. The chapter 08 image has not been booted here because of the GCC 16/C23 incompatibility. Do not write host debugfs fault-injection controls, trigger host SysRq or use broad random failure settings on a machine that shares a kernel with unrelated work. A container shares its host kernel and is not a disposable kernel boundary for crash or fuzz tests.

Separating a race report from a repair#

Suppose a synthetic KCSAN report names a writer storing a new pointer and a reader loading it on another CPU. First identify whether the pointer itself, the pointed-to fields, or both are shared. If the writer initializes fields and then publishes the pointer, the reader must observe the required initialization ordering. If the writer later removes and frees the object, ordering alone is insufficient: a reader can still hold the old pointer. A release/acquire pair can address a publication relationship under its rules, while a reference count, lock or RCU grace-period design can address lifetime. The right design depends on how readers obtain and release the object and whether they can sleep.

A test of the repair should arrange the relevant concurrent paths in a disposable guest, run the same reproducer before and after the change, and include teardown and allocation-failure cases. If KCSAN no longer reports the race, record the trial count and configuration; sampling can miss the old interleaving. Check that the proposed fix does not introduce a lock-order inversion or use a sleeping operation in atomic context. A regression assertion on object state, combined with source review and a sanitizer run, supplies stronger evidence than a report's disappearance alone.

Fault injection has similar attribution challenges. If a function makes three allocations and an interval setting causes a later eligible allocation to fail, an observed -ENOMEM does not identify which allocation failed without additional evidence. An earlier per-task fail_nth setting, cache exclusion, task filter or remaining-failure count can change the outcome. Instrument only the isolated test environment, verify the expected call and error, and assert that every successfully acquired resource was released. A failure-injection configuration that was never reached tests no unwind path.

Sanitizer reports themselves must be preserved with their full context. Allocation and free stacks can be missing because the mode or configuration does not retain them; report suppression can reduce repeated messages; a prior corruption can alter later evidence. Record the first relevant report, kernel and compiler identity, configuration, workload and whether the kernel continued. Avoid hiding warnings as a way to obtain a clean test log. A missing stack or uncertain origin is an explicit investigation limit.

Practice#

Read-only configuration inventory

On your own authorized host or disposable guest, inspect the running kernel identity and any exposed configuration without changing it:

uname -r
if [ -r /proc/config.gz ] && command -v gzip >/dev/null 2>&1; then
  gzip -cd /proc/config.gz | grep -E '^CONFIG_(KASAN|KCSAN|UBSAN|KMSAN|KFENCE|DEBUG_KMEMLEAK|FAULT_INJECTION)=' || true
elif [ -r "/boot/config-$(uname -r)" ]; then
  grep -E '^CONFIG_(KASAN|KCSAN|UBSAN|KMSAN|KFENCE|DEBUG_KMEMLEAK|FAULT_INJECTION)=' "/boot/config-$(uname -r)" || true
else echo 'running-kernel configuration not available at these paths'; fi

A missing config, option, permission or tool is a valid observation. A configuration file in /boot should be matched to the running build; it is not proof that every instrumented path ran or that no report was suppressed. Do not write debugfs, trigger a warning or allocate deliberately invalid kernel memory.

Source-only alternative: use the invented worker/free sequence and make a reference timeline, then evaluate the quoted fault decision with task filtering enabled and a nonmatching task, followed by a zero remaining count. Explain the separate fail_nth path and list A/B/C cleanup after a simulated -ENOMEM. Deliver this table or the actual configuration inventory, clearly marking the fictional report and unrun injection.

Knowledge check#

1. Why does adding READ_ONCE not repair the synthetic use-after-free?
2. Why can a clean KCSAN or KFENCE run not prove the absence of races or heap errors?
3. What does should_fail_ex do when the task filter rejects the current task on the normal path?
4. Why must an allocation failure after acquiring B release B and A?
5. Does lockdep replace KASAN or an object-lifetime argument?

Answers#

1. It does not keep the pointed-to object allocated or synchronize worker teardown. The ownership and cancellation or reference protocol must be correct.
2. Both depend on execution and sampling or instrumentation coverage. A missed interleaving or unsampled allocation can remain unreported.
3. It returns false before later normal checks; the earlier fail_nth branch is separate and must also be considered.
4. The function owns those acquired resources on the error path. Returning without releasing them leaks references or objects; freeing them before asynchronous users stop can cause a lifetime bug.
5. No. Lockdep reasons about observed lock dependencies and rules; it does not establish safe reclamation or detect every invalid memory access.

Checkpoint#

Submit a synthetic object-lifetime timeline and a bounded fault-decision and cleanup table. Ready means the report is labeled fictional, all acquired resources have an owner on failure, and a missing sanitizer report is not called proof. Retry with the allocation and teardown order if READ_ONCE is treated as reclamation. This is not certification.

Sources#

• KASAN report v6.12
• Fault injection decision v6.12
• failslab v6.12
• KASAN documentation
• KCSAN documentation