Evidence-based cross-subsystem capstones

Before you start#

Review workqueues, memory lifetime, durability, TCP, performance investigations, and patch review. Linux v6.12 is the historical source baseline. The artifacts and numerical observations below are explicitly synthetic; no guest, hardware, crash or network impairment test is claimed.

Learning goals#

Connect source-level invariants to falsifiable hypotheses, construct bounded tests and recovery plans, compare alternatives across CPU/memory, storage and network/security, and distinguish a source-reading exercise from independently reviewed engineering evidence.

Quick vocabulary: a work item is a request for a function to run asynchronously; a worker is the kernel thread that runs it. A folio is a unit of cached file data. NAPI is Linux's network receive polling mechanism. A storage flush asks the lower device stack to make earlier writes durable; the actual guarantee depends on the device and backend contract.

Source walkthrough#

A capstone is an investigation with a defined question, evidence and limits, not a claim that one person has mastered an entire subsystem. Begin by identifying the operation, objects, context and ownership at each boundary. Preserve exact build and workload identity, distinguish measured data from source-derived expectations, and write down what would disprove the leading hypothesis. A plausible stack or counter correlation is a starting point, not a diagnosis.

In kernel/workqueue.c, queue_work_on manipulates the pending bit under local interrupt protection. This exact v6.12 GPL-2.0-only excerpt, copyright Ingo Molnar and contributors, omits surrounding declarations and return:

	if (!test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work)) &&
	    !clear_pending_if_disabled(work)) {
		__queue_work(cpu, wq, work);
		ret = true;
	}

The first condition attempts to set the work item's pending bit; the next checks whether disabled work clears that pending state. Only when both conditions permit it does the function queue the work and report true. The complete function saves and restores local interrupt state. A false result does not automatically mean that an owning object can be freed: work might already be pending, disabled or require a separate synchronization protocol. Queue state and object lifetime are related but different invariants.

In fs/ext4/fsync.c, ext4_sync_file contains this exact v6.12 SPDX GPL-2.0 excerpt in its journaled path; surrounding error and journal code are omitted:

	ret = file_write_and_wait_range(file, start, end);
	if (ret)
		goto out;

The first statement initiates or waits for relevant data writeback and saves the result. A nonzero result takes the exit path instead of being treated as successful data synchronization. The caller's file reference, inode/mapping lifetime, journal operation, optional flush and writeback-error accounting all matter to the complete result. This branch constrains a durability investigation but does not prove that a particular device completed a flush.

Capstone A: workqueue, CPU and memory lifetime#

Synthetic initial artifact: a fictional driver allocates object O, queues work W referring to O, then receives a remove request. A made-up report says “read of freed object O in worker W”; a timeline records queue at 10 ms, remove start at 12 ms, free at 13 ms and worker field read at 14 ms on another CPU. All times share an assumed monotonic clock. The report and objects are invented teaching data, not a KASAN report from this workspace.

The observed synthetic order supports a use-after-free hypothesis if the pointer in W is the same O and the timestamps and symbolization are reliable. It does not establish whether the queue operation succeeded, whether O was already referenced, or whether the recorded free actually reclaimed the relevant allocation. An alternative is a stale or mis-symbolized report, object reuse or an unrelated pointer with the same numeric address. The quoted pending-bit logic shows why a false queue return cannot by itself establish that no earlier work is outstanding.

Trace the actual target's allocation, work initialization, queue call, callback and remove path. Record which context can sleep, which lock protects publication and which reference keeps O alive. cancel_work_sync or a workqueue flush can be relevant if the work's contract permits it, but the exact call and locking must be checked: waiting while holding a lock needed by the worker can deadlock. RCU can delay reclamation for appropriate readers but does not automatically cancel queued work. The fix must establish either a retained reference through the final callback or a teardown protocol that prevents new queueing, synchronizes existing work and only then frees O.

In kernel/workqueue.c, v6.12, SPDX GPL-2.0-only, cancel_work_sync calls this helper with cflags of 0; the complete function is:

static bool __cancel_work_sync(struct work_struct *work, u32 cflags)
{
	bool ret;

	ret = __cancel_work(work, cflags | WORK_CANCEL_DISABLE);

	if (*work_data_bits(work) & WORK_OFFQ_BH)
		WARN_ON_ONCE(in_hardirq());
	else
		might_sleep();

	/*
	 * Skip __flush_work() during early boot when we know that @work isn't
	 * executing. This allows canceling during early boot.
	 */
	if (wq_online)
		__flush_work(work, true);

	if (!(cflags & WORK_CANCEL_DISABLE))
		enable_work(work);

	return ret;
}

Cancellation sets the disable bit while it cancels, flushes a running instance when the workqueue is online, and then re-enables the work because the flag is clear. The flush can sleep, so a caller must not hold a lock that the worker needs. The excerpt does not show who frees the object, so it does not by itself resolve the synthetic report.

A bounded reproduction belongs in a dedicated disposable guest built from a known source and compatible compiler, with a narrowly scoped trusted test and KASAN or KCSAN as appropriate. Allocate a throwaway guest image and snapshot, cap iterations and elapsed time, record configuration and console output, and terminate/recreate the guest if it hangs. Do not use the host, a shared-kernel container or a physical device. Compare the same test before and after a proposed fix; include normal queue completion, queue rejected/already pending, remove before queue, repeated remove and allocation failure. Confirm the work callback cannot access freed state and no cancellation deadlock appears. This is a test design, not an executed reproducer.

A reviewable artifact is a timeline with each reference and lock owner, the failing and corrected state transitions, test output and independent reviewer objections. Falsify the stale-work hypothesis if a reliable trace shows W completed and its reference was released before O was freed, while the same object generation is identified. If no trace establishes identity or ordering, leave the cause unresolved rather than adding a defensive lock without an invariant.

Capstone B: storage durability and recovery#

Synthetic initial artifact: an application writes 4 KiB, calls fsync, receives success and then reports that data is absent after a power-loss test. The test log claims the call completed in 8 ms but does not show a directory fsync, filesystem configuration, device flush behavior or the capture method. The numbers and log are invented. A successful file fsync can have a different contract from persisting a newly created directory entry; the application protocol and filesystem semantics must be examined before blaming ext4.

Trace the file descriptor through do_fsync, VFS dispatch, ext4_sync_file, journal and block-layer completion for the selected configuration. Identify whether the file was new, renamed or replaced, whether the relevant directory was synchronized, and whether the device honors the ordering and flush operations requested. A file reference protects the open file during the syscall; folios, journal transactions and block requests have separate lifetimes. A writeback error can be reported later through the file's error accounting. The quoted branch shows one error cannot simply be ignored on the journaled path, but does not prove the synthetic application called that exact path.

Leading hypotheses include an incomplete application durability protocol, a filesystem or block-layer error, a device or virtual backend that does not honor flush semantics, and a flawed crash-test harness that inspected the wrong image. Counterevidence to the first hypothesis would be a verified file and directory synchronization sequence with all return values checked. Counterevidence to a flush hypothesis would include correctly correlated flush completion and a device contract known to preserve the data, while acknowledging that an incomplete capture can still hide failure. A single successful fsync return is not sufficient to rank those hypotheses.

A real crash-consistency reproduction must use a disposable guest and throwaway image, with an independently saved clean copy, bounded operations, known virtual storage cache settings, an explicit stop/crash mechanism confined to that guest and a recovery procedure to restart from the test image. Record exact kernel, filesystem, mount options and application operations; never power-cycle or corrupt a host device. Test before and after a minimal proposed change, with file-only and file-plus-directory synchronization controls, injected error handling where the selected guest supports it, and repeated trials. Verify that the crash harness actually interrupted the intended interval. These are design requirements, not a command to run on the host.

A candidate fix might correct the application's directory synchronization protocol if evidence proves that omission, or correct a demonstrated error-propagation bug in a specific filesystem path. It must preserve existing API and durability semantics. The artifact is a traceable operation log, error table, recovery images or hashes where appropriate, source mapping and a regression matrix. A reviewer should be able to reject the proposed fix if the failure persists under a fully synchronized protocol or if the harness inspected a stale image.

Capstone C: network, driver and security boundaries#

Synthetic initial artifact: a TCP service reports a 120 ms interval after receiving a request and before responding. Its worker used 15 ms CPU, the application queue depth was eight, and an unrelated firewall counter rose by one during the same second. These numbers are invented. The remaining wall time may include scheduling, a lock, application queueing or another wait; the firewall counter alone does not identify this connection or show packet loss.

Trace a representative packet from a supported network device through receive queues and NAPI, routing and TCP socket state to an application receive, then the server's work queue and response. inet_csk_accept in inet_connection_sock.c handles a listening socket's accept queue under its socket lock; that queue differs from a child socket's receive queue and an application's own jobs. Security hooks or netfilter policy can deny operations or packets at distinct points. A counter scoped to a broad rule may include other connections. The driver, socket and application each own different objects and callbacks; removal or close can race with in-flight work.

Hypothesis one is application queue or lock contention after complete receive. Hypothesis two is delayed network delivery before receive. The supplied timestamps place the described 120 ms after receive, so a network hypothesis must explain a separate unmeasured interval or challenge the timestamp definition. A third hypothesis is CPU quota or host scheduling. A controlled dedicated guest test can tag requests, record client and server timestamps with a known clock relationship, observe queue transitions and relevant loss or retransmission evidence, and compare low-load with a fixed higher-load run. It must avoid changing host networking, injecting loss on a shared network, or disabling firewall policy.

A bounded candidate change could reduce a demonstrated application lock bottleneck or fix a verified driver lifetime error. Regression tests should cover partial reads, concurrent connections, timeouts, close during work, policy-denied traffic and resource pressure. Preserve latency distributions, loss and observer overhead; do not infer that raising an accept backlog fixes a post-receive application queue. The artifact is a correlation table, competing explanations, falsification criteria and a proposed change reviewed against security and lifetime invariants. A reviewer can reject the hypothesis if matched requests execute immediately and the measured delay precedes receive.

One complete worked solution#

For capstone A's supplied sequence, the minimum source-consistent solution is: O is created before 10 ms; W is queued at 10 ms; a remove path frees O at 13 ms; W reads it at 14 ms. Under the stated same-object and reliable-timing assumptions, the read is after free by 1 ms. The failed invariant is that O must remain allocated while W may dereference it. The two most direct repair families are to hold a reference until W finishes or to stop new submissions and synchronously cancel/flush W before the final free. Neither is automatically correct without checking callback locking and who owns the last reference.

The hypothesis is falsified by evidence that the 14 ms read refers to a different object generation or that the timeline is wrong; a successful queue call alone does not settle it. Before the fix, a bounded test must reproduce the invalid access or an equivalent ownership assertion. After the fix, repeat the same test, normal operation and error/teardown controls under relevant configurations. Measure whether synchronization adds unacceptable latency, and request independent review of deadlock and lifetime reasoning. Because these artifacts have not been produced, this lesson solves the synthetic reasoning exercise only; it does not claim a real kernel bug was fixed.

Practice#

Safe artifact calculation

Run this userspace calculation; it performs no kernel fault or device operation:

python3 - <<'PY'
queued, removed, freed, read = 10, 12, 13, 14
print('synthetic read after free:', read - freed, 'ms')
print('queue-to-read:', read - queued, 'ms; removal-to-free:', freed - removed, 'ms')
PY

The expected first result is 1 ms. That arithmetic is valid only under the stated same-clock and same-object assumptions. A missing Python interpreter is not evidence about the hypothetical bug. The program does not recreate a kernel race.

Source-only alternative: use the four supplied timestamps and the two quoted source branches to construct the same ownership and error tables by hand. Choose one of the other two capstones and write one falsifiable alternative and one safety boundary. Deliver an evidence packet, not a claim of executed guest tests.

Knowledge check#

1. Why can a false queue_work_on return not prove the owning object is free of outstanding work?
2. What invariant is violated in the synthetic workqueue timeline?
3. Why does a successful file fsync not answer every directory-entry durability question?
4. Why does an unrelated firewall counter increase not prove this TCP request was dropped?
5. What additional evidence distinguishes source-only understanding from a reviewed engineering fix?

Answers#

1. Work can already be pending or disabled, and the caller needs the actual workqueue and lifetime contract. The return value alone does not synchronize all earlier activity.
2. The object is freed at 13 ms while work that dereferences it runs at 14 ms, under the supplied identity and clock assumptions.
3. Application operations and directory metadata have their own durability requirements; the filesystem and storage contract must be examined and all relevant errors checked.
4. The counter can include other traffic and has a wider time and rule scope. It is not correlated with the request or its socket.
5. A real reproducer, before/after and negative-control results, exact build and environment, cleanup and risk analysis, and independent review are needed for a bounded fix claim.

Checkpoint#

Submit one complete capstone report with source mapping, ownership and context, two hypotheses, falsification criteria, isolated test/recovery, before/after controls and review questions. Ready means the worked synthetic solution is correct and no unrun result is presented as evidence. Remediate missing ownership or causal claims by returning to the relevant source path and repeating the analysis. Progress is not certification.

Sources#

• Workqueue source v6.12
• ext4 fsync v6.12
• TCP accept v6.12
• Workqueue documentation
• Kernel testing overview