End-to-end storage and network investigations

Before you start#

Review ext4 and journaling, block I/O completion, TCP, NAPI, pressure and cgroups, and experimental design. The source baseline is upstream Linux v6.12. All incident timings and observations below are explicitly synthetic, not measurements from this workspace.

Learning goals#

Construct two competing-hypothesis investigations across storage and networking, follow relevant ownership and failure paths, and specify tests that could disprove a proposed cause before recommending a bounded fix.

Source walkthrough#

A user-visible delay crosses boundaries. A source path tells us which mechanisms are possible under stated conditions; timestamps, counters and controlled changes tell us which mechanisms actually occurred. A blocked task's wall time can include off-CPU waiting, while CPU time measures charged execution. One counter cannot identify a unique cause. The two cases below use invented numbers with explicit clock and scope; they are worked reasoning exercises, not fabricated production logs.

Storage entry and error branches

fsync(fd) reaches do_fsync, which obtains a file reference with fdget, calls vfs_fsync, and releases that reference with fdput; an invalid fd returns -EBADF. In fs/sync.c, v6.12, SPDX GPL-2.0, do_fsync is the static helper behind fsync and fdatasync; the complete function is:

static int do_fsync(unsigned int fd, int datasync)
{
	struct fd f = fdget(fd);
	int ret = -EBADF;

	if (fd_file(f)) {
		ret = vfs_fsync(fd_file(f), datasync);
		fdput(f);
	}
	return ret;
}

fdget and fdput bracket the call with a file reference, and -EBADF is the only error this helper creates itself. Whatever vfs_fsync returns is passed back unchanged, so the excerpt does not show what the filesystem does with data or journal errors.

In fs/sync.c, vfs_fsync_range contains this exact v6.12 SPDX GPL-2.0 excerpt; surrounding inode and dirty-time handling are omitted:

	if (!file->f_op->fsync)
		return -EINVAL;

The VFS checks that the open file's operations provide an fsync method. Without one, it returns -EINVAL rather than calling a null callback. With one, the following code dispatches to the filesystem's method. The struct file reference and f_mapping->host identify the open file and inode; a pathname string alone does not establish their lifetime. Errors from the filesystem and writeback must be preserved rather than reported as successful durability.

For ext4 with a journal, ext4_sync_file checks forced shutdown and read-only state, waits for relevant file data writeback, waits for the applicable journal transaction and issues a block flush when required. It checks and advances the file's writeback error state before returning. The no-journal path differs. Filesystem mount options, device cache and flush semantics, prior errors and configuration affect the result; a completed write is not equivalent to a completed and correctly propagated fsync.

TCP accept entry and error branches

In net/ipv4/inet_connection_sock.c, inet_csk_accept holds the listening socket lock and checks for a pending connection. This exact v6.12 GPL-2.0-or-later excerpt omits surrounding state checks and the later wait:

		error = -EAGAIN;
		if (!timeo)
			goto out_err;

If the accept queue is empty and the computed receive timeout is zero, such as for the relevant nonblocking case, the function takes its error path with -EAGAIN rather than sleeping. Otherwise it can wait for a connection and handle an error from that wait. On a successful path it removes a request from the accept queue, obtains the child socket and later releases the listener lock and relevant request reference. TCP Fast Open has additional ownership handling in the source; the simple path must not be generalized to every handshake state. An empty accept queue is not itself proof of network loss.

Case 1: synthetic fsync tail latency#

Assume a single host with a journaled ext4 filesystem on a supported block device. A service reports that one request's fsync took 180 ms wall time; a task CPU-time measurement over the same interval increased by 4 ms. Synthetic timestamps from one monotonic clock show the request entering fsync at 1,000 ms and returning successfully at 1,180 ms. During that interval, the cgroup's memory-pressure total increases by 20 ms and its CPU throttling counter does not change. These invented values establish only the stated observations: 176 ms was not charged as that task's CPU time, but the categories of that time are not yet known. PSI total does not label the exact fsync wait or establish a causal share.

Hypothesis A is that data writeback and journal/flush completion dominate the tail. The VFS holds the file reference while ext4 waits; dirty folios can be written through the filesystem and block layer, and completion returns through block and driver paths. A block request and its buffers have their own lifetime and synchronization, distinct from the calling task's file reference. A slow flush can lengthen fsync even if the application used little CPU. But a long block completion alone does not prove it was the specific request or transaction on this call's critical path.

Hypothesis B is that memory pressure causes direct reclaim or writeback congestion before or during the fsync, with storage latency a secondary effect. The 20 ms PSI increase is consistent with some stall in the cgroup, but could belong to another task and is smaller than the 180 ms wall interval. Other explanations include journal contention, CPU runqueue delay, writeback errors, virtual device scheduling, a slow userspace stage outside the measured syscall, or a different filesystem path. A cgroup memory high event, if measured, is not a timestamped proof that this specific call waited for reclaim.

A controlled disposable-guest test would retain the same file size, data, mount options, kernel, device model and correctness check while recording syscall start/finish, task scheduling, relevant writeback and block completion IDs, journal events, cgroup statistics and loss indicators. Compare a matched baseline against one deliberately different constraint only when the guest and recovery are under your control. Falsify the flush hypothesis if the relevant flush and journal completion finish well before the fsync return while the task remains runnable but unscheduled, assuming trace coverage is complete enough. Falsify the reclaim hypothesis if no relevant reclaim activity or pressure occurs in repeated matched slow calls with adequate instrumentation. Either observation needs careful clock and correlation validation.

A bounded candidate fix might reduce unnecessary synchronous writes in the application while preserving its durability contract, or correct a demonstrated writeback accounting bug in a specific kernel path. The regression plan must verify data and metadata durability under the required crash model in a throwaway guest, preserve error propagation, and compare tail latency across repeated trials. Disabling fsync or ignoring errors would change semantics, not establish a performance fix. No crash-consistency test or guest run has been performed here.

Case 2: synthetic TCP server delay#

Assume a server accepts a connection at 2,000 ms, receives a request by 2,020 ms, and sends a response at 2,140 ms on one monotonic clock. The application reports 120 ms between its completed receive and response, of which the worker accumulates 15 ms CPU time. A simultaneous socket snapshot shows a nonzero receive queue for another connection, and an application backlog metric reports eight queued jobs. These are synthetic observations. They do not establish packet loss, a full accept queue, a firewall drop or that the measured worker spent all remaining time on the network.

Hypothesis C is application queueing or CPU scheduling: the request was received, but the worker waited for a thread, lock or CPU. Check when the application enqueued and dequeued the same request, worker runnable/sleep transitions, cgroup CPU quota and lock contention. A nonzero socket receive queue indicates unread data for that socket at the observation time; it is not a wire-loss counter. A queue on another connection is especially weak evidence for this request. The accept queue contains pending established children and is conceptually distinct from a per-connection receive queue and from an application's own job queue.

Hypothesis D is TCP or network-path delay before application receive. Follow the socket operation and actual TCP state, retransmission evidence, routing and neighbour state, netfilter counters if available, device queue/NAPI processing and timestamps. NAPI may process packets in a polling context and the application can be scheduled later; neither a NAPI event nor a retransmission alone proves that it caused the 120 ms after receive completed. An nftables drop counter can increase because of unrelated traffic and does not prove this request was dropped. Conversely, no increase in a sampled counter does not establish that every packet arrived.

A controlled test in a dedicated guest can send a fixed request stream, tag requests with application-level IDs, and correlate client send, server receive, application dequeue and response completion using documented clock relationships. Compare a low-load baseline with the same workload at the suspected queue depth; record loss, retransmission, CPU pressure and controller limits. Falsify the application-queue hypothesis if matched slow requests show immediate dequeue and execution while a measured network interval accounts for the delay. Falsify a claimed retransmission cause if the correlated connection has no relevant retransmission and the delay occurs entirely after complete receive, subject to measurement coverage. A bounded fix might reduce demonstrated lock contention or correct a queue-handling defect, followed by concurrency, timeout, partial-read and connection-close regression tests. It must not blindly raise backlog or disable filtering.

Evidence, ownership and cross-layer uncertainty#

An end-to-end report identifies the request and its owner at each stage. A task can sleep while a block request or socket is owned by another subsystem; completion may happen on another CPU. Locks, references and callback lifetimes differ at each boundary. In the storage case the fd reference protects the open file during do_fsync, while filesystem and block structures have their own lifetimes. In the TCP case the listener socket lock protects accept-queue operations in the shown path; after acceptance the child socket has its own state and ownership. A timer or softirq can advance network work independently of the server thread.

Clock alignment and lost records can create false orderings. A log timestamp from a remote client cannot be subtracted from a server timestamp without an established clock relationship. A counter delta summarizes an interval and can include other tasks or devices. A trace with dropped events can omit the actual completion or wakeup. Keep a table of observed facts, source-derived possible mechanisms, assumptions and unknowns. The source excerpts constrain explanations, but they do not establish that either synthetic incident occurred.

Turning the hypotheses into a reviewable report#

For the storage case, the report should map each observed interval to a specific identifier: process and thread, fd or inode, filesystem and device, and any block request or journal transaction that instrumentation can safely correlate. If the fsync range is not associated with the observed block request, a slow request on the same disk is only background context. The filesystem may already have written some data before the syscall begins, and the journal commit and flush can overlap other tasks. Preserve both successful return and any recorded writeback error; a zero return on one fd does not describe unrelated files.

For the network case, use connection identity and an application request ID rather than assuming that a packet timestamp or a queue counter belongs to the measured transaction. Distinguish socket receive buffering, an accept queue of pending connections and the application work queue. A server can accept promptly and then wait on an application lock; a client can observe high latency even though the server spent little CPU time. If clocks differ between hosts, use a synchronization error bound or compare intervals measured on one host rather than subtracting incomparable timestamps.

Both investigations need negative controls and failure handling. A trace buffer overflow, missing request ID or unrecorded device completion is a gap in the evidence, not a zero-duration event. Repeating the test with reduced instrumentation can estimate observer impact, while a separate correctness check ensures that a proposed optimization does the same work. A reviewable fix names the proven bottleneck, the smallest change, possible regressions and rollback criteria. If the proposed cause cannot be distinguished from an alternative with available data, the proper result is an unresolved hypothesis and a next measurement.

Practice#

Safe private-file observation

The following uses one newly created temporary file and removes it automatically. It writes and fsyncs only that file; it does not change host configuration or simulate a crash:

python3 - <<'PY'
import os, tempfile, time
try:
    with tempfile.TemporaryDirectory(prefix='fsync-study-') as d:
        path = os.path.join(d, 'data')
        with open(path, 'wb') as f:
            f.write(b'kernel-study\n')
            f.flush()
            start = time.monotonic_ns()
            os.fsync(f.fileno())
            elapsed = (time.monotonic_ns() - start) / 1e6
        with open(path, 'rb') as f:
            assert f.read() == b'kernel-study\n'
        print('verified bytes: 13; fsync wall ms:', elapsed)
except OSError as e:
    print('private-file operation failed:', e.errno, e.strerror)
PY

A successful run reports the verified content and one wall-time sample. It does not identify the filesystem path, prove physical media persistence, or reproduce the synthetic 180 ms incident. A missing temporary filesystem, permission denial, unsupported fsync or I/O error is a legitimate failure to record. One sample is not a benchmark. The context manager cleans only its own temporary directory.

Source-only alternative: produce two incident tables from the supplied synthetic timestamps. For storage, list the 180 ms wall interval, 4 ms task CPU and at least two possible explanations; for TCP, distinguish the 120 ms post-receive interval, another connection's receive queue and application backlog. Add one falsification criterion per hypothesis and the VFS and accept error branches. Deliver the tables or the real private-file result, never attributing either synthetic incident to a measured cause.

Knowledge check#

1. Why does a 180 ms fsync with 4 ms task CPU not prove a 176 ms device flush?
2. What does vfs_fsync_range return when the file lacks an fsync operation?
3. Why is a receive queue on another TCP connection not proof of loss for the measured request?
4. What does the nonblocking empty accept-queue branch return?
5. Why would simply disabling fsync be an invalid performance fix for a required durability contract?

Answers#

1. Off-CPU wall time can include scheduling, reclaim, locks, journal waits and device I/O. Correlated events and matching request identities are needed to divide it.
2. The cited branch returns -EINVAL; it does not dispatch a nonexistent method.
3. It describes unread data on a different socket at one time. It neither measures wire loss nor identifies the measured request's path.
4. When the computed timeout is zero, the quoted branch takes the error path with -EAGAIN.
5. It removes or weakens the required durability operation rather than showing the same correct work became faster. Any fix must preserve the stated contract and error handling.

Checkpoint#

Submit a two-case report with observations, competing hypotheses, source paths, one falsification criterion per hypothesis and a bounded regression plan. Ready means synthetic data stays labeled, CPU time is not equated with device time, and queue types are distinguished. Retry the timelines if a correlation is treated as proof. This is not certification.

Sources#

• VFS sync v6.12
• ext4 fsync v6.12
• TCP accept v6.12
• TCP receive v6.12
• PSI documentation