fsync, ordering, failure, and crash consistency

Before you start#

Read device mapper and MD RAID, page-cache writeback, and ext4 journaling. This chapter uses Linux v6.12 as a historical source baseline; filesystem, mount options, storage stack, device firmware and architecture matter. The lab uses only a temporary directory and ordinary user permissions.

Learning goals#

Follow dirty data through writeback and synchronization, distinguish file content from directory-entry durability, interpret late writeback errors, and design an atomic replacement protocol with explicit assumptions.

Source walkthrough#

A buffered write can copy bytes into page-cache folios and return before backing storage has persisted them. A dirty folio has changes requiring writeback; writeback initiates I/O, and completion reports an outcome for that I/O. Reclaim can initiate or wait for writeback to free memory, but reclaim is not an application durability protocol. A filesystem maps logical file ranges to storage, decides which metadata must be updated, and selects the relevant synchronization and ordering operations. The block layer and device then perform their parts. Each stage can report failure. Successful memory copying, successful submission, I/O completion, and survival after power loss are four different claims.

In fs/sync.c, vfs_fsync_range, v6.12 contains this excerpt (GPL-2.0; surrounding code omitted):

	if (!file->f_op->fsync)
		return -EINVAL;
	if (!datasync && (inode->i_state & I_DIRTY_TIME))
		mark_inode_dirty_sync(inode);
	return file->f_op->fsync(file, start, end, datasync);

First, the VFS rejects an object without an fsync method. For a full sync, dirty timestamp state can be marked for synchronous handling. The final statement delegates to that file's filesystem operation with a byte range and the datasync distinction. This generic function does not itself encode all ext4, XFS, network filesystem or device rules. fsync requests synchronization of file data and associated metadata; fdatasync can omit metadata not required for later data retrieval. Neither automatically synchronizes the directory entry naming a newly created or renamed file. The Linux fsync(2) manual documents these scopes explicitly: fdatasync still needs to persist file-size changes, and an explicit directory fsync is needed to persist a directory entry.

For a concrete implementation, ext4_sync_file checks shutdown and read-only state, handles a no-journal case, and for its journaled path calls file_write_and_wait_range, then ext4_fsync_journal, with a flush when needed. An error can prevent subsequent success. This is an ext4 path, not a universal call chain. xfs_file_fsync also begins with file writeback and wait, but its log and device-flush decisions use XFS state and topology. Filesystems that do not support a requested synchronization can return an error.

file_write_and_wait_range is in mm/filemap.c. Its final lines (GPL-2.0-only; earlier writeback and waiting omitted) are:

	err2 = file_check_and_advance_wb_err(file);
	if (!err)
		err = err2;
	return err;

The function samples and advances the file's writeback-error cursor. It preserves an earlier error if present; otherwise it reports the sampled error. A successful earlier write can therefore be followed by an fsync error when delayed writeback fails. Error reporting has per-open-file-description observation semantics and can cover errors since a cursor was sampled, rather than identifying one exact userspace write. The errseq_t mechanism uses an error plus sequence state; see lib/errseq.c and include/linux/errseq.h. Do not assume every duplicated fd has an independent error cursor: duplicates share an open file description. The kernel VFS writeback-error documentation explains that generic reporting can reach file descriptions open at the time of an error because page-cache tracking does not associate each dirtied page with its specific descriptor.

From cache to nonvolatile storage#

A file write can dirty a folio in memory. Writeback turns the dirty range into filesystem and block I/O; a block request can be split, merged or transformed by stacked devices before the driver sends commands. A completion indicates that an I/O request finished according to the device and stack's contract, but a storage device can acknowledge a write while data remains in a volatile write cache. A device flush requests that previously completed writes reach the device's nonvolatile domain under its advertised behavior. Force unit access (FUA), where supported and used, requests the applicable write complete with stronger persistence semantics. REQ_PREFLUSH and REQ_FUA are block-layer flags, not magic instructions that overcome a controller or device that lies about persistence. The block flush state machine can sequence operations according to queue capabilities; see block/blk-flush.c and include/linux/blk_types.h.

A typical ordering question is: must data A reach persistent storage before metadata B exposes it? The answer depends on filesystem mode, synchronization operation, journal implementation and the storage stack. A flush can order a relevant set of writes but cannot by itself make an arbitrary collection of file modifications into an atomic transaction. RAID, device mapper, a hypervisor, remote storage and hardware caches add layers. If one layer discards a flush or fails to honor its contract, a higher-layer success claim can be invalidated. A power-loss-protected cache can change the physical persistence domain; it does not remove the need to understand the device's stated guarantees.

Suppose a program writes new over old and write returns 3. At this point the current cache may contain new; the disk might still contain old. After a successful file fsync, the filesystem has reported completion of its synchronization contract for the file under the stack's assumptions. That result is stronger than reading back new from cache, but it is still not a proof that unrelated files or a newly created directory entry are durable. If fsync returns EIO, the program must not announce that the replacement was safely saved. This is a hypothetical timeline, not a power-loss observation.

Atomic replacement is a protocol#

For a file in a directory on a filesystem supporting the necessary operations, a common replacement sequence is:

1. Create a new temporary file in the same target directory, using a unique name and restrictive initial permissions where appropriate.
2. Write the complete new contents, handling short writes and errors. Set required metadata before synchronization.
3. Flush any language-runtime buffer, then call fsync on the temporary file and check the result.
4. Rename the temporary file over the destination on the same filesystem, checking the rename result. Rename provides namespace atomicity under its applicable contract; it does not by itself make the directory change crash durable.
5. Open and fsync the containing directory, check that result, and only then report the desired durability level.

Creating the temporary file in the target directory avoids a cross-filesystem rename and simplifies which directory needs syncing. If moving between directories on the same filesystem, both directory updates and their synchronization requirements need consideration. If the filesystem does not support directory fsync or reports an error, the application cannot silently claim the protocol succeeded. The recipe does not atomically update multiple unrelated files; use a transactional design, journal or manifest appropriate to the application. Permissions, ACLs, ownership, symlinks and concurrent writers require explicit policy.

A truncate-in-place update is different. If a process truncates a file to zero, writes new bytes and crashes, readers or recovery may see an empty or partial result depending on timing and filesystem behavior. Synchronizing the file after the completed write improves the requested persistence of that file, but it cannot retroactively make the prior intermediate state invisible. A temporary-file-and-rename protocol provides a different namespace transition. File size is metadata required to retrieve the new contents; fdatasync is not a license to ignore size changes.

msync(MS_SYNC) requests synchronization for an applicable mapped range; mapping type, filesystem behavior and error handling matter. It is not a replacement for handling directory entry persistence. syncfs operates on the filesystem associated with an fd and can report filesystem-wide errors; it is not an application transaction boundary. close releases a descriptor and can report errors, but a successful close is not a substitute for an explicitly checked fsync when durability is required. On Linux, do not blindly retry close after an error: the descriptor number may already have been released and reused. These interfaces have different scopes and are not interchangeable synonyms.

Failure, sequencing, and verification#

Consider two files A and B. The application writes A, fsyncs A, then writes B, and crashes before fsyncing B. A may meet its synchronization contract while B does not. Even if both fsync calls succeed, a crash between the two can leave an intermediate application state. A database-style transaction needs a protocol that defines valid recovery states. For a single replacement, crash points before file fsync, after file fsync, after rename, and after directory fsync have different implications. A reader observing a rename in a live system does not prove the new directory entry will survive a later power failure.

A failed writeback can surface long after the original write call. errseq_t lets a later observer detect errors since its last sample without requiring the kernel to retain an unbounded list of failed writes. It is not an audit log and its cursor does not prove which exact bytes failed. An application must retain enough context to decide whether to retry, warn the user, switch storage, or preserve an older version. Repeating fsync after an error does not automatically establish that the first operation's intended data were repaired; the application must know what it is retrying.

A crash experiment requires an isolated guest, a disposable disk image, an independent copy of the test data, known cache settings, a controlled failure point, a separate recovery boot and an oracle describing acceptable outcomes. Killing an application is not the same as abruptly losing power: the guest kernel can continue writeback. Killing a virtual machine can also leave host-side caches operating, depending on the virtual disk configuration. Never power off or corrupt the host for this lesson. The ordinary lab below tests syscall behavior and read-back only, not crash survival.

Observing errors without overclaiming#

An application should preserve the first meaningful failure while still attempting necessary cleanup. If write reports a short count, it must account for the completed bytes before deciding whether to retry. If fsync fails, closing the descriptor remains necessary, but a later successful close does not undo the fsync failure. If rename fails, the old destination can remain in place and the temporary file may need cleanup; an application must not assume a replacement occurred. If directory fsync fails after a successful rename, the new name may already be visible, but the program must report that its requested crash-durability condition was not established. Recovery code must tolerate this distinction between visibility and durability.

In fs/ext4/fsync.c, v6.12, SPDX GPL-2.0 (surrounding code omitted), the journaled path ends by checking the file's writeback error state:

out:
	err = file_check_and_advance_wb_err(file);
	if (ret == 0)
		ret = err;

The check runs after the data wait and any journal commit, so it can surface a buffered-write failure even when those steps returned zero; a nonzero earlier result is kept instead. The excerpt does not show how the error cursor was sampled, whether another descriptor already observed the error, or what happened to the failed data.

A test with strace is useful for confirming that this process issued calls in the intended order and checking their return values. It cannot look inside a device's volatile cache, prove a remote server honored stable writes, or establish what will be recovered after power loss. Log the filesystem and mount context alongside the syscall sequence. A controlled crash experiment would need a separately validated storage and recovery setup; absence of an error in a source-only walkthrough or read-back is a narrower result.

The directory is a separate object#

The replacement recipe uses two objects whose state can be persisted at different times: the temporary file's inode and data, and the directory containing the name. File fsync first asks the filesystem to commit the former under its contract. Rename then changes the directory's mapping from a name to an object. Synchronizing the directory asks for that namespace change to be persisted. A reader can observe the new name immediately after rename even if a crash before directory sync later exposes a different permitted state. That is why a successful read after rename is a useful live-system observation but not a recovery test.

The recipe also has an application error boundary. If setting a required mode, ownership, ACL or other metadata fails before the file fsync, continuing to rename can publish an object that does not meet the program's intended requirements. If the rename fails, the old destination may still exist and the temporary file may need removal. If the directory fsync fails, the name may already be visible, so reporting a simple "nothing changed" is equally misleading. Record which operations succeeded and choose a recovery policy that tolerates an uncertain crash outcome. Other filesystems, remote servers and operating systems can have different contracts; a same-directory rename and successful sync calls are assumptions to verify for the actual deployment.

On a layered block device, flush behavior has to propagate through every participating layer. An encryption or RAID target can change the I/O path; a hypervisor can add a host cache; the drive can have its own volatile cache. The application does not directly control all these layers. A meaningful assurance therefore combines the filesystem and block-layer contract with the configuration and reliability of the storage beneath it. A local source reading can explain where a flush is requested, but cannot establish the physical persistence properties of a particular device.

Practice#

Source-only route: in a checkout with git -C "$KERNEL_SRC" describe --tags --exact-match returning v6.12, inspect:

rg -n -A20 'int vfs_fsync_range\(' "$KERNEL_SRC/fs/sync.c"
rg -n -A22 'int file_write_and_wait_range\(' "$KERNEL_SRC/mm/filemap.c"
rg -n -A55 'int ext4_sync_file\(' "$KERNEL_SRC/fs/ext4/fsync.c"

Trace the missing-fsync-method error, data wait, error-cursor sampling and ext4 journal branch. Write down which statement belongs to the VFS and which belongs to ext4; do not assert that the same filesystem path was executed on your machine.

Run this unprivileged observation in a disposable guest or a normal temporary directory:

python3 - <<'PY'
import os, tempfile
with tempfile.TemporaryDirectory() as d:
    tmp, dst = d + '/new', d + '/result'
    with open(tmp, 'wb') as f:
        f.write(b'new contents\n'); f.flush(); os.fsync(f.fileno())
    os.replace(tmp, dst)
    dfd = os.open(d, os.O_RDONLY | os.O_DIRECTORY)
    try: os.fsync(dfd)
    finally: os.close(dfd)
    with open(dst, 'rb') as f: print(f.read().decode().strip())
PY

Expected new contents if the temporary filesystem supports file and directory fsync. An OSError is a meaningful failure: report the operation and filesystem, not a successful durability result. The temporary directory is cleaned automatically. If available, save the program as replace-lab.py inside a temporary directory and use strace -e trace=write,fsync,rename,renameat,renameat2,close python3 replace-lab.py; tracing observes calls and return values, not power-loss behavior. Do not install or enable tracing as root for this exercise. Deliver a timeline labeling the file sync, rename, directory sync and each possible error; include the read-back or exception.

Knowledge check#

1. Why can a successful write be followed by EIO from fsync?
2. What does file_check_and_advance_wb_err contribute, and what does it not identify?
3. Why must a durable replacement sync the containing directory after rename?
4. What distinguishes a preflush or FUA operation from a multi-file transaction?
5. Why is reading back the new file not proof of power-loss survival?

Answers#

1. Buffered writeback can happen later and fail after the copy to page cache returned successfully.
2. It samples and advances writeback error state for the open file description; it is not a complete log mapping each error to one userspace write.
3. The file sync covers the file under its contract; the rename changes directory metadata, which needs its own synchronization for the stated replacement protocol.
4. Flush and FUA address storage ordering or persistence for relevant writes. They do not provide atomic commit and recovery for arbitrary changes across files.
5. The read can be satisfied from caches, and no crash or storage failure has been induced or recovered in the experiment.

Checkpoint#

Save the replacement timeline and lab output or error. Pass when you can identify each separate persistence boundary, explain a late writeback error, and name the same-filesystem and directory-sync assumptions. Revisit the ext4 fsync path and retry the timeline if any step is missing. This is practice, not certification.

Sources#

• VFS sync operations, Linux v6.12.
• File writeback and error sampling.
• ext4 fsync.
• XFS fsync.
• Block flush implementation.
• Writeback error sequence implementation.