ext4, extents, JBD2, and recovery

Before you start#

Read mounts and filesystem namespaces and page-cache writeback. This chapter describes upstream v6.12 as a historical baseline. All filesystem creation, mounting and crash experiments belong in disposable guest disks; the observation below needs only an ordinary temporary file.

Learning goals#

Trace logical-block lookup through extents, explain the journal transaction and fsync boundaries, distinguish data modes, and state what recovery can and cannot guarantee.

Source walkthrough#

A file presents byte offsets; ext4 converts them to filesystem logical blocks and maps those through extents to filesystem blocks. An extent describes a run of logical blocks and corresponding physical filesystem blocks, efficiently representing a contiguous run without one separate mapping entry per block. Extent trees and the ext4_map_blocks request represent this relationship; delayed allocation can defer physical block selection, and unwritten extents have different initialization semantics. Filesystem block numbers are not automatically physical disk-sector numbers after partitioning, device mapping and other layers. See fs/ext4/extents.c, fs/ext4/inode.c, and the [ext4 extent documentation](https://docs.kernel.org/6.12/filesystems/ext4/dynamic.html# extent-tree).

In ext4_ext_map_blocks, the incoming map->m_lblk is the logical starting block and map->m_len is the requested length. Its first extent lookup reads (GPL-2.0; surrounding code omitted):

	path = ext4_find_extent(inode, map->m_lblk, NULL, 0);
	if (IS_ERR(path)) {
		err = PTR_ERR(path);
		goto out;
	}

A failed tree lookup becomes an error and branches to cleanup; the returned error pointer is not an extent. On success the rest of the function considers the located extent, possible holes and request flags. It can return an existing mapping or, where the caller requests and conditions permit, allocate or convert blocks. A read of a hole is not proof that an allocated physical block contains zeroes. The exact behavior depends on flags and extent state.

Journal transaction and sync#

JBD2 groups journaled metadata changes into transactions. A handle represents a participant's access and reserved credits; acquiring a handle does not mean the transaction is committed. jbd2_journal_commit_transaction in fs/jbd2/commit.c transitions through states including T_LOCKED, T_FLUSH, T_COMMIT, T_COMMIT_DFLUSH, T_COMMIT_JFLUSH, T_COMMIT_CALLBACK, and T_FINISHED. These names represent an implementation sequence, not one synchronous syscall per state. Commit makes a valid transaction available for journal recovery subject to I/O and barrier guarantees; checkpoint later writes journaled metadata to its home locations so journal space can be reused. Commit and checkpoint are not the same event.

In the journaled branch, ext4_sync_file first waits for the relevant file data, then waits for the relevant journal transaction. This excerpt shows the error boundary (GPL-2.0; other branches omitted):

	ret = file_write_and_wait_range(file, start, end);
	if (ret)
		goto out;

If data writeback reports failure, the function does not claim success by continuing as if that data were written. Later ext4_fsync_journal and any needed block-device flush can also return errors, and the function checks recorded writeback errors. A no-journal branch and read-only or forced-shutdown handling exist too. Application code must check fsync's return, including errors reported after an earlier write appeared successful.

In ext4_fsync_journal, v6.12, SPDX GPL-2.0 (surrounding code omitted):

static int ext4_fsync_journal(struct inode *inode, bool datasync,
			     bool *needs_barrier)
{
	struct ext4_inode_info *ei = EXT4_I(inode);
	journal_t *journal = EXT4_SB(inode->i_sb)->s_journal;
	tid_t commit_tid = datasync ? ei->i_datasync_tid : ei->i_sync_tid;

	/*
	 * Fastcommit does not really support fsync on directories or other
	 * special files. Force a full commit.
	 */
	if (!S_ISREG(inode->i_mode))
		return ext4_force_commit(inode->i_sb);

	if (journal->j_flags & JBD2_BARRIER &&
	    !jbd2_trans_will_send_data_barrier(journal, commit_tid))
		*needs_barrier = true;

	return ext4_fc_commit(journal, commit_tid);
}

The transaction ID comes from i_datasync_tid for datasync and from i_sync_tid otherwise. Non-regular files take the forced full-commit branch, and needs_barrier is raised only when the journal uses barriers and the transaction will not already send one. The excerpt does not show the fallback from ext4_fc_commit to jbd2_complete_transaction.

The sequence below shows how ext4_sync_file moves between these steps.

In the usual data=ordered mode, ext4 orders associated data writeout before committing the metadata that exposes it; metadata is journaled. data=journal journals file data as well as metadata, with different performance and feature tradeoffs. data=writeback has weaker data-ordering guarantees. These labels do not guarantee an application's multi-file transaction or prevent faulty hardware from lying about flushes. Filesystem options, fast-commit behavior, device ordering, and kernel configuration affect details; consult the ext4 documentation.

Consider a hypothetical update of a file whose extent covers logical blocks 8–11 at physical filesystem blocks 100–103. Logical block 10 maps to block 102 within that extent. A modification dirties cached bytes and metadata may need a journal transaction. An fsync waits for relevant data and journal work, reporting any error. If power fails before a transaction's valid commit record, recovery must not treat that incomplete transaction as committed; a committed journal transaction can be replayed by JBD2 recovery. fs/jbd2/recovery.c implements recovery. This example does not predict which earlier data bytes survive an arbitrary crash. Creating or renaming a file can require syncing the containing directory as well as the file when an application needs a durable name.

From buffered modification to recoverable metadata#

A write may first modify page-cache data and mark it dirty. Delayed allocation allows ext4 to defer selection of physical blocks, potentially combining writes into larger allocations. Therefore the application returning from write cannot derive a stable physical extent or durability from its buffer and offset alone. When writeback maps blocks, extent lookup may find an existing initialized extent, an unwritten extent, or a hole. The flags and state determine whether allocation or conversion is needed. A failed lookup or allocation must be reported through the appropriate writeback and synchronization paths; a successfully copied userspace buffer is not proof the disk had space.

A journal handle reserves credits for metadata operations within a transaction. Credits bound the expected journal resources; they are not disk blocks for application data and not a durability receipt. Several handles can contribute work to a transaction. Commit must coordinate participants and ordered data according to the journal mode, write required journal records and complete its commit protocol. Checkpoint can occur later. If a crash happens, the recovery implementation scans the journal and replays transactions according to its integrity and ordering rules; it does not reconstruct an application's intended operation beyond what was actually recorded.

Consider replacing a configuration file safely. An application may write a new temporary file, fsync it, rename it over the old name, then fsync the containing directory, checking each result. The file sync and directory sync address different objects. This is a pattern to investigate against the actual filesystem and error model, not a universal proof in the presence of broken hardware, unsupported operations or unhandled failures. If the rename succeeds in memory but directory synchronization fails, the application must not report the durable replacement as established. A journal's metadata consistency does not supply the missing application-level error handling.

Recovery boundaries and error reporting#

A journal record's existence is not enough to infer that its transaction committed. Recovery must distinguish valid committed transactions from incomplete or invalid journal contents and replay according to JBD2's rules. The commit procedure has ordering and flush interactions because storage can cache and reorder writes. Even a valid commit record cannot compensate for a device that violates its advertised persistence semantics. Fast commits add another mechanism in supported configurations, so a diagram that depicts every fsync as one full traditional commit is also too broad.

If the journal aborts or ext4 enters forced shutdown, subsequent operations can report errors rather than silently continuing as though the filesystem were healthy. The excerpted ext4_sync_file checks forced shutdown, and later retrieves recorded writeback errors. A write can have returned a positive count because bytes reached the page cache, followed by an error at fsync when writeback is attempted or a prior error is reported. Applications need a policy for this outcome: retrying blindly can be unsafe if the previous operation partially completed, while ignoring the failure creates a false durability claim.

Extent mapping has its own concurrency concerns. A logical range can be involved in allocation, truncation or unwritten-extent conversion while another operation looks it up. The inode and extent code use locking and transaction rules appropriate to each operation. A sketch saying logical block 10 maps to physical block 102 is valid only for the stated hypothetical mapping and moment; it is not a license for an application to retain that block number after file changes. Block relocation, holes, delayed allocation and the stacked block device all complicate any attempt to treat a filesystem block address as a persistent hardware address.

To test crash consistency rather than merely call fsync, a researcher would need a disposable guest disk image, a precisely described failure injection point, a recovery boot, expected allowed outcomes, repeated trials and a record of cache and flush settings. Randomly killing a host process is not equivalent to power loss: the host kernel may continue writing data after the process exits. The exercise here intentionally avoids claiming such a test. It teaches the ordering question and leaves crash injection to a dedicated disposable laboratory.

Extent state and delayed allocation#

An ext4 extent tree has an ext4_extent_header with a magic value, entry count, maximum entry count and depth. At a leaf, struct ext4_extent records a logical start (ee_block), encoded length/state (ee_len) and high/low components of a physical filesystem-block start. Internal index entries point to lower tree blocks. These are on-disk little-endian structures and require validation; do not use a tree depth or entry count from untrusted disk as an unchecked array bound. The extent-status cache tracks in-memory status such as written, unwritten, delayed and hole ranges to accelerate lookup; it is not itself a guarantee that an uncommitted cached state survived a crash. See fs/ext4/ext4_extents.h and fs/ext4/extents_status.c.

A delayed-allocation range can contain dirty cached data before ext4 chooses physical blocks. An unwritten extent reserves blocks but is not yet treated as initialized file data; conversion to initialized state follows the filesystem's write completion and journaling rules. Reverse mapping is a different question from file logical-to-physical extent lookup: the latter does not by itself provide a complete list of every owner of a physical block. Allocation, truncation and conversion require appropriate inode/extent and journal synchronization.

For a hypothetical 4 KiB filesystem block, an initialized extent maps logical blocks 8–11 to physical filesystem blocks 100–103. File byte offset 10 * 4096 + 123 lies at logical block 10, offset 123, and maps through that extent to filesystem block 102, offset 123. A hole or unwritten extent needs its own read semantics; the arithmetic cannot establish that those bytes are initialized data. A stacked device can translate filesystem block 102 further, so it is not necessarily raw sector 102 on a physical disk.

The mapping below restates the hypothetical 4 KiB example as a table. The numbers are arithmetic from the chapter, not a dump of a real inode.

JBD2 handles, commit, and recovery#

A JBD2 handle associates a task's journaled work with a transaction and reserved credits; nested handle use and stopping follow the journal API, and the caller must not mistake a handle return for a completed durable commit. In v6.12 transaction_t.t_state is declared as T_RUNNING, T_LOCKED, T_SWITCH, T_FLUSH, T_COMMIT, T_COMMIT_DFLUSH, T_COMMIT_JFLUSH, T_COMMIT_CALLBACK, and T_FINISHED. The implementation's lock comments and transitions matter; this list is not a separate userspace transaction protocol. See include/linux/jbd2.h and fs/jbd2/transaction.c.

In include/linux/jbd2.h, v6.12, SPDX GPL-2.0-or-later (surrounding code omitted):

	enum {
		T_RUNNING,
		T_LOCKED,
		T_SWITCH,
		T_FLUSH,
		T_COMMIT,
		T_COMMIT_DFLUSH,
		T_COMMIT_JFLUSH,
		T_COMMIT_CALLBACK,
		T_FINISHED
	}			t_state;

The enumerators appear in declaration order, and the field comment says that only the kjournald2 thread alters t_state. This is an implementation list, not a userspace protocol, and the excerpt does not show which transitions a given commit takes.

In jbd2_journal_commit_transaction, v6.12, SPDX GPL-2.0+, an error while submitting data buffers is handled explicitly:

	err = journal_submit_data_buffers(journal, commit_transaction);
	if (err)
		jbd2_journal_abort(journal, err);

The function submits the relevant transaction data buffers and aborts the journal on a reported error. It does not convert that error into a successful commit. Later commit phases write and wait for journal records and coordinate the relevant barriers; checkpointing to home locations is a later concern. A crash before a valid commit is recovered differently from a crash after a valid committed transaction, and journal recovery can replay eligible committed metadata. Hardware that loses acknowledged writes or ignores required flushes invalidates assumptions about what survived. Orphan handling addresses filesystem cleanup for inodes needing it after interruption; it is not an application-level multi-file transaction. See fs/jbd2/recovery.c and fs/ext4/orphan.c.

data=ordered orders associated data before the exposing metadata commit under its rules; data=writeback lacks that same ordering; data=journal journals file data as well as metadata. None guarantees that an arbitrary application replacement spanning several files is atomic. In ext4_fsync_journal, v6.12 selects i_datasync_tid for datasync or i_sync_tid for full sync; non-regular files force a full commit in that function, while regular-file handling calls ext4_fc_commit with the selected transaction ID and evaluates whether a barrier is needed. Fast commit is conditional on support and eligibility, and a full commit can be required. ext4_sync_file checks writeback and later errors. The general application-level directory-sync and device-cache limits are explained in durability and crash consistency.

For the 4 KiB example, imagine metadata for a new extent is part of transaction T. A crash before T has a valid journal commit does not justify assuming recovery will replay T as committed. After a valid commit and subject to correct storage ordering and successful recovery, journal replay can restore the committed metadata even if checkpointing had not yet written all home locations. Neither statement alone proves the intended file bytes or directory name survived: data mode, fsync results, directory synchronization, storage cache behavior and the crash point all matter. This is a source-based hypothetical, not a crash experiment.

Credits, unwritten extents, and failure evidence#

A handle's credits reserve journal resources for its intended metadata work; they are not an application byte count or proof of disk-space availability. A task can use nested journal handles under JBD2's rules, and stopping a handle releases its participation rather than necessarily forcing an immediate transaction commit. A caller must check the return from the relevant start and stop operations. If the journal aborts after an I/O error, later operations can report failures; an earlier page-cache write does not erase that failure.

An unwritten extent is allocated but represents data that must not be exposed as initialized before the required conversion. The write path and completion/journal ordering coordinate that transition. The extent-status cache is useful for lookup but is not a second durable copy of the extent tree. For the example mapping logical block 10 to physical filesystem block 102, a later truncate or extent conversion can change the mapping; retaining that number as a permanent device address is unsafe. Inspect the mapping under the relevant locks and transaction state, and distinguish filesystem block units from lower block-device sectors.

A completed journal commit is not the same as checkpoint completion or application-level atomicity. If the device reports a write complete but loses volatile cache contents, even a theoretically ordered journal protocol can fail its persistence assumptions. The chapter's temporary-file lab does not power-cycle the guest or test replay. Use the separate durability chapter for the file-and-directory sync protocol and report crash outcomes only after an isolated image-based recovery experiment with a defined oracle.

Practice#

Source exercise: rg -n -A24 'int ext4_ext_map_blocks' "$KERNEL_SRC/fs/ext4/extents.c" and rg -n -A65 'int ext4_sync_file' "$KERNEL_SRC/fs/ext4/fsync.c" in a v6.12 checkout. Mark the lookup error, data-writeback error and journal/flush branches. Build a two-column timeline of commit versus checkpoint using fs/jbd2/commit.c.

Inside a disposable guest, the following uses only an automatically removed temporary file, not a mount or crash:

python3 - <<'PY'
import os, tempfile
with tempfile.TemporaryFile() as f:
    f.write(b'journal lesson\n')
    f.flush()
    os.fsync(f.fileno())
    f.seek(0)
    print(f.read().decode().strip())
PY

If the filesystem supports the operation, expect journal lesson. A flush or fsync error is a failed experiment and must be recorded. Python's flush transfers its userspace buffer to the kernel; fsync requests filesystem synchronization. Reading back is not proof of crash recovery, and the temporary directory's filesystem may not be ext4. Determine the guest filesystem with findmnt -T /tmp if available; report its type. The file is removed at close. Deliver the extent arithmetic, sync timeline, and a list of assumptions required for a durability claim.

Knowledge check#

1. What do m_lblk and m_len describe in the extent lookup?

2. What is the difference between committing a journal transaction and checkpointing it?

3. How do ordered and journal data modes differ?

4. What must the application do if fsync reports an error?

5. What does the cited JBD2 commit path do when submitting transaction data buffers fails?

Answers#

1. The logical start block and requested number of blocks, respectively; they are not a raw device address.

2. Commit establishes a recoverable journal transaction under the storage contract; checkpoint moves journaled metadata to home locations and enables log-space reuse.

3. Ordered mode journals metadata with associated data ordered before metadata commit; journal mode also journals file data.

4. Treat the synchronization as failed and apply its recovery policy. A previous successful write does not erase the later error.

5. It calls jbd2_journal_abort with the error; the failed submission is not converted into evidence of a successful journal commit.

Checkpoint#

Save the extent calculation and timeline. Pass when you identify lookup failure, data writeback, journal commit, checkpoint and the separate directory durability issue. Re-read the source and correct the timeline if any are conflated. This is not certification.

Sources#

• Extent mapping, Linux v6.12.
• ext4 fsync implementation.
• JBD2 commit implementation.
• JBD2 recovery implementation.
• ext4 administration documentation.