Device mapper, MD RAID, encryption, and stacked storage

Before you start#

Read NVMe, SCSI, and storage. Linux v6.12 is the historical source baseline. This chapter's observation is read-only; all destructive array, encryption or failure experiments belong on disposable guest images. Stacking, target features, configuration and userspace tooling determine the actual path.

Learning goals#

Follow a device-mapper bio through a target and clone completion; explain table switching and teardown; contrast thin snapshots, encryption, integrity and MD RAID; reason about a failed component without treating redundancy as backup.

Source walkthrough#

Device mapper (DM) exposes a virtual block device backed by a table of targets. A mapped_device owns the active mapping and associated queue and lifecycle state; a dm_table describes target ranges; each dm_target has a type, target-specific private data and operations. Targets implement different transformations, such as linear mapping, encryption, integrity verification, or thin provisioning. A table is a mapping policy and associated references, not a copy of all data. Some targets use bio-based mapping and others request-based paths; the path below is specifically the bio-based implementation. See drivers/md/dm.c, drivers/md/dm-table.c, and drivers/md/dm-linear.c.

For a bio-based mapping, dm_submit_bio finds the active table and splits or clones work where necessary for target boundaries. __map_bio selects the target and installs clone_endio before invoking its mapping operation. One result means the target has taken responsibility; another means DM should submit the remapped clone. The result switch includes (GPL-2.0-only; surrounding cases omitted):

	case DM_MAPIO_REMAPPED:
		dm_submit_bio_remap(clone, NULL);
		break;

This status tells DM to submit the remapped clone. It is not a completion status and does not prove the lower device has accepted or completed I/O. In contrast, DM_MAPIO_SUBMITTED means the target has assumed ownership for submission. DM_MAPIO_KILL and DM_MAPIO_REQUEUE go through different cleanup and pending-I/O accounting in this function. The clone_endio callback obtains the clone status and can invoke the target's end-I/O callback before DM finishes its own bookkeeping and updates the original I/O. The original bio must not be completed while a necessary clone still has an outstanding operation.

For the linear target, linear_map switches the device and sector (GPL-2.0; lines from dm-linear.c):

	bio_set_dev(bio, lc->dev->bdev);
	bio->bi_iter.bi_sector = linear_map_sector(ti, bio->bi_iter.bi_sector);

The first statement selects the target's lower block device; the second applies the linear mapping to the sector. The function then returns DM_MAPIO_REMAPPED, so the core can submit it. It does not itself guarantee a successful lower-device read or write.

Worked mapping and a failed clone#

Suppose a linear target covers virtual sectors 0–999 and maps them to an underlying device starting at sector 5000. A bio beginning at virtual sector 120 and covering eight sectors maps to lower sector 5120 through 5127. This assumes the target begins at virtual sector zero and no other transformation applies. A request spanning a target boundary must be handled in pieces under the table and queue limits; applying the first target's arithmetic to all of it would be wrong. Sectors are 512-byte units in block-layer addressing, not necessarily the physical-sector size.

Assume DM makes one clone and the lower device later returns an I/O error. The clone's bi_status reaches clone_endio, which gives any target completion method its defined opportunity to interpret the result, then accounts for that clone in the enclosing dm_io. When required pieces are finished, the original bio is completed with the resulting status. The target may specify a different result or requeue behavior; inspect that target instead of assuming a single policy. If allocation of a clone or table lookup fails earlier, there is no successful lower I/O to report. The request's pages and DM bookkeeping must remain valid until every relevant completion has finished.

Suspension, table changes, and teardown#

A table can be replaced while a mapped device exists, but not by freeing a live table under concurrent users. dm_swap_table serializes with suspend_lock and checks that the mapped device is suspended before proceeding. The key guard in dm.c, GPL-2.0-only, is:

	/* device must be suspended */
	if (!dm_suspended_md(md))
		goto out;

The function initializes its error result to ERR_PTR(-EINVAL) before this check; failing the guard does not publish a new table. Its comment says the old table is returned to the caller for destruction after a successful swap. Suspend and resume have options and target callbacks, and the implementation coordinates in-flight work and target state. A successful suspend is a lifecycle boundary with specific semantics, not a generic filesystem fsync. A safe teardown must prevent new references, deal with deferred and in-flight I/O, and release the old table and targets only after their users are finished. Userspace tools coordinate much of this protocol; inventing an unsynchronized table pointer replacement is not equivalent.

Thin, crypt, verity, and trust#

The thin-provisioning target allocates blocks from a shared data pool using metadata mappings. Thin volumes and snapshots can share data until writes require different mappings. Pool exhaustion or metadata errors can cause I/O to wait or fail depending on target state and policy. A thin snapshot is not a separate backup: pool loss or shared metadata corruption can affect both origin and snapshot. See drivers/md/dm-thin.c.

dm-crypt encrypts data according to its configured cipher, key and sector mapping. A key must be protected and managed across activation, rotation, crash recovery and backups; the kernel target does not magically protect a key from all privileged code in the running system. Encryption provides confidentiality under its configuration and threat model, but unauthenticated encryption alone does not establish that ciphertext has not been maliciously altered. An integrity mechanism must be selected and composed appropriately. See drivers/md/dm-crypt.c.

dm-verity verifies read-only data against a cryptographic hash tree rooted in a trusted digest; see drivers/md/dm-verity-target.c. It detects mismatches under its verification and trust assumptions, but does not encrypt the data and does not make an untrusted root digest trustworthy. Availability still requires handling a failed verification and possibly recovering from another trusted copy. Integrity and encryption answer different questions; neither is RAID redundancy. A practical system may stack them, but exact ordering and key/digest provisioning determine the resulting security properties.

MD arrays and failure#

Linux MD manages software arrays using personalities such as RAID1 and RAID5. struct mddev holds array-level state and a personality's operations decide how to handle the bio. In md_handle_request, a suspended array either waits or, for REQ_NOWAIT, returns a would-block error. The function obtains a live active_io reference before calling the personality's make_request, and releases it afterward. A reference here protects the relevant active-I/O lifecycle; it does not mean the device has completed a physical write. Different personalities implement mirroring, striping and parity differently.

For a hypothetical two-device RAID1 array, a read can be served from an eligible mirror, while a write has to be coordinated across the required members under the personality's policy. If one component fails, the array may remain available in a degraded state; a later replacement can be rebuilt from surviving data. Whether a specific request is retried or fails depends on the actual error, array state and implementation. The degraded state reduces fault tolerance, and an undetected or unrecoverable error on the remaining member can prevent recovery. RAID redundancy does not protect against accidental deletion, malicious modification, shared corruption or every controller failure; a separately tested backup addresses a different risk.

RAID5 stripes data and parity across members. A partial-stripe write can involve reading old data/parity and writing new data/parity, depending on the implementation and workload. The parity write hole is the risk that a crash between related updates leaves parity inconsistent with data. MD has mechanisms and options intended to address aspects of this problem, such as bitmap, journal or other supported policies, but a bitmap alone should not be described as making all data writes atomic. Inspect the configured personality and relevant documentation; drivers/md/raid5.c contains RAID5 implementation. Rebuild restores redundancy from surviving members; it is not equivalent to restoring a historical known-good backup.

Table ranges, completion, and accounting#

A DM table can have more than one target covering separate virtual ranges. The target's begin and length determine which mapping applies; a linear target's lower starting sector is relative to that target's virtual beginning. For example, if a target begins at virtual sector 100 and maps to lower sector 5000, virtual sector 120 maps to lower sector 5020, not 5120. The earlier example used a target beginning at zero. A bio that crosses from one target range into the next may need to be split so each clone obeys the appropriate mapping and target limits. This is a reason to record the table layout before doing sector arithmetic.

In drivers/md/dm-linear.c, v6.12, SPDX GPL-2.0-only (surrounding code omitted), the sector translation is a helper that linear_map calls:

static sector_t linear_map_sector(struct dm_target *ti, sector_t bi_sector)
{
	struct linear_c *lc = ti->private;

	return lc->start + dm_target_offset(ti, bi_sector);
}

dm_target_offset subtracts the target's begin from the incoming sector, so the lower start is added to an offset relative to that target, not to the table's first sector. The helper does not split requests that cross target boundaries; that happens before linear_map runs.

The target map result is an ownership boundary. With DM_MAPIO_SUBMITTED, the target has taken responsibility for the I/O. With DM_MAPIO_REMAPPED, the core submits the remapped clone. With a kill or requeue result in the cited function, DM frees target-I/O bookkeeping and decrements the pending count using the corresponding status. Neither the upper bio nor the old table can be freed merely because one mapping callback returned: pending lower operations and their end callbacks still govern lifetime. clone_endio can also ask a target-specific end-I/O method how to handle completion, so a generic diagram is not a substitute for reading that target.

Consider an array with two cloned operations, one for each lower range. If one succeeds and the other fails, a correct upper completion must reflect the relevant failure after all required completions are accounted for. It cannot declare success when the first clone happens to finish. If a target chooses requeue, the request must remain owned and be retried under the applicable policy; an unbounded retry can affect availability. Multipath and thin provisioning can deliberately queue under some conditions, making operational timeouts and recovery policy important.

Failure domains and repair#

For MD RAID1, the surviving member can permit continued operation after one member failure, but this does not prove every block on that survivor is readable. Rebuild reads surviving data and writes it to the replacement. An unreadable source region, a second failure, or an interrupted rebuild can leave the array degraded or data unavailable. The specific array metadata, bitmap and recovery settings affect behavior. A backup held outside the same failure domain is necessary for recovering an earlier version after a user deletes or overwrites a file.

For parity RAID, a full-stripe write and a partial-stripe update can require different read/modify/write work. If power fails between data and parity updates, parity can disagree with data; which recovery mechanism can detect or correct it depends on configuration. A write-intent bitmap helps locate regions needing resynchronization, but does not by itself provide a complete transactional record of every data and parity value. A RAID journal or other consistency mechanism has its own setup and failure boundaries. Do not equate an array marked clean with independently verified application data.

The same failure-domain reasoning applies to crypt and verity. A valid verity root hash is useful only if it is obtained from a trusted channel and associated with the intended image version. Encryption keys need a recovery and rotation plan; losing the only key can make intact ciphertext unavailable. Authenticated encryption or integrity targets can add verification, but their metadata, key management and stacking order matter. None of those mechanisms restores a deleted file without an independent copy. An inventory command can reveal names and levels, but it does not establish that backups exist or that a key is recoverable.

Practice#

Source-only route:

git -C "$KERNEL_SRC" describe --tags --exact-match
rg -n -A48 'static void __map_bio\(' "$KERNEL_SRC/drivers/md/dm.c"
rg -n -A12 'int linear_map\(' "$KERNEL_SRC/drivers/md/dm-linear.c"
rg -n -A34 'bool md_handle_request\(' "$KERNEL_SRC/drivers/md/md.c"

Expect v6.12, the target result switch, the two linear mapping statements, and the suspended/active-I/O checks. Work through virtual sector 120 with lower start 5000; mark where a clone error returns to original-I/O accounting. For an MD suspended bio with REQ_NOWAIT, identify the would-block path rather than claiming it waited.

Read-only guest inventory:

printf '%s\n' 'MD status:'
if [ -r /proc/mdstat ]; then cat /proc/mdstat; else echo 'not visible'; fi
printf '%s\n' 'DM and MD block devices:'
for p in /sys/block/dm-* /sys/block/md*; do
  [ -e "$p" ] || continue
  printf '%s\n' "$p"
  for f in dm/name md/level md/array_state; do
    [ ! -r "$p/$f" ] || printf '  %s: %s\n' "$f" "$(cat "$p/$f")"
  done
done

/proc/mdstat may show no configured arrays; no dm-* or md* entries is a valid result. Container visibility and permissions can hide devices. A DM name or MD level does not prove a particular encryption key, integrity mode, health, or successful recovery. Do not run mdadm --create, cryptsetup, dmsetup mutation, format, mount, or fail a component on the host. Deliver the sector calculation, one clone-error timeline and an inventory or explicit absence.

Knowledge check#

1. What does DM_MAPIO_REMAPPED ask DM to do, and is it a successful completion?
2. Where does virtual sector 120 map when a linear target starts at virtual zero and lower sector 5000?
3. Why must dm_swap_table require suspension, and what happens if its guard fails?
4. Does encryption by itself establish integrity? Does a thin snapshot establish backup?
5. What does REQ_NOWAIT do in the illustrated suspended MD path?

Answers#

1. DM submits the remapped clone; lower completion has not occurred yet.
2. Lower sector 5120, assuming that specified target and no other mapping.
3. The table and target lifetimes must be coordinated with I/O. The checked function returns its error result rather than swapping when the device is not suspended.
4. No to both: confidentiality and integrity are distinct, and a snapshot sharing a pool can share its failures.
5. It invokes the would-block error path rather than waiting for the suspended array to resume.

Checkpoint#

Save the mapping arithmetic, error timeline and inventory. Pass when you can distinguish target acceptance, clone completion, table lifetime, RAID rebuild, encryption, verification and backup. If a boundary is unclear, inspect __map_bio, clone_endio and md_handle_request and revise the timeline. This is practice, not certification.

Sources#

• Device mapper core, Linux v6.12.
• Linear target.
• Device mapper tables.
• Thin target.
• dm-crypt target.
• dm-verity target.
• MD core.
• RAID5 implementation.
• Thin provisioning kernel documentation.
• dm-crypt kernel documentation.
• dm-verity kernel documentation.
• MD RAID kernel documentation.
• dm-raid kernel documentation.