NVMe, SCSI, SATA, and storage-driver lifecycles

Before you start#

Read bio, scatter-gather, and DMA. The source baseline is Linux v6.12; hardware queues, transports, interrupt modes, configuration and virtualization differ. A QEMU virtio-blk disk is not an NVMe controller. The observation below is read-only and explicitly handles absent devices.

Learning goals#

Follow a PCI NVMe request through submission and completion, contrast SCSI command dispatch and error handling, explain the SATA/libata relationship, and identify lifetime and recovery boundaries.

Source walkthrough#

The block layer presents requests, but a storage driver translates them into a protocol appropriate for its controller. PCI NVMe uses submission queues (SQs) and completion queues (CQs) in host memory. A submission entry describes a command; a completion entry returns a command identifier (CID), status and result. The driver updates a queue tail and writes a doorbell register to notify the controller when required. A CQ phase bit helps distinguish newly written entries from old entries when the ring wraps. Queue depth, number of queues, interrupt vector allocation and polling are negotiated or configured; they are not universal constants. See drivers/nvme/host/pci.c and include/linux/nvme.h.

nvme_queue_rq is the PCI driver's blk-mq callback. It obtains the driver queue from hctx->driver_data and the request's NVMe private data using blk_mq_rq_to_pdu. Its first safety checks include this excerpt (GPL-2.0-only; surrounding code omitted):

	if (unlikely(!test_bit(NVMEQ_ENABLED, &nvmeq->flags)))
		return BLK_STS_IOERR;

A disabled queue cannot accept the request. The function also checks controller readiness and calls nvme_prep_rq; a preparation failure is returned. On successful preparation it takes sq_lock, copies the command to the submission queue, writes the submission doorbell according to the batching state and releases the lock. BLK_STS_OK is acceptance, not a successful storage transfer. Preparation and DMA mapping have their own error cleanup and configuration-dependent details.

On the completion side, nvme_handle_cqe reads the command ID from a CQ entry. Asynchronous event notifications are special and are not ordinary request completions. For a normal command, it looks up the request and handles a missing ID:

	req = nvme_find_rq(nvme_queue_tagset(nvmeq), command_id);
	if (unlikely(!req)) {

The following source warns about an invalid completed ID and returns; it does not dereference an absent request. Otherwise nvme_try_complete_req, batching and nvme_pci_complete_rq participate in completion. The exact disposition depends on status, multipath, retry policy and queue state; read drivers/nvme/host/core.c as well as the PCI transport. MSI-X is a common notification mechanism where configured and available, but queues can be polled, and not every completion produces a dedicated interrupt.

An NVMe request and its failure paths#

Imagine a request has been assigned tag 7 on an enabled I/O queue. The driver prepares an NVMe command, maps the request data under its DMA contract, writes a command whose CID identifies the request, and rings the appropriate doorbell. The controller later places a CQ entry with a matching CID and phase. The driver consumes the entry, finds the request through the tag set, interprets status and completes or otherwise handles it. When the CQ wraps, the phase changes so old entries are not treated as fresh completions. The numeric CID in a real request includes driver-defined encoding and generation details; the example does not assert that it is always just the integer 7.

Suppose the completion is delayed. nvme_timeout checks controller and PCI state and polls for a potentially missed completion before taking further recovery actions. If polling discovers that the request is no longer in flight, the timeout handler returns BLK_EH_DONE for that case. Other states can lead to abort, reset, deletion or timer handling; they are not one unconditional reset sequence. A request may be requeued when the relevant NVMe core policy permits retry, or failed when it does not. Device removal and reset must coordinate queues, DMA and in-flight completions before associated memory and controller state are released. A request reference or tag is not permission to use a controller after its teardown contract ends.

SCSI: command, host, and completion#

SCSI is a command architecture and Linux midlayer, not merely an old physical connector. A scsi_cmnd describes a command descriptor block (CDB), result and associated request/device state. scsi_device identifies a logical unit in the Linux model; Scsi_Host represents the host adapter, and its host template supplies low-level callbacks. Commands can be transported by different physical and virtual transports. A disk visible as /dev/sdX can therefore sit behind different adapters or translation layers; its name alone does not identify a wire protocol.

In scsi_dispatch_cmd, the midlayer checks device and host state, then calls the low-level queuecommand. The removed-device branch is explicit (GPL-2.0-only; surrounding code omitted):

	if (unlikely(cmd->device->sdev_state == SDEV_DEL)) {
		/* in SDEV_DEL we error all commands. DID_NO_CONNECT
		 * returns an immediate error upwards, and signals
		 * that the device is no longer present */
		cmd->result = DID_NO_CONNECT << 16;
		goto done;
	}

The done path calls scsi_done; the command is completed with a no-connect host result instead of sent to a deleted device. A blocked device can return a busy indication and be requeued rather than being treated as a successful hardware command. A low-level driver that accepts the command later invokes the appropriate completion interface. Completion is guarded against duplicate completion in scsi_done_internal, and scsi_io_completion evaluates result and good bytes. Sense data can describe a target's condition; retry decisions depend on status, sense, command policy and remaining retries. A partial result can complete good bytes and leave a remainder to retry or fail. The error-handling thread and escalation in drivers/scsi/scsi_error.c handle additional timeouts and recovery, not every successful request.

For a hypothetical SCSI read of 8192 bytes, a host adapter may report 4096 good bytes and a condition affecting the remainder. The midlayer's completion logic accounts for good bytes and evaluates whether remaining bytes can be retried or must fail. It must not report a complete successful read merely because the command was queued. If the device has entered SDEV_DEL before dispatch, the source path above returns a no-connect result without calling the low-level queue callback. This is a source-based scenario, not a measured hardware event.

SATA and topology#

SATA uses ATA commands and a SATA transport, while Linux libata can expose ATA disks through SCSI translation to the SCSI midlayer. drivers/ata/libata-scsi.c contains the translation layer. Seeing a SCSI device in sysfs does not imply the physical disk uses native SCSI commands end-to-end. Similarly, NVMe-over-fabrics has transports and failure modes different from PCI NVMe even though the NVMe core shares concepts. A virtual disk may use virtio, emulated SATA, emulated NVMe, or another configured device. Identify the actual bus and driver before selecting a trace.

An interrupt is a notification, not the complete lifetime of a request. NVMe polling can consume completions without the normal interrupt path, and SCSI low-level drivers have their own completion mechanisms. Reset or unplug can race with a completion already being processed. The driver must stop new work, synchronize in-flight users and DMA, and finish or cancel outstanding requests according to the applicable code. Freeing an SQ, CQ, command buffer or request-private data merely because an administrative reset was requested is unsafe if hardware or a completion handler can still use it.

NVMe queue state and concurrency#

An NVMe submission queue and completion queue are circular buffers, so head and tail movement need synchronization with both CPUs and the controller. In the PCI driver, sq_lock protects the code that copies the command and updates the submission doorbell in the displayed path. This lock is not the lock for every controller state, request reference, or completion queue. Queue setup, interrupt handling, reset and removal have their own synchronization. A driver must also respect ordering between placing a command in shared memory and making it visible to the controller; copying a structure into memory without the proper device protocol is not a complete submission.

The CQ phase bit lets the consumer decide whether a slot belongs to the current pass through the ring. When the head wraps, the expected phase toggles. The controller and driver still need correct memory ordering when reading the status and associated completion fields. A CID identifies a command in a queue context; the PCI code finds the request through the associated tag set and validates that a request exists. A stale or invalid completion cannot be treated as permission to access arbitrary memory. AENs are handled separately because they do not have an ordinary request in the same sense.

In drivers/nvme/host/pci.c, v6.12, SPDX GPL-2.0 (surrounding code omitted), the completion-queue check is an inline predicate:

static inline bool nvme_cqe_pending(struct nvme_queue *nvmeq)
{
	struct nvme_completion *hcqe = &nvmeq->cqes[nvmeq->cq_head];

	return (le16_to_cpu(READ_ONCE(hcqe->status)) & 1) == nvmeq->cq_phase;
}

The low bit of the status word is the phase tag, and the predicate compares it with the driver's expected phase for the current pass through the ring. This excerpt does not advance the head, read the other completion fields, or show the ordering around those reads.

Suppose a timeout fires just as a completion is arriving. The timeout code polls, then checks whether the request is still in flight. If the completion was processed, the handler can return without starting the same recovery action on an already-completed request. If it remains in flight, controller state and other conditions determine the next action. PCI error recovery can also make an immediate reset inappropriate, so the handler may reset its timer. This race illustrates why timeout, completion and reset cannot each independently free the request and its DMA state.

SCSI results and recovery are layered#

A SCSI result has information from multiple layers: the transport or host can report connectivity problems, while the target can return status and sense data describing a command condition. The command's successful submission to queuecommand is distinct from the target's later response. A busy return at dispatch means the low-level driver did not accept the command in that call; it is not a completed successful read. scsi_dispatch_cmd converts unexpected busy return values into a host-busy classification in the shown code, preserving a defined result for the midlayer.

In scsi_io_completion, good_bytes tells the midlayer how much was correctly handled. It calls scsi_end_request to account for progress, then handles a remaining portion according to result and retry rules. An error with a no-retry command can cause the remaining bytes to be ended with an error. When a command has no reported error but bytes remain, the code can requeue that remainder. This does not mean every sense condition should be retried indefinitely. Retry limits, error-handler state, device removal and command semantics constrain the decision.

A recovery sequence can escalate from command-specific handling to an abort or broader reset, but the actual low-level driver and transport define what can be attempted. During that time, new commands may be blocked or queued and already-completed commands still need their callbacks and references handled. If the device is removed, the midlayer must avoid continuing to issue new commands to a deleted object. The SDEV_DEL branch is one observable source boundary; it is not a complete proof that all possible removal races are handled by that one check.

Choosing a useful observation#

A QEMU guest with only a virtio-blk device might show vda under /sys/block and no NVMe controller or SCSI logical unit. That is an expected topology, not a failed source exercise. A guest configured with emulated NVMe can expose an NVMe controller and namespace, but model strings alone do not show how many hardware queues or interrupt vectors were configured. A device reachable through a SCSI host can be virtual, SATA via libata, USB mass storage, or another transport. Follow sysfs driver and bus links and the guest's device configuration before attributing a protocol.

For a reproducible performance experiment, record the guest kernel configuration, controller model and QEMU version, number of queues, CPU count, scheduler and workload. The SQ/CQ discussion predicts where to look, not a throughput number. A timeout or reset experiment requires a disposable guest, a recoverable backing image and explicit fault injection; merely reading sysfs does not trigger or prove the recovery sequence. If the device is absent, the source-only diagrams remain valid as conditional descriptions of the code.

Practice#

Source-only route:

git -C "$KERNEL_SRC" describe --tags --exact-match
rg -n -A32 'static blk_status_t nvme_queue_rq' "$KERNEL_SRC/drivers/nvme/host/pci.c"
rg -n -A40 'static inline void nvme_handle_cqe' "$KERNEL_SRC/drivers/nvme/host/pci.c"
rg -n -A75 'static int scsi_dispatch_cmd' "$KERNEL_SRC/drivers/scsi/scsi_lib.c"

Trace an enabled NVMe queue and the disabled-queue error separately. Find the special AEN branch and the invalid-CID guard. In SCSI, mark SDEV_DEL, blocked-device return, and low-level queuecommand. Do not infer a live device from source alone.

Read-only guest observation:

printf '%s\n' 'NVMe controllers:'
for p in /sys/class/nvme/nvme*; do
  [ -e "$p" ] || continue
  printf '%s: ' "$p"; [ ! -r "$p/model" ] || cat "$p/model"
done
printf '%s\n' 'SCSI logical units:'
for p in /sys/class/scsi_device/*; do
  [ -e "$p" ] || continue
  printf '%s\n' "$p"
  for f in vendor model type; do [ ! -r "$p/device/$f" ] || printf '  %s: %s\n' "$f" "$(cat "$p/device/$f")"; done
done
printf '%s\n' 'Block devices:'
ls /sys/block

An empty NVMe list means none is visible in this namespace/configuration, not that the kernel lacks NVMe support. An empty SCSI list is likewise a valid result. The displayed device model and path do not establish a request trace, command status, or physical-vs-virtual hardware by themselves. Do not issue format, reset or firmware commands. Deliver a two-column source diagram of NVMe and SCSI acceptance/completion and your observed lists or absence.

Knowledge check#

1. What does the NVMe submission doorbell communicate, and is ringing it completion?
2. Why does nvme_handle_cqe check for a missing request after reading the CID?
3. What does the SCSI SDEV_DEL branch do before it calls scsi_done?
4. Can a SATA disk appear through Linux's SCSI layer, and why?
5. Why is polling relevant to an NVMe timeout?

Answers#

1. It notifies the controller about submission-queue progress; device execution and CQ completion happen later.
2. A completion identifier may not resolve to a live request; the code warns and returns instead of dereferencing a missing request.
3. It sets a DID_NO_CONNECT host result and completes the command through the done path rather than dispatching it to the low-level driver.
4. Yes. libata translates between the SCSI-facing interface and ATA operations for supported devices.
5. The handler checks for a completion that might have been missed before proceeding with further timeout recovery.

Checkpoint#

Keep both source diagrams and the read-only device inventory. Pass when each diagram separates acceptance, actual completion, retry/failure, and teardown, and when you can explain an absent NVMe device without calling a virtio disk NVMe. Revisit the two driver functions and correct your diagram if needed. This is practice, not certification.

Sources#

• PCI NVMe driver, Linux v6.12.
• NVMe core.
• NVMe protocol structures.
• SCSI dispatch and completion.
• SCSI error handling.
• libata SCSI translation.