Buffered I/O, direct I/O, and the file-operation boundary

Before you start#

Read pathname resolution, and revisit the page cache if needed. All source references use Linux v6.12, a historical baseline. Filesystem, architecture, alignment and configuration affect the applicable path.

Learning goals#

Trace the VFS read boundary into a generic cached read, distinguish it from iomap direct I/O, and reason correctly about short results and alignment failure.

Source walkthrough#

A descriptor identifies an open struct file; its f_op points to filesystem or device operations. The VFS checks whether the open permits the requested operation and dispatches to a suitable method. For a regular file, the filesystem chooses its implementation. The read boundary in fs/read_write.c, vfs_read contains (GPL-2.0; other checks omitted):

	if (file->f_op->read)
		ret = file->f_op->read(file, buf, count, pos);
	else if (file->f_op->read_iter)
		ret = new_sync_read(file, buf, count, pos);

The older read method is preferred when present; otherwise new_sync_read constructs a kiocb and an iov_iter for the iterator method. Before dispatch, vfs_read checks file mode, user address access and the requested region. After a positive return it accounts the bytes and notifies relevant watchers. A driver or filesystem can provide a different method: VFS dispatch alone does not prove a disk operation occurred. The corresponding write path in that file has its own mode checks, write accounting and file-operation dispatch.

For files using generic_file_read_iter, mm/filemap.c routes normal cached reads to filemap_read, which locates folios, initiates reads when necessary and copies available bytes to the destination iterator. The file position advances by bytes transferred, not by the requested count. A read of 4096 bytes at offset 999 in a 1000-byte file can return one byte and advance to 1000; the next read returns zero at EOF. A positive short read is not automatically an error. A caller needing a complete record must loop, handle zero, and decide what an interruption or error means.

The generic iterator also contains a direct-I/O branch. Its final choice includes (GPL-2.0-only; preceding direct-I/O code omitted):

		if (retval < 0 || !count || IS_DAX(inode))
			return retval;
		if (iocb->ki_pos >= i_size_read(inode))
			return retval;

If none of those conditions holds, the implementation falls through to filemap_read for the remainder. The source comment names compressed Btrfs extents as one reason a direct read can be short. This is a property of this generic implementation; a filesystem may use a different iterator. DAX has separate direct access semantics and does not use that buffered fallback in this branch.

The sequence below follows a regular file that uses the generic iterator. Filesystems with their own file operations take a different route.

Direct I/O, iomap, and ownership#

Direct I/O generally transfers data without the ordinary page-cache data-copy route, but it is not a universal promise to bypass every cache in the machine. Filesystems must coordinate overlapping cached and direct operations. In the generic branch, kiocb_write_and_wait is called before mapping->a_ops->direct_IO, so existing dirty data in the affected range is handled before the direct read; an error stops that path. fs/iomap/direct-io.c implements iomap direct-I/O support used by participating filesystem paths; iomap describes logical ranges and backing mappings. A filesystem supplies mapping operations and may use iomap for buffered or direct operations. Do not assume all filesystems use the same method or that the cited generic direct_IO callback is an iomap call chain.

Direct I/O can require alignment of the file offset, memory address and length; requirements vary by filesystem and device. Some combinations fail with EINVAL, and some filesystem/kernel combinations can fall back to buffered I/O. Linux's open(2) documentation specifically warns that misaligned O_DIRECT I/O may either fail or fall back to buffered I/O, depending on filesystem and kernel version; applications must not infer which occurred from O_DIRECT alone. O_DIRECT is not an fsync substitute: requesting a direct data path does not itself establish the full ordering and durability of file data, metadata, directory entries and volatile device caches. Use the filesystem's documented synchronization contract, and check errors from both writes and sync calls. See open(2) and fsync(2).

For a hypothetical writer supplying 8192 bytes, suppose the call returns 4096. The application owns the remaining bytes and must decide whether and how to retry from the next offset. If a later call fails, it must not report that all 8192 bytes were written. If a direct request has an improperly aligned offset and is rejected, no successful transfer may be assumed merely because the descriptor was opened. Asynchronous interfaces add a further distinction between submission and completion; the buffer must stay valid under the API's lifetime contract until completion.

Position, partial progress, and interfaces#

A synchronous read of a seekable file normally uses the open file description's current position; pread supplies an explicit offset and leaves that shared position alone. Stream devices and pipes have different position semantics. The VFS dispatch point handles several types of file, so it cannot assume every operation is backed by an inode with ordinary disk blocks. A file operation can return an error before transferring bytes, or a positive byte count when some progress was made. The caller must consult that particular interface's rules instead of assuming that every short result means the same cause.

Suppose an application reads an eight-byte header from a stream. A return of three bytes leaves five bytes to obtain; it does not imply the next read will return those five immediately, nor that a later zero return can be ignored. Maintain a buffer offset of three, request only the five remaining bytes, and handle interruption, error or EOF according to the protocol. For a regular file, EOF is a natural explanation of a short read; it is not the only possible explanation in all environments. write has the same practical requirement to check the returned count. Even when an application loops successfully, buffered data can still fail later during writeback or fsync.

Direct I/O also has a lifecycle. A synchronous call does not return successful completion until its interface's synchronous work is complete, while an asynchronous request can return a submission result before a completion is delivered. The latter requires the buffer and any relevant metadata to remain valid for the period specified by the interface. Overlapping buffered and direct writes require filesystem coordination and application care; blindly mixing them can cause confusing performance or consistency behavior. Direct I/O's alignment requirements must be discovered for the target filesystem and kernel rather than guessed from a typical 4096-byte page size. Device sector sizes, filesystem constraints and user buffer alignment are separate inputs.

What the filesystem adds#

The iov_iter is a cursor over the caller's I/O buffers, while kiocb carries I/O state including the file, position and flags. They make it possible for an iterator method to operate over several kinds of buffer without each filesystem reimplementing the syscall interface. They do not eliminate the need to handle a partial copy or a changed iterator state. generic_file_read_iter records the number returned by direct I/O, updates the position on a nonnegative result, and adjusts the iterator as the code requires before deciding whether to return or fall through. Inspecting just the final filemap_read call would hide this state management.

Suppose a direct read requests 8192 bytes and returns 4096 before the end of a file for a reason where the implementation permits buffered fallback. The generic code has already advanced the I/O position by the direct result and reduced the remaining count. It may then call filemap_read for the rest, carrying the earlier result. This is a hypothetical illustration of the conditions in that implementation, not a guarantee for every filesystem or every short direct read. An error, no remaining count, DAX, or EOF takes a return branch instead. The Btrfs compressed-extent comment explains one possible short direct read; it does not prove that all compressed reads follow this path.

A user buffer can be valid when the syscall starts and still encounter a fault while bytes are copied. Some interfaces can return a positive count for the portion already transferred, while others report an error if no progress was made; read the implementation and syscall contract. An application must not fill the remainder of a data structure with assumed bytes after a short transfer. Similarly, a write returning a count means that the interface accepted that many bytes under its semantics, not necessarily that a remote endpoint consumed them or that a disk persisted them.

For a storage experiment, an actual O_DIRECT probe would need an aligned buffer, aligned offsets and lengths, a suitable guest filesystem and a method to discover its constraints. Python's ordinary bytes object does not guarantee the needed address alignment. A failure to satisfy those prerequisites is an experiment setup problem, not evidence that all direct I/O is broken. The safe read lab deliberately tests only the observable short-read contract.

Iterator state and direct-I/O completion#

vfs_read checks the file mode and user range before selecting a file operation. new_sync_read adapts a synchronous request to a kiocb and iov_iter; the filesystem's read_iter owns the applicable method behavior. A kiocb carries the file, current position and flags, while the iterator describes the remaining buffers and advances as data are copied or transferred. A positive short count represents progress; an error before progress can be negative. A caller requesting eight bytes and receiving three must advance its application buffer by three before asking for the remaining five. For an ordinary shared file position, the position advances by the successful count; pread supplies a separate offset.

In new_sync_read, v6.12, SPDX GPL-2.0 (surrounding code omitted):

static ssize_t new_sync_read(struct file *filp, char __user *buf, size_t len, loff_t *ppos)
{
	struct kiocb kiocb;
	struct iov_iter iter;
	ssize_t ret;

	init_sync_kiocb(&kiocb, filp);
	kiocb.ki_pos = (ppos ? *ppos : 0);
	iov_iter_ubuf(&iter, ITER_DEST, buf, len);

	ret = filp->f_op->read_iter(&kiocb, &iter);
	BUG_ON(ret == -EIOCBQUEUED);
	if (ppos)
		*ppos = kiocb.ki_pos;
	return ret;
}

The wrapper builds the kiocb and iov_iter, and writes kiocb.ki_pos back to the caller's position only when a ppos pointer was supplied. The BUG_ON shows that this synchronous adapter does not expect -EIOCBQUEUED. The excerpt does not show the mode checks in vfs_read or what a read_iter method does with a short copy.

IOCB_NOWAIT requests a nonblocking path where supported; needing a lock, allocation or I/O that cannot proceed immediately can produce -EAGAIN. An internal -EIOCBQUEUED return from an asynchronous kernel path means completion will occur later through its callback contract; it is not a byte count and is not automatically the userspace-visible return from every API. The specific synchronous wrapper and async interface determine what userspace sees. A short read at EOF, a NOWAIT refusal and a later async completion are different states. Inspect fs/read_write.c and fs/iomap/direct-io.c.

In v6.12 iomap_dio_bio_end_io, GPL-2.0, the initial completion handling is:

	if (bio->bi_status)
		iomap_dio_set_error(dio, blk_status_to_errno(bio->bi_status));
	if (!atomic_dec_and_test(&dio->ref))
		goto release_bio;

A failing bio records an errno in the shared DIO operation; the helper uses compare-and-exchange to retain an error when submission and completions race. The reference decrement determines whether all relevant pieces have completed; if not, this bio is released without completing the whole DIO. The last completion can wake a synchronous waiter or arrange inline, caller-deferred or workqueue completion according to flags. iomap_dio_complete calls the filesystem completion method when supplied, handles short read/EOF and position, ends inode DIO accounting and frees the DIO object. Buffer/page references and the kiocb must survive the applicable async operation until that completion contract ends.

For a hypothetical direct read starting at offset 1000, request 4096 bytes from a file whose relevant EOF is 1500. A successful short read can return 500 and advance that operation's position to 1500. A subsequent read can return zero. If the operation is queued asynchronously, the buffer cannot be reused just because submission returned; wait for the completion result. A negative completion is not a 500-byte success. These are conditional examples, not a claim that a particular filesystem supports the request's alignment.

Direct I/O alignment can constrain buffer address, file offset and length, varying with filesystem and kernel. Where supported, statx with STATX_DIOALIGN can report relevant constraints, including memory and offset alignment fields in the v6.12 UAPI; absent support or an unset result is not a universal 4096-byte rule. Direct I/O avoids the ordinary buffered data-copy path in applicable implementations but is not automatically zero-copy end-to-end: page pinning, bounce buffers, device mappings and filesystem work can still occur. It does not itself guarantee durable writes. For syscall semantics, distinguish a successful short read (which returns the bytes read and advances the file offset by that count) from a negative error; a zero return means EOF for a nonzero request on an ordinary regular file. See read(2).

Mixing overlapping buffered, mmap and direct I/O needs the filesystem's coherence rules. The generic read branch waits on relevant dirty data before direct read; iomap direct-write completion attempts post-write page-cache invalidation after filesystem end-I/O, including possible unwritten-extent conversion. Its source comment explicitly describes limitations when clean pages are repopulated by readahead or faults. Truncate and invalidation locks coordinate particular races, not a blanket guarantee that arbitrary concurrent application writes are ordered. The existing ordinary-file lab tests short read semantics without issuing direct I/O to a host device.

Buffer ownership across asynchronous completion#

The iomap DIO object aggregates multiple bios, their error state and a reference count. A submitted bio can complete on a different CPU while the submitting task is still constructing work, so the first recorded error uses a compare-and-exchange. Only the final reference transition enters the operation-level completion branch. Individual bio cleanup and whole-operation cleanup therefore occur at different times. A filesystem end-I/O callback can convert unwritten extents or perform other metadata work before the final result is delivered; this is why an async buffer cannot be reused at the earlier submission boundary.

In iomap_dio_set_error, v6.12, SPDX GPL-2.0 (surrounding code omitted):

static inline void iomap_dio_set_error(struct iomap_dio *dio, int ret)
{
	cmpxchg(&dio->error, 0, ret);
}

Because cmpxchg replaces only a zero value, the first errno recorded in dio->error wins. Later bios can still complete and drop references, so the stored value is not a count of failed bios. The excerpt does not show how the final completion turns that value into the caller's result.

The flow below orders the completion decisions in iomap_dio_bio_end_io as read in v6.12, in source order.

For an overlapping direct write and cached read, the implementation coordinates page-cache writeback and invalidation at particular points, but the source explicitly describes cases where post-write invalidation can fail and the direct write still worked. Application synchronization is still needed to establish its intended ordering when it mixes interfaces. O_DIRECT is a request subject to filesystem and alignment behavior, not a way to eliminate all copies, caching or error handling. If statx does not report DIO alignment, record that lack of information rather than guessing a device-safe address or testing against a host block device.

Alignment is not the only source of a short result. A filesystem can encounter EOF, a user-buffer fault, a signal or a lower-layer error after some progress, and each interface defines how it reports the bytes already transferred. The result must be interpreted alongside the updated iterator and position; the requested count is not a receipt for that many bytes. The example's file-offset arithmetic is intentionally separate from any claim about a host device's direct-I/O support.

Practice#

Source exercise: run rg -n -A35 'ssize_t vfs_read\(' "$KERNEL_SRC/fs/read_write.c" and rg -n -A55 'generic_file_read_iter\(' "$KERNEL_SRC/mm/filemap.c" in a v6.12 checkout. Identify mode validation, iterator dispatch, direct branch and fallback. Compare with the filesystem's own file_operations before claiming it uses this function.

In a disposable guest run:

python3 - <<'PY'
import os, tempfile
with tempfile.TemporaryFile() as f:
    f.write(b'abc'); f.flush(); f.seek(0)
    print(repr(os.read(f.fileno(), 8)))
    print(repr(os.read(f.fileno(), 8)))
PY

Expected b'abc' then b'': a short successful read followed by EOF, not an eight-byte zero-filled record. The file is automatically removed. This experiment uses buffered I/O and does not measure the cache or direct I/O. For a failure case, run the separate command below and record EBADF; never treat the failed operation as a byte count. Deliver a trace of requested count, returned count, and offset for both reads, plus an alignment and durability checklist for a hypothetical direct I/O request.

python3 - <<'PY2'
import os, tempfile
with tempfile.TemporaryFile() as f:
    fd = os.dup(f.fileno())
    os.close(fd)
    try: os.read(fd, 1)
    except OSError as e: print(e.errno)
PY2

On Linux the expected error number is 9 (EBADF), because the duplicated descriptor was closed before the read.

Knowledge check#

1. What does vfs_read do if there is no read method but there is read_iter?

2. What does a positive return smaller than the requested count mean?

3. Why is direct I/O not automatically durable?

4. When can the generic direct-read branch fall through to a buffered read?

5. Why does one completed bio not necessarily finish an iomap DIO operation?

Answers#

1. It calls new_sync_read, which adapts the synchronous request to the iterator interface.

2. That many bytes were transferred; the caller must handle the remainder and a subsequent EOF or failure.

3. Bypassing the usual page-cache data path does not by itself synchronize all relevant metadata and storage caches.

4. The code falls through when the direct result is nonnegative, bytes remain, the file is not DAX and the position has not reached EOF. Applicability depends on the filesystem's operations.

5. Other bios or submission references can remain. The completion path records any error and decrements the DIO reference; operation-level completion occurs only when the relevant final reference is reached.

Checkpoint#

Save the read trace and direct-I/O checklist. Pass when the trace shows 3 then 0 returned bytes, and the checklist distinguishes alignment, cache coherence, completion and durability. Re-read the two dispatch functions and repeat the observation if uncertain. This is not certification.

Sources#

• VFS read and write, Linux v6.12.
• Generic read and page-cache path.
• iomap direct I/O.
• iomap operations documentation.