VFS objects, file descriptors, and open files

Before you start#

Read file mappings and the page cache. This chapter follows Linux v6.12 source. The Python observation is unprivileged and works on an ordinary x86-64 guest; descriptor numbers shown are examples, not guaranteed allocations. The tag is a historical teaching baseline, not a production update recommendation.

Learning goals#

Distinguish a name, inode, path, open file description and descriptor table. Follow the openat2 success and unwind paths, and predict how dup, file position and final close interact.

Source walkthrough#

Keep this ownership chain in mind as you read: a pathname is looked up to a path (mount plus dentry); that path is held by an open file description (struct file); and a descriptor number is a per-process table entry that refers to that description. dup adds another descriptor entry to the same description, while a separate open normally creates another description.

The Virtual Filesystem (VFS) presents common operations while allowing filesystems to implement them differently. A struct inode is the kernel's representation of a filesystem object: it includes metadata, operation pointers and a mapping for cached data. A struct dentry represents a name in a directory lookup context; a negative dentry can record a name without a corresponding inode. Multiple names can refer to one inode, as with hard links. A struct path pairs a dentry with a mount, because the same dentry can be reached through different mount attachments. An inode is not a pathname and a pathname can change while an open file remains usable. See include/linux/fs.h and include/linux/dcache.h.

An open file description is represented by struct file. It holds the opened path, operation table, status flags, credentials and, for ordinary seekable files, a current position. It has its own references and lifetime. A process's files_struct contains an fdtable mapping descriptor numbers to file pointers; tasks may share the table with CLONE_FILES. A descriptor is a small index into that table, not an inode number and not the open file itself. The descriptor's close-on-exec flag belongs to the table entry, while the open file's status flags and position belong to the shared description. The structures and table management are in include/linux/fdtable.h and fs/file.c.

The openat2 syscall validates the userspace open_how before reaching do_sys_openat2 in fs/open.c. That function converts the requested flags, copies the name through getname, reserves a descriptor and calls do_filp_open. Here is the central branch (GPL-2.0-only; surrounding lines omitted):

	fd = get_unused_fd_flags(how->flags);
	if (fd >= 0) {
		struct file *f = do_filp_open(dfd, tmp, &op);

The reserved number is not yet an installed file. do_filp_open performs pathname and open work; it returns either an open struct file or an encoded error pointer. The next fragment handles that distinction:

		if (IS_ERR(f)) {
			put_unused_fd(fd);
			fd = PTR_ERR(f);
		} else {
			fd_install(fd, f);

On failure, put_unused_fd returns the reserved slot and PTR_ERR turns the error pointer into the syscall's negative error. On success fd_install publishes the file in the slot. In both cases the later putname(tmp) releases the copied pathname. If descriptor reservation itself fails, no do_filp_open is attempted. If copying the name fails, there was no descriptor to release. This ordering prevents both leaked slots and installing an error pointer as a file. do_filp_open has its own path-walk and file cleanup responsibilities; the outer function cannot simply free an arbitrary dentry.

Worked example: shared position and final close#

Imagine a six-byte file containing abcdef. open returns descriptor 3 and a new file description F with offset 0. dup(3) creates descriptor 4 referring to F, not a new description. Reading two bytes through 3 returns ab and advances F's offset to 2. Reading two through 4 then returns cd and moves the same offset to 4. Closing 3 removes that descriptor's reference, but 4 still refers to F; reading through 4 returns ef. Closing 4 drops the remaining descriptor reference, although other kernel references can delay final release. In contrast, a separate open normally creates a second description with its own offset. pread supplies an explicit offset and does not advance the shared position. Device files and special files may have different seek behavior.

Reference counts answer when an object may be freed, not which mutations are safe without locks. A file can remain open after its name is unlinked: the name and the open reference have distinct lifetimes. Conversely, recording a bare struct file * without obtaining the required reference can become a use-after-free when another thread closes the descriptor. fs/file_table.c contains file-reference release machinery. Do not infer that the integer fd still names the same object after close: the slot can be reused.

Following references through failure and reuse#

The open_how validation step matters because flag combinations and structure size determine whether the request is valid before path lookup begins. Once the name is copied, the kernel has a stable kernel-side copy for that lookup, but the directory tree itself can still change. Descriptor reservation happens before opening, so the outer function has a specific cleanup obligation if lookup, permission checking, filesystem open, or a later operation returns an error. put_unused_fd handles that reserved number. fd_install on success transfers the file reference into the descriptor table; a caller must not also drop the same ownership as though installation had failed. The details of table publication and concurrency belong to fs/file.c.

Suppose a process closes fd 3, then opens an unrelated file and gets fd 3 again. The number is the same but the file description is different. A thread that merely stored integer 3 and uses it later can accidentally operate on the new object unless its application-level coordination prevents reuse. Similarly, after fork, parent and child typically have separate descriptor tables whose entries refer to the same open file descriptions; closing an entry in one table does not remove the other entry. Tasks created with CLONE_FILES, in contrast, can share a table, making descriptor-table changes visible among those tasks. These distinctions are separate from whether two descriptors share f_pos.

The lifetime of a dentry is also separate from the lifetime of a path string. The dentry cache can retain entries beyond a single lookup, and references held by a path or open file keep relevant objects alive according to their rules. A negative dentry is not an inode containing an empty file. A mount reference matters because lookup must know which attachment gives meaning to the dentry. This is why a diagram that draws only fd directly to filename hides essential relationships.

Open flags and file operation ownership#

The file description contains an operation table because an fd can designate more than a regular disk file. A pipe, socket, device node, directory and regular file can all be reached through descriptor APIs, but their supported methods and position semantics differ. The VFS chooses an operation through the file's methods and applies common checks. The filesystem or driver implements the object-specific behavior. A successful open therefore means that the selected implementation accepted that open under the conditions checked; it does not imply that a subsequent read, write, seek or sync will succeed.

O_CLOEXEC illustrates why descriptor state and file state must not be mixed up. It requests that the descriptor be closed across a successful exec. The close-on-exec setting is associated with a descriptor entry; duplicating a descriptor does not mean every duplicate has the same descriptor flags. By contrast, O_APPEND is a file status flag on the open file description and can be shared by descriptors referring to it. Consult the interface and operation-specific behavior before building an atomicity argument around append writes.

A realistic failure sequence can happen after path lookup. Suppose the kernel reserved fd 5 and found a pathname, but the filesystem's open method rejects the operation. do_filp_open returns an error pointer; do_sys_openat2 releases slot 5 and then the copied filename. A later unrelated open is free to reuse 5. There is no valid file to put into fd 5 in this case. The inner open machinery is responsible for releasing the partial path and file objects it acquired, while the outer function releases the resources it owns. When reviewing a change, annotate that boundary: releasing both layers' ownership twice is as incorrect as failing to release either.

An open descriptor is also a useful capability-like handle in an informal sense: it points at an already-opened object and remains meaningful even if its pathname changes. It is not, by itself, a complete security capability model. Subsequent operations can perform further checks, a filesystem can change state, and the descriptor's authority derives from the actual open mode and kernel rules. Passing an fd to another process shares access to the relevant open description; it does not transfer the pathname's historical meaning or a promise that all future operations will work.

Descriptor lookup: borrowed and counted references#

fget obtains an owned file reference for the caller to release with fput; the lookup and concurrent close rules are implemented in fs/file.c. fdget uses __fget_light, whose result is a struct fd with flags in its word. In v6.12, the unshared-files-table branch can borrow a pointer instead of incrementing f_count (GPL-2.0; preceding checks omitted):

		return BORROWED_FD(file);
	} else {
		file = __fget_files(files, fd, mask);

The else path uses a counted lookup and returns CLONED_FD when successful. FDPUT_FPUT records whether fdput must drop that reference; fdget therefore does not always increment f_count. The borrowed form has a limited scope: its source comment prohibits retaining the file across return to userspace, calling filp_close on it in that interval, or cloning the current task before the matching put. A borrowed pointer is not a general-purpose lifetime reference.

fdget_pos first obtains an fd result, then, when the file has FMODE_ATOMIC_POS and the source's sharing/iteration condition applies, takes file->f_pos_lock and sets FDPUT_POS_UNLOCK. fdput_pos unlocks when that flag is set and then performs the normal fdput. The lock serializes relevant shared-position operations; it is not a lock on the whole inode or every read/write implementation. pread uses an explicit position and has a different position contract. See include/linux/file.h.

Closing a slot and releasing the file#

The fdtable can be resized and published using RCU-aware mechanisms while other lookups run. Table updates use files->file_lock where required; lookup code must use its documented reference and RCU rules rather than retain a raw pointer to an old array. Closing a descriptor removes a table entry so the numeric slot can be reused, but other entries or kernel references can still hold the same struct file. A concurrent thread that saves only fd number 3 can accidentally act on a different file after close and reuse. A counted file reference has a different lifetime from that integer.

The v6.12 filp_close calls filp_flush, then fput; filp_flush invokes a file's flush method when present. This can happen for a close of each descriptor referring to a shared file, so flush is not the same as final release and can report an error for that close. When the final f_count is dropped, fput can schedule ____fput as task work; for an interrupt, kernel thread, or task-work failure it can use delayed work instead. __fput performs final cleanup and calls f_op->release when supplied. Some special paths use different synchronous handling, so "last close immediately runs release in the close syscall" is not universal. See fs/open.c and fs/file_table.c.

In fs/open.c, v6.12, SPDX GPL-2.0-only (the rest of the close implementation omitted), the close syscall takes a different route from filp_close:

SYSCALL_DEFINE1(close, unsigned int, fd)
{
	int retval;
	struct file *file;

	file = file_close_fd(fd);
	if (!file)
		return -EBADF;

	retval = filp_flush(file, current->files);

	/*
	 * We're returning to user space. Don't bother
	 * with any delayed fput() cases.
	 */
	__fput_sync(file);

Notice that the descriptor is cleared first, then the flush runs, and __fput_sync drops the reference without deferring release to task work. The excerpt does not show the restart-error mapping that follows, and it does not prove that every close releases the file synchronously: other references can keep the struct file alive.

In fs/file_table.c, v6.12, SPDX GPL-2.0-only (the preceding comment block omitted), the synchronous helper is:

void __fput_sync(struct file *file)
{
	if (atomic_long_dec_and_test(&file->f_count))
		__fput(file);
}

Notice that __fput runs only when this decrement removes the last count. The excerpt does not show __fput, the release method or the comment's warning about where this helper may be called.

Extend the earlier example: parent opens F at fd 3, duplicates it as 4, then forks. Parent and child have separate descriptor tables in the ordinary fork case, but their entries still refer to F and its shared offset. Parent closes 3 and opens G; the freed slot may be reused as fd 3. The child can still use its own fd 3 for F, while parent fd 3 now refers to G. Closing parent fd 4 does not destroy F while child references remain. FD_CLOEXEC is a per-descriptor flag checked at successful exec; file status flags and position belong to the open file description. A process sharing a table via CLONE_FILES has a different table-sharing relationship.

A useful failure analysis distinguishes three events: descriptor reservation can fail before an open, do_filp_open can fail and cause the reserved slot to be returned, and a later close can report a flush error even though the descriptor slot has been removed. On Linux, retrying a failed close blindly can close a newly reused fd. An application that needs durability must check explicit synchronization too; flush/release are not universal substitutes for fsync.

A concurrent close is not a stable fd lookup#

Suppose thread A stores integer 3 for later and thread B closes fd 3, then opens G and receives the reused slot 3. A subsequent syscall by A performs a new table lookup and can operate on G. The fact that A originally saw F at that number does not pin F. Kernel code that needs a file beyond a protected borrowed scope must obtain the appropriate counted reference. RCU protects the table lookup protocol during concurrent publication and resizing, but it does not grant indefinite lifetime to a raw file pointer after the read-side/reference contract ends.

The files_struct count check in __fget_light uses acquire ordering to avoid observing an old table entry together with a newer unshared count while another thread closes and drops a shared table. This is an implementation detail of the optimization, not a rule that userspace can skip synchronization when sharing descriptor numbers. Similarly, fdget_pos may lock the shared position only under the source's specific FMODE_ATOMIC_POS and sharing/iteration condition. A release callback, close-time flush and position lock each solve a different problem. The fork-and-dup example provides a concrete way to keep them separate.

The close-on-exec bit belongs to each descriptor slot. A successful exec closes the slots marked for closure, but an unmarked duplicate can keep the same open description alive. The shared description's file status flags and position remain separate from those per-slot bits. If an application needs a newly duplicated fd to be close-on-exec without a race, it should use the appropriate atomic duplication/open interface rather than assume a later flag-setting call cannot race another thread's exec.

An fd passed to another process through a Unix socket can also refer to the same open description, so position sharing is not limited to duplicate descriptors in one process. The receiver obtains its own descriptor slot and must close it. This does not make the two processes share their entire descriptor tables or make their numeric fd values equal.

Practice#

Source exercise: rg -n -A30 'static long do_sys_openat2' "$KERNEL_SRC/fs/open.c" in a checkout whose git describe --tags --exact-match prints v6.12. Draw the reserved-slot, error-pointer and installed-file branches, including putname.

In the disposable guest, run:

python3 - <<'PY'
import os, tempfile
with tempfile.TemporaryFile() as f:
    f.write(b'abcdef'); f.flush(); f.seek(0)
    other = os.dup(f.fileno())
    try:
        print(os.read(f.fileno(), 2), os.read(other, 2))
        print(os.read(other, 2))
    finally:
        os.close(other)
PY

Expected output is b'ab' b'cd' followed by b'ef': both descriptors share the same position. The code deliberately lets the context manager close the original descriptor. As a counterexample, run the independent-open test below: each description starts at its own offset, so the first two-byte read from each is ab. If the first program fails, record the exception and do not infer the result. The temporary file is removed on close. Deliver a diagram of F, both descriptors, and offsets 0, 2, 4, 6, plus the open failure unwind.

python3 - <<'PY2'
import os, tempfile
with tempfile.TemporaryDirectory() as d:
    p = d + '/f'
    with open(p, 'wb') as f: f.write(b'abcdef')
    a = os.open(p, os.O_RDONLY); b = os.open(p, os.O_RDONLY)
    try: print(os.read(a, 2), os.read(b, 2))
    finally: os.close(a); os.close(b)
PY2

Expected b'ab' b'ab': the two independent opens have separate file descriptions and offsets. The temporary directory is removed on exit.

Knowledge check#

1. Why can two pathnames name the same inode without being the same dentry?

2. What happens when do_filp_open returns an error after fd reservation?

3. Why does reading through the duplicate start at offset 2?

4. Does closing one descriptor necessarily destroy F or the inode?

5. Does fdget always increment the file count, and when does fdput_pos unlock?

Answers#

1. A dentry represents a name and lookup location; hard links can associate different names with the same inode.

2. put_unused_fd releases the reserved slot, the encoded error is returned, and putname releases the copied pathname.

3. dup refers to the same open file description F, which owns the shared position advanced by the first read.

4. No. Other descriptor or kernel references may keep F alive, and the inode's lifetime has separate references and filesystem rules.

5. No. An unshared table can yield a borrowed file; a shared-table lookup can return a counted reference. fdput_pos unlocks only when FDPUT_POS_UNLOCK was set, then performs the corresponding put.

Checkpoint#

Save the diagram and explain all four offset states and both do_sys_openat2 branches. If you cannot distinguish the descriptor, file and inode, redraw the diagram and repeat the experiment. This demonstrates readiness to continue, not certification.

Sources#

• openat2 open and unwind, Linux v6.12.
• VFS inode and file structures.
• Descriptor table implementation.
• Descriptor table types.
• File lifetime implementation.