libfuse, SSHFS, and userspace filesystems

Before you start#

Review VFS objects, pathname lookup, filesystem I/O, and SSH channels. The source examples use libfuse 3.16.2 commit 7a92727d97c10290b3501d86a194738973edb61d and SSHFS 3.7.3 commit c91eb9a9a992f1a36c49a8e6f1146e45b5e1c8e7. They are independently pinned historical projects, not a tested runtime combination.

Learning goals#

Explain the relationship between a FUSE request, lookup reference, open handle and reply. Trace a synthetic unlink-while-open lifecycle and an interrupted remote request without mounting a filesystem, opening /dev/fuse or making an SSH connection.

Source walkthrough#

FUSE lets a kernel filesystem interface communicate with a userspace daemon over a connection associated with /dev/fuse. The kernel can issue requests for operations such as lookup, open, read and write; the daemon implements the corresponding behavior and returns protocol replies where required. A process using a mounted filesystem does not directly call the daemon's C functions. The kernel VFS, FUSE connection, daemon, optional network client and remote server are separate components with different object and failure lifetimes.

The following exact excerpt from libfuse 3.16.2 lib/fuse_lowlevel.c, fuse_reply_err, has the file's GNU LGPLv2 notice:

int fuse_reply_err(fuse_req_t req, int err)
{
	return send_reply(req, -err, NULL, 0);
}

req identifies the pending low-level request and err is the error value expected by this API. The helper passes its negation to the internal reply function with no payload. The exact sign and ownership contract are specific to the API; a daemon must not invent a second reply for the same completed request. A failed send or disconnected connection still requires the library and daemon to release their own state under the relevant contract. This function alone does not prove that a remote operation did or did not execute.

The next exact excerpt from SSHFS 3.7.3 sshfs.c, sftp_request_wait, carries the file's GNU GPL notice. The earlier immediate-error check and later response-type handling are omitted:

	while (sem_wait(&req->ready));
	if (req->error) {
		err = req->error;
		goto out;
	}

The request object contains a readiness semaphore and an error field. This path waits for readiness and then checks the request's error; if set, it copies the error and branches to cleanup instead of decoding a successful response. The surrounding function also checks response types and status values. Request state and connection ownership must remain valid while a response or failure can arrive. An SFTP operation is a protocol request to a server, not the client executing a remote Linux syscall or a guarantee of identical POSIX semantics.

Negotiation and request identity#

At connection initialization the kernel and daemon negotiate protocol version and supported capabilities. The available features depend on kernel, libfuse, daemon and configuration. INIT negotiation is not evidence that every optional operation or cache behavior is enabled. A request normally has a unique identifier within the connection so replies and interruptions can be associated with the right request. Unique IDs are scoped to that connection and lifetime; reusing an identifier after disconnect must not let an old network reply satisfy a new request.

The low-level libfuse API exposes request and inode-oriented operations directly, while the high-level API supplies a different abstraction and handles more path-related translation. The chosen API, threading model and implementation determine what state an application must manage. A session loop can dispatch requests to worker threads under supported modes, so a daemon cannot assume callbacks never overlap. Shared inode tables, remote handles and pending-request maps need synchronization and explicit references. A lock protecting a table does not automatically keep an external server connection alive after the lock is released.

A FUSE lookup can return a node identifier and generation with attributes under the protocol's rules. The kernel's lookup count represents references associated with successful lookups; later FORGET or batch forget operations tell the daemon to drop counts. A lookup reference is not the same as an open file handle. A daemon must account for inode reuse and generation so a stale node ID is not silently interpreted as a different object. A forgotten inode can still have other reasons to remain live, such as an open handle, depending on the filesystem's design and API contract.

An OPEN reply can carry a daemon-defined file handle, often called fh, that later operations return. It can represent a local descriptor, remote SFTP handle or another daemon object; it is not inherently a process-wide kernel FD. READ and WRITE operate on the relevant open state and offsets. FLUSH is associated with closing a file descriptor and is not automatically the same guarantee as fsync; it can occur more than once with duplicated descriptors. FSYNC requests a synchronization operation with the filesystem's documented semantics. RELEASE closes an open handle under FUSE's lifecycle, while FORGET concerns lookup references. Conflating these operations can cause premature deletion or false durability claims.

Worked object and handle timeline#

Start with a synthetic daemon inode N, lookup count zero and no open handle. At T0 LOOKUP succeeds and the daemon models nlookup=1. At T1 OPEN creates handle H, so the daemon retains separate open state. At T2 the name is unlinked. Path lookup by that name can now fail, but the open handle H may still refer to the object under the filesystem's semantics. At T3 a READ through H returns previously available data if the daemon and backing storage permit it. At T4 RELEASE disposes of H. At T5 FORGET reduces the remaining lookup count from one to zero, allowing final inode cleanup once all other references are gone.

The ordering is a teaching example, not a captured kernel sequence. Actual request order, batching, caching and filesystem behavior can differ. A daemon must not free N solely at unlink while H or a lookup reference still needs it. Nor should it assume that FORGET has an ordinary data reply: the low-level API's fuse_reply_none path releases the request without sending a normal reply for that operation. A count underflow, duplicate release or reused node ID is an implementation bug that needs explicit checking and tests.

Now add a delayed SSHFS read. At T2 a FUSE read request R causes an SFTP request with remote identifier 17. Before the response arrives, an interrupt or disconnect changes the local request's state. The daemon must coordinate cancellation, its pending SFTP map and request ownership so that a late response cannot produce a second terminal FUSE reply or access freed request memory. Depending on timing and supported operation semantics, the remote server may already have performed work even if the local client sees an error. This is a synthetic race, not a claim that every interrupt cancels the remote operation.

Replies, interruption and disconnect#

A low-level request is normally answered exactly once with the appropriate terminal reply, with protocol-specific exceptions such as FORGET that do not take a normal reply. A daemon must distinguish a request that is still pending from one already completed or canceled. Error paths must release request-associated memory and references; a second callback arriving after completion must not send a duplicate reply. The precise interrupt contract depends on the operation and API: an interrupt is a request to cancel or report interruption, not proof that the underlying side effect never occurred.

The pinned fuse_lowlevel.c includes do_forget, which invokes the registered forget callback when present and otherwise calls fuse_reply_none. It also has request dispatch and reply helpers. The daemon owns its own inode and handle tables, while libfuse owns protocol machinery under its API. An input with an invalid length or unsupported operation must be rejected or handled according to the protocol; a hand-written parser must check lengths before indexing a buffer. The private model below is deliberately not a FUSE parser.

In fuse_lowlevel.c, do_forget, libfuse 3.16.2, GNU LGPLv2 notice (the daemon's registered forget callback is outside this excerpt):

static void do_forget(fuse_req_t req, fuse_ino_t nodeid, const void *inarg)
{
	struct fuse_forget_in *arg = (struct fuse_forget_in *) inarg;

	if (req->se->op.forget)
		req->se->op.forget(req, nodeid, arg->nlookup);
	else
		fuse_reply_none(req);
}

The handler forwards nlookup to the daemon's forget callback when one is registered. Without one, it calls fuse_reply_none. libfuse does not decrement any daemon table in this path, and the excerpt does not show whether the callback frees the inode.

In the same file, fuse_reply_none, with the same notice:

void fuse_reply_none(fuse_req_t req)
{
	fuse_free_req(req);
}

The helper frees the request without sending a reply message, which is why FORGET has no ordinary terminal reply. It does not free the daemon's inode or handle state; that remains the daemon's responsibility.

If the daemon exits or the connection is aborted, pending operations can fail and later operations can observe errors such as a disconnected endpoint under the relevant kernel path. A stale mountpoint or client-side cache does not prove the daemon is still serving requests. Cleanup of remote SSH processes and descriptors is separate from kernel connection teardown. The Linux v6.12 FUSE source is a kernel-side entry point, but no claim is made that this kernel or any mounted filesystem was executed here.

Caches, permissions and remote semantics#

Entry and attribute caches can retain information about names and metadata for configured durations; page caching and optional writeback behavior add other layers. Invalidation and timeouts affect what a caller observes after a remote change. A cached success can be stale, while disabling one cache does not remove every other buffering or network race. Direct I/O, writeback cache and invalidation support are negotiated or configured features, not universal FUSE behavior. A durability claim needs a specific daemon, filesystem, protocol and server contract.

Permission checks can be performed by the kernel under options such as default_permissions, by the daemon, or through a combination under the actual setup. allow_other changes who can access the mount when permitted by policy; it is not an automatic security improvement. A daemon acting with access to remote credentials can become a confused deputy if it does not correctly associate requests with caller identity and authority. Namespace visibility, mount options and underlying filesystem permissions also matter. The model does not alter any FUSE configuration.

SSHFS translates filesystem operations into SFTP requests and interprets responses. SFTP is a remote protocol with its own handles, status codes, path behavior and server implementation. Rename atomicity, permissions, UID/GID mapping, symlink behavior, caching and reconnect semantics can differ from an application developer's assumptions about a local POSIX filesystem. A successful write response is not automatically evidence that a remote disk survived power loss. Network backpressure can stall requests; an interrupted local caller does not prove the remote server did not apply a write.

A remote handle and a FUSE fh can be related without being the same type or lifetime. A daemon may map one to the other and must release both appropriately. If a connection is lost, a remote handle may no longer be valid even if the local fh number remains in memory. Reconnecting and reusing that number without a verified protocol-specific recovery scheme can attach a request to the wrong remote state. Record the connection generation, pending request IDs and error stage in a controlled test; do not claim that a source sketch reproduces SSHFS's full reconnection behavior.

Negotiated features and observable errors#

A protocol feature bit is meaningful only after both ends have negotiated the relevant version and capability and the daemon has implemented the corresponding behavior. An implementation can reject an operation as unsupported, return an error for a particular object, or disconnect before a reply. Those outcomes have different implications for retry and caching. A daemon should retain enough request context to make one terminal decision while ensuring that a late worker cannot act on a freed request. A transport reconnection creates a new context; a remote identifier from the old connection is not automatically valid there.

The model's invalid forget count is a counterexample for local bookkeeping, not a real FUSE wire packet. Real protocol input also includes lengths, opcodes, identifiers and operation-specific fields. A parser must reject truncated or malformed input before dereferencing those fields, and its tests need to cover the supported ABI and negotiated protocol version. This chapter deliberately supplies no mounting or raw device test, so its source inspection establishes only the cited functions and conditional relationships.

Practice#

This pure model checks reference counts and rejects an invalid negative forget count. It does not mount, contact SSH, open /dev/fuse or parse kernel protocol bytes:

python3 - <<'PY'
state = {'lookup': 1, 'open': 1, 'linked': True}
state['linked'] = False
print('synthetic read after unlink:', state['open'] > 0)
state['open'] -= 1
forget = 1
if forget < 0 or forget > state['lookup']:
    raise ValueError('invalid forget count')
state['lookup'] -= forget
print('synthetic reclaimable:', state['lookup'] == 0 and state['open'] == 0)
try:
    invalid = 2
    if invalid > state['lookup']:
        raise ValueError('invalid forget count')
except ValueError as error:
    print('rejected:', error)
def decode_toy(data):
    if len(data) < 2 or data[0] != len(data) - 1:
        raise ValueError('bad toy length')
    return data[1:]
print('toy payload:', decode_toy(bytes([2, 65, 66])))
try:
    decode_toy(bytes([3, 65]))
except ValueError as error:
    print('malformed rejected:', error)
PY

Expected output shows a modeled readable open handle, then reclaimable state, then a rejected invalid count, a toy payload and a rejected malformed length. This is an abstract counter and framing example, not a FUSE protocol implementation or a guarantee that an unplugged remote object remains readable. If Python is unavailable, use the T0–T5 table and calculate both counts by hand. No filesystem mount, remote connection or host configuration is changed.

For a source-only alternative, draw N and H at each stage and mark lookup count, open count and name state. Add request R and SFTP identifier 17; show the terminal reply state when an interrupt races a delayed response. Explain the libfuse error reply and why the SSHFS response wait is not itself a remote syscall.

Knowledge check#

1. Why does unlinking a name not necessarily permit immediate destruction of an object with an open handle?
2. How do RELEASE and FORGET differ in the synthetic model?
3. What does fuse_reply_err pass to its internal reply function, and why must request ownership still be tracked?
4. Why can a delayed SFTP response after an interrupt be dangerous to a daemon's request bookkeeping?
5. Why does an SSHFS write response not automatically prove remote power-loss durability?

Answers#

1. An open handle and lookup references can retain state independently of the pathname. The daemon needs its own valid lifecycle rules even after the name is removed.
2. RELEASE concerns the open handle H; FORGET reduces a lookup reference count for N. They are separate obligations and can arrive under different timing conditions.
3. It passes the negated error and an empty payload to the internal reply helper. A daemon must still avoid a second terminal reply and release associated state under the API contract.
4. The local request may already have completed or been freed. A late response must be correlated with the correct connection and request state so it cannot cause duplicate reply or use-after-free.
5. SFTP, server, filesystem, caching and device persistence each have their own semantics. A protocol response is not by itself proof that all lower layers committed data to stable storage.

Checkpoint#

Submit the N/H lifecycle table, delayed-response race diagram and either private counter-model output or source-only analysis. Ready means distinguishing lookup counts from open handles and EOF/error from durability, and explaining the two source excerpts. Retry with a local conceptual object before adding the remote request.

Sources#

• libfuse 3.16.2 low-level implementation
• libfuse 3.16.2 public low-level interface
• SSHFS 3.7.3 request implementation
• Linux v6.12 FUSE device implementation