Pipes, Unix sockets, SysV/POSIX IPC, and shared memory

Before you start#

Read durability and crash consistency and VFS objects and descriptors. This chapter studies upstream Linux v6.12, a historical baseline. Its Python lab creates only process-owned pipes, sockets, temporary files and a uniquely named shared-memory object, then cleans them up.

Learning goals#

Explain pipe buffering and end lifetime, transfer an open file reference over a Unix socket, distinguish SysV and POSIX IPC objects, and identify the synchronization and cleanup obligations of shared memory.

Source walkthrough#

A pipe connects writers and readers through a bounded kernel buffer. struct pipe_inode_info holds the pipe mutex, reader and writer wait queues, ring state, counts of readers and writers, and an array of struct pipe_buffer entries. A pipe buffer refers to a page and describes an offset, length, flags and operations for that entry. pipe_read and pipe_write in fs/pipe.c use this shared structure; definitions are in include/linux/pipe_fs_i.h. A descriptor points to an open file description for an end; duplicating it or inheriting it across fork can keep that end alive. Closing one integer descriptor is not necessarily closing the last writer or reader.

These implementation details are Linux v6.12 specifics. The portable interface guarantees described here are POSIX/Linux API behavior; pipe capacity and some limits are configurable and should not be taught as a fixed byte count. Linux documents PIPE_BUF as 4096 bytes, while POSIX guarantees at least 512; query or consult the target platform when a protocol depends on the value.

The ring's finite capacity is not PIPE_BUF. Capacity determines how much can be buffered before writers must wait or fail. PIPE_BUF is an atomicity bound for qualifying pipe writes: writes no larger than that bound are not interleaved with other writers under the interface's rules. It does not guarantee that one read returns exactly one write. Larger writes can be partial or interleaved. Blocking and O_NONBLOCK, available space, signals and competing readers affect the result. A stream has no application record boundaries just because a writer made separate system calls.

The v6.12 pipe_write path checks readers while holding the pipe mutex. From fs/pipe.c, GPL-2.0, surrounding code omitted:

	if (!pipe->readers) {
		send_sig(SIGPIPE, current, 0);
		ret = -EPIPE;
		goto out;
	}

The condition tests whether any read ends remain. With none, it generates SIGPIPE, sets -EPIPE, and jumps to cleanup rather than appending data. A process with the default signal disposition may terminate before it can inspect the error; one that ignores or handles SIGPIPE can observe EPIPE. The complete function unlocks and performs its wakeup bookkeeping through its exit paths. Later checks handle a reader disappearing while a writer is waiting. A zero-length write has a separate early return; do not generalize the excerpt to every possible call.

In pipe_read, the reader holds pipe->mutex, obtains head and tail, selects pipe->bufs[tail & mask], confirms the buffer, and copies up to the smaller available or requested length. It advances buf->offset and reduces buf->len; once the buffer is empty, it updates the tail. If the ring is empty and no writers remain it exits, returning zero when it has copied nothing. If writers remain and no bytes were copied, a nonblocking read returns -EAGAIN; a blocking reader can wait. The code handles partial progress and writer wakeups separately. This normal path is for the relevant pipe configuration; notification and packet modes have additional branches.

For a concrete sequence, create a pipe with one read and one write descriptor. Write abc; the ring now contains three bytes. Reading two can return ab, leaving c for a later read. Duplicate the writer and close the original: the duplicate keeps a writer open, so an empty pipe is not yet EOF. Close the duplicate; once buffered c is consumed and no writers remain, a read returns zero for EOF. If instead all readers close while a writer remains, a subsequent nonzero write triggers the signal/error behavior above. These states describe endpoint references, not merely two original fd numbers.

Unix sockets and passing a file reference#

Unix-domain sockets offer local stream, datagram and sequenced-packet interfaces. A stream is a byte stream; application framing is still necessary. A datagram preserves message boundaries subject to size and queue limits. A socketpair creates connected local endpoints without a global pathname. Pathname sockets and abstract-namespace sockets have different naming and filesystem-lifetime rules; access controls and network namespace configuration matter. Kernel socket buffers (sk_buff) carry data and ancillary control information while queued.

SCM_RIGHTS sends references to open files, not the sender's descriptor integers as globally meaningful identifiers. The receiver installs new descriptors in its own table referring to the transmitted open file descriptions. As with dup, shared file position and file status flags can matter; descriptor flags such as close-on-exec belong to the new descriptor. Linux does not generally copy the sender's FD_CLOEXEC setting: use MSG_CMSG_CLOEXEC on receive to request close-on-exec atomically where available. Close received fds after use and consider truncation and malformed ancillary data. Passing a descriptor can transfer access beyond the receiver's ability to reopen the pathname, so the sender must treat it as an authority transfer. Credentials (SCM_CREDENTIALS, peer credentials and related socket options) follow their own validation and policy; a pathname alone is not proof of a peer identity. See unix(7) and recvmsg(2).

unix_attach_fds, Linux v6.12, GPL-2.0-or-later, moves the file-list reference into the socket buffer:

	UNIXCB(skb).fp = scm->fp;
	scm->fp = NULL;

The first statement assigns the list to the queued buffer's Unix control block. The second clears the source pointer, recording that ownership was transferred; leaving both owners believing they own the same reference would risk double release. The function first checks excessive in-flight references and can fail with -ETOOMANYREFS; it can also return -ENOMEM if preparation fails. The full send and socket-buffer cleanup paths determine how a failure is unwound. On receive, scm_detach_fds installs descriptors subject to available space and errors. Unix sockets have garbage-collection machinery for cycles of in-flight socket references: a socket can carry a descriptor for another socket, so ordinary user-visible descriptor closure alone is not a complete explanation of all queued-reference lifetimes. Consult the implementation rather than assuming every queued descriptor is immediately destroyed when one process exits.

In net/unix/af_unix.c, v6.12, SPDX GPL-2.0-or-later (surrounding code omitted), the receive side moves the list back out of the socket buffer:

static void unix_detach_fds(struct scm_cookie *scm, struct sk_buff *skb)
{
	scm->fp = UNIXCB(skb).fp;
	UNIXCB(skb).fp = NULL;

	unix_destroy_fpl(scm->fp);
}

This mirrors unix_attach_fds: the buffer's pointer is cleared, so one owner remains. The call to unix_destroy_fpl releases the garbage-collector edge and vertex bookkeeping for the list. The descriptors themselves are installed later by scm_detach_fds, which this excerpt does not show.

A hypothetical sender opens a file with offset 0, reads two bytes, and sends its fd. The receiver's newly allocated descriptor refers to the same open description with offset 2. If the receiver reads two more bytes, the shared position can advance to 4 for the sender as well. The sender's integer 5 need not be the receiver's integer. If the receiver's ancillary buffer is too small, control data can be truncated; it must inspect flags and close whatever descriptors it actually received. Message delivery, fd installation and eventual close are distinct events.

SysV and POSIX IPC are families, not one API#

System V IPC includes message queues, semaphore sets and shared-memory segments. SysV objects are addressed through keys for discovery and kernel identifiers for operations, with ownership and permission metadata. Their lifetime can exceed the process that created them until explicitly removed or until the relevant IPC namespace is destroyed. A SysV shared-memory segment can be marked for removal while attached; existing attachments and final destruction follow the segment's rules. An attachment is an address-space mapping, not an automatic mutex. ipc/shm.c, ipc/msg.c, and ipc/sem.c implement these families.

POSIX message queues have names and descriptors with message priorities and queue limits; Linux implements them in ipc/mqueue.c. POSIX named semaphores and shared-memory objects have separate interfaces and name/unlink lifetimes. On Linux, POSIX shared-memory objects are commonly exposed through a tmpfs mounted at /dev/shm, but that is a deployment convention and namespace view, not a guarantee that every system exposes the same mount. Linux shmem supplies memory-backed file infrastructure used by several features; see mm/shmem.c. Unlinking a name prevents later lookup by that name, while existing references or mappings can remain usable until released according to the API. Permissions, mount or IPC namespaces, resource limits, and user credentials affect access.

A named POSIX semaphore is not the same object as a SysV semaphore set, and a POSIX message queue is not a Unix datagram socket. Their APIs differ in naming, permissions, limits and removal. A user should remove only the uniquely named objects created for their exercise; commands that indiscriminately remove system-wide IPC objects can disrupt unrelated processes. A namespace isolates certain resource views but does not remove every authority or shared-resource concern.

Shared bytes need a protocol#

Suppose process A and process B map the same four-byte counter. Both read 7, both compute 8, and both write 8. The final value can be 8 even though two increments were intended. Shared memory shares bytes; it does not automatically make a read-modify-write atomic or establish ordering between a data buffer and a flag. Use a process-shared mutex, semaphore, atomic protocol or another suitable synchronization mechanism with the required memory-ordering rules. A process-shared primitive must be initialized for that use, live as long as all participants, and handle a participant dying while holding it where necessary.

A producer can write a payload and signal a semaphore; the consumer waits, reads the payload under the protocol, and acknowledges or releases space. The design must define buffer bounds, ownership of slots, what happens on peer crash, and how to clean up names and references. A plain integer flag without appropriate atomics and ordering can fail even if a casual test happens to work. Conversely, using a pipe or Unix socket can provide a communication boundary without shared writable data, but still requires handling short transfers, closure and errors.

Capacity, ownership, and a slow peer#

Suppose a writer produces records faster than a reader consumes them. Once pipe capacity is exhausted, a blocking writer can sleep until the reader makes room, a signal interrupts the operation, or reader state changes. A nonblocking writer instead receives an immediate result governed by the requested length and available space; it must handle a short write or EAGAIN under the interface rules. An application that assumes an entire message was written can lose its framing even if each subsequent read succeeds. Multiple writers must agree on record size and serialization; PIPE_BUF atomicity for qualifying writes does not create a general message queue.

A Unix socket can also fill its send queue or lose its peer. Stream sends can be partial and require a loop; datagram size and queue limits impose different behavior. The sender cannot infer that the receiving application processed a message merely because the kernel accepted it. With SCM_RIGHTS, the queued kernel reference can keep an open file description alive after the sender closes its own descriptor. The receiver owns the descriptors it actually obtains, and must close them even on parsing or application errors. Truncated ancillary data requires careful cleanup and cannot be treated as a complete list of the sender's intended descriptors.

For shared memory, naming and mapping are also separate. A process may successfully open a POSIX shared-memory object but fail to map it due to address-space or resource limits. A mapping can continue after a name is unlinked, while a later process cannot discover it through the removed name. An application needs an explicit cleanup owner and crash policy; blindly unlinking a guessed name can affect another program. These differences are especially important when several containers see different /dev/shm mounts or IPC namespaces.

Practice#

Source-only route: with a v6.12 checkout run:

rg -n -A45 'pipe_write\(' "$KERNEL_SRC/fs/pipe.c"
rg -n -A22 'static int unix_attach_fds\(' "$KERNEL_SRC/net/unix/af_unix.c"
rg -n -A35 'void scm_detach_fds\(' "$KERNEL_SRC/net/core/scm.c"

Trace no-readers to SIGPIPE/EPIPE, and trace the file-list pointer from scm to the socket buffer and then to descriptor installation. Name who owns each reference after the move; do not infer runtime execution from the source.

Run this unprivileged Python 3 lab. It creates no persistent shared global object:

python3 - <<'PY'
import array, os, socket, tempfile
from multiprocessing import shared_memory
r, w = os.pipe()
os.write(w, b'abc'); os.close(w)
print('pipe:', os.read(r, 2), os.read(r, 2), os.read(r, 2)); os.close(r)
a, b = socket.socketpair()
try:
    with tempfile.TemporaryFile() as f:
        f.write(b'abcdef'); f.flush(); f.seek(2)
        a.sendmsg([b'F'], [(socket.SOL_SOCKET, socket.SCM_RIGHTS, array.array('i', [f.fileno()]))])
        data, anc, flags, _ = b.recvmsg(1, socket.CMSG_SPACE(array.array('i').itemsize))
        assert not (flags & socket.MSG_CTRUNC)
        received = array.array('i'); received.frombytes(anc[0][2][:received.itemsize])
        fd = received[0]
        try: print('fd:', data, os.read(fd, 2), 'offset:', f.tell())
        finally: os.close(fd)
finally: a.close(); b.close()
shm = shared_memory.SharedMemory(create=True, size=4)
print('shared name:', shm.name, flush=True)
try:
    other = shared_memory.SharedMemory(name=shm.name)
    try:
        shm.buf[:4] = b'LAB!'; print('shared:', bytes(other.buf[:4]))
    finally: other.close()
finally: shm.close(); shm.unlink()
PY

Expected pipe output is b'ab' b'c' b''; the third read is EOF after the last writer closes. The received fd reads b'cd' and the original file description's position becomes 4. The shared-memory line prints b'LAB!'; its generated name varies and is printed for identifying this exercise's object. If Python, Unix ancillary data or POSIX shared memory is unavailable, report the exception and run the source-only route; do not remove unrelated IPC objects. If the process is killed before its finally block, its printed or debugger-recorded unique shared-memory name is the only object to remove with the corresponding API; do not run blanket IPC cleanup.

For a counterexample, run the following in a separate Python process. Python normally ignores SIGPIPE, so os.write raises BrokenPipeError; another program with the default disposition may instead terminate on the signal.

python3 - <<'PY'
import os
r, w = os.pipe(); os.close(r)
try:
    try: os.write(w, b'x')
    except BrokenPipeError: print('no readers: EPIPE')
finally: os.close(w)
PY

Expect no readers: EPIPE. This tests endpoint lifetime, not pipe capacity or PIPE_BUF interleaving. Deliver the three pipe states, the shared-offset explanation and the shared-memory cleanup record.

Knowledge check#

1. Why does closing the original write fd not necessarily make a reader see EOF?
2. What does PIPE_BUF promise, and what does it not promise about reads?
3. What is transferred by SCM_RIGHTS, and why can file offsets be shared?
4. How do SysV segment removal and POSIX shared-memory unlink differ from freeing every existing mapping immediately?
5. Why does mapping the same counter not make two increments atomic?

Answers#

1. Duplicated or inherited write ends can still be open. EOF follows buffered data exhaustion and closure of all write ends.
2. It bounds atomic writes under the pipe rules; it is not the whole pipe capacity and does not impose one-read-per-write framing.
3. References to open file descriptions are sent and new descriptors are installed. The shared description can carry a shared position.
4. Marking a SysV segment for removal or unlinking a POSIX name affects the object's naming/removal state, while existing attachments or references can outlive that operation under their API rules.
5. Two processes can read the same old value and overwrite each other's update; synchronization and memory ordering are required.

Checkpoint#

Save the endpoint timeline, fd-transfer result and unique-object cleanup outcome. Pass when you distinguish pipe EOF from EPIPE, a descriptor number from an open description, and a shared mapping from a synchronization protocol. Re-read pipe_write and unix_attach_fds and repeat the corresponding portion if unsure. This is practice, not certification.

Sources#

• Pipe implementation, Linux v6.12.
• Pipe structures.
• Unix-domain sockets.
• Ancillary data and descriptor installation.
• SysV shared memory.
• POSIX message queues.
• Shared-memory filesystem implementation.