Sockets, sk_buff, and the networking stack

Before you start#

Read futex, epoll, and io_uring and Unix sockets and pipes. The source baseline is upstream Linux v6.12, a historical teaching version. The lab uses a socketpair inside one process and performs no external network access. TCP, protocol, driver and offload paths vary by configuration, namespace and architecture.

Learning goals#

Follow a socket descriptor to protocol dispatch, explain socket creation and failure cleanup, distinguish an skb's metadata from its shared packet data, and trace one conditional TCP/IPv4 route through local output and receive.

Source walkthrough#

A socket fd is a process-local descriptor table index. Its open struct file has socket file operations and private data pointing to struct socket. The socket contains a type, a state and ops, a struct proto_ops table selected for the address family and socket type. For many Internet sockets, socket->sk points to a struct sock containing protocol state, locks, queues and memory accounting. sk->sk_prot points to a struct proto supplying transport-level operations. These are related dispatch layers, not interchangeable names: family-level socket methods route operations to the appropriate protocol implementation, which manages transport state. Definitions and dispatch live in net/socket.c, include/net/sock.h, and net/ipv4/af_inet.c.

socket(AF_INET, SOCK_STREAM, 0) selects an address family, stream type and default protocol for that combination. __sys_socket checks flags, calls socket creation, then maps the created socket to an fd. __sock_create selects the registered family under RCU, obtains a module reference before leaving the RCU critical section, invokes the family's create operation, and establishes appropriate ownership for the socket operations. An unsupported family, module reference failure, allocation failure or security hook can return an error. If family creation fails, the family must release its allocated sock->sk; the generic code clears the potentially dangling pointer before its cleanup path. That contract is important when a protocol module supplies the create callback.

In sock_map_fd, v6.12, GPL-2.0-or-later, the fd-reservation branch is:

	int fd = get_unused_fd_flags(flags);
	if (unlikely(fd < 0)) {
		sock_release(sock);
		return fd;
	}

A failure to reserve an fd releases the newly created socket and returns the error. On success, sock_alloc_file creates a file for the socket; if that fails it releases the socket, and the caller returns the reserved fd via put_unused_fd. Only a successful file is installed with fd_install. Thus neither a failed file pointer nor a permanently reserved fd is published as a working socket. These paths manage different resources, so duplicating cleanup without understanding the ownership transfer can double-release them.

In net/socket.c, v6.12, GPL-2.0-or-later (surrounding code omitted), the family's create callback runs after its module reference is taken:

	err = pf->create(net, sock, protocol, kern);
	if (err < 0) {
		/* ->create should release the allocated sock->sk object on error
		 * but it may leave the dangling pointer
		 */
		sock->sk = NULL;
		goto out_module_put;
	}

The contract is that the family's create releases its own sk on error; the generic code then clears sock->sk so the later release path cannot touch a dangling pointer. The excerpt does not show the out_module_put and out_release unwinds or which families follow the contract.

The common server sequence is socket, bind, listen, then accept: bind selects a local address and port subject to namespace, privileges and collisions; listen establishes a passive endpoint; accept returns a new connected socket while the listener remains available. A client normally calls connect, then sends and receives. For a nonblocking connect, an in-progress result is not proof that the connection succeeded; readiness and the socket error must be checked. sendmsg and recvmsg dispatch through socket operations and eventually protocol methods; a file close drops one reference, and the final release invokes protocol cleanup. A duplicate fd, queued packet, timer or other reference can affect actual lifetimes. Calls can fail at every stage, including EADDRINUSE on bind or ECONNREFUSED on a connection attempt.

Packet storage: metadata, data, and references#

The sk_buff (skb) is a kernel packet-buffer descriptor, not simply a C array containing a complete Ethernet frame. head points at the start of allocated linear storage, data identifies the current data start, and tail and end mark the used-data and allocation bounds using the representation defined by the build. Headroom before data can accommodate headers; tailroom follows the used linear data. len describes total skb data, while data_len accounts for non-linear data such as page fragments or a fragment list. Network and transport header offsets locate protocol headers relative to the skb head when set. A packet can have data outside the linear head, so code cannot assume data + len is one dereferenceable contiguous region. See include/linux/skbuff.h.

An skb descriptor has its own users reference count. The shared packet-data area has a separate dataref in skb_shared_info. skb_clone creates another descriptor sharing packet data instead of copying the payload; modifying shared data requires the applicable copy-on-write or unsharing helpers. In __skb_clone, Linux v6.12, GPL-2.0-or-later, the following two consecutive statements are central:

	atomic_inc(&(skb_shinfo(skb)->dataref));
	skb->cloned = 1;

The first increases the shared data reference; the second marks the original descriptor as cloned. Earlier statements copy header metadata and pointers and initialize the clone's descriptor reference. A clone is therefore not a private payload copy. skb_clone can fail, including when orphaning fragments or allocating the descriptor fails; the caller must handle NULL. Releasing the last descriptor reference and releasing the last shared data reference are separate steps. A reference can preserve lifetime without granting exclusive write access.

truesize is a memory-accounting estimate associated with skb storage, not necessarily the payload length or exact wire size. Socket memory accounting and receive/send buffer limits constrain queued packets. A receive queue can drop or refuse packets under pressure; a successful send into a local queue is not proof that a remote application consumed them. skb ownership and destructor callbacks determine when socket accounting and references are released. A queueing path can transfer ownership to another layer; after a successful ownership transfer the previous caller must obey that API rather than freeing or modifying the skb again.

A conditional TCP/IPv4 path#

For a connected IPv4 TCP stream using the normal Internet socket operations, sendmsg reaches inet_sendmsg and the transport's send method, then tcp_sendmsg/tcp_sendmsg_locked. TCP copies or references data according to the chosen path and its memory rules, constructs queued skbs and invokes push logic. The transport's output path can reach tcp_write_xmit, tcp_transmit_skb, and the IPv4 output operation ip_queue_xmit. A route and destination object select an output path, then IP output and device queueing can reach __dev_queue_xmit. Policy hooks, segmentation, congestion, socket memory, route availability, qdisc configuration and errors can change this path. See net/ipv4/tcp.c, net/ipv4/tcp_output.c, and net/ipv4/ip_output.c.

On a physical NIC, the chosen driver eventually handles transmit descriptors and device completion under its DMA and queue contracts. A packet arriving from a physical device can be processed through interrupt and NAPI/softirq-related paths, depending on driver and configuration. The loopback driver instead reinjects locally transmitted packets into the receive stack; it is not a physical NIC and does not prove that an interrupt or wire transmission occurred. See net/core/dev.c and drivers/net/loopback.c.

For a locally delivered IPv4 packet, ip_rcv validates and routes the packet through the configured hooks; ip_local_deliver and ip_protocol_deliver_rcu dispatch by IP protocol. TCP's IPv4 entry tcp_v4_rcv finds an appropriate socket and validates state before established or other TCP handling. TCP eventually makes in-order bytes available on the receive queue and wakes readers under its locking and scheduling rules. This is one conditional source map, not a guarantee that every packet follows every named function or that the source was executed in the lab. IPv6, forwarded packets, XDP drops, tunnels and non-TCP sockets take different branches.

A stream example and failure cases#

Suppose one endpoint writes abc and then def to a stream. The receiver may read ab, then cdef, or another permitted grouping. TCP and Unix stream sockets preserve byte order, not the sender's application write boundaries. A send can return a short count, especially with nonblocking operation; a caller must continue from the returned offset or report failure. sendall in Python handles repeated sends but can still raise an exception. A receive returning zero bytes for a nonzero requested length on a stream indicates orderly EOF after peer shutdown and queued bytes are consumed. It is not the rule for zero-length UDP datagrams.

Consider socket creation when the process has no free fd. get_unused_fd_flags fails before a socket file is installed, and sock_map_fd calls sock_release. Consider instead a socket successfully connected but then closed by its peer: a later send can fail with a socket error or signal depending on protocol and flags, and a receive can report EOF or an error according to how the connection ended. There is no universal ordering that makes close wait for the peer application to process all bytes. Closing the only application descriptor does not imply that all network packets and protocol timers vanished at that instant.

Memory pressure supplies a different failure mode. A packet clone can fail without invalidating the original skb; a caller must not dereference the NULL clone or assume the clone's payload became independent. In transmit or receive paths, locks, RCU and references protect different relationships: RCU family lookup does not by itself keep a loadable protocol module available after the read-side section, hence the module reference in socket creation. An skb data reference keeps storage alive, not a socket's state lock held. Track the actual owner at each call boundary.

From socket calls to protocol ownership#

bind and listen are operations on a socket object, but neither is synonymous with a successful connection. For an Internet stream listener, bind associates a local address and port under the relevant checks; listen changes the endpoint into a passive listener and sets a backlog subject to kernel limits. An accept operation obtains a new connected socket and file/descriptor for a queued connection. If descriptor or file allocation fails, the kernel must release the newly created socket or reserved fd through the applicable accept cleanup path; the listener is not automatically the new connection. A client-side connect can return an error immediately, complete synchronously, or report an in-progress state for nonblocking use. Use the actual syscall result and later socket error, not writable readiness alone, to decide that connection setup succeeded.

sendmsg uses a message header that can describe multiple buffers and optional control data. The iovec is a userspace description, not a promise that one skb or one network packet will be emitted. A receive method copies or otherwise exposes data under its interface's rules and can return fewer bytes than the requested buffer. Ancillary data has its own length and truncation handling. For a stream, zero from a nonzero-length receive indicates orderly end of that direction once queued bytes are exhausted; an error return is a separate outcome. When all file references are eventually released, the socket release path invokes protocol cleanup, but queued work and protocol references have their own lifecycle rules.

The sk_buff header offsets are important because different layers need different views of one packet. At one stage data can point at a transport header; after another operation it can point at payload or another header. Helpers such as pull, push and header accessors adjust or interpret the layout under their documented preconditions. A driver must not dereference a header that is absent or resides in non-linear fragments without first ensuring the necessary bytes are available. Clone sharing makes this more subtle: a metadata change to one descriptor is not the same as an exclusive write to shared packet bytes.

For a hypothetical skb with 128 bytes of linear allocation, data at offset 32 and tail at offset 80, the linear used region is 48 bytes and there are 32 bytes of headroom. If an additional page fragment contributes 100 bytes, total data length can exceed that 48-byte linear span. The exact skb fields, headroom and length must be read with their helpers; this arithmetic does not claim a measured packet. truesize can be larger than 148 because memory accounting includes storage and metadata overhead. A socket receive limit is therefore not simply a count of payload bytes.

In a physical receive path, a NIC driver supplies buffers and eventually hands received skbs to the networking stack; hard interrupt, polling and deferred processing vary by driver and configuration. A backlog or protocol queue may defer work when a socket is owned by userspace. RCU can protect a lookup while a reference keeps an object alive afterward; neither changes the requirement to respect socket locking when updating state. A local loopback socketpair demonstration tests byte-stream behavior only and cannot establish the precise physical ingress or egress path.

Receiving and releasing an skb#

A packet's journey can end before transport delivery. IP validation, a configured policy hook, a missing route, transport checksum checks, lack of a matching socket, or receive-queue pressure can each stop progress at a different boundary. The drop path must release the skb and any references it owns; an application waiting on recv is not guaranteed a separate notification for every incoming packet dropped before queueing. An accepted packet can instead be queued for a socket, where the socket's memory accounting and callback rules apply until userspace consumes it or teardown releases the queued data. These are distinct outcomes from the original sender's local send return.

A packet descriptor can be shared while the socket state changes concurrently. An skb reference keeps that descriptor alive only as its reference contract specifies; a shared data reference keeps packet storage alive separately. Socket lock and RCU rules protect other relationships. If an asynchronous callback needs a struct sock after a lookup, it must hold the appropriate reference or other protection; keeping an skb alone is not a universal guarantee that an arbitrary socket pointer remains valid. Conversely, freeing a cloned descriptor does not free shared packet data while another data reference remains. These distinctions let you review an error path without assuming that one reference count controls the whole stack.

Practice#

Source-only route in a v6.12 checkout:

rg -n -A25 'static int sock_map_fd' "$KERNEL_SRC/net/socket.c"
rg -n -A35 'static struct sk_buff \*__skb_clone' "$KERNEL_SRC/net/core/skbuff.c"
rg -n 'inet_sendmsg|tcp_sendmsg_locked|tcp_v4_rcv' "$KERNEL_SRC/net/ipv4/af_inet.c" "$KERNEL_SRC/net/ipv4/tcp.c" "$KERNEL_SRC/net/ipv4/tcp_ipv4.c"

Confirm the checkout tag. Mark the reserved-fd cleanup, clone data reference, and transport dispatch locations. Draw a path with explicit conditional branches; do not claim these rg matches constitute an execution trace.

Run this bounded own-process Unix stream experiment:

python3 - <<'PY'
import socket
a, b = socket.socketpair()
try:
    a.sendall(b'abcdef')
    a.shutdown(socket.SHUT_WR)
    first = bytearray()
    while len(first) < 2:
        part = b.recv(2 - len(first))
        if not part: raise RuntimeError('unexpected EOF')
        first.extend(part)
    rest = bytearray()
    while True:
        part = b.recv(4)
        if not part: break
        rest.extend(part)
    print(repr(bytes(first)), repr(bytes(rest)), repr(b''))
    b.setblocking(False)
    # EOF persists after the peer has shut down its write side.
    print('EOF:', b.recv(1) == b'')
finally: a.close(); b.close()
x, y = socket.socketpair(); x.setblocking(False)
try:
    try: x.recv(1)
    except BlockingIOError: print('empty stream: EAGAIN')
finally: x.close(); y.close()
PY

Expected b'ab' b'cdef' b'' and EOF: True on this local sequence; stream segmentation is not guaranteed for arbitrary sockets or larger writes. The second socketpair is a counterexample to assuming data is immediately ready: its nonblocking recv(1) prints empty stream: EAGAIN before any send. No remote address is contacted. Deliver a descriptor-to-socket diagram, the two read values and the EOF observation. If socketpair is unavailable, record the error and use the source route.

Knowledge check#

1. How do struct socket and struct sock participate in dispatch?
2. What happens to the socket when fd reservation fails in sock_map_fd?
3. What does the skb clone's dataref increment protect, and what does it not guarantee?
4. Why can two stream writes be received in a different grouping?
5. Why is loopback not evidence of physical NIC interrupt handling?

Answers#

1. socket->ops supplies socket-family operations, while socket->sk and its protocol methods hold transport-specific state and behavior.
2. The function releases the socket and returns the fd allocation error without installing a file.
3. It preserves a reference to shared data; it does not make the payload exclusive or hold a socket lock.
4. A stream preserves ordered bytes, not application write boundaries; each read can return a different available amount.
5. Loopback reinjects traffic through software and need not use a physical NIC, DMA, or its interrupt path.

Checkpoint#

Save the dispatch and skb ownership diagram plus the local output. Pass when it shows the creation-error unwind, distinct skb and shared-data references, and conditional TCP/IP and loopback paths. Re-read the cited functions and retry if any ownership transfer or stream result is unclear. This is practice, not certification.

Sources#

• Socket creation and syscall dispatch, Linux v6.12.
• Socket protocol structures.
• Internet socket operations.
• skb structures.
• skb clone and release.
• TCP output.
• IPv4 receive.