TCP state, queues, congestion, and loss recovery

Before you start#

Read sockets and the network stack. The source baseline is upstream Linux v6.12, a historical teaching version. TCP behavior depends on options, congestion-control selection, namespace settings, route and workload. The loss example is explicitly synthetic; the lab uses only an own-process IPv4 loopback connection.

Learning goals#

Follow a connection and byte stream through TCP send and receive paths; distinguish receiver flow control from congestion control; reason about missing bytes, retransmission, closure and request lifetime without treating TCP segments as application messages.

Terms used in this lesson#

• MSS: maximum segment size, the TCP data size used when segmenting a stream. skb: Linux socket buffer, the kernel packet metadata and data container; it does not necessarily map one-to-one to a wire packet.
• SACK: selective acknowledgment, which reports received sequence ranges beyond the cumulative ACK. RACK: time-based loss detection using transmission and delivery timing. TLP: tail loss probe, a probe sent near the end of outstanding data to elicit feedback. RTO: retransmission timeout.
• cwnd: congestion window; rwnd: receiver-advertised window; ssthresh: slow-start threshold used by congestion-control algorithms.

Source walkthrough#

TCP offers an ordered reliable byte stream, subject to connection failure. A listening socket in LISTEN is not itself each accepted connection. On an incoming SYN, the IPv4 path can use tcp_v4_conn_request and request-socket machinery to track a pending handshake. On a valid completion, tcp_check_req and tcp_v4_syn_recv_sock can produce a child socket; accept retrieves an established child while the listener remains a separate socket. SYN cookies, TCP Fast Open, backlog pressure, retransmission and configuration add alternate paths; do not assert that every SYN necessarily allocates or retains the same request object. See net/ipv4/tcp_ipv4.c and net/ipv4/inet_connection_sock.c.

TCP sequence numbers count stream bytes modulo 2^32; SYN and FIN each consume sequence space. SND.UNA denotes the oldest byte sent but not cumulatively acknowledged, SND.NXT the next sequence to send, and RCV.NXT the next byte expected in order. These names describe protocol state; kernel fields and helper abstractions must be read in context. An ACK normally acknowledges all sequence space before its acknowledgement number. A receiver can buffer later out-of-order data while withholding contiguous delivery until the gap is filled. TCP's byte positions are not application record numbers.

For a connected socket, tcp_sendmsg takes the socket lock and calls tcp_sendmsg_locked. The function appends data to send skbs, handles memory and blocking conditions, and pushes pending work. A central excerpt from net/ipv4/tcp.c, Linux v6.12, GPL-2.0-or-later, is:

	if (copied) {
		tcp_tx_timestamp(sk, sockc.tsflags);
		tcp_push(sk, flags, mss_now, tp->nonagle, size_goal);
	}

Only after a positive amount is copied does this path request timestamp processing and invoke tcp_push. The surrounding code handles no-space waits and returns partial progress if some data was already copied before an error. tcp_push can defer transmission due to corking or other conditions, and __tcp_push_pending_frames invokes tcp_write_xmit when the socket is not closed. The caller's socket lock and execution context matter; the function is not a promise that bytes were acknowledged before send returns.

tcp_write_xmit selects eligible queued data under congestion, receiver-window, pacing, segmentation and other constraints. Its transmission branch (GPL-2.0-only; later lines omitted) is:

		if (unlikely(tcp_transmit_skb(sk, skb, 1, gfp)))
			break;

A nonzero transmit result breaks this loop; it is not counted by the immediately following tcp_event_new_data_sent as a successful new send. That event follows successful transmit submission in this branch and advances relevant sender state. tcp_transmit_skb can clone an skb so TCP retains state needed for retransmission while a lower-layer instance travels through IP. A lower-layer acceptance still is not an acknowledgement from the remote application.

In __tcp_transmit_skb, which tcp_transmit_skb calls, net/ipv4/tcp_output.c, v6.12, GPL-2.0-only (surrounding code omitted), the clone decision is:

	if (clone_it) {
		oskb = skb;

		tcp_skb_tsorted_save(oskb) {
			if (unlikely(skb_cloned(oskb)))
				skb = pskb_copy(oskb, gfp_mask);
			else
				skb = skb_clone(oskb, gfp_mask);
		} tcp_skb_tsorted_restore(oskb);

The original queued skb stays with TCP while a descriptor goes down the stack, so retransmission state survives the send. A payload copy happens only when the original is already cloned (pskb_copy); otherwise the clone shares data. The excerpt stops before the NULL check that returns -ENOBUFS.

Receive, acknowledgements, and a missing segment#

On the established receive path, tcp_rcv_established has fast and slow branches. In a slow branch it validates incoming state, calls tcp_ack, processes urgent data where relevant, and calls tcp_data_queue for segment text. ACK processing updates send-side state and can free acknowledged retransmission-queue data; data queuing handles in-order, duplicate and out-of-order cases under socket and queue rules. Network receive processing can run in softirq-related context, while userspace operations use socket locking; exact deferral and backlog behavior depend on the path. A packet with invalid flags or failed validation can be dropped rather than delivered.

In the IPv4 entry point, tcp_v4_rcv, v6.12, GPL-2.0-or-later (header pull, socket lookup and state handling omitted), the first checks are:

	drop_reason = SKB_DROP_REASON_NOT_SPECIFIED;
	if (skb->pkt_type != PACKET_HOST)
		goto discard_it;

	/* Count it even if it's bad */
	__TCP_INC_STATS(net, TCP_MIB_INSEGS);

Packets not addressed to this host are discarded before TCP looks at them, and the segment counter increments before header validation, so a bad segment still appears in the input count. The excerpt does not show the tcphdr pull, checksum handling, socket lookup, or the state machine that follows.

Use a synthetic example with the receiver's RCV.NXT = 1000 and three 100-byte segments covering [1000,1100), [1100,1200), and [1200,1300). The first arrives: RCV.NXT becomes 1100 and its bytes can become available in order. The middle segment is lost. The third arrives and can be placed in the out-of-order queue; the cumulative ACK remains 1100 because byte 1100 is missing. A duplicate ACK can communicate that the contiguous boundary has not advanced; if SACK is negotiated and used, SACK blocks can also describe later received ranges. Eventually, a loss-detection mechanism or retransmission timeout causes the missing range to be retransmitted. Once bytes [1100,1200) arrive and are accepted, TCP can join the buffered later range, advance RCV.NXT to 1300, and deliver the contiguous bytes. Exact ACK timing, packet count and recovery choice are not fixed by this example. No packet loss was induced in the lab.

A duplicate segment may contain data already received and must not cause duplicate stream bytes. A corrupt checksum or unacceptable sequence/state can cause a drop. If the sender cannot recover within its configured policy, the connection can fail; reliable stream semantics do not mean an indefinitely retried connection can never error. An application still has to process short reads, EOF and errors, and supply its own message framing.

Congestion, flow control, and recovery#

The receiver advertises a receive window (rwnd) reflecting how much sequence space it can accept under the protocol's rules. A zero window can stop ordinary new data, with probe behavior used to discover later window opening. This protects receiver capacity. The congestion window (cwnd) limits how much the sender may put into the network based on congestion-control state; it is not the same as the receiver's advertised memory. Available transmission is constrained by both windows, bytes already in flight, pacing, segmentation and other rules. ssthresh is a congestion-control threshold used in algorithms' transitions; its exact meaning and updates depend on the algorithm and state.

SACK reports noncontiguous received ranges when negotiated and can help the sender identify which data is outstanding. RACK uses transmission timing and delivery information for loss inference; tail loss probe (TLP) can send a probe to elicit feedback before a full retransmission timeout in applicable cases. The retransmission timeout (RTO) remains a recovery mechanism when feedback does not arrive. These mechanisms are conditional, interact with reordering and negotiated options, and should not be reduced to a universal rule that a fixed number of duplicate ACKs always triggers a particular retransmission. See net/ipv4/tcp_input.c, net/ipv4/tcp_output.c, and net/ipv4/tcp_timer.c.

CUBIC and BBR are different congestion-control algorithms, not protocol versions. CUBIC is implemented in tcp_cubic.c; BBR's availability depends on the build and its selection can be changed under relevant policy. The default is a build and runtime configuration fact, not a claim that every v6.12 machine always uses CUBIC or BBR. Record the selected algorithm before interpreting measurements. Offloads can make a local capture show larger packets than appear on the wire; an skb is not necessarily one physical segment.

Shutdown, timers, and ownership#

A FIN closes one direction of the byte stream and consumes sequence space. shutdown(SHUT_WR) lets an application stop sending while still receiving; the peer reads EOF only after previously queued bytes in that direction have been delivered. Closing both directions and the kernel's connection state machine can involve FIN_WAIT, CLOSE_WAIT, LAST_ACK and TIME_WAIT, depending on which endpoint closes and how. TIME_WAIT protects protocol correctness for delayed segments and connection reuse; it is not proof that an application fd remains open. A reset (RST) indicates an abortive condition and has different consequences from orderly FIN. Timers, retransmission limits and socket references can outlive the user's last fd; the actual close path and namespace configuration matter.

Backpressure can make a blocking send wait for memory or window progress, or make a nonblocking send return EAGAIN or partial progress. The send queue's contents are not all necessarily on the wire. Zero-copy send options introduce an additional buffer-lifetime contract: userspace must wait for the relevant completion notification before reusing memory where the API requires it. Such completion concerns buffer reuse; it does not mean the remote application has processed the bytes. These conditions should be recorded separately in a performance or correctness investigation.

Tracking sender state through acknowledgement#

Suppose the sender starts with SND.UNA = SND.NXT = 1000 and the receiver has advertised enough window. After new bytes [1000,1100) are transmitted, SND.NXT can advance to 1100 while SND.UNA remains 1000. An acknowledgement of 1100 advances the cumulative acknowledged boundary to 1100, allowing TCP to release corresponding retransmission state when all other references and rules permit. If the application has already called send, those bytes may still occupy send-side memory before acknowledgement. A successful send count reports local acceptance under the socket API, not an end-to-end durable receipt.

If bytes [1100,1200) are sent but an ACK has not yet arrived, SND.UNA remains 1100. Sending [1200,1300) depends on the current send and receive windows, congestion state and other constraints. An ACK carrying 1100 again does not advance the cumulative boundary, although accompanying SACK information can describe later received data. A retransmitted segment uses sequence space already assigned to the original bytes; it does not create a second copy in the ordered application stream. Sequence-number wrap is handled with TCP's serial-number comparison rules, so ordinary unbounded integer comparisons are not a correct implementation of every TCP check.

Connection establishment and cleanup limits#

A SYN received by a listening socket can create a request socket representing an incomplete connection. The SYN-ACK and final ACK transition the state toward a full child socket, and a listener's accept queue holds children until accepted or otherwise removed. The backlog is bounded and subject to implementation and configuration; a server that stops accepting can encounter overflow behavior even if it continues receiving SYNs. SYN cookies and TCP Fast Open are examples of alternate paths, so a trace must record whether those features were active. tcp_v4_syn_recv_sock creates and initializes a child for its applicable path; errors and races with duplicate packets require their own reference and cleanup handling.

After a connection is established, an application can close its sending direction while continuing to read. If A sends abc then shuts down writing, B must first receive the queued bytes before observing EOF from A's direction. B can still send a reply if its own sending direction remains open. A reset differs: depending on state and API timing, the peer can observe an error rather than a clean EOF. A timeout can eventually terminate a connection even though TCP normally retransmits, and the application must decide what a partially completed exchange means.

What a congestion measurement can and cannot say#

Congestion control responds to evidence about network delivery; receive flow control reflects the peer's advertised buffer capacity. A small send rate can result from either, from application pacing, from socket memory, from CPU limits, or from a bottleneck outside TCP. Observing a small congestion window alone does not prove a particular bottleneck without workload and timing context. In a controlled test, record the selected algorithm, RTT, MSS, advertised window, retransmission events, pacing and offload settings as available. ss -tin is a useful snapshot when supported, but its fields are not a full packet history.

CUBIC's window evolution and BBR's model-based approach are implemented by different congestion-control modules; their exact behavior depends on the v6.12 code, socket state and configuration. Neither eliminates receiver-window limits, socket-buffer limits or the need to handle network errors. RACK, TLP, SACK and RTO interact with recovery state; a lab that merely transfers six bytes over loopback does not validate any one of those loss mechanisms. The synthetic missing-segment diagram is an explanation of ordered delivery and cumulative acknowledgement, not a measured experiment.

Reading a connection state snapshot#

A snapshot of a listener and a connected socket can show different queues and states even though both are associated with the same local port. The listener's pending connection queues relate to accept and handshake work; a child socket's send and receive queues relate to its own byte stream. A server that has accepted a child may close that child while continuing to accept other clients. A client that half-closes its write side can still be waiting for a response. Confusing listener state with child state leads to incorrect conclusions about backlog pressure and connection termination.

For a transfer that ends with an error after some bytes were accepted, the application must decide whether retrying the higher-level operation is safe. TCP can report ordered bytes but cannot tell an application whether the peer committed a database update or processed a request before the connection failed. An idempotency token, application acknowledgement or transaction protocol may be needed for that decision. Similarly, TIME_WAIT is a TCP state supporting protocol correctness, not an application-level receipt. A loopback success demonstrates the test's byte exchange only, not that a network will behave the same under loss or that a remote service would commit the data.

Practice#

Source-only route:

rg -n -A45 'int tcp_sendmsg_locked\(' "$KERNEL_SRC/net/ipv4/tcp.c"
rg -n -A30 'static bool tcp_write_xmit\(' "$KERNEL_SRC/net/ipv4/tcp_output.c"
rg -n -A35 'void tcp_rcv_established\(' "$KERNEL_SRC/net/ipv4/tcp_input.c"
rg -n 'tcp_ack\(|tcp_data_queue\(' "$KERNEL_SRC/net/ipv4/tcp_input.c"

Confirm v6.12 and follow the push and established slow path, including the surrounding branches; the first 30 lines of a long function are not its complete behavior. Draw the synthetic sequence intervals, cumulative ACK and out-of-order queue. State which losses and recovery actions are assumed rather than observed.

Run an own-process loopback server and client. It binds only 127.0.0.1 and an OS-selected ephemeral port:

python3 - <<'PY'
import socket
listener = socket.socket(); listener.settimeout(2)
listener.bind(('127.0.0.1', 0)); listener.listen(1)
client = socket.socket(); client.settimeout(2)
try:
    client.connect(listener.getsockname())
    peer, _ = listener.accept(); peer.settimeout(2)
    try:
        client.sendall(b'abcdef'); client.shutdown(socket.SHUT_WR)
        chunks = []
        while True:
            data = peer.recv(2)
            if not data: break
            chunks.append(data)
        print('bytes:', b''.join(chunks), 'EOF: true')
        peer.sendall(b'ok'); peer.shutdown(socket.SHUT_WR)
        reply = bytearray()
        while True:
            data = client.recv(2)
            if not data: break
            reply.extend(data)
        print('reply:', bytes(reply), 'EOF:', True)
    finally: peer.close()
finally: client.close(); listener.close()
PY

Expected bytes: b'abcdef' EOF: true and reply: b'ok' EOF: True. Reads use two-byte buffers but TCP may return shorter chunks; the loop assembles the stream. A timeout or unavailable loopback stack is a valid failure to report. If ss is installed, ss -tin is a read-only snapshot, but this short-lived connection may already be gone and other displayed sockets are not this test. Do not add packet loss, firewall rules or host sysctls. Deliver the observed stream and the separately labeled synthetic loss timeline.

Knowledge check#

1. Why is an accepted socket different from the listening socket?
2. In the synthetic loss example, why does the cumulative ACK stay at 1100 after bytes 1200–1299 arrive?
3. How do cwnd and rwnd limit different risks?
4. Does tcp_transmit_skb returning success prove that the peer application read the bytes?
5. What does FIN half-close mean, and how does it differ from RST?

Answers#

1. The listener maintains passive connection state and accepts child sockets representing individual connections; it can continue listening after one accept.
2. The range beginning at 1100 is missing, so the receiver has not obtained contiguous bytes beyond that boundary, even if it buffers later bytes.
3. cwnd constrains network flight based on congestion control; rwnd reflects the receiver's advertised acceptance window. Both can limit sending.
4. No. It indicates success at that transmit boundary, not remote acknowledgement or application consumption.
5. FIN closes one stream direction orderly after prior bytes; RST is an abortive reset with different error and state effects.

Checkpoint#

Save the loopback output, a state/sequence diagram and the source path with alternative branches marked. Pass when you distinguish send acceptance from ACK and application delivery, correctly advance the synthetic ACK after the gap, and explain both flow and congestion control. Revisit the receive and transmit functions and retry if needed. This is practice, not certification.

Sources#

• TCP send and push, Linux v6.12.
• TCP transmission and retransmission.
• TCP ACK and receive queues.
• IPv4 TCP connection handling.
• TCP timers.
• CUBIC.