OpenSSH, authentication, channels, and agents

Before you start#

Review Unix sockets and pipes, userspace process control, credentials, and PAM boundaries. This chapter reads OpenSSH portable V_9_8_P1, commit 6849957945754e6551e515f41e8cf3937cda222d, as a historical source snapshot. It is not a current deployment recommendation or a claim about the host's SSH configuration.

Learning goals#

Separate server identity from user authentication, follow channel window and closure state, and explain what an agent can do without exporting a private key. Track descriptors and process references through a synthetic disconnect without connecting to a real server or agent.

Source walkthrough#

SSH is a protocol family with three layers: transport (server authentication, confidentiality and integrity), user authentication (client user to server), and connection (multiplexed logical channels) (RFC 4251 §§ 1, 4.1). Client and server exchange identification and negotiate algorithms and keys. The client must establish that the server host key belongs to the intended server through an appropriate trust process. Encryption to an unverified endpoint does not by itself establish that endpoint's identity. User authentication is a separate step: proving that a user or client holds an acceptable credential does not replace checking the server host key.

The following exact excerpt from OpenSSH portable V_9_8_P1 channels.c, channel_input_window_adjust, has the file's BSD-style redistribution notice. Earlier parsing and later return are omitted:

	if ((new_rwin = c->remote_window + adjust) < c->remote_window) {
		fatal("channel %d: adjust %u overflows remote window %u",
		    c->self, adjust, c->remote_window);
	}
	c->remote_window = new_rwin;

c is the channel object found for the received channel identifier. adjust is the parsed increment to the peer-advertised remote window. The addition is checked for unsigned wraparound before the new value is assigned; an overflowing adjustment is fatal on this source path. A window grants a bounded amount of channel data that can be sent before further adjustment. This code is not a captured session and does not describe every channel type or transport error.

The next exact excerpt from ssh-agent.c, process_sign_request2, carries that file's BSD-style redistribution notice. Later signing and response cleanup are omitted:

	if ((r = sshkey_froms(e->request, &key)) != 0 ||
	    (r = sshbuf_get_stringb(e->request, data)) != 0 ||
	    (r = sshbuf_get_u32(e->request, &flags)) != 0) {
		error_fr(r, "parse");
		goto send;
	}

e is a socket entry containing the request buffer. The handler parses a key description, data to sign and flags; any parse failure logs the error and branches to the response path instead of proceeding with a partially decoded request. The surrounding function allocates buffers and later releases temporary state according to its error path. The agent's ability to sign does not require returning private-key material to the requesting client. Access to an agent socket can nevertheless confer signing authority subject to the agent's constraints and policy.

Transport, host identity and user identity#

After initial identification, SSH negotiates transport algorithms and performs key exchange. The server proves possession of a host private key using the protocol, and the client decides whether the resulting host key is trusted for the intended destination. Host-key verification is a trust decision about the server endpoint, not an account password check. A changed key can mean legitimate key rotation, a configuration change or an attack; it needs verification through a trusted administrative channel. Automatically disabling checks to make an error disappear destroys the evidence needed to distinguish those cases.

After a protected transport is established, user authentication decides whether a requested account may authenticate by an offered method under server policy. The auth2.c source is an entry point for the selected OpenSSH server's authentication dispatch. Public-key, password and other methods have different policies and dependencies. Authentication success is not a promise that every channel request or operating-system operation will be authorized. A server can deny an account, command, forwarding request or resource after transport setup.

Portable OpenSSH has platform and configuration-dependent privilege separation and sandbox paths. A privileged monitor and a less-privileged preauthentication child can split responsibilities; a session process can later run under the authenticated user's context according to policy. Their process boundaries, descriptor passing and cleanup are implementation and build dependent. Do not infer one universal fork tree or claim that a source-only inspection tested the host's sandbox. The selected monitor.c is a starting point for a specific configured path, not proof that every platform uses identical isolation.

A server host-key mismatch and a user-authentication denial arise at different phases. In a synthetic case, a client expects host key H1 but receives H2; it must resolve that trust discrepancy before treating the endpoint as the intended server. In another case the host key is accepted but the server denies user authentication. Changing a username or password cannot establish that H2 is trustworthy, and accepting H2 without verification cannot make denied user credentials valid. The two cases need separate evidence and should not be combined into “SSH failed.”

Channels and flow control#

One authenticated transport can carry multiple logical channels. Each channel has local and remote identifiers, buffers, state, window information and possibly local descriptors. Channel numbers are scoped to the connection; they are not global process IDs or host file descriptors. A channel may represent a session, forwarding stream or another supported type. An optional pseudo-terminal is a terminal interface with its own line discipline and job-control behavior; it is not required for every SSH session (RFC 4254).

For a concrete model, assume channel A begins with a remote window of 8 bytes and channel B with a window of 4. The sender transmits six bytes on A and four on B, leaving windows 2 and 0 respectively. A can send up to its remaining allowance subject to packet-size and other rules; B must wait for a valid window adjustment before sending more channel data. If B receives an adjustment of 3, its modeled allowance becomes 3. This is synthetic arithmetic and not a measured OpenSSH trace. A slow consumer, transport loss or exhausted window can create backpressure without implying that the remote process has exited.

EOF and close are different channel events. EOF indicates that no more data will be sent in one direction under the channel protocol; the channel remains open and the other direction can still carry data. Close participates in terminating the channel; each side responds unless it already sent close, and the channel number is reusable after both sent and received close. Exit status is a separate channel request (recommended by RFC 4254) and EOF alone says nothing about success. Local pipes or sockets can reach EOF at a different time from remote process termination. Implementations handle half-close and outstanding data under the protocol rules (RFC 4254 §§ 5.3, 6.10).

In channels.c, OpenSSH portable V_9_8_P1, channel_input_ieof, under the file's BSD-style notice (the parsing and proxy branches are omitted):

	chan_rcvd_ieof(ssh, c);

	/* XXX force input close */
	if (c->force_drain && c->istate == CHAN_INPUT_OPEN) {
		debug("channel %d: FORCE input drain", c->self);
		c->istate = CHAN_INPUT_WAIT_DRAIN;
		if (sshbuf_len(c->input) == 0)
			chan_ibuf_empty(ssh, c);
	}
	return 0;

The handler passes EOF to chan_rcvd_ieof, whose body is not shown, and only the forced-drain case moves input to CHAN_INPUT_WAIT_DRAIN. The excerpt says nothing about the output side or about channel close.

Consider a race: channel A receives EOF while output remains buffered for the local reader. The implementation must follow its channel state machine so buffered data is handled or an error is reported before resources are released under the applicable rules. If the transport disconnects first, both channels can fail regardless of their individual windows. A stale channel ID from the old transport is not automatically valid in a later connection. The channels.c window and EOF handlers are useful entry points, but a single function does not verify every state transition and cleanup path.

Agent authority and descriptor lifetime#

An SSH agent typically listens on a Unix socket and accepts protocol requests from clients with access under its environment and policy. A client can ask it to sign appropriate data using a loaded identity; the agent need not reveal the private key. A forwarded agent connection can allow a remote environment to request signatures while forwarding is active. That delegates meaningful signing authority even if it does not copy the key itself. Constraints, confirmation, destination restrictions and platform support vary with the agent and configuration. Treat the socket and the context in which it is forwarded as security boundaries.

The sign-request handler allocates buffers, parses a key and data, applies selected checks and eventually produces a response or failure. Malformed input follows an error branch and must not be treated as a valid signature. The request buffer, parsed key, optional confirmation state and response each have lifetimes; a disconnect can occur while a request is pending. A callback or event handler must not write to a closed or reused socket descriptor as though it still belonged to the original peer. Cleanup of temporary buffers and identity references is a separate concern from whether the cryptographic signature operation succeeded.

In ssh-agent.c, OpenSSH portable V_9_8_P1, process_sign_request2, under the file's BSD-style notice (the surrounding key and destination checks are omitted):

	if (id->confirm && confirm_key(id, sig_dest) != 0) {
		verbose_f("user refused key");
		goto send;
	}

A key marked for confirmation must pass confirm_key before sshkey_sign runs, and a refusal goes to the response path without a signature. The excerpt does not show the destination checks or the security-key branch, so it does not list every constraint the agent applies.

A local session can also use pipes for standard input, output and error, or a PTY when requested. Closing one descriptor does not necessarily terminate a process if another reference remains; a child process can outlive a transport unless the server's session and process cleanup handle it. The server must reap children and close descriptors according to its implementation and platform. A channel close, process exit, transport disconnect and child reaping are related but distinct events. A source trace should name which object owns each descriptor and what happens on partial setup failure.

Worked lifecycle and trust table#

At T0 a client verifies host key H1 under its chosen trust policy and then completes user authentication for account U. At T1 channels A and B open on the same protected connection. At T2 B's remote window reaches zero, while A still has two bytes of allowance. At T3 A sends EOF in one direction but may still receive buffered output; B waits. At T4 the transport disconnects before B receives an adjustment. Both channel owners must handle transport failure and release local resources, but neither the unobserved remote command's status nor every byte's delivery can be inferred from the local disconnect alone.

The server may have created child processes and the client may have local input/output descriptors. Process exit and descriptor closure can race: a child can exit while output remains buffered, or a network failure can occur while the child is still running. A manager of these resources needs a clear owner for each close and wait operation. A numeric FD or PID can be reused after cleanup, so a late event must be associated with the correct live channel or process object. This reasoning applies to the synthetic model; no SSH session is run by the practice below.

If the user requested agent forwarding, the remote side's access to a forwarding channel should be considered separately from the trust in H1. An accepted host key identifies an endpoint under the chosen policy; it does not mean every process on that endpoint is trustworthy. The forwarding socket or channel may be accessible to code running with relevant permissions there. An authentication failure, channel denial or agent parse error should be reported at its own boundary, without exposing secret material in logs.

Error boundaries in a multiplexed connection#

An implementation must decide what to do when a channel request is malformed, a local descriptor becomes unwritable or the transport itself fails. A malformed window adjustment can be a protocol error even though another channel still appears healthy. A local pipe error, by contrast, can affect one channel while the shared transport is still alive. The exact response is determined by the protocol and OpenSSH state machine, not by the arithmetic model alone. Record whether the error belongs to transport setup, authentication, a channel request, local I/O or the remote command.

The agent has a separate event loop and socket-entry lifetime from the transport channel. A forwarded connection can disappear while the agent is processing a request. The consumer of the response must associate it with the original connection and avoid writing a completed response into a newly reused descriptor. This is a general ownership invariant, not a claim that the quoted parse block alone proves every possible race safe. For a real source investigation, follow allocation, request parsing, response and teardown under the selected build, then verify with a separately authorized isolated test.

Practice#

This pure model uses integers and lists. It does not start SSH, contact a host, enumerate keys, use an agent or read configuration:

python3 - <<'PY'
windows = {'A': 8, 'B': 4}
for channel, count in [('A', 6), ('B', 4)]:
    if count > windows[channel]:
        raise ValueError('window exhausted')
    windows[channel] -= count
print('synthetic remaining:', windows)
try:
    if 1 > windows['B']:
        raise ValueError('window exhausted')
except ValueError as error:
    print('synthetic blocked:', error)
windows['B'] += 3
print('synthetic B after adjustment:', windows['B'])
PY

Expected values are A=2, B=0, a blocked B send and B=3 after adjustment. This is arithmetic, not a protocol implementation, encryption test or runtime evidence. If Python is unavailable, calculate the same numbers from the worked model. No host key, user identity, agent socket or network connection is accessed.

As a source-only alternative, draw the two channels and their remaining windows at T0–T4. Mark EOF, close and transport failure separately, then write one sentence explaining why a changed host key and denied user authentication occur at different boundaries. Deliver the diagram and an FD ownership table with all identifiers marked synthetic.

Knowledge check#

1. Why is an encrypted connection to an unverified host key not enough to establish server identity?
2. What does remote_window constrain in the selected channel path, and why is overflow checked?
3. How do channel EOF and close differ from an exit status?
4. Why can agent forwarding be sensitive even when the agent does not export a private key?
5. What happens in the quoted sign-request path if parsing an input field fails?

Answers#

1. Encryption protects a channel to whichever endpoint participated; a separate trust decision is needed to associate the host key with the intended server.
2. The peer-advertised window bounds channel data that may be sent under the protocol. Unsigned wraparound could turn an invalid adjustment into misleading state, so the checked branch terminates on overflow.
3. EOF ends data in a direction, close participates in ending the channel, and exit status reports a process result when provided. They are related but not interchangeable.
4. A party with access to the forwarded agent interface may be able to request signatures under its constraints. The absence of private-key export does not eliminate delegated signing authority.
5. The function logs the parsing error and goes to its response path rather than continuing with incomplete key, data or flags. Surrounding cleanup and response handling still matter.

Checkpoint#

Submit the two-channel window timeline, EOF/close/exit distinction and host-key versus user-authentication table. Ready means calculating the remaining windows, identifying agent signing authority and naming an error/cleanup path. Retry with one channel before adding multiplexing and a disconnect.

Sources#

• OpenSSH portable V_9_8_P1 channel source
• OpenSSH portable V_9_8_P1 agent source
• OpenSSH portable V_9_8_P1 authentication
• OpenSSH portable V_9_8_P1 privilege monitor