Bash, coreutils, pipelines, and job control

Before you start#

Review shell habits, process creation and waiting, pipes and IPC, and userspace libraries. The source examples use GNU Bash 5.2 and GNU coreutils 9.5 as separate historical baselines. They do not imply that the host runs these versions or that every shell implements Bash extensions.

Learning goals#

Trace parsing, expansion, redirection, process creation and descriptor cleanup for a pipeline. Explain the difference between a builtin and an external executable, foreground jobs and process groups, ordinary pipeline status and pipefail. Follow a concrete coreutils read/write loop and handle short I/O, EOF and errors without treating a synthetic diagram as a captured process trace.

Source walkthrough#

A shell is a userspace interpreter and process coordinator, not the kernel and not an essential single implementation of the command line. It reads text, tokenizes and parses it into structured commands, applies expansions and redirections, and executes a builtin, function or external program according to the command and its environment. An abstract syntax tree (AST) records structure such as a pipeline or redirection; the original text is not simply passed intact to the kernel as a command. When the shell starts an external program, the kernel receives an executable path, arguments and environment through an execution interface, not shell metacharacters to interpret.

Bash 5.2's execute_cmd.c is GPL-3.0-or-later according to the file notice. The following excerpt is inside execute_pipeline; its loop header and later cleanup are omitted:

      if (pipe (fildes) < 0)
	{
	  sys_error (_("pipe error"));
#if defined (JOB_CONTROL)
	  terminate_current_pipeline ();
	  kill_current_pipeline ();
	  UNBLOCK_CHILD (oset);
#endif /* JOB_CONTROL */
	  last_command_exit_value = EXECUTION_FAILURE;

The two entries in fildes are the new pipe's read and write descriptors when pipe succeeds. This excerpt instead handles creation failure: it reports an error; when job control is compiled in, it terminates the partially constructed pipeline, handles the child-signal mask and records failure. The following source comments say unwind protections take care of open descriptors before control returns to the top level. A pipe failure after some children started cannot be treated as if no resources were acquired. The surrounding implementation tracks which ends each child must close, and its configuration branches matter; this short excerpt is not a complete Bash execution trace.

Coreutils 9.5's src/cat.c is GPL-3.0-or-later as stated in its file header. In simple_cat, the following exact lines occur in the input loop; surrounding read, write and error branches are omitted:

      size_t n_read = safe_read (input_desc, buf, bufsize);
      if (n_read == SAFE_READ_ERROR)
        {
          error (0, errno, "%s", quotef (infile));
          return false;
        }

In the pinned 9.5 source this local variable is ssize_t and the call is read, not safe_read; cat.c includes full-write.h and uses full_write for output. This exact source distinction matters: do not attribute retry-on-EINTR behavior to the read call here. The separate gnulib safe_read interface does retry EINTR, but it is not the call shown in this simple_cat implementation. The conditional detects a negative read result; error reports the input name and current errno, and the helper returns false. The following branch treats a zero result as EOF and returns success for this input. The loop otherwise uses full_write for the block and invokes write_error if the required count was not written. A pipe experiment using cat is not proof that this particular helper ran on the host.

Parse first, then expand words#

For printf '%s\n' "$name" | cat > result, the parser recognizes two commands connected by a pipe and a redirection on the second command. Quotes and redirection operators have syntactic meaning at parse time. "$name" later undergoes parameter expansion within double quotes, preserving its expansion as one argument even if it contains spaces or wildcard characters. Single quotes preserve the literal format string. The shell does not recursively parse arbitrary characters produced by $name as new shell syntax; treating data as code through eval is a separate and dangerous operation.

Bash has a defined expansion order that includes brace expansion where applicable, tilde expansion, parameter and variable expansion, arithmetic and command substitution, word splitting, filename expansion and quote removal. Details and exceptions matter, particularly for "$@", assignments, redirection operands and contexts where splitting is suppressed. Quoting an expansion protects it from ordinary word splitting and pathname expansion; it does not validate its meaning or make an option-looking argument harmless to the called program. Use -- where a particular utility documents it, and avoid treating untrusted strings as shell code. A wildcard that matches filenames is expanded before the program receives its arguments, subject to shell options; an unmatched pattern's behavior can vary with those options.

Command substitution runs commands and substitutes their standard output, with trailing newlines removed by shell rules. It is not a safe binary-data transport: shell variables cannot reliably hold NUL bytes. Exit status of a substitution can be obscured by an enclosing command, and subshell state changes do not normally update the parent shell. Distinguish the shell's own parsing error, an expansion failure, redirection failure, executable lookup failure and the invoked program's status. Those failure stages imply different commands were or were not run.

A builtin such as cd must affect the current shell if its change is to persist there. An external cd process could only change its own working directory. Other builtins can run in a pipeline or subshell context where changes do not propagate to the interactive parent. Bash offers options that alter some pipeline behavior; do not assert that every stage always runs in a separate process under every option. An external command generally requires a child execution context and exec of a selected program; a shell can choose optimizations when semantics permit. PATH lookup selects a candidate based on environment and shell rules. An absolute executable path avoids that lookup but still does not prove file contents or trustworthiness.

Redirection and descriptor ownership#

For the model printf 'abc\n' | cat > result, imagine an initially available pipe pair read end 3 and write end 4. The producer's standard output, descriptor 1, is connected to the write end; the consumer's standard input, descriptor 0, is connected to the read end. The consumer's output redirection opens or creates result in the working directory subject to permissions, umask and shell options, and makes its descriptor 1 refer to that file. The exact intermediate descriptor numbers and ordering are illustrative, not guaranteed values from a host trace.

After duplicating the needed ends, each execution context closes redundant pipe descriptors. The parent shell also closes its own copies as appropriate. A pipe reader sees EOF only after all references to its write end have been closed and buffered bytes have been consumed. If a forgotten write descriptor remains open, the reader can wait despite the intended producer exiting. Conversely, when all read ends are closed, a writer can encounter EPIPE and SIGPIPE according to signal handling. Descriptor duplication creates another reference to the same underlying open-file description; closing one integer does not necessarily close all copies. This is why cleanup and ownership are essential even in a tiny command line.

In Bash 5.2 execute_cmd.c, GPL-3.0-or-later per its notice, the parent's side of the pipe loop in execute_pipeline drops its copies after handing the left-hand command to execute_command_internal (surrounding code omitted):

      if (prev >= 0)
	close (prev);

      prev = fildes[0];
      close (fildes[1]);

close (fildes[1]) removes the parent's copy of the new pipe's write end, one of the references that must disappear before a reader sees EOF. The excerpt does not show how the child arranges its own descriptors, the unwind protections above it, or the job-control branches.

Redirections are processed in order. command >out 2>&1 first sends standard output to out and then duplicates that current destination for standard error. command 2>&1 >out first sends standard error to the old standard output, then changes standard output; the destinations differ. If opening out fails, the intended command may not run. A shell builtin can temporarily redirect its own descriptors and must restore them afterward; an external command's child needs its own correct descriptors before execution. File opening can truncate a target, so practice with private files, not valuable data. Here the lab uses a newly created temporary directory and does not operate on host service configuration.

Now let a producer write four bytes and exit with status 7 while a consumer reads the bytes and exits 0. Without pipefail, Bash normally reports the status of the last pipeline component, so the pipeline status is 0. With set -o pipefail, it reports the rightmost nonzero component's status, here 7. This does not mean all consumers completed successfully in a general multi-stage pipeline, nor does it establish that output reached durable storage. PIPESTATUS captures component statuses in Bash, but subsequent commands can change it, so copy it immediately if needed. A downstream early exit can also change an upstream status through SIGPIPE; test that case separately rather than assuming every failure indicates the same cause.

Jobs, signals and process lifetime#

A job is a shell's bookkeeping for one pipeline or command group. With interactive job control, a shell normally arranges a process group for a foreground pipeline, manages the terminal's foreground process group and can regain terminal control when the job stops or exits. A session and controlling terminal are kernel concepts; a process group is not identical to a single process or a job table entry. Background jobs do not automatically have the same right to read the controlling terminal. Terminal-generated signals are directed according to foreground process-group rules. The exact behavior depends on interactive mode, terminal availability and shell settings; the noninteractive lab below does not exercise a terminal job-control session.

Children terminate independently. SIGCHLD notifies a parent about child state changes, but signals can coalesce and do not substitute for collecting each relevant status with wait operations. Bash blocks and unblocks child-related signals in parts of pipeline construction to protect its job bookkeeping against a child exiting at an inconvenient point. A race can otherwise arise if a short-lived child exits before the shell has recorded it. A signal handler or event loop must preserve its own invariants; application-level logic should not assume one signal corresponds to exactly one child. The kernel maintains child status until an appropriate wait or other configured behavior disposes of it.

The POSIX shell language describes a portable subset and allows implementation variation. Bash provides extensions such as arrays, [[ ... ]], PIPESTATUS and pipefail; a script that depends on them should explicitly invoke Bash rather than claim /bin/sh portability. Likewise, GNU coreutils options are not automatically present in BusyBox or another Unix implementation. Record bash --version and cat --version before comparing observed behavior with a pinned source release. A successful test on the host demonstrates that host's executable and settings, not a complete source trace for Bash 5.2 or coreutils 9.5.

The coreutils loop and failure paths#

In the plain-copy branch, simple_cat reads into a caller-supplied buffer, checks SAFE_READ_ERROR, returns success on zero, and otherwise asks full_write to transfer the read count to standard output. The buffer contains only the number of initialized bytes returned by the read; writing its entire capacity would leak stale data or produce unwanted bytes. Read can be short before EOF, particularly for pipes and devices, so the next loop iteration must request more. A robust write helper advances after partial writes and handles an interruption according to its contract. An error after some output leaves a partial file or stream; retrying an entire operation can duplicate bytes.

In coreutils 9.5 at the pinned commit, src/cat.c, GPL-3.0-or-later per its notice, the end of the simple_cat loop reads (surrounding code omitted):

      /* End of this file?  */

      if (n_read == 0)
        return true;

      /* Write this block out.  */

      if (full_write (STDOUT_FILENO, buf, n_read) != n_read)
        write_error ();

In this excerpt a zero count returns success and a failed write is reported by write_error. The read call and its error branch lie above these lines; at this commit the file uses safe_read and tests SAFE_READ_ERROR, which does not match the plain read described earlier in this section.

The actual GNU coreutils source includes gnulib-derived helper interfaces and generated build machinery; a source checkout may require its own dependencies before rebuilding. This chapter reads the pinned 9.5 source but does not build that project. The src/cat.c source includes more optimized and option-dependent paths than the simple loop. The Bash parser and Bash execution source are entry points for different phases, not a claim that one search reveals every macro or conditional path.

For the modeled four-byte request, suppose cat reads two bytes twice. It sends each returned count through the full-write helper; a partial underlying write may require another attempt. When the producer closes its final write reference and the queued bytes are drained, read returns zero. If instead an input read reports an error, the explicit error branch reports failure rather than treating it as EOF. If output cannot accept data, the write branch reports an error or signal behavior according to the environment. A successful consumer exit says its own operation succeeded under its interface; it does not certify the producer's exit, file durability, or the correctness of the producer's data.

Practice#

Run the following in an ordinary noninteractive Bash process. It creates only a private directory and removes it afterward; it does not modify shell startup files or the host terminal. It requires Bash and coreutils-style cat:

work=$(mktemp -d)
bash --noprofile --norc -c '
  set -o pipefail
  { printf "abc\\n"; exit 7; } | cat > "$1/out"
  status=$?
  printf "pipefail status=%s\n" "$status"
  printf "content="; cat "$1/out"
  { printf "xyz\\n"; exit 7; } | cat > "$1/out2"
  statuses=("${PIPESTATUS[@]}")
  printf "components=%s,%s\n" "${statuses[0]}" "${statuses[1]}"
  set +o pipefail
  { printf "ok\\n"; exit 7; } | cat > "$1/out3"
  printf "default status=%s\n" "$?"
' _ "$work"
rm -rf -- "$work"

Expected output shows status 7 with pipefail, content abc, component statuses 7,0, and status 0 after disabling pipefail, assuming the private file writes succeed. PIPESTATUS must be copied immediately because later commands replace it. The exact buffering, process IDs and descriptor numbers are not measured by this test. If the temporary directory or output cannot be created, the consumer can fail and the result differs; report the error rather than forcing a permission change. The shell exits and the temporary directory is removed. The deliberately failing producer is a private child only.

For a source-only alternative, draw the two-stage AST and the modeled descriptor graph with read end 3 and write end 4. Starting with producer status 7 and consumer status 0, calculate both pipeline statuses, then describe why retaining an extra write end prevents EOF. Distinguish this worked model from a captured trace. Deliver the diagram, status calculation and one explanation of how read error differs from EOF.

Knowledge check#

1. Why does quoting "$name" change argument formation, and why is it still not validation of the argument's meaning?
2. Which descriptor references must close before a pipe reader can observe EOF?
3. Why does producer | consumer normally return 0 when the producer returns 7 and the consumer returns 0, and what changes with pipefail?
4. What does the quoted Bash pipe-failure branch protect against, and why is a signal not a substitute for waiting?
5. Why must cat use the returned read count rather than its buffer capacity, and what distinguishes EOF from a read error?

Answers#

1. Double quotes suppress ordinary splitting and filename expansion of that parameter expansion, preserving it as one argument in this context. The receiving utility can still interpret the resulting string as an option or a dangerous pathname; quoting is not semantic validation.
2. All open references to the pipe's write end must close, including copies in the parent or unrelated children, and buffered data must drain. Closing only the intended producer's descriptor is insufficient if another writer reference remains.
3. The ordinary Bash pipeline status is the final command's status, here the consumer's 0. With pipefail, the rightmost nonzero component determines the result, here 7. Component statuses must be captured before another command changes PIPESTATUS.
4. A pipeline may already have children and descriptors when pipe fails. Bash reports the failure and conditionally terminates the partial job while its unwind mechanism closes resources. Child signals can coalesce; the shell must still account for child state through waits and its job table.
5. Only the returned count denotes initialized input bytes. A zero count represents end of that input in the helper, while the separate error sentinel causes an error report and failure. A positive short read is neither necessarily EOF nor an error.

Checkpoint#

Submit the pipeline AST, descriptor ownership graph, status table and either the private lab output or the source-only calculation. Ready means explaining redirection order, all-writers-closed EOF, a partial write and the producer/consumer status distinction. If uncertain, return to pipes, redraw the model and retry without touching host shell configuration. This is a learning artifact, not certification.

Sources#

• Bash 5.2 execute_cmd.c, pinned source
• Bash 5.2 parser source
• Bash 5.2 substitution source
• GNU coreutils 9.5 cat.c
• GNU coreutils 9.5 copy.c
• Bash 5.2 official release archive