Trace and debug userspace boundaries

Before you start#

Review system calls, ELF and dynamic linking, process creation, and build tools. This chapter inspects strace 6.10 and GDB 15.1 source as independent historical examples. It observes only a self-authored child program when the relevant host tools and permissions are available.

Learning goals#

Distinguish system-call tracing, library-call interception and source-level debugging. Explain entry and exit stops, tracee identity, error translation, symbol and debug-information matching, and the effects of tracing on a program's timing. Construct a bounded observation with a stated filter and a synthetic alternative that is not presented as captured output.

Source walkthrough#

strace observes supported system-call and signal events through operating-system tracing facilities, commonly ptrace on Linux. The tracer and tracee are different tasks; when a tracee stops, the tracer can inspect the stop, decode relevant registers or supplied syscall information, print a representation and resume it. Exact stop types, architecture ABI, permissions and strace options matter. A system-call trace is not a C function trace: a library function can execute entirely in userspace, call several syscalls, use a different syscall than its name suggests, or be optimized away. A traced syscall name is the tracer's decoded interpretation of the observed event under its supported ABI.

The sequence is a teaching model of those stops, not a captured trace. Signals, seccomp filters and syscall restarts add or change stops, so use it for the ordering question only.

The following exact excerpt is from strace 6.10 src/strace.c, trace_syscall; the file is LGPL-2.1-or-later. The enclosing branch and later exit path are omitted:

		int res = syscall_entering_decode(tcp);
		switch (res) {
		case 0:
			return 0;
		case 1:
			res = syscall_entering_trace(tcp, sig);
		}

tcp points to strace's per-tracee control block, not the application's C thread-control block. At an entry event the decoder determines how to continue. A zero result returns immediately; a result of one invokes the entry tracing path and can update the signal parameter. Other results continue according to the surrounding code. The function later finishes entry processing and has a separate exit branch. This is conditional control flow in one release, not proof that each syscall produces exactly two visible lines under every filter, signal, restart or seccomp configuration. The tracee may terminate, execute a new image or enter an architecture-specific stop in between.

GDB 15.1 gdb/infrun.c, wait_for_inferior, carries a GPL-3-or-later notice. This exact excerpt omits surrounding loop and event handling:

      ecs.ptid = do_target_wait_1 (inf, minus_one_ptid, &ecs.ws, 0);
      ecs.target = inf->process_target ();

GDB waits for a target event, records the returned process/thread identity in the execution-control state and records the target associated with the inferior. The following code handles the event and determines whether to wait again. An inferior is GDB's representation of a debugged program; its threads, address spaces and state may change at fork, exec or exit. The wait result is not itself a source-level stack trace. GDB needs a matching executable and debug information and must interpret the target's state under the correct architecture.

Entry, exit and errors#

For a simple x86-64 Linux example, a process requests openat on a nonexistent private pathname. The architecture's syscall ABI supplies the syscall number and arguments in its designated registers; the kernel may return a negative error encoding in the raw register. The C library wrapper normally converts that to userspace -1 and sets thread-local errno to ENOENT. A trace line such as openat(AT_FDCWD, "missing", O_RDONLY) = -1 ENOENT is a useful decoded representation, but its pathname and flags are what the tracer decoded, not proof of why the file was absent or what another process saw.

The steps above are a source-based model of where a negative kernel result is converted into strace's fields. They are not a captured trace.

In strace 6.10 src/syscall.c, get_error fills the tracer's per-tracee result fields. The file carries an LGPL-2.1-or-later notice. Lines 1507-1522 are quoted exactly:

static void
get_error(struct tcb *tcp, const bool check_errno)
{
	if (ptrace_syscall_info_is_valid()) {
		if (ptrace_sci.exit.is_error) {
			tcp->u_rval = -1;
			tcp->u_error = -ptrace_sci.exit.rval;
		} else {
			tcp->u_error = 0;
			tcp->u_rval = ptrace_sci.exit.rval;
		}
	} else {
		tcp->u_error = 0;
		arch_get_error(tcp, check_errno);
	}
}

When ptrace syscall information is valid, a negative raw result becomes u_rval = -1 and a positive u_error (the negated raw value). Otherwise an architecture-specific helper reads the error. These are strace's internal fields, not the libc wrapper's return value. The excerpt does not say why the call failed, and it does not cover the other architecture paths.

A signal may interrupt a blocking syscall. Depending on the interface, signal disposition, restart rules and observation point, the tracer may report an interruption or an internal restart-related state followed by resumed work. Do not count every printed entry as a completed business operation. strace -f can follow new tasks created by a traced process when supported; without it, a child may perform the operation outside the selected trace. Threads have distinct Linux task identities even where they share memory and file tables. A trace filter can reduce output, but it can also hide a prerequisite call or a failure path. Record the filter, tool version and whether child following was enabled.

At exec, the process image changes while the traced task's identity and descriptor state have rules of their own; close-on-exec descriptors close as part of a successful exec. A failed exec leaves the old image running. A tracer has to distinguish syscall-entry and exit state across these transitions. Seccomp can add stops or deny syscalls according to the configured environment; tracing behavior is not one universal entry/exit sequence. In the pinned src/syscall.c, entry decoding and error formatting are separate paths. Decoding an error name requires the correct ABI and return convention; a trace for one architecture cannot be interpreted by blindly applying another's registers.

In the same strace 6.10 release, src/strace.c, the event loop chooses how to resume a tracee. The file carries an LGPL-2.1-or-later notice. Lines 3962-3974 are quoted exactly; the switch that follows is omitted:

	unsigned int restart_op;
	unsigned int restart_sig = 0;
	enum trace_event te = wd ? wd->te : TE_BREAK;
	/*
	 * Copy wd->status to a non-const variable to workaround glibc bugs
	 * around union wait fixed by glibc commit glibc-2.24~391
	 */
	int status = wd ? wd->status : 0;

	if (current_tcp && has_seccomp_filter(current_tcp))
		restart_op = seccomp_filter_restart_operator(current_tcp);
	else
		restart_op = PTRACE_SYSCALL;

By default the resume operation is PTRACE_SYSCALL. With a seccomp filter the operator comes from seccomp_filter_restart_operator. A seccomp-filtered tracee can therefore produce a different stop pattern from the simple entry and exit model. The excerpt does not show which operator a particular filter needs, and it does not show what the omitted switch does with each event.

Descriptor annotations such as path decoding can be useful but have limits. A descriptor number can be closed and reused, and a pathname can be renamed or unlinked while an open file remains accessible. A traced openat path is interpreted relative to its directory descriptor and process namespace, not necessarily the observer's current directory. /proc visibility, permission restrictions and races can limit descriptor details. Avoid collecting unrelated processes' command lines, environment variables, credentials, file contents or network payloads. A narrow trace of a self-authored child is sufficient for this lesson.

Function tracing and debugger information#

ltrace commonly observes selected dynamically linked library calls by instrumenting supported linkage paths. It is not a universal inventory of every function invocation. A function inlined by the compiler has no independent call to intercept; a statically linked call or a call bypassing a supported PLT interception path may not appear. Architecture, loader behavior, IFUNC resolution, interposition, optimizations and secure-execution restrictions can further change visibility. A missing ltrace line is therefore not proof that no equivalent computation occurred. Conversely, a logged library call is not necessarily a syscall: the call may be answered from a cache or by a userspace fast path.

In the inspected ltrace mirror's pinned handle_event.c, handle_event dispatches event types and handle_breakpoint handles a breakpoint event with process and breakpoint state. The file's notice is GPL-2-or-later. This is a selected source entry point from the identified mirror commit, not a claim that its entire interception implementation or every architecture was audited. It supports the distinction between a tool's breakpoint-based observation and a complete compiler-level function log.

GDB can stop a self-authored process at a breakpoint, inspect registers and memory, unwind frames, and associate machine addresses with source using available symbol tables and debug information. DWARF describes types, variables and source locations; a build ID or other debug-file association can help locate matching separate debug information. The executable, shared libraries, debug files and source revision must correspond. A stack trace built from mismatched files may have plausible names and wrong line numbers. A stripped executable may retain enough dynamic symbols for loading while lacking local symbols and detailed source locations.

Address-space layout randomization can place a position-independent executable or DSO at a different address on each run. A debugger accounts for mappings and load bias when associating runtime addresses with file offsets and symbols. A raw instruction address without the corresponding mapping and build identity is not portable evidence. Optimized code can inline functions, eliminate variables, reorder instructions or place a variable in different locations at different points; optimized out is an honest debugger result, not proof the source variable never existed. Frame-pointer omission, tail calls and incomplete unwind information can also limit a backtrace.

A debugger's all-stop and non-stop modes, target support and multithreading affect which threads are paused while another is inspected. A paused thread can hold an application mutex or a resource needed by another task. Breakpoints and ptrace stops change scheduling, introduce latency and can hide or expose races. Source-level stepping may jump among lines because optimized instructions do not preserve a one-to-one order. Record compiler options such as -g and optimization, the exact executable and whether the process was started under the debugger. Do not claim that one clean debugging run proves the absence of races.

A worked state and synthetic trace#

Suppose an own-child program calls open("missing-demo-file", O_RDONLY) in an empty private directory and then returns status 2 when it sees failure. For a synthetic, architecture-scoped explanation, imagine x86-64 system-call entry selecting openat with a directory descriptor equivalent to AT_FDCWD, a pointer to the NUL-terminated pathname and flags corresponding to read-only access. The process's current directory makes the relative name resolve inside that private directory. If no matching file exists, the kernel returns the raw negative error for ENOENT; the libc wrapper presents -1 and sets the calling thread's errno to ENOENT.

A possible human-readable line is openat(AT_FDCWD, "missing-demo-file", O_RDONLY) = -1 ENOENT. This line is a teaching model, not captured output. The exact flags may include implementation or architecture details, and some libc or program choices may invoke a different syscall. After printing a message the program exits 2; exit_group, a signal, or a runtime cleanup can produce other visible events depending on execution. A filtered trace of only open calls would not show every later write or exit. A debugger breakpoint on main shows a different boundary: the program has entered userspace code but may not yet have executed the failing open.

The failure is not a race-free proof about the whole machine. Another actor could create a matching file between a check and an open. The lab avoids that ambiguity by using a newly created private directory and a self-authored child, but even then a result belongs to that run. If the tracer reports permission denied, security policy, ptrace availability or a sandbox restriction may be responsible. Do not change Yama, an LSM, kernel settings or privileges to make a demonstration work. Use the synthetic source-only case and state that runtime tracing was unavailable.

Observability and ownership#

A tracer retains per-tracee state while tasks enter, exit, fork or change images. It must stop using a control block when the corresponding tracee state is released, and a debugger must reconcile events with its current inferior and thread records. These internal objects are not the same as the process's file descriptors or application heap objects. Filtering a task out of displayed output does not mean the operating system stopped tracking it. When a tracer dies or detaches, the tracee's continuation or signal behavior depends on the mechanism and options; do not build production recovery assumptions from an example command.

A trace can contain sensitive paths, arguments, data and timing. Store only the minimum necessary evidence in a private directory, and avoid tracing existing services or other users' tasks. Even for your own child, a syscall filter is an information-selection choice: it can omit the very event that caused a later symptom. Compare a hypothesis against the selected evidence, note missing context, and avoid equating correlation with causation. If a failure disappears under the debugger, timing perturbation is one possible explanation rather than proof of a compiler defect.

Permissions can be affected by the tracer's relationship to the child, user identity, dumpability, namespaces, Yama, other Linux security modules and sandbox policy. Starting your own child under a tracer is still subject to those rules. A denied operation is a valid observation. Do not attach to unrelated PIDs, run the tool with elevated privileges, weaken security settings or collect host secrets to force a result. The source-only analysis remains useful but is not runtime evidence.

Practice#

This lab builds one small private program, attempts only an open in its own empty temporary directory, and runs GDB on that child if installed. strace and ltrace are optional; absent tools or denied tracing are reported rather than worked around. No existing process is attached:

work=$(mktemp -d)
cat > "$work/demo.c" <<'C'
#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>
int main(void) {
    int fd = open("missing-demo-file", O_RDONLY);
    if (fd < 0) { printf("open failed: %d\n", errno); return 2; }
    close(fd);
    return 0;
}
C
cc -g -O0 "$work/demo.c" -o "$work/demo"
(cd "$work" && ./demo) || printf '%s\n' 'expected missing-file exit'
if command -v strace >/dev/null; then
  (cd "$work" && strace -e trace=openat ./demo) || printf '%s\n' 'trace or program returned nonzero'
else printf '%s\n' 'strace unavailable'; fi
if command -v ltrace >/dev/null; then
  (cd "$work" && ltrace ./demo) || printf '%s\n' 'ltrace or program returned nonzero'
else printf '%s\n' 'ltrace unavailable'; fi
if command -v gdb >/dev/null; then
  (cd "$work" && gdb -q -nx -batch -ex 'break main' -ex run -ex bt --args ./demo) || printf '%s\n' 'GDB unavailable or tracing denied'
else printf '%s\n' 'GDB unavailable'; fi
rm -rf -- "$work"

The ordinary run should print an error number and exit 2; the || message records that expected failure. On Linux, ENOENT is normally displayed as 2, but compare with the host headers rather than assuming a universal ABI. A permitted GDB run should stop at main and show at least its frame; warnings or denial are valid outcomes. If strace is available, its output may include loader-related opens as well as the missing file. The program is compiled with debug information and no optimization for a clearer demonstration, not as proof that production code behaves identically. The temporary directory is removed afterward.

Source-only alternative: use the synthetic openat line and the entry/exit sequence above. Label each part as argument, decoded return or error; explain the raw-kernel versus libc convention and name one event hidden by an open-only filter. Deliver this table and the GDB matching-source requirements. Do not describe the synthetic line as an observed trace.

Knowledge check#

1. Why does an absent ltrace line not prove that a library's functionality was never used?
2. What is the relationship between a raw Linux syscall error, a libc return value and thread-local errno in the worked example?
3. Why must GDB's executable, debug information and source revision match before trusting a line number?
4. What can a filter on openat hide, and why can a syscall restart complicate line counting?
5. What should a learner do if ptrace is denied for the private child?

Answers#

1. Inlining, static linking, direct calls, IFUNC selection or unsupported interception paths can bypass the events ltrace observes. The tool has a specific coverage model, not a complete record of all computations.
2. The kernel's raw return encodes an error under the architecture ABI. The libc wrapper normally returns -1 and sets that thread's errno to ENOENT; strace formats its own decoded representation of the event.
3. Addresses, code layout and line tables vary across builds. A mismatched debug file or source tree can associate a real instruction with the wrong name or line even when the resulting backtrace looks plausible.
4. It can hide reads, writes, signals, child creation and exits. An interruption and restart can yield multiple observed stops or representations for one higher-level operation, depending on tool and configuration.
5. Record the denial, tool version and scope, then use the supplied source-only explanation. Do not elevate privileges or change host security policy to force tracing.

Checkpoint#

Submit a labeled synthetic or captured own-child observation, the tool/filter/version record, and a matching-debug-information checklist. Ready means distinguishing syscall and function boundaries, explaining the error translation, and identifying at least one source of missing or misleading trace evidence. Retry with the single failed open and no child following if the event sequence is unclear.

Sources#

• strace 6.10 trace loop
• strace 6.10 syscall decoding
• GDB 15.1 inferior event loop
• GDB 15.1 symbol-file implementation