Userspace investigations and the project atlas

Before you start#

Review portal boundaries, desktop investigation, application sandboxes, and event loops and transactions. This chapter's incident data is synthetic. The atlas is a course map, not a claim to cover every Linux userspace project or a tested combination of its historical baselines.

Learning goals#

Follow an asynchronous portal request through cancellation and cleanup, compare alternative hypotheses across subsystems, design controlled tests with negative controls, and choose a project-specific route to independent evidence and review.

Source walkthrough#

A userspace incident can span a process, D-Bus connection, file descriptor, kernel object and service policy. Evidence from one layer rarely establishes the state of every other layer. We use xdg-desktop-portal 1.18.4, commit 11c8a96b147aeae70e3f770313f93b367d53fedd, as a bounded source example. A request object is a GObject with its own reference and exported D-Bus lifetime; a selected document and any stored fd are separate resources. A cancellation can race with a worker preparing a response.

The first exact excerpt is from xdg-desktop-portal 1.18.4 src/file-chooser.c, send_response_in_thread_func. The file notice grants LGPL-2.0-or-later. Surrounding result construction is omitted:

  if (response != 0)
    goto out;

After acquiring the request lock and reading stored response and options, a nonzero response follows the out path before the success-only document registration loop. At out, the function builds its result, emits a response only if the request is still exported, unexports in that case and completes the task. The branch does not say that all possible cancellation races are solved, nor that a user actually selected or denied a file in this lesson.

The second exact excerpt is from xdg-desktop-portal 1.18.4 src/request.c, request_unexport, also LGPL-2.0-or-later. Surrounding fd retrieval and object release are omitted:

  request->exported = FALSE;
  g_dbus_interface_skeleton_unexport (G_DBUS_INTERFACE_SKELETON (request));
  g_object_unref (request);

The code marks the request no longer exported, removes its D-Bus skeleton and drops a reference. Immediately before these statements it checks a stored fd and closes it if appropriate. The GObject reference and fd are distinct resources. A local pointer or stale request path does not retain a valid exported object after cleanup. A caller must still account for references held by tasks or other callbacks; this short excerpt is not a complete race audit.

Capstone A: canceled file selection#

The following incident is synthetic, not a captured portal log. At 0 ms an application sends a file chooser request for an authored temporary document. At 12 ms the backend reports a nonzero response. At 15 ms the application closes its request. At 20 ms a worker reaches the response code. No document registration or response emission has been supplied as an observed fact. The question is whether a file grant was created or a response was emitted after the request stopped being exported.

Hypothesis A is that the nonzero response correctly skipped document registration. Hypothesis B is that a success response was processed before cancellation, so registration occurred under a different ordering. Both are compatible with a vague report saying “the dialog disappeared.” The actual source reads the stored response while holding the request lock and follows the nonzero branch before the success-only registration loop. At the output point, emission is conditional on request->exported; the source does not make timestamps from separate processes a total order. To predict a concrete result, an investigator needs the lock ordering and exported state observed at the relevant moments, not just wall-clock labels from separate processes.

For the supplied state where the worker reads nonzero response and sees the request already unexported at the guarded output check, the source-based answer is: it skips the success registration loop and does not emit through that guarded block. The task still completes in the shown source path. If a different callback already unexported the request, the worker must not unexport it a second time through this conditional block. This is a worked source interpretation under explicitly supplied state, not a runtime trace and not proof of every reference or fd cleanup path.

An alternative explanation is that a success response was read before cancellation and a document registration was already underway. A controlled isolated test would use a self-authored file, record request identity, response, lock-protected state, export/unexport and document-registration outcome, and vary cancellation timing. A negative control would let a valid selection complete. The test should close its own connections and descriptors and remove its own temporary files; it must not contact the user's existing desktop portal or expose personal documents. No such test ran here.

A hypothetical defect that emitted a response after unexport might be addressed by preserving the export-state check and object reference until the operation's terminal decision, but a real patch must identify the actual broken ordering first. Before and after tests should cover canceled, successful and backend-error responses, duplicate cancellation, client disconnect and fd cleanup. A reviewer should inspect lock ownership and callbacks for deadlock or double-unref. Adding a blanket retry or broad filesystem grant would not address the invariant and could increase authority.

Capstone B: audio graph and device loss#

A second synthetic incident: an application successfully creates a 48 kHz stereo stream, queues 240 frames, and later receives a device-invalidated result. Its window remains visible. One hypothesis is a route or device change after initialization; another is that the client violated buffer acquire/release ordering and received a different error that was misreported. The audio state, application window and desktop session are separate objects. A visible window is not evidence that a PipeWire node remains connected. A queued buffer establishes application-side queueing, not that the server scheduled it or that sound reached a physical endpoint.

Start with the earlier audio chapter and its pinned Wine pulse_get_render_buffer source branch, then follow the selected server's node and connection lifecycle using the audio chapters' independently recorded baselines. Kernel bridge points include device descriptors and ALSA operations, not a universal single call chain. A controlled test would use a separately prepared virtual audio environment, an authored test signal and explicit cleanup; it should compare valid ordering, an oversized request and a controlled endpoint removal. No host microphone or speaker is needed or used here.

The deliverable is a state table with client interface, buffer ownership, server node, endpoint and last returned result. A minimal change proposal must target the demonstrated layer: correcting application release ordering differs from fixing a server reconnection race. Regression evidence should cover normal playback, invalid size, disconnect during a pending callback and teardown while references remain. A source-only explanation cannot assert that a real device recovered or that a particular latency was measured.

Capstone C: socket readiness and storage contention#

A third synthetic incident: a service receives a partial HTTP request, while a separate database writer holds an immediate SQLite transaction. The client sees a delayed response. One hypothesis is that the request parser is waiting for the remainder of a valid line; another is that parsing completed and the application is waiting on a database lock. The Nginx NGX_AGAIN parser branch and SQLite pager busy-handler branch in the previous chapter are versioned source entry points, but they belong to different processes and state machines. A socket readiness notification means an operation may make progress without blocking; it does not establish that the HTTP parser completed, nor that a separate SQL transaction committed.

A falsifiable test separates the conditions: send a complete authored request to a local isolated test service while holding the database writer, then release the writer; separately send an incomplete request with no database contention. Observe the application and connection state, busy result and timeout. A safe private SQLite model was run in the previous chapter; no Nginx service or cross-process incident was run here. A proposed fix must preserve request parsing and transaction semantics, include before/after and duplicate-request controls, and avoid retrying a non-idempotent write without an application-level strategy.

Evidence, gates, and remediation#

For every capstone, submit a versioned source map, initial facts, competing hypotheses, object and fd ownership, normal and failure timeline, minimal change proposal, and regression matrix. Mark each row source-derived, synthetic, actually observed or unknown. A source-only pass requires a coherent branch explanation and counterexample; runtime engineering additionally requires a reproducible isolated environment, raw results and cleanup. An independently reviewed change requires reviewers to examine correctness, security, compatibility and test coverage. None of these gates is a certificate of expertise.

If an answer confuses a PID with a permanent request identity, redraw the object-lifetime table and trace the disconnect branch. If a test treats absence of a log as proof that an operation never happened, add a direct observation or explain the remaining uncertainty. If a proposed fix only suppresses an error, state which invariant remains untested and retry. Independent expertise develops through repeated source reading, controlled measurements, debugging, contributions and review over time, not through a chapter count.

Practice#

This model evaluates the supplied portal state; it contacts no bus or portal:

python3 - <<'PY'
response = 1
exported = False
register_document = response == 0
emit_response = exported
print('synthetic registration:', register_document)
print('synthetic response emission:', emit_response)
PY

Both outputs are false for the supplied state. Change response to zero and exported to true to represent a different branch; this model still cannot validate document registration, locking or a real D-Bus signal. If Python is unavailable, make the same state table by hand.

For a source-only alternative, use the supplied cancellation timeline and the two exact excerpts. Explain the skipped registration loop, conditional emission and separate request/fd references; then state how the success-before-cancel hypothesis differs. Submit the three capstone artifacts and a test matrix without opening a desktop, audio device, network service or user file.

Knowledge check#

1. What does a nonzero chooser response do before the success-only registration loop?
2. Why does an unexported request not imply every task reference is gone?
3. What observation separates incomplete HTTP input from SQLite writer contention?
4. Why does a visible window not prove an audio endpoint remains available?
5. What additional evidence separates source understanding from runtime engineering?

Answers#

1. The inspected worker jumps to its output path before that loop. Its later emission depends on the request's exported state.
2. Unexporting a D-Bus skeleton and dropping one GObject reference are separate from references held by tasks or callbacks. The fd also has its own close path.
3. A complete controlled request and database transaction result can distinguish parser progress from a lock conflict; a vague delayed response cannot.
4. Window presentation and audio routing use different processes, objects and failure paths. One can continue while the other fails.
5. A controlled build and isolated execution with exact versions, raw results, negative controls and cleanup are needed. Independent review adds a further evidence gate.

Checkpoint#

Submit the portal state and ownership table, three hypothesis pairs and a regression plan. Pass when the supplied portal case is answered, the other two cases have falsifiable controls, and synthetic evidence is not called a measured trace. Revisit the source branches and retry if unexport is treated as freeing every reference. This is not certification.

Sources#

• Portal 1.18.4 file chooser
• Portal 1.18.4 request lifecycle
• Nginx 1.26.2 HTTP parser
• SQLite 3.46.1 pager

The userspace project atlas#

This inventory lists every appended chapter in fixed reading order, grouped by the project families they teach. WORKED means a selected source path was explained, not that every feature was tested. SURVEYED means a bounded overview; INDEPENDENT FOLLOW-ON identifies work or source verification still needed. Source revisions are historical, independently chosen and not a compatible deployment matrix. Runtime verification is limited to the individual author records: models and private native exercises do not establish service, hardware, guest, Wine or Windows behavior. Authorship and selected source checks are recorded by the three authors; independent specialist review and runtime validation remain follow-on work.

97–100: libc, dynamic loading, shells and process control

WORKED selected examples; glibc 2.40 and musl 1.2.5 are separate baselines. Entry: glibc read. Prerequisite: syscall entry and ELF. Next: trace an owned short process through loader and read errors.

Lessons: 97 userspace architecture and source baselines, 98 glibc musl syscalls threads and allocation, 99 dynamic loader linker symbol versioning, 100 shells bash coreutils and process control.

101–104: toolchains, packages, D-Bus and debugging

WORKED selected paths; Meson 1.5.0, Ninja 1.12.1 and systemd v256 are independently recorded. Entry: sd-bus dispatch. Prerequisite: processes and shared libraries. Next: correlate an owned method call with its reply and fd lifetime.

Lessons: 101 build tools gcc clang meson ninja, 102 packages apt dpkg rpm pacman, 103 dbus daemon broker and sd bus, 104 userspace observability strace ltrace gdb.

105–112: systemd services, logs, devices and sessions

WORKED selected v256 source paths for all eight chapters, including execution, journal, udev and logind lifetimes; see the userspace-core author record for bounded checks. Entry: systemd v256 manager. Prerequisite: D-Bus and cgroup v2. Next: model one failed transaction and inspect job cleanup in source.

Lessons: 105 systemd pid1 manager and unit loading, 106 systemd jobs dependencies and transactions, 107 systemd service types readiness and restart, 108 systemd socket timer path activation, 109 systemd cgroups sandboxing and user managers, 110 journald journalctl and coredumps, 111 udev device events rules and lifetimes, 112 logind seats sessions and display login.

113–120: identity, network and storage services

WORKED selected source paths: Linux-PAM 1.6.1, polkit 124, NetworkManager 1.48.10, systemd v256, OpenSSH portable V_9_8_P1, UDisks 2.10.1, GVfs 1.54.2, cryptsetup 2.7.5, libfuse 3.16.2 and SSHFS 3.7.3 are separate historical pins. Entries: PAM dispatch, NetworkManager device, and LUKS2 keyslot. Prerequisite: credentials, filesystems and service lifetimes. INDEPENDENT FOLLOW-ON: validate one authorized scenario in a disposable environment.

Lessons: 113 pam nss authentication and accounts, 114 polkit authorization agents and dbus policy, 115 networkmanager networkd and dhcp, 116 dns resolvconf systemd resolved, 117 openssh server client and agent, 118 udisks gvfs mounting and removable storage, 119 cryptsetup luks dmcrypt and key management, 120 fuse libfuse sshfs and userspace filesystems.

121–128: graphics, Wayland, input and X11

WORKED selected paths; Mesa 24.2.0, Wayland 1.23.0 and Weston 14.0.0 are separate historical baselines in the desktop author record. Entry: Wayland core protocol 1.23.0. Prerequisite: DRM, fd passing and IPC. Next: trace one buffer from commit to release in an isolated compositor.

Lessons: 121 userspace graphics pipeline and libdrm, 122 mesa dri gallium and drivers, 123 vulkan opengl egl gbm, 124 wayland protocol objects and wire, 125 wayland client surfaces buffers events, 126 compositors weston wlroots mutter kwin, 127 input libinput xkb seat routing, 128 x11 xorg xwayland.

129–136: color, fonts, toolkits, portals and sessions

WORKED selected paths; GLib 2.82.2 and xdg-desktop-portal 1.18.4 are independently pinned. Entry: portal request. Prerequisite: event loops and Wayland objects. Next: compare a canceled portal request with a successful selected-file grant.

Lessons: 129 graphics synchronization color hdr, 130 text fontconfig freetype harfbuzz pango, 131 gtk gobject glib event loop, 132 qt qobject events and platform plugins, 133 desktop launchers xdg dbus activation, 134 portals desktop security and accessibility, 135 desktop login display managers and sessions, 136 desktop screenshot screenlock clipboard.

137–144: audio, Bluetooth and media

WORKED selected source paths: alsa-lib 1.2.12, PipeWire 1.2.0, WirePlumber 0.5.5, PulseAudio 17.0, BlueZ 5.78, GStreamer 1.24.8 and FFmpeg 7.0.2 are separate pins; the capture chapter also uses portal 1.18.4 and WebRTC commit 3710345e71ad3a7c1f4ed4efef152fd4fc24869a. Entries: PipeWire stream, BlueZ transport, and GStreamer pad. Prerequisite: PCM frames and graph scheduling. INDEPENDENT FOLLOW-ON: test a negotiated format and disconnect with an isolated endpoint.

Lessons: 137 linux audio userspace and alsa lib, 138 pipewire core objects protocol, 139 pipewire graph scheduling buffers realtime, 140 wireplumber session policy, 141 pipewire pulse jack and compatibility, 142 bluez bluetooth userspace audio, 143 gstreamer ffmpeg media pipelines, 144 desktop capture webrtc and media security.

145–148: integrated sessions, application sandboxes, containers and VMs

WORKED selected source paths; Flatpak 1.16.0, bubblewrap 0.10.0, Podman 5.3.0, runc 1.2.0, QEMU 9.1.0 and libvirt 10.6.0 are separate pins. Entry: runc init. Prerequisite: namespaces and process lifecycle. Next: model a failed setup handshake and enumerate every pipe owner.

Lessons: 145 desktop session end to end investigation, 146 flatpak bubblewrap and app sandboxes, 147 oci runc podman containers, 148 qemu libvirt and userspace virtualization.

149–152: browsers, printing, servers and capstones

WORKED selected branches; Chromium 129, CUPS 2.4.10, Nginx 1.26.2 and SQLite 3.46.1 are independent pins. Entry: CUPS IPP parser. Prerequisite: IPC, event loops and policy. Next: compare malformed protocol input with an authorized denial in a private harness.

Lessons: 149 browsers webengines and sandbox boundaries, 150 printing cups and ipp, 151 servers event loops nginx sqlite, 152 userspace incident capstones and atlas.

153–156: Wine architecture, PE loading, ntdll and wineserver

WORKED selected Wine 10.0 paths. Entry: Wine handle management. Prerequisite: ELF/PE and object lifetime. Next: trace a handle lookup failure and last-reference cleanup.

Lessons: 153 wine architecture prefixes and trust, 154 pe coff loader dlls and imports, 155 ntdll unix boundary and syscalls, 156 wineserver processes threads and handles.

157–160: Wine waits, memory, files, registry, services and sockets

WORKED selected Wine 10.0 paths. Entry: Wine virtual memory. Prerequisite: synchronization and file ownership. Next: compare an auto-reset event with a semaphore under a controlled test.

Lessons: 157 wine synchronization waits and futexes, 158 wine memory virtualalloc and seh, 159 wine files registry paths and security, 160 wine services winsock and ipc.

161–164: Wine ABI, windowing and Direct3D translators

WORKED selected Wine 10.0, DXVK 2.5.3 and vkd3d-proton 2.14 paths; no tested compatibility matrix. Entry: DXVK D3D11 device. Prerequisite: graphics synchronization and PE ABI. Next: trace a resource reference through a queued present.

Lessons: 161 wine wow64 cross architecture abi, 162 wine windowing wayland x11 input, 163 wine direct3d wined3d opengl, 164 dxvk vkd3d proton vulkan.

165–168: Wine audio, COM, Proton and conformance

WORKED selected Wine 10.0 and Proton 9.0-4 paths; Mono and Winetricks are SURVEYED only. Entry: Wine apartment. Prerequisite: audio buffers and IPC. Next: specify a same-thread COM mismatch test with balanced cleanup.

Lessons: 165 wine audio media and devices, 166 wine com ole dotnet and installers, 167 proton steam runtime and compatibility, 168 wine debugging conformance and capstone.

Extending this map

Start from a project release and its actual source entry, then record the revision, license, object ownership, normal and failure branches, configuration and tests. Follow the kernel boundary through the corresponding versioned Linux source, but do not infer a universal call chain from a name. Build a minimal owned reproducer in an appropriate isolated environment, compare a negative control, record raw results and ask for review. Update the family entry when new evidence changes its depth; leave unverified versions and missing hardware explicitly pending. This atlas covers course projects, not all Linux applications or every version of a listed project.