Journals, logging, and userspace coredumps

Before you start#

Review service management, userspace tracing, file durability, and credentials. The source baseline is systemd v256 commit 5c79cdec10a547a866764a66e1e14898112a00cd, independent of the Linux v6.12 kernel baseline and the host's installed tools. This chapter uses invented records and never reads the host journal or triggers a crash.

Learning goals#

Trace journal ingestion and append failure, distinguish client-supplied fields from trusted metadata, and explain timestamps, cursors, rate limits and storage limits. Describe the conditional userspace coredump path and the privacy and evidence limits of a memory image. Construct a synthetic log sequence without presenting it as a captured system event.

Source walkthrough#

systemd-journald is a userspace service that receives records from several configured sources. Native journal messages, syslog-compatible messages, service stdout/stderr streams and kernel messages reach it through different interfaces and carry different context. Their availability depends on configuration, boot state and permissions. A process that writes to a stream is not necessarily the process that originally opened or inherited that stream, so a reader should not assume every line's process metadata uniquely identifies the code that produced its bytes.

The following exact excerpt from systemd v256 src/journal/journald-server.c, server_dispatch_message, is LGPL-2.1-or-later under its SPDX notice. Surrounding initialization and later storage dispatch are omitted:

        if (n == 0)
                return;

        if (LOG_PRI(priority) > s->max_level_store)
                return;

n is the number of supplied field vectors; an empty message is discarded on this path. The priority comparison can also prevent storage when the message exceeds the configured maximum level. The Server object holds configuration and runtime state; the input vectors and ClientContext have ownership and lifetime rules across dispatch. A message accepted by a socket is therefore not automatically a durable journal entry. The same function also has storage-disabled and rate-limit branches, so lack of a record cannot by itself prove that the application never wrote anything.

The next exact excerpt is from the same pinned journald-server.c, within the journal write path, also LGPL-2.1-or-later. The preceding append call and later error handling are omitted:

        if (r >= 0) {
                server_schedule_sync(s, priority);
                return;
        }

Here r is the result of journal_file_append_entry. On nonnegative return, the server schedules synchronization according to its priority and configuration, then leaves this path. That is not a claim that the record has already survived power loss at the instant the function returns. The following source logs an append failure, determines whether a retry is appropriate and may rotate and vacuum before trying again; repeated failure can stop the attempt. Storage policy, filesystem behavior and device guarantees all constrain durability.

In src/journal/journald-server.c, server_write_to_journal, v256, LGPL-2.1-or-later under its SPDX notice (the first append and the second append are omitted):

        log_debug_errno(r, "Failed to write entry to %s (%zu items, %zu bytes): %m", f->path, n, iovec_total_size(iovec, n));

        if (!shall_try_append_again(f, r))
                return;
        if (vacuumed) {
                log_ratelimit_warning_errno(r, JOURNAL_LOG_RATELIMIT,
                                            "Suppressing rotation, as we already rotated immediately before write attempt. Giving up.");
                return;
        }

        server_rotate_journal(s, TAKE_PTR(f), uid);
        server_vacuum(s, /* verbose = */ false);

A failed first append goes to shall_try_append_again, which decides whether a retry is appropriate. If the server already rotated immediately before this write attempt, the function logs a rate-limited warning and gives up. Otherwise it rotates, vacuums and looks for a journal to retry on. The excerpt stops before that second append. The function returns void, so a failed retry gives the caller no status; the excerpt does not show whether the retry succeeds.

Origins, credentials and timestamps#

Native messages can carry fields supplied by the sender, including a message string and application identifiers. Journald can add trusted metadata derived from peer credentials and process context; conventionally trusted fields begin with an underscore and are not simply accepted as arbitrary sender assertions. Their interpretation still depends on the source and the point at which credentials were observed. A PID can be reused, a process can exit before metadata is collected, and a stream inherited by a child can complicate attribution. Record a boot identifier and relevant process lifetime before joining a PID to another source of evidence.

Realtime timestamps describe wall-clock time and can change when the clock is corrected. Monotonic timestamps describe an elapsed-time clock within an appropriate boot context and are not directly comparable across different boots without additional information. _BOOT_ID is useful for grouping records by boot, but neither the identifier nor a timestamp proves that an event's content is true. A clock adjustment can make two wall-clock timestamps appear out of order even when monotonic ordering within one boot differs. The v256 journald-native.c and server code show separate ingestion and dispatch stages; exact credential and field availability depends on the transport and configuration.

A Unix datagram sender may supply application fields, while the receiver can obtain operating-system credentials through supported socket mechanisms. That credential is evidence of a peer at the transport boundary, not a cryptographic signature over the semantic truth of every field. A process with permission to log can still log an incorrect message. Forwarded syslog data, kernel messages and service streams have their own trust boundaries. Never treat a user-controlled MESSAGE= string as if it were the same kind of evidence as a manager-derived credential field.

Files, indexes and queries#

Journal files contain structured objects and indexes rather than just newline-delimited text; the v256 journal file implementation contains the append machinery. Conceptually, field and data objects represent field names and values, entry objects associate data with a particular record, and index structures support lookups and ordering. Headers, offsets, object sizes and integrity checks matter when a reader processes an on-disk file. Compression may be used depending on configuration and data. A malformed, truncated or corrupted file must be handled as an error; a matching string in raw bytes is not equivalent to a validated indexed entry. The exact on-disk layout and supported features are version-specific.

Rotation replaces the active journal file under relevant conditions, while vacuuming removes eligible older files according to retention and size policy. A cursor is an opaque position token used by journal APIs and journalctl to resume around an entry; applications should not parse it as a timestamp or assume it remains valid after rotation and deletion. Following a journal means waiting for new records subject to access policy and process lifetime. Matching on a field selects records with that field, not a globally complete causal history. A user can have permission to see some journals and not others. This lesson does not read any host journal, service logs or other users' metadata.

Forward Secure Sealing can provide integrity-related evidence for configured journals under its documented assumptions and key-handling procedure. It is not enabled universally, is not default encryption, and does not independently establish that an authorized sender's message was truthful. Protect the verification key and understand the coverage of sealing before claiming tamper evidence. Transport authentication, journal file integrity, log completeness and confidentiality are different properties. A journal can be locally readable according to permissions even if its entries are sealed.

A synthetic record and missing evidence#

Consider two invented entries in the same boot: entry A has realtime 100.000 seconds and monotonic 20.000 seconds; entry B has realtime 99.900 seconds and monotonic 20.200 seconds. Under the stated assumption that the realtime clock stepped backward between the records, monotonic order places A before B while wall-clock values decrease. The values are teaching data, not captured logs. If the boot identifiers differed, these monotonic values could not be ordered as one continuous clock without additional evidence. If a process ID in A is 400 and a later process also has PID 400, PID equality alone does not prove the same process lifetime.

Now suppose the application's next 50 records exceed a configured rate limit. The v256 source's journal_ratelimit_test result can suppress dispatch; when the result indicates prior suppression, a driver message can report dropped messages under the source's rules. A query that finds no individual records in that interval cannot establish that the application produced none. The suppression message itself is subject to storage and observation conditions. A truncated journal, permission denial, filter mismatch, rotation, storage-disabled mode or crash before synchronization provides further competing explanations.

For a concrete investigation, define a question such as “did this particular invocation emit a ready message before its worker exit?” Identify the boot, unit, process lifetime, applicable stream and permissions. Then distinguish a service's READY=1 manager notification from a human-readable journal message that says “ready”; they are different channels unless the application and manager evidence connect them. Record filters and missing intervals. A journal entry is useful evidence, not a complete distributed trace or a substitute for an application's own durable transaction record.

A query is a bounded observation#

A journal query has an input set: accessible journal files, a time range, match expressions and the reader's permissions. If a cursor refers to an entry that was vacuumed, the reader cannot recover the removed data simply by parsing the cursor. A follow operation can be interrupted by a reader exit, permission change or journal rotation. A consumer that resumes from a stored cursor needs a policy for gaps and duplicates, not an assumption that each entry will be delivered exactly once across every failure. A cursor can help resume an observation, but its practical use depends on which files still exist and on the API's seek behavior.

Suppose an investigator filters for a unit and sees one message after a restart. The filter can exclude relevant messages attributed to a helper or to the manager, and rate limiting can suppress other entries. The helper could have written through a stream established by a different process, so a PID field should be interpreted in the context of that transport. The investigator can compare a manager job result, process lifetime, application record and independent data when authorized, but should state which evidence is missing. No one field turns the query into a complete causal trace.

Journal storage may be volatile or persistent according to configuration and availability. Rotation and vacuum policies apply to eligible files; an append failure, filesystem error or service shutdown can interrupt the expected path. Encryption at rest, if required, is a separate storage and key-management decision. Even when a configured integrity mechanism detects certain modifications, it does not fill a missing interval or authenticate the intent of the program that logged the message. Those distinctions help decide whether a journal is suitable for a given incident question.

Crash collection and privacy#

Linux can be configured through core_pattern to invoke a userspace handler for certain crashes. A distribution may route eligible crashes through systemd-coredump and socket activation; this is configuration-dependent and is not guaranteed on every Linux host. The handler can collect metadata and, subject to configured size, storage and resource limits, retain or process a core image. A coredump can contain stack, heap, mapped files, tokens, private data and other secrets. Access to that image requires careful authority and retention decisions. A crash report without the full memory image still may reveal process metadata.

The pinned src/coredump/coredump.c implements selected processing paths for the userspace collector. A failure to obtain space, exceed a configured limit, access metadata or write storage can change the resulting artifact. A successful handler invocation is not proof that an entire untruncated core was stored. Debugging also requires the matching executable, libraries, architecture and debug symbols; a plausible backtrace from mismatched builds can be misleading. The coredump's crashing thread and other threads can reflect an intermediate state rather than the root cause.

In src/coredump/coredump.c, save_external_coredump, v256, LGPL-2.1-or-later under its SPDX notice (the size-budget computation and compression branch omitted):

        r = copy_bytes(input_fd, fd, max_size, 0);
        if (r < 0)
                return log_error_errno(r, "Cannot store coredump of %s (%s): %m",
                                context->meta[META_ARGV_PID], context->meta[META_COMM]);
        truncated = r == 1;

The contract of copy_bytes is that it returns zero when input ends before the byte limit and a positive value otherwise. This excerpt records only the value 1 as truncated, so the flag depends on that exact return value. A truncated image can still be compressed or stored, and the return checks above it handle storage errors. The excerpt does not show the image contents or whether a matching debug binary exists.

Do not cause a host crash merely to create evidence for this lesson. Do not modify core_pattern, read other people's coredumps, collect secrets or change journald settings. A separately prepared disposable guest could be used for a future crash exercise only with explicit setup and cleanup, but that has not been run here. The chapter 08 guest itself is not boot verified. The supplied synthetic sequence is the intended source-only route.

Practice#

This private Python model creates two invented records, sorts them by monotonic value and demonstrates a missing interval. It does not invoke journalctl, contact journald, read host logs or create a coredump:

python3 - <<'PY'
records = [
    {'boot': 'demo', 'realtime': 100.0, 'mono': 20.0, 'message': 'A'},
    {'boot': 'demo', 'realtime': 99.9, 'mono': 20.2, 'message': 'B'},
]
for entry in sorted(records, key=lambda item: item['mono']):
    print('synthetic', entry['message'], entry['realtime'], entry['mono'])
print('synthetic omitted records: 50')
PY

Expected output orders A before B by the chosen synthetic monotonic values and labels 50 records as omitted. This demonstrates sorting supplied data only; it does not implement journald's rate limiter or establish that a real clock stepped. If Python is unavailable, use the same two values and explain the ordering by hand. No persistent files or system settings are changed.

Deliver a provenance table identifying sender-controlled fields, manager-derived credentials, boot and clock assumptions, and one possible reason a record is absent. As a source-only alternative, use the A/B and 50-record scenario above; label the values synthetic, explain the priority and storage early returns, and state what would be needed to support a real durability or coredump claim.

Knowledge check#

1. Why can a message accepted from a client fail to appear as a stored journal entry?
2. What does the successful append branch establish, and why does it not establish immediate power-loss durability?
3. Why can realtime and monotonic timestamps order the synthetic A/B records differently?
4. What do trusted metadata fields establish, and why can PID reuse or inherited streams complicate attribution?
5. Why are sealing, encryption and truthfulness separate claims?

Answers#

1. The server can discard empty input, filter by priority, disable storage, rate limit, or encounter an append or storage failure. Query permissions and filters can also hide a stored entry.
2. The append returned nonnegative and the server scheduled synchronization. It does not prove that every filesystem and device persistence step completed at that moment.
3. The example assumes a backward realtime adjustment while the within-boot monotonic values increase. Different boot identifiers or missing clock context would require additional evidence.
4. The fields can reflect credentials and context obtained by the journal service under its rules; they do not guarantee the truth of sender text. A PID can be reused and a child can inherit a stream whose metadata reflects an earlier connection.
5. Sealing can provide integrity evidence under its configured assumptions; it does not encrypt records by default or prove that a permitted sender's content was factually correct.

Checkpoint#

Submit the synthetic two-clock ordering, provenance table and missing-record counterexample, with the source-only label or private model output. Ready means explaining the priority and append branches, separating synchronization from durability, and identifying coredump privacy and matching-symbol requirements. Retry with only the two timestamps before adding rate limits and crash collection.

Sources#

• systemd v256 journal server
• systemd v256 native journal ingestion
• systemd v256 coredump implementation
• systemd v256 journal service manual
• systemd v256 journalctl manual