Socket, timer, and path activation

Trace activation triggers, descriptor ownership and service start.

14 min readChapter 108 of 168Intermediate → advanced
In this chapter

Before you start#

Review unit loading, jobs and transactions, service readiness, and file descriptors. The source baseline is systemd v256 commit 5c79cdec10a547a866764a66e1e14898112a00cd, independently pinned from the historical Linux kernel baseline. No host timer, socket unit or path unit is activated in this chapter.

Learning goals#

Explain how socket, timer and path units trigger jobs without assuming one event equals one successful service invocation. Track listener and accepted-connection descriptor ownership, compare calendar and monotonic timer semantics, and explain why a path watch must check state as well as receive events. Analyze failure and cleanup in the selected source.

Source walkthrough#

Triggers and their ownership boundaries
Socket
Listener and accepted connection descriptors have separate owners.
Timer
Clock base, deadline and accuracy settings determine scheduling.
Path
A condition is checked, watched and checked again around a race window.
Job
A requested service job can fail, merge or remain pending.
Conceptual model. A conceptual comparison, not a captured activation; a trigger is not proof of completed service work.

Activation separates an event source from the service that handles work. A socket unit can hold a listening socket and request a service when activity arrives. A timer unit can request a service when a configured deadline is reached. A path unit can watch for a filesystem condition and request a service when its rules indicate work. In each case a trigger results in manager-side state and potentially a job; it does not prove that the service started, completed successfully or processed exactly one application event.

The following exact excerpt from systemd v256 src/core/socket.c, socket_enter_running, is LGPL-2.1-or-later under the file's SPDX notice. The explanatory comment, error object and following branches are omitted:

c
        _cleanup_close_ int cfd = cfd_in;
        _cleanup_(sd_bus_error_free) sd_bus_error error = SD_BUS_ERROR_NULL;
        int r;

The function receives a connection descriptor and places it in a local variable with automatic close cleanup unless ownership is transferred. The separate bus-error object also has cleanup. cfd can be negative for an Accept=no path, so it must not be interpreted as a valid accepted connection in every invocation. The surrounding code suppresses activation when a stop is pending, handles rate limits and distinguishes cases before either assigning or refusing the descriptor. This is ownership in the manager's source path, not a captured socket activation.

The next exact excerpt from v256 src/core/path.c, path_enter_waiting, is also LGPL-2.1-or-later. Earlier checks and the later recheck are omitted:

c
        r = path_watch(p);
        if (r < 0) {
                log_unit_warning_errno(UNIT(p), r, "Failed to enter waiting state: %m");
                path_enter_dead(p, PATH_FAILURE_RESOURCES);
                return;
        }

The path object attempts to install watches. A negative return is logged and moves the unit into a resource-failure state; it does not pretend that a watch exists. The following source explicitly rechecks the condition after creating watches because the path may have changed between the first check and watch installation. This check-watch-recheck pattern closes a particular observation gap; it does not turn inotify into a lossless transaction log of every filesystem operation.

Socket ownership and two connections#

With a suitable stream socket unit and Accept=no, the manager retains the listening socket and activates the associated service, passing listening descriptors according to the socket activation interface. The service accepts connections itself. A socket unit can remain active while the service exits or restarts, allowing the listener and its queued connections to remain available subject to backlog, timeouts and configuration. This is not a promise that every request survives a failure: accepted connections already owned by a failed service can close, the backlog can fill, and clients still need error handling.

With Accept=yes, the manager accepts a connection and starts a service instance for that connection under the documented template and configuration rules. The service receives the accepted connection rather than taking responsibility for accepting all clients from the same listener. The manager's listener, the accepted descriptor in the service and any duplicated descriptors have separate references. Closing one process's descriptor does not automatically close a copy held elsewhere. Descriptor names, counts and lifetime depend on the socket unit, service and selected API; do not assume every unit passes exactly one descriptor.

The sd_listen_fds family uses environment information including LISTEN_PID and LISTEN_FDS; supported interfaces can also provide LISTEN_FDNAMES. The receiving process must validate that the descriptors are the kind and number it expects. The API's close-on-exec behavior and ownership rules matter if the process starts children. A descriptor can remain open across a fork and delay EOF or retain a listener unexpectedly; after it is closed the integer can be reused for a different object. A file descriptor store is a separate optional facility with configured limits and lifecycle rules, not an unlimited permanent persistence service.

Consider a synthetic Accept=no listener L. At T0 the manager owns L, service S is inactive, and client C1 connects. At T1 the manager requests S; the connection waits subject to the listening backlog. At T2 S receives a reference to L and accepts C1 as descriptor A. At T3 S fails while handling A: A can close with S, while the manager's reference to L remains if the socket unit remains active. Client C2 connects at T4 and may wait for a later service start, but can fail if the queue is full or policy stops activation. This timeline is conceptual and does not establish the behavior of a particular host socket unit.

Accept=no listener and an accepted descriptor
Accept=no listener and an accepted descriptor1. Client to Listener (manager): connect while service is inactive. 2. Listener (manager) to Service: start service with listener. 3. Service to Listener (manager): accept() in service. 4. Service to Service: service fails while holding accepted fd. 5. Listener (manager) to Client (return): listener remains if unit stays active.ClientListener (manager)Serviceconnect while service isinactivestart service withlisteneraccept() in serviceservice failswhile holdingaccepted fdlistener remains if unitstays active
  1. ClientListener (manager)

    connect while service is inactive

  2. Listener (manager)Service

    start service with listener

  3. ServiceListener (manager)

    accept() in service

  4. ServiceService

    service fails while holding accepted fd

  5. Listener (manager)Client

    Return. listener remains if unit stays active

Time runs downward. Dashed arrows show returns.

Read the text version
  1. Client → Listener (manager): connect while service is inactive
  2. Listener (manager) → Service: start service with listener
  3. Service → Listener (manager): accept() in service
  4. Service → Service: service fails while holding accepted fd
  5. Listener (manager) → Client (return): listener remains if unit stays active
Conceptual model. A synthetic conceptual timeline for the v256 socket path, not a captured socket activation; the listener and the accepted connection have separate references.

For an Accept=yes contrast, the manager accepts C1 as A1 and C2 as A2 and schedules separate instances under its rules. A failure of the C1 instance does not make A1 become C2's descriptor. Cleanup must close A1 in every unsuccessful ownership path. The quoted _cleanup_close_ variable supports that kind of failure handling until ownership moves elsewhere. The source also checks a pending stop and trigger rate limits; a manager can refuse a connection rather than queue an unlimited number of service starts.

In src/core/socket.c, socket_dispatch_io, v256, LGPL-2.1-or-later under its SPDX notice (earlier poll-error handling omitted):

c
        if (p->socket->accept &&
            p->type == SOCKET_SOCKET &&
            socket_address_can_accept(&p->address)) {

                cfd = socket_accept_in_cgroup(p->socket, p, fd);
                if (cfd == -EAGAIN) /* Spurious accept() */
                        return 0;
                if (cfd < 0)
                        goto fail;

                socket_apply_socket_options(p->socket, p, cfd);
        }

        socket_enter_running(p->socket, cfd);

The accept path is conditional: it runs only when the socket unit requests accepting, the port is a socket and its address allows accepting. A spurious -EAGAIN returns without starting anything. The connection descriptor then goes to socket_enter_running on this path, and in the Accept=no case cfd remains negative. The excerpt does not show whether a service instance starts or succeeds.

Timers measure different notions of time#

A calendar timer uses civil-time expressions interpreted with a timezone and realtime clock rules. A monotonic timer uses elapsed-time bases such as time relative to boot, activation or another configured event. Suspend behavior depends on the selected clock and relevant timer options; do not assume that every monotonic timer counts suspended time in the same way. Clock adjustments, timezone changes, daylight-saving transitions and a machine being powered off can affect calendar scheduling. The v256 src/core/timer.c calculates calendar and monotonic candidates under separate branches.

AccuracySec= permits scheduling coalescing within its documented bounds rather than guaranteeing the service begins at the exact specified instant. RandomizedDelaySec= adds a randomized scheduling component under its rules; it should not be described as the same mechanism as accuracy coalescing. Persistent= applies to supported calendar-timer behavior and can cause a missed calendar event to be caught up when the timer becomes active, subject to recorded state and conditions. It is not a guarantee that every missed occurrence produces one separate job. Check the v256 timer manual source for option applicability and defaults.

For a concrete model, assume a calendar expression schedules 09:00 each day in UTC, no randomized delay, no extra accuracy delay, and a machine continuously running with a correct clock. A deadline on Tuesday is Tuesday 09:00 UTC; a change in the user's local timezone does not change the stated UTC instant. If the clock or configured timezone differs, the answer can differ. If the machine was off at the deadline, persistent catch-up may request work after activation when its conditions are met, but that request is not evidence of one execution per missed day. This is arithmetic under explicit assumptions, not a captured timer event.

The v256 timer_enter_waiting obtains the unit to trigger and follows a failure path if it has vanished. It records a fresh time snapshot and iterates enabled timer values, distinguishing calendar from monotonic cases. The manager later handles the selected event and requests a job. A timer firing while its service is already active need not create a new independent service instance for every elapsed interval. If the triggered service fails, inspect its job and result; the timer's scheduling state alone does not prove useful work completed.

In src/core/timer.c, timer_enter_waiting, v256, LGPL-2.1-or-later under its SPDX notice (the monotonic branches and earlier lookup omitted):

c
                        r = calendar_spec_next_usec(v->calendar_spec, b, &v->next_elapse);
                        if (r < 0)
                                continue;

                        /* To make the delay due to RandomizedDelaySec= work even at boot, if the scheduled
                         * time has already passed, set the time when systemd first started as the scheduled
                         * time. Note that we base this on the monotonic timestamp of the boot, not the
                         * realtime one, since the wallclock might have been off during boot. */
                        rebased = map_clock_usec(UNIT(t)->manager->timestamps[MANAGER_TIMESTAMP_USERSPACE].monotonic,
                                                 CLOCK_MONOTONIC, CLOCK_REALTIME);
                        if (v->next_elapse < rebased)
                                v->next_elapse = rebased;

For a calendar value, the code asks calendar_spec_next_usec for the next realtime instant. If that instant is earlier than the manager start time, it is moved to that start time. The comment ties this to RandomizedDelaySec= working after boot. This rebasing applies to one calendar value on this path; it does not show Persistent= catch-up, and it does not establish one job per missed occurrence.

Calendar and monotonic bases
Calendar
calendar_spec_next_usec yields a realtime instant; a value earlier than manager start is moved to that start time.
Monotonic
A base such as boot, startup or the trigger unit is shifted to the timer clock and offset by the configured value.
Missed events
Persistent= catch-up uses separate state and is not shown by this branch.
Job count
A computed next time does not by itself establish one job per elapsed occurrence.
Source map. A v256 source map of two timer branches, not a captured timer event; clocks, time zones and unit settings select the actual instant.

Paths, coalescing and races#

A path unit uses filesystem event monitoring and state checks for its configured condition. The specific directives have different meanings: existence, nonempty files, directory state and change notifications are not interchangeable. An inotify event can be coalesced with another and a queue can overflow; consumers that need complete state should be designed to rescan or otherwise reconcile authoritative state. A path unit is not a durable journal of every write and does not guarantee a distinct job for each event.

Suppose a watched file is written twice rapidly while the triggered service is already running. The event mechanism or manager may combine observations, and the service may receive a single opportunity to examine the resulting file state. The service should not infer “one invocation means exactly one write.” If it needs to process a queue, the queue needs its own durable identifiers and acknowledgment rules. If an event is lost or the watch cannot be created, a periodic reconciliation or explicit recovery plan can be necessary; this chapter does not claim that any particular unit has such a plan.

The selected path_enter_waiting code checks whether the triggered unit is already active, evaluates the path condition, creates watches and rechecks. That second check addresses the race in which the file changes after the first check but before the watch is installed. A watch creation failure enters PATH_FAILURE_RESOURCES; a running service or pending job affects the next state. The surrounding code also defers some trigger notifications to avoid unsafe reentry while a job is being installed or finished. This is event-loop and state-machine reasoning, not an assertion that no filesystem race remains.

In src/core/path.c, path_enter_waiting, v256, LGPL-2.1-or-later under its SPDX notice (the condition checks, watch setup and recheck omitted):

c
        /* If the triggered unit is already running, so are we */
        trigger = UNIT_TRIGGER(UNIT(p));
        if (trigger && !UNIT_IS_INACTIVE_OR_DEACTIVATING(unit_active_state(trigger))) {
                path_set_state(p, PATH_RUNNING);
                path_unwatch(p);
                return;
        }

If the triggered unit is not inactive or deactivating, this branch records PATH_RUNNING, removes the watches and returns before the condition is checked. It therefore does not start a second run while the trigger unit is active. The excerpt does not show how file events that arrive during that time are handled elsewhere.

Path check, watch and recheck
  1. Running trigger

    An active trigger records PATH_RUNNING, removes watches and returns.

  2. First check

    A satisfied path condition enters the running path.

  3. Install watches

    A failure enters PATH_FAILURE_RESOURCES and logs a warning.

  4. Recheck

    The condition is checked again because it may change before watches exist.

  5. Waiting

    Otherwise the path unit waits for later notifications.

Source map. A v256 source map of path_enter_waiting branches, not a captured inotify trace; the recheck narrows the race but does not make events lossless.

Activation descriptors are an interface contract between the manager and a particular service invocation. Before using them, a service should confirm the expected descriptor type and address and keep track of which component closes each reference. A helper process inheriting an unintended listener can keep that listener alive after the main process exits, changing the behavior a client observes. Close-on-exec flags and deliberate descriptor passing affect this lifetime; they do not replace the service's own shutdown protocol.

A combined failure table#

For a socket event, ask whether the manager still owns the listener, whether a connection was already accepted, who closes the accepted descriptor, whether a start job was enqueued and what the service actually returned. For a timer, record the clock base, timezone, selected deadline, accuracy/randomization settings, missed-event policy and service result. For a path, record the watched condition, watch-install result, initial and rechecked state, and any evidence of queue overflow. These are different inputs to the same broad job mechanism; they should not be replaced by a single count called “events handled.”

A trigger can race with a stop job. In the quoted socket path, a pending stop suppresses new connection handling and may flush or refuse work according to the branch. A timer's target can vanish. A path watch can fail for resource reasons. Even after a valid trigger, job installation can fail due to conflicts or unit state. The service can start and then fail before processing data. Each stage needs its own error and ownership explanation. A successful unit-file verification does not execute these paths or establish that a real network client, clock event or file write was handled.

Practice#

This bounded own-process socket example models the descriptor ownership idea with a private Unix stream socket. It does not use systemd, a host service, a TCP port or a unit file. Python's context managers close both ends and remove the private path:

sh
python3 - <<'PY'
import os, socket, tempfile
with tempfile.TemporaryDirectory() as directory:
    path = os.path.join(directory, 'listener.sock')
    with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as listener:
        listener.bind(path)
        listener.listen(2)
        with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as client:
            client.connect(path)
            connection, _ = listener.accept()
            with connection:
                client.sendall(b'A')
                print('accepted data:', connection.recv(1))
        print('listener remains open:', listener.fileno() >= 0)
PY

Expected output shows accepted data: b'A' and listener remains open: True. The connection closes before the final listener check, illustrating separate descriptors within one process; it does not demonstrate systemd descriptor passing or service restart. If Unix sockets are unavailable or denied, record the error and use the supplied L/C1/C2 timeline. No host network, clock, timer or path-unit configuration is changed.

For a source-only deliverable, draw the manager listener L, accepted connection A and two clients in the worked timeline. Mark which reference survives the service failure under the stated assumptions. Add the UTC deadline and the two-write coalescing example, clearly labeling all as models. Explain the path_watch failure and the reason for rechecking after watch installation.

Knowledge check#

  1. How do Accept=no and Accept=yes differ in who accepts a stream connection and which descriptor a service receives?
  2. Why can the listening socket remain available after a service fails while an accepted connection does not?
  3. What is the difference between the assumed UTC calendar deadline and a monotonic elapsed-time trigger?
  4. Why does the path source recheck after installing a watch, and why is that not a guarantee of one job per write?
  5. What does the quoted path_watch error branch do?

Answers#

  1. With the stated stream-socket setup, Accept=no passes a listener to a service that accepts clients. Accept=yes has the manager accept connections and activate service instances with accepted descriptors, subject to the unit and implementation rules.
  2. The manager can retain its listener reference while the service's accepted descriptor is closed as the service exits. Backlog, socket-unit state and client timeouts still limit what survives.
  3. The calendar example names a civil-time instant under an explicit UTC timezone and clock assumption. A monotonic trigger is based on elapsed time from its configured origin, with suspend and clock behavior determined by the selected clock and settings.
  4. A change can occur between the first state check and watch installation. The recheck detects an applicable condition after that window, but event coalescing, overflow, active services and job semantics still prevent a one-write-to-one-job guarantee.
  5. It logs the error, moves the path unit to a resource-failure state and returns. It does not claim a working watch was established.

Checkpoint#

Submit the listener/accepted-FD ownership timeline, the UTC deadline calculation and the two-write path case, plus either the private socket output or the supplied source-only analysis. Ready means identifying who retains each descriptor, separating a trigger from job completion, and explaining the watch recheck and failure branch. Retry with one client and no restart before adding the second client.

Sources#

A little further beneath the surface.

Check your understanding with the checkpoint. Marking a chapter as read saves a bookmark, not a proficiency assessment.