Service types, readiness, and restart policy

Before you start#

Review unit loading, job transactions, process exit and wait, and Unix sockets. This chapter reads systemd v256 at commit 5c79cdec10a547a866764a66e1e14898112a00cd; it does not assume that the host runs this release or that an earlier QEMU lab booted.

Learning goals#

Compare service start types by the event they use as readiness evidence. Follow the v256 service state machine through process creation, notification, failure and cleanup. Explain how a restart policy and start limit interact, and design a bounded own-process notification model without modifying a manager.

Source walkthrough#

A service unit describes how a manager starts and supervises a process or group of processes. Its Service object extends common unit state with commands, tracked process identities, timers, result codes and service-specific states. The manager's job and the service state machine are related but distinct: a start job can wait while a service is activating, then complete or fail as the service transitions. The manager's event loop processes process exits, timeouts and notification messages asynchronously; the service program can run concurrently and can exit before a later observation is made.

The following exact excerpt from systemd v256 src/core/service.c, service_enter_start, carries the file's LGPL-2.1-or-later SPDX notice. The surrounding branch and subsequent checks are omitted:

        if (s->type == SERVICE_FORKING) {
                s->control_command_id = SERVICE_EXEC_START;
                c = s->control_command = s->exec_command[SERVICE_EXEC_START];

                s->main_command = NULL;

The code selects a control command for the forking service type and clears main_command on this branch. The manager must later identify and supervise the intended main process under the configured rules; the original control process and the eventual main process are not automatically the same identity. Other service types take a different branch in the surrounding function. If a required command is absent, later source reports an error and follows failure handling. Process identity and PidRef lifetime matter because a numeric PID can be reused after exit; the temporary PidRef declared at function entry has an explicit cleanup annotation in this version.

The next exact excerpt from the same pinned service.c, service_enter_running, is also LGPL-2.1-or-later. Earlier result and process-watch handling are omitted:

        if (s->result != SERVICE_SUCCESS)
                service_enter_signal(s, SERVICE_STOP_SIGTERM, f);
        else if (service_good(s)) {

A non-success result enters the service's signal-based stop path rather than marking it successfully running. On the success branch, service_good determines whether the subsequent running transition is appropriate. The following code handles queued notification states and arms a timer, including a failure path if timer setup fails. The state machine cannot safely be summarized as “fork once, then always running.” The exact callback, timer, result and cleanup paths depend on type, state and configuration.

Readiness types are different contracts#

Type=simple treats service startup as progressing without waiting for the program to prove that it has executed successfully or is ready to serve requests. Type=exec waits for a successful execution transition and can report failures such as a nonexistent executable earlier in the start job. Successful execve means that the new program image was installed; it does not mean the application has opened its database, bound its own network listener or completed initialization. Both types need an application-level check if a client requires stronger readiness.

Type=notify waits for an accepted readiness notification such as READY=1, subject to the manager's configuration and notification authorization. The application chooses when to send it, so correctness depends on that program's actual readiness criteria. Type=notify-reload adds a notification-based reload protocol: the service reports RELOADING=1 with MONOTONIC_USEC=, then reports READY=1 when reload completes. These messages describe the application's assertion, not an independent health test (v256 sd_notify(3)). A message is not automatically trusted merely because a process can send a datagram: NotifyAccess= controls which service processes may provide notifications under the manager's credential checks. An auxiliary process exiting before its message can be attributed is a race the service designer must consider.

In the v256 service state machine src/core/service.c (LGPL-2.1-or-later), service_notify_message acts on a READY=1 tag only after service_notify_message_authorized accepts the sender. The branch below is the readiness part; the preceding STOPPING= branch is omitted. Lines 4409-4420 are quoted exactly:

        } else if (strv_contains(tags, "READY=1")) {

                s->notify_state = NOTIFY_READY;

                /* Type=notify services inform us about completed initialization with READY=1 */
                if (IN_SET(s->type, SERVICE_NOTIFY, SERVICE_NOTIFY_RELOAD) &&
                    s->state == SERVICE_START)
                        service_enter_start_post(s);

                /* Sending READY=1 while we are reloading informs us that the reloading is complete. */
                if (s->state == SERVICE_RELOAD_NOTIFY)
                        service_enter_running(s, SERVICE_SUCCESS);

For Type=notify and Type=notify-reload, a READY=1 received while the service is still starting moves it to the post-start path. A READY=1 received while a reload notification is pending completes the reload. The branch does not test the application's own health. For other types the same tag does not by itself change the start state in this branch, so Type=exec readiness still means only the execution event.

Type=dbus uses acquisition of the configured bus name as a readiness event. Name ownership is not the same as successful operation of every method; a service can own the name and still reject calls or fail a backend. Type=forking follows a traditional daemon pattern where a control process forks and the manager needs to identify the main process, using a configured PID file or other supported mechanisms when appropriate. GuessMainPID= is a heuristic in applicable cases, not a guarantee of correct identity for every process tree. A stale PID file or early parent exit can create misleading evidence.

Type=oneshot waits for its configured start command or commands to finish. With appropriate RemainAfterExit= configuration, the unit can remain active after the process exits even though no main process is running. Without it, the active state can be transient. Type=idle changes dispatch timing relative to queued jobs; it is not a general-purpose application readiness protocol. These descriptions are simplified around the v256 systemd.service manual; exact directives, defaults and conditions should be checked there for a specific unit.

A concrete start timeline#

Imagine two synthetic services with the same executable. Service A is Type=exec; service B is Type=notify. At T0 the manager installs a start job for each. At T1 the executable starts successfully but has not yet connected to its database. For A, the successful exec event can satisfy the relevant start criterion even though a client request would fail. For B, the manager continues waiting for an accepted READY=1. At T2 B finishes initialization and sends that message from an authorized process. The manager can then mark B ready under the notification protocol. This is a teaching timeline, not a measurement of a host manager.

Now change the case: B sends READY=1 before its database connection is usable. The manager cannot infer application truth from a message alone; it records the readiness assertion under the configured protocol. If B never sends an acceptable notification, its start timeout can expire and move it through failure and cleanup. If the sender exits immediately, attribution and delivery can race with its exit. An application that delegates initialization to a helper must design the notification sender and process lifetime according to NotifyAccess= and the manager's documented rules. An untrusted process must not be able to assert readiness for another service.

A successful manager start job is useful evidence of the configured readiness criterion, not a universal health check. A Type=oneshot command returning zero might have completed its immediate task but left a later external dependency unavailable. A Type=dbus service can acquire a name before a client operation finishes. A process can exit immediately after readiness, moving the service to a failure or inactive state. A client should observe the job result, current unit state and any application-specific response it requires, rather than collapsing these into one boolean.

Spawn, process ownership and cleanup#

During startup, the manager selects commands and establishes execution context: credentials, file descriptors, environment, sandbox options and resource constraints are configured through relevant unit settings. The spawned process can fail before or during execution, and the manager must distinguish a failure to spawn from a program that executed and later returned an unsuccessful status. For Type=exec, the execution-status communication path is part of how the manager learns whether execution reached the intended stage. Exact descriptors and helper processes depend on v256 configuration and execution path; a diagram of one path is not a universal syscall sequence.

A PidRef represents a process identity under systemd's implementation, helping the manager avoid relying only on a recycled numeric PID where supported. It has its own ownership and cleanup rules. A control process, main process, notification sender and other processes in the unit's cgroup can be different tasks. A stale stored PID or a helper that outlives the expected parent can complicate supervision. The manager watches process exits and changes state through the event loop. A callback handling an exit must check current service state because a stop, timeout or restart may already be pending.

Stopping a unit can involve signals and cgroup cleanup according to KillMode=, timeout and other settings (v256 systemd.kill(5)). A signal being sent does not establish that every descendant has terminated. The service may ignore a graceful signal, exit with a status classified as success or failure, or leave work to be cleaned up later. A manager can escalate after configured timeouts. Do not extrapolate a single process tree or a single kill mode to every unit. An application that starts asynchronous helpers still needs a clear protocol for their shutdown and resource ownership.

service_enter_running first preserves a result if one has not already been recorded, unwatches the relevant control PID, and chooses a path based on result and service health. In its normal running branch it handles queued reload or stop notifications before setting running state; a failure to arm the running timer re-enters failure handling. That ordering illustrates why an arriving notification and a process-exit event must be interpreted with the current state. The manager's event loop and object lifetime protect its own state transitions, but application data shared by its service threads needs its own synchronization.

Restart, limits and backoff#

Restart= selects which categories of termination should lead to another start attempt under the unit's configuration. SuccessExitStatus=, RestartPreventExitStatus= and RestartForceExitStatus= affect classification or restart decisions in documented combinations; they are not a promise that every failure will be retried indefinitely. A service can be deliberately stopped, fail a start condition, hit a rate limit or be in a state where a particular restart rule does not apply. Inspect the result and job rather than treating the existence of Restart= as proof of recovery.

In the same v256 file, service_shall_restart decides from explicit stops and exit-status lists before it reads the Restart= setting. The file is LGPL-2.1-or-later. Lines 1889-1916 are quoted exactly:

static bool service_shall_restart(Service *s, const char **reason) {
        assert(s);
        assert(reason);

        /* Don't restart after manual stops */
        if (s->forbid_restart) {
                *reason = "manual stop";
                return false;
        }

        /* Never restart if this is configured as special exception */
        if (exit_status_set_test(&s->restart_prevent_status, s->main_exec_status.code, s->main_exec_status.status)) {
                *reason = "prevented by exit status";
                return false;
        }

        /* Restart if the exit code/status are configured as restart triggers */
        if (exit_status_set_test(&s->restart_force_status,  s->main_exec_status.code, s->main_exec_status.status)) {
                /* Don't allow Type=oneshot services to restart on success. Note that Restart=always/on-success
                 * is already rejected in service_verify. */
                if (s->type == SERVICE_ONESHOT && s->result == SERVICE_SUCCESS) {
                        *reason = "service type and exit status";
                        return false;
                }

                *reason = "forced by exit status";
                return true;
        }

The order matters. A manual stop wins first, a prevented exit status blocks restart, and a forced exit status allows restart except for a successful Type=oneshot service. Only then does the Restart= switch decide. A true result is therefore a decision to attempt another start, not a guarantee that the attempt will run; the visual below lists the later gates.

RestartSec= provides a delay before a restart. The v256 manual also documents RestartSteps= and RestartMaxDelaySec= for increasing delays over configured steps. The exact computed delay and supported options must be checked against this version and unit settings; do not assume newer directives exist. Start-rate limiting can prevent repeated attempts within a configured interval. A restart loop can otherwise amplify load on a failing database or service dependency. An external recovery plan should account for the rate limit, evidence retention and downstream effects rather than simply clearing failures repeatedly.

For a synthetic timeline, suppose an authorized notify service starts at time 0 with a five-second start timeout and never sends READY. At time 5 the manager records a timeout and begins the configured stop/cleanup path. Assume cleanup completes at time 6 and the applicable restart delay is two seconds: the next attempt can be scheduled no earlier than time 8 under these simplified assumptions. If the configured start limit is hit, that retry may be refused instead. Real timings, signal delivery and configured backoff can differ. No process or service was actually restarted to produce these numbers.

A watchdog notification is another distinct assertion: when enabled and configured, a service must send its expected keepalive within the manager's deadline. Missing a deadline can produce a watchdog-related failure and interact with restart policy. Receiving a keepalive says that an authorized sender sent a message at that time; it does not establish that every application subsystem is healthy. A blocked worker, an overloaded event loop or an incorrectly placed notification can make a superficial heartbeat misleading. Use an application-specific check and record the watchdog interval and actual result before drawing a health conclusion.

Practice#

The following own-process Unix datagram model sends a readiness-shaped message between two sockets in a private temporary directory. It does not use NOTIFY_SOCKET, connect to systemd or simulate credential validation by a manager. It requires Python 3 and creates no persistent service or timer:

python3 - <<'PY'
import os, socket, tempfile
with tempfile.TemporaryDirectory() as directory:
    path = os.path.join(directory, 'notify.sock')
    receiver = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
    sender = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
    try:
        receiver.bind(path)
        receiver.settimeout(1)
        sender.sendto(b'READY=1', path)
        print('private message:', receiver.recv(128).decode())
        try:
            receiver.recv(128)
        except TimeoutError:
            print('no second notification before timeout')
    finally:
        sender.close()
        receiver.close()
PY

Expected output shows the private message and the bounded timeout. A missing Unix socket capability or permission denial is a valid result; do not change host policy. The directory and socket file are removed by the context manager after the descriptors close. This toy model demonstrates a message and timeout only; it does not reproduce systemd's notification credential checks, service state machine, process tracking or restart behavior.

Source-only alternative: use the A/B timeline above, annotate the event each type treats as readiness, then calculate the synthetic timeout and retry at times 5, 6 and 8. Add a counterexample in which READY arrives before the application is actually usable. Deliver the annotated timeline and identify which values are assumptions rather than captured manager output.

Knowledge check#

1. What does a successful Type=exec start establish, and what can still be unready?
2. Why can READY=1 from a helper fail to establish readiness, even when the datagram was sent?
3. How does the quoted SERVICE_FORKING branch distinguish control and main command bookkeeping?
4. Why is a PID number alone insufficient as a durable identity for a supervised process?
5. What can prevent an apparently configured restart from occurring at the next nominal delay?

Answers#

1. It establishes the configured execution transition, not that the application has finished initialization or can handle a particular request. A separate readiness protocol or application check may be needed.
2. The message must be received and accepted under notification access and process-attribution rules. It can arrive too early, from an unauthorized process or after a race with the sender's exit; its contents are also an application assertion.
3. The branch selects the configured start command as a control command and clears main_command. Later logic must establish the intended main process under the forking service's rules.
4. Numeric PIDs can be reused after exit. A process identity reference and lifecycle tracking help distinguish the supervised process from a later unrelated task, subject to the platform and implementation.
5. Start limits, the exit classification, explicit stop, failed conditions, cleanup duration or policy can change or prevent a retry. A nominal restart delay is not a promise of an unconditional attempt at exactly that wall-clock instant.

Checkpoint#

Submit the exec-versus-notify timeline, timeout/retry calculation, and either the private datagram output or the supplied source-only analysis. Ready means distinguishing successful exec from application readiness, naming notification authorization and process identity, and explaining one cleanup or limit failure. If uncertain, retry with a single service and no restart before adding the timeout.

Sources#

• systemd v256 service state machine
• systemd v256 manager jobs
• systemd v256 service manual source
• systemd v256 execution manual source