Jobs, dependencies, and transactions

Before you start#

Review the unit loader, D-Bus requests, and process lifetimes. The following traces systemd v256 commit 5c79cdec10a547a866764a66e1e14898112a00cd. It describes a historical userspace implementation; it does not claim that this version ran as the host manager.

Learning goals#

Distinguish dependency pull-in, ordering and failure propagation; follow transaction construction, order checking and job installation; and explain why an accepted request is not a completed job. Analyze a four-unit graph and its cycles without enqueuing any host jobs.

Source walkthrough#

A unit is a managed object with configuration and an activation state. A job is a requested operation on a unit, such as starting or stopping it, with its own waiting, running and result lifecycle. A transaction is the manager's temporary collection of candidate jobs and their relationships while it decides whether a request can be installed. The transaction is not a general atomic filesystem or database transaction: a subsequently started service can perform external effects, and failure does not automatically undo arbitrary work.

The v256 src/core/transaction.c, transaction_verify_order, is LGPL-2.1-or-later according to its SPDX notice. The following exact excerpt omits the generation setup and final return:

        HASHMAP_FOREACH(j, tr->jobs) {
                r = transaction_verify_order_one(tr, j, NULL, g, e);
                if (r < 0)
                        return r;
        }

The transaction owns or references a map of candidate jobs. The loop visits each job and invokes an ordering check using the current generation and error object. A negative return propagates an error rather than declaring the graph valid. The surrounding source explains that cycle checking may attempt to fix a cycle by dropping a job; whether this is possible depends on which jobs are essential and on the actual transaction. This excerpt does not assert that every ordering cycle can be repaired or that a removed job's service was running.

A second exact excerpt from the same file, within transaction_activate, shows how the caller handles that check. The SPDX license is LGPL-2.1-or-later; surrounding loop and logging are omitted:

                r = transaction_verify_order(tr, &generation, e);
                if (r >= 0)
                        break;

                if (r != -EAGAIN)
                        return log_warning_errno(r, "Requested transaction contains an unfixable cyclic ordering dependency: %s", bus_error_message(e, r));

A nonnegative result leaves the verification loop. -EAGAIN is a special retry signal here after a possible graph change; another error is reported as an unfixable ordering problem. The caller can then proceed to merge jobs and other transaction stages if ordering is acceptable. Error handling must release temporary transaction resources under the surrounding caller's ownership rules. The excerpt is a conditional code path from one release, not a measured boot trace.

In the v256 src/core/manager.c, manager job requests construct a transaction and invoke transaction_add_job_and_dependencies. The latter, in the pinned transaction source, recursively adds appropriate jobs for the requested unit's dependency atoms under flags and conditions. It can load referenced units, handle conflicts, and return errors. It does not follow that every relationship creates the same type of job or that every dependency is required for the anchor request. The manager then validates and installs an acceptable result. src/core/job.c contains job state and completion handling; its lifetime is separate from the lifetime of the unit it references.

Relationships answer different questions#

Wants= expresses a weaker pull-in relationship: starting one unit generally queues a start job for the wanted unit, but a failure to start it does not by itself invalidate the requesting unit's start. Requires= is stronger, but its failure behavior is conditional: if the required unit fails and the requiring unit has After= on it, the requiring unit is not started; an explicit stop or restart of the required unit is propagated with or without ordering. Requisite= fails immediately if the named unit is not already started; it does not pull it in. BindsTo= couples active state more closely, especially with After=, while PartOf= propagates stop/restart operations from the listed unit to the configuring unit. See the v256 systemd.unit manual for the conditions and direction of each relationship.

The v256 transaction builder turns dependency atoms into jobs. In src/core/transaction.c (LGPL-2.1-or-later), transaction_add_job_and_dependencies ends with the strong pull-in loop quoted below. Lines 1030-1040 are exact; the following ignored-atom loop is omitted:

        /* Finally, recursively add in all dependencies. */
        if (IN_SET(type, JOB_START, JOB_RESTART)) {
                UNIT_FOREACH_DEPENDENCY(dep, ret->unit, UNIT_ATOM_PULL_IN_START) {
                        r = transaction_add_job_and_dependencies(tr, JOB_START, dep, ret, TRANSACTION_MATTERS | (flags & TRANSACTION_IGNORE_ORDER), e);
                        if (r < 0) {
                                if (r != -EBADR) /* job type not applicable */
                                        goto fail;

                                sd_bus_error_free(e);
                        }
                }

The table in src/core/unit-dependency-atom.c maps Requires= to the strong start atom and Wants= to the ignored one. Errors from the strong loop abort construction unless they are -EBADR; errors from the ignored loop are logged and dropped. The excerpt shows construction only. It does not show default dependencies, a running service's later failure, or any ordering edge.

Before= and After= order jobs that are in the same transaction; they do not by themselves pull another unit in. Conversely, Requires= does not automatically mean After=. A requirement without an ordering edge can result in concurrently scheduled start jobs. If a service must not start until a required unit's start job has completed, express both the appropriate dependency and ordering, subject to the required unit's actual readiness semantics. A service reported started by a weak readiness type may still not be ready for application traffic. Conflicts= expresses a negative relationship and can cause a stop job or a transaction conflict; ordering still has to be considered. These distinctions and examples are documented in the v256 systemd.unit manual.

Defaults add relationships for many unit types unless disabled or overridden under documented rules. They make the effective graph larger than the explicit lines in one fragment. Template instances, targets, device units and socket activation can contribute additional dependencies. The v256 unit manual source describes those directives for this baseline. A graph drawn from one file alone is therefore a hypothesis until the selected fragment, drop-ins and generated/default relationships are accounted for.

Worked four-unit graph#

Consider synthetic units app.service, database.service, metrics.service and ready.target. The app has Requires=database.service, Wants=metrics.service, and After=database.service. The target has Wants=app.service and After=app.service. Assume all four fragments load successfully, no other dependencies exist, and the app is requested through the target. The manager constructs candidate start jobs for the target, app, database and metrics. Database's job is ordered before app's because of the explicit After=. The target follows app under its explicit order. Metrics is pulled in but has no specified ordering relative to app in this simplified graph.

If metrics fails, the app's weaker wants relationship does not automatically turn that failure into the same hard prerequisite failure as database. If database's ordered start job fails, the app's required relationship and ordering affect whether its own start can proceed according to the documented rules. The manager's unit and job results must be checked: a start job finishing successfully does not always mean a later application request succeeds. No claim is made that an actual machine ran this graph.

Now introduce a soft ordering cycle by adding After=ready.target to metrics and After=metrics.service to the target, while metrics is only wanted. The candidate ordering constraints can form a cycle. The v256 transaction code may be able to drop an eligible nonessential job and retry; it is not guaranteed to choose a particular job in every larger graph. If all jobs in an ordering cycle are essential to the anchor transaction, the verifier can report an unfixable cycle instead. The exact result depends on the full relationship graph and transaction flags. This is a distinction between a possible repair and an unconditional promise that “systemd fixes cycles.”

A different counterexample is a Requires= edge with no After= edge. Both jobs can be present but lack an ordering constraint between them. Observing the database start first in one run is not evidence that the configuration required that order. Add the appropriate After= and retest the model. Do not add a reverse ordering edge casually: two individually plausible lines can form a cycle when combined with defaults or target dependencies.

Jobs, replacement and observation#

A manager can receive a new request while a previous job is waiting or running. Job modes affect whether an incoming job replaces, fails against, isolates or otherwise interacts with existing jobs under the documented interface. Transaction merging must reconcile compatible jobs and reject or drop unmergeable candidates according to policy. A unit state transition can finish a waiting job, fail it, or cause another dependent job to become runnable. The manager's event loop processes these transitions; external service processes, D-Bus clients and kernel events can race with one another. A job's pointer or numeric identifier should not be retained after its completion without the interface's ownership and correlation rules. The v256 D-Bus API documents StartUnit()'s returned job object path, Subscribe(), and JobRemoved(); subscribe before issuing the request, then correlate the returned job path/ID with the removal event.

In the v256 transaction source src/core/transaction.c (LGPL-2.1-or-later), transaction_merge_jobs begins with a conflict check for jobs on the same unit. Lines 221-248 are quoted exactly; the second merge step is omitted:

        /* First step, check whether any of the jobs for one specific
         * task conflict. If so, try to drop one of them. */
        HASHMAP_FOREACH(j, tr->jobs) {
                JobType t;

                t = j->type;
                LIST_FOREACH(transaction, k, j->transaction_next) {
                        if (job_type_merge_and_collapse(&t, k->type, j->unit) >= 0)
                                continue;

                        /* OK, we could not merge all jobs for this
                         * action. Let's see if we can get rid of one
                         * of them */

                        r = delete_one_unmergeable_job(tr, j);
                        if (r >= 0)
                                /* Ok, we managed to drop one, now
                                 * let's ask our callers to call us
                                 * again after garbage collecting */
                                return -EAGAIN;

                        /* We couldn't merge anything. Failure */
                        return sd_bus_error_setf(e, BUS_ERROR_TRANSACTION_JOBS_CONFLICTING,
                                                 "Transaction contains conflicting jobs '%s' and '%s' for %s. "
                                                 "Probably contradicting requirement dependencies configured.",
                                                 job_type_to_string(t),
                                                 job_type_to_string(k->type),
                                                 k->unit->id);

When two candidate job types cannot merge, the function first tries delete_one_unmergeable_job. A successful drop returns -EAGAIN, so the caller collects garbage and tries the merge again. If nothing can be dropped, the transaction fails with a conflicting-jobs error. That message suggests contradictory requirement dependencies without naming the responsible line. This is one conditional path; it does not say which job a particular merge keeps.

An accepted D-Bus start request means the manager accepted or installed a job, not that the service has finished starting. A client that wants completion evidence can subscribe to relevant job-removal notifications and correlate the returned job identity and result. Subscription and request ordering matter: if a client submits a very fast job before establishing its observation mechanism, completion can occur before the client is listening. The client also has to handle disconnect and manager restart; a missing signal is not automatically a success or failure result. Unit active state, job result and application readiness answer different questions.

A critical-chain or boot timing view can help identify ordering and durations in a particular boot, but it is not a complete causal explanation for every delay. Parallel jobs can overlap; a service can return readiness early; clocks, cached work and external dependencies vary. A service duration displayed on one machine does not establish a universal boot dependency or performance bound. Preserve the boot identifier, tool version and relevant graph when making an observation, and do not publish sensitive unit names or logs without review.

Interpreting job outcomes#

A start job can be queued behind another ordered job while the associated unit is inactive. Once runnable, it invokes the unit operation and may remain in progress while the unit's asynchronous state changes. A unit can have a failed state after a job has disappeared from the manager's active job list; the removal is a lifecycle event, not erasure of every diagnostic. Conversely, another request can later reset or change unit state. An observer must timestamp the request and result and avoid assuming that a current status report describes the same job it watched earlier.

A client asking to stop a unit that is already inactive can obtain a different job path and result from a client requesting a start. A new conflicting request can replace or be rejected against an existing job depending on job mode and current state. If a transaction contains both a start and a stop for the same unit, merge rules and dependency direction matter; writing two names in a diagram is not enough to determine a result. Source inspection of transaction_merge_jobs and the particular mode is needed before predicting which candidate survives. Do not invoke these requests on the host to answer the exercise.

In the four-unit example, a useful event record would contain the requested target, generated jobs, each explicit ordering edge, the manager's acceptance result, the job removal result, and the observed unit state. If database's job is still waiting, app's After= constraint explains why app has not started under the stated graph. If database reports job success and app later fails its own initialization, the same ordering edge does not make database responsible for that later failure. The graph constrains possible ordering; it does not prove causation. Record application logs and independent evidence before drawing a causal conclusion.

Failure and cleanup#

A transaction can fail while loading a dependency, detecting an ordering cycle, merging jobs or checking a conflicting job mode. In these cases, the manager must discard or release temporary transaction state without claiming that the requested job completed. After installation, an asynchronously failing service produces a job result and unit state through a different path. Some related jobs may already have run, and service processes may have changed external state; a failed transaction or job is not proof of a globally rolled-back system.

Unit and job relationships are also lifetime relationships. The transaction tracks candidate jobs and dependency edges while verifying and merging them. Installed jobs are then managed through the manager's job structures, and completion removes or invalidates them according to the implementation. A callback or D-Bus client must not assume a job object remains accessible forever just because it once received a path or ID. A new job can reuse a conceptual unit name while being a different operation. Record the specific request, identifier, result and observation scope.

Practice#

This private graph exercise uses only Python and the synthetic relationships above. It does not call a manager or start units. The first graph is acyclic; the second adds a reverse edge and demonstrates a cycle. Python's TopologicalSorter is a teaching tool, not systemd's transaction algorithm.

python3 - <<'PY'
from graphlib import TopologicalSorter, CycleError
base = {'database': set(), 'metrics': set(),
        'app': {'database'}, 'ready': {'app'}}
print('acyclic:', sorted(TopologicalSorter(base).static_order()))
cycle = {key: set(value) for key, value in base.items()}
cycle['metrics'].add('ready')
cycle['ready'].add('metrics')
try:
    list(TopologicalSorter(cycle).static_order())
except CycleError:
    print('cycle detected')
PY

Expected output lists the four names and reports cycle detected; sorting the displayed list deliberately discards scheduling order, so use the declared dependency sets to explain ordering. If Python is absent, use the supplied graph and trace ready → metrics → ready as a cycle. The script does not decide which systemd job would be dropped and does not change the host. Deliver a separate table for pull-in edges, ordering edges and the hard/soft assumptions.

Knowledge check#

1. Why does Requires=database.service not by itself guarantee that database's start finishes before app's start?
2. What does the quoted verification loop do on a negative result, and what special retry is visible in its caller?
3. Why is an accepted start request not proof that the service is ready?
4. How can a client miss a fast job completion notification, and how should it reduce that race?
5. Why does a transaction error not guarantee that every external effect was rolled back?

Answers#

1. Requirement and ordering are separate relationships. Without an appropriate After=, jobs can be included without that explicit ordering; actual defaults and other edges must also be inspected.
2. It propagates the negative result from a job's ordering check. The activation loop retries on -EAGAIN after a possible graph change, while other errors are reported as unfixable for that path.
3. Acceptance concerns a job request. The job can fail later, and the unit's chosen readiness type may report start completion before the application can serve the request the client cares about.
4. Completion can occur between submitting a job and subscribing to its notification. Subscribe first, correlate the returned job identity and handle disconnect or manager restart instead of interpreting silence as success.
5. Transactions organize and install jobs, while services can perform external work after jobs run. Partial operations, already completed jobs or service side effects are not universally reversed by a later error.

Checkpoint#

Submit the four-unit pull-in and ordering graph, the soft-cycle counterexample and a table separating job acceptance, job result and application readiness. Ready means explaining why Requires= and After= answer different questions and identifying when the source retry is conditional. If unclear, remove metrics and the target, solve the two-unit graph, then add them back and retry.

Sources#

• systemd v256 transaction implementation
• systemd v256 manager job construction
• systemd v256 job implementation
• systemd v256 unit dependency manual