udev events, rules, and device lifecycles

Before you start#

Review device model and sysfs, device ownership, service activation, and manager unit loading. This chapter reads systemd v256 commit 5c79cdec10a547a866764a66e1e14898112a00cd; the Linux v6.12 kernel is a separate historical source baseline. No host uevent, rule, device permission or hardware state is changed.

Learning goals#

Trace a device event from a kernel notification through userspace rule evaluation, worker cleanup and recorded properties. Distinguish matching from assignment and an event sequence from proof that a device is ready. Explain removal races and the authority of rule programs without executing host udev rules.

Source walkthrough#

A kernel device can produce a uevent, a notification about an action such as adding, changing or removing a device. The kernel exports device events through its kobject uevent mechanism; on Linux, udevd normally receives them through a netlink socket. udevd queues events and dispatches workers that apply configured rules, update device metadata and perform selected actions. The kernel device model, udev's userspace database, a systemd device unit and a desktop storage service such as UDisks are separate components with different responsibilities. A successful udev rule does not mean a filesystem is mounted or a desktop application has opened the device.

The following exact excerpt from systemd v256 src/udev/udev-rules.c, udev_rules_apply_to_event, has the file's LGPL-2.1-or-later SPDX license. Assertions and final return are omitted:

        LIST_FOREACH(rule_files, file, rules->rule_files)
                LIST_FOREACH_WITH_NEXT(rule_lines, line, next_line, file->rule_lines) {
                        r = udev_rule_apply_line_to_event(line, event, &next_line);
                        if (r < 0)
                                return r;
                }

The UdevRules object contains ordered rule files, each containing rule lines. The function walks those lines and calls the line evaluator with the current event and a pointer that can influence which line is visited next. A negative result propagates out instead of silently declaring the rule pass successful. A rule can contain several tokens and matching or assignment logic; the excerpt is not a complete parser specification. The event, rules and device references must remain valid while a worker evaluates them, and a device can disappear while processing is underway.

The next exact excerpt from v256 src/udev/udev-worker.c, in the event-processing path, is also LGPL-2.1-or-later. The following RUN handling and database update are omitted:

        r = udev_event_execute_rules(udev_event, worker->rules);
        if (r < 0)
                return r;

The worker invokes the event rule engine with its rule set and checks the result. On failure it returns the error rather than continuing as though all rules succeeded. The surrounding source disables a watch before processing, can run configured actions, reinstalls an inotify watch where appropriate and updates the device database. It explicitly tolerates one missing-device case when a watch is added because a device may already have been removed. The two lines do not establish that every later database write or device node operation succeeded.

In src/udev/udev-worker.c, worker_process_device, v256, GPL-2.0-or-later SPDX (the rule evaluation and database update omitted):

        if (udev_event->inotify_watch) {
                r = udev_watch_begin(worker->inotify_fd, dev);
                if (r < 0 && r != -ENOENT) /* The device may be already removed, ignore -ENOENT. */
                        log_device_warning_errno(dev, r, "Failed to add inotify watch, ignoring: %m");
        }

This is the removal-race branch. A failed watch addition is logged as a warning and processing continues, except that -ENOENT is expected because the device may already be gone. The rule-engine error returned earlier does not reach this branch. The excerpt does not show whether the later database update succeeds, and it does not establish that a removed device produces no further properties.

From event to worker and consumers#

The kernel's device model and netlink uevent mechanism report an action and properties associated with a device. udevd receives and queues events and dispatches work according to its event and device ordering rules. A worker processes an sd_device-related representation and the current rule set; its references and any opened descriptors must remain valid until processing and cleanup are complete. Rules can derive additional properties, create or adjust supported symlinks and permissions, invoke builtins or configured helper programs, and update database information under /run/udev/data according to configuration and device type.

The sequence number carried by a uevent is a kernel-assigned event counter useful for detecting/order-checking events observed by consumers; it does not certify that probing, firmware loading, filesystem availability and every userspace consumer have completed. Parent and child devices can have dependencies, and an add can race with a remove or a change. A worker that reads an attribute after the kernel device disappears can receive an error even if an earlier add event was valid. A later event can supersede a property from an earlier one. Treat a device object and a sequence number as scoped evidence, not a permanent identity for a physical device across unplug and replug.

systemd can expose device units for devices with the appropriate tagging and configuration. A rule can add SYSTEMD_WANTS to request unit activation under documented conditions, but a property assignment is not by itself proof that a service started or succeeded. UDisks or another desktop service can later make separate policy decisions about a block device and mounting. The kernel block device, udev's processing result, the manager's device-unit state and an application's mount request are separate boundaries. Do not infer a mount or an authorization decision from a udev property alone.

Rule matching and assignment#

A rule line consists of match conditions and assignments or actions. A match such as SUBSYSTEM=="block" tests a property; an assignment such as ENV{EXAMPLE}="yes" updates a property under that rule's semantics. != is a negative match, while += and -= have list-oriented meanings where supported. In the inspected v256 parser, OP_ASSIGN_FINAL is associated with :=; for keys that support final assignment, it prevents later applicable assignments from changing that value. Operator validity and behavior are key-specific; the parser can warn about or reinterpret an inappropriate operator. A learner should not assume that every key accepts every operator.

In src/udev/udev-rules.c, parse_operator, v256, GPL-2.0-or-later SPDX (the rule-line parser operator table, with the rest of the file omitted):

static UdevRuleOperatorType parse_operator(const char *op) {
        assert(op);

        if (startswith(op, "=="))
                return OP_MATCH;
        if (startswith(op, "!="))
                return OP_NOMATCH;
        if (startswith(op, "+="))
                return OP_ADD;
        if (startswith(op, "-="))
                return OP_REMOVE;
        if (startswith(op, "="))
                return OP_ASSIGN;
        if (startswith(op, ":="))
                return OP_ASSIGN_FINAL;

        return _OP_TYPE_INVALID;
}

The table tests the two-character comparison forms before the single =, and recognizes := as its own operator. It returns only an operator value. Whether a key accepts OP_ASSIGN_FINAL, and what it does with that value, is decided later by key-specific branches. For example, the inotify-watch option handler in the same file checks OP_ASSIGN_FINAL before setting its final flag. The operator table itself evaluates no rule.

Rules are processed in documented file and line order. Parent-matching attributes and the current device's attributes are not interchangeable; multiple parent-match constraints on a line have documented matching relationships. A GOTO can change the next rule line within the supported rule-file control flow. IMPORT can obtain properties from a supported source, a builtin or an external program depending on syntax. PROGRAM and RUN involve executable code under udev's authority and timing constraints; they are not harmless string substitutions. Running a rule evaluator against a real device can have side effects, which is why this chapter does not call udevadm test or trigger host events.

ENV{...} denotes a property in the rule context, not a promise that an arbitrary shell environment remains available forever. TAG associates a device with a tag under the rule engine's rules. SYSTEMD_WANTS is one possible property used at the device/manager boundary; it does not replace unit dependency analysis or service authorization. A rule program should be short and bounded under the service's documented execution environment rather than treated as an enduring daemon supervisor. A later service activation has its own identity, unit and lifetime.

The v256 udev manual source describes version-specific syntax. The pinned rule source's OP_ASSIGN_FINAL handling provides an implementation entry point, but a single token enum does not prove all assignments share identical finalization logic. Read the branch for the key in question and record the actual rule set and device properties when diagnosing a real system.

A worked add/remove timeline#

Use this synthetic event table. At T0 an ADD event for a virtual teaching device has SUBSYSTEM=demo, ACTION=add, SEQNUM=10 and property READY=0. At T1 a rule matching the subsystem assigns ENV{EXAMPLE}="seen"; a later matching line assigns a permitted final value for the same supported key under its key-specific rules. At T2 the worker would update its device database if the rule pass and update operations succeeded. At T3 a REMOVE event with sequence 11 arrives. This is invented state, not a host event and not an assertion that a particular real device has these properties.

The table answers only bounded questions. A rule with ACTION=="add" should not be assumed to match the remove event. A consumer that observed EXAMPLE=seen at T2 must still handle the device disappearing at T3. If the worker cannot reinstall a watch because the device has already gone away, the inspected code has a specific -ENOENT handling branch. A different error can be logged, and a database update can also fail. A saved path or property does not keep the kernel device physically present. A reference to a userspace device object controls that object's lifetime, not hardware availability.

Now imagine a helper in a rule fails or times out before producing an expected property. A later rule that depends on that property may not match, and a consumer may see incomplete or different metadata. The actual outcome depends on the rule operator and error handling; do not assume all helper failures abort all rule files. If a final assignment was made for a key that supports :=, a later ordinary assignment to the same key is constrained by that key's implementation. Inspect the exact v256 branch rather than generalizing from the synthetic label to every rule key.

Rule order as a state transition#

Imagine two synthetic rule files in documented order. The first tests ACTION=="add" and assigns an example property. The second tests that property and assigns another value. If the first match fails on a remove event, the second line's input state differs from the add case. This is a sequence of state transitions within one event, not a global rule that the property remains true for every future event or every instance of the same device model. A database entry from an earlier event can also be updated or removed under the device manager's rules.

A parent attribute lookup introduces another identity boundary. The event's device and an ancestor are different nodes in a device hierarchy; a rule that matches a parent attribute must use the documented parent-matching semantics. Two properties that happen to have the same string value do not establish that they came from the same ancestor. In a real review, retain the source of each property and the ordered rule file that tested or changed it. The private model below deliberately implements none of the real parent matching and cannot validate a production rule.

If a helper program fails, record its exit status and the rule context rather than editing device permissions to make a downstream application work. A permission or ownership assignment is an authority change, and a helper can act on a path that disappears or is replaced while the event is in flight. An appropriate review checks whether the source and target refer to the intended device and whether a later REMOVE event invalidates the result. Such checks need controlled test hardware or a disposable environment and are not performed by this model.

Ownership, removal and security#

A device event is not a long-running worker's property forever. The manager, worker, event object, rule set, sd_device reference, optional netlink connection and watch descriptors each have lifetimes. The selected worker path can obtain a reference to a netlink object if it was initialized during rule execution, and it handles watch teardown and restoration around processing. A cleanup path must release temporary references even when a rule or database update fails. A stale pointer in a worker is a memory-lifetime problem; a valid device object whose hardware has been unplugged is a separate availability problem.

A rule that executes a helper crosses an authority boundary. The helper runs in the udev service's configured context, which can be more privileged than the user who later accesses a device. Device properties or names from external hardware must not be blindly treated as shell code or trusted path components. Long-running work belongs in a properly supervised service when the documented design calls for one, not an unbounded rule helper. The exact sandbox and restrictions of the host udev service vary; no universal privilege claim follows merely from the name “udev.”

For a real incident, record the kernel and systemd versions, the action, device path, relevant rule-file order, selected properties and error stage. Avoid collecting serial numbers or other sensitive device identifiers unnecessarily. A rule parser warning, an absent sysfs attribute, a failed helper and a post-rule service failure are different diagnoses. udevadm output and device databases can change concurrently with hotplug. This lesson's mock data is sufficient for the reasoning exercise and does not require inspecting a real device or changing its permissions.

Practice#

This private Python model applies two simple in-memory rules to the invented ADD and REMOVE rows. It does not parse the full udev language, execute a helper, read sysfs or invoke udevadm test:

python3 - <<'PY'
events = [
    {'ACTION': 'add', 'SUBSYSTEM': 'demo', 'SEQNUM': 10},
    {'ACTION': 'remove', 'SUBSYSTEM': 'demo', 'SEQNUM': 11},
]
for event in events:
    result = dict(event)
    if result['ACTION'] == 'add' and result['SUBSYSTEM'] == 'demo':
        result['EXAMPLE'] = 'seen'
    print('synthetic', result['ACTION'], result.get('EXAMPLE', 'not assigned'))
PY

Expected output says the synthetic add is seen and the remove is not assigned. This intentionally does not model udev rule ordering, :=, workers, netlink or database state; those are explained from the pinned source above. If Python is unavailable, use the T0–T3 table and perform the same match by hand. No device node, host rule file, service or hardware is touched.

For the source-only deliverable, draw kernel event, manager queue, worker, rule pass and consumer as distinct stages. Use the invented add/remove values, mark the rule-engine error return and the removed-before-watch counterexample, and explain why a userspace reference cannot make a removed device available. Label all state values synthetic.

Knowledge check#

1. What does a negative return from udev_event_execute_rules do in the quoted worker path?
2. Why does an ADD sequence number not prove that all probing and userspace consumers are finished?
3. How do == and := differ, and why must the latter be checked for the particular key?
4. What is the difference between a valid sd_device reference and an available physical device?
5. Why is executing a rule helper a security and lifetime boundary rather than just parsing text?

Answers#

1. The worker returns the error from the selected path instead of continuing to the immediately following operations as though rule evaluation succeeded. Other surrounding cleanup and manager behavior must still be inspected for the exact failure.
2. It identifies an event within a particular observation; asynchronous probing, later changes, removal and userspace work have their own lifecycles. A sequence does not certify all of them are complete.
3. == is a match operator. := requests final assignment for keys that support it; supported operators and final-value behavior are key-specific, so the relevant parser and evaluator branch matter.
4. A reference can retain a userspace object in memory while the kernel device or physical hardware disappears. Later reads or watch operations may fail even though the object pointer remains valid.
5. A helper is executable code running with the udev service's configured authority and timing limits. Its inputs can come from device properties, it can fail or outlive assumptions, and it has side effects distinct from evaluating a match.

Checkpoint#

Submit the synthetic add/remove property table and the event-to-worker diagram, with the rule error and removal race annotated. Ready means distinguishing matching from assignment, object lifetime from device availability, and a device property from a completed service or mount. Retry with only one add and one matching line before adding removal and final assignment.

Sources#

• systemd v256 udev rule implementation
• systemd v256 udev worker
• systemd v256 udev manager
• systemd v256 udev manual source