Workqueues, kernel threads, and deferred work

Before you start#

Read interrupts and softirqs, timers and timekeeping, and scheduler core. This chapter uses upstream v6.12 source. The source-only race analysis is sufficient without a guest module. The optional host observation reads process metadata; it does not change workqueue settings, load modules or signal kernel threads.

Learning goals#

Follow a work item from queueing to callback execution; distinguish work item, workqueue, pool-workqueue and worker pool; reason about a requeue/cancel teardown race; and explain the cooperative stop protocol for a kernel thread.

Why defer work?#

An interrupt handler often cannot perform a long or potentially sleeping operation in its immediate context. A workqueue can arrange for a callback to run later under the workqueue's execution rules. Deferred execution changes the time and context of the work; it does not automatically keep the producer's device, buffer or enclosing object alive. A callback needs a valid ownership and synchronization protocol from the moment it is queued until it has finished or been canceled.

A work_struct represents a work item and includes its callback and internal state. A delayed_work combines work with a timer-related delay. A workqueue_struct is a logical queue and policy domain, not necessarily a dedicated thread. Concurrency-managed workqueues, often abbreviated cmwq, use worker pools and pool_workqueue associations to connect a logical workqueue to execution resources and accounting. A worker pool can serve multiple workqueues. A queue name in a trace does not imply that one permanently dedicated task executes only that queue's work.

Bound workqueues associate work with a CPU or locality according to their configuration; unbound workqueues use a different pool and affinity arrangement intended for flexible execution. CPU hotplug, affinity scopes and pool configuration affect the actual worker. A caller specifying a CPU to queue_work_on must satisfy its CPU-lifetime contract; the v6.12 source warns that a CPU going away can lead to execution elsewhere. An ordered workqueue provides ordering guarantees under its API, but it is not a global order across unrelated queues or a substitute for protecting shared data.

Source walkthrough#

In v6.12 kernel/workqueue.c, queue_work_on contains this excerpt (SPDX GPL-2.0-only; local IRQ save/restore and surrounding statements omitted):

if (!test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work)) &&
    !clear_pending_if_disabled(work)) {
	__queue_work(cpu, wq, work);
	ret = true;
}

The pending bit participates in preventing duplicate pending instances of the same work item. The disabled-work check can also prevent queueing; the function returns true when this call queued the work and false when it did not. False is not a generic assertion that the callback has already completed successfully: it can mean the item was already pending or work was disabled. When a worker claims an item, v6.12 clears its pending bit before invoking the callback, so a later queue attempt can schedule a subsequent execution while the callback is still running. Callers must use the workqueue API contract and their own lifetime protocol, not inspect internal bits as a substitute. The surrounding code saves and restores local interrupt state; __queue_work selects the appropriate pool-workqueue and uses pool synchronization (queue implementation).

On the ordinary worker path in this revision, worker_thread takes the pool lock, selects work and calls process_scheduled_works after assignment; that helper calls process_one_work. The pool lock protects shared worker and queue bookkeeping, but the worker releases it around the callback as described below. Rescuers and BH workers have additional paths, so this sequence is a bounded example rather than a universal stack trace.

A pending item and a currently executing callback are related but not identical states. A callback can be executing after its pending state has been cleared, and under the API's rules a later queue attempt can arrange another execution. The v6.12 workqueue implementation handles non-reentrancy and worker bookkeeping under its documented conditions; do not infer from one successful queue call that the object can be freed as soon as the producer returns. delayed_work adds a timer phase, so canceling only the embedded ordinary work without using the appropriate delayed-work API can miss part of the lifecycle.

In the same file, process_one_work records the current work and callback, removes the item from a worker's scheduled list, clears the pending state, releases the pool lock and invokes the callback. Its source contains this short statement (SPDX GPL-2.0-only; tracing, lock transitions and surrounding code omitted; full function):

worker->current_func(work);

The function's annotations and body show that the pool lock is released and later reacquired around execution; the callback is not simply invoked while holding the pool's raw lock. The source also notes that a work item may be freed from its own callback, so it avoids using the work object carelessly afterward. That special possibility does not authorize an external producer to free an item while another producer or worker can still access it. A caller must establish a complete ownership protocol for its particular design.

Ordinary workqueue callbacks run in worker context where sleeping can be permitted, subject to the callback's locks and API constraints. v6.12 also has BH workqueues with different execution-context restrictions; do not generalize sleepability to every workqueue type. An unbound queue, a high-priority queue and a real-time task are not synonyms. A worker can be delayed by other work, scheduling, resource pressure or a blocked dependency; deferred work is not a hard deadline guarantee.

Concurrency, ordering and reclaim#

Concurrency management aims to keep appropriate worker concurrency without requiring a separate dedicated thread for every work item. max_active controls aspects of active work for a queue under the documented scope; the meaning and effective limits depend on bound/unbound configuration and kernel version. An ordered queue is the appropriate interface when the API's ordering guarantee is needed; setting a numerical limit on an unrelated configuration is not automatically equivalent across all CPUs and pool arrangements. An item waiting behind a callback that itself waits for the same ordered queue can deadlock.

Flags express specialized requirements. Freezable workqueues participate in system freeze handling; high-priority workqueues use high-priority worker pools but do not promise SCHED_FIFO semantics. WQ_MEM_RECLAIM supplies a rescuer mechanism so that work needed for memory reclaim has a forward-progress path under the documented constraints. A rescuer is not infinite memory, a guarantee that arbitrary circular dependencies resolve, or permission to wait on work that cannot run until the current callback finishes. The workqueue documentation describes these choices and their limits.

A delayed callback can be queued, canceled or requeued while teardown proceeds. Its enclosing object must survive every producer and callback that can touch it. If the callback can queue itself, a teardown flag protected by the appropriate lock or another documented disabling protocol must stop future producers before final synchronization. Merely removing the item from a list owned by the driver does not cancel the kernel work item. Conversely, calling a synchronous cancel while holding a lock that the callback needs can deadlock: the canceler waits for the callback, while the callback waits for the lock.

Cancellation, flushing and a race#

cancel_work_sync waits for the relevant ordinary work to be canceled or finish under its documented contract; cancel_delayed_work_sync handles delayed work's timer and work components (the v6.12 implementation routes it through delayed cancellation, source). The documentation explicitly says cancel_work_sync(&dwork->work) must not be used for delayed work. flush_work and flush_workqueue wait for relevant work according to their respective scope and generation rules; drain_workqueue has a different queue-wide purpose and restricts new external queueing while draining. None of these APIs magically stops an unrelated producer from queueing again after the synchronization returns. disable_work_sync and related interfaces have their own disable-count semantics in this release; choose a protocol appropriate to the actual API and object (v6.12 workqueue API docs).

Consider an object D with a delayed work item. At time 0 a device callback queues D's work. At time 1 a worker starts running it, and the pending state is no longer a simple indicator that no callback exists. At time 2 another producer decides to queue it again while removal begins. If removal frees D after observing only a false queue result, the worker or later execution can dereference freed memory. If removal calls synchronous cancellation but leaves the producer enabled, the producer can queue after cancellation returns. Both are lifetime failures.

A safer conditional sequence is: under the driver's own synchronization publish a stopping state and prevent new producer entry; disable or detach the external source; release any lock the worker needs; call the matching synchronous cancel or disable operation; synchronize other callbacks and references; then free D. The actual ordering depends on the device and APIs. If the callback can rearm a separate timer_list, chapter 22's timer_shutdown_sync and workqueue teardown contract may both be needed. The stopping flag needs appropriate locking or memory ordering; a plain unsynchronized boolean is not a universal solution. Never call a synchronous wait on the currently executing item from that same callback if it would wait for itself.

Dedicated kernel threads#

A kernel thread is a kernel-managed task executing a thread function, not a userspace pthread with a normal userspace address space. kthread_create creates a thread in a stopped state under its interface; kthread_run combines creation and wakeup. A thread can cooperate with kthread_should_stop, kthread_should_park and wait mechanisms according to its implementation. Parking is different from permanent exit. A thread that blocks forever on a condition unrelated to its stop wakeup may prevent a stop from completing; the thread function must be designed to observe stop requests and release its resources.

In v6.12 kernel/kthread.c, kthread_stop contains these statements (SPDX GPL-2.0-only; tracing and intervening operations omitted):

set_bit(KTHREAD_SHOULD_STOP, &kthread->flags);

wake_up_process(k);
wait_for_completion(&kthread->exited);

The caller sets the stop request, unparks and wakes the task, then waits for the thread's exit completion. The surrounding function acquires and releases a task reference and returns the thread function's result (v6.12 implementation). This is a cooperative request and synchronization, not a force-kill that can safely interrupt arbitrary code. Calling it from the thread itself, or while holding a lock that the target needs to exit, can deadlock. A pointer to a kthread needs a valid lifetime; its numeric PID or a stale task pointer is not a reference.

The thread side of that request is kthread_should_stop in v6.12 kernel/kthread.c, SPDX GPL-2.0-only (the exported-symbol line and kernel-doc comment omitted):

bool kthread_should_stop(void)
{
	return test_bit(KTHREAD_SHOULD_STOP, &to_kthread(current)->flags);
}

The function reads the stop flag of the calling kthread and returns; it neither blocks nor wakes anything. A thread therefore sees the request only when its own loop calls it, or when a wait it uses returns, which is why a thread blocked on an unrelated condition can stall kthread_stop. The excerpt does not show how the waker reaches a sleeping thread; that happens on the kthread_stop side, as the sequence below shows.

Source-only lifecycle example#

Imagine work item W is initially idle. Producer P queues W and receives true. Worker K claims it and begins the callback. While K executes, a second request can result in another queued execution under the applicable workqueue rules; the same work item's non-reentrancy and queueing semantics must be preserved. Teardown T sets a protected stopping state, prevents P from issuing further requests, then invokes the matching synchronous cancellation without holding the callback's lock. When cancellation returns under its contract and all other users have released references, T may free the enclosing object.

Change one event: P reads a stale stopping flag and queues W after T's cancellation returned. T's prior wait does not protect against that new producer. Correct the design by synchronizing the producer gate and teardown, not by adding another unsynchronized flush_work. This is a conditional reasoning exercise, not evidence that a particular device driver follows that protocol.

Queue results and ownership#

Suppose producer P calls queue_work_on twice before a worker begins. The first successful call makes one instance pending. If the second sees the pending bit already set, it returns false; that does not tell P whether the worker will run soon, whether a prior callback is still executing, or whether the surrounding object has another owner. After a worker claims the item and clears the relevant pending state, a later queue request can be meaningful under the documented API. The exact interleaving must be established through the queue and object protocols rather than inferred from a single boolean.

Now suppose teardown sets a stop flag but reads and writes it without any synchronization with P. Even if teardown calls cancel_delayed_work_sync, P can fail to observe the flag and issue a later queue attempt. A correct design specifies how producers acquire access to the object, how the stop state becomes visible, which callbacks are detached, and how outstanding references are drained. If the stop flag and producer gate share a lock, the teardown must release that lock before waiting on work that might acquire it. This relationship is specific to the example and does not prescribe one universal lock for all workqueue users.

Practice#

Source-only route: draw W's idle, pending, executing, possible requeue and synchronized-teardown states. Mark the owner of the enclosing object at each transition, the producer gate and the point at which freeing becomes safe. Add the stale-producer counterexample and explain why waiting while holding the callback's lock deadlocks. Cite the two workqueue source points above and the kthread_stop reference/wait sequence.

Read-only host route: as an ordinary user, inspect the processes visible in your namespace:

ps -e -o pid=,comm= | awk '$2 ~ /^kworker/ || $2 ~ /^ksoftirqd/ { print; if (++n == 12) exit }'
cat /proc/loadavg

A container may hide kernel threads or expose no matching entries. A name such as kworker is not proof of which work item ran, its queue, its callback context or the kernel source version. Process names and PIDs can be sensitive; review output before sharing. /proc/loadavg is a system-level summary, not a workqueue backlog counter. Do not signal these tasks, inspect other users' private process data, write workqueue sysfs knobs or insert a host module. No files are created and there is no cleanup.

Deliver the lifecycle diagram and either a labeled read-only observation or the visibility limitation. If you need callback timing for a real investigation, use a disposable guest and appropriately configured tracing with a bounded workload; those tools and their overhead need a separate experiment. This chapter does not claim that a guest module or workqueue callback was run.

Knowledge check#

1. Why can queue_work_on return false without proving the callback has completed?
2. What is the difference between a logical workqueue and a worker pool?
3. Why does synchronous cancellation not by itself prevent a later producer from requeueing work?
4. What does the quoted kthread_stop sequence require from the thread function?
5. Why is WQ_MEM_RECLAIM not a cure for every dependency cycle?

Answers#

1. The item can already be pending or disabled, and an executing callback is a different state. The return reports this queue attempt, not a general completion result.
2. The workqueue is a policy and synchronization domain; pool-workqueues connect it to worker pools that execute work and can be shared. There need not be one dedicated thread per queue.
3. A producer that remains able to queue after the wait returns can create new work. Teardown must first gate or synchronize those producers and preserve object lifetime.
4. It sets a stop request, wakes the thread and waits for exit. The function must cooperate, notice the request under its design and reach a return; it is not forcibly terminated.
5. A rescuer supports forward progress under specified reclaim constraints, but cannot resolve arbitrary circular waits or replace resources that the callback itself requires.

Checkpoint#

Deliver the work-item lifecycle and teardown diagram with the stale-producer counterexample. Pass when you distinguish pending from executing, select the delayed-work cancellation interface, explain the no-self-wait rule and describe cooperative kthread stop. If uncertain, inspect queue_work_on, process_one_work and kthread_stop, then revise the ownership timeline. This is not certification.

Sources#

• v6.12 workqueue implementation: queueing, worker execution and cancellation.
• v6.12 workqueue interfaces.
• v6.12 kernel threads: stop and park.
• Workqueue documentation.
• v6.12 timer teardown.