SMP, topology, affinity, and CPU placement

Before you start#

Read scheduler core, fair scheduling and EEVDF, and real-time and deadline. The source baseline is upstream v6.12. The optional host experiment temporarily pins only its own short-lived Python process to a CPU already in its allowed mask, then restores the original mask in finally. It does not change cgroups, CPU online state or IRQ affinity. The source-only mask example needs no runtime experiment.

Learning goals#

Distinguish logical CPUs, cores, packages, SMT and NUMA nodes; explain the intersection of affinity, cpusets and usable CPUs; trace a v6.12 affinity update and its race handling; and separate CPU placement, memory policy, utilization estimates and hardware frequency.

Topology is a model of a particular machine#

An SMP system has multiple logical CPUs that can execute tasks. A logical CPU is the scheduler-visible execution unit; it is not necessarily a physical core. Simultaneous multithreading can expose multiple logical CPUs sharing parts of one core. Cores can belong to packages, caches can be shared at different levels, and NUMA nodes group memory and CPU locality according to platform descriptions. A virtual machine can expose a virtual topology that does not mirror its host. CPU IDs need not be contiguous, and a CPU that exists in a topology description may be offline or unavailable to a particular task.

The scheduler uses topology information to build scheduling domains and groups that describe balancing relationships. A domain may represent an SMT, shared-cache or wider span depending on architecture, configuration and topology. A group is an aggregation used in that balancing model, not necessarily a user-visible cgroup. Domains can be rebuilt as CPUs or cpusets change. The scheduler domains documentation and v6.12 topology code explain the mechanisms; they do not promise one hierarchy for every machine.

A cpumask is a set of CPU IDs. Several sets matter simultaneously: possible, present, online or active CPUs; the task's requested affinity; and the cpuset constraints that apply to it. Their precise treatment differs for userspace tasks, kernel threads, hotplug and specific paths. An allowed mask is not a promise that all listed CPUs are idle or equal capacity. A thread is normally the affinity unit: setting one thread's mask does not automatically change every thread in a process. A newly created child inherits relevant scheduling and affinity state under the creation rules, but later changes can diverge.

Source walkthrough#

In v6.12 kernel/sched/syscalls.c, __sched_setaffinity includes this excerpt (SPDX GPL-2.0-only; allocation and surrounding statements omitted):

cpuset_cpus_allowed(p, cpus_allowed);
cpumask_and(new_mask, ctx->new_mask, cpus_allowed);

The first call obtains the CPUs currently allowed by the task's cpuset constraints. The second computes the intersection of the requested mask and that allowed set into a temporary mask. This is an example of a kernel-owned intermediate set, not a pointer that userspace may retain. The function then checks deadline-affinity constraints where applicable and calls __set_cpus_allowed_ptr. Empty or otherwise invalid effective masks can fail; allocation can fail before this excerpt. The syscall wrapper also copies the userspace mask and handles faults.

The same function rechecks constraints after applying the mask because a cpuset can change concurrently. The following excerpt is from that race branch (same source and SPDX notice; surrounding comments and statements omitted):

cpumask_copy(new_mask, cpus_allowed);

retval = -EINVAL;

These lines do not stand alone: the branch checks whether the applied mask is still a subset of the newly read cpuset, may further restrict the replacement using a prior user mask, and calls the affinity-setting helper again before returning the error. The purpose is to recover from a concurrent constraint change rather than silently assume the first intersection stayed valid. Temporary masks are freed on the relevant exit labels. An error return is not a license to infer that absolutely no intermediate state changed; inspect the branch and re-read the effective affinity if this race matters.

The outer sched_setaffinity path obtains a task reference, checks ownership or capability when required, invokes security hooks, and prepares the user-mask context before the internal operation. The task can exit or constraints can change during an operation, so lookup, reference lifetime and locks matter. ESRCH, EPERM, EINVAL, ENOMEM or EFAULT can arise from different checks and circumstances. An empty allowed intersection is one possible EINVAL cause, not the only one. The syscall acts on the identified task/thread; it is not an instruction to migrate unrelated processes.

In v6.12 kernel/sched/core.c, SPDX GPL-2.0-only, __set_cpus_allowed_ptr_locked selects the destination CPU after the mask checks; the comment above the call and the earlier checks are outside this block:

	dest_cpu = cpumask_any_and_distribute(cpu_valid_mask, ctx->new_mask);
	if (dest_cpu >= nr_cpu_ids) {
		ret = -EINVAL;
		goto out;
	}

This is a concrete -EINVAL source: when no CPU in the new mask is valid, the update returns before __do_set_cpus_allowed and before any migration is queued. cpu_valid_mask is a kernel mask, distinct from the task's cpuset; the cpuset intersection happened earlier in __sched_setaffinity. The excerpt does not show the comment about spreading a cpuset-wide change over a roughly random CPU, nor the affine_move_task path that follows a successful choice.

Placement on wakeup and during balance#

When a task wakes or is initially placed, generic scheduler code invokes a class-specific select_task_rq path under its applicable synchronization. The fair implementation, select_task_rq_fair in v6.12 kernel/sched/fair.c, uses the previous CPU, wake flags and domain information among other inputs. The source asserts the task's pi_lock for stable affinity information on this path. Wake-affine and idle-CPU decisions attempt to balance locality, current load and available CPUs; they are heuristics with configuration and topology dependencies, not promises to run next to the waker.

An idle CPU can pull suitable work, and periodic or event-driven balancing can move tasks across relevant scheduling domains. The exact triggers differ with scheduler configuration, nohz behavior and CPU state. A task can be migrated between runqueues only with appropriate locking and ordering; moving a running task can require a stopper mechanism. In v6.12 scheduler core, migration_cpu_stop is documented as running in a high-priority stopper context to push a task off a CPU. It rechecks conditions because the destination or source CPU may have changed state. Migration is a coordinated state transition, not a write of a new CPU number while both runqueues proceed independently.

Affinity updates can race with CPU hotplug or cgroup changes. A CPU that was allowed at the time a userspace mask was constructed can be offline or disallowed when the kernel applies it. Likewise, a successful setting can later be narrowed by external constraints. A robust observer reads the effective mask after a change and handles errors; it does not assume that a previous snapshot remains valid forever. This chapter does not instruct you to offline a CPU or alter a cpuset, because those operations affect other work and require a disposable environment and recovery plan.

A worked intersection and migration#

Assume a hypothetical task requests CPUs {0,1,2,3}, its cpuset currently permits {2,3,4}, and only {0,1,2,4} are online and usable for the example. The simple intersection is {2}. If the relevant kernel path accepts the request, CPU 2 is the only candidate in this simplified state; that does not guarantee immediate execution there. If the task was running on CPU 1, an affinity update may require migration through the appropriate scheduler path. Until synchronization completes, reading isolated fields without the required locks can observe a transition, not a stable final state.

Now request only {0,1} while the cpuset remains {2,3,4}. The requested and permitted sets have no overlap, so an affinity update can fail with EINVAL instead of selecting an arbitrary prohibited CPU. If the cpuset changes between the first intersection and the later recheck, the v6.12 branch described above can revise the effective mask and return EINVAL. The example assumes a normal userspace task and omits policy-specific constraints; kernel threads and deadline reservations can impose different rules. No real CPU IDs or host state are being claimed here.

A second timeline separates wakeup and migration. Task A sleeps after last running on CPU 2. A waker makes it runnable; its class evaluates a target under the allowed set and current topology. Suppose CPU 2 is busy and CPU 4 is allowed, so the policy chooses CPU 4. Enqueueing there makes A a candidate; it runs only when CPU 4 selects it. If CPU 4 becomes unavailable during the transition, hotplug and affinity machinery must handle the race according to the source's locking and retry rules. The scheduler may choose a different allowed CPU. This hypothetical case is not evidence of the exact heuristic or timing on your machine.

Capacity, memory and energy are distinct#

On heterogeneous systems, CPUs can have different capacities, frequency ranges or thermal constraints. Scheduler capacity and thermal pressure are estimates or inputs used by relevant paths; the exact normalization and accounting depend on architecture and kernel configuration. On a homogeneous machine or virtual machine the values may be equal, unavailable or abstracted. A CPU's advertised maximum frequency is not a guarantee of its current effective throughput, and one scheduler utilization value is not an application benchmark.

PELT tracks decaying load and utilization signals; util_est can supplement estimates for tasks whose recent behavior is not yet fully reflected. These signals can feed placement and frequency decisions under applicable configurations. Energy-aware scheduling, when enabled and supported by an energy model and appropriate topology, considers energy-related costs; it is not universally active and does not guarantee minimum energy for every workload. CPU frequency governors and thermal controls are related systems with their own update cadence and constraints. Do not infer that a wake-affine decision was based on energy merely because the machine has a frequency governor.

NUMA memory policy concerns where memory is allocated or preferred; CPU affinity constrains where a task may run. The two can interact with locality but neither automatically enforces the other. A task pinned to a CPU can access memory allocated on a different node, and a memory policy does not necessarily pin the task to a CPU. Automatic NUMA balancing, page migration and cpuset memory-node constraints add configuration-dependent behavior. A useful experiment records both CPU placement and memory placement instead of assuming one from the other.

Observe a transition without overclaiming#

Imagine a diagnostic records a task on CPU 1 just before an affinity change and CPU 2 shortly afterward. Those snapshots are compatible with migration, but they do not show the complete sequence or prove which scheduler callback made the placement decision. The task may have blocked, awakened, and then run on CPU 2; another observation could catch it while a migration is being coordinated. A causal investigation needs appropriately configured tracepoints or other evidence, timestamps, task identity, effective masks and any relevant hotplug or cpuset changes. Numeric PIDs can be reused, so a long recording also needs a way to distinguish process lifetimes. Tracing adds overhead and is outside the read-only exercise below. A source map explains which transitions are possible; it is not a substitute for observing the specific execution.

Practice#

Source-only route: calculate the two intersections in the worked example. Draw the state before and after a successful move from CPU 1 to CPU 2, then add a cpuset-change race and identify the __sched_setaffinity recheck and error. Label online/active assumptions and explain why a CPU in the mask may still be busy. This is supplied source-based reasoning, not a measured migration.

Read-only topology observation: on a Linux host, these commands inspect the current shell and visible topology. Output can reveal CPU layout and container allocation, so review it before sharing. Missing sysfs files or restricted proc access are meaningful limitations, not a reason to escalate privileges.

printf 'shell pid=%s\n' "$$"
sed -n '/^Cpus_allowed_list:/p; /^Mems_allowed_list:/p' "/proc/$$/status"
cat /sys/devices/system/cpu/online
cat /sys/devices/system/cpu/possible
ls /sys/devices/system/node 2>/dev/null | head

The allowed list can be a subset of online CPUs, and the visible NUMA directories need not identify physical hardware from within a virtual machine or container. A proc snapshot cannot establish where each page of memory lives. If files are unavailable, retain the error and use the source-only route.

Optional own-process pin: run this Python 3 program as your ordinary user. It changes only the Python process and attempts to restore the original effective mask in finally; the parent shell is not pinned. The selected CPU comes from the original allowed set, not a hard-coded host CPU.

python3 - <<'PY'
import os
original = os.sched_getaffinity(0)
if not original:
    raise SystemExit('empty affinity mask; no change attempted')
cpu = min(original)
print('original:', sorted(original), 'chosen:', cpu)
try:
    os.sched_setaffinity(0, {cpu})
    print('temporary:', sorted(os.sched_getaffinity(0)))
finally:
    try:
        os.sched_setaffinity(0, original)
        print('restored effective:', sorted(os.sched_getaffinity(0)))
    except OSError as error:
        print('restore failed; process will exit:', error)
PY

Normally the temporary set contains one CPU and the restored effective mask matches the original. A concurrent cpuset or CPU availability change can cause the set or restoration to fail or be narrowed; the finally block reports the error and the short-lived process exits. Do not respond by changing a host cpuset or online state. A successful affinity update does not prove which CPU executed every instruction or where memory was allocated. No files are created. Deliver the source-only intersection table, actual masks if you ran the exercise, and one reason the two are not interchangeable.

Knowledge check#

1. Why can a machine with eight logical CPUs have fewer than eight physical cores?
2. What does the cpumask_and excerpt compute, and why is a later cpuset recheck needed?
3. Why can an affinity call return EINVAL even though the requested mask contains valid CPU numbers?
4. How is CPU affinity different from NUMA memory policy?
5. Why do PELT, capacity and frequency values not guarantee that the scheduler chooses a particular CPU?

Answers#

1. SMT can expose multiple logical CPUs per core; virtual topology can also differ from physical hardware. The actual topology must be inspected rather than inferred from a count.
2. It intersects the request with the currently allowed cpuset. A concurrent cpuset update can invalidate the earlier result, so v6.12 rechecks and handles the race.
3. The request may have no usable intersection with the task's constraints, or another affinity or policy validation can fail. Valid IDs alone do not establish permission or online availability.
4. Affinity constrains CPU execution; NUMA policy controls aspects of memory allocation or placement. Either can change without guaranteeing the other.
5. They are inputs or estimates in configuration-dependent paths among affinity, topology, load, thermal and policy constraints. They are not a complete deterministic trace or a throughput guarantee.

Checkpoint#

Deliver the intersection and race table, plus the Python masks if the optional experiment ran. Pass when you explain the first intersection, empty-mask error, post-update race and one distinction between CPU and memory placement. If uncertain, reread __sched_setaffinity and select_task_rq_fair, then repeat the source-only model. This is not certification.

Sources#

• v6.12 affinity syscalls: masks, checks and cpuset race.
• v6.12 scheduler core: migration and wakeup paths.
• v6.12 fair scheduler: wake placement.
• v6.12 topology implementation.
• Scheduler domains.
• cgroup v2.