cgroup controllers, PSI, limits, and delegation

Before you start#

Review cgroup v2 membership, fair scheduling, memory reclaim, and block I/O. This chapter uses upstream v6.12 and cgroup v2. Controller availability, hierarchy and kernel configuration vary. All calculations are synthetic; the lab only reads the current process's visible cgroup.

Learning goals#

Distinguish proportional CPU weight from quota, compare memory protection and throttling with hard limits, explain task-count and I/O controls, and interpret PSI as stall time rather than utilization or root cause.

A quick mental model#

A cgroup is a group in a tree that the kernel uses to organize processes and distribute or account for resources. A task is a schedulable process thread, so task counts can exceed process counts. Settings higher in the tree can constrain every group below them. Pressure means work is losing time because a resource is contended; it is a symptom to investigate, not a diagnosis. The controller files expose different kinds of policy and measurements, so first identify the group and its ancestors, then compare relevant counters over time. See the kernel's cgroup v2 hierarchy rules and PSI definitions.

Source walkthrough#

Controllers implement resource distribution and accounting over the cgroup hierarchy described in cgroup v2 documentation. Settings are constrained by ancestors, enabled controllers and applicable domain or threaded topology. A control file's presence does not establish that the workload is using that controller as expected: check membership, parent settings, effective values and relevant statistics. Resource limits are operational controls, not a complete sandbox or a guarantee of equal performance across hardware.

The pids controller counts tasks, including threads, against hierarchical limits. In kernel/cgroup/pids.c, pids_can_fork uses this exact v6.12 GPL-2.0-only excerpt; surrounding declarations and return are omitted:

	pids = css_pids(cset->subsys[pids_cgrp_id]);
	err = pids_try_charge(pids, 1, &pids_over_limit);
	if (err)
		pids_event(pids, pids_over_limit);

The first line gets the pids-controller state from the task's CSS set. The next attempts to charge one task; a failure identifies an over-limit group and increments the relevant event accounting before the function returns the error. Fork or clone can fail with -EAGAIN at a pids limit. The cancellation path uncharges an already accepted fork if later creation fails, and release uncharges a departing task. A limit can be exceeded by administrative migration or by lowering the limit below current usage; it prevents further disallowed creation, not an automatic killing of existing tasks.

The charge walks the hierarchy. In pids_try_charge, v6.12, SPDX GPL-2.0-only (lines omitted where marked):

static int pids_try_charge(struct pids_cgroup *pids, int num, struct pids_cgroup **fail)
{
	struct pids_cgroup *p, *q;

	for (p = pids; parent_pids(p); p = parent_pids(p)) {
		int64_t new = atomic64_add_return(num, &p->counter);
		int64_t limit = atomic64_read(&p->limit);
	/* ... */
		if (new > limit) {
			*fail = p;
			goto revert;
		}
	/* ... */
	}

	return 0;

revert:
	for (q = pids; q != p; q = parent_pids(q))
		pids_cancel(q, num);
	pids_cancel(p, num);

	return -EAGAIN;
}

Each level that has a parent is charged in turn; the loop body does not run for the root. The first counter over its limit records itself as the failure point, and the revert loop cancels the charges already made, including the failing level, before -EAGAIN is returned. The excerpt does not show the callers, so it does not show that a particular fork reached this path.

Pressure Stall Information (PSI) accounts time during which tasks are stalled for CPU, memory or I/O. In kernel/sched/psi.c, this exact v6.12 SPDX GPL-2.0 excerpt is from psi_task_change; surrounding declarations are omitted:

	group = task_psi_group(task);
	do {
		psi_group_change(group, cpu, clear, set, true);
	} while ((group = group->parent));

The code finds the task's PSI group and applies the state change to it and its parent chain for the selected CPU. It is accounting for task state transitions, not measuring which exact application function caused a stall. Surrounding code skips the idle task and updates task flags. PSI depends on configuration and runtime enablement; a missing pressure file is an availability finding rather than a zero-pressure measurement.

CPU weight, quota and topology#

cpu.weight expresses a relative share among sibling cgroups competing for CPU under the fair scheduler and applicable hierarchy. It is work-conserving: if a sibling has no runnable work, another eligible group can use otherwise available CPU, subject to quota, cpuset, scheduler class and ancestor constraints. A weight of 200 versus 100 is not a reservation of two physical CPUs or a promise of exactly two-thirds of total machine capacity under every workload. Real-time and deadline tasks and optional scheduling mechanisms have their own rules; inspect the controller and scheduler configuration before applying a fair-scheduling approximation.

cpu.max expresses a maximum aggregate runtime quota and period, or max for no limit at that level (v6.12 CPU interface). Consider a synthetic quota of 100,000 microseconds per 200,000-microsecond period, with no tighter ancestor and two runnable fair tasks permitted to run on two CPUs. If both run continuously in parallel, they can consume the aggregate 100 ms of CPU time in about 50 ms of wall time, after which the group can be throttled until replenishment. On one CPU the same aggregate budget takes about 100 ms of wall time to consume. Actual enforcement involves period alignment, runtime distribution and scheduler behavior; this arithmetic is an illustration, not a promise of exact observed latency. cpu.max.burst, when supported and configured, permits limited accumulated burst behavior; it does not remove the quota's long-term hierarchical constraints.

Now put two sibling groups under an unconstrained parent, both continuously runnable on a single shared CPU, with weights 200 and 100 and no other competing runnable work. A rough steady-state fair-share expectation is two-thirds versus one-third. If the first group hits a tighter quota, its actual share can be less; if the second sleeps, the first can use spare time. Weight and quota answer different questions. Read cpu.stat for usage and throttling counters and compare over a measured interval rather than infer throttling from a single instantaneous CPU percentage.

The cpuset controller restricts eligible CPUs and memory nodes. cpuset.cpus.effective and cpuset.mems.effective reflect resources actually granted after parent constraints and online topology; requested lists can differ from effective lists. A weight cannot use a CPU outside the effective set. CPU hotplug, NUMA policy, memory placement and affinity further affect performance. Two tasks with the same quota can behave differently if one is confined to a busy CPU or remote memory.

Memory: accounting, pressure and limits#

memory.current reports charged memory for a cgroup and descendants under the controller's accounting rules. memory.stat breaks down categories, but shared pages, page cache, kernel memory and changing ownership make a simple sum of process resident sets an unreliable equality check. A page shared across processes is not necessarily charged separately to each; charges can outlive the task that first touched a page. Read the v2 documentation and relevant kernel version before comparing categories across systems.

memory.high is a pressure and throttling boundary (v6.12 memory interface): exceeding it causes reclaim and throttling work in relevant allocation paths, rather than an immediate promise to kill a task. memory.max is a hard limit with reclaim and, when necessary and applicable, cgroup OOM handling. The OOM killer can select a victim; memory.oom.group can request group-oriented treatment under its documented exceptions. memory.swap.max limits swap use separately where swap accounting and the controller are available. None of these values is a guarantee that every allocation below a limit succeeds: system-wide pressure, allocation context and other constraints still matter.

For a synthetic group with memory.high 1 GiB and memory.max 1.5 GiB, suppose two processes together raise charged usage to 1.1 GiB. Crossing high can make allocating tasks perform reclaim and experience delay; it does not by itself mean either task was OOM-killed. If usage approaches max and reclaim cannot make enough space for a permitted allocation, a memcg OOM can occur. One task can trigger the allocation while the selected victim is another eligible task, depending on policy and circumstances. A temporary overage, shared pages, reclaimability and ancestor limits complicate the outcome; inspect memory.events, memory.events.local, memory.stat and pressure over time before assigning blame.

memory.min and memory.low are protection controls with hierarchical and overcommit semantics, distinct from high and max. Giving every sibling a large protection value cannot create RAM; effective protection depends on parent budgets and actual usage. A high value is not a hard reservation, and a max value is not a target the kernel will always maintain without stalls. The memory controller source contains the accounting and reclaim paths for the pinned release.

I/O and other controllers#

The I/O controller can apply device-specific bandwidth or IOPS ceilings with io.max and distribute service with supported weight mechanisms. io.weight, io.cost and io.latency have different semantics and require suitable underlying device and scheduler/controller support. A configured limit on one major:minor device does not automatically describe every layer of a stacked device, network filesystem or page-cache hit. Buffered writeback attribution uses cgroup writeback support and filesystem/device behavior; shared inodes and ownership changes can make attribution differ from a naive “the last writer owns every byte” model. Check io.stat, the backing device graph and filesystem support before claiming a policy applies to the intended work.

The hugetlb controller accounts and limits configured huge-page resources separately from ordinary memory accounting; page size and available pools matter. The RDMA controller can limit supported RDMA resource objects where configured, not general network bandwidth. These controllers may be absent in a particular kernel or hierarchy. A missing file cannot be treated as an unlimited setting without first establishing that the controller exists and is enabled.

cgroup.events exposes lifecycle information including populated for the cgroup subtree. A populated transition can help a manager know whether tasks remain, but it is not proof that no external references, file descriptors or asynchronous work exist. The freezer interface requests freezing and reports state; completion is asynchronous and task behavior and hierarchy matter. Local and hierarchical event files must be distinguished: a parent's aggregate event count can change because of a descendant. These are lifecycle observations, not substitutes for verifying process exit and resource cleanup.

PSI: stalls, windows and attribution#

PSI reports some when at least some non-idle work is stalled for a resource and full when all relevant non-idle work is stalled under that resource's accounting semantics. CPU, memory and I/O have different definitions; system-level CPU full has special semantics and must not be compared blindly with a cgroup's values. The rolling averages cover defined windows, and the total field accumulates stall time in microseconds. A snapshot is not a utilization percentage, a count of individual blocked tasks, or proof that one component caused the stall. Use deltas over a recorded interval and correlate independent evidence.

For example, high memory PSI during a request slowdown supports the hypothesis that memory stalls overlap the incident. It does not by itself establish whether the source was a low cgroup high limit, system-wide reclaim, a slow backing device, compaction or a workload change. Compare memory.events high/max counters, memory statistics, CPU throttling and I/O evidence, then vary one controlled condition in a disposable environment. PSI triggers can notify userspace when a configured stall threshold in a time window is crossed; creating a trigger writes configuration to a pressure file and is outside this read-only lab. Sampling and instrumentation can perturb the workload, and averages smooth short spikes.

Reclaim path and a throttling race#

The memory-high discussion has a concrete v6.12 implementation boundary in mem_cgroup_handle_over_high. It reads the current task's accumulated pages over high and returns immediately if the count is zero. Otherwise it obtains the memory cgroup associated with the task's memory context, clears the task's pending count and enters reclaim processing. The source explicitly avoids continuing fruitless high-limit reclaim for a dying task. It calls reclaim_high with a batch-sensitive target, calculates a delay for memory and swap overage, and clamps that delay. Small penalties can be skipped; forward progress and retry conditions also affect whether the task sleeps. This is not a claim that every byte over high causes a fixed delay or an OOM kill.

The CPU bandwidth code has a corresponding race worth understanding. In throttle_cfs_rq, the scheduler takes the bandwidth lock and attempts to assign a small amount of runtime before committing to throttle. If runtime became available concurrently, it avoids dequeuing the runqueue; otherwise it adds the runqueue to the throttled list and proceeds with throttling under the scheduler's synchronization. That recheck prevents a stale “out of runtime” observation from unnecessarily leaving runnable work throttled for a period. cpu.stat increments and observed delays depend on the full bandwidth implementation and the sampling interval, not just the existence of this function.

The bandwidth recheck is in throttle_cfs_rq, v6.12, SPDX GPL-2.0 (the else branch and later code omitted):

	raw_spin_lock(&cfs_b->lock);
	/* This will start the period timer if necessary */
	if (__assign_cfs_rq_runtime(cfs_b, cfs_rq, 1)) {
		/*
		 * We have raced with bandwidth becoming available, and if we
		 * actually throttled the timer might not unthrottle us for an
		 * entire period. We additionally needed to make sure that any
		 * subsequent check_cfs_rq_runtime calls agree not to throttle
		 * us, as we may commit to do cfs put_prev+pick_next, so we ask
		 * for 1ns of runtime rather than just check cfs_b.
		 */
		dequeue = 0;
	} else {
	/* ... */

When __assign_cfs_rq_runtime finds runtime under cfs_b->lock, the function sets dequeue to 0 and skips adding the runqueue to the throttled list. The comment explains why it asks for 1 ns rather than a plain check. The excerpt does not show the dequeue walk or the period timer, so it does not measure how long a throttled group waits.

These two paths also show why a limit value is not a complete performance explanation. A cgroup can experience memory-high reclaim while a CPU quota simultaneously limits the reclaiming task's execution time; an I/O bottleneck can slow reclaim further. The relative timing and hierarchy determine the observed effect. A useful experiment holds workload and parent limits stable, measures counters over a defined interval, and includes a comparison where one suspected constraint differs. The source establishes possible mechanisms; it does not measure their contribution to a particular incident.

Practice#

Read-only own-cgroup sample

This Python program reads the unified membership and only proceeds automatically when it finds a cgroup2 mount whose mount root is /. It resolves the candidate beneath that mount and refuses paths escaping it. Non-root mount roots and unusual namespace views require manual interpretation; it reports them instead of guessing. It writes no cgroup controls.

python3 - <<'PY'
import os, re, time
print('kernel:', os.uname().release, 'time:', time.time())
rows = open('/proc/self/cgroup').read().splitlines()
entry = next((r.split(':', 2)[2] for r in rows if r.startswith('0::')), None)
mount = None
for line in open('/proc/self/mountinfo'):
    left, sep, right = line.partition(' - ')
    if sep and right.split()[0] == 'cgroup2':
        fields = left.split()
        unescape = lambda s: re.sub(r'\\([0-7]{3})', lambda m: chr(int(m.group(1), 8)), s)
        root, point = unescape(fields[3]), unescape(fields[4])
        if root == '/': mount = point; break
if entry is None or mount is None:
    print('unified membership or unambiguous root mount unavailable; inspect mountinfo manually')
else:
    base = os.path.realpath(mount)
    target = os.path.realpath(os.path.join(base, entry.lstrip('/')))
    if os.path.commonpath([base, target]) != base or not os.path.isdir(target):
        print('membership path is not safely resolvable in this view')
    else:
        print('candidate:', target, '(namespace and mount view must still be verified)')
        for name in ('cgroup.controllers', 'cgroup.events', 'cpu.stat', 'memory.current', 'memory.events', 'cpu.pressure', 'memory.pressure', 'io.pressure'):
            try:
                with open(os.path.join(target, name)) as f: print(name + ':', f.read().strip())
            except OSError as e: print(name + ': unavailable', e.errno)
PY

Run it twice with an interval appropriate for your investigation and record both timestamps, kernel version and workload changes. Differences in cumulative counters can support a hypothesis; a single value does not prove causation. A mount namespace or cgroup namespace can still make the candidate view misleading even with a root mount; confirm it against mountinfo and the process's membership. Missing controllers, permission errors, an unavailable unified mount and zero changes are valid findings. No privileged host change, cgroup migration or pressure trigger is required.

Source-only alternative: calculate the 100 ms quota consumed by two CPUs over 50 ms, then contrast the memory-high case with a max/OOM case and explain the pids charge failure. Deliver a table with assumptions, expected counters to inspect and at least two competing causes for a PSI increase. These are predictions, not observed throttling or reclaim.

Knowledge check#

1. Why can a group with twice a sibling's weight receive more than two-thirds of CPU when the sibling sleeps?
2. How can a 100 ms aggregate quota be exhausted in 50 ms of wall time?
3. Does exceeding memory.high immediately imply a cgroup OOM kill?
4. Why can a thread creation return -EAGAIN even if the number of application processes looks small?
5. What does memory PSI show, and what can it not identify by itself?

Answers#

1. Weight divides contested capacity among runnable eligible groups. Spare capacity can be used by another group subject to quota, affinity and ancestor constraints.
2. Two tasks running on separate CPUs each accumulate CPU time concurrently. Together they can consume 100 ms of aggregate budget in 50 ms, under the stated assumptions.
3. No. High triggers pressure and reclaim/throttling behavior; max and allocation failure have different OOM-related semantics.
4. The pids controller counts tasks, including threads, hierarchically. The charge can fail at an ancestor even when the number of visible process leaders is small.
5. It reports time spent in accounted stall states over specified windows or as an accumulated total. It does not identify the unique cause or equal memory utilization.

Checkpoint#

Submit the quota and memory examples, a pids failure timeline and a PSI interpretation with an alternative cause. Ready means weight, quota, high, max and stall accounting are distinct and any actual observations include their hierarchy and timing. Revisit the controller source and retry if a snapshot is used as proof of causation. This is not certification.

Sources#

• PIDs controller v6.12
• PSI implementation v6.12
• Memory controller v6.12
• cgroup v2 documentation
• PSI documentation