Reclaim, swap, pressure, and the OOM killer

Before you start#

Read physical page allocation, page faults and COW, and address spaces and page tables. This chapter describes upstream v6.12. Its reclaim and OOM examples are synthetic. The host exercise only reads counters; do not create memory pressure, configure swap, force reclaim or kill any process.

Learning goals#

Distinguish reclaimable clean file data, dirty/writeback state and anonymous swap backing. Explain classic LRU and multi-generation LRU as configuration-dependent alternatives, trace a reclaim and OOM decision, and interpret pressure and memory counters without equating an OOM with an empty machine.

Why reclaim exists#

The page allocator needs eligible pages that meet an allocation's node, zone, order, watermark and accounting constraints. Reclaim attempts to make pages available by evaluating memory that may be evicted, written back, swapped or otherwise released. The unit encountered by many v6.12 MM paths is a folio, which can represent one or more base pages. Its references, mappings, dirty state, writeback state and backing object all affect whether it can be reclaimed. A list position is not permission to free an object without checking those relationships.

Classic LRU-like reclaim organizes pages or folios using active/inactive and anonymous/file distinctions under its implementation. It approximates recency and reclaimability; the names do not promise perfect least-recently-used ordering. Multi-generation LRU, when built with CONFIG_LRU_GEN and enabled under applicable runtime settings, tracks generations and uses additional aging and feedback. A v6.12 system need not use MGLRU simply because the source contains it. Check configuration and actual enablement before interpreting its generation model. The multi-generation LRU documentation describes its design.

kswapd is a kernel reclaim thread associated with memory-node behavior; it can work in the background when watermarks indicate pressure. A task allocating memory can also enter direct reclaim when its allocation context permits. Direct reclaim consumes time in the allocating task's path and can involve filesystem or I/O activity depending on GFP constraints. Neither mechanism guarantees that every allocation succeeds or that a particular workload's latency is bounded. A nonblocking allocation can fail without being allowed to perform the same work.

Source walkthrough#

In v6.12 mm/vmscan.c, shrink_folio_list processes isolated folios. This excerpt is upstream v6.12 (SPDX GPL-2.0; surrounding list and accounting code omitted):

if (!folio_trylock(folio))
	goto keep;

The reclaim path tries to acquire the folio lock. If it cannot, it follows a keep path instead of acting as though it exclusively owns the folio. Later branches examine evictability, whether unmapping is allowed, reference state and dirty/writeback conditions. Other locks and references protect other relationships; the folio lock does not by itself establish that all mappings or pins disappeared. A folio can be activated, retained, sent for writeback, demoted, swapped or freed under different conditions. The exact branch depends on the reclaim context and configuration.

The same function calls folio_check_dirty_writeback and records dirty and writeback counts. Dirty data requires preservation according to the backing object's rules; writeback means I/O is in progress, not necessarily that data has reached durable media or that the folio can be freed immediately. Reclaim may avoid waiting in some paths to prevent long stalls, or may initiate writeback where permitted. A clean file-cache folio can often be discarded if it is no longer needed and its mapping/reference rules allow it, because the file can supply data on a later fault. An unevictable, pinned or otherwise referenced folio is not made free merely by calling it old.

In v6.12 mm/oom_kill.c, out_of_memory begins with this excerpt (SPDX GPL-2.0-only; declaration and later branches omitted):

if (oom_killer_disabled)
	return false;

The function can return without selecting a victim when the OOM killer is disabled. Other branches can discover that memory was freed, handle a current task already exiting, respect allocation context, determine the constraint, or select a victim. A call into this function is not identical to killing the allocating task. The caller and OOM context determine subsequent behavior. This short branch also shows that an OOM path has error and policy cases beyond the common story “find the largest process.”

Aging, eviction and working sets#

Reclaim scans candidates and uses access information, reference bits and feedback to estimate which memory can be evicted with acceptable cost. Recently used memory can be protected or promoted under the applicable classic or MGLRU algorithm. A later refault of evicted file data is evidence that reclaim discarded something the workload used again; workingset accounting can use such feedback. It is not proof that the next access will have the same cost or that a single counter identifies the optimal victim. Page-table accessed bits, reverse mappings and folio references require architecture and locking rules.

A dirty folio contains modified data that has not yet been handled according to its backing-store writeback rules. Writeback indicates an operation in progress; a write can fail or remain blocked. A clean folio with no disqualifying references may be removed from the page cache and freed, while a mapped folio may first need unmapping and TLB coordination. A pinned folio may resist migration or reclamation even if its contents are old. Reclaim's folio_trylock failure is one ordinary race: another actor owns the folio lock, so the scanner keeps it for now rather than assuming that waiting is always appropriate.

In mm/vmscan.c, v6.12, SPDX GPL-2.0 (surrounding list, congestion and statistics code omitted), shrink_folio_list samples the dirty and writeback state of each folio:

		folio_check_dirty_writeback(folio, &dirty, &writeback);
		if (dirty || writeback)
			stat->nr_dirty += nr_pages;

Notice that the call records state; it does not free anything. A folio that is dirty or under writeback is counted as pending work, which is why the earlier sections treat those states separately from "old". The excerpt does not show which branch later decides to write, keep or discard the folio.

MGLRU changes aging and candidate selection, but it does not erase dirty data, reference counts, writeback or backing-store constraints. Classic active/inactive terminology remains useful when reading many counters and paths; it should not be used as a complete description of all MGLRU decisions. Likewise a synthetic “oldest page” diagram cannot tell you the order of actual eviction from a running system with changing cgroups, nodes and workloads.

Anonymous memory and swap#

Anonymous private memory has no ordinary file from which its current modified contents can always be reconstructed. When swap is configured and a path permits it, reclaim can arrange swap backing and use swap-cache and swap-entry state to preserve data while physical memory is reclaimed. A swap PTE encodes a non-present mapping state that points into the relevant swap machinery rather than being an ordinary present PFN mapping. On a later fault, do_swap_page in v6.12 mm/memory.c participates in resolving that entry, coordinating swap cache, folio state, locking and PTE revalidation. The exact path can involve I/O, races or errors; a swap entry does not guarantee the data can be read successfully.

A swap-out operation must preserve data and ownership until the necessary write or compressed-storage operation is complete. Swap cache helps coordinate a folio associated with a swap entry; it is not simply an anonymous file descriptor. Another task can fault or unmap while I/O is in progress, so references, locks and page-table checks prevent stale state from being installed. If swap I/O fails or a backing device disappears, the kernel must follow the relevant error path; do not assume that “swap exists” implies every anonymous page is safely recoverable.

zswap is a compressed cache in front of configured swap backing under its implementation and policy. zram is a compressed RAM-backed block device that can be configured as swap or used for other block purposes. They are different layers: using zswap does not mean zram is present, and zram as swap is itself a swap device. Compression consumes RAM and CPU and does not make capacity infinite. The zswap documentation and zram documentation describe their separate interfaces.

swappiness expresses a relative cost preference in reclaim policy, not a percentage of RAM that must fill before swapping. Its effect depends on the kernel, workload, available swap, cgroup settings and reclaim path. A value alone cannot predict the next folio selected. Never change host swappiness or create swap merely to reproduce a textbook explanation.

A synthetic reclaim pass#

Imagine a scan considers five isolated base-page-sized folios under a context that permits the relevant operations. F1, F2 and F3 are clean file-cache folios with no blocking references or mappings. D is a dirty file folio not yet under writeback. A is an anonymous folio whose current contents require swap backing. A successful hypothetical pass might discard F1–F3, queue or otherwise arrange writeback for D, and allocate a swap entry and begin preserving A. At the end of that instant, the three clean folios may have become free; D and A cannot be counted as free merely because I/O was requested. Their later completion, references and rechecks determine the outcome.

Change the assumptions: D's writeback fails or remains blocked, and no swap space is available for A. Reclaim may retain or reactivate them and seek other candidates; it cannot safely discard modified contents just to satisfy the allocation. If an order-2 allocation requires four contiguous base pages, even freeing three scattered folios is insufficient to prove success. A cgroup can also reach its memory.max while the host has free RAM outside that cgroup's accounting and constraints. These are explanatory states, not a captured reclaim log or a promise about exactly five folios in a real scan.

OOM selection and constraints#

An out-of-memory condition is a failure to satisfy an allocation under its constraints after applicable recovery, not necessarily a machine with literally zero free RAM. A memory-cgroup limit, NUMA policy, cpuset, eligible zone, allocation order or watermark can constrain the available memory. The OOM path considers its oom_control, applicable domain and policy. A global OOM and a memory-cgroup OOM can have different candidate sets and consequences. The OOM documentation and cgroup v2 documentation provide context.

The kernel uses badness and eligibility rules, including oom_score_adj and protected or unkillable tasks, when selecting among applicable candidates. oom_score_adj is not a guarantee that a named process will be selected in every OOM context. An allocating task already exiting or carrying a fatal signal can be handled specially to help release memory. The OOM reaper can asynchronously unmap eligible victim memory; sending a signal does not instantaneously release every page, file, pinned mapping or reference. A victim may take time to exit, and some memory cannot be immediately reaped. Other policies, such as panic-on-OOM, can change behavior.

In mm/oom_kill.c, v6.12, SPDX GPL-2.0-only (surrounding code omitted), out_of_memory has an early branch for a task that will free its own memory:

	if (task_will_free_mem(current)) {
		mark_oom_victim(current);
		queue_oom_reaper(current);
		return true;
	}

Notice that this branch never runs the victim search. It marks the allocating task and hands its memory to the reaper, and it returns before select_bad_process. The excerpt does not show the other early returns, the oom_kill_allocating_task option or the cgroup-specific constraints that the sequence above omits.

Consider this synthetic log summary: “memory cgroup limit reached; allocation order 0; victim in that cgroup selected.” It supports a hypothesis about a constrained cgroup allocation, not that all host RAM was exhausted. To investigate a real event, preserve the full kernel and cgroup context, allocation flags, node/zone information, victim details and timing. A high-order failure without a kill can be fragmentation; a cgroup OOM can occur with host free memory; a global OOM can involve other constraints. Never reproduce these cases by pressuring a shared host or killing unrelated processes.

Practice#

Source-only route: make a table for F1–F3, D and A with initial state, proposed action, whether the page is immediately free and one failure condition. Then analyze the synthetic cgroup OOM summary and state why it does not establish total host RAM exhaustion. Add the folio_trylock failure branch and the disabled-OOM branch from the source walkthrough. These are worked models, not real logs.

Read-only host route: run as an ordinary user:

head -35 /proc/meminfo
grep -E '^(pgscan|pgsteal|pswpin|pswpout)' /proc/vmstat | head -30
if [ -r /proc/pressure/memory ]; then cat /proc/pressure/memory; else printf 'memory PSI not exposed\n'; fi

The counters can be host-wide, namespace-restricted or affected by unrelated workloads. PSI reports time-related pressure measures under its documented windows; a value is not a direct count of OOM kills. /proc/vmstat names and availability vary by configuration. Review output before sharing because it can disclose capacity and activity. An absent PSI file is a visibility limitation. Do not run stress tools, exhaust memory, alter swap, cgroups or sysctls, or clear caches. No files are created and no cleanup is needed.

For a counterexample, a nonzero SwapFree value does not prove that a particular constrained allocation can use swap or that a dirty file folio is already durable. Deliver the source-only table and one labeled observation or visibility limitation. If deeper testing is required, use a disposable guest with explicit resource limits and recovery, not this host.

Knowledge check#

1. Why does a failed folio_trylock lead to a keep path rather than immediate free?
2. What is the difference between dirty and writeback state?
3. Why is zswap not the same component as a zram swap device?
4. Why can a cgroup OOM occur when the host still has free memory?
5. Why doesn't selecting an OOM victim immediately guarantee that every associated page becomes free?

Answers#

1. Another actor may hold the folio lock or change its state. The reclaim path follows its synchronization rules and keeps the candidate for now instead of assuming exclusive ownership.
2. Dirty indicates modified data requiring appropriate handling; writeback indicates an I/O operation is in progress. Completion, errors and durability are separate questions.
3. Zswap is a compressed cache in front of swap backing; zram is a compressed RAM-backed block device that can itself be configured as swap. Their roles and configurations differ.
4. The allocation may be constrained by its memory cgroup or another domain even if memory exists elsewhere. Global free memory is not the same as the cgroup's available budget.
5. Exit and reaping are asynchronous and references, pins, shared mappings or I/O can keep memory alive. A signal is not an instantaneous universal free operation.

Checkpoint#

Deliver the five-folio state table and constrained-OOM interpretation, with labeled read-only counters if available. Pass when you distinguish dirty from writeback, swap cache from backing storage, and a constrained OOM from total physical exhaustion. If uncertain, retrace shrink_folio_list, do_swap_page and out_of_memory and revise the states. This is not certification.

Sources#

• v6.12 reclaim implementation.
• v6.12 OOM handling.
• v6.12 swap fault path and swap cache.
• Multi-generation LRU.
• zswap and zram.
• OOM handling.