NUMA policy, huge pages, and memory placement

Before you start#

Read reclaim, swap and OOM, physical page allocation, page faults and COW, and SMP topology and affinity. The source baseline is upstream v6.12. A single-node machine or restricted container is a valid environment. The optional exercise maps a small anonymous range in its own process and reads its own proc information; it changes no host NUMA policy, cpuset, huge-page pool or global THP setting.

Learning goals#

Separate CPU placement from NUMA memory policy and first touch. Trace a v6.12 VMA allocation policy and a huge-page collapse revalidation/failure path. Compare transparent huge pages, multi-size THP and hugetlb without promising that a request produces a large mapping or improves performance.

Nodes, CPUs and policy#

NUMA organizes memory and CPUs into nodes with different locality or access costs. A node is not the same object as a CPU: it can contain several CPUs, memory, both, or platform-specific resources. A task's CPU affinity constrains where its threads execute; memory policy and cpuset memory-node constraints influence where eligible allocations come from. Pinning a task to a CPU does not automatically pin its pages to the corresponding node, and a memory policy does not guarantee execution on that node's CPU. A virtual machine's reported topology may not match the host's physical layout.

Linux supports memory policies including default/local behavior, preferred and bind-style node choices, and interleave policies. The exact result depends on policy scope, nodemask, cpuset restrictions, allocation type, available memory, fallback rules and configuration. A preferred node can fall back where permitted; a binding restriction can cause an allocation failure rather than unrestricted fallback. “First touch” is a useful shorthand for an anonymous page being allocated when first faulted under the then-applicable context and policy, not a universal guarantee that every page remains on the first CPU's node forever. Shared mappings, preallocation, migration and explicit policies change the story.

Upstream v6.12 includes MPOL_WEIGHTED_INTERLEAVE in addition to ordinary interleave, with weighting controls and restrictions described in its NUMA memory-policy documentation. Do not assume that older distribution kernels support the same mode or controls, or that a requested policy is valid for every node mask. The VMA, current task and shared-object policy can each matter. The process must also be permitted to use the relevant nodes under its cpuset and allocation context.

Source walkthrough#

In v6.12 mm/mempolicy.c, get_vma_policy contains this excerpt (SPDX GPL-2.0-only; interleave-index handling omitted):

pol = __get_vma_policy(vma, addr, ilx);
if (!pol)
	pol = get_task_policy(current);

The first call asks for a policy associated with the VMA/address under its applicable operations. If it obtains none, the function falls back to the current task's policy. The subsequent branch adjusts an interleave index for ordinary or weighted interleave. Some shared policies carry an extra reference that callers must conditionally release; the function's source comment describes that ownership. A policy pointer is not safe merely because it was once returned, and a fallback policy is not proof that the preferred node had free memory.

The same file's vma_alloc_folio_noprof, reached through the vma_alloc_folio wrapper in the relevant configuration, gets the policy, allocates with folio_alloc_mpol_noprof, and calls mpol_cond_put(pol) before returning. The caller must hold the documented mmap/VMA protection, supply a valid address and handle a null allocation result. This is a policy-aware allocation path, not an unconditional promise to allocate from the current CPU's node. Flags, order, allowed nodes and fallback rules also participate.

A different path is huge-page collapse. In v6.12 mm/khugepaged.c, collapse_huge_page includes this excerpt (SPDX GPL-2.0; surrounding lock and cleanup code omitted):

result = alloc_charge_folio(&folio, mm, cc);
if (result != SCAN_SUCCEED)
	goto out_nolock;

Before this allocation the function drops the mmap read lock because allocation and compaction can take time. A failed allocation or charge follows the no-lock cleanup path instead of installing a huge mapping. On success it reacquires protection and revalidates the VMA and PMD before proceeding; later code uses page-table and anonymous-memory synchronization, installs a huge mapping on the successful path and releases a prepared folio on applicable failures. The VMA or page tables may have changed while the lock was dropped. Allocating a large folio is therefore not equivalent to successfully collapsing the mapping.

A two-node allocation timeline#

Consider a synthetic system with node 0 and node 1, each with a CPU and memory. A task begins on CPU 0 and creates a private anonymous mapping. Under an assumed local/default policy, its first write to page A occurs while it runs on CPU 0, and allocation succeeds from node 0. The task then moves to CPU 1 and first writes a different page B; under the same simplified assumptions B may be allocated from node 1. A remains on node 0 unless a later policy or migration mechanism moves it. This illustrates a possible first-touch relationship, not a rule that a migration of the thread instantly migrates all memory.

Now change the policy: a bind policy and cpuset allow only node 0. A first touch while the thread runs on CPU 1 can still allocate from node 0 if permitted and available; CPU and memory location are distinct. If node 0 lacks eligible memory and the applicable policy forbids fallback to node 1, the allocation can fail or enter the relevant reclaim/OOM handling despite free memory elsewhere. A preferred policy can have different fallback behavior. Exact outcomes depend on the policy, kernel and allocation flags, so the table is not a prediction for an arbitrary machine.

Automatic NUMA balancing can use hinting faults and migration decisions to improve locality under its configured policy. It can move pages or influence task placement, but it is not a guarantee that all memory follows a thread, that every page is migratable, or that a migration improves a particular workload. Pinned pages, shared access, device mappings, hotplug and memory pressure constrain movement. Record node masks, task affinity, mapping type and time when investigating placement; a single numa_maps snapshot does not describe the full history.

Transparent huge pages and mTHP#

Transparent huge pages, THP, allow suitable memory to be backed by larger folios or mappings without the application explicitly allocating from a hugetlb pool. On common x86-64 configurations, a PMD-sized mapping can cover 2 MiB with 4 KiB base pages. A large folio and a huge page-table mapping are related but distinct: multi-size THP can use folios larger than a base page while still using PTE mappings. Do not equate every THP allocation with a PMD-sized hardware translation or assume every architecture has the same supported sizes.

At fault time the kernel may allocate an appropriate large folio, fall back to base pages, or fail according to policy, resources and context. khugepaged can scan and collapse eligible regions later under its configured behavior. madvise(MADV_HUGEPAGE) communicates a preference for an eligible range; it does not reserve contiguous memory, override every system policy or guarantee promotion. MADV_COLLAPSE is a separate interface with its own synchronous behavior and constraints; this exercise does not invoke it. Check v6.12 THP documentation and the actual size/configuration controls.

A huge folio or mapping can be split when needed by memory management, mapping changes or other operations. Collapse can require allocation, charging, copying and synchronization, and can fail if pages are pinned, shared incompatibly, swapped, recently changed or otherwise unsuitable. The source's SCAN_* outcomes distinguish reasons, but not every failure is visible as a single userspace errno in every path. A VMA recheck after dropping mmap_lock is essential because an unmap or permission change could invalidate the earlier candidate. A correct cleanup must release the prepared folio and locks appropriate to the branch reached.

In mm/khugepaged.c, v6.12, SPDX GPL-2.0 (surrounding lock and cleanup code omitted), collapse_huge_page takes the write lock and checks the candidate again before any page-table change:

	mmap_write_lock(mm);
	result = hugepage_vma_revalidate(mm, address, true, &vma, cc);
	if (result != SCAN_SUCCEED)
		goto out_up_write;
	/* check if the pmd is still valid */
	result = check_pmd_still_valid(mm, address, pmd);
	if (result != SCAN_SUCCEED)
		goto out_up_write;

Notice that both checks use the same SCAN_* result convention and jump to a cleanup label rather than continuing. The excerpt does not show hugepage_vma_revalidate itself, the later PMD installation, or the cleanup performed at out_up_write.

A worked coverage calculation: with 4 KiB base pages, 2 MiB / 4 KiB = 512 base pages. If an eligible aligned region is represented by one PMD-sized huge mapping, one translation can cover that 2 MiB rather than requiring 512 base-page PTE translations for the same span. That can reduce translation overhead in some workloads. It does not guarantee a 512-fold speedup: TLB organization, access pattern, cache behavior, allocation cost, fragmentation, NUMA placement and split/collapse overhead all matter. A PTE-mapped multi-size folio does not necessarily have the same TLB coverage as a PMD mapping.

In include/linux/huge_mm.h, v6.12, SPDX GPL-2.0 (surrounding definitions omitted):

#define HPAGE_PMD_ORDER (HPAGE_PMD_SHIFT-PAGE_SHIFT)
#define HPAGE_PMD_NR (1<<HPAGE_PMD_ORDER)
#define HPAGE_PMD_MASK	(~(HPAGE_PMD_SIZE - 1))

Notice that HPAGE_PMD_NR is derived from the PMD shift and PAGE_SHIFT, so the 512 in the calculation above is the x86-64 4 KiB case, not a universal constant. The excerpt does not show whether a particular VMA, architecture or configuration enables PMD-sized THP at all.

Hugetlb is a different contract#

Hugetlb pages use explicitly managed huge-page resources and accounting. Persistent and surplus pools, reservations, supported sizes and permissions depend on system configuration and policy. A reservation is not the same as successfully faulting every requested page, and an application can encounter allocation or fault failure if the pool or reservation conditions cannot be met. Hugetlb pages are not swapped out under ordinary memory pressure as documented; this differs from treating all THP behavior as a single nonswappable category. See hugetlb documentation and v6.12 mm/hugetlb.c.

THP's opportunistic allocation and splitting mechanisms are not the same as explicit hugetlb pool accounting. Both can reduce page-table overhead for suitable workloads, but they have different APIs, reservation behavior and failure modes. A HugePages_Free counter concerns the hugetlb pool; it does not establish whether an anonymous VMA was promoted to THP. Conversely, AnonHugePages in an applicable /proc/PID/smaps entry reports an accounting value for that mapping, not a universal guarantee about every folio or hardware translation in the process. KernelPageSize and MMUPageSize also require architecture- and mapping-specific interpretation; do not equate either mechanically with all THP sizes.

Compaction can help form contiguous memory for larger allocations, but pinned or unmovable pages, fragmentation, cgroup charging, NUMA restrictions and latency policy can cause failure or fallback. Memory hotplug and page migration introduce additional lifetime and locality constraints. An application that merely asks for huge pages must tolerate the documented fallback or error behavior; it cannot assume that total free RAM proves the necessary physical block is available.

A collapse race and a failure#

Imagine a VMA covering an aligned 2 MiB region backed by base pages. A collapse scanner confirms an initial candidate, drops the mmap read lock and allocates a large folio. Before it reacquires the lock, another thread unmaps half the range. The v6.12 source revalidates the VMA after reacquisition; if the range is no longer suitable, it follows a failure path and releases the prepared folio rather than installing a huge PMD into a stale VMA. This is a source-based race model, not a captured trace. If allocation itself fails, the earlier SCAN_SUCCEED check directs control to cleanup before the later revalidation.

A second failure concerns policy: if the allowed nodes contain no eligible large block and the relevant allocation cannot fall back, compaction or reclaim may be unable to satisfy the request. The kernel can fall back to smaller pages or report a failure according to the specific interface and path. Neither the policy example nor the collapse example justifies changing a shared host's global THP mode or huge-page pool.

Practice#

Source-only route: draw the two-node first-touch timeline and a second row with node-0-only binding. Record the assumptions about allowed nodes and fallback. Calculate 512 base pages per 2 MiB under the stated x86-64 example, and trace the collapse allocation/revalidation failure. Label all of it as model evidence, not an observed placement or promotion.

Bounded own-process host route: this Python program allocates a private anonymous 2 MiB mapping, requests MADV_HUGEPAGE only for that mapping if the Python interface exposes it, touches one byte per host base page, and reads its own mapping's proc metadata. It never changes global THP policy or allocates a hugetlb pool. Python and the kernel may reject or ignore the hint; that is a valid outcome.

python3 - <<'PY'
import ctypes, mmap, os
size = 2 * 1024 * 1024
page = os.sysconf('SC_PAGE_SIZE')
region = mmap.mmap(-1, size, flags=mmap.MAP_PRIVATE | mmap.MAP_ANONYMOUS,
                   prot=mmap.PROT_READ | mmap.PROT_WRITE)
try:
    address = ctypes.addressof(ctypes.c_char.from_buffer(region))
    if hasattr(region, 'madvise') and hasattr(mmap, 'MADV_HUGEPAGE'):
        try:
            region.madvise(mmap.MADV_HUGEPAGE)
            print('MADV_HUGEPAGE hint accepted')
        except OSError as error:
            print('hint rejected:', error)
    else:
        print('madvise hint unavailable')
    for offset in range(0, size, page):
        region[offset] = 1
    found = False
    mapping_start = None
    try:
        with open('/proc/self/smaps', encoding='ascii') as f:
            selected = False
            for line in f:
                first = line.split(None, 1)[0] if line.strip() else ''
                if '-' in first and all(c in '0123456789abcdef-' for c in first):
                    lo, hi = (int(x, 16) for x in first.split('-'))
                    selected = lo <= address < hi
                    if selected:
                        print('own VMA:', line.strip())
                        found = True
                        mapping_start = lo
                elif selected and line.startswith(('KernelPageSize:', 'MMUPageSize:', 'AnonHugePages:')):
                    print(line.strip())
    except OSError as error:
        print('smaps unavailable:', error)
    if not found:
        print('mapping entry not visible')
    try:
        with open('/proc/self/numa_maps', encoding='ascii') as f:
            matches = [line.strip() for line in f
                       if mapping_start is not None and int(line.split()[0], 16) == mapping_start]
            print('own numa_maps:', matches[0] if matches else 'entry not visible')
    except OSError as error:
        print('numa_maps unavailable:', error)
finally:
    region.close()
PY
cat /sys/kernel/mm/transparent_hugepage/enabled 2>/dev/null || printf 'THP mode not exposed\n'

The mapping is deliberately small, but allocation can still fail under a tight limit; record an error rather than retrying with more memory. A hint being accepted means the call succeeded, not that collapse occurred. Alignment, enabled sizes, memory pressure and policy affect the result. smaps or numa_maps can be absent or restricted, and these views may change while read. Addresses, node layout and system mode can be sensitive; review before sharing. KernelPageSize, MMUPageSize and AnonHugePages do not universally reveal every underlying large-folio or TLB detail. The finally block closes the mapping; no files or persistent settings are created.

Deliver the source-only node and coverage tables and the labeled observation or visibility failure. As a counterexample, an accepted MADV_HUGEPAGE hint with zero AnonHugePages is possible and does not imply that the hint modified a global setting.

Knowledge check#

1. Why can a task on CPU 1 allocate memory from node 0?
2. What does get_vma_policy do if the VMA-specific lookup returns no policy?
3. Why does collapse_huge_page revalidate the VMA after allocation?
4. Why does a 2 MiB PMD mapping not imply a 512-fold application speedup?
5. How do hugetlb pool accounting and THP promotion differ?

Answers#

1. CPU affinity and memory policy are separate. A bind policy, cpuset constraint, fallback rule or prior allocation can place memory on another node.
2. The cited v6.12 code falls back to the current task's policy; subsequent allocation still depends on allowed nodes, flags and availability.
3. The function drops the mmap lock for an allocation that may take time. Another thread can unmap or change the candidate, so the old VMA/PMD observation must be checked again before installation.
4. TLB organization, workload locality, cache behavior, allocation and split costs, and NUMA placement all affect performance. The arithmetic describes coverage, not a speedup.
5. Hugetlb uses explicit pool/reservation and accounting rules, whereas THP is an opportunistic mechanism for eligible mappings. Their allocation and failure behavior are not interchangeable.

Checkpoint#

Deliver the two-node policy table, 512-page calculation and collapse race, plus labeled own-process observations if run. Pass when you distinguish policy from affinity, explain a revalidation/failure cleanup and state why an accepted hint is not proof of a huge mapping. If uncertain, retrace get_vma_policy, vma_alloc_folio_noprof and collapse_huge_page, then revise the model. This is not certification.

Sources#

• v6.12 NUMA policy implementation.
• v6.12 huge-page collapse and huge-memory paths.
• v6.12 hugetlb implementation.
• NUMA memory policy.
• Transparent huge pages.
• Hugetlb pages.