Livepatch, CPU/memory hotplug, and RAS

Before you start#

Review kernel modules, RCU and object lifetime, SMP and CPU placement, physical memory, and crash capture. Linux v6.12 is the historical source baseline. Support depends on architecture, configuration, hardware and the particular patch or device. No livepatch, hotplug, memory poisoning or crash operation is run here.

Learning goals#

Trace livepatch enablement and consistency transition, distinguish CPU and memory offline rollback, and interpret reliability/availability/serviceability (RAS) reports without assuming that a corrected event is harmless or an uncorrectable event always has the same outcome.

Source walkthrough#

Livepatch can redirect selected kernel functions to replacement implementations while a supported kernel is running. A klp_patch describes a patch module and its objects; a klp_object describes the kernel or module being patched; function entries identify old and new implementations and associated metadata. Symbol matching, module identity, relocations and the exact build matter. A livepatch is not an arbitrary binary rewrite, a guarantee that every bug can be fixed live, or a replacement for supported updates and reboot planning.

The consistency model coordinates task transitions so that a task does not execute an incompatible mixture of old and new functions in an unsafe state. Depending on architecture and configuration, reliable stack traces help determine when tasks can switch. Sleeping tasks and long-running kernel paths can delay completion. RCU and ftrace-related lifetime rules protect selected internal structures; they do not make any two arbitrary function versions semantically compatible. Forcing a transition can bypass safety checks and leave tasks executing incompatible states, so it is an exceptional, risky operation, not routine cleanup.

Livepatch enablement and unwind#

In kernel/livepatch/core.c, klp_enable_patch validates the patch, module and function lists, checks that livepatch is initialized, takes klp_mutex, checks compatibility and obtains a module reference. It initializes patch state and calls __klp_enable_patch. The following exact v6.12 GPL-2.0-or-later excerpt is from __klp_enable_patch; surrounding validation and callback handling are omitted:

	klp_start_transition();
	patch->enabled = true;
	klp_try_complete_transition();

The first call starts the consistency transition, the second records the patch as enabled, and the third tries to complete the transition. A successful enable call does not mean every task necessarily switched instantaneously. Earlier, klp_init_transition prepares transition state and a write barrier orders transition flags before the patched function stack becomes visible to the ftrace handler. Loaded objects receive pre-patch callbacks and function patching; unloaded modules can require later matching when they appear. A failing pre-patch callback or klp_patch_object takes the error path and calls klp_cancel_transition instead of claiming a completed enable.

The outer function's error label starts patch cleanup while the mutex is held, releases the mutex and finishes cleanup. The module reference and patch object lifetime must survive any transition and callback using them. A failed module-reference acquisition returns -ENODEV; an incompatible patch returns -EINVAL; an already active transition can return -EBUSY in the inner function. These cases are not interchangeable with a task that remains temporarily in the old state after successful enable. See transition.c for per-task transition and completion mechanisms.

In kernel/livepatch/transition.c, v6.12, SPDX GPL-2.0-or-later, klp_cancel_transition is what the err: label of __klp_enable_patch calls; the complete function is:

void klp_cancel_transition(void)
{
	if (WARN_ON_ONCE(klp_target_state != KLP_TRANSITION_PATCHED))
		return;

	pr_debug("'%s': canceling patching transition, going to unpatch\n",
		 klp_transition_patch->mod->name);

	klp_target_state = KLP_TRANSITION_UNPATCHED;
	klp_complete_transition();
}

It requires the transition target to be patched, changes the target to unpatched and calls the completion path. It does not show how far each task has actually switched back; that per-task logic is elsewhere in the same file.

A hypothetical patch changes a function used by a long-sleeping task. Initialization and callbacks succeed, so the patch is enabled, but that task's stack is not yet known to be in a safe state to switch. Other tasks may progress while the transition remains incomplete. Waking or naturally returning the task can allow safe progress under the actual consistency rules; blindly forcing the transition can violate the old/new state assumptions. A livepatch author must review data structures, calling conventions, module dependencies and callbacks, not only replace the text of one function. This example is conceptual and no patch was loaded.

CPU hotplug and callback rollback#

CPU hotplug moves a CPU through a state machine with registered startup and teardown callbacks. The per-CPU hotplug state stores current and target states, callback results and rollback information. Offlining interacts with scheduler placement, per-CPU work, interrupts, timers, RCU and architecture-specific shutdown. A task or interrupt cannot simply be assumed to migrate successfully because another CPU exists; affinity, housekeeping requirements and callback support matter. Locks and hotplug synchronization prevent conflicting state transitions and protect users of CPU topology.

In kernel/cpu.c, cpu_down_maps_locked explicitly returns -EOPNOTSUPP if the platform does not support CPU offline and -EBUSY if hotplug is disabled or no suitable housekeeping CPU is available for the control work. Callback failure at an intermediate state can require rollback through the state machine. The precise callbacks and rollback ordering depend on registered states and architecture; a partially completed offline request is not a reason to assume the CPU is safely absent from all users.

In kernel/cpu.c, v6.12, the file header says the code is "licenced under the GPL" and the file has no SPDX tag. cpuhp_down_callbacks runs the teardown range and, on failure, resets the state and runs the startup range back; the complete function is:

static int cpuhp_down_callbacks(unsigned int cpu, struct cpuhp_cpu_state *st,
				enum cpuhp_state target)
{
	enum cpuhp_state prev_state = st->state;
	int ret = 0;

	ret = cpuhp_invoke_callback_range(false, cpu, st, target);
	if (ret) {
		pr_debug("CPU DOWN failed (%d) CPU %u state %s (%d)\n",
			 ret, cpu, cpuhp_get_step(st->state)->name,
			 st->state);

		cpuhp_reset_state(cpu, st, prev_state);

		if (st->state < prev_state)
			WARN_ON(cpuhp_invoke_callback_range(true, cpu, st,
							    prev_state));
	}

	return ret;
}

The rollback runs only when the state ended below prev_state; WARN_ON reports an unexpected rollback failure but does not repair it. The excerpt does not show what each callback undoes, so a driver's own cleanup still has to be verified separately.

Consider an illustrative CPU-down sequence where callbacks A and B succeeded, but callback C refuses a transition. The hotplug framework must run the applicable rollback callbacks for completed states and report the error, restoring a coherent state under its rules. A driver author must implement its own callback's resource ownership and reversal correctly; the framework cannot infer how to restore a device's private work. Checking a final online mask alone is useful but does not prove every driver handled a failed transition without leaks.

Memory hotplug and a pinned-page failure#

Adding memory creates or exposes memory resources; onlining makes eligible memory available to the allocator, while offlining attempts to remove it from use before physical removal. A memory block is governed by section, zone and platform constraints. The kernel isolates page ranges, migrates movable pages and handles notifications; long-term pins, unmovable allocations, holes, zone boundaries, signals and device-specific memory can prevent success. ZONE_DEVICE describes specialized device-associated memory with different rules, not ordinary hotpluggable system RAM in every respect.

In mm/memory_hotplug.c, offline_pages validates alignment and checks for holes and a compatible zone, disables per-CPU page lists and LRU caching, isolates the range, and notifies interested subsystems before scanning and migrating movable pages. This exact v6.12 GPL-2.0-only excerpt is from the later scan result; surrounding loop and cleanup are omitted:

		if (ret != -ENOENT) {
			reason = "unmovable page";
			goto failed_removal_isolated;
		}

The scan's -ENOENT has a specific, local meaning in this loop: this scan found no further movable pages; it is not a general success code for memory offlining. A different return takes the unmovable-page failure path. The failed_removal_isolated label undoes page-range isolation and sends MEM_CANCEL_OFFLINE; subsequent cleanup re-enables LRU caching and per-CPU page lists before returning the error. Earlier failures enter at different labels according to what was acquired. The failure is a rollback of this attempted transition, not proof that every earlier migration was invisible or that a pinned page can always be identified from one log line.

A synthetic example: a memory block contains pages in use by several tasks, including one page that cannot be moved because of an active long-term pin. The offline attempt isolates the range and asks subsystems to prepare. Migration of movable pages can occur, but the unmovable page prevents the range from becoming offline. The failure path restores the isolation state and notifies cancellation; an administrator sees a failed operation, not a safely removed block. The owner of the pin must be identified and released under its real contract before a retry can be meaningful. This example does not assert that a particular pin always produces this exact return code; the source branch and actual diagnostics must be correlated.

RAS, reporting and recovery#

RAS means reliability, availability and serviceability. EDAC can report certain memory-controller and related hardware errors; x86 machine-check architecture (MCA), firmware-first APEI/GHES, PCIe Advanced Error Reporting (AER) and architecture-specific facilities report other classes of events. Their coverage and firmware coordination are platform-specific. A corrected error means some reporting layer indicates correction under its semantics; repeated corrected errors can still indicate degrading hardware. An uncorrectable error's effect depends on severity, affected data, containment and recovery policy. One counter is not a complete health diagnosis.

Persistent storage such as pstore/ramoops can preserve selected logs across a reboot when configured and functioning; a missing record may reflect lack of support, overwriting or a failure before persistence. A watchdog may reset a stuck system, but can also cause a reboot loop. Recovery planning includes replacing failed hardware, testing backups, preserving logs and verifying that a service returns to a known state. It is not enough to assert that a machine rebooted. Relevant entry sources include EDAC and pstore.

CPU rollback and the boundary of a patch#

The CPU hotplug source records per-CPU state, target, callback result and rollback state. In the teardown path, the framework invokes callbacks for the transition and, when an applicable callback fails, invokes the corresponding rollback path for states that were already traversed. The source in cpuhp_down_callbacks and _cpu_down distinguishes work handled by the target CPU's hotplug thread from later cleanup on the control side. The control path holds the CPU write lock during the state change and releases it before post-unplug cleanup, while a broader add/remove lock still protects the operation. This is a state-machine protocol, not a promise that a driver can omit its own cleanup.

For a synthetic callback sequence, suppose the target CPU successfully leaves states C and B but teardown at A returns an error. The applicable rollback invokes startup callbacks to restore traversed states before the error is reported. If a callback's inverse cannot restore the resource it changed, the final online mask alone does not establish correctness. A controller should capture the exact failing state, callback return and final topology, and then inspect the device-specific resource ownership. The source's platform-disabled and housekeeping checks can reject before this callback sequence, producing a different outcome from an intermediate callback failure.

Livepatch has a similarly important boundary between registration, redirection and safe transition. A klp_object can refer to the base kernel or a named module; function descriptors identify replacements, and relocation and symbol resolution must match the intended build. Pre- and post-patch callbacks may manage state around an object, but cannot make an incompatible data layout safe by declaration. A module that loads after patch registration can require later object handling. The patch module reference prevents premature unload while the framework needs it, and transition structures must remain valid while tasks and ftrace handlers can observe them.

A forced transition is especially dangerous when a task's stack still contains an old function that expects old shared state. Redirecting a later call to new code can create a mixed-version invariant that neither implementation supports. Reliable stack tracing and the per-task consistency model reduce that risk where available; a warning that the architecture lacks reliable support means completion can stall. A stalled transition is an operational event to investigate with the exact patch and tasks, not a reason to assume that force is harmless.

RAS evidence also needs a timeline. A rising corrected-memory-error count on one controller can motivate hardware diagnostics and a maintenance plan, but it does not identify an application crash without matching address, time and platform evidence. A fatal PCIe AER report may involve a device, link, upstream bridge or firmware handling; resetting an assigned device can affect other users. A pstore record can preserve a previous boot's message, but its retention capacity and reset path may omit earlier context. Combine independently identified reports with inventory and recovery tests before recommending a hardware replacement or attributing a kernel failure.

Practice#

Read-only inventory

On an authorized host or disposable guest, inspect existing state without writing any sysfs or firmware control:

uname -r
for f in /sys/devices/system/cpu/online /sys/devices/system/cpu/offline; do [ -r "$f" ] && printf '%s: %s\n' "$f" "$(cat "$f")"; done
for d in /sys/devices/system/memory/memory[0-9]*; do [ -r "$d/state" ] && printf '%s: %s\n' "${d##*/}" "$(cat "$d/state")"; done | head -n 12
for d in /sys/devices/system/edac /sys/fs/pstore /sys/kernel/livepatch; do [ -d "$d" ] && ls -ld "$d" || printf '%s unavailable\n' "$d"; done

Online CPU and memory-block states are snapshots. An absent EDAC, pstore or livepatch directory can reflect configuration, hardware, permissions or mount state; it does not prove the machine has no relevant errors or that hotplug is safe. Do not write online, memory state, livepatch controls, poison pages or trigger SysRq. Do not use sudo to force access.

Source-only alternative: trace the synthetic pinned-page case through isolation, notification, unmovable scan result and cancellation cleanup. Separately trace livepatch callback failure to klp_cancel_transition and the outer cleanup. Deliver those timelines or the read-only inventory, distinguishing an enabled patch from a completed per-task transition and marking the example unrun.

A future destructive hotplug or livepatch test requires a separately prepared disposable guest, supported kernel and hardware model, a clean snapshot, a dedicated controller, bounded test inputs and a recovery procedure to terminate and recreate the VM. This lesson does not supply or execute such a host operation.

Knowledge check#

1. Why does successful livepatch enablement not imply every task has switched immediately?
2. What happens when a pre-patch callback fails in the shown enable path?
3. Why can a pinned page prevent memory offlining?
4. What does the isolated-range failure path do before returning?
5. Does one corrected EDAC event prove a machine is healthy?

Answers#

1. The consistency model transitions tasks at safe points, and a task with an incompatible active stack can delay completion. Architecture and stack reliability matter.
2. The inner error path cancels the transition; the outer function performs its cleanup and returns the error. It must not be described as a successfully completed patch.
3. A long-term pin or other unmovable use can prevent migration or isolation of the required range. Actual eligibility and errors depend on the page and configuration.
4. It undoes isolation, sends cancellation notification and re-enables the disabled LRU and per-CPU page-list mechanisms through the subsequent labels.
5. No. It is a reported correction under a particular platform's semantics; trends, other events and independent diagnostics determine the health assessment.

Checkpoint#

Submit a livepatch enable/error timeline and a memory-offline failure/rollback timeline. Ready means module and patch references, per-task transition, pinned-page limitations and RAS reporting scope are distinguished, with no claim of an unrun hotplug test. Retry the source labels if any acquired state lacks a cleanup step. This is not certification.

Sources#

• Livepatch core v6.12
• Livepatch transition v6.12
• Memory hotplug v6.12
• CPU hotplug v6.12
• CPU hotplug documentation v6.12
• Livepatch documentation
• Memory hotplug documentation
• EDAC documentation
• APEI documentation
• PCIe AER documentation
• x86 machine-check documentation
• pstore documentation