Oops, panic, vmcore, kdump, and kernel debugging

Before you start#

Review interrupt and exception context, object lifetime, ELF and symbols, and trace evidence. The source baseline is upstream Linux v6.12. Chapter 08's QEMU image has not been booted in this workspace because of a GCC 16/C23 compatibility problem; no working guest or crash capture is assumed here. Source-only analysis remains a valid route.

Learning goals#

Distinguish warning, oops and panic; trace the conditional crash-kernel path; match a vmcore and stack trace to the correct build; and explain which observations are missing from a filtered or failed crash capture.

Source walkthrough#

The kernel's printk infrastructure stores records in a ring buffer. Records can include a log level, timestamp and message; console printing and readers can see records after the originating event, depending on buffering, console behavior and scheduling. Ratelimited messages may be suppressed. A timestamp's clock and per-CPU ordering matter: two adjacent messages do not automatically establish a causal chain, and a missing earlier line does not prove that no earlier event happened. The printk implementation is the source entry point; the actual log of a particular system is separate evidence.

A WARN reports a condition while execution may continue, subject to configuration such as panic-on-warn. BUG signals a serious kernel invariant failure, with architecture and configuration affecting the resulting trap. An oops is a report of a kernel fault or serious condition; the kernel can sometimes terminate the affected task and continue, but state may already be inconsistent. A panic is a fatal path that does not promise normal recovery. Taint flags record conditions such as certain warnings, external modules or proprietary modules; a taint is context for investigation, not proof that a particular module caused the fault.

Lockup, hung-task and RCU-stall reports each describe different progress failures. A watchdog can report a CPU that has not made expected scheduling or interrupt progress; hung-task detection concerns tasks blocked for too long; an RCU stall concerns delayed grace-period progress. Thresholds, detector configuration, virtualization and severe load affect interpretation. One warning can be a symptom of another problem, and the first visible message need not be the first causal event.

Panic and crash-kernel handoff#

In kernel/panic.c, panic disables local interrupts and preemption and uses panic_cpu to select the CPU that carries out the main panic path. Other CPUs may stop or be stopped; an NMI and a nested panic complicate the exact sequence. It formats and prints the panic message, gives kgdb an opportunity where configured, and conditionally attempts a crash-kernel handoff. This exact v6.12 GPL-2.0-only excerpt omits surrounding comments and later code:

	if (!_crash_kexec_post_notifiers)
		__crash_kexec(NULL);

The local Boolean chooses whether to try the crash path before panic notifiers. The call does not guarantee that a crash kernel exists, that it starts, or that a dump is saved. With the option set, the later path runs notifiers and dump-related work before trying __crash_kexec; the source warns that notifiers can make an already unstable kernel less reliable. Ordering is therefore a trade-off between collecting more information and reaching the capture kernel reliably.

The same function elects one CPU to run the main panic path. In kernel/panic.c, v6.12, SPDX GPL-2.0-only (the longer comment above the exchange is omitted):

	old_cpu = PANIC_CPU_INVALID;
	this_cpu = raw_smp_processor_id();

	/* atomic_try_cmpxchg updates old_cpu on failure */
	if (atomic_try_cmpxchg(&panic_cpu, &old_cpu, this_cpu)) {

The branch wins only for the first CPU whose exchange succeeds. The source's else if (old_cpu != this_cpu) branch, which stops the other CPU, is outside this excerpt. The excerpt does not show whether the other CPUs actually stopped, nor what happens to a crash kernel if a second CPU panics during the attempt.

The second stage is __crash_kexec. This exact v6.12 GPL-2.0-only excerpt, copyright Eric Biederman and contributors, omits register setup and later unlock:

	if (kexec_trylock()) {
		if (kexec_crash_image) {

The trylock avoids racing another CPU's kexec image replacement and avoids blocking indefinitely on an unavailable lock in a damaged system. Only if the lock is acquired and a crash image is loaded does the inner path prepare registers, save vmcore information, shut down architecture-specific crash state and invoke the machine kexec operation. If either condition is false, this function can return without transferring control. The code is not proof that capture storage is available or that all other CPUs stopped cleanly.

Capture, dump and missing memory#

A typical kdump setup reserves memory for a crash kernel at boot with an appropriate crashkernel configuration, loads a compatible capture kernel and initramfs into the reserved area, and configures where its userspace will save evidence. On panic, the crashed kernel can transfer execution to that kernel. The capture environment reads the crashed memory through the vmcore interface, commonly represented as an ELF core file, and writes it to selected storage. Exact reservation syntax, architecture, firmware, IOMMU, memory availability, kexec permissions and distribution integration vary. Merely having a crashkernel argument does not prove the reservation succeeded or that a crash image is loaded.

Dump tools can filter or omit page categories to reduce size. Missing pages, excluded user data, unavailable devices, incomplete writes and capture-kernel failures limit analysis. A vmcore can contain credentials, application data and memory from other users; store and share it only under an appropriate authorization and retention policy. An ELF note or a readable header does not establish that all memory ranges are present or that all CPU registers were captured. The kdump documentation describes the configuration and capture stages.

Consider a hypothetical host with reserved crash memory but no loaded crash image. Panic reaches the early __crash_kexec attempt, obtains the lock, finds kexec_crash_image absent and returns to panic processing. No vmcore follows from that attempt. If a crash image exists but capture storage fails, the handoff may have occurred without a usable saved dump. If a dump is filtered, a debugger may report a missing page even when the original kernel had mapped it. These are source-based conditional cases, not observations from a machine in this workspace.

Symbols, build identity and KASLR#

Symbolization requires the exact kernel build's uncompressed vmlinux with appropriate debug information, its configuration and build identity. Matching a release string alone is insufficient: distribution backports, compiler version, LTO, configuration and local patches can change layout. Build IDs where present help match artifacts, but they do not recover missing debug information. Load matching module binaries and symbols for module addresses, and account for each module's runtime load address. A symbol from a similar build can produce a plausible but wrong source line.

KASLR can relocate the running kernel relative to link-time addresses. Crash tools and vmcore metadata can supply relocation information; live debugging requires the actual runtime offsets and appropriate permission. Do not simply subtract a guessed constant from every address, especially across modules. scripts/faddr2line can resolve a function-plus-offset when the matching binary and suitable debug data exist; addr2line resolves addresses in an ELF image under its own assumptions. Tool versions and architecture support matter.

GDB can inspect a matching vmlinux, symbols and a supported remote target or core-file workflow. The kernel's GDB scripts add helpers when configured and loaded, but they do not make a damaged stack reliable. Frame pointers, ORC or other unwind data, architecture, compiler and actual stack contents affect backtraces. An exception or interrupt can switch stacks; memory corruption can overwrite return addresses. A line in an unwind is evidence to test against registers, disassembly and object lifetime, not an unquestionable complete call chain.

A synthetic oops investigation#

The following is invented teaching data, not an observed log. The names are hypothetical and are not claimed as Linux v6.12 symbols:

[  42.000] BUG: unable to handle kernel NULL pointer dereference
[  42.001] CPU: 2 PID: 410 Comm: demo-worker
[  42.002] RIP: demo_submit+0x20/0x80 [demo_module]
[  42.003] Call Trace: demo_worker -> process_one_work

The message suggests a kernel-mode access involving a null address, but the instruction and registers are needed to establish which pointer was null. CPU 2 and PID 410 identify the reported execution context at that point, not necessarily the task that originally corrupted the object. The module label means a matching module binary and its runtime load information are needed to resolve demo_submit+0x20; the invented offset cannot be used against upstream source. process_one_work suggests a workqueue context if the trace and symbols are valid, but does not establish which producer queued the work or whether the work item's lifetime was correct.

Competing hypotheses include a missing error check, a use-after-free whose memory happened to contain zero, a teardown race and a corrupt stack. The first test is to inspect the faulting instruction, relevant register values and exact matching source. A null pointer dereference at a line does not by itself explain why the pointer became null. A warning earlier in the ring buffer could be relevant, unrelated or already truncated. Preserve the full log, taint flags, kernel build and module list before altering the system; never manufacture a precise source line from this synthetic text.

Debugger and capture limitations#

A vmcore contains a snapshot after a failure and possible memory filtering, not an execution recording. A lock seen held in the dump might be held because another CPU stopped during panic or because it was genuinely involved in the original deadlock. A pointer can be stale and allocator metadata can already be damaged. Compare task state, lock context, object references and the faulting instruction; keep hypotheses separate from facts. If a frame cannot be unwound across a corrupt stack or missing page, mark the remaining call chain unknown.

For a live disposable x86-64 QEMU guest, a debugger stub must bind to loopback explicitly, for example -gdb tcp:127.0.0.1:1234 -S. -S stops the guest at startup until the debugger continues it. This is only a QEMU option example: it is not a complete boot command and no guest was started. Use a matching vmlinux, a dedicated disposable guest and a protected local environment. Remote debugger access can control guest execution; do not expose the stub on all interfaces. Rebuild the chapter 08 lab with a verified compatible toolchain before assuming an image exists.

A crash test must be deliberately confined to a throwaway guest with a known disk image, reserved crash kernel, capture destination and a way to terminate and recreate the VM. Save a clean image copy, record the selected VM and recovery steps, and verify that the capture output stays within the guest or a dedicated test directory. Do not trigger host SysRq, host kexec, host panic or arbitrary crash reproducers. This lesson does not supply or run a destructive crash trigger; planning a recovery path is not evidence that kdump works.

From a register to a defensible explanation#

A concrete investigation begins with the faulting instruction, not the nearest function name. On x86-64, a report can include RIP, a fault address, register values and an error code. The exact meaning of the error code depends on the exception; compare it with architecture documentation and the decoded instruction. If the instruction reads a field at an offset from a register containing zero, that supports a null base-pointer hypothesis. It does not identify why that register is zero. If it instead uses a nonzero pointer into freed memory, a null-pointer headline from another message would be misleading. Registers can be unavailable or partially damaged, and a synthetic example cannot establish the state of an actual CPU.

Next identify the owner of the object used by that instruction. Was it obtained under a lock, reference or RCU read-side section? Could a work item or interrupt access it after teardown? Match source and machine code because optimization can move operations, inline functions or merge paths. A function-plus-offset points inside the generated binary, not necessarily to a unique source statement. A matching debug file can improve symbolization, but inlined frames and optimized-out variables remain limits. A debugger's inability to read a page could mean the page was filtered from the dump, corrupted or genuinely absent; those possibilities need separate evidence.

A crash capture plan should be tested before it is needed. In an isolated guest, verify that a reserved region and loaded capture kernel are visible by the guest's supported interfaces, that the capture initramfs can access the intended dedicated destination, and that the resulting vmcore is readable with matching symbols. Only then can a later destructive crash exercise test the whole chain. Failure at reservation, loading, handoff or writing produces different evidence. A successful boot of the capture kernel alone does not prove that its output contains every page or that the original fault is diagnosable.

Practice#

Read-only log and ELF inspection

On your authorized host or disposable guest, query the existing log only if access is already permitted:

uname -a
if command -v dmesg >/dev/null 2>&1; then dmesg --color=never 2>/dev/null | tail -n 20 || true; else echo 'dmesg unavailable'; fi
if command -v readelf >/dev/null 2>&1; then readelf --version | head -n 1; else echo 'readelf unavailable'; fi

A restricted or empty log is a valid result; the pipeline can hide the reason, so if needed run dmesg --color=never separately without sudo and record its error. Do not share log lines containing secrets or other users' data. If you already possess a matching, authorized kernel ELF, readelf -h -n /path/to/vmlinux reads its ELF header and notes; substitute the actual file. A missing build ID or missing debug sections limits matching, and readelf does not prove the file matches the running kernel. This is read-only inspection, not crash testing.

Source-only alternative: use the invented oops and the two panic/crash snippets. Make a table of known text, hypotheses and missing evidence; trace both “no loaded image” and “loaded image but no saved vmcore” outcomes. Deliver that table or an authorized ELF identity record, and explicitly mark the synthetic trace as not measured.

Knowledge check#

1. Why can a WARN, an oops and a panic lead to different outcomes?
2. What happens in __crash_kexec if the lock cannot be acquired or no image is loaded?
3. Why is a matching release string insufficient for source-line symbolization?
4. What can a filtered vmcore not establish about an omitted page?
5. Why does demo_submit in the supplied oops not identify a real upstream function?

Answers#

1. They describe different conditions and paths; policy such as panic-on-warn can change an outcome, and an oops can leave the system unreliable even if execution continues.
2. The inner handoff is not performed on that path. The function can return without starting a crash kernel; a panic path may continue its remaining steps.
3. Configuration, compiler, patches, modules and relocation can change addresses and layout. The exact binary, debug information, runtime addresses and build identity matter.
4. It cannot establish the page's contents from an omitted range; absence from the dump is not proof that the page was absent in the original machine.
5. It is explicitly invented teaching data. A real source mapping requires the exact module binary, runtime address and matching symbols.

Checkpoint#

Submit the synthetic evidence table and crash-handoff branches, including at least two plausible causes and the matching-artifact requirement. Ready means oops is not equated with panic, capture failure is distinguished from a valid vmcore, and unknown stack or memory is left unknown. Revisit the source and retry if a guessed source line is presented as fact. This is not certification.

Sources#

• Panic implementation v6.12
• Crash handoff v6.12
• printk implementation v6.12
• kdump documentation
• GDB kernel debugging