BPF tracing, BTF, CO-RE, and observability limits

Before you start#

Review BPF verification, ftrace and perf, and RCU and object lifetime. This lesson uses upstream Linux v6.12 as a historical source baseline. Program types, attachment points, BTF and permissions depend on configuration and the actual running kernel. The worked event sequence is synthetic. The observation lab does not load or attach BPF programs or print target arguments.

Learning goals#

Compare BPF tracing attachment points, explain the scope of BTF and CO-RE, trace a ring-buffer reservation and release, and identify correlation and loss errors in an entry/exit latency measurement.

Source walkthrough#

A BPF tracing program observes an event or function through an attachment mechanism and runs under that mechanism's context and verifier constraints. An fentry or fexit program can attach through a BPF trampoline to supported function entry or exit points, using available type information and attachment rules. A kprobe attaches to a supported kernel instruction location or symbol; a kretprobe observes a supported return path subject to its own limitations. A tracepoint attaches to an explicitly defined kernel event; a raw tracepoint exposes a different context interface from a formatted trace event. An uprobe observes a supported userspace executable location and depends on the actual binary and mapping. These mechanisms answer different questions and do not have identical coverage, argument formats, return behavior or overhead.

A particular function may be inlined, renamed, unavailable to a trampoline, blacklisted for kprobes, or absent in a different build. A return probe can miss instances under pressure or fail to observe an abnormal path. A tracepoint can be present but represent a broader or narrower event than the investigation needs. Kernel BTF, CONFIG_BPF, architecture support, privileges and active LSM policy determine which attachment is possible. The presence of a function name in a header alone does not prove an attachable site exists on the running kernel.

In kernel/trace/bpf_trace.c, the following exact v6.12 SPDX GPL-2.0 excerpt is from the tracing helper bpf_get_current_task_btf; preceding code and the function prototype metadata are omitted:

BPF_CALL_0(bpf_get_current_task_btf)
{
	return (unsigned long) current;
}

The helper takes no explicit BPF arguments and returns the current task pointer in the internal helper calling convention. Its adjacent prototype identifies the result as a trusted pointer to a BTF-described task type. That return type is a verifier contract, not a promise that arbitrary task fields can be dereferenced, stored forever, or used in every program type. The current task is the task executing at the hook; it may be a worker rather than the application whose high-level operation ultimately caused the work. The tracing helper and trampoline implementation are source entry points, not proof that a particular attachment ran.

BTF and CO-RE are compatibility tools#

BPF Type Format describes types and related metadata for a build. A type ID is meaningful within the relevant BTF object and build; the same numeric ID is not a universal stable identifier across kernel versions. BTF can describe structure members, function prototypes and other types, but a field's existence and offset do not establish that its meaning, lifetime, locking rules or initialization condition stayed the same. Build options can remove a type or change a function's availability. Kernel source headers from one release are not evidence of the layout of a distribution kernel with backports.

Compile Once, Run Everywhere (CO-RE) uses relocation information in a BPF object and target BTF, usually processed by a compatible userspace loader such as libbpf, to adapt supported type and field references. A relocation can adjust a field offset when the target type matches the expected shape. It cannot make a removed semantic contract reappear, invent an unsupported helper, guarantee a function is attachable, or prove a changed field still means the same thing. Kfuncs and helpers have version, program-type, context, privilege and lifetime constraints; even a relocated program can fail verification or attachment. Record the exact loader, compiler, object and kernel builds. libbpf and bpftool are separate projects or components with their own version histories; pin a release or commit for any reproducible actual experiment.

A sensible load sequence obtains or opens the BPF object, resolves relocations against the target's BTF, loads programs and maps through the kernel verifier, then attaches the selected program using an appropriate link or attachment API. A link fd can own an attachment; closing it can detach when no pin or other reference retains it, depending on the attachment. Pins can extend object lifetime beyond the loader process. Failure at relocation, verifier load, link creation or later event delivery are different outcomes, each requiring its own cleanup. A zero error from object loading is not proof that the event fired or the consumer read every record.

Ring-buffer reservation and release#

The BPF ring buffer is a bounded shared communication mechanism between BPF producers and a userspace consumer. A producer reserves a record, fills it, and submits or discards it. Reservation can fail when the requested record is too large, there is insufficient space, or a context-specific locking condition prevents reservation. In particular, the v6.12 implementation uses a trylock in NMI context and can return NULL even when a simplistic free-space calculation appears favorable. The program must test the returned pointer; it cannot write through NULL or assume every event has a record.

In kernel/bpf/ringbuf.c, __bpf_ringbuf_reserve contains this exact v6.12 excerpt; preceding position calculations and following record initialization are omitted. This file has no file-level SPDX tag or separate license notice in the inspected v6.12 source; consult the kernel licensing guidance and source provenance rather than assigning it a more specific identifier without evidence:

	if (new_prod_pos - cons_pos > rb->mask ||
	    new_prod_pos - pend_pos > rb->mask) {
		raw_spin_unlock_irqrestore(&rb->spinlock, flags);
		return NULL;
	}

The first comparison checks producer advancement relative to the consumer; the second checks the span from the oldest pending record. If either exceeds the ring's allowed range, the code releases the lock with saved interrupt state restored and returns NULL. On success, the following code initializes a busy record and publishes the producer position with release ordering. This lock and publication protocol protect ring metadata; they do not automatically count dropped events for the application. A program that cares about loss needs a separate suitable counter and must handle failure to update or read that counter as well.

bpf_ringbuf_submit calls the commit helper with discard false; bpf_ringbuf_discard uses discard true. Their verifier prototypes mark the record argument with OBJ_RELEASE: the reserved reference is consumed by the operation. A producer must not submit twice, use the released record afterward, or leave a reservation outstanding on a control-flow path. The commit clears the busy state and applies wakeup policy. A consumer observes committed records under the ring's ordering contract; a successfully reserved record is not yet a completed event until it is submitted. Ringbuf reserve failure, explicit discard and userspace consumer lag are different observations.

Both calls reach one shared helper, bpf_ringbuf_commit. In kernel/bpf/ringbuf.c, v6.12, the inspected file has no file-level SPDX tag; its GPL-2.0 status comes from the kernel's COPYING notice. The excerpt is the opening of the function; the wakeup branch and the rest of its body are omitted:

static void bpf_ringbuf_commit(void *sample, u64 flags, bool discard)
{
	unsigned long rec_pos, cons_pos;
	struct bpf_ringbuf_hdr *hdr;
	struct bpf_ringbuf *rb;
	u32 new_len;

	hdr = sample - BPF_RINGBUF_HDR_SZ;
	rb = bpf_ringbuf_restore_from_rec(hdr);
	new_len = hdr->len ^ BPF_RINGBUF_BUSY_BIT;
	if (discard)
		new_len |= BPF_RINGBUF_DISCARD_BIT;

	/* update record header with correct final size prefix */
	xchg(&hdr->len, new_len);

The discard argument only ORs a discard bit into the header length, and the busy bit is XORed in the same step. The excerpt stops before the wakeup branch, so it does not show how a consumer treats discarded records or how a failed reservation is counted. Submit and discard therefore share one release path, but they are different outcomes for the consumer.

Correlation with overlapping work#

Suppose an investigator wants the latency of a function from entry to exit. A common design records a timestamp at entry, looks it up at exit, computes a delta, emits a record and deletes the state. A process ID shared by all threads is insufficient when two threads in one process enter concurrently. A thread ID is better for a strictly non-reentrant per-thread operation, but can still collide with recursive or nested entries on the same thread. A robust key may need task identity plus a nesting depth, an operation-specific object or request identifier, or a verified non-reentrancy assumption. A map entry must be deleted on completion and bounded or expired on missing exits; task-ID reuse can otherwise pair an old timestamp with a different task.

Use this synthetic sequence in microseconds: thread A enters operation X at 100; thread B enters Y at 110; A exits at 150; B exits at 210. Correct durations are 50 and 100 microseconds. If the map is keyed only by shared process ID, B can overwrite A's start: A would be paired with 110 and appear to take 40 microseconds, while B may find the entry deleted. Keying by thread ID distinguishes A and B in this example, but recursive entry on A would still need a depth or request identity. Migration between CPUs does not change a thread's identity; using CPU number as the only key can pair unrelated work, and per-CPU state must account for migration.

Now suppose A's 50-microsecond result is submitted but B's 100-microsecond event fails ring-buffer reservation. The consumer sees only 50 microseconds and may incorrectly report a mean of 50 for two operations. The actual mean of these two synthetic durations is 75 microseconds. A drop counter can show a missing record but cannot recover B's lost latency or prove that drops are independent of duration. If slower operations coincide with bursts, loss can systematically bias a histogram. A stronger design records attempts, successful submissions and reservation failures, uses a bounded correlation map with cleanup, and compares its counts with independent evidence.

If a task exits or the instrumented function never reaches the observed return path, an entry can remain in the map. Timeouts or explicit lifecycle cleanup can remove stale entries, but choosing a timeout too short can discard legitimately slow operations. A kernel address used as a key can be reused after an object is freed, so its lifetime and generation matter. On map insertion failure, a corresponding exit should not be interpreted as a zero-latency event. Record missing-start, missing-end, insertion and output failures separately rather than folding them into a successful latency count.

Maps, clocks and interpretation#

Per-CPU maps reduce contention by giving CPUs separate values for supported map types, but userspace must aggregate all relevant CPU slots correctly. An event can enter on one CPU and exit on another, so per-CPU entry state is not automatically suitable for task-based correlation. Histograms aggregate observations into buckets and lose individual event detail; bucket boundaries, units, overflow and update failures must be documented. Atomic operations or appropriate synchronization are needed where multiple contexts update shared values; a BPF verifier's acceptance does not establish that the chosen aggregation matches the intended metric.

A BPF ring buffer and a perf-event buffer are different transport mechanisms with different layouts and loss reporting. Both can drop or fail to deliver records when consumers cannot keep up, and their ordering and per-CPU properties must be understood separately. Sampling reduces volume but can bias results if selection depends on timing or workload phase. A tracepoint may already be filtered by attachment scope, namespace or authorization. Record the denominator: attempts, filtered events, map failures, output failures and successfully consumed events do not necessarily match.

A monotonic timestamp supports intervals under its clock's semantics, but the exact helper and clock must be identified. Some monotonic clocks exclude suspend while boot-time clocks include it; a latency crossing suspend can differ accordingly. Cross-CPU comparisons require the selected clock's guarantees, and converting nanoseconds to microseconds can lose precision. Wall-clock timestamps can jump when time is adjusted. State the clock and interval, rather than mixing two time domains because both values look like integers.

Practice#

Read-only BTF and tooling inventory

On your own authorized Linux host or a disposable guest, inspect existing metadata without loading, attaching or pinning a program:

uname -r
if [ -r /sys/kernel/btf/vmlinux ]; then ls -l /sys/kernel/btf/vmlinux; else echo 'vmlinux BTF unavailable or restricted'; fi
if command -v bpftool >/dev/null 2>&1; then bpftool version; else echo 'bpftool unavailable'; fi
if [ -r /proc/sys/kernel/unprivileged_bpf_disabled ]; then cat /proc/sys/kernel/unprivileged_bpf_disabled; else echo 'BPF policy indicator unavailable'; fi

A readable BTF file does not prove an intended function exists or that CO-RE relocations, verification and attachment will succeed. A tool version does not identify the running kernel's exact type semantics. Missing BTF, bpftool or permission is a legitimate result; record the actual kernel and tool versions without changing sysctls or using sudo. Do not list other users' program details unless separately authorized, and do not print captured arguments, payloads or secrets.

Source-only alternative: reproduce the A/B timeline, first with a shared PID key and then with distinct thread IDs. Calculate 50 and 100 microseconds and the biased 50-microsecond observed mean after B's synthetic output failure. Add a table for ring reservation success, NULL and submit/discard ownership. Deliver the table or actual inventory, stating that no BPF program was loaded or attached and that the two-event sequence was invented teaching data.

Optional disposable-guest attachment

Only in a dedicated disposable x86-64 QEMU guest you control, with upstream v6.12 booted, a mounted tracefs exposing syscalls:sys_enter_getpid, BPF and tracepoint support enabled, and bpftrace v0.21.3 installed from its separately versioned project, you may run the following. It attaches only in that guest and filters on a process created for this experiment. The guest needs the capabilities and policy required for this attachment; if unavailable or denied, stop rather than changing a sysctl or disabling security. Do not run it on the host or against another user's process.

python3 -c 'import os,time; time.sleep(3); [os.getpid() for _ in range(100)]; time.sleep(2)' &
guest_lab_pid=$!
trap 'kill "$guest_lab_pid" 2>/dev/null || true; wait "$guest_lab_pid" 2>/dev/null || true' EXIT
sudo timeout 8s bpftrace -e 'tracepoint:syscalls:sys_enter_getpid /pid == $1/ { @calls = count(); }' "$guest_lab_pid"
wait "$guest_lab_pid" 2>/dev/null || true
trap - EXIT

The program reports an aggregate count only; it captures no syscall arguments. A result near 100 is possible if attachment completes before the child starts its calls, but startup delays, policy, event availability and tracing overhead can change or prevent that result. A missing tracepoint, incompatible bpftrace, denied attachment or timeout is an explicit result; do not call it a successful capture. timeout terminates bpftrace, whose process exit closes its unpinned attachment; the trap terminates and reaps the controlled child if interrupted. Confirm the bpftrace process has exited, then shut down and discard the guest. No guest attachment was run for this chapter. Pin the bpftrace release and exact guest image in any real report; the example does not promise compatibility with other versions.

Knowledge check#

1. Why can CO-RE relocate a field correctly while the program still measures the wrong thing?
2. What must a producer do when ring-buffer reservation returns NULL, and what happens to a successful reservation?
3. Why is a process ID alone insufficient for the interleaved two-thread example?
4. How can a dropped slow event bias the observed mean?
5. Why can per-CPU entry state fail when a task migrates before exit?

Answers#

1. CO-RE addresses supported type-layout differences. Field meaning, function availability, lifetime, helper rules and application intent can still differ.
2. It must not write through the null pointer and should account for the failed output where possible. A successful reservation must be submitted or discarded exactly once under the verifier and API contract.
3. Both threads share the process ID and can overwrite or delete one another's timestamp. Thread identity distinguishes this example, but nested calls still need additional correlation.
4. The consumer sees only the retained 50-microsecond event although the two-event mean is 75. A drop counter reports loss but cannot reconstruct the missing duration.
5. Entry and exit can execute on different CPUs, so the exit's per-CPU slot may not contain the entry state. The key and storage must match the operation's lifetime and migration behavior.

Checkpoint#

Submit the synthetic correlation timeline, loss calculation and reservation ownership table. Ready means you account for nested calls, cleanup, failed output and clock semantics while distinguishing BTF layout from semantic compatibility. Revisit the source branches if a NULL reservation or stale entry is treated as a successful event. This is not certification.

Sources#

• BPF tracing helpers v6.12
• BPF ring buffer v6.12
• BPF trampoline v6.12
• BTF documentation
• BPF ring buffer documentation