ftrace, tracepoints, perf, and measurement semantics

Before you start#

Review interrupts, scheduling, clocks, and BPF verifier basics. This chapter uses upstream Linux v6.12 as a historical source baseline. Tracing and perf availability, PMUs, permissions and distribution policy vary. The lab reads available metadata and optionally measures only its own bounded command; it does not enable host tracing.

Learning goals#

Distinguish tracing mechanisms and their contexts, follow tracepoint and perf-event permission paths, calculate a conditional multiplex estimate, and account for lost data and timing uncertainty.

Source walkthrough#

A tracepoint is a named instrumentation site compiled into kernel code, with arguments supplied by its caller (tracepoint documentation). A trace event builds a record format and filtering interface around a tracing event; not every possible probe and tracepoint is interchangeable with a trace event. ftrace function tracing instruments supported function-entry sites, while function-graph tracing can follow supported function entry and return to estimate nesting and duration. Dynamic ftrace uses patchable call sites and architecture-specific machinery to reduce disabled overhead; enabling it changes the executed path and still has overhead. A kprobe can dynamically instrument supported instruction locations, but blacklists, instruction boundaries, optimization and context constrain placement.

The tracepoint fast path in include/linux/tracepoint.h uses a static key. This exact v6.12 GPL-2.0-only excerpt is from the generated trace_##name inline function; continuation backslashes are part of the source and surrounding macro lines are omitted:

		if (static_key_false(&__tracepoint_##name.key))		\
			__DO_TRACE(name,				\

The condition checks whether the tracepoint's static key is enabled; the next line begins dispatch to the trace machinery when it is. The remaining macro passes arguments and condition, and optional lockdep checks follow. Enabled probes execute in the calling context, which can be interrupt, atomic or other constrained context; a consumer cannot assume it may sleep. Disabled sites still have a cost and configured instrumentation can perturb caches, timing and scheduling. A tracepoint name is not a promise of an eternal stable semantic ABI.

tracefs buffers and scope#

tracefs exposes tracing controls and event metadata when mounted and accessible (event tracing documentation). A trace instance can provide a separate set of many tracing controls and buffers, but not every tracing setting or underlying facility is necessarily instance-local. Global tracer or event changes can affect other users. This lesson does not write tracing_on, event enable, current_tracer, filters or buffer settings on the host. A dedicated disposable guest is the right place to configure an instance after understanding its cleanup and shared resources.

Tracing buffers are finite. Producers reserve and commit records (ring buffer design), and consumers read records according to the tracing ring buffer's per-CPU and overwrite or non-overwrite behavior. A slow consumer, high event rate, oversized event, disabled tracing interval or buffer policy can cause missing events. Per-CPU streams and clocks complicate ordering: timestamps need a known clock and synchronization assumptions before comparing events from different CPUs. Head and tail positions, reader pages, nested writers and overwrite rules belong to the actual ring-buffer implementation; a trace file with plausible lines is not proof that no records were lost.

A trace event can report a scheduler switch, for example, but it does not automatically establish the cause of the next task's latency. A kprobe at a function entry can count calls that fail later. A function-graph duration can include preemption or instrumentation effects depending on settings and interpretation. Always state the event, filter, clock, CPU scope, loss indicators and kernel identity before using a trace as evidence. When raw arguments or paths are captured, they can expose sensitive data and should not be collected or shared without authorization.

perf event creation and permissions#

perf_event_open creates an event fd from struct perf_event_attr (v6.12 perf UAPI), a target pid or CPU, grouping and flags. The attribute's type and config select an event; sample_type, period or frequency, exclusion flags and read format determine what is collected. A software event, hardware PMU event and tracepoint event have different sources and semantics. A PMU may not support a requested event or may have limited counters; event names and encodings vary across CPU models and virtualization. A sampling overflow is a sample of execution under a particular mechanism, not a complete instruction trace.

In kernel/events/core.c, perf_event_open begins with this exact v6.12 SPDX GPL-2.0 excerpt, copyright Thomas Gleixner and contributors; surrounding validation is omitted:

	err = security_perf_event_open(&attr, PERF_SECURITY_OPEN);
	if (err)
		return err;

The first line invokes a security check for the requested attributes; the next returns a denial or other error before an event is successfully created. The preceding code validates flags and copies the userspace attributes. The following code performs additional checks, including access to kernel measurements when they are not excluded, sampling constraints and lockdown-related restrictions for certain register data. Later allocation and attachment can also fail. A successful security hook alone is not proof that the event fd exists or that a PMU scheduled the event.

An event fd retains a reference to its event and associated context. Group leaders, child events, output redirection, mmap mappings and task exit can affect lifetime and teardown. A process must close its own event fd and unmap its buffers when done; an event associated with another task or CPU needs the appropriate authorization. perf_event_paranoid, capabilities, LSM policy and container restrictions can deny access (perf security documentation). A denied event is a valid diagnostic outcome; changing the host sysctl or using elevated privilege is not part of this lesson.

Counting, sampling and the mmap ring#

A counting event accumulates an event-specific value over the time it is scheduled. Sampling produces records at a configured period or frequency, subject to PMU and kernel limits. A count of CPU cycles is not elapsed wall time: frequency scaling, descheduling, stalled cycles and architecture influence interpretation. Sampling can have skid, where the recorded instruction pointer is near rather than exactly at the instruction responsible for an overflow. Precise-event support and its limits are PMU-specific. Symbolization needs matching binaries, kernel build IDs and debug symbols; a symbol from a different build can produce a plausible but wrong stack.

For the perf mmap interface (ring-buffer documentation), the metadata page contains producer and consumer position information such as data_head and data_tail; userspace follows the documented memory-ordering protocol when reading committed records and publishing consumption. Record headers and sample layouts depend on configured attributes. Ring buffers can lose records when producers outrun the consumer, and loss records or counters need interpretation. A sample absent from a buffer may have been lost, filtered, never generated, or outside the measurement interval. Treating the recorded sample count as the exact number of underlying events without accounting for sampling and loss is incorrect.

The producer and consumer positions are fields of struct perf_event_mmap_page, v6.12, SPDX GPL-2.0 WITH Linux-syscall-note (the comment above these fields omitted):

	__u64   data_head;		/* head in the data section */
	__u64	data_tail;		/* user-space written tail */
	__u64	data_offset;		/* where the buffer starts */
	__u64	data_size;		/* data buffer size */

data_head and data_tail sit beside the offset and size of the data area. The comment above them requires a memory barrier between reading record data and storing the tail, which this excerpt does not reproduce. Advancing data_tail is when the kernel may reuse that space, so it should follow processing of the record.

Hardware counters can be multiplexed when there are more requested events than available PMU slots. The read format can report time_enabled and time_running. For an illustrative event with raw count 300,000, enabled time 2 seconds and running time 1 second, a common scaled estimate is 300,000 * 2 / 1 = 600,000. This assumes that periods during which the event ran are representative of the periods it did not; phase changes or short bursts can make that estimate misleading. If running time is zero, the estimate is undefined, not zero or infinity to be silently reported as a measurement. Multiplexing and scaling should be disclosed.

The enabled and running times come from calc_timer_values, v6.12, SPDX GPL-2.0 (surrounding code omitted):

static void calc_timer_values(struct perf_event *event,
				u64 *now,
				u64 *enabled,
				u64 *running)
{
	u64 ctx_time;

	*now = perf_clock();
	ctx_time = perf_event_time_now(event, *now);
	__perf_update_times(event, ctx_time, enabled, running);
}

The helper samples perf_clock, computes the event-context time and passes it to the time-update code, which fills the enabled and running values. It returns times rather than a scaled count, so the scaling is left to the reader. The excerpt does not show how the scheduler chooses which events are on the PMU.

Now suppose a synthetic trace producer emitted 1,000 events and a loss indicator reports 100 lost. If the loss accounting covers the same interval and event stream, at most 900 are available under that simple model. If slow events are more likely to generate bursts and overflow the buffer, the retained 900 are not necessarily an unbiased sample. One cannot reconstruct the missing latency distribution by multiplying every bucket by 1000/900. The numbers are teaching data, not a live trace. Compare independent counters, reduce capture scope in a disposable environment, and report the uncertainty.

Choosing an instrument#

Use a static tracepoint when its event semantics match the question and the running build exposes it; it often has a more intentional interface than a probe into an arbitrary internal function, though semantics can still change. Use function tracing to investigate call flow only after considering recursion, instrumentation coverage and overhead. A kprobe can investigate an otherwise uninstrumented point but is tied to a symbol or instruction and kernel build. perf can count or sample hardware and software events and can consume tracepoints, but its CPU, thread and privilege scopes differ from a tracefs capture.

For a scheduling-delay hypothesis, a task's wakeup and subsequent switch-in tracepoints can suggest a runnable wait interval if paired by the correct task identity and clock. Missing records, migration, task-ID reuse and event filters can invalidate the pairing. CPU cycles sampled inside a function answer a different question from wall-clock latency between wakeup and execution. A controlled comparison and consistent build identity matter more than a large number of uncontextualized samples.

A measurement timeline and reference cleanup#

Imagine measuring a process for two seconds with two hardware events in a group. The PMU cannot schedule both continuously, so one event reports only one second of running time. Its scaled count is conditional on the observed interval representing the unobserved interval; if the workload has a startup phase followed by idle time, a scheduler that happened to run the counter mainly during startup can produce a severe overestimate. Event grouping can help coordinate related measurements when the PMU can schedule the group, but a group that never runs has no valid scaling denominator. Always preserve raw count, enabled time and running time rather than keeping only the scaled number.

An event created by perf_event_open can be initially disabled, then enabled and read using its fd under the selected attributes. A sampling event can additionally map a ring buffer and receive overflow records. On a failure after some setup, close the event fd and unmap successful mappings in the owning process; an open fd, a mapping and a parent/child event relationship have distinct references. The kernel serializes relevant event-context changes and uses reference counts to avoid freeing an event while another path still uses it. These lifetime rules do not make it safe for a collector to read an incomplete record or advance the consumer tail before processing data.

The trace ring and the perf ring are not identical formats. A tracefs event stream has its own per-CPU buffering and trace clock, while perf sample records use the configured perf attributes and mmap metadata. A collector must use the corresponding documented protocol, including memory ordering around the producer head and consumer tail. If a lost-record indication is observed, retain it alongside the sample interval and do not silently normalize a latency distribution as though all losses were random. If no loss indicator is visible, that also does not prove the event was enabled for the entire interval or that every relevant code path reached the instrumented site.

A practical comparison starts with a concrete question such as whether runnable wait time increased for one task. Record the kernel build and CPU topology, baseline workload and trace clock, then compare a controlled run with the same event selection and collection overhead. A change in perf cycles can reflect more work, frequency changes or different scheduling; a change in wakeup-to-run interval can reflect CPU contention, affinity or missing events. These hypotheses need additional evidence. Avoid turning one tracepoint or one PMU counter into an unconditional diagnosis.

Practice#

Read-only metadata and own-command perf

On an authorized host or disposable guest, inspect read-only availability. These commands do not enable events or change tracing controls:

uname -r
if [ -r /proc/sys/kernel/perf_event_paranoid ]; then cat /proc/sys/kernel/perf_event_paranoid; else echo 'perf policy unavailable'; fi
if [ -r /sys/kernel/tracing/available_events ]; then head -n 10 /sys/kernel/tracing/available_events; else echo 'tracefs event list unavailable or restricted'; fi
if command -v perf >/dev/null 2>&1; then perf --version; else echo 'perf unavailable'; fi

A readable event list proves metadata is visible, not that recording is allowed or loss-free. perf is a separately versioned userspace tool: record its version and kernel build when reproducing a result. If you are authorized to measure your own process, the following bounded command requests a software event and does not configure global tracefs:

if command -v perf >/dev/null 2>&1; then perf stat -e task-clock -- sh -c ':'; else echo 'perf unavailable'; fi

A permission denial, unavailable event, missing tool, restricted container or unsupported PMU is a legitimate result. The command's task-clock is CPU time for that tiny workload, not a reliable benchmark of shell performance; startup and measurement overhead dominate. Do not change perf_event_paranoid, enable host tracing or use sudo to make it pass.

Source-only alternative: use the 300,000 count with enabled/running times of 2 and 1 seconds to calculate the conditional 600,000 estimate, and explain why phase bias can invalidate it. Separately use the synthetic 1,000/100 loss example and name two reasons missing events may not be random. Deliver the calculations or actual read-only metadata and own-command result, with kernel/tool versions and a statement of what was not observed.

Knowledge check#

1. What does the tracepoint static key control in the quoted fast path?
2. Why can perf_event_open fail after the initial security hook returns zero?
3. When is the scaled 600,000 count only an estimate?
4. Why is CPU time not necessarily the same as wall-clock latency?
5. Why can 100 lost records bias a latency distribution rather than merely reduce its size?

Answers#

1. It gates dispatch of the enabled tracepoint path. The enabled callback executes in the call site's context, and neither the key nor a named event establishes loss-free capture.
2. Attribute checks, kernel-access restrictions, PMU support, allocation and attachment can fail later. A single hook's zero return is not the final fd result.
3. Scaling assumes the event rate during scheduled PMU time represents the unscheduled interval. Workload phases and uneven multiplexing can violate that assumption.
4. CPU time measures execution charged to a task or event; wall time can include descheduling, blocking and other waiting, and cycles depend on frequency and PMU semantics.
5. Loss can correlate with bursts or expensive events. The missing events need not have the same distribution as the retained samples.

Checkpoint#

Submit the multiplex calculation and loss-bias explanation, plus the source permission branch or actual own-command result. Ready means each observation states scope, time, loss and limitations without enabling host tracing. Retry with a timeline if CPU execution, wall time and sampled records are conflated. This is not certification.

Sources#

• Tracepoint fast path v6.12
• Perf event core v6.12
• Tracing ring buffer v6.12
• ftrace documentation
• Perf ring-buffer documentation