Experimental design for kernel performance

Before you start#

Review scheduling and CPU placement, memory pressure, cgroup controllers, and perf and tracing. Source examples use upstream Linux v6.12. The real exercise measures only its own bounded userspace process and temporary file; its results are not portable kernel benchmarks.

Learning goals#

Form a discriminating hypothesis, distinguish throughput, wall time, CPU time and tail latency, interpret PSI and multiplexed perf counters, and design a controlled comparison with raw data and uncertainty.

Source walkthrough#

Performance is a property of a system under a particular workload, environment and measurement method. Begin with a question such as “did this change increase the 99th-percentile request latency for workload W on configuration C?” Define the unit of work, correctness requirement, measurement window and expected mechanism before choosing a tool. A faster result that drops requests or returns incorrect data is not an improvement for the same workload. Record kernel commit and config, compiler and build flags, userspace versions, hardware or VM topology, storage, CPU placement, cgroups and background activity.

Wall-clock latency measures elapsed time from a defined start to end. CPU time measures time charged to execution; off-CPU waiting, preemption, locks, I/O and scheduling can make wall time much larger. Throughput measures completed work per time interval; it can improve while individual tail latency worsens. A percentile summarizes a distribution: a p99 estimate from a small or biased sample can be unstable, and a median alone hides rare long pauses. Event counts and samples are useful evidence but require their own units, scope and loss accounting.

PSI is a state-based signal

In kernel/sched/psi.c, psi_task_change updates the task's PSI flags and propagates a change to the task's group and parent groups. This exact v6.12 SPDX GPL-2.0 excerpt omits declarations and the initial idle-task check:

	group = task_psi_group(task);
	do {
		psi_group_change(group, cpu, clear, set, true);
	} while ((group = group->parent));

The first line finds the task's pressure-accounting group. The loop applies the state change on the selected CPU and walks to each parent. PSI derives stall information from task-state accounting; the displayed rolling averages and cumulative totals summarize pressure, not resource utilization or a unique root cause. Configuration and cgroup hierarchy determine what is available. A pressure increase can correlate with a latency increase without identifying which reclaim, device or scheduling mechanism caused it.

A perf read has scope and failure paths

In kernel/events/core.c, perf_event_read_local is a kernel-side local-read interface with context restrictions. This exact v6.12 SPDX GPL-2.0 excerpt omits surrounding comments and checks:

	if (event->attr.inherit) {
		ret = -EOPNOTSUPP;
		goto out;
	}

The condition rejects an inherited event because this local atomic-context read cannot collect all child counters. The error path goes to a common exit that restores interrupt state. Other checks reject a per-task event not belonging to current, a per-CPU event for another CPU, or a pinned event not running on this CPU. When allowed, the function reads the active PMU event and can return enabled and running times. This is not a claim that every userspace perf read follows exactly the same local function or that a successful read covers all CPUs and child tasks.

Workload and environment control#

A reproducible workload states input size, access pattern, concurrency, request rate, warmup, duration and output validation. Cold-cache and warm-cache runs answer different questions. Reusing a page-cache-resident file can measure userspace and cache behavior rather than storage media; a first read can include setup, page faults or device activity. Cache state must be described, not “fixed” by dropping the host's caches. Randomization or alternating A/B order can reduce systematic order bias, but it does not eliminate drift or replace enough repetitions.

CPU frequency scaling, turbo behavior, thermal limits, power management, interrupt placement, shared caches, background daemons, virtual CPU scheduling, NUMA placement and cgroup throttling can all affect results. A benchmark's affinity and cpuset effective values matter, but pinning is not automatically fair or representative of production. Memory allocation and page placement can differ between trials. Build flags, debug instrumentation and sanitizers change both code and overhead. Record these factors and choose controls appropriate to the hypothesis; do not change host frequency, throttle settings, cache state or sysctls for this lesson.

A warmup period can allow code paths, caches and allocators to reach a chosen state, but discarding warmup can hide an important startup regression. State whether startup is in scope. Measure the same successful work in both variants: same input and output, same concurrency, same target and timeout. Random seeds and datasets should be retained where relevant. If the workload is nondeterministic, describe how it was generated and whether the randomization was reproducible.

A synthetic paired A/B experiment#

Suppose a hypothetical benchmark produces paired wall times in milliseconds: baseline A is 100, 102, 99, 101, 100; candidate B is 98, 99, 100, 98, 99. Each pair uses the same input and completes with the same verified output. The median is 100 ms for A and 99 ms for B, a 1 ms or 1% reduction relative to the baseline median. Pairwise changes are -2, -3, +1, -3 and -1 ms. These values are synthetic teaching data, not measurements made here.

The candidate looks slightly faster in this small example, but one pair is slower and five samples do not establish a stable effect across workloads. A background task, different cache state or thermal drift could produce a similar difference. Alternate or randomize run order, collect more independent repetitions, preserve raw samples and use an uncertainty method appropriate to the experiment. Report the effect size and uncertainty, not only the fastest candidate run. If the difference is smaller than measurement noise, report that the evidence is inconclusive rather than declaring a win or equivalence without a defined equivalence margin.

A valid negative control can use the same code path without the suspected change or a workload where the proposed mechanism should not matter. A control that changes multiple inputs is harder to interpret. If A reads cached data and B reads uncached data, the comparison does not isolate a kernel patch. If B returns early or processes fewer bytes, matching elapsed time says little about equivalent performance. Preserve output hashes or another correctness check alongside timing, and investigate failures before calculating speedup.

Multiplexing, sampling and observer effects#

A PMU has a limited number of counters and may multiplex events. Suppose an event counts 300,000 occurrences while it runs for one second during a two-second enabled interval. Scaling by enabled divided by running gives a conditional estimate of 600,000. The estimate assumes the measured periods represent the unmeasured periods. If the event runs only during a high-load phase, it can overestimate the full interval; if time_running is zero, the ratio is undefined. Save raw values and times and disclose scaling rather than presenting the estimate as an exact count.

Sampling records a subset of execution or overflows, not every operation. PMU skid can place a sample near rather than exactly at the responsible instruction; precise-event support is hardware-specific. Lost ring-buffer records, filtering, sampling frequency and symbol mismatch can bias a profile. A function with many samples may be hot because it is called often or because each call is expensive; a sampled stack alone does not distinguish those mechanisms. Matching build IDs, binaries and debug symbols are required for reliable attribution.

Instrumentation can perturb the measurement. A tracepoint, high-frequency perf sample, BPF map update, stack capture or verbose log consumes CPU, memory bandwidth and buffer space. Compare with and without the observer where safe, keep event selection narrow, and record lost samples and overhead. An uninstrumented baseline and a heavily traced candidate are not comparable as a patch experiment. A perf permission failure is a data point about the environment, not a reason to relax host policy.

Correlation, hypothesis and intervention#

Suppose request latency rises at the same time as memory PSI. That correlation supports investigating memory stalls, but it does not distinguish cgroup high-limit reclaim, global reclaim, I/O delays, allocation bursts or a changed workload. Examine time-aligned memory events, CPU throttling, I/O statistics, application work and relevant source paths. Then design a controlled intervention in a disposable environment: for example, compare the same workload under two explicitly documented memory limits while holding other conditions stable. A successful intervention strengthens a causal explanation only if it changes the hypothesized mechanism and alternative explanations remain controlled.

A scheduler counter, a PSI aggregate and a function trace can disagree without any being broken: they may measure different scopes or clocks. CPU time charged to a process excludes time while it waits; PSI counts defined stall conditions over a group; wall time spans the user-visible interval. A tracing timestamp and a userspace clock need a known relationship before subtraction. No single counter proves a root cause. Keep raw evidence and a written causal model so another person can challenge the inference.

Choosing the unit of evidence#

A request latency distribution should define when the timer starts and ends. For a storage request, application submission-to-completion includes queueing, scheduling and filesystem behavior; device service time is a narrower interval. A write acknowledged by a buffered interface can finish before durable storage. Comparing those two latencies as though they measured the same contract would produce a false conclusion. Likewise, throughput expressed as bytes per second should say whether it counts submitted, completed, verified or durable bytes. Correctness failures and timeouts need their own counts rather than being removed silently from the denominator.

For a paired test, assign each A/B pair a shared input and environment window, then randomize or alternate which variant runs first. Compute the difference for each pair before summarizing so that a slowly changing machine state is less likely to masquerade as a treatment effect. Pairing cannot correct a systematic difference such as A using a debug build and B an optimized build. Record build flags, warmup policy, trial order, raw samples and a confidence or uncertainty method. Repeating one long run is not always equivalent to independent repetitions because samples within a run can be correlated.

A concrete diagnostic matrix can compare wall time, process CPU time, CPU throttling and memory pressure for each trial. If wall time increases while CPU time is stable and throttling rises, quota is a plausible contributor; if memory pressure also rises, reclaim or I/O can be another explanation. Neither counter alone proves the cause. Check parent cgroups and workload correctness, then vary one factor in a disposable test environment. If the suspected counter does not change while latency changes, revise the hypothesis rather than forcing the evidence to fit it.

The bounded temporary-file exercise is intentionally smaller than a kernel performance investigation. It checks that every read returns the same bytes and records a distribution, but repeated reads can be served by cache and the SHA-256 verification and interpreter contribute to the measured interval. A faster sample is not proof of a faster filesystem or device. Its value is practicing raw-data retention, monotonic timing, correctness checking and cautious interpretation before adding more complex instruments.

Practice#

Bounded real userspace measurement

This Linux/Python 3 exercise writes a small deterministic temporary file, verifies every read, and records seven wall and process-CPU durations. It does not drop caches or alter kernel knobs. It includes Python, temporary filesystem and page-cache effects; it is not a storage-device or kernel benchmark.

python3 - <<'PY'
import hashlib, os, platform, statistics, tempfile, time
payload = bytes(range(256)) * 512
expected = hashlib.sha256(payload).digest()
wall, cpu = [], []
print('kernel:', platform.release(), 'python:', platform.python_version())
with tempfile.TemporaryDirectory(prefix='kernel-study-') as directory:
    path = os.path.join(directory, 'data.bin')
    with open(path, 'wb') as f:
        f.write(payload)
    for _ in range(7):
        w0, c0 = time.perf_counter_ns(), time.process_time_ns()
        with open(path, 'rb') as f:
            data = f.read()
        assert len(data) == len(payload) and hashlib.sha256(data).digest() == expected
        wall.append((time.perf_counter_ns() - w0) / 1e6)
        cpu.append((time.process_time_ns() - c0) / 1e6)
    print('bytes per read:', len(payload), 'verified reads:', len(wall))
    print('wall ms raw:', wall)
    print('wall median/range ms:', statistics.median(wall), min(wall), max(wall))
    print('CPU ms raw:', cpu)
PY

Expected structural output is the kernel and Python versions, 131,072 bytes, seven verified reads, seven wall and CPU samples and a median/range. The actual times depend on the machine and must not be copied as universal expected values. A temporary-directory or filesystem permission error means the exercise did not complete; an assertion failure means the work was not verified. The context manager removes only the temporary directory it created. Repeated reads can be cached, and hashing and Python overhead are included. No cold-cache claim follows.

Optional perf stat -e task-clock -- true can measure its own bounded command where perf is already installed and permitted; a missing tool or permission denial is a legitimate result. Do not install perf, change perf_event_paranoid, use sudo, drop caches or alter host CPU settings. Record the tool version if used.

Source-only alternative: calculate the synthetic A/B median and pairwise differences, then evaluate the 300,000 count with two seconds enabled and one second running. Explain one workload phase that makes scaling misleading. Deliver those calculations or your own raw exercise output, with the correctness check and the distinction between real and synthetic data.

Knowledge check#

1. Why is the synthetic 1% median improvement not automatically a reliable patch effect?
2. What does the PSI source excerpt update, and why does PSI not prove a root cause?
3. Why does perf_event_read_local reject an inherited event?
4. When can a 600,000 scaled count misrepresent the true event total?
5. Why can lower CPU time coexist with higher wall-clock latency?

Answers#

1. Five noisy pairs, one slower case, possible order and environmental effects, and missing uncertainty limit the conclusion. Correctness and controlled repeated comparisons are required.
2. It propagates a task state change to its PSI group and ancestors. The resulting stall aggregate can reflect several mechanisms and needs correlated evidence and a controlled test.
3. The local atomic-context read cannot collect all child counters, so the cited branch returns -EOPNOTSUPP and restores interrupt state on exit.
4. When the time the counter ran is not representative of the full enabled interval, such as a phase change or uneven multiplexing. The scaling is an estimate, not a measured exact total.
5. The task can spend less time executing while waiting longer for I/O, locks, reclaim or CPU scheduling. Wall time includes those off-CPU intervals.

Checkpoint#

Submit raw samples and correctness output from the bounded exercise, or the synthetic A/B and multiplex calculations. Ready means the same work is verified, raw data and effect size are retained, uncertainty and alternative causes are stated, and no fastest-run-only conclusion is made. Reproduce the comparison under controlled conditions before claiming a kernel improvement. This is not certification.

Sources#

• PSI implementation v6.12
• perf event core v6.12
• perf UAPI v6.12
• PSI documentation
• perf ring-buffer documentation