Traffic control, qdiscs, XDP, and network BPF

Before you start#

Read netfilter and nftables, NAPI and offloads, and sockets and skbs. The source baseline is upstream Linux v6.12, a historical version. The lab only inspects loopback configuration; it does not attach programs or change qdiscs.

Learning goals#

Explain queueing, shaping, policing and marking; trace qdisc dispatch and a specific driver XDP redirect; distinguish tc skb programs from native, generic and offloaded XDP; and account for AF_XDP and packet-buffer ownership.

Source walkthrough#

Traffic control (tc) configures queueing disciplines, classes, classifiers and actions. An egress qdisc can enqueue packets awaiting transmission, dequeue according to its policy, and maintain backlog and drop statistics. A classless qdisc applies one policy without user-visible child classes; a classful qdisc can classify traffic into classes with separate scheduling parameters or child qdiscs. A classifier chooses an action or class according to configured rules. An ingress attachment can classify or police arriving skbs but does not provide an ordinary egress backlog for delaying packets in the same way. Exact behavior depends on the qdisc, attachment and device.

Shaping intentionally schedules transmission to constrain rate, generally requiring queueing. Policing measures traffic against a rule and can drop or mark excess traffic rather than buffering it for later. Pacing schedules packets over time based on a sender or qdisc policy; it is not synonymous with a strict administrative rate limit. A queue can drop on overflow or apply an active queue-management algorithm. Explicit Congestion Notification (ECN), when enabled and applicable to an ECN-capable packet, marks congestion rather than necessarily dropping that packet; it does not guarantee that every overload can be signaled without loss. fq, fq_codel and HTB address different scheduling problems and must be understood with their actual configuration. See net/sched/sch_generic.c, sch_fq_codel.c, and sch_htb.c.

The core's __qdisc_run, Linux v6.12, GPL-2.0-or-later, initializes a quota and repeatedly calls qdisc_restart. Its loop includes:

	while (qdisc_restart(q, &packets)) {
		quota -= packets;
		if (quota <= 0) {

Each successful restart reports a packet count that reduces the per-run quota. On exhaustion, the following code marks a missed run for a lockless qdisc or schedules another run for the other case, then exits the loop. This is a bounded service pass, not a claim that all qdiscs dequeue in FIFO order. net_tx_action and other paths can run scheduled qdiscs; locking and no-lock state coordinate concurrent producers and runners. The selected driver can still stop its transmit queue or return a resource condition after dequeue, requiring the core's retry/requeue rules.

The quota logic is a small loop with four states, shown here as a flow of the excerpt above.

Three packets and queue policy#

Imagine three same-sized ECN-capable packets A, B and C arrive at a configured egress queue while the device can transmit only one immediately. A leaves; B and C wait. A simple FIFO queue can send B then C when resources return, but a fair-queueing scheduler can place packets into flows and choose a different order across flows. If a configured active queue manager decides congestion should be signaled, and a packet and policy permit ECN marking, B might be marked rather than dropped. If the queue reaches a hard capacity, C can still be dropped. The exact result depends on qdisc, queue state, flow hashing, ECN capability and configuration; these values are a teaching scenario, not a measurement of fq_codel defaults.

HTB can organize classes with configured rates and ceilings, subject to its implementation and token accounting; it does not create physical bandwidth beyond the device. fq and fq_codel address flow scheduling and queue-delay behavior with different mechanisms. A qdisc statistic named drops or overlimits has meaning defined by the implementation and attachment. One snapshot cannot establish why every packet was dropped. Counters can include unrelated traffic, and hardware offload can change which software path observes packets.

XDP placement and actions#

XDP runs a BPF program at a receive-oriented hook. Native/driver XDP runs in a driver-specific early path, commonly before a normal skb is constructed; generic XDP runs in the networking core on an skb-based path and has different costs and capabilities. Hardware offload executes a supported program on capable device hardware, subject to driver and verifier/offload constraints. These are not interchangeable execution contexts. A driver and configuration may not support native XDP, zero-copy AF_XDP or any hardware offload. A program's successful verification does not prove attachment to a device or a particular packet path.

The principal XDP actions are XDP_PASS (continue into the appropriate stack path), XDP_DROP, XDP_TX (transmit through the applicable device path), XDP_REDIRECT (use redirect information set by a helper), and XDP_ABORTED (exceptional failure). Drivers must handle ownership for each result. XDP runs early for an applicable ingress packet, but it is inaccurate to say every BPF program or every packet bypasses netfilter: a passed packet can continue through later stack hooks, while a dropped or redirected packet may not traverse the same path. tc BPF at ingress or egress ordinarily acts on an skb and can classify or take configured actions at those attachment points; its helpers, metadata and lifetime rules are not identical to XDP's xdp_buff context.

drivers/net/virtio_net.c provides one representative XDP redirect branch, Linux v6.12, GPL-2.0-or-later:

		err = xdp_do_redirect(dev, xdp, xdp_prog);
		if (err)
			return XDP_DROP;

The driver asks the redirect machinery to process this packet; a nonzero error turns this driver's result into drop. Its success branch marks redirect work for a later flush. This is one virtio-net implementation, not a claim about every driver's cleanup or every redirect target. xdp_do_flush must run at the proper point in the NAPI cycle for the bulk queues and RCU protection described in net/core/filter.c. A redirect helper's selected target does not itself prove a packet reached that target; map lookup, target state, queue capacity and flush can fail.

A devmap can redirect toward network devices, a cpumap can redirect work toward CPUs, and an XSK map can direct eligible traffic to AF_XDP sockets. Each has map, queue and ownership constraints. The action XDP_REDIRECT without valid helper-produced redirect state is not a complete forwarding policy. XDP_TX transmits using the supporting driver path, and a failed conversion or transmit must release the packet buffer according to the driver. XDP_ABORTED is generally surfaced as an exception/drop path; it should not be treated as a normal successful pass.

Packet bounds and AF_XDP#

An XDP program works with packet pointers such as data and data_end; verifier-visible bounds checks must precede accesses. Helpers that change packet head, tail or layout can invalidate previous pointer assumptions, requiring pointers to be reloaded and bounds checked again. A verifier rejection, unsupported helper or unavailable offload is a normal setup failure, not a reason to disable security controls. The BPF verifier, runtime, maps and privileges are covered in later chapters; a verified program can still make a semantically wrong policy decision.

AF_XDP uses a userspace memory region (UMEM) divided into frames and shared rings for fill, receive, transmit and completion. Ownership is transferred through ring operations: userspace supplies frames, the kernel/driver can fill them, userspace consumes received descriptors, and transmitted frames become reusable only after the appropriate completion. Returning a frame to a fill ring while it is still being read or transmitted can corrupt data. Ring producer/consumer synchronization and memory accounting matter. Copy and zero-copy modes depend on kernel, driver, device and setup; the existence of AF_XDP does not prove hardware zero-copy support. See net/xdp/xsk.c.

A hypothetical redirect sends a received frame to a devmap target. The program first validates packet bounds, selects a key and invokes the redirect helper, then returns XDP_REDIRECT. The driver executes the redirect and eventually flushes its batch. If the map has no suitable target or the redirect call fails, the driver's error path can drop and account the frame; it must not simultaneously hand the same buffer to normal receive and recycle it. If it succeeds, ownership proceeds to the redirect path, whose target and completion rules govern release. This state example is not an observed packet trace.

Queue state, classification, and the driver boundary#

A qdisc's backlog is the work it currently holds, measured in the implementation's packet and byte accounting, not the NIC's entire outstanding DMA ring. A classifier can choose a class or action based on packet fields, metadata, marks or other configured state. In a classful hierarchy, the parent scheduler's selection and a child qdisc's own order both matter. A packet that matches a class can still be dropped later because of a child queue limit, policing action, driver failure or policy. Conversely, a drop counter at one layer does not automatically include drops before that layer received the skb.

Suppose a traffic contract allows 1 megabit per second and a burst allowance. A shaper can send an initial permitted burst, then queue eligible packets until its rate budget allows further transmission; the exact token calculation and units depend on the configured qdisc. A policer using a comparable threshold can instead apply its configured excess action immediately. If excess is dropped, a sender may retransmit or adapt depending on its transport, but the policer itself does not perform TCP congestion control. ECN marking is meaningful only when the packet and configured algorithm permit it; not-ECT traffic or a full queue can still be dropped. The scenario gives intuition, not a benchmark or an assertion about a default HTB configuration.

__qdisc_run repeatedly requests work with qdisc_restart, bounded by a quota, while the surrounding scheduling and locking code coordinates producers and the runner. If the transmit queue is stopped, work can remain pending until the driver wakes it. If a driver returns busy without consuming the skb, the core must preserve ownership for its defined retry path. A device removal or qdisc replacement must coordinate queued skbs and concurrent execution before freeing state. A qdisc drain, a NIC completion and an application acknowledgement are separate events; a throughput counter at any one layer cannot prove the others occurred.

Redirect is a multi-stage ownership transfer#

The redirect helper first records target information, including a map lookup result where applicable, in per-CPU redirect state. Returning XDP_REDIRECT tells the driver to use that state; the helper did not by itself complete packet delivery. xdp_do_redirect places the frame on the appropriate bulk queue or returns an error. The driver then invokes xdp_do_flush before napi_complete_done in the documented NAPI sequence. The v6.12 source comment explains that map-entry protection relies on the single NAPI processing interval and RCU conditions. Deferring the flush past that interval can break both delivery and lifetime assumptions.

The NAPI poll function in the same virtio-net driver shows where that flush sits. In drivers/net/virtio_net.c, Linux v6.12, GPL-2.0-or-later (lines omitted):

	virtnet_poll_cleantx(rq, budget);

	received = virtnet_receive(rq, budget, &xdp_xmit);
	rq->packets_in_napi += received;

	if (xdp_xmit & VIRTIO_XDP_REDIR)
		xdp_do_flush();

The flush runs only when the receive loop has set VIRTIO_XDP_REDIR, and after virtnet_receive returns but before the completion check. The excerpt does not show the completion logic or the flush points of other drivers, so it is not a rule for every driver.

In net/core/filter.c, Linux v6.12, GPL-2.0-or-later, the driver-facing redirect function chooses between the AF_XDP path and the frame path:

int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp,
		    struct bpf_prog *xdp_prog)
{
	struct bpf_redirect_info *ri = bpf_net_ctx_get_ri();
	enum bpf_map_type map_type = ri->map_type;

	if (map_type == BPF_MAP_TYPE_XSKMAP)
		return __xdp_do_redirect_xsk(ri, dev, xdp, xdp_prog);

	return __xdp_do_redirect_frame(ri, dev, xdp_convert_buff_to_frame(xdp),
				       xdp_prog);
}

xdp_do_redirect reads the redirect target recorded by the helper. Only an XSKMAP target takes the AF_XDP path; other types convert the buffer to an xdp_frame for the frame path. The excerpt does not show queue admission, target validation or the error values, so a zero return should not be read as delivery.

The flush function in the same file drains whichever per-type lists were used in this context:

void xdp_do_flush(void)
{
	struct list_head *lh_map, *lh_dev, *lh_xsk;

	bpf_net_ctx_get_all_used_flush_lists(&lh_map, &lh_dev, &lh_xsk);
	if (lh_dev)
		__dev_flush(lh_dev);
	if (lh_map)
		__cpu_map_flush(lh_map);
	if (lh_xsk)
		__xsk_map_flush(lh_xsk);
}

Device, CPU-map and XSK redirects have separate flush lists here. Work that was queued but never reaches its matching flush is not delivered by this function alone, which is why the driver's flush call matters.

For a devmap target, the eventual transmit device and queue must be usable. A cpumap target instead hands work toward another CPU under that map's processing model. An XSK map redirects to an AF_XDP socket bound with compatible queue and UMEM configuration. A missing entry, incompatible queue, full target ring or unsupported driver mode can lead to a failed redirect or drop. XDP_TX and XDP_REDIRECT have different target selection, even though both require the original receive buffer to have a well-defined new owner. Returning XDP_PASS follows the applicable normal stack path; it does not grant the XDP program a second opportunity to reuse the same frame after handing it off.

An AF_XDP UMEM frame might begin in the fill ring, become owned by a receive operation, appear on the receive ring, and then be processed by userspace. After userspace is done, it can return an appropriate frame to the fill ring. On transmit, userspace publishes a transmit descriptor and waits for the completion-ring indication before reusing the frame under that API. Completion returns the frame for reuse; it does not by itself prove successful transmission. These ring entries are not deep copies of all packet bytes. A program with multiple producer or consumer threads must follow the ring's synchronization contract rather than assume the ring is automatically multi-producer safe. A process crash, socket close or detach has its own cleanup behavior and is not proof that a live peer has already stopped reading a frame.

tc and XDP observations at different points#

A tc ingress BPF classifier normally sees an skb after the stack has constructed one; native XDP can run earlier on a driver receive buffer. Generic XDP is implemented in an skb-oriented path, and hardware-offloaded XDP can run in a device with a restricted supported instruction and helper set. For the same original wire packet, a native XDP drop can prevent an skb-based tc or netfilter observer from seeing it. A pass can allow later processing, subject to all other routing and policy conditions. An XDP redirect can select another endpoint rather than simply invoking the original stack path. These are conditional placement relationships, not a universal bypass rule.

A program must validate that a header fits between data and data_end before reading its fields. After a helper changes packet layout, previously checked pointers may no longer establish the required bounds. Reload the pointers and repeat the checks. A verifier rejection can arise from an unsafe access, unsupported helper, resource limit or attachment restriction; it is useful evidence that this program was not accepted for that environment. A verified program can still drop the wrong traffic or use an incorrect map key. The read-only tc lab does not load code, verify a program or prove the placement of a live BPF attachment.

Interpreting a read-only qdisc snapshot#

A tc -s qdisc show result names qdiscs attached to the selected interface and can report sent bytes, packets, drops, overlimits and backlog. Those fields are cumulative and qdisc-specific; a later snapshot must be compared over a known interval, and a zero backlog at the instant of inspection does not prove that the queue was never occupied. The lo interface often uses noqueue, so the absence of a shaping hierarchy is an observation about loopback, not proof that the kernel lacks HTB or fq_codel. Read-only tc filter show can return an empty list even if another interface or another network namespace has filters.

A useful experiment design would separate policy from capability: first identify the attachment point and program mode, then use a controlled disposable guest and a known packet source, and finally compare counters at the ingress, redirect target and egress. The present lab intentionally performs no attachment or traffic generation. A displayed hardware-offload flag is not a captured execution trace, and a successful qdisc query does not establish that an XDP program ran. When a tool or permission is missing, the source-only redirect timeline remains a valid exercise with its hypothetical status clearly marked.

Practice#

Source-only route:

rg -n -A22 'void __qdisc_run\(' "$KERNEL_SRC/net/sched/sch_generic.c"
rg -n -A18 'case XDP_REDIRECT:' "$KERNEL_SRC/drivers/net/virtio_net.c"
rg -n -A24 'void xdp_do_flush\(' "$KERNEL_SRC/net/core/filter.c"

Confirm the checkout is v6.12. Identify the qdisc quota and reschedule branch, the virtio redirect failure, and the flush call's scope. Work through A/B/C and the failed redirect; state the configuration assumptions and ownership at each step.

Read-only loopback inspection:

if command -v ip >/dev/null 2>&1; then ip -details link show dev lo; else echo 'ip unavailable'; fi
if command -v tc >/dev/null 2>&1; then
  tc -s qdisc show dev lo || echo 'qdisc inspection unavailable'
  tc -s filter show dev lo ingress || echo 'ingress filter inspection unavailable'
  tc -s filter show dev lo egress || echo 'egress filter inspection unavailable'
else echo 'tc unavailable; use source-only route'; fi

An empty filter list or a noqueue qdisc is a valid result; commands and permissions vary by environment. Counters are snapshots and can include unrelated traffic. No BPF program is loaded or attached, and no qdisc is modified. Deliver the queue timeline and observed qdisc/filter state or explicit unavailability.

Knowledge check#

1. What happens when __qdisc_run exhausts its quota?
2. How do shaping and policing differ?
3. What does the shown virtio-net branch do if xdp_do_redirect fails?
4. Why must packet bounds be revalidated after a layout-changing helper?
5. Why is an AF_XDP completion relevant before a userspace frame is reused?

Answers#

1. The code marks a missed run or schedules another run according to qdisc locking mode, then breaks out of this pass.
2. Shaping schedules traffic over time, typically with a queue; policing measures and applies an action such as drop or mark to excess traffic.
3. It returns XDP_DROP in this branch rather than reporting successful redirect.
4. A helper can invalidate old packet pointers or boundaries, so subsequent accesses need fresh verifier-visible checks.
5. The kernel or device may still own or use the transmitted frame until the relevant completion transfers reuse responsibility back.

Checkpoint#

Save the three-packet queue sketch, redirect success/failure timeline and inspection output. Pass when you distinguish qdisc scheduling, tc skb processing, XDP placement and AF_XDP frame ownership without equating an offload request with hardware support. Re-read the two cited functions and revise any uncertain boundary. This is practice, not certification.

Sources#

• Generic qdisc run, Linux v6.12.
• fq_codel.
• HTB.
• virtio-net XDP paths.
• XDP redirect and flush.
• tc BPF classifier.
• AF_XDP implementation.