net_device, NAPI, queues, and offloads

Before you start#

Read routing and neighbours and sockets and skbs. The source baseline is upstream Linux v6.12, a historical teaching version. Driver, hardware, architecture, configuration, traffic and PREEMPT_RT change the details. All commands below read existing state; they do not change interfaces or generate traffic.

Learning goals#

Follow a representative receive poll and a transmit ownership decision, explain the NAPI budget and interrupt race, distinguish skb and DMA buffer lifetimes, and interpret offload and queue counters without assuming that an skb equals one wire packet.

Source walkthrough#

A struct net_device represents a network interface in a network namespace. It holds interface state, feature flags, queues and a net_device_ops table. A driver implements selected operations, including ndo_start_xmit for transmission; virtual devices have operations too. A multi-queue device has transmit and receive queues whose mapping to CPUs, interrupts and hardware is driver and configuration dependent. A netdev is not necessarily a physical NIC: loopback, veth and tunnels are also netdevs. See include/linux/netdevice.h and net/core/dev.c.

NAPI is a kernel mechanism for scheduling a driver's poll method, usually to process batches of receive work and often transmit completions. A driver registers a napi_struct with a poll callback and enables it when the device is ready. In a typical interrupt-driven path the interrupt handler acknowledges or masks the relevant interrupt and schedules NAPI; the poll method processes descriptors up to its budget. NAPI is commonly run from network softirq processing, but threaded NAPI, busy polling and PREEMPT_RT alter execution context. It is not correct to assume every poll runs in the original hardware interrupt handler, or that every netdev uses the same interrupt mechanism.

The generic __napi_poll, Linux v6.12, GPL-2.0-or-later, contains:

	if (napi_is_scheduled(n)) {
		work = n->poll(n, weight);
		trace_napi_poll(n, work, weight);

The core checks scheduled state before invoking this registered callback with the NAPI weight as its budget, and records a trace event after it returns. The surrounding comments explain why the scheduled-state test avoids a race with netpoll. The core warns if work exceeds weight. If work is less than the weight, this code returns from this pass; if the whole budget is consumed, the core can keep or repoll the NAPI instance according to its state. A returned work count is driver-reported work under that callback's contract, not necessarily a count of wire frames visible at every later layer.

A concrete driver is Intel e1000e, not a universal NIC implementation. Its interrupt path uses napi_schedule_prep, masks the relevant interrupts and calls __napi_schedule. Its poll method cleans transmit completions and calls its receive cleaner. The completion branch in drivers/net/ethernet/intel/e1000e/netdev.c, Linux v6.12, GPL-2.0, begins:

	if (likely(napi_complete_done(napi, work_done))) {

Only when napi_complete_done returns true does the following driver code consider re-enabling interrupts; it also checks device-down state and its MSI-X configuration. When transmit cleanup is incomplete or receive work consumed the full budget, this driver returns the budget instead. This prevents an unconditional "work less than budget means always enable IRQ" rule: busy polling, a missed scheduling event and teardown can change the result. A device event arriving around poll completion must not be lost; correct scheduling, masking and re-enable ordering are a driver/core contract.

Receive buffer and GRO lifetime#

A physical receive queue supplies buffers the device can DMA into. The driver must establish the correct DMA mapping and ownership, inspect completed descriptors, and synchronize or unmap as required before CPU access. A page pool can recycle pages for receive workloads, but recycling is valid only when references and DMA ownership permit it. An skb may retain a page or fragment after the driver moves on to another descriptor. Returning that page to a pool too early can let the NIC overwrite data still used by the stack. Conversely, losing the last reference without the appropriate return or release leaks memory or mappings. Exact page-pool use is driver-specific; e1000e is cited for NAPI scheduling, not as a claim that all of its receive paths use the same page-pool design. See net/core/page_pool.c.

A driver can pass a received skb to napi_gro_receive. Generic receive offload (GRO) can combine eligible packets into a larger logical skb before later processing. The core's GRO machinery flushes batches under its rules and delivers normal skbs onward; see net/core/gro.c. Not all packets are eligible, and GRO does not mean the wire carried one giant frame. After delivery, protocol headers, routing, sockets and later policy still determine the result. A drop can occur in a driver, backlog, protocol validation, policy or socket queue; counters at those points are not interchangeable.

For a hypothetical poll budget of 64, suppose the e1000e cleaner reports 20 receive units and transmit cleanup is complete. The driver can attempt napi_complete_done(…, 20) and, only if it returns true and the device is up, re-enable the relevant interrupt. If it reports 64, the code returns the budget and does not follow that completion branch in this invocation. Whether there were exactly 64 physical frames, GRO aggregation, an immediately arriving 65th packet or another scheduling event depends on the actual driver and workload. This is a state example from source, not a measured interrupt trace.

Transmit ownership and backpressure#

A qdisc or direct transmit path selects an skb and transmit queue, then the netdev core calls the driver's ndo_start_xmit through its transmit helpers. NETDEV_TX_OK means the driver consumed responsibility for the skb under that interface, even if the driver internally dropped it and accounted an error. It does not mean the peer received a frame. NETDEV_TX_BUSY means the driver did not consume the skb and the caller must retain or retry it according to the networking core's contract; a driver returning BUSY after freeing or permanently consuming the same skb would violate ownership. A driver normally stops a queue before it cannot accept further packets and wakes it after resources return, rather than using BUSY as ordinary flow control.

In include/linux/netdevice.h, v6.12, GPL-2.0-or-later (the mask define and typedef omitted), the transmit result values are:

enum netdev_tx {
	__NETDEV_TX_MIN	 = INT_MIN,	/* make sure enum is signed */
	NETDEV_TX_OK	 = 0x00,	/* driver took care of packet */
	NETDEV_TX_BUSY	 = 0x10,	/* driver tx path was busy*/
};

The enum's own comments describe who holds the skb after the call: OK means the driver took care of it, and BUSY means the driver did not. Neither comment says the peer received a frame. The enum does not show the queue stop and wake calls, or how the core retries after BUSY.

Transmit completion can release descriptors, DMA mappings, skb references and queue accounting. The precise ordering belongs to the driver and device. Byte Queue Limits (BQL), when supported and used, feed byte-based completion and queued-byte accounting into queue control; BQL is not a universal cap for every virtual or physical device. A stopped queue must be woken when appropriate, and teardown must prevent a completion or interrupt from using freed rings. The driver must synchronize NAPI, interrupts and outstanding DMA before releasing device state. A network-device unregister is not permission to immediately free a page that another skb still references.

Aggregation, segmentation, and steering#

GRO combines suitable receive packets for later processing. Generic segmentation offload (GSO) represents a larger packet that will later be segmented, while TCP segmentation offload (TSO) delegates TCP segmentation to capable hardware. A local capture before segmentation can show a large skb although the NIC emits multiple MTU-sized frames. Checksum offload can leave a partial checksum for later completion; a capture at that point can appear to have a bad checksum even when the wire packet is correct. Conversely, a hardware or software checksum error can be real. Record the capture point, offload settings and device capabilities before making a claim about wire packets.

RPS can steer receive processing in software to another CPU; RFS uses flow-related locality to improve processing placement; XPS helps select transmit queues based on CPU or receive-queue mappings where configured. These mechanisms interact with interrupt affinity, RSS hardware distribution, CPU topology and application placement. None guarantees that a particular flow remains on one CPU forever. Queue counters, /proc/net/softnet_stat and /proc/interrupts observe different stages and can include unrelated system traffic. Hexadecimal per-CPU softnet fields and their interpretation are kernel-version dependent; do not label every increment a NIC hardware drop without checking the implementation.

A three-stage example illustrates why counters differ. A NIC receives two small packets, GRO combines them into one later skb, and a socket drops that skb due to receive memory pressure. Depending on device and accounting, the NIC may count two packets, the later stack can process one aggregate, and the socket can report a drop in its own units. This is a conceptual example; it does not prescribe exact counter increments for every driver. TSO is the reverse direction in the sense that one large software representation can yield multiple wire segments, but it is not the same algorithm as GRO.

The completion race in more detail#

napi_complete_done does more than clear a simple Boolean. In the v6.12 core it first refuses completion in certain netpoll-service or busy-poll states. It can flush GRO state and deliver the accumulated normal list, remove the NAPI instance from the poll list under local interrupt protection, and update scheduling state with a compare-and-exchange loop. If another event set NAPIF_STATE_MISSED, the code retains scheduled state, reschedules the instance and returns false. If a configured timeout defers hard interrupts, it can arrange the timer and return a value that tells the driver not to unmask immediately. These details explain why the e1000e code checks the return from napi_complete_done: treating the callback as an unconditional acknowledgement could lose or mishandle work that arrived near completion. Read the exact state and timer branches in net/core/dev.c.

Completion also marks an ownership boundary for teardown. napi_disable() waits for NAPI ownership to be released, but does not necessarily wait for the poll callback itself to return. A driver must therefore avoid accessing private state after napi_complete_done() if teardown can free that state; this is especially important when polling runs in a thread. See the NAPI control API documentation.

In the cited e1000e interrupt handler, the driver reads the interrupt cause and can return IRQ_NONE if the cause is absent, the adapter is down, or the interrupt is not asserted for that path. Its comment describes interrupt auto-masking when the cause register is read. It then schedules the registered NAPI instance if napi_schedule_prep succeeds. Other e1000e interrupt modes and unrelated drivers have their own masking and acknowledgement details. A reader who assumes that every device uses this register or the same handler will build an incorrect race diagram. The relevant invariant is that the device and driver must arrange for outstanding work to be noticed while avoiding uncontrolled repeated interrupts.

In the legacy e1000_intr handler, drivers/net/ethernet/intel/e1000e/netdev.c, v6.12, GPL-2.0 (interrupt-cause handling omitted), the scheduling step is:

	if (napi_schedule_prep(&adapter->napi)) {
		adapter->total_tx_bytes = 0;
		adapter->total_tx_packets = 0;
		adapter->total_rx_bytes = 0;
		adapter->total_rx_packets = 0;
		__napi_schedule(&adapter->napi);
	}

The poll is scheduled only when napi_schedule_prep succeeds, so an interrupt that arrives while the instance is already scheduled does not start a second poll. The excerpt does not show the cause-register read or the masking, which vary by device and interrupt mode.

The budget is a per-poll contract, not a fixed global limit on all work done by the CPU. Receive processing consumes the callback's reported budget according to the driver contract; completion cleanup and special modes require reading that driver. A return equal to the budget tells the core that the instance can require another poll, even if the final descriptor happened to exhaust the ring exactly at that boundary. A driver must follow NAPI's rules for that ambiguous case instead of both completing the instance and claiming that the full budget was consumed. A return below budget and a true completion result are separate observations. Busy polling can alter who drives the next poll and whether interrupts should immediately be re-enabled.

Queue ownership under load#

Consider a transmit ring with two free descriptor slots and a driver that needs one slot per small packet in this simplified example. It accepts packet A and B, updates its producer state, and stops the netdev transmit queue before accepting a third packet if the device has no usable space. A device completion later lets the driver reclaim descriptors, finish the appropriate DMA mappings, release skb references and wake the queue. The exact order and number of descriptors per packet are hardware-specific; the model is useful because it exposes the ownership boundary. If an skb is accepted, the qdisc cannot continue treating it as an unconsumed buffer; if the driver returns NETDEV_TX_BUSY, it must not have consumed the skb.

A queue stop/wake bug can look like packet loss to an application even when no individual transmit operation reports a permanent error: packets may remain queued while the device becomes idle. Conversely, waking a queue before ring resources are safe to reuse risks concurrent reuse of DMA descriptors. BQL can help prevent excessive bytes from accumulating below the qdisc when a driver uses the accounting API correctly, but it cannot fix an incorrect completion handler or guarantee fairness between flows. Inspect per-queue counters, driver state and the timing of stop, completion and wake events before assigning blame to the qdisc.

Steering changes where work runs, not the packet's ultimate correctness contract. RSS can distribute hardware receive flows among queues before software sees them; RPS can enqueue processing to another CPU; RFS adds a locality-oriented mapping; XPS can influence transmit queue selection. CPU affinity and migration can make these mappings change over time. An increment in a softnet backlog-drop field, a driver receive-miss counter and a socket receive-buffer drop refer to different points in the pipeline. Compare deltas over a known interval and identify the field's v6.12 definition before concluding where a particular packet was lost.

One receive batch, three observation points#

The e1000e poll function obtains its adapter from the napi_struct, optionally cleans completed transmit work, and calls the configured receive-cleaning method with the budget and a work_done accumulator. When either transmit cleanup is incomplete or receive work reaches the budget, the function returns the budget. Otherwise it attempts completion and conditionally re-enables interrupts. The driver-specific callback is e1000e_poll; the core's __napi_poll does not itself know that the device is e1000e or directly clean its ring. That separation lets another driver register a different callback while following the common NAPI contract.

Now suppose the device reports four received frames and a GRO path aggregates two compatible packets. A hardware or driver receive counter can count the four frames, the NAPI callback can report work in its driver's units, and a later observer can encounter fewer skb objects. A TCP socket may then consume bytes from an aggregated representation. None of these numbers is necessarily wrong. To investigate a mismatch, capture deltas at each specific point with offload configuration and time interval recorded; do not infer dropped frames from unequal counts alone. Interrupt counts measure notifications, not packets, and interrupt moderation can further change their relationship.

Practice#

Source-only route:

rg -n -A65 'static int __napi_poll\(' "$KERNEL_SRC/net/core/dev.c"
rg -n -A35 'napi_complete_done\(napi, work_done\)' "$KERNEL_SRC/drivers/net/ethernet/intel/e1000e/netdev.c"
rg -n 'napi_gro_receive|gro_normal_list' "$KERNEL_SRC/net/core/gro.c" "$KERNEL_SRC/net/core/dev.c"

Confirm a v6.12 checkout. Trace the generic budget branch and e1000e's conditional interrupt re-enable; name which steps are driver-specific. Draw ownership for an skb accepted by a driver versus one returned NETDEV_TX_BUSY.

Read-only observation, with no packet generation:

if command -v ip >/dev/null 2>&1; then ip -s link show; else echo 'ip unavailable'; fi
if [ -r /proc/net/softnet_stat ]; then head -4 /proc/net/softnet_stat; else echo 'softnet statistics unavailable'; fi
if [ -r /proc/interrupts ]; then head -8 /proc/interrupts; else echo 'interrupt statistics unavailable'; fi
if command -v ethtool >/dev/null 2>&1 && [ -d /sys/class/net/lo ]; then
  ethtool -k lo || echo 'features unavailable or not permitted'
  ethtool -l lo || echo 'channel information unavailable or not permitted'
  ethtool -S lo || echo 'driver statistics unavailable or not permitted'
else echo 'ethtool or loopback unavailable'; fi

ip -s normally reports packet, byte, error and drop counters for interfaces; they are cumulative snapshots, and concurrent traffic changes them. The softnet file contains per-CPU hexadecimal fields, while interrupt output is architecture/environment dependent. Loopback may not implement ethtool queries, a container may hide physical devices, and permission failures are valid observations. Do not enable or disable features. Deliver the budget/IRQ diagram, a NETDEV_TX_BUSY ownership explanation and either the counters or explicit unavailable results.

Knowledge check#

1. Why does __napi_poll check scheduled state before calling the callback?
2. When does the shown e1000e poll branch re-enable its interrupt?
3. What is the ownership difference between NETDEV_TX_OK and NETDEV_TX_BUSY?
4. Why can a capture show one large skb even when the wire uses multiple frames?
5. Why can a page-pool page not be recycled as soon as a receive descriptor is cleaned?

Answers#

1. The core's comment identifies a race with netpoll; only the entity that sees the scheduled state under the applicable synchronization invokes the poll method.
2. Only after its work conditions allow completion, napi_complete_done returns true, and the device state permits re-enabling the relevant interrupt.
3. OK transfers skb consumption responsibility to the driver; BUSY leaves the skb unconsumed for the core's retry handling.
4. GSO/TSO or GRO can change the software representation at the capture point relative to wire segmentation.
5. An skb or other consumer can still hold the page, and DMA ownership and synchronization must be settled before reuse.

Checkpoint#

Save the poll timeline, transmit ownership diagram and read-only observations. Pass when you can explain the full-budget case, completion/re-enable condition, buffer lifetime and offload observation limits. If a tool or physical NIC is missing, complete the source route and document the limitation. This is practice, not certification.

Sources#

• NAPI and netdev core, Linux v6.12.
• Network device structures and operations.
• e1000e interrupt and poll implementation.
• GRO implementation.
• Page pool implementation.
• NAPI documentation.