Routing, neighbours, policy, and netlink

Before you start#

Read UDP, IPv4, and IPv6. Source references use upstream Linux v6.12, a historical teaching baseline. Routing tables, policy rules, neighbor entries and netlink permissions are network-namespace and configuration dependent. The lab is read-only and sends no packet to its test destination.

Learning goals#

Follow policy and FIB lookup, distinguish a route from neighbor reachability, explain netlink message validation and dump semantics, and reason about a gateway-resolution failure without changing host routes.

Source walkthrough#

A route lookup answers how to send a packet under a flow's context: destination, source where supplied, output interface, mark, type of service, namespace and applicable policy. A forwarding information base (FIB) stores routes and associated next-hop information. Policy rules can select tables or other actions before a particular table's longest-prefix match is evaluated. A default route is a /0 match in a table; it is not automatically selected if a higher-priority rule selects a different table or returns an error. Source address selection interacts with route and interface configuration; the destination alone does not fully determine the chosen source on every machine.

For IPv4 output, net/ipv4/route.c builds a flowi4 and invokes fib_lookup on relevant paths. In policy-enabled builds, __fib_lookup, Linux v6.12, GPL-2.0-or-later, invokes the rules machinery:

	err = fib_rules_lookup(net->ipv4.rules_ops, flowi4_to_flowi(flp), 0, &arg);

The network namespace supplies the IPv4 rule operations, the flow supplies lookup keys, and the result/error is written through the lookup argument. Later code translates -ESRCH to -ENETUNREACH in this function. Rule actions can call fib_table_lookup for a selected table; the IPv4 implementation is in net/ipv4/fib_trie.c. A route can be local, unicast, unreachable or another type; an explicit output interface and special route cases can change normal lookup behavior. These details mean the one-line excerpt is a policy boundary, not a complete route lookup.

IPv6 has separate flow and route machinery. net/ipv6/route.c includes ip6_route_output_flags, fib6_rule_lookup calls and fib6_table_lookup. Do not assume the IPv4 trie implementation is the IPv6 implementation merely because both perform prefix selection. IPv6 source-address selection also considers address scope, availability and policy; inspect the actual environment. A dst_entry represents cached route/destination state and carries lifetime rules. Some fast paths use RCU-protected pointers and others take references; a route can be invalidated or replaced as configuration changes. A pointer seen under RCU cannot be retained indefinitely without the required reference or protection. A cache is an optimization, not proof that the route will never change.

Worked policy and prefix selection#

Consider a synthetic namespace with these rules and routes:

• Priority 100: packets with mark 7 consult table 100.
• Table 100: 198.51.100.0/24 via gateway 192.0.2.1 on interface A.
• Main table: 198.51.100.0/25 via interface B, plus a default route.
• A later rule consults the main table if the earlier rule does not produce a final result.

A packet to 198.51.100.42 with mark 7 matches the priority-100 rule. If table 100's /24 route is usable and returns a final unicast result, it selects gateway 192.0.2.1 on A. The /25 in the main table is longer, but it is not compared globally against a successful result in a different table selected by the earlier rule. Without mark 7, the main table can select its /25. A rule can instead continue, reject or select another action; this simplified example specifies its own assumptions. The TEST-NET addresses are illustrative and the example does not transmit to them.

A gateway route means the link-layer destination usually corresponds to the next-hop gateway, not directly to the final IP destination. An on-link route can instead require neighbor resolution for the destination itself. A usable FIB result is only the first step: interface state, neighbor discovery, link-layer transmission and the remote network can still fail. A route lookup cannot prove that a remote server is reachable.

Neighbour state and packet lifetime#

A struct neighbour represents protocol-to-link-layer reachability state associated with a device and neighbor table. IPv4 commonly uses ARP; IPv6 uses Neighbor Discovery messages carried by ICMPv6. The neighbor state machine includes NUD_INCOMPLETE, REACHABLE, STALE, DELAY, PROBE and FAILED, as well as other states. STALE can still have a usable link-layer address; it is not synonymous with a failed TCP connection. A reachable neighbor can become stale, enter delay/probe when used, and be refreshed or eventually fail depending on timers and configured probes.

In __neigh_event_send, v6.12, GPL-2.0-or-later, the code takes neigh->lock with bottom halves disabled and can start resolution:

			WRITE_ONCE(neigh->nud_state, NUD_INCOMPLETE);
			neigh->updated = now;

It records an incomplete state and update time before arranging a probe timer on that branch. The surrounding code checks current state, dead objects and probe configuration. If no applicable probes are configured in another branch, it marks the neighbor failed, unlocks and frees the skb with a failure reason. When resolution is incomplete, packets may be queued within configured limits; excess queued skbs can be dropped. The neighbor lock protects state and queue operations in that path; it does not make a caller's unreferenced neighbor pointer valid forever. neigh_resolve_output and neighbor reference-release rules matter at the caller boundary.

Extend the earlier synthetic example: the FIB chooses gateway 192.0.2.1, but A has no usable neighbor entry for it. The stack initiates ARP resolution and can queue a packet while waiting. If replies arrive and validate, the neighbor entry can become usable and queued traffic can proceed. If configured probes fail, the entry becomes failed and queued packets can be dropped or errors propagated according to the path. The route can still be present in the FIB throughout. A neighbor entry's state concerns link-layer next-hop reachability, not an end-to-end TCP handshake or application health.

Netlink: messages, validation, and snapshots#

Netlink provides socket-based structured messages between userspace and the kernel. NETLINK_ROUTE carries route, link, address and neighbor operations and notifications. A struct nlmsghdr contains length, type, flags, sequence number and port ID. A netlink port ID identifies a socket endpoint in its namespace; although it can resemble a process ID, an application must not assume it is always the PID. Sequence numbers help correlate requests and replies, while asynchronous notifications can arrive independently. Port ID, sequence and sender address must be interpreted according to the message and socket API, not assumed to authenticate arbitrary payloads by themselves.

Attributes are typed length-prefixed records. Their lengths and alignment must be checked before advancing through a buffer; nested attributes need their own bounds and policy checks. The kernel parser also validates messages, but a userspace parser must validate the bytes it receives rather than trusting an advertised length. rtnl_linkprop, Linux v6.12, GPL-2.0-or-later, includes:

	err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFLA_MAX, ifla_policy, extack);
	if (err)
		return err;

The parser receives the fixed header size, attribute table, maximum type, policy and extended-ack context. If validation fails, the handler returns the error before using parsed attributes. This specific handler is an example of validation, not the implementation of every route or link request. At the outer dispatch boundary, rtnetlink_rcv_msg checks message type and a minimum payload length; non-GET operations normally require CAP_NET_ADMIN under that code's namespace capability check. GET visibility and individual operation permissions still depend on the request and environment.

In net/core/rtnetlink.c, v6.12, GPL-2.0-or-later (the handler lookup and dump branch are omitted), the outer dispatch checks are:

	type -= RTM_BASE;

	/* All the messages must have at least 1 byte length */
	if (nlmsg_len(nlh) < sizeof(struct rtgenmsg))
		return 0;

	family = ((struct rtgenmsg *)nlmsg_data(nlh))->rtgen_family;
	kind = rtnl_msgtype_kind(type);

	if (kind != RTNL_KIND_GET && !netlink_net_capable(skb, CAP_NET_ADMIN))
		return -EPERM;

A message too short for its rtgenmsg header returns success without doing anything, and a non-GET request must hold CAP_NET_ADMIN in the socket's namespace. GET requests skip that capability check, so reads and writes are gated differently. The excerpt does not show the per-handler validation, which still happens after this point.

An NLM_F_ACK request can receive an NLMSG_ERROR message whose embedded error is zero for an acknowledgement or negative for an error. Multipart dumps use NLM_F_MULTI and finish with NLMSG_DONE or an error. A receiver must iterate all messages in each datagram, check lengths and alignment, verify the sequence for its request, and handle error and termination messages. NLM_F_DUMP_INTR signals an interrupted/inconsistent dump where a fresh dump may be necessary. Even a completed dump is not a transactionally frozen history of all concurrent network changes: routes, links and neighbors can change while the reader processes results. Notifications can help a client track changes, but recovery from missed messages or interrupted dumps needs resynchronization.

Concurrency, namespaces, and source selection#

Routes and neighbor entries can be replaced while packets and readers are in flight. RCU enables certain readers to inspect structures without a heavyweight lock on every lookup; writers and reference counting must still ensure that storage remains alive until readers finish. A neighbor object's reference is a lifetime claim, while its lock protects fields and queue state; holding one is not automatically the other. A packet queued for unresolved neighbor delivery also retains whatever destination and skb state the queueing path requires. Error and queue-limit paths must release it exactly once.

Network namespaces have their own interfaces, routes, rules and neighbor tables. A container's view can be empty or different from the host's, and a process with privileges in one user/network namespace does not thereby have unrestricted host authority. ip route get performs a kernel route lookup for supplied parameters; it does not open a connection or prove that ARP, Neighbor Discovery or remote delivery succeeded. Adding a mark, source or output interface changes the lookup question. For a reproducible investigation, record these parameters and the namespace rather than comparing two unexplained route outputs.

Address selection and route selection can influence each other. A socket bound to a source address or interface constrains the flow; an unbound socket may receive a source address chosen for the selected path. Policy routing can use a source or mark, and IPv6 scope rules can restrict which sources are eligible. The final answer is not always reducible to "longest prefix across the entire machine." Similarly, a local route delivers to the host's own stack rather than resolving a gateway, and point-to-point or tunnel devices can use link-layer behavior different from Ethernet ARP. The synthetic gateway example applies only to the specified Ethernet-like setup.

What to record when a lookup disagrees with expectations#

Start with the namespace and the complete lookup inputs. A route selected for destination 198.51.100.42 with mark 7 and a bound output interface is not necessarily the route selected for the same destination with no mark and no interface constraint. Record rule order, the consulted table, route type, prefix, next hop, output interface and chosen source where exposed. A throw, unreachable or prohibited route can change whether lookup continues or returns an error; do not treat all non-unicast entries as an implicit default route. Policy routing is an ordered decision process, not a single global longest-prefix contest.

Then examine the next-hop state separately. A STALE neighbor can still be used while confirmation proceeds; an INCOMPLETE entry may have packets waiting for resolution; a FAILED state reflects failed resolution under the applicable policy and timing. An empty neighbor table may simply mean no recent traffic has needed neighbor resolution. None of these states proves that a remote transport port accepts connections. A read-only table dump and a synthetic walk are therefore useful even when no network traffic is authorized or the ip utility is unavailable.

A netlink dump as a state machine#

A robust client begins with an aligned request header and a chosen sequence number, then receives one or more netlink datagrams. Each datagram can contain multiple messages, so the parser advances by the aligned message length only after checking that the header and declared length fit in the remaining buffer. It distinguishes data records, NLMSG_ERROR acknowledgements or failures, and NLMSG_DONE. An attribute parser similarly checks each attribute's minimum header size and bounds before examining its payload. The kernel's nlmsg_parse call protects a kernel handler; it does not validate the client's own parsing implementation.

The kernel documentation specifies that NLM_F_DUMP_INTR can appear on any message in a dump, including NLMSG_DONE; clients should retry when it is observed because the result may omit objects. This is stronger than checking the flag only on the terminator. See Introduction to Netlink: dump consistency.

Suppose a route dump yields records A and B, then an interrupted-dump indication. The client may display those records as partial observations, but should not replace its entire authoritative view with them as though no other routes existed. It can discard or mark the partial result and request another dump according to its design. A successful second dump is still a view taken while the system can change. Sequence matching separates replies from unrelated notifications, and the socket's netlink port ID should be learned from its actual binding rather than assumed to equal the process ID. A read-only ip invocation performs this machinery on the user's behalf; its formatted text does not expose every validation decision made by the client.

Practice#

Source-only route:

rg -n -A28 'int __fib_lookup\(' "$KERNEL_SRC/net/ipv4/fib_rules.c"
rg -n -A35 'int __neigh_event_send\(' "$KERNEL_SRC/net/core/neighbour.c"
rg -n -A22 'static int rtnl_linkprop\(' "$KERNEL_SRC/net/core/rtnetlink.c"
rg -n 'fib6_rule_lookup|fib6_table_lookup' "$KERNEL_SRC/net/ipv6/route.c"

Verify the tag is v6.12. Trace the rule lookup, the incomplete/failed neighbor branches, and nlmsg_parse's error return. Work through the mark-7 example and state why the main-table /25 does not override a successful earlier table result. This route is source and arithmetic only; it changes no network state.

If the ip tool is available, run these read-only commands in your current namespace or disposable guest:

if command -v ip >/dev/null 2>&1; then
  ip -details rule show
  ip -details route show table all
  ip -details neigh show
  ip route get 198.51.100.42 || echo 'lookup failed or no route'
else
  echo 'iproute2 unavailable; use source-only route'
fi

The final command asks the kernel for a route lookup to a numeric documentation address; it does not send a packet or contact a remote host. Output depends on policy, namespace and permissions. An empty neighbor table does not prove a route is absent; a route result does not prove the gateway has resolved or that the destination is reachable. A Network is unreachable, permission error or missing ip tool is a valid observation. Do not add routes, create neighbors, change firewall rules or send probes. Deliver the synthetic table selection, the local read-only output or error, and a sketch of the netlink dump completion and interruption handling.

Knowledge check#

1. Why can table 100's /24 win before a /25 in the main table?
2. What address normally needs neighbor resolution when a route uses a gateway?
3. What does NUD_STALE mean, and why is it not equivalent to a failed TCP connection?
4. What does the nlmsg_parse error branch prevent in the quoted handler?
5. Why must a netlink dump client handle NLMSG_DONE, errors and NLM_F_DUMP_INTR?

Answers#

1. Policy rules choose a table/action in priority order; longest-prefix matching occurs within the table actually consulted, and a final earlier result can prevent consulting main.
2. The link-layer next hop, normally the gateway address on that interface, needs resolution rather than the final remote destination.
3. A cached link-layer mapping may still be usable but needs confirmation under the neighbor state machine; it says nothing by itself about an application's TCP session.
4. It returns a parse error before using attributes that failed the fixed-header, length or policy validation.
5. A dump can span messages, end in an error, or be marked interrupted; a client must recognize termination and resynchronize rather than treating a partial set as complete.

Checkpoint#

Save the policy-selection calculation, neighbor failure timeline and read-only lookup output or unavailability. Pass when your explanation distinguishes rule selection, prefix lookup, gateway resolution, netlink validation and end-to-end reachability. Re-read the three source branches and retry the example if any boundary is conflated. This is practice, not certification.

Sources#

• IPv4 route lookup, Linux v6.12.
• IPv4 policy rules.
• IPv4 FIB trie.
• IPv6 route lookup.
• Neighbour state and queues.
• Routing netlink handlers.
• Netlink message implementation.