Linux,
inside out.

This book guides a computer science graduate from basic Unix and C through Linux kernel internals, userspace services, desktops, applications and Wine toward independent systems investigation. It is a long apprenticeship, not a certification or a guarantee of expertise. The similarly named Linux From Scratch project teaches readers to build a Linux distribution; this is a separate Linux kernel curriculum, not its official guide. No finite book covers every driver, architecture, application, configuration, or release.

Baseline and evidence#

The kernel historical teaching baseline is upstream Linux v6.12, with versioned documentation. It is not a recommendation to deploy an unpatched kernel. A distribution may backport fixes and behavior without changing the apparent upstream version; a current kernel may have moved or replaced the code. Userspace and Wine projects have separate independently pinned source versions; they are not a jointly tested compatibility matrix. Pin tools separately when an exercise needs a particular version. When comparing another version, use Git to inspect the change and read its surrounding configuration rather than assuming names and paths are unchanged.

An annotated source path establishes what that version can do under stated conditions. It does not establish which path your machine executed. A runtime trace describes one observed execution, with its instrumentation, workload and losses; it does not establish all possible executions. Separate direct evidence, inference, synthetic examples and unanswered questions in your notes. The chapter diagrams are teaching models, not captured executions unless explicitly stated. The figure below sorts the evidence kinds used in this curriculum and what each can and cannot show.

What makes a Linux system#

Linux is the kernel: it manages processes, memory, devices and interfaces used by programs. A shell, compiler, C library and utilities are userspace programs; many systems use GNU components, but neither GNU nor any particular shell is the kernel. The C library often supplies the convenient function that prepares a system call and translates a negative kernel error into userspace errno. A distribution selects a kernel, configuration, patches, libraries, package manager, services and update policy. Firmware and a bootloader prepare a machine and hand control to the kernel according to an architecture-specific contract; they are not interchangeable with PID 1, the first userspace process.

This boundary matters when debugging. A shell's command not found, a dynamic loader's missing-library error, a rejected system call, and a device driver's error may look like one failed command, yet require different evidence. Identify the layer and version before attributing a failure to Linux itself.

A multi-year study plan#

These are planning ranges for roughly 5–10 focused hours per week, not promises. The original kernel phases total approximately 25–50 months and 1,090–2,150 hours; the userspace and Wine phases add roughly 24–48 months and 1,040–2,050 hours; overlap, prior systems work, interruptions and hardware availability change elapsed time. Some investigations take far longer. A source-only learner can complete reading gates while recording runtime work as pending.

Read the first four parts in order. Later, specialize and revisit foundations when a dependency is unclear. The kernel research atlas and userspace project atlas are starting maps for choosing a next question, not a complete list of Linux expertise.

Prerequisite matrix#

Weekly study routine#

A practical five-session cycle is read, trace, experiment, explain, review. First read the chapter and source with unfamiliar terms recorded. Next choose a single operation and follow a normal path and a meaningful error branch; record the configuration and how any indirect call is selected. Then run a bounded exercise appropriate to your route, or analyze the supplied source-only example. Write down expected results before looking at observations. Explain the path in your own words and draw the objects and their owners. Finally answer the knowledge check without the text, compare your explanation to the cited source, and revise the notebook with unresolved questions. An hour of careful tracing can be more useful than a day of collecting names.

Gates at part boundaries#

Each gate asks for an artifact and an explanation to another person or to your future self. Use the chapter checkpoints for detailed criteria. If you cannot explain a dependency, return to its prerequisite, work a smaller example and retry; do not mark a failed runtime experiment as a pass merely because source reading was possible.

These are evidence gates, not examinations administered by kernel maintainers. Source-only completion demonstrates source analysis under the stated assumptions; it does not demonstrate a boot, a measured result, operational competence, or expert qualification.

Choose a lab route#

Source-only requires a browser or a checkout of the tagged source and time to draw and explain paths. Supplied worked cases provide evidence even without a compiler or VM. Own-host read-only exercises use ordinary permissions, your own short-lived processes and observations such as /proc; results describe your host kernel, not automatically v6.12. Avoid exposing other users' process details in shared notes.

Disposable guest work uses an x86-64 QEMU guest, a separate kernel, initramfs or disk image, and an ordinary unprivileged host account. Label host and guest terminals, prompt, working directory and image paths before every command. Never give a physical disk, host mount, host firewall, host sysctl or host kernel module to a destructive exercise. Keep a clean image copy and retain logs; stop the VM and discard only lab copies to recover. Filesystem mutation, networking changes, crashes, fuzzing, hotplug and module insertion belong in a disposable guest or disposable image. A guest is not a substitute for backups or a security boundary against a hostile workload.

Chapter 08 specifies packages, resources and a reproducible intended build and boot. Its v6.12 kernel has not been successfully booted in the recorded author environment. Attempts using GCC 16 failed on C23-related build errors with both EFI and non-EFI configurations; the documented Ubuntu 24.04/GCC 13 alternative remains unverified here. Preserve the exact error, config and tool versions if your attempt fails; do not assume the VM will start immediately. No PID 1 or guest /proc output is claimed as observed by the authors.

Wine route: use only self-authored test programs in a new private prefix when the relevant tools are available; a Wine prefix is not a security sandbox. The authors did not run Wine, Windows conformance or the PE labs. Record a source-only result when those tools are unavailable.

Hardware and architecture contrast may require a suitable board, serial access, recovery media and device-specific documentation. arm64 source can be studied without hardware; an arm64 guest needs its own build, machine and firmware choices. Check access and recovery before trying hardware. Build time, disk space and RAM depend on configuration and parallelism; chapter 08 gives concrete starting budgets. A C toolchain, Git and optionally QEMU, BusyBox and readelf support different exercises; none is required for the source-only route.

Reading the code and keeping a notebook#

Start at a named interface and locate its definition, callers and relevant data structure. For each call, check the active Kconfig option, architecture, compile-time alternatives and runtime state. Follow function-pointer initialization to a specific operations table before claiming a target; macros, inline functions, generated files and static calls make a text search incomplete. Read both the callee and its callers, then trace resource acquisition and unwind in reverse order. Compare tags with Git history when a symbol is absent. A search returning no matches is a question about scope and spelling, not proof that a behavior cannot happen.

Use this notebook template for every investigation:

Question and expected observation:
Source tag/commit, files and symbols:
Architecture, CONFIG options, compiler, boot arguments and tool versions:
Caller, execution context and indirect-call selection:
Objects, owners, references, locks and lifetime:
Normal path and conditional branches:
Error, cancellation or teardown path:
Evidence: source / synthetic model / actual command and captured output:
Interpretation, limits, competing explanation and next check:

For example, ask why an invalid descriptor fails on a read. At v6.12 inspect ksys_read in fs/read_write.c, record how a descriptor is looked up and what happens if lookup fails, and distinguish the caller's descriptor number from the referenced open-file object. State whether you only inspected source or ran a userspace program, and capture that program's host kernel and return value. A trace of one failed call cannot establish the behavior of all filesystems or all versions. As you learn locking and references, add their exact lifetime and context rather than retroactively calling an early sketch complete.

How to read and measure progress#

Each chapter has prerequisites, goals, a source walkthrough, practice, questions, answers, a checkpoint and sources. Use search and the glossary to return to a term; the outline follows chapter headings. Theme, reading size, focus mode and completion marks are stored locally in your browser. A completion mark is a bookmark, not proof of competence. Printing produces a paper-friendly chapter view. For further work consult the kernel development process guide and the relevant subsystem maintainers' documentation, then verify against the version you actually use.