execve, ELF loading, and process images

Before you start#

Read system-call entry and user copies, ELF and the ABI, and init and userspace. This chapter reads upstream v6.12; the optional host program is synthetic and runs only as your ordinary user. It creates no setuid executable, needs no privileges, and does not replace your shell because the shell starts it as a separate process.

Learning goals#

Follow an execve request through filename, binary-parameter state, handler selection and ELF or script loading. Explain process identity, address-space and descriptor changes; distinguish the kernel ELF loader from the userspace dynamic linker; and identify error and cleanup behavior on both sides of the point of no return.

What exec changes#

execve asks the kernel to replace the calling process's program image. A successful call does not return to the old program's next instruction. The calling process does not need to call fork first: a shell commonly forks a child and the child then executes a command, but another program can replace itself directly. The process identity visible as PID normally remains the same across a successful exec. In a multithreaded process, exec has important thread-group effects: other threads are removed and the executing thread becomes the surviving process image under the kernel's exec rules. Do not assume each old thread continues into the new program.

The new image has a different address-space layout, program mappings, stack and execution entry. Arguments and environment strings are copied under limits and become inputs to the new program. Some process attributes persist, while others are reset or recalculated. Descriptors normally remain open unless marked close-on-exec; signals, credentials and other attributes have their own rules. This is not a complete portable list. The system-call ABI and loader behavior depend on architecture, binary format, configuration and security policy.

A pathname is resolved in the caller's filesystem view. Execution needs a suitable file and permission under the applicable credentials, mount and security rules. The file can be an ELF binary or a script handled by an enabled binary-format handler. A file existing at a path is not proof that it is executable, for the correct architecture, or accompanied by its interpreter. See v6.12 fs/exec.c and v6.12 fs/binfmt_elf.c.

Source walkthrough#

do_execveat_common in fs/exec.c receives a filename and userspace argument descriptions. It handles an error-encoded filename before proceeding, allocates and prepares a struct linux_binprm, copies arguments, and invokes bprm_execve. The linux_binprm object carries information about the candidate binary, including the opened file, a header buffer, argument state, credentials and execution progress. It is kernel-owned preparation state, not a durable userspace handle.

The following excerpt is from do_execveat_common in upstream v6.12 fs/exec.c (SPDX GPL-2.0-only; surrounding error checks omitted):

retval = bprm_execve(bprm);
out_free:
	free_bprm(bprm);

The call attempts the binary execution path and returns a status to this code when applicable. free_bprm releases the preparation object and resources owned by it on this exit route; additional cleanup, including the filename reference, follows. The out_free label is also reached by earlier failures after allocation. This excerpt does not imply that all failures occur before the old image has been dismantled or that a successful exec returns to old userspace. Resource ownership changes as the operation advances, so cleanup must match the stage reached.

Inside bprm_execve, the kernel prepares credentials, checks unsafe execution state, calls security hooks and invokes exec_binprm. The binary-handler search examines registered handlers and the header buffer. The script handler in v6.12 fs/binfmt_script.c checks for #!, parses an interpreter path and arranges for that interpreter to be opened. The following excerpt from load_script is upstream v6.12 (SPDX GPL-2.0-only; surrounding parsing and argument manipulation omitted):

file = open_exec(i_name);
if (IS_ERR(file))
	return PTR_ERR(file);

The interpreter name is not a function pointer into the script. open_exec attempts to obtain a suitable executable file; an error pointer is recognized and its encoded error returned rather than dereferenced. On success the handler stores the interpreter file in bprm and the exec machinery continues with it. A script can be present and executable while this open fails because its interpreter does not exist in the caller's mount namespace. A CRLF line ending or malformed shebang can also alter the interpreted path or format; inspect the actual bytes and applicable handler rules rather than guessing from a shell message.

ELF segments and the userspace loader#

For an ELF candidate, load_elf_binary checks the magic, supported object type and architecture, and reads program headers. PT_LOAD entries describe segments to map with appropriate permissions and offsets. PT_INTERP, when present and accepted, names a program interpreter such as the dynamic loader. The kernel opens and validates that interpreter and establishes the program image according to the selected architecture and ELF rules. ELF sections are useful to linkers and debuggers, but program headers guide the loadable image; the new process does not receive one separate mapping per section merely because a section table lists it.

The kernel also constructs the initial userspace stack with argument and environment strings and an auxiliary vector, or auxv. The auxv supplies machine- and image-related information to the new userspace runtime under the ABI. Stack layout, alignment, word size and architecture-specific entries are ABI-dependent. Address-space randomization can change chosen addresses when enabled and applicable; one observed address is not a portable constant or proof that randomization is active. The PT_INTERP program interpreter receives control first and is responsible for completing the dynamically linked program’s runtime setup. How the executable and interpreter segments are mapped and represented at that point follows the ABI and architecture implementation; do not treat their names as one object. See the System V ELF ABI program-header and dynamic-linking rules.

The dynamic linker is userspace code. For a dynamically linked program, it maps or locates required shared objects and processes dynamic relocations according to its own implementation and policy. It helps arrange thread-local storage, applies relevant protections such as RELRO where supported, and runs startup routines and constructors before the application reaches its normal main. The exact order and features depend on the loader and binary; Linux's kernel does not implement every userspace library's initialization sequence. A static binary can avoid PT_INTERP, but may still have application-level runtime dependencies.

This boundary explains a common failure. Suppose /lab/app exists and has execute permission, but its PT_INTERP names /lib64/ld-linux-x86-64.so.2, which is absent from the process's root. The kernel can return ENOENT while opening the interpreter. A shell may report “No such file or directory” for /lab/app even though the application file is present. readelf -l can reveal the interpreter path without executing the file. Conversely EACCES can result from file permissions, directory search, a noexec mount or security policy; ENOEXEC can indicate an unrecognized or unsuitable executable format. These are diagnostic hypotheses, not a universal one-to-one mapping from message to cause.

Credentials, descriptors and the point of no return#

Exec involves credential preparation and security checks. Set-user-ID or file capabilities can affect the resulting credentials under their specific rules, but mount flags, no_new_privs, tracing, LSM policy and other conditions can prevent or constrain such transitions. This chapter's experiment does not create privileged files or try those transitions. It is incorrect to assume that executing a file automatically grants its owner identity or that a successful file open grants authority to execute it.

Before the point of no return, a failed preparation can generally unwind acquired resources and report an error to the old program. This lets it handle a missing file, a bad format or another early error. During begin_new_exec, the implementation crosses a point after which the old process image cannot safely continue. The v6.12 bprm_execve error path checks bprm->point_of_no_return; if an error occurs after that point and no fatal signal is already pending, it forces a fatal signal. Thus “exec is transactional” is only a useful approximation for earlier failures, not a promise that every late failure restores the old image. Inspect free_bprm and the appropriate error labels for the resources owned at each stage.

In fs/exec.c, v6.12, SPDX GPL-2.0-only (surrounding code omitted):

	/*
	 * Ensure all future errors are fatal.
	 */
	bprm->point_of_no_return = true;

This line sets the flag that bprm_execve checks later. The excerpt does not show the state changes made before it, and those determine what can no longer be undone.

In fs/exec.c, v6.12, SPDX GPL-2.0-only (the success path before out: omitted):

out:
	/*
	 * If past the point of no return ensure the code never
	 * returns to the userspace process.  Use an existing fatal
	 * signal if present otherwise terminate the process with
	 * SIGSEGV.
	 */
	if (bprm->point_of_no_return && !fatal_signal_pending(current))
		force_fatal_sig(SIGSEGV);

When the error happens after the commit and no fatal signal is pending, the kernel forces SIGSEGV instead of returning to the old program. The excerpt does not show the cleanup that follows, and earlier failures return normally.

File descriptors show another lifetime boundary. A descriptor with FD_CLOEXEC set is closed as part of successful exec processing. Other open descriptors normally continue to refer to their open file descriptions, subject to the kernel's exec and security rules. The file object and position can therefore outlive the old program image. A failed early exec generally leaves the old program able to inspect its descriptors, but a late fatal failure must not be treated as an ordinary recoverable return. Descriptor numbers can be reused by the dynamic loader or new program after exec; a numeric fd is not a permanent identity for an open object.

Threads and descriptor-table boundaries#

An exec performed by one thread must coordinate with the other members of its thread group before the new single program image can proceed. A descriptor table might also have been shared with another task through clone flags. The kernel's exec path has to handle those relationships before close-on-exec processing can safely apply to the executing image. This is why a superficial explanation that simply “clears a few pointers and jumps to main” misses important synchronization and ownership work. It also explains why a program should not use the close of a descriptor in one context as proof that every other reference to the underlying open file has disappeared. Follow the actual begin_new_exec and file-table paths for a particular version and sharing configuration.

A worked image replacement#

Suppose a single-threaded process has PID 400 in its namespace and opens a temporary file as descriptor 3. It sets FD_CLOEXEC on descriptor 3 and calls execve on a valid dynamically linked program. During preparation the kernel holds references to the executable and linux_binprm state. After successful format loading and the irreversible transition, close-on-exec closes descriptor 3 and the new image begins through its interpreter. The PID observed by the new program remains 400 in that namespace. The old main never continues. A loader could later open a file and reuse descriptor number 3, so seeing a live descriptor 3 after startup would not by itself prove the original descriptor survived.

Change one condition: the interpreter named by PT_INTERP is absent. If the failure occurs during the early preparation path, execve returns an error to the old image, the old program can report ENOENT, and normal cleanup releases the candidate's temporary references. This is a conditional worked example; resource limits, tracing, security policy and exact format can change the outcome. In a different late failure after the point of no return, the old program is not guaranteed to recover and the kernel's fatal-signal path matters.

Practice#

Source-only route: make a three-column timeline for the worked example: preparation before the point of no return, transition, and new-image startup. Place the opened executable, linux_binprm, descriptor 3 and dynamic interpreter in their relevant stages. Add the missing-interpreter alternative and identify which conclusion is conditional on the failure occurring before the point of no return. This is source-based evidence, not a captured execution.

Host own-process route: create a private directory and compile this synthetic program. It opens a temporary file, marks the descriptor close-on-exec, and executes a new image of itself once. The shell that launched it is not replaced. The file is unlinked immediately, so no data is left in /tmp if the process succeeds or fails after that point.

lab_dir=$(mktemp -d) || exit 1
cd "$lab_dir" || exit 1
cat > observe.c <<'C'
#define _POSIX_C_SOURCE 200809L
#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
int main(int argc, char **argv) {
    if (argc == 3 && strcmp(argv[1], "after") == 0) {
        int fd = atoi(argv[2]);
        errno = 0;
        int result = fcntl(fd, F_GETFD);
        printf("after exec: pid=%ld fd=%d result=%d errno=%d\n",
               (long)getpid(), fd, result, errno);
        return 0;
    }
    char path[] = "/tmp/exec-observe-XXXXXX";
    int fd = mkstemp(path);
    if (fd < 0) { perror("mkstemp"); return 1; }
    if (unlink(path) < 0) { perror("unlink"); close(fd); return 1; }
    if (fcntl(fd, F_SETFD, FD_CLOEXEC) < 0) { perror("fcntl"); close(fd); return 1; }
    char number[32];
    snprintf(number, sizeof number, "%d", fd);
    printf("before exec: pid=%ld fd=%d\n", (long)getpid(), fd);
    fflush(stdout);
    execl(argv[0], argv[0], "after", number, (char *)NULL);
    perror("execl");
    close(fd);
    return 1;
}
C
cc -std=c11 -Wall -Wextra -Werror -o observe observe.c || exit 1
./observe
readelf -h ./observe
readelf -l ./observe | grep -A 1 INTERP || true

Typically the two printed PIDs match and fcntl after exec reports -1 with EBADF; exact numbers depend on the host. A descriptor number can be reused during startup, so an unexpected successful fcntl is not sufficient evidence that the original file survived. The output is a host observation, not evidence that the host ran v6.12 code. readelf may show INTERP for a dynamic executable, or none for a static build. If compilation, temporary-file creation or exec fails, preserve the diagnostic and don't claim the second image ran.

As a counterexample, create an executable script naming a deliberately absent interpreter in the private directory, then execute it:

printf '#!/definitely-absent-teaching-interpreter\n' > missing-interp
chmod 700 missing-interp
./missing-interp
printf 'status=%s\n' "$?"

Expect failure despite the script existing; wording and exit status are shell-dependent. Verify ls -l missing-interp and inspect its first line before assuming the script path itself is missing. Do not create the interpreter or modify host loader files. Save observations and remove only the created files:

rm observe observe.c missing-interp
cd /tmp
rmdir "$lab_dir"

Knowledge check#

1. Why does successful execve normally preserve a PID yet not return to the old program?
2. What does the quoted script-loader open_exec failure tell you about a missing interpreter?
3. Why is the dynamic linker a separate component from the kernel's ELF loader?
4. What changes after the point of no return if a later exec stage fails?
5. Why is a live descriptor with the old number after exec not conclusive evidence that the original open file survived?

Answers#

1. Exec replaces the calling process's image rather than creating another process. The new image begins under the same process identity in the relevant namespace, while the old instructions no longer resume on success.
2. The handler attempts to open the interpreter and returns the encoded error if that fails. The script itself may exist even though the interpreter path does not.
3. The kernel validates and maps the executable and interpreter under its ELF rules; userspace loader code then handles shared libraries, dynamic relocations and runtime startup according to that loader's implementation.
4. The old image cannot necessarily be restored. The v6.12 error path checks point_of_no_return and can force a fatal signal instead of returning normally to the old program.
5. Close-on-exec closes the original descriptor, but the new runtime can open another object and reuse its integer number. An object identity or reference must be tracked separately from the number.

Checkpoint#

Deliver the three-stage timeline and either the source-only interpreter failure analysis or the host output. Pass when you distinguish PID from image, describe close-on-exec and reuse, and explain one early and one late error path. If uncertain, retrace do_execveat_common, load_script and the point_of_no_return branch, then repeat only the safe private-directory experiment. This is not certification.

Sources#

• v6.12 fs/exec.c: preparation, handler search and error cleanup.
• v6.12 fs/binfmt_script.c: interpreter parsing and open.
• v6.12 fs/binfmt_elf.c: ELF validation and mapping.
• v6.12 ELF UAPI definitions.
• Linux userspace API documentation.