libc, system calls, threads, and allocation

Before you start#

Review userspace boundaries, system-call entry, tasks and shared resources, and locks. This chapter reads glibc 2.40 at commit 3d1aed874918c466a4477af1da35983ab036690e and musl 1.2.5 at its verified tag object 0784374d561435f7c787a555aeab8ede699ed298. These independent historical releases are examples, not a tested distribution or compatibility matrix.

Learning goals#

Explain a libc wrapper's error convention, thread-local state, the ownership transition during thread creation and joining, and the distinction between a thread cache and a locked allocator arena. Compare selected glibc and musl source without projecting one implementation's internals onto the other. Identify the conditions required before freeing shared application data.

Source walkthrough#

An application calls the C library using its platform's userspace ABI. A wrapper may enter the kernel using a system call, but a library function is not necessarily a system call: allocation can reuse a previously obtained chunk, and a lock can sometimes complete without kernel assistance. On Linux, the kernel convention for a raw error differs from the usual C interface convention of returning -1 and setting errno. The wrapper and architecture-specific support translate that boundary. errno is meaningful after an interface reports failure; a successful call need not clear an old value. pthread functions often return an error number directly rather than setting errno, so consult that interface instead of assuming one rule for everything.

The glibc 2.40 Linux __libc_read implementation carries an LGPL-2.1-or-later notice. The function below is exact; aliases and includes are omitted:

ssize_t
__libc_read (int fd, void *buf, size_t nbytes)
{
  return SYSCALL_CANCEL (read, fd, buf, nbytes);
}

The signed result can hold a count or -1; the descriptor selects an entry in the calling process and the buffer must be writable for the requested transfer. The macro names the operation and carries its three arguments. Its cancellation-aware machinery is significant when a thread has a pending cancellation at a cancellation point: it is not merely an ordinary C call to a function named read in the kernel. Its exact expansion and system-call instruction depend on architecture and configuration. A failed invalid descriptor normally becomes -1 with errno set to EBADF; a short positive read is not an error. If the caller immediately makes another library call, it must save the error first if it needs to report the original cause.

musl 1.2.5's independently implemented src/unistd/read.c calls syscall_cp(SYS_read, fd, buf, count) rather than glibc's macro. The release's COPYRIGHT identifies musl's MIT license; individual files and third-party pieces should still be checked. Both wrappers name a cancellation-aware path, but their internal functions, TLS and allocation designs are not interchangeable. A shared public API does not make the private struct pthread, internal symbol names, or allocator metadata a stable cross-libc ABI. An executable linked for one interpreter cannot generally be assumed to run unchanged with a different loader merely because both systems implement POSIX interfaces.

In musl 1.2.5, src/unistd/read.c, the file has no separate notice; the release's COPYRIGHT file states the MIT license for musl as a whole. Include lines are omitted:

ssize_t read(int fd, void *buf, size_t count)
{
	return syscall_cp(SYS_read, fd, buf, count);
}

The wrapper forwards its arguments to syscall_cp, the cancellation-aware helper named above. The excerpt does not show that helper, its cancellation handling or its error conversion, so it supports only the claim about this one wrapper.

The next exact excerpt is from glibc 2.40 nptl/pthread_create.c, LGPL-2.1-or-later, in the __pthread_create_2_1 error path. The surrounding branch and comments are omitted:

	  pd->setup_failed = 1;
	  lll_unlock (pd->lock, LLL_PRIVATE);

After an omitted upstream explanatory comment, the same branch continues:

	  pid_t tid;
	  while ((tid = atomic_load_acquire (&pd->tid)) != 0)
	    __futex_abstimed_wait_cancelable64 ((unsigned int *) &pd->tid,
						tid, 0, NULL, LLL_SHARED);

This branch is conditional: the new thread ran far enough to exist, but setup of requested attributes failed while a stopped-start protocol was in use. pd is the thread descriptor, not the application's arbitrary data. Marking setup_failed tells the child its startup cannot proceed normally; unlocking releases it from the stopped-start coordination. The creator then repeatedly loads pd->tid with acquire ordering and waits while it is nonzero. A futex wait can wake or return without the desired condition being true, so the loop reloads the predicate. The kernel's clear-child-tid mechanism and glibc's ownership protocol allow the creator to wait for termination before reclaiming the descriptor in this error case. The surrounding code also adjusts thread bookkeeping. It would be unsafe to free pd simply because the initial attribute setup failed while the child might still use it.

From creation to join#

In glibc's Native POSIX Thread Library, pthread_create arranges a descriptor and stack, prepares attributes, and invokes a creation path that uses Linux task creation with sharing flags appropriate for a thread. The actual flag set and fallback vary with kernel support and release; do not substitute a remembered universal clone call for the selected source. A thread control block supports thread-local storage (TLS), the current thread's runtime data, and architecture-specific access conventions. The new thread starts in glibc's start_thread, performs library setup, then calls the application-supplied function. The function's argument is a pointer supplied by the creator; the creator must make sure the pointed-to object remains alive until the new thread no longer accesses it.

The inspected create_thread code builds a clone_args object using flags including CLONE_VM, CLONE_FILES, CLONE_SIGHAND and CLONE_THREAD for the selected sharing, CLONE_SETTLS for the new thread's TLS, and CLONE_CHILD_CLEARTID for termination notification. It supplies addresses in pd, the stack and the thread pointer to __clone_internal; an error return is converted to an error number. These are the fields and flags in this release's path, not a guarantee about every architecture's low-level syscall or every libc. The same source declares a pthread_create symbol version for GLIBC_2_34 and conditional older compatibility symbols. This is versioned binary interface machinery, not a promise that glibc's internal descriptor layout is public or portable.

In glibc 2.40 nptl/pthread_create.c, LGPL-2.1-or-later per its notice, the flag set and clone_args for a new thread are built as follows (surrounding code omitted):

  const int clone_flags = (CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SYSVSEM
			   | CLONE_SIGHAND | CLONE_THREAD
			   | CLONE_SETTLS | CLONE_PARENT_SETTID
			   | CLONE_CHILD_CLEARTID
			   | 0);

  TLS_DEFINE_INIT_TP (tp, pd);

  struct clone_args args =
    {
      .flags = clone_flags,
      .pidfd = (uintptr_t) &pd->tid,
      .parent_tid = (uintptr_t) &pd->tid,
      .child_tid = (uintptr_t) &pd->tid,
      .stack = (uintptr_t) stackaddr,
      .stack_size = stacksize,
      .tls = (uintptr_t) tp,
    };
  int ret = __clone_internal (&args, &start_thread, pd);

The flags match those named above, and CLONE_CHILD_CLEARTID is the kernel mechanism the setup-failure wait relies on. The excerpt does not show how __clone_internal reports failure on each architecture, nor the kernel side of thread exit.

An ordinary joinable thread can finish before its parent calls pthread_join. Its execution has ended, but join-related bookkeeping is retained so that a valid join can obtain the result and release resources. A detached thread has a different reclamation policy and must not be joined as though it were joinable. A successful join provides a synchronization boundary for the joined thread's completed actions under the POSIX interface; it does not repair unrelated data races that occurred earlier. Joining oneself, joining an invalid or already-consumed thread, and failure to create a thread are separate cases. A program must check the returned status and avoid joining an uninitialized handle if creation failed.

The kernel schedules tasks, while libc implements the user-facing pthread abstraction and coordinates user memory. A futex is an interface for waiting on a userspace word when an expected value matches, with a wake operation for waiters. It is a building block, not a complete predicate or a guarantee that the condition is true on return. Consider a queue guarded by a mutex: the consumer checks ready == 0, waits in a loop, and reacquires the mutex before checking again. The producer changes the queue and predicate under the agreed synchronization, then signals. Spurious wakeups and another consumer taking the item make a single if insufficient. The application should use its supported pthread condition-variable interface rather than copying an internal glibc futex sequence.

A thread-local errno location allows two threads to observe different errors without writing the same ordinary global integer. It does not mean every object accessed by the threads is private. A heap pointer, a shared descriptor table, and memory supplied as the thread argument may be shared; the application must synchronize accesses and arrange lifetimes. A descriptor close by one thread can affect operations in another, and descriptor-number reuse complicates reasoning. Record which thread owns a close, cancellation and cleanup action rather than assuming a thread has its own process-wide files.

Allocation is an ownership protocol#

In glibc 2.40 malloc/malloc.c, __libc_malloc initializes the allocator if needed, checks a normalized request, and may return a chunk from the calling thread's cache when configured and when the matching bin is nonempty. The cached fast path does not imply a system call for that request. On a multithreaded fallback, the source obtains an arena, calls _int_malloc, may retry another arena after failure, and unlocks the acquired arena mutex before returning. The single-thread path has separate handling. An arena is allocator state managing chunks, not a separate application object owner or necessarily one arena per thread.

In glibc 2.40 malloc/malloc.c, LGPL-2.1-or-later per its notice, the thread-cache check in __libc_malloc returns before the arena path (surrounding code omitted):

  if (tc_idx < mp_.tcache_bins
      && tcache != NULL
      && tcache->counts[tc_idx] > 0)
    {
      victim = tcache_get (tc_idx);
      return tag_new_usable (victim);
    }

When the matching bin is nonempty, this branch returns before the single-thread test and arena_get, so no arena mutex is taken on this path. The excerpt does not show allocator initialization, the arena fallback or the bookkeeping inside tcache_get, and it says nothing about how often the cache hits in a real program.

For an oversized invalid request, checked_request2size can return zero; the shown source sets ENOMEM and returns NULL. Other allocation failures also require checking. malloc(0) has implementation-defined practical details under the standard interface; do not infer a useful writable byte from a non-null zero-size result. A returned pointer belongs to the caller until ownership is deliberately transferred or it is freed. After free, application access is invalid even if the allocator retains the physical memory or a later allocation returns the same numerical address. realloc failure must be handled without discarding the original pointer; on failure it generally remains the caller's allocation for a nonzero request.

A large request may be satisfied via an mmap-based path depending on current allocator thresholds and state. The threshold is tunable and can change; a single size does not prove a particular backing path. Freed small chunks may be held for reuse, so a process's resident memory need not immediately fall. Thread caches and arenas can reduce contention for some workloads but add retained memory and do not prove a performance result for a real program. Exact bin sizes, hardening, locking and reuse are implementation and configuration details.

musl's 1.2.5 src/malloc/mallocng/malloc.c is a different allocator, organized around its own metadata, groups and slots. Its pthread_create.c has a separate thread-start implementation. Do not use glibc's tcache or arena details as a description of musl merely because both expose malloc and pthread APIs. These source comparisons establish design differences, not a benchmark or a universal security ranking.

Cancellation and observable API boundaries#

A deferred cancellation request becomes effective only at an appropriate cancellation point when cancellation is enabled. The distinction matters for ownership: a thread blocked in a cancellation-aware read might leave its normal function without reaching the ordinary free or unlock statements. Cleanup handlers, registered before acquiring or publishing the resource and removed in the correct order, can release resources on cancellation. They must themselves respect the application's lock ordering and cannot safely free an object that another thread still owns. Asynchronous cancellation has much narrower safe-use conditions and should not be inferred from a wrapper's name.

When the wrapper returns a short positive count, those bytes are valid even if a later call fails. The caller maintains an offset and checks the next result; errno from an earlier failure is not evidence that the positive result failed. A blocking call may also be interrupted or restarted depending on the interface and signal configuration. The application should write the error path around the documented interface, not compare a raw kernel negative error code to the public read return. Runtime tools such as strace expose system calls but do not show all userspace allocator cache hits, TLS accesses or lock acquisitions, so an absent syscall does not establish that no library work occurred.

A worked two-thread state#

Suppose main allocates a struct result and starts one joinable worker with a pointer to it. Initially the worker owns no exclusive lifetime right: main agrees to keep the allocation alive through join. The worker calls read(-1, buffer, 1), gets -1, and copies its own errno value EBADF into the result while holding an application mutex. Meanwhile main's errno might contain ENOENT from an earlier failed open; the worker's update to its thread-local error does not overwrite main's value. Main waits for the worker with pthread_join, checks the join result, reads the stored result, destroys any no-longer-used synchronization object and frees the allocation.

A tempting counterexample is to free the result immediately after pthread_create because the function returned success. The worker may not have executed yet, so it can dereference freed memory. Another is to use a condition variable once without looping: an unrelated wake can make main inspect a result before the producer set the predicate. A correct implementation protects both publication and lifetime; a lock alone does not keep a freed allocation alive. Cancellation introduces another path: if a thread can be canceled while holding resources, install appropriate cleanup handlers or disable cancellation for a carefully bounded critical region. Avoid acquiring locks in an inconsistent order during cleanup.

Practice#

This own-process program creates one worker, demonstrates a failed read and retains a heap object through successful join. It changes no system service or setting. It requires a C compiler and pthread support. Use a private temporary directory and remove only that directory:

work=$(mktemp -d)
cat > "$work/threads.c" <<'C'
#include <errno.h>
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
struct result { int err; };
static void *worker(void *arg) {
    struct result *r = arg; char byte;
    if (read(-1, &byte, 1) == -1) r->err = errno;
    return NULL;
}
int main(void) {
    struct result *r = malloc(sizeof *r);
    pthread_t thread;
    if (!r) return 1;
    r->err = 0;
    int rc = pthread_create(&thread, NULL, worker, r);
    if (rc) { free(r); return 2; }
    rc = pthread_join(thread, NULL);
    if (rc) { /* do not free memory a possibly live worker could use */ return 3; }
    printf("worker EBADF: %s\n", r->err == EBADF ? "yes" : "no");
    free(r);
    return 0;
}
C
cc -Wall -Wextra -pthread "$work/threads.c" -o "$work/threads"
"$work/threads"
readelf -l "$work/threads" | grep 'interpreter' || true
rm -rf -- "$work"

Expected output includes worker EBADF: yes; the interpreter line is architecture and toolchain dependent and may be absent for a static build. If compilation or linking fails, record the error and compiler version; do not install system packages or change the host. The error branch intentionally does not free memory after an unexpected failed join because the worker could still access it; process exit reclaims the private process resources. This lab does not prove the kernel's futex or allocator implementation. As a source-only alternative, use the worked state above: list who can access r at create, during execution, after successful join, and after free, and explain why the premature-free counterexample is invalid. Deliver the four-state ownership table and the two error conventions.

Knowledge check#

1. Why can a failed read in one thread leave another thread's errno unchanged?
2. In the quoted glibc setup-failure branch, why wait for pd->tid to become zero before reclaiming the descriptor?
3. Does a cache hit in __libc_malloc prove a new kernel mapping was created? What does the fallback lock protect?
4. Why must the waiter recheck a condition after waking, and why does that not by itself prove safe object lifetime?
5. What does the cited musl wrapper establish, and what cannot be inferred from its matching public read API?

Answers#

1. Each thread accesses its own TLS-backed errno state. Only inspect the value after the relevant interface indicates failure, and remember that pthread APIs commonly return their errors directly.
2. A child can already be using pd despite failure to finish requested setup. The creator signals the stopped child and waits for termination using the predicate loop before reclaiming its descriptor; a wake alone does not establish that the predicate changed.
3. No. The cache can return a previously retained chunk without a syscall. The fallback obtains and later unlocks allocator arena state; this internal lock does not protect arbitrary application data stored in the returned allocation.
4. A wake can be spurious or another thread can consume the condition. The predicate must be checked under the intended synchronization; references and ownership must separately keep the object containing that predicate alive.
5. It establishes that this musl release's Linux read wrapper calls syscall_cp with the listed arguments. It does not establish that musl shares glibc's internal cancellation implementation, ABI-private structures, tcache or arena behavior.

Checkpoint#

Submit the create/run/join/free ownership table, explain the quoted failure branch and provide either the program output or the supplied source-only state analysis. Ready means distinguishing pthread return codes from errno, naming the predicate checked after wake, and explaining a safe allocation lifetime. If uncertain, retry with one joinable worker and the locking chapter; source-only completion does not certify runtime behavior.

Sources#

• glibc 2.40 Linux read wrapper
• glibc 2.40 thread creation
• glibc 2.40 allocator
• musl 1.2.5 read wrapper
• musl 1.2.5 mallocng
• musl 1.2.5 thread creation