Virtual memory, Unix signals, and Windows exceptions

Before you start#

Review address spaces and page tables, page faults, Wine's NT boundary, and wineserver handles. Wine wine-10.0 is pinned to commit b073859675060c9211fcbccfd90e4e87520dc2c2. Linux v6.12 is a separate historical baseline. The memory addresses and fault sequence below are invented; no Wine executable or host fault was run.

Learning goals#

Distinguish reservation, commitment, decommitment and release; trace Wine's allocation and free validation; explain file-view and protection lifetimes; and connect a Unix signal to a Windows-facing exception without claiming identical semantics.

Source walkthrough#

A process virtual address space is a set of mappings and permissions, not a contiguous block of physical RAM. Windows-facing APIs such as VirtualAlloc expose reserve and commit as distinct states. Windows documents commit as charging memory/paging-file commitment while physical pages are generally supplied on access; reservation excludes other allocations from the range. Decommit removes commitment while preserving the reservation, and release gives up the reservation subject to API constraints. Page size and allocation granularity are distinct: a reservation may have coarser alignment than page commitment. See VirtualAlloc and Freeing Virtual Memory.

Wine implements these semantics over host mappings, its own bookkeeping and server coordination where required. Linux mmap and mprotect are host mechanisms, not identical Windows commit guarantees. Linux overcommit, host resource limits, page faults and backing-store behavior mean a successful reservation or protection change must not be described as a universal guarantee that every future physical page allocation succeeds. Wine's virtual-memory implementation tracks views and protection state in addition to making host operations.

Allocation, parameters and remote process requests#

NtAllocateVirtualMemory receives a process handle, an in/out base pointer and size pointer, zero-bits constraint, allocation type and protection. The caller must supply valid storage for those pointers and interpret outputs only under the returned status. Wine validates size, flags and constraints before dispatching. This exact Wine 10.0 excerpt is from NtAllocateVirtualMemory in dlls/ntdll/unix/virtual.c; its file notice grants LGPL-2.1-or-later. Surrounding tracing and other checks are omitted:

    if (!*size_ptr) return STATUS_INVALID_PARAMETER;

A zero size is rejected immediately with the stated NT status. It is not a request to reserve the entire address space. Subsequent checks reject unsupported type bits and certain zero_bits values. For the current process, the function computes a limit and calls allocate_virtual_memory; for another process it builds a server-mediated APC request and updates output pointers only if that request reports success. A failure to queue the request returns before a successful allocation result exists. This conditional remote-process branch is why “every allocation is one local mmap” is incorrect.

The local implementation needs to reconcile requested ranges with existing Wine file_view bookkeeping and host address-space availability. Conflicting addresses, unsupported flags, invalid protection, host mapping failure or memory limits can produce different status values. The implementation's virtual_mutex and uninterrupted-section helpers coordinate access to view structures and signal state in relevant paths; an arbitrary raw pointer into a view is not a lifetime reference after a concurrent release. Error cleanup must release mappings or metadata acquired by the particular branch. A requested address is a hint or constraint under the API and flags, not a promise that Wine will silently replace an unrelated mapping.

Decommit and release are different transitions#

In the same virtual.c, NtFreeVirtualMemory rounds the range, enters an uninterrupted section protected by virtual_mutex, finds the view and validates its relationship to the request. This exact Wine 10.0 LGPL-2.1-or-later excerpt omits validation and neighboring placeholder cases:

    case MEM_DECOMMIT:
        status = decommit_pages( view, base - (char *)view->base, size );
        break;
    case MEM_RELEASE:
        if (!size) size = view->size;
        status = free_pages( view, base, size );
        break;

The decommit case calls a helper for pages within the view. The release case uses the full view size when the supplied size is zero, then calls a different helper to free the range. The earlier validation rejects missing views, inappropriate view types and certain non-base or out-of-range requests. Later code releases the mutex and restores signal state. This excerpt does not establish that every partial release is valid or that an invalid pointer is safe to dereference. Callers must check the returned status and stop using released addresses.

The decommit helper is short. In dlls/ntdll/unix/virtual.c, Wine 10.0, LGPL-2.1-or-later, decommit_pages is the complete function:

static NTSTATUS decommit_pages( struct file_view *view, size_t start, size_t size )
{
    if (!size) size = view->size;
    if (anon_mmap_fixed( (char *)view->base + start, size, PROT_NONE, 0 ) != MAP_FAILED)
    {
        set_page_vprot_bits( (char *)view->base + start, size, 0, VPROT_COMMITTED );
        return STATUS_SUCCESS;
    }
    return STATUS_NO_MEMORY;
}

The range is remapped as PROT_NONE with a fixed anonymous mapping, and Wine's committed bit is cleared. The function does not unmap the range or delete the view. That is consistent with decommit keeping the reservation, although the reservation rules themselves are not shown here.

Whole-range release takes a different helper. In the same file, free_pages deletes the view when the requested size equals the view size and otherwise removes pages from it. The later partial-release lines are omitted:

static NTSTATUS free_pages( struct file_view *view, char *base, size_t size )
{
    NTSTATUS status;

    if (size == view->size)
    {
        assert( base == view->base );
        delete_view( view );
        return STATUS_SUCCESS;
    }
/* ... */

A whole-view release discards the view bookkeeping, while decommit keeps it. Reading only the switch in the earlier excerpt would not show this difference. The excerpts also do not show the validation in NtFreeVirtualMemory, so the path actually reached depends on that earlier code.

Worked reserve and commit timeline#

Assume a synthetic Wine process reports a 64 KiB allocation granularity and a 4 KiB page size. It reserves a 64 KiB range at 0x10000000, then commits two pages beginning there with read/write protection. Under the example's successful-operation assumptions, the first 8 KiB is committed and the remaining 56 KiB remains reserved. The addresses and success are invented, not an observed Wine run. A request with size zero follows the quoted STATUS_INVALID_PARAMETER branch; an overlapping incompatible reservation can fail with a conflict status under the relevant implementation path.

Next the application changes the first page to read-only using the appropriate protection API. A write to that page would violate the intended protection and can lead through a host fault and Wine exception handling, depending on actual mapping and signal configuration. It should not be deliberately triggered on the host for this lesson. Decommitting the two pages returns them to reserved state if the operation succeeds; it does not release the entire reservation. A later successful release of the reservation invalidates the old range for the application's use. A stale pointer into it is not made valid merely because a future unrelated allocation reuses the same numeric address.

If the commit step fails, the caller must not use the pages as committed memory. Depending on which operation created the reservation and the exact failure branch, the reservation may remain and need explicit cleanup; never assume every failure automatically releases all earlier successful state. Record each operation's returned base, rounded size and status. For a remote process request, the server and target process can exit or reject the operation; the output pointers are not a substitute for checking the result.

Views, files and copy-on-write#

Wine can create mappings of files or image sections and associate them with file_view structures and protection metadata. A view's base and size describe an address-space interval; a mapping object and any backing file have separate references and lifetimes. A shared mapping can expose updates according to the mapping and synchronization contract, while copy-on-write creates private modifications rather than changing the shared backing in the same way. A mapped image's section protections, relocation and initialization requirements differ from a generic anonymous reservation.

Unmapping one view does not necessarily close every handle to the mapping object or file; closing a handle does not necessarily invalidate all existing views. Conversely, retaining a stale pointer after its view is unmapped is invalid even if the underlying file still exists. Host mmap errors, file truncation, access permissions and Wine's view validation can cause different failures. Cross-process mappings and synchronization require explicit object references and architecture-specific address-space constraints; a raw address meaningful in one process is not automatically meaningful in another.

Thread stacks use reserved and committed regions and guard behavior under the applicable Wine implementation. A guard page can detect stack growth or an invalid access, but its handling, re-arming and protection semantics must be established for the actual path. A guard fault is not a general-purpose memory-safety detector, and it is not safe to infer that every adjacent page belongs to the same reservation. Stack overflow recovery can itself need stack space, so exception behavior is sensitive to architecture and runtime configuration.

Signals and Windows-facing exceptions#

On Linux, a memory access violating a mapping or protection can cause SIGSEGV; some backing-object failures (for example, accessing beyond a mapped file's end) can produce SIGBUS; an illegal instruction can produce SIGILL. The exact signal and code depend on the architecture and fault. See Linux mprotect(2) and mmap(2). Wine's x86-64 signal implementation receives host context and constructs or dispatches Windows-facing exception state for supported cases. The signal handler executes in constrained context; register state, stack selection and nested faults matter.

Windows exception processing includes debugger first-chance notification and application exception dispatch; vectored handlers and frame-based SEH participate according to the platform's dispatch rules. A first-chance notification is not automatically a crash: a debugger or handler can handle the exception. Conversely, a translated signal does not guarantee a valid handler or safe resumption. See Microsoft's debugger exception handling and x64 exception handling. Wine must preserve enough thread context for the selected architecture and respect its dispatcher and unwind contracts.

On 32-bit x86, Windows exception handling has historically used per-thread registration chains; x86-64 exception unwinding instead relies on table-based metadata such as .pdata and .xdata where applicable. These are architecture-specific ABI concepts, not interchangeable linked lists. The TEB, thread stack and unwind metadata must remain valid while handlers run. An application with corrupt stack or unwind information can yield an incomplete or misleading backtrace. Linux's signal delivery and Wine's Windows-facing exception dispatch are separate stages, not the Linux kernel implementing SEH directly.

Race and cleanup reasoning#

Imagine thread A reading from a mapped view while thread B releases it. Without an application-level ownership protocol, A can access an address after it becomes invalid or after a new mapping reuses that address. Wine's internal mutex protects its own view metadata during operations; it is not a general lock automatically held by arbitrary application code using the mapped bytes. The application needs synchronization around use and release. A protection update can also race with another thread's access, producing a fault whose timing depends on host scheduling.

A robust investigation records the exact Wine build, architecture, returned allocation addresses and sizes, operation statuses, mapping type and thread sequence. It distinguishes a host signal from the translated NT exception and does not infer the original allocation merely from a reused address. When a page fault is involved, source reading can identify possible validation and error branches, but actual behavior needs a controlled runtime environment and matching binaries. A private prefix is not a sandbox, and this chapter's source exercise intentionally does not trigger faults.

Diagnosing a protection failure without guessing#

Suppose a thread has reserved a region and committed only its first two pages. A pointer into the third page lies inside the reserved address range, but that alone does not make a read valid: reservation excludes competing allocations from the range while commitment and protection describe access to individual pages. Now suppose the second page is committed read-only and an application writes to it. That is a different cause from an access to the uncommitted third page, although both may ultimately surface through an exception path. The investigation needs the faulting address, requested access, current view, page state and protection, as well as the instruction and architecture. A fault address on its own does not prove an out-of-bounds heap access.

The region and the Linux mappings may change concurrently. Wine's virtual-memory lock protects its internal view bookkeeping during relevant operations; it is not an application-level guarantee that another thread will not release an allocation immediately after a caller inspects it. Programs need their own ownership protocol for sharing pointers and must stop users before releasing their storage. A retained numeric address after release does not retain the original allocation. If a later allocation uses that address, a stale pointer can silently address a different object, which makes this bug more subtle than an immediate fault.

A mapped file view introduces another independently owned resource: the section or backing file can remain referenced by a view after the application closes a handle, depending on the operation and object references. Writing a private copy-on-write page changes that view's private contents; it does not establish durability of a file update. Shared mappings have different visibility and writeback requirements. A valid protection transition also does not make the underlying backing store infallible: a host mapping can fail due to address conflicts or resource limits, and an application must test the returned status before publishing a pointer. Keep the failed result, requested range and process identity together in a diagnostic; reporting only “out of memory” discards important distinctions.

Practice#

Synthetic reservation table

This Python program calculates the stated reservation sizes; it does not allocate or protect memory at the example addresses:

python3 - <<'PY'
reserved = 64 * 1024
page = 4096
committed = 2 * page
print('reserved bytes:', reserved, 'committed bytes:', committed, 'remaining reserved bytes:', reserved - committed)
print('after decommit: committed bytes:', 0, 'reservation bytes:', reserved)
print('after release: reservation bytes:', 0)
PY

Expected values are 65,536 reserved, 8,192 committed and 57,344 remaining before decommit. The counterexample is a zero-size allocation request: the cited function returns STATUS_INVALID_PARAMETER, so the model must not mark any pages committed. The program is arithmetic only; it does not test Windows commitment, Linux overcommit, signal delivery or Wine.

Source-only alternative: draw four states—reserved, two pages committed, decommitted and released—and attach a status and pointer-validity condition to each. Add the zero-size failure and remote-process branch described above. Deliver the table and distinguish a hypothetical Unix fault from a measured Windows-facing exception. Do not execute invalid memory accesses or unknown PE files on the host.

Knowledge check#

1. Why is reservation different from commitment in the worked range?
2. What does a zero-size NtAllocateVirtualMemory request return in the cited path?
3. Why does decommit not necessarily release the address reservation?
4. Why can Wine's internal view lock not replace application synchronization around a pointer?
5. Why is a first-chance exception not automatically a process crash?

Answers#

1. Reservation establishes an address-space range; commitment establishes usable committed state under the relevant implementation. They have different alignment and lifetime rules.
2. The function returns STATUS_INVALID_PARAMETER before the subsequent allocation path.
3. The cited switch calls a decommit helper for pages within the view, while release uses a separate free path. Actual success and range rules still need checking.
4. The lock protects Wine's bookkeeping during relevant operations; application threads can still race accesses against release unless their own ownership protocol prevents it.
5. It is an early notification/handling stage. A debugger or handler may handle the exception; whether execution resumes safely depends on the actual context and handlers.

Checkpoint#

Submit the four-state reservation table, zero-size failure and signal-to-exception boundary diagram. Ready means no stale pointer is used after release, Linux memory behavior is not claimed identical to Windows commitment, and synthetic faults are not presented as captured crashes. Retry the allocation/free source branches if a failed operation is treated as success. This is not certification.

Sources#

• Wine 10.0 virtual-memory implementation
• Wine 10.0 x86-64 signal handling
• Wine 10.0 loader
• Wine 10.0 server object management