ntdll, Unix calls, and syscall translation

Before you start#

Review Linux syscall entry, C pointers and lifetime, Wine architecture, and PE loading. This lesson pins upstream Wine wine-10.0 to commit b073859675060c9211fcbccfd90e4e87520dc2c2. Linux v6.12 is a separate historical kernel baseline; no Wine/kernel version combination was runtime tested here.

Learning goals#

Distinguish Windows-facing NT calls, Wine's internal Unix-call dispatch and Linux syscalls; trace a file-open success or failure through actual Wine 10.0 code; and explain why ABI, thread state and error translation matter.

Source walkthrough#

Windows-facing software commonly calls an API in a higher-level DLL, which can call an Nt* routine exported by ntdll. An NT status value describes success or failure under Windows-facing conventions. That interface is not the same as a Linux syscall number or Linux errno. Wine implements compatible behavior using PE-facing code, Unix-side functions, shared data and, where appropriate, wineserver. Some requests are handled locally; some require server-managed state; some ultimately invoke host system calls. There is no universal rule that every Nt* call becomes one server request or one Linux syscall.

A calling convention is an agreement about argument registers, stack layout, alignment, return values and preserved registers. Windows x86-64 and Unix System V x86-64 conventions differ, as does Linux's kernel syscall ABI. Passing a pointer or integer in the wrong register can corrupt state even if both sides use 64-bit addresses. Wine's architecture-specific assembly and generated interfaces bridge these boundaries. Other architectures, WoW64 configurations and build options have different paths; the x86-64 discussion below is not a universal Wine implementation.

Two internal dispatch mechanisms#

Wine's source distinguishes Windows-facing syscall dispatch from calls into Unix libraries. In dlls/ntdll/unix/loader.c, initialization connects __wine_syscall_dispatcher, __wine_unix_call_dispatcher and the Unix-call function table to the appropriate PE-facing state. These are Wine implementation interfaces, not a claim that a Windows syscall number is a Linux syscall number. The selected table, architecture and build configuration determine what a given dispatch means.

In include/wine/unixlib.h, the non-arm64ec __wine_unix_call wrapper contains this exact Wine 10.0 excerpt. The file's notice grants LGPL-2.1-or-later, copyright Alexandre Julliard and contributors; the surrounding conditional is omitted:

    return __wine_unix_call_dispatcher( handle, code, args );

The wrapper forwards a Unix-library handle, operation code and argument pointer to the configured dispatcher and returns an NTSTATUS. The handle selects a Wine Unix-library interface; the code and argument structure are meaningful only under that interface's agreement. The nearby arm64ec branch calls a different entry point. This line does not show the internal implementation of every NT service, and it does not mean arbitrary pointers or codes are safe. A caller and callee must agree on layout, lifetime and architecture-specific calling conventions.

In dlls/ntdll/unix/signal_x86_64.c, handwritten assembly for the Unix-call dispatcher saves selected Windows-visible registers and vector state, switches to a Wine-managed syscall frame/stack, adapts arguments and calls a function through a table. It later restores state and returns through the appropriate path. The source also handles Linux-specific thread-segment state and special return conditions. The comment “kernel stack” in this userspace compatibility implementation must not be confused with entering the actual Linux kernel's privileged stack. The assembly is architecture-specific and sensitive to unwinding, alignment, signals and thread-local state.

Windows code that issues a direct syscall instruction rather than using an exported interface is another compatibility problem. Wine's handling depends on architecture, build and implementation; do not assume every hard-coded Windows syscall number is a Linux syscall or is transparently supported. A syscall instruction that actually enters Linux is interpreted by Linux's ABI, not by a Windows kernel. Never test malformed binaries or privileged direct syscalls on the host to explore this distinction.

TEB, PEB, TLS and exceptions#

A Windows-facing thread has a thread environment block (TEB), and the process has a process environment block (PEB). These store Windows-visible thread and process information, loader state and other fields used by compatible code. Unix thread-local storage and the host thread's signal context are related but distinct mechanisms. Wine has to maintain the correct association when switching between PE-facing and Unix-facing code, including when a callback or signal interrupts execution. A pointer into a TEB is not a substitute for a retained server object reference, and its validity depends on the corresponding thread lifetime.

Structured exception handling (SEH) is a Windows-facing exception mechanism. On Unix, Wine's architecture-specific signal handling can translate selected host faults and thread contexts into Windows-compatible exception behavior. A host signal is not automatically identical to a Windows exception, and signal handlers run under restrictive context rules. The source has architecture-specific logic for recovering register state and unwinding across dispatcher frames. A failure in that bridge can affect error handling or stack walking even if the original API call appears unrelated to signals.

A safe point in this discussion means a point at which a particular transition or callback's invariants permit an action, not a universal promise that a thread can be suspended or its memory freed anywhere. Register save/restore, stack alignment, unwind metadata and signal masks are part of the implementation contract. A developer changing a handwritten assembly bridge must understand the precise architecture ABI and error paths; ordinary C type checking cannot validate all of those obligations.

A concrete NT file request#

The pinned Wine Unix file implementation defines NtOpenFile as a call to NtCreateFile with the FILE_OPEN disposition and selected default arguments. NtCreateFile receives an output handle, access mask, object attributes, I/O status block, sharing mode, disposition and options. An object name is a Windows-facing name; it must be translated or handled by an appropriate Wine object path before a host file operation can occur. The access and sharing rules have semantics beyond the numeric host fd.

The following exact Wine 10.0 excerpt comes from NtCreateFile; the file's notice grants LGPL-2.1-or-later, copyright Erik Bos, Eric Pouech, Alexandre Julliard and contributors. Surrounding redirect and error handling are omitted:

        status = nt_to_unix_file_name( &new_attr, &unix_name, disposition );

The call translates the selected NT name and disposition into a Unix-name result or an NT status. A successful translation does not by itself open the file or prove host access. The subsequent code handles a special device-type status through a server object path, treats a not-found result differently for create-capable dispositions, and calls open_unix_file only on the applicable success path. It frees the temporary Unix name after that call. The code later fills the I/O status information. Exact name, device, sharing and disposition rules require reading the rest of the function and its helpers.

The wrapper NtOpenFile in Wine 10.0, LGPL-2.1-or-later forwards to the create path:

NTSTATUS WINAPI NtOpenFile( HANDLE *handle, ACCESS_MASK access, OBJECT_ATTRIBUTES *attr,
                            IO_STATUS_BLOCK *io, ULONG sharing, ULONG options )
{
    return NtCreateFile( handle, access, attr, io, NULL, 0, sharing, FILE_OPEN, options, NULL, 0 );
}

The wrapper fixes the disposition to FILE_OPEN and passes NULL for the optional arguments. It does not show name translation, sharing checks or create-disposition handling inside NtCreateFile.

Consider a synthetic request to open an existing prefix-mapped file with FILE_OPEN. If name translation succeeds and the host open succeeds under the process's Unix credentials and policy, Wine can return a suitable Windows-facing handle and status through the implementation. If the host reports a permission error at a path that uses errno_to_status, the mapping in the same source maps EACCES to STATUS_ACCESS_DENIED. If the target is absent, name translation or host opening can return a not-found status; it is not automatically permission denial. An IO_STATUS_BLOCK and output handle must be interpreted only according to the returned status and the API contract.

For a device-like name, nt_to_unix_file_name can return STATUS_BAD_DEVICE_TYPE; the shown source then calls server_open_file_object and frees the temporary NT-name buffer before returning. That is a conditional server path, not proof that every file call passes through wineserver. For a create-capable disposition, a missing file can be treated differently from FILE_OPEN, which specifically requests an existing object. These branches explain why a superficial mapping such as “NtOpenFile equals Linux open” loses object namespace, sharing, status and lifecycle details.

Error translation is a contract, not a lookup table for everything#

Unix functions commonly return -1 and set thread-local errno; Wine's errno_to_status maps selected host errors into NT status values for paths that call it. In the pinned code, EACCES, EPERM and EROFS map to STATUS_ACCESS_DENIED, while ENOENT maps to STATUS_OBJECT_NAME_NOT_FOUND and EBADF to STATUS_INVALID_HANDLE. The mapping is many-to-one and context-sensitive: once several host errors map to the same NT status, the Windows-facing caller cannot reconstruct the original host cause from that status alone. Other Wine paths can construct statuses directly without using this helper.

In dlls/ntdll/unix/file.c, errno_to_status, Wine 10.0, LGPL-2.1-or-later (later cases omitted):

NTSTATUS errno_to_status( int err )
{
    TRACE( "errno = %d\n", err );
    switch (err)
    {
    case EAGAIN:    return STATUS_SHARING_VIOLATION;
    case EBADF:     return STATUS_INVALID_HANDLE;
    case EBUSY:     return STATUS_DEVICE_BUSY;
    case ENOSPC:    return STATUS_DISK_FULL;
    case EPERM:
    case EROFS:
    case EACCES:    return STATUS_ACCESS_DENIED;
    case ENOTDIR:   return STATUS_OBJECT_PATH_NOT_FOUND;
    case ENOENT:    return STATUS_OBJECT_NAME_NOT_FOUND;
    case EISDIR:    return STATUS_INVALID_DEVICE_REQUEST;
    case EMFILE:
    case ENFILE:    return STATUS_TOO_MANY_OPENED_FILES;
/* ... */

Several host errors, including EACCES, EPERM and EROFS, produce the same NTSTATUS. The excerpt does not show whether a given caller uses this helper at all, or statuses created without an errno.

A Windows-facing library can translate an NT status again into a Win32 last-error value. That is another boundary; it is not safe to compare the numeric Linux errno, NTSTATUS and Win32 error as if they shared a numbering scheme. A trace may show a host permission error followed by a Windows-facing access-denied result, but it must be associated with the actual request and process. Error translation does not bypass Linux DAC, LSM, mount or namespace policy; the Wine process acts with its host authority.

Object and buffer lifetime across the boundary#

The NT call's argument pointers, output buffers and object attributes must remain valid for the synchronous portion of the call, and asynchronous operations can impose additional completion and buffer-lifetime rules. The function table and argument structure definitions are an ABI between Wine components built to cooperate; mixing binaries from incompatible Wine builds can cause protocol or layout failures. The server protocol also has a version check during process initialization. A Wine update must not be assumed safe to mix with a still-running older server or arbitrary third-party DLLs.

When a path conversion allocates a temporary Unix string, the implementation must free it after the host operation or on error. A server handle is a separately managed object reference; a host fd used by an operation can have its own duplication and closure rules. A successful lookup may need a reference until the operation completes, while a failed lookup must not be treated as a valid handle. Inspect the exact helper and cleanup labels rather than adding a generic close to every error path.

For performance or debugging, one NT request can cause multiple host operations, and a server round trip can be interleaved with client-side work. A Linux syscall trace alone may show openat, mmap, socket activity or futex operations, but it does not automatically identify the Windows API, object semantics or application intent. Conversely, a Wine debug channel can be filtered or contain sensitive paths. Correlate timestamps, process/thread identity, source version and object ownership before drawing a call-chain conclusion.

Following one error without conflating layers#

Suppose a Windows-facing caller requests an existing file, Wine resolves its NT name to a Unix path, and the relevant host operation returns EACCES. For a path that calls the inspected helper, Wine maps that errno to STATUS_ACCESS_DENIED. A higher-level wrapper may then translate the NT status to its own error convention. Those are successive representations of one failure; they do not imply that Linux saw a Windows syscall number or that the Windows caller received the original errno. Capture the specific implementation path before drawing that connection because Wine also returns statuses from validation or server operations without first receiving a host errno.

Now consider the same request with a malformed object attribute. The inspected NtCreateFile checks required attributes and can return an invalid-parameter status before normal path conversion and host opening. A server protocol failure or an unsupported device path is yet another category. These cases demonstrate why the stage, input ownership and actual status are necessary debugging evidence. A successful local model of an errno mapping cannot establish that a Wine process ran, reached the helper or made any particular host syscall.

Verifying a boundary claim#

A concrete boundary report should name the exported entry point, the actual Wine binary version, the selected architecture and the path that produced the result. If the report contains only a Linux open failure, it cannot identify which Windows API requested it without correlation. If it contains only an NT status, the many-to-one error mapping can hide the original host cause. A source walkthrough explains possible branches, while a controlled runtime trace would be needed to establish which branch occurred for a particular application. Keep those evidence types separate.

Practice#

Source-based status table

This program models only the cited error mapping; it does not invoke Wine, access a Windows prefix or issue a privileged syscall:

python3 - <<'PY'
mapping = {'EACCES': 'STATUS_ACCESS_DENIED', 'EPERM': 'STATUS_ACCESS_DENIED',
           'ENOENT': 'STATUS_OBJECT_NAME_NOT_FOUND', 'EBADF': 'STATUS_INVALID_HANDLE'}
for error in ('EACCES', 'EPERM', 'ENOENT', 'EBADF'):
    print(error, '->', mapping[error])
PY

Expected output maps the first two errors to the same access-denied status and the others to distinct statuses. This is an authored model of selected cases in errno_to_status, not executable Wine code or a complete mapping. A counterexample to reversing the mapping is STATUS_ACCESS_DENIED: it can represent more than one source errno. If Python is unavailable, complete the table by hand.

Source-only alternative: use the supplied NtCreateFile path. Write separate rows for name translation success followed by host open, a missing name under FILE_OPEN, and the special device-object branch. Add a cleanup column for the temporary name and state which result is still unknown without a real host operation. Deliver the table and ABI diagram without executing a PE binary, privileged syscall or malformed input.

Knowledge check#

1. Why is __wine_unix_call_dispatcher not the Linux syscall dispatcher?
2. What does successful NT-to-Unix name translation establish, and what does it not establish?
3. Why can STATUS_ACCESS_DENIED not always be reversed to one host errno?
4. Why does the x86-64 assembly save and restore register and thread state?
5. Does the special device-object branch prove every NtCreateFile request uses wineserver?

Answers#

1. It is a Wine userspace dispatch interface with a Wine library handle, operation code and arguments. A later host operation may enter Linux through a separate ABI.
2. It establishes that the translation step returned success on that path. The host open, permissions, sharing rules and handle creation can still fail.
3. The cited mapping converts multiple host errors, including EACCES and EPERM, to the same status. Other Wine paths can also produce that status directly.
4. The PE-facing and Unix-facing conventions and thread contexts differ. Correct preservation, stack alignment and restoration are needed for calls, returns, signals and unwinding.
5. No. It is a conditional branch for a particular translation status; the normal path can call the Unix-file helper, and other operations have their own implementations.

Checkpoint#

Submit the source-based file-request and error-mapping tables plus a diagram of the NT, Wine and Linux boundaries. Ready means the Unix-call dispatcher is not confused with a Linux syscall, temporary names have a cleanup owner, and unobserved runtime behavior remains unknown. Revisit the pinned code if every NT call is routed through a single assumed path. This is not certification.

Sources#

• Wine Unix-call interface, 10.0
• Wine Unix file implementation, 10.0
• Wine x86-64 signal and dispatcher implementation, 10.0
• Wine Unix loader setup, 10.0