Event-driven servers and embedded databases

Before you start#

Review epoll and asynchronous I/O, file durability, userspace process control, and userspace observability. Nginx release-1.26.2 is pinned to Git mirror commit 37fe98355461d2f03d73e6a8e82ac4e4cd85d711; SQLite version-3.46.1 is pinned to Git mirror commit f3d536d37825302e31ed0eddd811c689f38f85a3. These versions are historical source examples, not a tested deployment pair. SQLite develops in Fossil; the Git mirror commit records Fossil origin c9c2ab54ba1f5f46360f1b4f35d849cd3f080e6fc2b6c60e91b16c63f69a1e33, a different identifier from the Git commit.

Learning goals#

Follow an incomplete HTTP request through a stateful parser, distinguish readiness from completion, trace a SQLite pager lock retry, and design an owned temporary database experiment that separates contention from I/O failure.

Source walkthrough#

Nginx commonly has a master process that manages configuration and worker processes that accept and process connections. A worker uses an event loop to respond to readiness notifications and timers; it need not dedicate one thread to every waiting connection. Readiness means an operation may make progress, not that a complete HTTP request or response exists. SQLite is an embedded database library: a connection lives inside its calling process rather than requiring a separate SQLite database server. Both systems still depend on kernel descriptors, storage and scheduling.

The first exact excerpt is from Nginx 1.26.2 src/http/ngx_http_parse.c, ngx_http_parse_request_line. The repository's license notice states BSD two-clause terms. The excerpt follows the parser's loop and omits its completed-request branch:

    b->pos = p;
    r->state = state;

    return NGX_AGAIN;

The parser saves the buffer position and current state, then returns NGX_AGAIN to request more input. The next call can continue from the saved state rather than treating a split network read as a malformed request. Earlier branches return an invalid-request result for prohibited input. A caller must preserve the request and buffer while waiting; NGX_AGAIN is neither a complete request nor an unconditional error.

The second excerpt is from SQLite 3.46.1 src/pager.c, pager_wait_on_lock. The file header disclaims copyright and places the source in the public domain. Surrounding lock-transition assertions are omitted:

  do {
    rc = pagerLockDb(pPager, locktype);
  }while( rc==SQLITE_BUSY && pPager->xBusyHandler(pPager->pBusyHandlerArg) );
  return rc;

The pager attempts a database lock, retries only when the result is SQLITE_BUSY and the configured busy handler requests another attempt, and returns the final result. A busy result represents a particular contention condition; it is not equivalent to every I/O error or a successful transaction. The permitted lock transitions and circumstances in which the busy handler is called are constrained by surrounding code. SQLite can bypass the handler when invoking it could contribute to deadlock; a configured handler does not guarantee a callback on every lock conflict (busy-handler documentation). A handler that stops retrying allows the caller to receive SQLITE_BUSY and decide whether to retry, roll back or report failure.

Nginx connections, requests, and events#

A listening socket accepts a connection, and a connection owns a socket descriptor and state for reading, writing, timers and event registration. An HTTP request is a separate object associated with that connection; keep-alive can permit more than one request over a connection at different times. Nginx uses memory pools for request and connection allocations. A request pool cannot be freed while an outstanding callback still expects its data, and a persistent connection's lifetime can extend beyond an individual request. An upstream connection, when proxying, has its own descriptor and failure path.

On Linux an epoll-based event module can report that a socket is readable or writable. The worker must still call the appropriate operation and handle a partial read, end of stream, error or EAGAIN. A readable event can mean EOF as well as available data. Timers can expire while data is arriving; whichever event the worker processes must follow the request and connection's actual state. A timeout is not a parser syntax error, and an invalid request is not merely a slow client. An event loop makes waiting scalable, but long blocking work in a worker can delay other connections assigned to that worker. Nginx's development guide describes its event and timer loop; Linux's epoll(7) documentation describes readiness notifications and the need to perform I/O afterward.

In Nginx's epoll module, events carry data associated with connections and an instance bit. A descriptor number can be closed and later reused by the kernel for a different socket. An event already queued for the old connection must not be applied blindly to the new object just because the integer fd is the same. Nginx encodes the read event's instance bit in data.ptr and checks it during dispatch to reject a class of stale events. This narrow guard does not make every callback or application module automatically race-free. This is a lifetime and generation issue, not a property guaranteed by fd numbers themselves.

In Nginx's epoll module, ngx_epoll_process_events, release 1.26.2, with the repository's BSD two-clause license notice (surrounding code omitted; the stale-event body is elided):

    for (i = 0; i < events; i++) {
        c = event_list[i].data.ptr;

        instance = (uintptr_t) c & 1;
        c = (ngx_connection_t *) ((uintptr_t) c & (uintptr_t) ~1);

        rev = c->read;

        if (c->fd == -1 || rev->instance != instance) {
/* ... */

The low bit of the stored pointer carries the instance value; the loop strips it, then compares it with the read event of the connection it found. The excerpt does not show the full handler, the write-event branch or any other event module.

Worked request in two chunks#

Suppose an owned test client sends the bytes GET /demo HTTP/1. followed later by 1\r\nHost: example\r\n\r\n. These are synthetic chunks, not a captured network trace. After consuming the first chunk, the request-line parser has not seen a complete line. Under the appropriate parser state and buffer assumptions, it saves its current position and state and returns NGX_AGAIN. The event loop waits for additional data. When the second chunk arrives, the parser resumes and can finish the line, after which header parsing continues. The precise intermediate state depends on the actual bytes and parser implementation; a network read boundary is not an HTTP message boundary.

As a counterexample, substitute an invalid character where the parser's current state rejects it. The parser can return an invalid-request status rather than waiting forever for another chunk. If the peer closes before completing the line, EOF handling is another path. A request timeout can occur even though the partial bytes were syntactically valid. The worker must release the request pool and close or reuse the connection according to its actual state; an upstream socket, if one was created, requires separate cleanup.

For the stale-event case, imagine connection A used fd 12 and is closed. Later connection B receives fd 12. A previously queued event identifies A's old instance. Applying that event to B solely by comparing the fd integer could operate on freed or unrelated state. The event module's instance check addresses a particular form of this problem, but it does not make every callback or application module automatically race-free. A useful trace includes connection identity, generation, event type and lifetime, not only the fd number.

SQLite's connection, pager, and journal#

A SQLite connection owns database state, prepared statements and configuration. A prepared statement compiles SQL into a virtual machine program; sqlite3_step advances it, and reset/finalize operations have their own lifetime rules. A statement can hold resources or a read transaction open longer than an application expects. A connection must not be closed or reused as if all statements had completed when they remain active. Threading guarantees depend on the SQLite build, connection mode and application discipline.

The B-tree layer organizes database pages; the pager coordinates page cache, locks and journaling through a virtual filesystem, or VFS. On Unix, the VFS maps operations to filesystem and locking mechanisms, but exact behavior depends on build and filesystem. A transaction is not merely a sequence of SQL strings: it has lock and journal state that must be committed or rolled back. An error during a write can require rollback and can leave the connection in a state the application must inspect. SQLITE_BUSY generally indicates a conflicting lock or transaction under a specific path; SQLITE_IOERR indicates a different class of lower-level I/O failure. SQLITE_LOCKED is another distinct result, generally describing a conflict within the same connection or shared cache (SQLite result codes).

Rollback-journal mode records information needed to restore prior database pages before changes are committed according to its protocol. WAL mode appends changes to a write-ahead log and uses a checkpoint to move eligible content back to the main database. Readers and writers interact differently in these modes, and a long-lived reader can delay checkpoint progress. A successful commit does not imply that the main database file alone contains every latest change while a WAL exists: committed changes may still reside in the -wal file, which must be kept with the database when copying. The -shm file coordinates clients and indexes the WAL; it is not durable database content (SQLite WAL documentation, WAL file format). Copying only the main file from a live database can therefore produce an incomplete backup.

Synchronization settings, filesystem behavior, hardware caches and failure models affect durability. SQLite's documented transaction guarantees rely on its VFS and storage assumptions; one successful local transaction or a call named fsync does not prove survival of every power failure. Do not test crash consistency by cutting power or corrupting a host database. The SQLite Unix VFS and WAL implementation are entry points for version-specific behavior, not universal filesystem proofs.

Worked two-connection contention#

Assume a private database with two connections A and B. A begins an immediate write transaction and changes a row without committing. B sets a zero busy timeout and attempts its own immediate write transaction. Under the stated single-writer conflict, B receives a busy error rather than obtaining the write transaction. A still owns its transaction and must commit or roll back. B's failure does not automatically roll back A, and B should not report its own write as committed.

Now let A roll back. B can attempt a new write transaction and, if no other conflicting lock or failure intervenes, succeed. That retry is a new operation after the state change, not proof that a previous failed write was secretly accepted. In WAL mode a reader may continue seeing its earlier snapshot while a writer commits; a checkpoint and the reader's lifetime affect what appears in the main database file. A reader's old result is not automatically evidence of database corruption. This timeline is a source-informed example; the practice below checks a private local contention case with the installed SQLite version.

A useful evidence table lists connection, transaction state, lock attempt, returned result and cleanup. For example: A owns the write transaction; B's zero-timeout attempt fails busy; A rolls back; B retries and owns a new transaction; B rolls back and both connections close. If an SQLITE_IOERR occurs instead, investigate filesystem and VFS errors rather than treating it as ordinary lock contention. An exception does not excuse leaving a connection or temporary directory open; cleanup belongs in finally or equivalent ownership logic.

Experimental interpretation#

Nginx's NGX_AGAIN and SQLite's SQLITE_BUSY both invite later work, but they occur in different state machines. The former in this parser means the request line needs more bytes; the latter in the pager concerns a lock attempt and a busy handler. Retrying a parser without additional data can spin; retrying a database write while a conflicting transaction remains open can repeatedly fail. An event loop should arrange a meaningful wakeup, timeout or backoff rather than busy-looping.

A source location is not an execution trace. A request may never reach this parser because connection setup fails, and a SQL operation may not invoke the shown lock transition under its selected journal mode. A responsible report records the actual version, configuration, workload, returned status and relevant ownership. The source paths explain possible branches but cannot by themselves establish performance, production safety or durability for a particular deployment.

A transaction error is not a response status#

Imagine an application behind an HTTP server attempts a database update. The client sends one request, the application parses its own input and opens a transaction, and a competing writer causes the database operation to return busy. The HTTP server may later send a response chosen by the application, but SQLite's result is not itself an HTTP status code. A retry policy must decide whether the operation is safe to repeat, whether a partial transaction needs rollback, and whether the client could have received a response before a network disconnect. Repeating a non-idempotent request without an application-level identifier can apply it twice even if the first response was lost.

An upstream timeout likewise does not prove the database rolled back. The upstream process may continue after the proxy gives up, or it may fail before committing. A disciplined investigation records transaction outcome at the application and database boundary as well as the proxy's connection state. A busy timeout can reduce transient contention but cannot repair a writer that never releases its transaction. In an owned experiment, release or roll back A before retrying B, then close both connections. The practice case provides that cleanup and avoids drawing conclusions about an actual HTTP deployment.

Practice#

This Python program creates a new private temporary database, uses two local connections and removes the directory on exit. It does not start Nginx or access an existing database:

python3 - <<'PY'
import sqlite3, tempfile
from pathlib import Path
print('Python SQLite version:', sqlite3.sqlite_version)
with tempfile.TemporaryDirectory(prefix='sqlite-lock-study-') as directory:
    path = Path(directory) / 'owned.db'
    a = sqlite3.connect(path, timeout=0)
    b = sqlite3.connect(path, timeout=0)
    try:
        a.execute('CREATE TABLE item (value INTEGER)')
        a.commit()
        a.execute('BEGIN IMMEDIATE')
        try:
            b.execute('BEGIN IMMEDIATE')
            print('unexpected: second writer acquired')
            b.rollback()
        except sqlite3.OperationalError as exc:
            print('second writer result:', exc)
        a.rollback()
        b.execute('BEGIN IMMEDIATE')
        print('retry after rollback: acquired')
        b.rollback()
    finally:
        a.close()
        b.close()
PY

A typical result reports a locked database for the second writer, followed by successful acquisition after A rolls back. The exact installed SQLite version and error text can differ; filesystem or policy errors are legitimate outcomes and should not be relabeled as a successful contention test. The context manager deletes only its newly created directory after connections close. The lab does not test WAL recovery, power failure or storage durability.

For a source-only alternative, use the supplied two-chunk HTTP request and A/B transaction table. Write the saved parser position and NGX_AGAIN after chunk one, then the successful continuation assumption after chunk two. For SQLite, identify A's transaction, B's busy result, rollback and retry; explain why an I/O error is a different branch. Deliver both tables without running a server or accessing a database.

Knowledge check#

1. Why does the parser save both buffer position and state before returning NGX_AGAIN?
2. Why can a stale event for an old fd be dangerous after the number is reused?
3. What controls retry in the quoted SQLite pager loop?
4. Why can copying only the main database file be insufficient in WAL mode?
5. Why does a successful private transaction not prove crash durability on every filesystem?

Answers#

1. The parser needs to continue from the consumed position and intermediate grammar state when more bytes arrive. The return does not represent a complete request.
2. The integer can now identify a different connection. Without an appropriate instance or generation check, an old callback can act on unrelated or freed state.
3. It retries only while the lock result is SQLITE_BUSY and the busy handler requests another attempt; otherwise it returns the last result.
4. Committed changes can remain in the WAL before checkpointing. A consistent backup must account for the database's actual state and documented backup mechanism.
5. Durability depends on VFS behavior, synchronization settings, storage and failure assumptions. A normal successful commit is not a power-failure experiment.

Checkpoint#

Submit the two-chunk parser timeline, stale-fd counterexample and A/B transaction ledger. Pass when incomplete input differs from invalid input, busy differs from I/O failure, and the local lab is not described as a crash-consistency test. Revisit the parser and pager branches and retry if a readiness event is treated as a complete request. This is not certification.

Sources#

• Nginx 1.26.2 HTTP parser
• Nginx 1.26.2 HTTP request handling
• Nginx 1.26.2 epoll module
• SQLite 3.46.1 pager
• SQLite 3.46.1 Unix VFS
• SQLite 3.46.1 WAL