PAM, NSS, authentication, and accounts

Before you start#

Review libc and threads, credentials, userspace boundaries, and logind sessions. This chapter reads glibc 2.40 commit 3d1aed874918c466a4477af1da35983ab036690e and Linux-PAM 1.6.1 commit 9438e084e2b318bf91c3912c0b8ff056e1835486. These separate historical source baselines are not a tested host configuration.

Learning goals#

Separate name and account lookup from authentication, account policy and session setup. Interpret NSS status and action rules, a reentrant lookup's buffer ownership, and a PAM application's handle and cleanup responsibilities. Analyze an invented authentication-success/account-denial case without calling a real authentication service.

Source walkthrough#

Name Service Switch, or NSS, is a libc mechanism for looking up categories of information using configured sources. A passwd lookup can map a name to a UID and associated account fields. A host-name lookup can use other sources under a separate configuration entry. A successful name-to-UID mapping establishes neither that a person proved possession of an authentication token nor that account policy permits a login. A remote NSS backend can be slow, unavailable or return no match; those outcomes have different meanings. The GNU C Library documents NSS status actions as success, notfound, unavail and tryagain; default actions normally return on success and continue on other statuses, but configuration can override them. In particular, NOTFOUND means the lookup completed without the requested value, whereas UNAVAIL describes an unavailable service. (glibc NSS actions)

The exact excerpt below is from glibc 2.40 nss/nsswitch.c, __nss_next2; the file carries an LGPL-2.1-or-later notice. It is within the ordinary status branch, with surrounding validation omitted:

       if (nss_next_action (*ni, status) == NSS_ACTION_RETURN)
	 return 1;

ni points to the current NSS action-list position, and status describes the backend result under this internal interface. The action lookup decides whether that result should stop the source traversal. If it is RETURN, this function returns 1 to indicate that no further source should be tried along this path. The same function has separate checks for all-values traversal and the end of the list. Its return is an internal traversal decision, not itself the public result of every getpwnam_r call. The surrounding backend implementation and configuration determine which status arrives.

Linux-PAM is a separate library and module framework. An application creates a pam_handle_t for a service and conversation, requests PAM operations, handles their returned status and eventually calls pam_end. Modules execute through a configured stack under the application's process and authority. The exact excerpt from Linux-PAM 1.6.1 libpam/pam_dispatch.c, _pam_dispatch, follows handler initialization; the release's Copyright permits BSD-style redistribution or, alternatively, GPL terms where no file-specific exception applies:

    case PAM_AUTHENTICATE:
	h = pamh->handlers.conf.authenticate;
	break;

choice selects the requested operation in the surrounding switch. For this case, h points at the configured authentication handler chain associated with this PAM handle, and break leaves the switch. Other operations select different chains. The dispatch machinery later walks handlers and combines their results according to configured control actions. This selection does not establish that account checks or session setup also succeeded. It is a source-reading branch, not an executed authentication attempt.

In libpam/pam_dispatch.c, _pam_dispatch_aux, Linux-PAM 1.6.1, under the BSD-style or GPL terms of the release Copyright notice (the surrounding loop and other action cases omitted):

	case _PAM_ACTION_OK:
	case _PAM_ACTION_DONE:

	    if ( impression == _PAM_UNDEF
		 || (impression == _PAM_POSITIVE && status == PAM_SUCCESS) ) {
                /* in case of using cached chain
                   we could get here with PAM_IGNORE - don't return it */
                if ( retval != PAM_IGNORE || cached_retval == retval ) {
		    impression = _PAM_POSITIVE;
                    status = retval;
                }
	    }
	    if ( impression == _PAM_POSITIVE ) {
		if ( retval == PAM_SUCCESS ) {
		    h->grantor = 1;
		}

		if ( action == _PAM_ACTION_DONE ) {
		    goto decision_made;
		}
	    }
	    break;

For the OK and DONE actions, the impression changes only when it is still undefined or is already positive with a success status. A remembered negative impression is not replaced here. A DONE action ends the decision only when the impression is positive. This is the state-machine basis for saying that a later sufficient success does not erase an earlier required failure. The mapping from return value to action comes from the handler action table, which the excerpt does not show.

Lookup status, buffers and identity#

A glibc NSS configuration can name sources for a database and attach action clauses to backend statuses. Typical internal categories include SUCCESS, NOTFOUND, UNAVAIL and TRYAGAIN, with actions such as returning or continuing. A missing account is not the same as a directory service being unreachable, and a temporary failure is not proof of permanent absence. The exact source order and action overrides are configuration-dependent. An administrator might intentionally stop on one result, while another configuration continues to the next source. Do not read or modify the host's NSS configuration as part of this exercise.

The reentrant getpwnam_r interface accepts caller-provided storage for a result structure and character buffer. A zero return with result set to pwd indicates a found entry; zero with a null result means no matching entry. An error such as insufficient buffer space is represented separately by the return code, commonly ERANGE for a buffer that must be enlarged; on error, result is null. The caller owns the supplied memory and must keep it alive while using pointers into that buffer. It must check the actual interface contract and implementation; neither an empty result nor a failed lookup should be silently converted to an authenticated identity. (getpwnam(3))

In nss/getXXbyYY_r.c, the template that builds getpwnam_r through nss/getpwnam_r.c, glibc 2.40, LGPL-2.1-or-later under its notice (the service lookup call and the final result mapping omitted):

      /* The status is NSS_STATUS_TRYAGAIN and errno is ERANGE the
	 provided buffer is too small.  In this case we should give
	 the user the possibility to enlarge the buffer and we should
	 not simply go on with the next service (even if the TRYAGAIN
	 action tells us so).  */
      if (status == NSS_STATUS_TRYAGAIN
#ifdef NEED_H_ERRNO
	  && *h_errnop == NETDB_INTERNAL
#endif
	  && errno == ERANGE)
	break;

A TRYAGAIN status with ERANGE means the caller buffer is too small. The comment says the lookup must not move on to the next service in that case, and the break leaves the loop so the caller can enlarge its buffer. The final result mapping then returns the errno value, so the buffer condition is not reported as a found or absent entry. The excerpt does not show the NOTFOUND or UNAVAIL branches.

Use a synthetic result table. With buffer capacity 8, assume the implementation reports ERANGE and provides no usable entry; the caller allocates a larger buffer and retries. With capacity 256, assume the return is zero and the result pointer references an entry for demo, UID 1200. The caller can now interpret that returned data for this lookup. A different run can return zero with a null result, meaning no match, or return a backend error. These values are invented for reasoning and are not observations of a host account. A caller that frees the buffer and retains pw_name has a dangling pointer even if the string appears unchanged briefly.

NSS modules implement backend interfaces for selected databases; they may query local files, a cache, a remote service or another configured mechanism. A cache can make repeated results fast but introduces its own invalidation, stale-data and availability questions. A UID-to-name mapping can change between lookup and later use, and a process's actual credentials are kernel state, not merely a displayed name. NSS results may be used by an application as input to policy, but successful lookup is not a password check and does not establish authorization to access a file or service.

PAM operations and conversation ownership#

A PAM-aware application starts a transaction with pam_start, supplying a service name, optional initial user and conversation function. The handle retains state for this transaction. The conversation callback mediates prompts and responses between modules and the application; applications must protect secrets, follow allocation ownership rules and avoid logging tokens. A module is code loaded under the application's execution context, with whatever authority and resources that context provides. Modules can contact services, change credentials or produce side effects. This is why the practice below does not call PAM, even with invented credentials.

pam_authenticate asks the authentication stack to establish the required authentication result. pam_acct_mgmt is a separate account-management phase that can reject an expired, disabled or otherwise disallowed account even after authentication succeeds. Credential establishment and session open/close are separate operations used by applications when appropriate, and password-changing operations have their own stack. An application must follow the relevant return codes, its PAM integration contract and cleanup path; calling authentication alone does not create a session or grant every privilege.

The pam_end function receives the handle and a final status so modules can release per-transaction data. In the inspected libpam/pam_end.c, module data cleanup precedes freeing handlers because module cleanup can need handle state. The application must not use the handle after successful teardown and must also close any session or delete credentials it established under the applicable PAM contract. If an earlier step fails, it still needs an appropriate pam_end path; that does not imply calling a session-close operation for a session that never opened.

A conversation can fail or be canceled; a module can return an error after a prompt, and the user or remote identity may change before a later authorization decision. An application must not retain a pointer to a temporary response buffer after its ownership ends. It should avoid leaving secrets in logs or collecting data it does not need. The code and configuration of a particular PAM stack determine which modules run and what they do; a source map of libpam's dispatcher is not proof that an installed system uses any specific module.

Controls and a worked stack#

The familiar control flags required, requisite, sufficient and optional are shorthand for PAM control behavior, but their effects depend on the operation, previous failures and configured stack. A required failure is remembered while processing generally continues; a requisite failure can terminate the relevant stack early. A sufficient success can stop further processing when the applicable prior-failure condition allows it; it must not erase an earlier required failure. Optional results affect the overall result under the documented rules, particularly when no other relevant module provides a result. Extended bracketed control syntax can express more precise mappings and jumps.

include and substack incorporate configuration but do not have identical control-flow boundaries. In particular, a terminating action or jump in a substack is constrained by the substack's semantics rather than blindly skipping arbitrary outer-stack entries. Exact behavior must be checked against the pinned Linux-PAM configuration documentation and the actual stack. A diagram showing only one line per module cannot model every extended control action or asynchronous conversation.

Consider an invented service with an authentication stack in which a required module succeeds, followed by a sufficient module that also succeeds and can end that authentication stack under its configured conditions. The application receives authentication success. It then invokes a distinct account stack, whose required module reports an expired-account denial. The application must treat the account phase as denied; it must not open a session merely because the password-like check succeeded. It passes the final relevant status to cleanup and calls pam_end after any required application-side cleanup. No real user, password, login, PAM module or host configuration is involved in this model.

A second invented authentication case has a required module fail, followed by a sufficient module that succeeds. The remembered required failure prevents treating the later success as a clean overall success under the ordinary shorthand semantics. A requisite failure instead can stop the applicable stack earlier. Which prompts were shown and which modules performed work are observable only in an actual authorized execution; the model makes no such claim. The application must treat returned codes as data and choose its response without exposing unnecessary information about which account or credential condition failed.

Trust boundaries and race conditions#

A lookup can return a UID while the backing account changes before a later action. An authentication transaction can succeed while a subsequent account check denies access. A session can be opened and then fail during a later application initialization step. These boundaries require explicit state in the application: handle created, authenticated, account approved, credentials established if applicable, session opened if applicable, then closed and ended. Do not call cleanup for a resource as though it was acquired when the corresponding operation failed; do not skip cleanup for one that was acquired.

A PAM application can run privileged or unprivileged depending on its design and environment. Its modules run in the process context and may have access to sensitive resources. Authentication success is not a universal authorization decision for every operation the process can perform. Kernel credentials, file permissions, capabilities, security modules and service policy remain separate checks. Conversely, a name displayed by a lookup is not a durable proof of the credentials attached to a running task. When a remote backend is unavailable, the configured action and service policy decide what happens; an application should not invent a success from a lookup failure.

A login program should also distinguish an unavailable identity backend from an expired account. Retrying a transient lookup can be appropriate under a bounded policy, whereas repeatedly prompting for credentials cannot repair an unavailable directory server by itself. The application should report a useful failure without exposing whether a sensitive account exists, and its logs should avoid passwords and conversation responses. Those are application design choices; the NSS and PAM sources cited above do not prescribe one universal user-facing message.

Practice#

This pure Python model contains invented boolean outcomes. It does not call NSS, PAM, a login manager, a password prompt or a host account database:

python3 - <<'PY'
def outcome(auth_required, auth_sufficient, account_allowed):
    auth_ok = auth_required and auth_sufficient
    return 'allow' if auth_ok and account_allowed else 'deny'
print('auth success, account denied:', outcome(True, True, False))
print('required failure, later success:', outcome(False, True, True))
print('all modeled checks succeed:', outcome(True, True, True))
PY

Expected output is deny, deny, then allow. This deliberately simplified function is not a PAM implementation: it does not implement control flags, conversations, modules, substacks or policy. Its purpose is to preserve the two-phase counterexamples. If Python is unavailable, use the same three rows by hand. No credentials or system configuration are accessed.

Source-only alternative: draw the synthetic buffer table (8 bytes with ERANGE, 256 bytes with a found record, and zero return with null result for absence), then trace the invented authentication-success/account-denial case through handle cleanup. Mark all values invented. Deliver a state table showing which operations succeeded, what must be released, and why a lookup is not authentication.

Knowledge check#

1. How can getpwnam_r return zero without returning an account, and who owns the result buffer?
2. Why is UNAVAIL different from NOTFOUND in an NSS source chain?
3. What chain does the quoted PAM dispatch case select, and what does it not establish about account policy?
4. Why must a prior required failure not be erased by a later sufficient success in the worked case?
5. Why is pam_end needed after an unsuccessful transaction when a handle was created?

Answers#

1. A zero return with a null result pointer can indicate no matching entry. The caller supplies and retains the structure and character buffer while using pointers into that buffer; a separate error code can require a retry with more space.
2. They represent different backend outcomes: an unavailable source is not evidence that the account is absent. Configured action clauses determine whether to continue or return for a particular status.
3. It selects the handle's configured authentication handler chain. Account management is a separate operation and can deny an otherwise authenticated transaction.
4. A required failure is remembered under the shorthand control semantics. Later sufficient success cannot convert that prior failure into an unqualified overall success for the modeled stack.
5. The handle can own module and transaction data even after an earlier operation fails. The application must follow the PAM cleanup contract and pass an appropriate status, without inventing a successfully opened session.

Checkpoint#

Submit the NSS buffer/result table and PAM operation/cleanup state diagram, with either the private model output or the supplied source-only answers. Ready means distinguishing absence from error, authentication from account approval, and handle lifetime from application session lifetime. Retry with a single lookup and a single account denial before adding control flags.

Sources#

• glibc 2.40 NSS traversal
• glibc 2.40 reentrant lookup template
• Linux-PAM 1.6.1 dispatcher
• Linux-PAM 1.6.1 handle cleanup
• Linux-PAM 1.6.1 configuration manual