Fonts, shaping, text, and rasterization

Before you start#

Read userspace graphics and Wayland surfaces. This chapter uses HarfBuzz 9.0.0, Pango 1.54.0 and FreeType 2.13.3 as separately pinned source examples. These versions are not a tested dependency matrix. The practice uses only a literal string and locally installed tools; no display connection is necessary.

Learning goals#

Follow font selection, text itemization, shaping and rasterization as separate steps. Explain clusters, fallback, bidirectional ordering and glyph ownership. Trace two real source paths and distinguish a font-dependent example from an observed rendering.

A useful map is: Fontconfig finds candidate font files; Pango organizes text into layout runs and lines; HarfBuzz turns compatible runs into positioned glyph IDs; FreeType loads glyph images for a renderer. These are roles in a common pipeline, not a guarantee that every application calls each library directly.

Source walkthrough#

A Unicode code point is a numerical value representing a character or another element of text encoding. UTF-8 represents code points using one or more bytes. Neither a byte nor a code point necessarily corresponds to one visible character. A grapheme cluster is an approximation to a user-perceived character: a base letter and combining accent can form one cluster even when represented by two code points. Editing software must choose appropriate cursor and deletion boundaries; counting bytes is insufficient, and even grapheme segmentation is distinct from shaping.

The byte layout of the worked literal below shows why counts differ:

A glyph is a particular drawable form in a font. A glyph ID is an index in a particular font face, not a Unicode code point and not a globally portable identifier. The same code point can select different glyphs in different fonts, while several code points can produce one ligature. Conversely one character can yield multiple glyphs. Shaping associates glyphs with input clusters so applications can relate visual output to source text. Cluster values are caller-supplied labels that help relate shaped glyphs to input; they are not universal character IDs. For UTF-8, byte offsets are a common choice, but callers may supply other values. Shaping and the selected cluster level can merge values. After shaping, HarfBuzz glyph-info codepoint fields hold glyph IDs in the font, not Unicode code points. See HarfBuzz cluster semantics and buffer API.

Fontconfig selects and configures fonts. A requested family, style, weight, language and other properties form a pattern; matching compares it with available font patterns and configuration rules. The result is a best match under that configuration, not proof that every character exists in that font. A fallback mechanism may select other fonts for missing coverage, but which layer performs fallback and how it splits runs depends on the application and toolkit. Fontconfig configuration and cache contents are installation-specific. A cache helps discover font metadata; it does not guarantee that a file remains unchanged or that every glyph can be loaded. See Fontconfig 2.15.0 matching source for a versioned reference to the matching machinery; the worked source excerpts below come from HarfBuzz and FreeType.

Itemization and direction

Pango provides higher-level text layout. It divides text into items or runs with consistent properties such as font, script, language, direction and shaping attributes. A script describes a writing system; it is not the same thing as a spoken language. An application may attach styling spans, and itemization must honor their boundaries as well as the text's properties. A font fallback change can split a run even when the logical text is continuous. The Pango 1.54.0 layout implementation manages layout lines, runs and their measurement.

Bidirectional text has a logical order used for storage and a visual order determined by the bidirectional algorithm and surrounding context. A run of Arabic or Hebrew mixed with Latin and digits is not handled correctly by reversing the entire UTF-8 byte string. Pango and its underlying libraries determine directional runs and visual placement while preserving links back to logical text. A cursor in a mixed-direction line can have distinct visual and logical neighbors. Line breaking, wrapping, ellipsization and selection need these mappings, and a small change in width can alter line breaks without changing the underlying string.

A layout has a lifetime independent of the pixels eventually rendered. Changing its text, attributes, width or font description can invalidate cached measurements and runs. A caller must not retain borrowed pointers into a layout after changing or destroying the owning object unless the API explicitly supplies a reference or copy. Pango objects are reference-counted through the relevant object APIs; reference counting keeps an object allocated but does not make unsynchronized mutation of one layout safe across threads. Applications should use the documented threading guarantees of their installed versions and avoid concurrently modifying shared layout state.

Shaping turns runs into positioned glyphs

HarfBuzz takes a font, a buffer of input text and segment properties such as direction, script and language, plus optional feature controls. It produces glyph IDs and positions: advances move the pen, and offsets position glyphs relative to that pen. GSUB tables can substitute glyph sequences, including ligatures or contextual forms; GPOS can adjust positioning, such as kerning and mark attachment. These transformations depend on the font's tables, selected features, script and shaping implementation. HarfBuzz does not choose all fallback fonts or paint pixels by itself. The application must not assume every font has the same ligatures, glyph IDs, widths or anchor data.

From HarfBuzz 9.0.0 src/hb-shape.cc, hb_shape_full, file copyright Red Hat and Google with its permissive license notice; earlier plan creation and later verification are omitted:

  hb_bool_t res = hb_shape_plan_execute (shape_plan, font, buffer, features, num_features);

  if (buffer->max_ops <= 0)
    buffer->shaping_failed = true;

  hb_shape_plan_destroy (shape_plan);

The first statement executes a plan against the specified font and buffer, producing a success value. The following condition marks shaping as failed if the buffer's operation budget is exhausted. The last statement drops the plan reference; cached plan creation earlier in the function does not remove the need to destroy this reference. The surrounding code can preserve an input copy for optional verification, checks buffer status, and destroys temporary state. Empty input has an early successful return. A caller should still check the appropriate result and buffer state, especially for allocation or shaping failure, rather than treating any buffer pointer as proof of valid glyph output.

The caller owns or references the HarfBuzz font, face and buffer according to their API contracts. A font object also carries scale and variation coordinates, which influence positions and selected outlines. The shaping buffer is mutable: after shaping, its glyph information and positions represent output rather than the original input in the same conceptual form. A caller retaining pointers into internal buffer arrays must obey their validity rules and must not mutate or destroy the buffer underneath another user. Reference counts manage lifetimes, not arbitrary concurrent access to mutable contents. A shape plan can be cached for matching properties, but a cache hit is not a promise that two different fonts or feature settings will shape identically.

Loading outlines and rasterizing

FreeType opens a font face, selects a size and loads glyphs. A face exposes metrics and a glyph slot; loading another glyph into that slot replaces its previous contents. Copy or convert data when it must outlive the next load. A size determines scale and can affect hinting, which adjusts outlines or metrics for a target pixel grid. Rasterization can turn an outline into coverage pixels; color glyphs and bitmap strikes introduce other paths. The eventual renderer blends that coverage with a chosen color and background. Rasterizing an outline does not itself place a full shaped line or implement bidirectional text.

From FreeType VER-2-13-3 src/base/ftobjs.c, FT_Load_Glyph, governed by the FreeType project license referenced by the file; declarations and subsequent driver operations omitted:

    if ( !face || !face->size || !face->glyph )
      return FT_THROW( Invalid_Face_Handle );

The condition requires a valid face with a size and glyph slot before loading can proceed. The return reports an invalid face handle; it is not an empty glyph successfully loaded. The following source assigns the slot and clears its previous contents before proceeding through driver-specific loading. Glyph-index validity is checked by font drivers, as the adjacent source comment says. Applications should check each returned error, release face and library resources on failure, and not read the slot as though a failed load produced valid pixels.

Advanced concurrency note: a single FT_Face must only be used by one thread at a time; separate face instances or explicit locking can be appropriate. This is not a claim that every FreeType operation is globally unsafe across threads. See FreeType face thread-safety guidance. Cached glyph bitmaps must include enough key information, such as font identity, size, variation settings, load flags and rendering mode, to avoid substituting a result produced under different conditions. A file path alone is not an immutable font identity if the file can change. Resource limits, unavailable fonts and malformed data are error cases; this lesson does not ask you to fuzz or open untrusted fonts.

Worked cluster table

The following is synthetic teaching data, not an observation of a specific font. Consider the logical string fi followed by a space and e plus a combining acute accent. In UTF-8, the ASCII characters occupy one byte each, while the combining mark occupies two. Suppose a particular font and enabled features replace fi with a single ligature glyph L, and place accent glyph M over base glyph E. One possible conceptual shaped result is:

The same synthetic table as a mapping, with the merged mark shown explicitly:

The table deliberately uses symbolic glyph names and invented advances. It is not a claim about glyph IDs or the output of HarfBuzz 9.0.0. Another font might have separate f and i glyphs, combine the accented character differently, lack an anchor, or cause fallback to another run. A different buffer cluster policy may yield a different mapping. The accent's zero advance in this example means the pen does not move for that glyph; its offset positions it relative to the base. It does not mean the glyph has zero width or is invisible.

For an RTL example, take a logical Arabic word followed by Latin digits. The system identifies directional and script runs, shapes the Arabic according to contextual forms and font support, and lays out runs according to bidi rules. The visual order is not obtained by reversing the original UTF-8 bytes. The exact ordering of punctuation and digits depends on the complete surrounding text and bidi rules; do not infer an entire paragraph from this simplified description. The cluster mapping lets selection and cursor logic connect the displayed glyphs back to the original logical offsets.

Fallback, missing glyphs, and layout consequences

A font matcher may return an apparently close family while lacking a particular script. The layout engine can split the text into font runs and use a fallback face for the missing portion. A missing glyph indicator, often represented by a font's notdef glyph, is evidence that the selected face did not supply the desired shape; it is not an instruction to substitute a guessed Unicode value for a glyph ID. The fallback face can have different ascent, descent and advances. Consequently line height and wrapping can change when a font package, language preference, font configuration or platform changes, even with identical text.

Font selection and shaping should be repeatable only within a well-described environment. Record the font file and face index, relevant variation settings, language, direction, enabled features, fontconfig configuration and library versions if comparing measurements. A family name is insufficient because it can resolve to different installed files. Even then, renderer settings such as hinting, subpixel policy, scale and color management can change pixels. A screenshot is not needed to establish the logical sequence from selection to glyph loading, and a single screenshot would not prove correctness for all scripts.

A practical failure path starts with a successful match, then a face-open error because the file became unavailable. The application should report or attempt an allowed fallback, release its match and face resources, and avoid reusing a partially initialized glyph slot. If shaping succeeds but rasterization fails for a glyph, the layout's logical mapping may still exist while the renderer cannot produce the requested image. Robust software distinguishes those failures instead of replacing the entire paragraph with arbitrary bytes or dereferencing an invalid slot.

Practice#

The following read-only query asks Fontconfig to match a generic family; it does not install fonts, rebuild caches or open a display. It uses only locally installed configuration and may select a different font on another system:

if command -v fc-match >/dev/null; then
  fc-match --version
  fc-match -f 'family=%{family}\nfile=%{file}\n' sans
else
  printf '%s\n' 'fc-match unavailable; use the supplied source-only table'
fi

Expected output identifies the installed Fontconfig version and a family and file when a match exists. An absent tool, empty match, or inaccessible font is an environment outcome, not proof about the pinned HarfBuzz source. The printed path is local metadata; avoid copying it into a public report if that would reveal private directory names. The source-only alternative is the supplied L/space/E/M table: calculate total advance as 9 + 4 + 7 + 0 = 20 units, identify the two glyphs associated with cluster byte 3, and explain why those symbolic IDs cannot be assumed for a local font. A counterexample is disabling a ligature or choosing a font without it: f and i can then remain separate glyphs while representing the same two input characters.

Knowledge check#

1. How can two code points form one grapheme but still produce more than one glyph?
2. What does HarfBuzz's cluster information connect, and why are its values not universal glyph IDs?
3. What happens to the previous FreeType glyph slot contents when another glyph is loaded?
4. Why can a matching font still require fallback?
5. Why is reversing UTF-8 bytes not a bidirectional layout algorithm?

Answers#

1. Graphemes describe user-perceived text units; shaping selects drawable forms. A base and combining mark can be one grapheme rendered with separate positioned glyphs.
2. Clusters relate output glyphs to input positions under the buffer's chosen indexing and cluster rules. Glyph IDs identify font-specific glyphs and may differ between faces.
3. The source clears the slot during the next load. A caller needing the earlier outline or bitmap must retain an appropriate independent copy or object before replacement.
4. Family and style matching selects a candidate, but that face may lack a requested character or script. A layout system can split a run and choose a fallback if available.
5. UTF-8 uses variable-length byte sequences, and bidi ordering operates on text and contextual direction. Byte reversal can corrupt encoding and ignores logical runs.

Checkpoint#

Submit the synthetic cluster table with its 20-unit total, a diagram of selection, itemization, shaping and rasterization, and one missing-font cleanup path. Pass when glyphs, graphemes and input clusters are distinct and the example is labeled font-dependent. If not, trace the two source excerpts and redo the table; this is practice, not certification.

Sources#

The source paths are HarfBuzz shaping, FreeType glyph loading, Pango layout, and Fontconfig matching. Their versions are separate baselines; local runtime versions may differ.