GStreamer, FFmpeg, and media pipelines

Before you start#

Read PipeWire buffers and scheduling and userspace graphics. GStreamer 1.24.8 and FFmpeg n7.0.2 are independent historical source baselines. The exercise uses rational arithmetic only; it does not decode files, access devices or use a network.

Learning goals#

Trace media through GStreamer elements and FFmpeg's packet/frame decoder interface. Explain negotiation, reference ownership, timestamps, flushing and end-of-stream. Diagnose backpressure without assuming that retrying two blocked operations without state progress will help.

Source walkthrough#

GStreamer represents processing components as elements connected by pads. A GstPipeline is a specialized bin that coordinates a collection of elements, state transitions, clock selection and a bus for application messages. A source produces data, filters transform it, and a sink consumes it, but real pipelines can branch, synchronize and dynamically add or remove pads. A pad has a direction and negotiated capabilities; linking two pads creates a possible route, not proof that every later buffer will be accepted. See GStreamer 1.24.8 gst/gstpad.c.

Caps describe media types and fields such as format, rate, channel count, dimensions or memory features. Negotiation establishes a compatible concrete format among elements; an element can reject caps or require renegotiation when conditions change. A buffer's metadata, timestamps and memory blocks describe that particular buffer and do not replace the negotiated format. A compatible caps intersection is necessary for a direct connection but does not guarantee that allocation, device initialization or later processing will succeed. An adapter element may be needed for conversion.

A GstBuffer contains one or more GstMemory blocks and metadata. Buffers and memory are reference-counted. A reference keeps an object alive but does not mean every holder may mutate it concurrently. Writability and copy-on-write rules determine whether a caller may change a buffer or must obtain a writable copy. Memory may be system memory, shared memory, a DMA-BUF or another allocator type; mapping it can fail or require synchronization. A pointer from a mapping remains valid only for the permitted mapping lifetime. See GStreamer 1.24.8 gst/gstbuffer.c.

Push, pull and thread boundaries

In push mode an upstream element sends data through a source pad to a downstream peer's chain function. In pull mode a downstream component requests ranges from upstream when the pad and element support that scheduling mode. A queue element can create a thread boundary and buffer data, decoupling producer and consumer until its configured limits are reached. Once a queue fills, policy determines whether the producer blocks, data is dropped, or another configured behavior occurs. A thread boundary does not make unbounded buffering safe or remove the need for shutdown and error handling.

From GStreamer 1.24.8 gst/gstpad.c, gst_pad_push, LGPL version 2 or later as stated in the file, with validation and tracer calls omitted:

  res = gst_pad_push_data (pad,
      GST_PAD_PROBE_TYPE_BUFFER | GST_PAD_PROBE_TYPE_PUSH, buffer);

The first line invokes the internal push path for this pad; the second specifies a buffer in push mode and supplies the buffer. The surrounding function validates a source pad and buffer, executes tracing hooks and returns the resulting flow status. A caller must account for transfer ownership under the API and must inspect results such as not-linked, flushing or error. A pad without a usable peer is a normal failure condition during dynamic graph changes, not proof that the entire process must crash. A buffer cannot be modified after ownership was transferred merely because a local variable still contains its address.

A source can be removed while downstream work is pending. Pad blocking, stream locks, state changes and flushing events are tools used by elements and applications to coordinate graph changes; they do not make arbitrary application callbacks safe without ownership discipline. A queue task can still be running while a UI thread requests shutdown. The application should stop new work, initiate the relevant state transition and flush, wait or otherwise coordinate with streaming tasks, and unref elements and buffers only after they cannot be accessed by those tasks. Exact APIs and locking are element-specific.

Events, messages and time

GStreamer events travel through the pipeline to describe stream control, including caps, segment changes, flushing and end of stream. Messages posted to the bus inform the application about errors, state changes and end-of-stream conditions. Events and bus messages have distinct routing and lifetimes: an EOS event reaching downstream and an EOS bus message observed by the application occur at different points. A bus error should trigger cleanup and reporting; a pipeline that posted an error should not be assumed to be playing successfully because a previous state-change request returned.

A GstClock provides a time source selected or supplied for a pipeline. Base time relates clock time to pipeline running time. A segment maps media timestamps into a running-time domain, accounting for start, rate and seek state. Presentation timestamp, or PTS, indicates when a frame should be presented in the relevant stream time base. Decode timestamp, DTS, can differ when decoding order and display order differ. A timestamp without its time base and segment is not a wall-clock time or a direct latency measurement.

A seek can send flush-start and flush-stop events to unblock streaming and clear queued state before a new segment. Old buffers arriving after a flush must not be interpreted as belonging to the new segment. EOS means no more input in that stream, but buffered elements or decoders may still need to drain before the application sees a final completion message. Cancellation and state changes can race with a late bus message; associate messages with the current pipeline lifetime and release their references even if the UI that requested playback has closed.

FFmpeg packets, frames and progress

FFmpeg's libavformat can open a media input and demultiplex it into AVPacket objects belonging to streams. Packets contain encoded data and timing information in stream-specific time bases. A decoder in libavcodec accepts packets using avcodec_send_packet and produces decoded AVFrame objects using avcodec_receive_frame. Frames can contain reference-counted buffers and can be passed to filters, encoders or a renderer under their own contracts. A decoder can produce zero, one or multiple frames for an input packet, and it can retain internal references after a successful send.

The public contract is documented in the FFmpeg 7.0 decoding API; implementation details can be traced in FFmpeg n7.0.2 libavcodec/decode.c, whose file notice is LGPL-2.1-or-later. Earlier state checks and later drain handling are omitted:

        if (!AVPACKET_IS_EMPTY(avci->buffer_pkt))
            return AVERROR(EAGAIN);
        ret = av_packet_ref(avci->buffer_pkt, avpkt);
        if (ret < 0)
            return ret;

The first condition detects an occupied internal packet buffer and returns EAGAIN rather than accepting another packet. The third line takes a reference to the supplied packet when the buffer is available; the final condition propagates failure to take that reference. The caller retains responsibility for its own packet reference and can unref it at the appropriate point after a successful send. An EAGAIN result means the application should attempt the complementary receive operation and follow the documented state machine, not discard the packet as though it was accepted.

The send/receive API is a state machine. If avcodec_send_packet returns EAGAIN, the packet was not accepted: call avcodec_receive_frame to consume available output, then retry that same packet. After a successful send, receive frames until EAGAIN says more input is needed. The API guarantees that send and receive will not both return EAGAIN without an intervening state change, so sleeping and repeating one blocked call cannot make progress. Decoder-specific delay and reordering mean that one packet does not map to one output frame. At end of input, send a null packet to begin draining, then receive delayed frames until AVERROR_EOF; during draining, EAGAIN is not the expected completion signal. A seek or discontinuity requires flushing decoder state under the API before reusing it for the new stream position.

In libavcodec/decode.c, n7.0.2, LGPL-2.1-or-later (inside decode_simple_internal; surrounding code omitted):

    // Some codecs (at least wma lossless) will crash when feeding drain packets
    // after EOF was signaled.
    if (avci->draining_done)
        return AVERROR_EOF;

Once end of stream has been recorded, the decoder returns AVERROR_EOF for further input instead of feeding the codec another packet. The comment gives a codec-specific crash as the reason. The excerpt does not show the drain loop or what a player should do with frames still pending.

A packet or frame must be unreferenced or freed when its owner no longer needs it. av_packet_unref releases the caller's packet contents; av_frame_unref releases frame references while retaining the frame object for reuse. A filter or encoder may take or retain a reference according to its own API, so the caller must read that interface's ownership contract. On an error after demux, decoder allocation or filter initialization, unwind only successfully created contexts and references. A reference count prevents premature free but does not allow unsynchronized concurrent mutation of the same frame metadata.

In libavcodec/decode.c, n7.0.2, LGPL-2.1-or-later (surrounding code omitted):

    } else
        ret = decode_simple_receive_frame(avctx, frame);

    if (ret == AVERROR_EOF)
        avci->draining_done = 1;

Both branches reach the same end-of-stream check, so an AVERROR_EOF from either one sets the drain flag. The excerpt does not show the receive-callback branch or frame cleanup.

Worked timestamp and reorder example

Use a synthetic video stream time base of 1/90,000 second and an audio stream time base of 1/48,000 second. Video PTS 9,000 represents 0.1 second; audio PTS 4,800 also represents 0.1 second. Their integers differ by almost a factor of two but represent the same time in their respective units. Rescale using rational arithmetic with an explicit rounding policy rather than comparing raw integers; see the FFmpeg timestamp API. At a 48 kHz rate, 480 audio frames span 0.01 second. This is a calculation, not an observed playback trace.

For reordering, imagine encoded packets arrive in decode order for frames whose presentation order differs. A decoder may consume a packet without immediately emitting the frame with the same apparent timestamp, and may emit a delayed frame only after later input or drain. DTS helps express decoding order while PTS expresses intended presentation. A sink uses timestamps, segment and clock state to decide when to present; sorting packets by PTS before decoding is not a general substitute for respecting codec dependencies. The exact sequence depends on the codec and container.

Suppose input ends after the final packet while the decoder retains one delayed frame. If the program stops immediately, that frame is lost. The proper drain sends the null input as documented, receives remaining frames and then observes EOF. If a seek occurs instead, stale buffered frames should not be presented under the new segment; flush decoder state and reset the application's timestamp mapping as required. A failed allocation or decoder error still requires packet, frame and context cleanup. A timestamp calculation alone cannot establish that any frame was displayed.

Framework choices and licensing

GStreamer and FFmpeg can be combined but have distinct abstractions. GStreamer emphasizes element graphs, pad negotiation and events; FFmpeg libraries expose demuxing, codec and format primitives that applications compose. A GStreamer plugin can internally use FFmpeg in some builds, but that is not a universal architecture. Installed plugin sets and codec availability vary. A source file's license does not settle the licensing of every optional library or build configuration: the quoted GStreamer file states LGPL version 2 or later, and the quoted FFmpeg file states LGPL 2.1 or later, while optional components and build options can introduce different terms. Consult the actual distributed build and project notices for licensing questions.

A container's stream time base, decoder state and output format must be tracked separately. A video frame can have a timestamp but no usable pixel buffer after an error; a buffer can be valid but have an unknown timestamp. A queue can hold references across a state transition. An application that responds to cancellation by freeing a context while a worker is still decoding risks a race regardless of which framework it uses. Establish a stop signal, unblock queues using the framework's flush mechanism, join or otherwise synchronize worker completion, then release owned contexts and references.

Backpressure and ownership across a queue

Imagine a producer has prepared packet P but the decoder returns EAGAIN. P remains the producer's unaccepted input; freeing it before a later successful send would lose the packet. The consumer calls receive and obtains frame F, takes or transfers the needed reference downstream, and unreferences its own frame before reusing the frame object. The producer can then retry P. If the downstream queue is full, the application must choose an explicit blocking, dropping or cancellation policy rather than allocate frames without limit. A seek changes the interpretation of queued timestamps, so the application flushes old work and avoids delivering F as though it belonged to the new segment. These are state transitions, not evidence that a particular codec was executed.

A pipeline can retain buffered data after an upstream input stops, so end-of-stream and completion are not the same instant. A downstream error can also arrive while another element still owns a buffer. The application must wait for the relevant bus or callback result and release its references according to the selected library's contract; a displayed frame count alone does not prove that every buffer was processed successfully.

Practice#

This private model compares two rational timestamps and simulates decoder progress. It reads no media and creates no pipeline:

python3 - <<'PY'
from fractions import Fraction
video = Fraction(9000, 90000)
audio = Fraction(4800, 48000)
print('video seconds:', video, 'audio seconds:', audio)
print('same time:', video == audio)
state = ['send packet accepted', 'receive until EAGAIN', 'send NULL drain', 'receive until EOF']
print('synthetic states:', ' -> '.join(state))
PY

Expected output shows both timestamps as 1/10 and True. A counterexample is comparing raw integers 9,000 and 4,800 without their time bases; that falsely labels equal times as different. The state list is a teaching model, not a decoder run. In your state table, include the alternate branch where send returns EAGAIN: receive output, retry the still-unaccepted packet, and record when receive returns EAGAIN or drain reaches EOF. Then account for packet and frame references.

Knowledge check#

1. What is the difference between caps and a buffer's metadata?
2. What does a queue element change about execution?
3. What does avcodec_send_packet return when its internal packet buffer is occupied?
4. Why can PTS and DTS differ?
5. Why must a decoder be drained at end of input?

Answers#

1. Caps describe negotiated media format and features; buffer metadata and memory describe a particular buffer under that format.
2. It can introduce a thread and buffering boundary. Limits, blocking and teardown still need explicit handling.
3. The quoted path returns EAGAIN without taking the new packet reference. The caller must make progress through the complementary operation.
4. Encoded frames can be decoded in a different order from presentation, so decoding and display timing are distinct.
5. The decoder can retain delayed frames; a null input and receive loop under the API recover them before EOF.

Checkpoint#

Submit a caps-to-buffer ownership diagram, the 1/10-second calculation and an FFmpeg send/receive/drain state table. Pass when no raw timestamp comparison ignores its time base and every packet/frame reference has a cleanup owner. If not, retrace the two excerpts and repeat the model. This is practice, not certification.

Sources#

Read GStreamer pad implementation, GStreamer buffer implementation, FFmpeg decoder implementation, and FFmpeg demux implementation. The examples are synthetic and no media was decoded.