Skip to main content
Application Note

Analyzing QLI GStreamer Video Encoder Pipelines

Platform: Qualcomm Linux, GStreamer 1.28+ Encoder: v4l2h264enc (V4L2-based hardware H.264 encoder) Applies to: Any GStreamer pipeline using a V4L2 video encoder

Introduction

GStreamer is a widely used open-source multimedia framework for building audio and video processing pipelines on Linux-based embedded systems. In applications such as surveillance cameras, industrial vision systems, and video conferencing devices, video is captured from a camera, encoded in hardware, and either stored to a file or streamed over a network in real time. Understanding the latency and frame integrity of such a pipeline is essential — not only for performance tuning but also for diagnosing unexpected behaviour in production. GStreamer’s built-in debug logging system provides a rich, timestamped record of everything the pipeline does internally, including buffer allocations, latency negotiation, frame handling, and timing information. This application note focuses on pipelines that use the v4l2h264enc element — a GStreamer wrapper around a V4L2 kernel-level hardware H.264 encoder, commonly found on ARM SoCs such as Qualcomm platforms. The techniques described here apply to any GStreamer pipeline using a V4L2-based encoder and can be adapted for other encoder elements with similar debug output. Specifically, this note addresses the following questions:
  • What latency does the encoder introduce, and why?
  • How many buffers does the encoder use, and on which side?
  • How long does it actually take to encode each frame?
  • Were any frames dropped during the recording or stream?
  • What does consistent PTS spacing actually prove — and what does it not prove?
All analysis is performed directly from GStreamer debug logs, without requiring additional instrumentation or source code modification. The log search patterns and shell commands provided work on any log captured with GST_DEBUG=6 or higher.

Overview

When a GStreamer pipeline uses a hardware encoder such as v4l2h264enc, the encoder introduces a measurable pipeline latency. This latency is not arbitrary — it is determined by the encoder driver’s buffer requirements and is fully observable from GStreamer debug logs. This app note explains:
  • How to read the encoder’s self-reported latency from logs
  • Why the latency has the value it does (buffer pool depth)
  • How to measure actual per-frame encode time
  • How to verify frame integrity (no drops) using PTS spacing and frame number sequence
To capture logs with sufficient detail, run the pipeline with:
Reference pipeline:

1. Encoder Self-Reported Latency

What to look for

Search the log for: gst_video_encoder_set_latency Example log line:

What it tells you

Note: This is a declared buffering latency, not a measured processing latency. It does not mean each frame takes 133 ms to encode. It means the encoder pipeline must be filled to a certain depth before output begins.

2. Why the Latency Has That Value — Buffer Pool Depth

The V4L2 queue model

V4L2 encoders use two internal queues. Note that V4L2 naming is from the device’s perspective — OUTPUT means output from the application to the device (i.e., encoder input), which is the reverse of what you might expect.

How the latency is calculated

The encoder latency is derived from the minimum number of CAPTURE (output) buffers the driver requires:
For example, at 30 fps with 4 required output buffers:
The encoder must keep 4 output slots populated simultaneously. It cannot release the first encoded frame until all 4 slots have been filled — meaning 4 input frames must have been processed first.

Where to verify in the gst log

Search for gst_v4l2_get_driver_min_buffers:
The output (src) side drives the latency. The input (sink) side imposes no buffering requirement.

3. GStreamer’s Minimum Buffer Floor

Why the sink pool shows 2 buffers even when the driver says 0

GStreamer enforces a minimum of 2 buffers on any V4L2 buffer pool, regardless of what the driver reports. This is a safety floor to prevent pipeline deadlock: one buffer can be held by the upstream element while another is queued with the driver.

Where to verify in the gst log

Search for increasing minimum buffers for the sink pad:
Compare with the src (output) pad, where the driver’s requirement of 4 is already above the floor of 2:

Summary of buffer allocation

4. Measuring Actual Per-Frame Encode Time

The encoder’s self-reported latency describes pipeline depth, not how long it takes to encode a single frame. To measure actual encode time, pair the handle_frame (input) and finish_frame (output) log entries for each frame number.

Log lines to use

Input — when the encoder receives a raw frame:
Output — when the encoder has finished encoding (use the frame PTS line):
Note: Use the “frame PTS” log line as the finish timestamp — it confirms the encoded buffer is fully formed. It appears immediately after “finish frame fpn N” (~7 µs later, negligible).

Calculation

Both timestamps are wall-clock values from the GStreamer log. handle_frame runs on the encoder’s input thread; finish_frame runs on its output thread — so this correctly measures real elapsed time across both threads.

Typical observations

Note: If steady-state encode time consistently exceeds the frame period, the encoder is slower than real-time and queues will grow — eventually causing latency build-up or frame drops in a live pipeline.

5. Verifying Frame Integrity — PTS Spacing

What PTS spacing tells you

The Presentation Timestamp (PTS) on each buffer indicates when the frame should be presented. For a 30 fps stream, consecutive frames should have PTS values spaced 33.33 ms apart.

Consistent PTS spacing means:

  • The content is correctly tagged as a 30 fps stream

It does NOT mean:

  • Frames were captured or encoded without drops
  • PTS values are assigned by the camera source element based on the declared framerate — surviving frames are always tagged sequentially regardless of whether intermediate frames were dropped

How to check PTS spacing

Extract PTS values from the muxer’s buffer enqueue log lines:
Then check spacing between consecutive values. All gaps should be within a small tolerance of the expected frame period (e.g., ±1 ms for 30 fps).

What anomalies mean

6. Verifying Frame Integrity — Frame Number Sequence

PTS spacing analysis can be misleading at startup and cannot distinguish between a dropped frame and a timestamp irregularity. The definitive check for frame drops is the frame number sequence in handle_frame logs.

What to look for

The encoder logs a monotonically increasing frame number for every frame it receives:
A gap in this sequence (e.g., Handling frame 4 followed by Handling frame 6) means frame 5 was dropped before reaching the encoder.

How to extract and verify

If no gap output is produced, all frames were delivered to the encoder in sequence with no drops.

Combining both checks

Quick Reference — Log Search Patterns

Summary

Conclusion

GStreamer debug logs contain all the information needed to fully characterise the latency and frame integrity of a hardware-encoded video pipeline — without modifying source code or adding external instrumentation. The key insight is knowing which log lines to look for and what they actually mean. The following points summarise what has been covered in this app note:
  • The encoder’s self-reported latency (gst_video_encoder_set_latency) directly reflects the number of output CAPTURE buffers the driver requires multiplied by the frame duration. For v4l2h264enc at 30 fps with 4 output buffers, this is a fixed 133.33 ms. Because min_latency equals max_latency, this delay is constant and predictable.
  • The 4 buffers responsible for this latency are output-side (CAPTURE queue) buffers, not input buffers. The input (sink) side requires only 2 buffers — a floor enforced by GStreamer to prevent deadlock, overriding the driver’s reported value of 0.
  • Actual per-frame encode time is measured by pairing handle_frame (input) and finish_frame / frame PTS (output) wall-clock timestamps for each frame number. During startup the encode time is elevated while the encoder fills its buffer pipeline; in steady state it is typically well below the frame period, confirming the encoder is not a throughput bottleneck.
  • PTS spacing at the muxer tells you only that frames are correctly timestamped for the declared framerate. It does not prove that no frames were dropped. Consistent 33.33 ms PTS gaps are expected even when frames have been lost, because remaining frames are tagged sequentially by the source element.
  • The definitive proof of zero frame drops is a contiguous handle_frame number sequence (0, 1, 2, … N) in the encoder input logs. Any gap in this sequence identifies exactly which frame was dropped and where in the pipeline the drop occurred.
Together these techniques give developers a repeatable, log-only methodology for validating encoder behaviour across different hardware platforms, resolutions, and framerates — making it straightforward to include latency and frame integrity evidence directly in engineering reports and application notes.