> ## Documentation Index
> Fetch the complete documentation index at: https://dragonwingdocs-staging.qualcomm.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Analyzing QLI GStreamer Video Encoder Pipelines

> A log-based method for analyzing latency, buffer depth, encode time, and frame integrity in GStreamer pipelines using V4L2 hardware encoders.

**Application Note**

# Analyzing QLI GStreamer Video Encoder Pipelines

**Platform:** Qualcomm Linux, GStreamer 1.28+

**Encoder:** `v4l2h264enc` (V4L2-based hardware H.264 encoder)

**Applies to:** Any GStreamer pipeline using a V4L2 video encoder

## Introduction

GStreamer is a widely used open-source multimedia framework for building audio and video processing pipelines on Linux-based embedded systems. In applications such as surveillance cameras, industrial vision systems, and video conferencing devices, video is captured from a camera, encoded in hardware, and either stored to a file or streamed over a network in real time.

Understanding the latency and frame integrity of such a pipeline is essential — not only for performance tuning but also for diagnosing unexpected behaviour in production. GStreamer's built-in debug logging system provides a rich, timestamped record of everything the pipeline does internally, including buffer allocations, latency negotiation, frame handling, and timing information.

This application note focuses on pipelines that use the `v4l2h264enc` element — a GStreamer wrapper around a V4L2 kernel-level hardware H.264 encoder, commonly found on ARM SoCs such as Qualcomm platforms. The techniques described here apply to any GStreamer pipeline using a V4L2-based encoder and can be adapted for other encoder elements with similar debug output.

Specifically, this note addresses the following questions:

* What latency does the encoder introduce, and why?
* How many buffers does the encoder use, and on which side?
* How long does it actually take to encode each frame?
* Were any frames dropped during the recording or stream?
* What does consistent PTS spacing actually prove — and what does it not prove?

All analysis is performed directly from GStreamer debug logs, without requiring additional instrumentation or source code modification. The log search patterns and shell commands provided work on any log captured with `GST_DEBUG=6` or higher.

## Overview

When a GStreamer pipeline uses a hardware encoder such as `v4l2h264enc`, the encoder introduces a measurable pipeline latency. This latency is not arbitrary — it is determined by the encoder driver's buffer requirements and is fully observable from GStreamer debug logs.

This app note explains:

* How to read the encoder's self-reported latency from logs
* Why the latency has the value it does (buffer pool depth)
* How to measure actual per-frame encode time
* How to verify frame integrity (no drops) using PTS spacing and frame number sequence

To capture logs with sufficient detail, run the pipeline with:

```bash theme={null}
export GST_DEBUG_NO_COLOR=1
export GST_DEBUG=6
export GST_DEBUG_FILE=/path/to/output.log
gst-launch-1.0 <your pipeline>
```

Reference pipeline:

```bash theme={null}
gst-launch-1.0 -e qtiqmmfsrc name=camsrc video_0::type=preview ! video/x-raw,format=NV12_Q08C,width=1920,height=1080,framerate=30/1,interlace-mode=progressive,colorimetry=bt601 ! queue ! v4l2h264enc capture-io-mode=4 output-io-mode=5 ! queue ! h264parse ! mp4mux ! queue ! filesink location="/opt/mux1.mp4"
```

## 1. Encoder Self-Reported Latency

### What to look for

Search the log for:

`gst_video_encoder_set_latency`

Example log line:

```text theme={null}
0:00:00.836238072  1553  1569  DEBUG  videoencoder
  gstvideoencoder.c:2987:gst_video_encoder_set_latency:
  <v4l2h264enc0> min_latency:0:00:00.133333332 max_latency:0:00:00.133333332
```

### What it tells you

| Field | Value | Meaning |
| - | - | - |
| `min_latency` | 133.33 ms | Declared minimum delay before first encoded output |
| `max_latency` | 133.33 ms | Declared maximum delay |
| `min == max` | — | Fixed, deterministic latency — no jitter |

Note: This is a declared buffering latency, not a measured processing latency. It does not mean each frame takes 133 ms to encode. It means the encoder pipeline must be filled to a certain depth before output begins.

## 2. Why the Latency Has That Value — Buffer Pool Depth

### The V4L2 queue model

V4L2 encoders use two internal queues. Note that V4L2 naming is from the device's perspective — OUTPUT means output from the application to the device (i.e., encoder input), which is the reverse of what you might expect.

| V4L2 Queue | GStreamer Pad | Direction |
| - | - | - |
| OUTPUT queue | sink pad | Raw frames going into the encoder |
| CAPTURE queue | src pad | Encoded bitstream coming out of the encoder |

### How the latency is calculated

The encoder latency is derived from the minimum number of CAPTURE (output) buffers the driver requires:

```text theme={null}
latency = num_capture_buffers × frame_duration
```

For example, at 30 fps with 4 required output buffers:

```text theme={null}
latency = 4 × (1/30 s) = 4 × 33.33 ms = 133.33 ms
```

The encoder must keep 4 output slots populated simultaneously. It cannot release the first encoded frame until all 4 slots have been filled — meaning 4 input frames must have been processed first.

### Where to verify in the gst log

Search for `gst_v4l2_get_driver_min_buffers`:

```text theme={null}
DEBUG  v4l2  gstv4l2object.c:954:gst_v4l2_get_driver_min_buffers:
  <v4l2h264enc0:sink> driver requires a minimum of 0 buffers    ← INPUT side
DEBUG  v4l2  gstv4l2object.c:954:gst_v4l2_get_driver_min_buffers:
  <v4l2h264enc0:src>  driver requires a minimum of 4 buffers    ← OUTPUT side
```

The output (src) side drives the latency. The input (sink) side imposes no buffering requirement.

## 3. GStreamer's Minimum Buffer Floor

### Why the sink pool shows 2 buffers even when the driver says 0

GStreamer enforces a minimum of 2 buffers on any V4L2 buffer pool, regardless of what the driver reports. This is a safety floor to prevent pipeline deadlock: one buffer can be held by the upstream element while another is queued with the driver.

### Where to verify in the gst log

Search for increasing minimum buffers for the sink pad:

```text theme={null}
INFO  v4l2bufferpool  gstv4l2bufferpool.c:603:gst_v4l2_buffer_pool_set_config:
  <v4l2h264enc0:pool0:sink> increasing minimum buffers to 2
```

Compare with the src (output) pad, where the driver's requirement of 4 is already above the floor of 2:

```text theme={null}
INFO  v4l2bufferpool  gstv4l2bufferpool.c:610:gst_v4l2_buffer_pool_set_config:
  <v4l2h264enc0:pool0:src> increasing minimum buffers to 4
```

### Summary of buffer allocation

| Pad | Driver Minimum | GStreamer Floor | Final Allocation |
| - | - | - | - |
| sink (input) | 0 | 2 | 2 |
| src (output) | 4 | 2 | 4 |

## 4. Measuring Actual Per-Frame Encode Time

The encoder's self-reported latency describes pipeline depth, not how long it takes to encode a single frame.

To measure actual encode time, pair the `handle_frame` (input) and `finish_frame` (output) log entries for each frame number.

### Log lines to use

Input — when the encoder receives a raw frame:

```text theme={null}
LOG  v4l2videoenc  gstv4l2videoenc.c:804:gst_v4l2_video_enc_handle_frame:
  <v4l2h264enc0> Handling frame N
```

Output — when the encoder has finished encoding (use the frame PTS line):

```text theme={null}
LOG  videoencoder  gstvideoencoder.c:2644:gst_video_encoder_finish_frame:
  <v4l2h264enc0> frame PTS H:MM:SS.nnnnnnnnn, DTS ...
```

Note: Use the "frame PTS" log line as the finish timestamp — it confirms the encoded buffer is fully formed. It appears immediately after "finish frame fpn N" (\~7 µs later, negligible).

### Calculation

```text theme={null}
encode_time(N) = timestamp(finish_frame for frame N) - timestamp(handle_frame N)
```

Both timestamps are wall-clock values from the GStreamer log. `handle_frame` runs on the encoder's input thread; `finish_frame` runs on its output thread — so this correctly measures real elapsed time across both threads.

### Typical observations

| Phase | Frames | Encode Time |
| - | - | - |
| Startup | 0 to (`N_buffers` - 1) | Elevated — encoder pipeline filling |
| Steady state | `N_buffers` onwards | Consistent, near or below frame period |

Note: If steady-state encode time consistently exceeds the frame period, the encoder is slower than real-time and queues will grow — eventually causing latency build-up or frame drops in a live pipeline.

## 5. Verifying Frame Integrity — PTS Spacing

### What PTS spacing tells you

The Presentation Timestamp (PTS) on each buffer indicates when the frame should be presented. For a 30 fps stream, consecutive frames should have PTS values spaced 33.33 ms apart.

### Consistent PTS spacing means:

* The content is correctly tagged as a 30 fps stream

### It does NOT mean:

* Frames were captured or encoded without drops
* PTS values are assigned by the camera source element based on the declared framerate — surviving frames are always tagged sequentially regardless of whether intermediate frames were dropped

### How to check PTS spacing

Extract PTS values from the muxer's buffer enqueue log lines:

```bash theme={null}
grep "mp4mux0:video_0> Enqueuing buffer" your.log \
  | sed 's/.*pts \([0-9:]*\.[0-9]*\).*/\1/'
```

Then check spacing between consecutive values. All gaps should be within a small tolerance of the expected frame period (e.g., ±1 ms for 30 fps).

### What anomalies mean

| Gap Observed | Likely Cause |
| - | - |
| \< frame period | Startup artifact during encoder pipeline fill |
| = 2× frame period | One dropped frame |
| = N× frame period | N-1 dropped frames |
| Irregular in first few frames only | Normal encoder startup behaviour |

## 6. Verifying Frame Integrity — Frame Number Sequence

PTS spacing analysis can be misleading at startup and cannot distinguish between a dropped frame and a timestamp irregularity. The definitive check for frame drops is the frame number sequence in `handle_frame` logs.

### What to look for

The encoder logs a monotonically increasing frame number for every frame it receives:

```text theme={null}
LOG  v4l2videoenc  ...gst_v4l2_video_enc_handle_frame:<v4l2h264enc0> Handling frame 0
LOG  v4l2videoenc  ...gst_v4l2_video_enc_handle_frame:<v4l2h264enc0> Handling frame 1
LOG  v4l2videoenc  ...gst_v4l2_video_enc_handle_frame:<v4l2h264enc0> Handling frame 2
...
```

A gap in this sequence (e.g., `Handling frame 4` followed by `Handling frame 6`) means frame 5 was dropped before reaching the encoder.

### How to extract and verify

```bash theme={null}
grep -a "Handling frame" your.log \
  | sed 's/.*Handling frame //' \
  | sort -n > frame_nums.txt

# Check for gaps
awk 'NR==1{prev=$1; next}
     {if ($1 != prev+1) print "Gap: frame " prev " -> " $1; prev=$1}
     END{print "Total: " NR " frames, last: " prev}' frame_nums.txt
```

If no gap output is produced, all frames were delivered to the encoder in sequence with no drops.

### Combining both checks

| Check | What It Proves |
| - | - |
| PTS spacing consistent | Content is correctly timestamped at target framerate |
| Frame numbers contiguous | No frames dropped between camera and encoder |
| Both pass | Pipeline delivered all frames correctly end-to-end |

## Quick Reference — Log Search Patterns

| What You Want to Find | Search String |
| - | - |
| Encoder reported latency | `gst_video_encoder_set_latency` |
| Driver buffer requirements | `gst_v4l2_get_driver_min_buffers` |
| GStreamer buffer floor override | `increasing minimum buffers` |
| Frame input to encoder | `Handling frame` |
| Frame output from encoder | `frame PTS` (after `finish_frame`) |
| Buffers arriving at muxer | `Enqueuing buffer.*pts` |

## Summary

```text theme={null}
Encoder latency    =  (driver min CAPTURE buffers) × (1 / framerate)

Actual encode time =  finish_frame timestamp  −  handle_frame timestamp

No frame drops     ⟺  handle_frame sequence is contiguous (0, 1, 2, … N)

PTS spacing confirms framerate tagging — NOT absence of frame drops
```

## Conclusion

GStreamer debug logs contain all the information needed to fully characterise the latency and frame integrity of a hardware-encoded video pipeline — without modifying source code or adding external instrumentation. The key insight is knowing which log lines to look for and what they actually mean.

The following points summarise what has been covered in this app note:

* The encoder's self-reported latency (`gst_video_encoder_set_latency`) directly reflects the number of output CAPTURE buffers the driver requires multiplied by the frame duration. For `v4l2h264enc` at 30 fps with 4 output buffers, this is a fixed 133.33 ms. Because `min_latency` equals `max_latency`, this delay is constant and predictable.
* The 4 buffers responsible for this latency are output-side (CAPTURE queue) buffers, not input buffers. The input (sink) side requires only 2 buffers — a floor enforced by GStreamer to prevent deadlock, overriding the driver's reported value of 0.
* Actual per-frame encode time is measured by pairing `handle_frame` (input) and `finish_frame` / frame PTS (output) wall-clock timestamps for each frame number. During startup the encode time is elevated while the encoder fills its buffer pipeline; in steady state it is typically well below the frame period, confirming the encoder is not a throughput bottleneck.
* PTS spacing at the muxer tells you only that frames are correctly timestamped for the declared framerate. It does not prove that no frames were dropped. Consistent 33.33 ms PTS gaps are expected even when frames have been lost, because remaining frames are tagged sequentially by the source element.
* The definitive proof of zero frame drops is a contiguous `handle_frame` number sequence (0, 1, 2, … N) in the encoder input logs. Any gap in this sequence identifies exactly which frame was dropped and where in the pipeline the drop occurred.

Together these techniques give developers a repeatable, log-only methodology for validating encoder behaviour across different hardware platforms, resolutions, and framerates — making it straightforward to include latency and frame integrity evidence directly in engineering reports and application notes.
