Run a LiteRT model on the NPU - Qualcomm Dragonwing Documentation

Use the LiteRT runtime to run quantized LiteRT models on the NPU of Qualcomm® Dragonwing™ devices.

Prerequisites

Before running the application, complete the following steps to connect to the target device, download the required artifacts, and set up the Python environment.

Enable Wi-Fi and SSH on the device. The device requires an internet connection to download the artifacts needed to run sample applications. If SSH and Wi-Fi are already configured, skip this step. Follow Setup an SSH connection to enable Wi-Fi and SSH on the device.

On the target device, download the download_artifacts.sh script, set executable permissions, and run it to download the model and label files to the device.

cd /tmp/

curl -L -O https://raw.githubusercontent.com/qualcomm/sample-apps-for-qualcomm-linux/refs/heads/main/qualcomm-linux/scripts/download_artifacts_2.x.sh && chmod +x download_artifacts_2.x.sh && ./download_artifacts_2.x.sh

Install the LiteRT runtime and other dependencies by setting up the Python environment on the target device.
python3 -m venv --system-site-packages python-venv
Activate the python environment:
source python-venv/bin/activate
Install the LiteRT runtime, Pillow, and OpenCV packages:
pip3 install ai-edge-litert==2.1.5 Pillow opencv-python==4.10.0.84

Run an object detection application

The following Python application performs object detection in real time on a video file using a quantized YOLOX LiteRT model. It displays annotated frames to a file or Wayland display, and is optimized for edge AI scenarios using hardware acceleration through the QNN LiteRT delegate.

Create and go to the ${HOME}/Downloads/qimsdk_samples/apps/ directory.

mkdir -p ${HOME}/Downloads/qimsdk_samples/apps/ && cd ${HOME}/Downloads/qimsdk_samples/apps/

Download the object_detection.py file.

curl -L https://raw.githubusercontent.com/qualcomm/sample-apps-for-qualcomm-linux/refs/heads/main/qualcomm-linux/applications/LiteRT/object_detection.py -o object_detection.py

For code explanation, see Object detection with OpenCV and LiteRT below.

Run the application:
- Output to file
- Output to display
python3 object_detection.py --output file
Download the output video to the host machine to review the results:
scp root@<IP_ADDRESS_OF_TARGET_DEVICE>:/tmp/output_object_detection.mp4 .
When prompted for a password, enter oelinux123.
In the terminal of the target device, run the object detection application:
python3 object_detection.py --output wayland

Object detection with OpenCV and LiteRT

To create an application similar to the object detection application described in the previous section, create an object_detection.py file as follows:

In the ${HOME}/Downloads/qimdk_samples/apps/ folder, create an object_detection.py file.

Add the following code to your object_detection.py file.

The postprocessing in the following code is compatible with object detection models from AI Hub. For custom models, update the postprocessing logic to match the model’s output format and requirements.

a. Import the required packages:

#!/usr/bin/env python3
import os
import cv2
import numpy as np
import argparse
import ai_edge_litert.interpreter as tflite
import gi
gi.require_version('Gst', '1.0')
from gi.repository import Gst

b. Handle output arguments:

parser = argparse.ArgumentParser(description="Run object detection and output to file or Wayland.")
parser.add_argument("--output", choices=["file", "wayland"], default="file",
                  help="Choose output mode: 'file' (default) or 'wayland'")
args = parser.parse_args()

c. Initialize and configure model parameters:

ARTIFACT_DIR = os.path.join(
   os.path.expanduser("~"), "Downloads", "qimsdk_samples")
MODEL_PATH = f"${ARTIFACT_DIR}/models/yolox_quantized.tflite"  # YOLOX quantized model
LABEL_PATH = f"${ARTIFACT_DIR}/labels/coco_labels.txt"
VIDEO_IN = f"${ARTIFACT_DIR}/media/video.mp4"
VIDEO_OUT = "/tmp/output_object_detection.mp4"
DELEGATE_PATH = "libQnnTFLiteDelegate.so"

FRAME_W, FRAME_H = 1600, 900
FPS_OUT = 30
CONF_THRES = 0.25
NMS_IOU_THRES = 0.50
BOX_SCALE = 3.2108588218688965
BOX_ZP = 31.0
SCORE_SCALE = 0.0038042240776121616

d. Load the model and set up the LiteRT delegate:

delegate_options = {'backend_type': 'htp'}
delegate = tflite.load_delegate(DELEGATE_PATH, delegate_options)
interpreter = tflite.Interpreter(model_path=MODEL_PATH, experimental_delegates=[delegate])
interpreter.allocate_tensors()

in_det = interpreter.get_input_details()
out_det = interpreter.get_output_details()
in_h, in_w = in_det[0]["shape"][1:3]

labels = [l.strip() for l in open(LABEL_PATH)]

e. Set up video capture and preprocessing:

try:
   cap = cv2.VideoCapture(VIDEO_IN)
   sx, sy = FRAME_W / in_w, FRAME_H / in_h
   frame_rs = np.empty((FRAME_H, FRAME_W, 3), np.uint8)
   input_tensor = np.empty((1, in_h, in_w, 3), np.uint8)

f. Create a GStreamer pipeline to stream frames to the Wayland display:

   if args.output == "file":
      fourcc = cv2.VideoWriter_fourcc(*"mp4v")
      out_writer = cv2.VideoWriter(VIDEO_OUT, fourcc, FPS_OUT, (FRAME_W, FRAME_H))
   else:
      Gst.init(None)
      pipeline = Gst.parse_launch(
         'appsrc name=src is-live=true block=true format=time caps=video/x-raw,format=BGR,width=1600,height=900,framerate=30/1 ! videoconvert ! waylandsink'
      )
      appsrc = pipeline.get_by_name('src')
      pipeline.set_state(Gst.State.PLAYING)

   frame_cnt = 0

g. Initialize the main loop to read the video, run inference on each frame, and draw bounding boxes on the output:

   # -------------------- Main Loop --------------------
   while True:
      ok, frame = cap.read()
      if not ok:
         break
      frame_cnt += 1

      cv2.resize(frame, (FRAME_W, FRAME_H), dst=frame_rs)
      cv2.resize(frame_rs, (in_w, in_h), dst=input_tensor[0])

      interpreter.set_tensor(in_det[0]['index'], input_tensor)
      interpreter.invoke()

      boxes_q = interpreter.get_tensor(out_det[0]['index'])[0]
      scores_q = interpreter.get_tensor(out_det[1]['index'])[0]
      classes_q = interpreter.get_tensor(out_det[2]['index'])[0]

      boxes = BOX_SCALE * (boxes_q.astype(np.float32) - BOX_ZP)
      scores = SCORE_SCALE * scores_q.astype(np.float32)
      classes = classes_q.astype(np.int32)

      mask = scores >= CONF_THRES
      if np.any(mask):
         boxes_f = boxes[mask]
         scores_f = scores[mask]
         classes_f = classes[mask]

         x1, y1, x2, y2 = boxes_f.T
         boxes_cv2 = np.column_stack((x1, y1, x2 - x1, y2 - y1))

         idx_cv2 = cv2.dnn.NMSBoxes(
               bboxes=boxes_cv2.tolist(),
               scores=scores_f.tolist(),
               score_threshold=CONF_THRES,
               nms_threshold=NMS_IOU_THRES
         )

         if len(idx_cv2):
               idx = idx_cv2.flatten()
               sel_boxes = boxes_f[idx]
               sel_scores = scores_f[idx]
               sel_classes = classes_f[idx]

               sel_boxes[:, [0, 2]] *= sx
               sel_boxes[:, [1, 3]] *= sy
               sel_boxes = sel_boxes.astype(np.int32)

               sel_boxes[:, [0, 2]] = np.clip(sel_boxes[:, [0, 2]], 0, FRAME_W - 1)
               sel_boxes[:, [1, 3]] = np.clip(sel_boxes[:, [1, 3]], 0, FRAME_H - 1)

               for (x1i, y1i, x2i, y2i), sc, cl in zip(sel_boxes, sel_scores, sel_classes):
                  cv2.rectangle(frame_rs, (x1i, y1i), (x2i, y2i), (0, 255, 0), 2)
                  lab = labels[cl] if cl < len(labels) else str(cl)
                  cv2.putText(frame_rs, f"{lab} {sc:.2f}", (x1i, max(10, y1i - 5)),
                              cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2)

      if args.output == "file":
         out_writer.write(frame_rs)
      else:
         data = frame_rs.tobytes()
         buf = Gst.Buffer.new_allocate(None, len(data), None)
         buf.fill(0, data)
         buf.duration = Gst.util_uint64_scale_int(1, Gst.SECOND, FPS_OUT)
         timestamp = cap.get(cv2.CAP_PROP_POS_MSEC) * Gst.MSECOND
         buf.pts = buf.dts = int(timestamp)
         appsrc.emit('push-buffer', buf)

h. Release the pipeline and notify the user on completion:

except KeyboardInterrupt:
   print(f"\nCtrl+C received. Cleaning up after {frame_cnt} frames...")

finally:
   # -------------------- Finish --------------------
   cap.release()
   if args.output == "file":
      out_writer.release()
      print(f"Done – processed video saved to {VIDEO_OUT}")
   else:
      appsrc.emit('end-of-stream')
      pipeline.set_state(Gst.State.NULL)
      print("Done – video streamed to Wayland sink")

​Prerequisites

​Run an object detection application

​Object detection with OpenCV and LiteRT

Prerequisites

Run an object detection application

Object detection with OpenCV and LiteRT