> ## Documentation Index
> Fetch the complete documentation index at: https://dragonwingdocs-staging.qualcomm.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Run a LiteRT model on the NPU

> Run object detection application on the NPU of a Qualcomm Dragonwing device using QNN delegate with Python.

Use the [LiteRT](https://ai.google.dev/edge/litert) runtime to run quantized LiteRT models on the NPU of Qualcomm® Dragonwing™ devices.

## Prerequisites

Before running the application, complete the following steps to connect to the target device, download the required artifacts, and set up the Python environment.

1. Enable Wi-Fi and SSH on the device.

   The device requires an internet connection to download the artifacts needed to run sample applications. If SSH and Wi-Fi are already configured, skip this step.

   Follow [Setup an SSH connection](https://dragonwingdocs.qualcomm.com/Technologies/Wi-Fi/setup-wi-fi#set-up-uart/ssh) to enable Wi-Fi and SSH on the device.

2. On the target device, download the `download_artifacts.sh` script, set executable permissions,
   and run it to download the model and label files to the device.

   ```shell theme={null}
   cd /tmp/
   ```

   ```shell theme={null}
   curl -L -O https://raw.githubusercontent.com/qualcomm/sample-apps-for-qualcomm-linux/refs/heads/main/qualcomm-linux/scripts/download_artifacts_2.x.sh && chmod +x download_artifacts_2.x.sh && ./download_artifacts_2.x.sh
   ```

3. Install the LiteRT runtime and other dependencies by setting up the Python environment on the target device.

   ```shell theme={null}
   python3 -m venv --system-site-packages python-venv
   ```

   Activate the python environment:

   ```shell theme={null}
   source python-venv/bin/activate
   ```

   Install the LiteRT runtime, Pillow, and OpenCV packages:

   ```shell theme={null}
   pip3 install ai-edge-litert==2.1.5 Pillow opencv-python==4.10.0.84
   ```

## Run an object detection application

The following Python application performs object detection in real time on a video file using a quantized YOLOX LiteRT model. It displays annotated frames to a file or Wayland display, and is optimized for edge AI scenarios using hardware acceleration through the QNN LiteRT delegate.

1. Create and go to the `${HOME}/Downloads/qimsdk_samples/apps/` directory.

   ```shell theme={null}
   mkdir -p ${HOME}/Downloads/qimsdk_samples/apps/ && cd ${HOME}/Downloads/qimsdk_samples/apps/
   ```

2. Download the `object_detection_2.x.py` file.

   ```shell theme={null}
   curl -L https://raw.githubusercontent.com/qualcomm/sample-apps-for-qualcomm-linux/refs/heads/main/qualcomm-linux/applications/LiteRT/object_detection_2.x.py -o object_detection.py
   ```

   For code explanation, see [Object detection with OpenCV and LiteRT](#create-object_detection-py) below.

3. Run the application:

   <Tabs>
     <Tab title="Output to file">
       ```shell theme={null}
       python3 object_detection.py --output file
       ```

       Download the output video to the host machine to review the results:

       ```shell theme={null}
       scp root@<IP_ADDRESS_OF_TARGET_DEVICE>:/tmp/output_object_detection.mp4 .
       ```

       When prompted for a password, enter `oelinux123`.
     </Tab>

     <Tab title="Output to display">
       In the terminal of the target device, run the object detection application:

       ```shell theme={null}
       python3 object_detection.py --output wayland
       ```
     </Tab>
   </Tabs>

<h2 id="create-object_detection-py">
  Object detection with OpenCV and LiteRT
</h2>

To create an application similar to the object detection application described in the previous section, create an `object_detection.py` file as follows:

1. In the `${HOME}/Downloads/qimdk_samples/apps/` folder, create an `object_detection.py` file.

2. Add the following code to your `object_detection.py` file.

   <Note>
     The postprocessing in the following code is compatible with object detection models from AI Hub.
     For custom models, update the postprocessing logic to match the model's output format and requirements.
   </Note>

   a. Import the required packages:

   ```python theme={null}
   #!/usr/bin/env python3
   import os
   import cv2
   import numpy as np
   import argparse
   import ai_edge_litert.interpreter as tflite
   import gi
   gi.require_version('Gst', '1.0')
   from gi.repository import Gst
   ```

   b. Handle output arguments:

   ```python theme={null}
   parser = argparse.ArgumentParser(description="Run object detection and output to file or Wayland.")
   parser.add_argument("--output", choices=["file", "wayland"], default="file",
                     help="Choose output mode: 'file' (default) or 'wayland'")
   args = parser.parse_args()
   ```

   c. Initialize and configure model parameters:

   ```python theme={null}
   ARTIFACT_DIR = os.path.join(
      os.path.expanduser("~"), "Downloads", "qimsdk_samples")
   MODEL_PATH = f"${ARTIFACT_DIR}/models/yolox_quantized.tflite"  # YOLOX quantized model
   LABEL_PATH = f"${ARTIFACT_DIR}/labels/coco_labels.txt"
   VIDEO_IN = f"${ARTIFACT_DIR}/media/video.mp4"
   VIDEO_OUT = "/tmp/output_object_detection.mp4"
   DELEGATE_PATH = "libQnnTFLiteDelegate.so"

   FRAME_W, FRAME_H = 1600, 900
   FPS_OUT = 30
   CONF_THRES = 0.25
   NMS_IOU_THRES = 0.50
   BOX_SCALE = 3.2108588218688965
   BOX_ZP = 31.0
   SCORE_SCALE = 0.0038042240776121616
   ```

   d. Load the model and set up the LiteRT delegate:

   ```python theme={null}
   delegate_options = {'backend_type': 'htp'}
   delegate = tflite.load_delegate(DELEGATE_PATH, delegate_options)
   interpreter = tflite.Interpreter(model_path=MODEL_PATH, experimental_delegates=[delegate])
   interpreter.allocate_tensors()

   in_det = interpreter.get_input_details()
   out_det = interpreter.get_output_details()
   in_h, in_w = in_det[0]["shape"][1:3]

   labels = [l.strip() for l in open(LABEL_PATH)]
   ```

   e. Set up video capture and preprocessing:

   ```python theme={null}
   try:
      cap = cv2.VideoCapture(VIDEO_IN)
      sx, sy = FRAME_W / in_w, FRAME_H / in_h
      frame_rs = np.empty((FRAME_H, FRAME_W, 3), np.uint8)
      input_tensor = np.empty((1, in_h, in_w, 3), np.uint8)
   ```

   f. Create a GStreamer pipeline to stream frames to the Wayland display:

   ```python theme={null}
      if args.output == "file":
         fourcc = cv2.VideoWriter_fourcc(*"mp4v")
         out_writer = cv2.VideoWriter(VIDEO_OUT, fourcc, FPS_OUT, (FRAME_W, FRAME_H))
      else:
         Gst.init(None)
         pipeline = Gst.parse_launch(
            'appsrc name=src is-live=true block=true format=time caps=video/x-raw,format=BGR,width=1600,height=900,framerate=30/1 ! videoconvert ! waylandsink'
         )
         appsrc = pipeline.get_by_name('src')
         pipeline.set_state(Gst.State.PLAYING)

      frame_cnt = 0
   ```

   g. Initialize the main loop to read the video, run inference on each frame, and draw bounding boxes on the output:

   ```python theme={null}
      # -------------------- Main Loop --------------------
      while True:
         ok, frame = cap.read()
         if not ok:
            break
         frame_cnt += 1

         cv2.resize(frame, (FRAME_W, FRAME_H), dst=frame_rs)
         cv2.resize(frame_rs, (in_w, in_h), dst=input_tensor[0])

         interpreter.set_tensor(in_det[0]['index'], input_tensor)
         interpreter.invoke()

         boxes_q = interpreter.get_tensor(out_det[0]['index'])[0]
         scores_q = interpreter.get_tensor(out_det[1]['index'])[0]
         classes_q = interpreter.get_tensor(out_det[2]['index'])[0]

         boxes = BOX_SCALE * (boxes_q.astype(np.float32) - BOX_ZP)
         scores = SCORE_SCALE * scores_q.astype(np.float32)
         classes = classes_q.astype(np.int32)

         mask = scores >= CONF_THRES
         if np.any(mask):
            boxes_f = boxes[mask]
            scores_f = scores[mask]
            classes_f = classes[mask]

            x1, y1, x2, y2 = boxes_f.T
            boxes_cv2 = np.column_stack((x1, y1, x2 - x1, y2 - y1))

            idx_cv2 = cv2.dnn.NMSBoxes(
                  bboxes=boxes_cv2.tolist(),
                  scores=scores_f.tolist(),
                  score_threshold=CONF_THRES,
                  nms_threshold=NMS_IOU_THRES
            )

            if len(idx_cv2):
                  idx = idx_cv2.flatten()
                  sel_boxes = boxes_f[idx]
                  sel_scores = scores_f[idx]
                  sel_classes = classes_f[idx]

                  sel_boxes[:, [0, 2]] *= sx
                  sel_boxes[:, [1, 3]] *= sy
                  sel_boxes = sel_boxes.astype(np.int32)

                  sel_boxes[:, [0, 2]] = np.clip(sel_boxes[:, [0, 2]], 0, FRAME_W - 1)
                  sel_boxes[:, [1, 3]] = np.clip(sel_boxes[:, [1, 3]], 0, FRAME_H - 1)

                  for (x1i, y1i, x2i, y2i), sc, cl in zip(sel_boxes, sel_scores, sel_classes):
                     cv2.rectangle(frame_rs, (x1i, y1i), (x2i, y2i), (0, 255, 0), 2)
                     lab = labels[cl] if cl < len(labels) else str(cl)
                     cv2.putText(frame_rs, f"{lab} {sc:.2f}", (x1i, max(10, y1i - 5)),
                                 cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2)

         if args.output == "file":
            out_writer.write(frame_rs)
         else:
            data = frame_rs.tobytes()
            buf = Gst.Buffer.new_allocate(None, len(data), None)
            buf.fill(0, data)
            buf.duration = Gst.util_uint64_scale_int(1, Gst.SECOND, FPS_OUT)
            timestamp = cap.get(cv2.CAP_PROP_POS_MSEC) * Gst.MSECOND
            buf.pts = buf.dts = int(timestamp)
            appsrc.emit('push-buffer', buf)
   ```

   h. Release the pipeline and notify the user on completion:

   ```python theme={null}
   except KeyboardInterrupt:
      print(f"\nCtrl+C received. Cleaning up after {frame_cnt} frames...")

   finally:
      # -------------------- Finish --------------------
      cap.release()
      if args.output == "file":
         out_writer.release()
         print(f"Done – processed video saved to {VIDEO_OUT}")
      else:
         appsrc.emit('end-of-stream')
         pipeline.set_state(Gst.State.NULL)
         print("Done – video streamed to Wayland sink")
   ```