Detect objects with sample_object_detection - Qualcomm Dragonwing Documentation

The sample_object_detection sample application detects and classifies objects in images using ROS 2 nodes. It uses Qualcomm® AI Engine Direct SDK (QNN) for model inference. The sample uses the Ultralytics YOLOv8 model to predict bounding boxes and classes of objects in an image.

NoteFor more information, see sample_object_detection on GitHub.

Figure: Sample effects of sample_object_detection

Pipeline flow for `sample_object_detection`

Figure: Pipeline flow for sample_object_detection

ROS nodes used in `sample_object_detection`

ROS node	Description
qrb_ros_camera	Qualcomm ROS 2 package that captures images with parameters and publishes them to ROS topics.
yolo_preprocess	Subscribes to image data, reshapes and resizes it, and republishes it to a downstream topic.
qrb_ros_nn_inference	Loads a trained AI model, receives preprocessed images, performs inference, and publishes results.
yolo_postprocess	Matches inference output with YOLO label files.
yolo_overlay	Subscribes to the YOLO postprocess and image data, and shows the object detection results as a ROS topic.

ROS topics used in `sample_object_detection`

ROS Topic	Type	Published By
`/cam0_stream1`	`sensor_msgs.msg.Image`	`camera_node`
`/qrb_inference_input_tensor`	`qrb_ros_tensor_list_msgs.msg.TensorList`	`yolo_preprocess_node`
`/resized_image`	`sensor_msgs.msg.Image`	`yolo_preprocess_node`
`/yolo_detect_tensor_output`	`qrb_ros_tensor_list_msgs.msg.TensorList`	`nn_inference_node`
`/yolo_detect_result`	`vision_msgs.msg.Detection2DArray`	`yolo_detection_postprocess_node`
`/yolo_detect_overlay`	`sensor_msgs.msg.Image`	`yolo_detection_overlay_node`

Noteyolo_detection_overlay_node synchronizes two inputs, /yolo_detect_result and /resized_image, using an exact-time policy. Both must be publishing for /yolo_detect_overlay to be produced.

Prerequisites

You have completed the settings in Set up the environment for running sample applications.
You have generated the YOLOv8 model using QAI-Hub on the host computer with the following steps.

Generate YOLOv8 model

Set up QAI-Hub on the host computer.

Install QAI-Hub.

## It is recommended to use python venv to avoid impacting the host environment.
python3 -m venv venv_qaihub
source venv_qaihub/bin/activate
## Install QAI-Hub related python packages.
pip3 install qai-hub==0.41.0
## Configure QAI-Hub token, replacing xxx with your own token got from QAI-Hub page.
qai-hub configure --api_token xxx
pip3 install "qai-hub-models[yolov8-det]==0.42.0"

Export the model on the host computer.

# For IQ-9075, after executing the following commands, the model file and coco8.yaml are generated in the host computer.
python -m qai_hub_models.models.yolov8_det.export --target-runtime "qnn_context_binary" --chipset "qualcomm-qcs9075-proxy" --skip-profiling --skip-inferencing --compile-options='--qairt_version=2.41'

SSH Session

# Push model files to the device's default path according to the preceding export log.
# On the device, create the path if it doesn't exist.
mkdir /opt/model

# On the host computer, scp all model files to the device.
scp <Host_Export_model_Path>/yolov8_det_float/yolov8_det.bin root@<device_ip>:/opt/model
scp <Host_Export_model_Path>/coco8.yaml root@<device_ip>:/opt/

For more information, see qrb_ros_tensor_process on GitHub.

Run out-of-the-box `sample_object_detection`

Try me

Run the sample on the development kit

Ensure that you have connected a camera to the development kit and set it up according to Set up the camera. The launch file starts the qrb_ros_camera node with camera ID 0 and a 640 × 480 stream, so you don’t need to run qrb_ros_camera separately.

Set up the runtime environment.

SSH Session

# Set up the runtime environment
source /usr/share/qirp-setup.sh -m
# Set the ROS_DOMAIN_ID
export ROS_DOMAIN_ID=123

Launch the sample with qrb_ros_camera.

SSH Session

ros2 launch sample_object_detection launch_with_qrb_ros_camera.py model:=/opt/model/yolov8_det.bin

Check the results on the host computer

In a terminal of the host computer, check the ROS topics with the name /yolo_detect_overlay in rqt.

Build and run `sample_object_detection`

The following steps build the sample_object_detection package using Dragonwing IQ-9075 Evaluation Kit as an example.

Try me

Build and package on the host computer

Build the sample application project.

cd <decompressed_workspace>/images/iq-9075-evk/qirpsdk_artifacts/iq-9075-evk/
tar -zxvf qirp-sdk_<qirp_version>.tar.gz
cd qirp-sdk
source setup.sh
# build sample
cd <qirp_decompressed_path>/qirp-samples/ai_vision/sample_object_detection
colcon build

Package and push the sample application to the device.

# package and push build result of sample
cd <qirp_decompressed_path>/qirp-samples/ai_vision/sample_object_detection/install/sample_object_detection
tar -czvf sample_object_detection.tar.gz lib share
scp sample_object_detection.tar.gz root@[ip-addr]:/opt/

Install and run on the development kit

Install the sample application.

SSH Session

# Install sample package
tar --no-overwrite-dir --no-same-owner -zxf /opt/sample_object_detection.tar.gz -C /usr/ros/jazzy/

Run the sample application with the steps in Run out-of-the-box sample_object_detection.

Robotics SDK(Linux)

​Pipeline flow for sample_object_detection

​ROS nodes used in sample_object_detection

​ROS topics used in sample_object_detection

​Prerequisites

​Generate YOLOv8 model

​Run out-of-the-box sample_object_detection

​Build and run sample_object_detection

Pipeline flow for `sample_object_detection`

ROS nodes used in `sample_object_detection`

ROS topics used in `sample_object_detection`

Prerequisites

Generate YOLOv8 model

Run out-of-the-box `sample_object_detection`

Build and run `sample_object_detection`