> ## Documentation Index
> Fetch the complete documentation index at: https://dragonwingdocs-staging.qualcomm.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Execute AI model inference with qrb_ros_nn_inference

[qrb\_ros\_nn\_inference](https://github.com/qualcomm-qrb-ros/qrb_ros_nn_inference) is a ROS2 package that runs neural network models on the device, so your robotics application gets AI-based perception without leaving the ROS graph.

**It provides the following functionalities:**

* Model inference API which supports three model formats:
  * `.tflite`
  * `.so`
  * `.bin`
* Model inference acceleration based on Qualcomm platforms.

## Pipeline flow for  `qrb_ros_nn_inference`

The following figure shows how a preprocess node feeds an input tensor to `qrb_ros_nn_inference`, which publishes the output tensor to a postprocess node.

<img src="https://mintcdn.com/qualcomm-staging/blZpKs_b9On5nj-y/archives/2.1/SDKs/QIR-SDK-2.0/media/80-65220-2-qirp-sdk-qsg/nn-inference-pipeline.svg?fit=max&auto=format&n=blZpKs_b9On5nj-y&q=85&s=845b8a09867f1cb5cebfcef7793998f8" alt="Preprocess node publishing an input tensor to the nn inference node, which publishes an output tensor to a postprocess node." style={{ width: '100%', display: 'block', margin: '0 auto' }} width="982" height="243" data-path="archives/2.1/SDKs/QIR-SDK-2.0/media/80-65220-2-qirp-sdk-qsg/nn-inference-pipeline.svg" />

<p style={{ textAlign: 'center', fontWeight: 'bold' }}>
  Figure : qrb\_ros\_nn\_inference pipeline
</p>

## ROS topics used in the `qrb_ros_nn_inference `pipeline

The following table lists the ROS topics in the pipeline, their message types, and the node that publishes or subscribes to each one.

<table>
  <thead>
    <tr>
      <th style={{ width: '30%' }}>ROS topic</th>
      <th style={{ width: '20%' }}>Message type</th>
      <th style={{ width: '20%' }}>Published/Subscribed by</th>
      <th style={{ width: '20%' }}>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`/qrb_inference_input_tensor`</td>
      <td>`qrb_ros_tensor_list_msgs::msg::TensorList`</td>
      <td>Subscribed by `qrb_ros_nn_inference`</td>
      <td>Subscribes to the input tensor for model inference.</td>
    </tr>

    <tr>
      <td>`/qrb_inference_output_tensor`</td>
      <td>`qrb_ros_tensor_list_msgs::msg::TensorList`</td>
      <td>Published by `qrb_ros_nn_inference`</td>
      <td>Publishes the output tensor of model.</td>
    </tr>
  </tbody>
</table>

## ROS node parameters used in the `qrb_ros_nn_inference `pipeline

The following table lists the node parameters, their types, and their default values.

<table>
  <thead>
    <tr>
      <th style={{ width: '20%' }}>Node parameter</th>
      <th style={{ width: '25%' }}>Type</th>
      <th style={{ width: '25%' }}>Default value</th>
      <th style={{ width: '30%' }}>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`backend_option`</td>
      <td style={{textAlign: 'center'}}>`string`</td>
      <td style={{textAlign: 'center'}}>`""`</td>
      <td>Hardware acceleration option for model inference. For valid values, see [qrb\_ros\_nn\_inference documentation](https://github.com/qualcomm-qrb-ros/qrb_ros_nn_inference/blob/main/qrb_inference_manager/Documentation.md#2.1).</td>
    </tr>

    <tr>
      <td>`model_path`</td>
      <td style={{textAlign: 'center'}}>`string`</td>
      <td style={{textAlign: 'center'}}>`""`</td>
      <td>Indicates the path of the model file.</td>
    </tr>
  </tbody>
</table>

## Prerequisites

* You have **Set up the device** according to [Set up the environment for running sample applications](./set-up-env-for-sample-app).
* On a `qcom-robotics-image` device (Overlay Config #1), ensure that you have installed the Qualcomm AI Engine Direct SDK (`qairt-sdk`) and the `qrb-ros-nn-inference` package. For instructions, see [Enable and evaluate Qualcomm Linux overlays on QIR SDK](./enable-and-evaluate-qualcomm-linux-overlays-on-QIR-SDK).
  On a `qcom-robotics-proprietary-image` device (Overlay Config #2), the package is already installed.
* You have downloaded the model from [Qualcomm AI hub](https://aihub.qualcomm.com/).
* You have prepared the preprocess node and postprocess node according to the [qrb\_ros\_nn\_inference example](https://github.com/qualcomm-qrb-ros/qrb_ros_nn_inference/tree/main/test).

## Run out-of-the-box `qrb_ros_nn_inference`

<Accordion title="Try me">
  <Steps>
    <Step title="Set up QIR SDK and ROS2 environment on the device">
      Start one terminal, and run the following commands.

      Value range of `ROS_DOMAIN_ID`: \[0, 232]

      ```bash theme={null}
      ssh root@[ip-addr]
      ```

      ```bash title="SSH Session" theme={null}
      source /usr/share/qirp-setup.sh
      export ROS_DOMAIN_ID=xx
      ```
    </Step>

    <Step title="Launch the nodes">
      Launch three nodes: preprocess node, `qrb_ros_nn_inference` node and postprocess node.

      ```bash title="SSH Session" theme={null}
      ros2 launch model_inference.launch.py
      ```
    </Step>
  </Steps>
</Accordion>

## Build and run `qrb_ros_nn_inference`

<Accordion title="Try me">
  <Steps>
    <Step title="Build qrb_ros_nn_inference on the host">
      ```bash theme={null}
      cd <qirp_decompressed_workspace>
      source setup.sh
      cd qirp-samples/platform/qrb_ros_nn_inference
      colcon build --merge-install --cmake-args ${CMAKE_ARGS}
      ```
    </Step>

    <Step title="Push qrb_ros_nn_inference to the device">
      <Note>
        **Note**

        Ensure that the QIR SDK is installed on the device.
      </Note>

      ```bash theme={null}
      cd qirp-samples/platform/qrb_ros_nn_inference/install
      tar czvf qrb_ros_nn_inference.tar.gz include lib share
      scp qrb_ros_nn_inference.tar.gz root@[ip-addr]:/opt/
      ssh root@[ip-addr]
      ```

      ```bash title="SSH Session" theme={null}
      tar --no-overwrite-dir --no-same-owner -zxf /opt/qrb_ros_nn_inference.tar.gz -C /usr/ros/jazzy/
      ```
    </Step>

    <Step title="Set up QIR SDK and ROS2 environment on the device">
      Start one terminal, and run the following commands.

      Value range of `ROS_DOMAIN_ID`: \[0, 232]

      ```bash theme={null}
      ssh root@[ip-addr]
      ```

      ```bash title="SSH Session" theme={null}
      source /usr/share/qirp-setup.sh
      export ROS_DOMAIN_ID=xx
      ```
    </Step>

    <Step title="Launch the nodes">
      Launch three nodes: preprocess node, `qrb_ros_nn_inference` node and postprocess node.

      ```bash title="SSH Session" theme={null}
      ros2 launch model_inference.launch.py
      ```
    </Step>
  </Steps>
</Accordion>

## Limitations

For the latest limitations, see [qrb\_ros\_nn\_inference](https://github.com/qualcomm-qrb-ros/qrb_ros_nn_inference) on GitHub.