> ## Documentation Index
> Fetch the complete documentation index at: https://dragonwingdocs-staging.qualcomm.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Understanding Real Time Linux Kernel

## **Real-Time (RT) kernel**

A real-time system is a deterministic system, where response to an event is expected in a set time.

A system is classified as compatible with RT if:

* It's devoid of unbounded latency.
* The maximum response time is calculated with precision.
* It meets the set criteria for scheduling of tasks (latency and deadline).

Linux can be configured as a real-time operating system (RTOS) in which real-time tasks have well-defined periodic execution cycles (cycle time) and meet execution criteria within specified limits (jitter).

To install the patches, see [Versions of PREEMPT\_RT patches](https://wiki.linuxfoundation.org/realtime/start).

> [!NOTE]
>
> The real-time support is for kernelspace process and not for userspace.

> [!NOTE]
>
> This section isn't applicable for QCS5430.

![Figure : Build sequence](https://mintcdn.com/qualcomm-staging/3pCxfvUGXlOT6fV1/Suites/Industrial-Edge-Suite/media/k2l-performance/RT-kernel-build-sequence.svg?fit=max\&auto=format\&n=3pCxfvUGXlOT6fV1\&q=85\&s=b45be73d67206c7462b94dea92986fba)

Figure : Build sequence

### **Set up the workspace**

The Qualcomm Linux kernel supports the LTS RT kernel (6.18.x), which is maintained through the Yocto recipe in the `meta-qcom` layer in the `recipes-kernel/linux/linux-qcom-rt_6.18.bb` file.

For more information about cloning the workspace and getting all Qualcomm Linux meta layers to use Qualcomm RT Linux, see [Sync](https://docs.qualcomm.com/doc/80-80023-254/topic/build_from_source_github_intro.html#sync).

### **Enable the RT kernel**

The Qualcomm Linux `meta-qcom` layer supports `linux-qcom-rt_6.18.bb` recipe that fetches and builds the Qualcomm Linux kernel for the supported machines by default.

The `meta-qcom` layer applies the changes on top of the existing layer. During the kernel build, `meta-qcom` layer enables `PREEMPT_RT` using `rt.config` based on the kernel version, and allows real-time configurations.

> [!NOTE]
>
> * Use `linux-qcom-rt_6.18.bb` for QLI.2.0.
> * Use `linux-qcom-next-rt_git.bb` for `qcom-next`.

For more information about supported machines, see [Build critical release tags](https://docs.qualcomm.com/doc/80-80023-300/topic/build_critical_release_tags.html).

### **Customize the RT kernel**

* If you are carrying any changes for the RT kernel, maintain them in the recipe as shown below:
  * Maintain the patch file under `recipes-kernel/linux/linux-qcom-6.18/<your_patch_file>.patch`
  * Append the patch file to `SRC_URI` in the `recipes-kernel/linux/linux-qcom-rt_6.18.bb` file.

```text theme={null}
SRC_URI += " \
    file://<your_patch_file>.patch \
    "
```

* To apply any external configurations on the RT kernel:
  * Maintain the configuration file in the `recipes-kernel/linux/linux-qcom-6.18/configs/<your_config>.cfg` recipe.
  * Append the configuration file to `SRC_URI` in the `recipes-kernel/linux/linux-qcom-rt_6.18.bb` file.

```text theme={null}
SRC_URI += " \
    file://configs/qcom_rt.cfg \
"
```

* To modify the kernel command-line, add the command-line parameters into the `meta-qcom/ci/base.yml` file to `KERNEL_CMDLINE_EXTRA` variable.

```text theme={null}
KERNEL_CMDLINE_EXTRA:append = " qcom_scm.download_mode=1 <new_parameter>"
```

### **Configure kernel settings for the RT kernel**

Optional and mandatory kernel configurations are used in the RT kernel.

To enable full preemption in the RT kernel, use `CONFIG_PREEMPT_RT`.

The `CONFIG_PREEMPT_RT` flag is enabled by default as part of the `rt.config` used in `linux-qcom-rt_6.18.bb` recipe.

The following example shows the kernel configuration:

```text theme={null}
zcat proc/config.gz | grep CONFIG_PREEMPT
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
CONFIG_PREEMPT_RT=y
```

Set the following kernel configuration options when affining the RT task:

* `CONFIG_NO_HZ_COMMON` - When enabled, it configures the kernel infrastructure for tickless operation.
* `CONFIG_NO_HZ_FULL` - When enabled, it configures the kernel to avoid sending scheduling-clock interrupts to CPUs with a single runnable task.
* `CONFIG_CPUSETS` - Use the `CONFIG_CPUSETS` configuration option to enable `cpuset`, where the CPU is grouped to form a set.

![Figure: RT kernel verification](https://mintcdn.com/qualcomm-staging/3pCxfvUGXlOT6fV1/Suites/Industrial-Edge-Suite/media/k2l-performance/compile-RT-kernel.svg?fit=max\&auto=format\&n=3pCxfvUGXlOT6fV1\&q=85\&s=a65ec124e6e67a99e1f54d1076eac7c1)

Figure: RT kernel verification

### **Build the RT kernel**

To build the RT kernel, run the following commands:

1. To ensure that you are in the KAS shell, run the following command:

```text theme={null}
kas shell meta-qcom/ci/qcs6490-rb3gen2-core-kit.yml:meta-qcom/ci/linux-qcom-rt-6.18.yml:meta-qcom/ci/qcom-distro-kvm.yml:meta-qcom/ci/lock.yml
```

> [!NOTE]
>
> The `qcom-distro-kvm.yml` configuration is supported with KVM only for QLI 2.0. The RT kernel (`PREEMPT_RT`) is hypervisor-agnostic. KVM is selected as the default hypervisor for QLI 2.0 due to the following reasons:
>
> * **Upstream community support** — KVM and its drivers are part of the mainline Linux kernel, enabling broader ecosystem contributions and long-term support.
> * **Standard VirtIO support** — KVM supports all upstream VirtIO devices and is compatible with standard VMMs such as QEMU.
> * **Open development** — Active upstream community with broader developer familiarity.

The kas build command automatically includes `xbl_config_kvm.elf` in the generated `qcomflash` package. No manual UEFI BDS configuration is required.

> [!NOTE]
>
> Copy the kas lock file from meta-qcom-releases to meta-qcom, see [Build a BSP image](https://docs.qualcomm.com/doc/80-80023-254/topic/build_from_source_github_intro.html#build-a-bsp-image).

2. To compile the BitBake Qualcomm Linux multimedia image, run the following command:

```text theme={null}
bitbake qcom-multimedia-image
```

### **Tune the RT kernel**

Tune the RT kernel to achieve a deterministic latency for RT tasks in the device.

Set the CPU cores that run RT tasks to run at maximum operating frequency while preventing thermal mitigation of CPU frequency. For example, in an idle sleep scenario, RT tasks face scheduling latency due to CPU wake time delay.

> [!NOTE]
>
> These optimizations are automatically applied by the [URM](./features-impacting-performance#understand-the-userspace-resource-manager). when RT tests(cyclictest) is launched.
>
> > Manual configuration is **not required** under normal operation.
> > The following steps are provided for reference, debugging, or custom setups.
>
> To configure the system before running the tests, do the following for QCS6490, Qualcomm Dragonwing<sup>™</sup> IQ-9075, and Qualcomm Dragonwing<sup>™</sup> IQ-615 Development Kit:
>
> 1. Disable the timer migration:
>    ```text theme={null}
>    echo 0 > /proc/sys/kernel/timer_migration
>    ```
> 2. Affine all kernel work queues in `/sys/devices/virtual/workqueue/*` to the housekeeping CPUs:
>    ```text theme={null}
>    for wq in /sys/devices/virtual/workqueue/*; do
>       [ -w "$wq/cpumask" ] && echo 7F > "$wq/cpumask"
>    done
>    # 7F = CPUs 0-6 (binary 0111 1111)
>    ```
> 3. Set CPU frequency governor to performance:
>    ```text theme={null}
>    for policy in /sys/devices/system/cpu/cpufreq/policy*; do
>       [ -w "$policy/scaling_governor" ] && echo performance > "$policy/scaling_governor"
>    done
>    ```
> 4. Disable RT accounting/throttling:
>    ```text theme={null}
>    echo -1 > /proc/sys/kernel/sched_rt_runtime_us
>    ```
> 5. Set IRQ affinity to housekeeping CPUs:
>    ```text theme={null}
>    ALLOW_CPUS="0,1,2,3,4,5,6"
>    cpu_list_to_mask() {
>       MASK=0
>       for cpu in $(echo $1 | tr ',' ' '); do
>          MASK=$((MASK | (1 << cpu)))
>       done
>       printf "%x\n" "$MASK"
>    }
>    MASK=$(cpu_list_to_mask "$ALLOW_CPUS")
>    echo "Setting IRQ affinity to CPUs: $ALLOW_CPUS (mask=0x$MASK)"
>    for irq in /proc/irq/[0-9]*; do
>       smp_file="$irq/smp_affinity"
>       [ -w "$smp_file" ] && echo "$MASK" > "$smp_file" 2>/dev/null
>    done
>    ```
>
> To configure the system before running the tests, do the following for the Dragonwing<sup>™</sup> IQ-8275 Development Kit:
>
> 1. Disable the timer migration:
>    ```text theme={null}
>    echo 0 > /proc/sys/kernel/timer_migration
>    ```
> 2. Affine all kernel work queues in `/sys/devices/virtual/workqueue/*` to the housekeeping CPUs:
>    ```text theme={null}
>    for wq in /sys/devices/virtual/workqueue/*; do
>       [ -w "$wq/cpumask" ] && echo F7 > "$wq/cpumask"
>    done
>    # F7 = CPUs 0–2,4-7 (binary 1111 0111)
>    ```
> 3. Set CPU frequency governor to performance:
>    ```text theme={null}
>    for policy in /sys/devices/system/cpu/cpufreq/policy*; do
>       [ -w "$policy/scaling_governor" ] && echo performance > "$policy/scaling_governor"
>    done
>    ```
> 4. Disable RT accounting/throttling:
>    ```text theme={null}
>    echo -1 > /proc/sys/kernel/sched_rt_runtime_us
>    ```
> 5. Set IRQ affinity to housekeeping CPUs:
>    ```text theme={null}
>    ALLOW_CPUS="0,1,2,4,5,6,7"
>    cpu_list_to_mask() {
>       MASK=0
>       for cpu in $(echo $1 | tr ',' ' '); do
>          MASK=$((MASK | (1 << cpu)))
>       done
>       printf "%x\n" "$MASK"
>    }
>    MASK=$(cpu_list_to_mask "$ALLOW_CPUS")
>    echo "Setting IRQ affinity to CPUs: $ALLOW_CPUS (mask=0x$MASK)"
>    for irq in /proc/irq/[0-9]*; do
>       smp_file="$irq/smp_affinity"
>       [ -w "$smp_file" ] && echo "$MASK" > "$smp_file" 2>/dev/null
>    done
>    ```
>
> The following example shows how to add a kernel command-line parameter to disable RCU callbacks (**rcu\_nocbs**) in `meta-qcom/conf/machine/<machine-name.conf>`:
>
> * CPU cores 7
> * IRQ affine to core 0-6
> * RCU no call back 7
>
> ```text theme={null}
> QCOM_RT_CPU        = "7"
> QCOM_IRQAFF        = "0-6"
> QCOM_RCU_NOCBS     = "7"
> QCOM_RCU_EXPEDITED = "1"
> QCOM_CPUIDLE_OFF   = "1"
> ```

### **Test the RT kernel**

A suite of tests is available in the Linux foundation RT test suite. The RT Linux kernel test obtains the following information:

* Real-time performance of the RT Linux kernel
* RT Linux kernel latencies and key performance indicators (KPIs)

> [!NOTE]
>
> Don't reboot the system during the RT Linux kernel test as it runs for over 24 hours.

Cyclictest tool is used for benchmarking the RT Linux kernel systems. It's used to evaluate the relative performance of the real-time systems. The Qualcomm Linux build has the cyclictest tool. For more information, see [Cyclictest](https://wiki.linuxfoundation.org/realtime/documentation/howto/tools/cyclictest/start).

This guide describes the following cyclictests:

* Cyclictest with no-load: System load isn't added to perform this test.
* Cyclictest with stress-ng (next-generation): Specific percentage of load is added to perform this test to measure the worst case system latencies.

1. Run the following cyclic test for QCS6490, Qualcomm Dragonwing<sup>™</sup> IQ-9075, and Qualcomm Dragonwing<sup>™</sup> IQ-615 development kits:
   ```text theme={null}
   cyclictest -a 7 -t 1 -m -l 100000000 -i 1000 -p 99 -h 100
   # -a 7 → pin threads to CPU 7 (RT cores)
   # -t 1 → 1 threads
   # -m → lock memory (avoid page faults)
   # -l 100000000 → long run
   # -i 1000 → 1 ms interval
   # -p 99 → RT priority
   # -h 100 → histogram up to 100 µs
   ```
2. Run the following cyclic test for the Dragonwing<sup>™</sup> IQ-8275 Development Kit:
   ```text theme={null}
   cyclictest -a 3 -t 1 -m -l 100000000 -i 1000 -p 99 -h 100
   ```

To access the RT test suite source code, see [rt-tests/rt-tests.git](https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git/). For more information, see [RT-Tests](https://wiki.linuxfoundation.org/realtime/documentation/howto/tools/rt-tests).

3. Note the latencies.

To run cyclic test with stress-ng on QCS6490, Qualcomm Dragonwing IQ-9075, and Qualcomm Dragonwing<sup>™</sup> IQ-615, do the following:

1. Set the target CPU load. Define the desired CPU load percentage. If not specified, the default load is 60%:
   ```text theme={null}
   LOAD=60
   ```
2. Run stress-ng on selected CPU cores in the background. Apply CPU load on cores 0, 1, 2, 4, 5, 6, and 7 using stress-ng. Each instance runs for one day and is pinned to a specific CPU core:
   ```text theme={null}
   for cpu in 0 1 2 3 4 5 6; do
       taskset -c $cpu stress-ng --cpu 1 --cpu-load "$LOAD" \
       --temp-path . -t 1d &
   done
   ```
3. Run the cyclic latency test. Execute cyclictest with high priority to measure scheduling latency. The test runs for approximately 27.78 hours:
   ```text theme={null}
   cyclictest -a 7 -t 1 -m -l 100000000 -i 1000 -p 99 -h 100 --mainaffinity 6
   ```
4. Note the worst-case latencies.

To run cyclic test with stress-ng on Dragonwing IQ-8275, do the following:

1. Set the target CPU load:
   ```text theme={null}
   LOAD=60
   ```
2. Run stress-ng on selected CPU cores:
   ```text theme={null}
   for cpu in 0 1 2 4 5 6 7; do
        taskset -c $cpu stress-ng --cpu 1 --cpu-load "$LOAD" \
        --temp-path . -t 1d &
   done
   ```
3. Run the cyclic latency test:
   ```text theme={null}
   cyclictest -a 3 -t 1 -m -l 100000000 -i 1000 -p 99 -h 100 --mainaffinity 2
   ```
4. Note the worst-case latencies.

### **RT test results**

| **Device** | **Distro and image** | **Cyclictest use case** | **Results** | **Comments** |
| - | - | - | - | - |
| QCS6490 RT core: Best core - 7 Boot Flow - KVM | `performance_linux-qcom-rt-6.18_qcom-distro-kvm and qcom-multimedia-image` | No load | `T: 0 ( 1687) P:99 I:1000 C:100000000 Min: 1 Act: 2 Avg: 1 Max: 5` | Completed; latencies are inline \< 20 µs |
| QCS6490 RT core: Best core - 7 Boot Flow - KVM | `performance_linux-qcom-rt-6.18_qcom-distro-kvm and qcom-multimedia-image` | Stress-NG | `T: 0 ( 1853) P:99 I:1000 C:100000000 Min: 1 Act: 2 Avg: 1 Max: 10` | Completed; latencies are inline \< 20 µs |
| Dragonwing IQ-615 RT core: Best core - 7 Boot Flow - KVM | `performance_linux-qcom-rt-6.18_qcom-distro-kvm and qcom-multimedia-image` | No load | `T: 0 (23693) P:99 I:1000 C: 100000000 Min: 1 Act: 2 Avg: 2 Max: 9` | Completed; latencies are inline \< 20 µs |
| Dragonwing IQ-615 RT core: Best core - 7 Boot Flow - KVM | `performance_linux-qcom-rt-6.18_qcom-distro-kvm and qcom-multimedia-image` | Stress-NG | `T: 0 (1590198) P:99 I:1000 C:100000000 Min: 1 Act: 2 Avg: 2 Max: 7` | Completed; latencies are inline \< 20 µs |

## **RT Benchmarks by Platform**

> [card] [RT Benchmarks IQX](https://dragonwingdocs-staging.qualcomm.com/Suites/Industrial-Edge-Suite/rt-benchmarks-iqx.md)
>
> Benchmark results for IQ-X series high-performance platforms

> [card] [RT Benchmarks IQ9](https://dragonwingdocs-staging.qualcomm.com/Suites/Industrial-Edge-Suite/rt-benchmarks-iq9.md)
>
> Benchmark results for the Dragonwing IQ-9075 platform

> [card] [RT Benchmarks IQ8](https://dragonwingdocs-staging.qualcomm.com/Suites/Industrial-Edge-Suite/rt-benchmarks-iq8.md)
>
> Benchmark results for the Dragonwing IQ-8275 platform

> [card] [RT Benchmarks IQ6](https://dragonwingdocs-staging.qualcomm.com/Suites/Industrial-Edge-Suite/rt-benchmarks-iq6.md)
>
> Benchmark results for the Dragonwing IQ-615 platform
