> ## Documentation Index
> Fetch the complete documentation index at: https://dragonwingdocs-staging.qualcomm.com/llms.txt
> Use this file to discover all available pages before exploring further.

# VM Management — RT with KVM

## **VM Management with Libvirt and Virsh**

Qualcomm platforms use **Libvirt** and **Virsh** for guest VM lifecycle management. Domain XML files define the VM configuration and are passed to `virsh` to create, start, and manage VMs.

***

### Setup Steps

#### 1. Enable libvirt daemons

```bash theme={null}
systemctl enable --now virtqemud virtqemud.socket
systemctl enable --now virtnetworkd virtnetworkd.socket
```

#### 2. Switch firewall backend to nftables

The `ip_tables` module is incompatible with `PREEMPT_RT` (its dependency `NETFILTER_XTABLES_LEGACY` requires `!PREEMPT_RT`). Use `nftables` instead:

```
# /etc/libvirt/network.conf
firewall_backend = "nftables"
```

#### 3. Ensure build includes iproute2 and `act_csum.ko`

* Add `iproute2` to the image recipe (provides the `tc` utility required by nftables)
* Enable `CONFIG_NET_ACT_CSUM=m` in the kernel config (provides `act_csum.ko`)

#### 4. Enable vhost-net in QEMU build

Add `vhost` to the QEMU PACKAGECONFIG in `meta-qcom-distro/conf/distro/include/qcom-base.inc`:

```diff theme={null}
-PACKAGECONFIG:append:pn-qemu = " pixman virtfs vnc-jpeg vnc-sasl"
+PACKAGECONFIG:append:pn-qemu = " pixman virtfs vnc-jpeg vnc-sasl vhost"
```

#### 5. Start default network and define VMs

```bash theme={null}
virsh net-start default
virsh net-list --all

# Copy kernel Image and rootfs to /var/rtvm/ and /var/hkvm/ first
virsh define rt-vm.xml
virsh define hk-vm.xml

virsh start rt-vm
virsh start hk-vm
virsh list --all
```

***

### Known Issues and Resolutions

> [accordion] Issue 1 — Failed to connect to hypervisor (virsh net-start default)
>
> **Cause:** `virtqemud` and `virtnetworkd` daemons and sockets are not running.
>
> **Resolution:**
>
> ```bash theme={null}
> systemctl start virtqemud virtqemud.socket
> systemctl enable virtqemud virtqemud.socket
> systemctl start virtnetworkd virtnetworkd.socket
> systemctl enable virtnetworkd virtnetworkd.socket
> ```

> [accordion] Issue 2 — ip\_tables module not found (PREEMPT\_RT incompatibility)
>
> **Cause:** `NETFILTER_XTABLES_LEGACY` depends on `!PREEMPT_RT`, so `ip_tables.ko` cannot be built on the RT kernel.
>
> **Resolution:** Set the libvirt network firewall backend to `nftables` in `/etc/libvirt/network.conf`:
>
> ```
> firewall_backend = "nftables"
> ```

> [accordion] Issue 3 — 'tc' not found after switching to nftables
>
> **Cause:** `iproute2` (which provides `tc`) and `act_csum.ko` are not included in the build.
>
> **Resolution:** Add `iproute2` to the image recipe and enable `CONFIG_NET_ACT_CSUM=m` in the kernel config.

> [accordion] Issue 4 — vhost-net support is not compiled in (virsh start fails)
>
> **Cause:** The `vhost` PACKAGECONFIG option is not enabled for QEMU.
>
> **Resolution:** Add `vhost` to `PACKAGECONFIG:append:pn-qemu` in `qcom-base.inc` (see Setup Step 4 above).

***

### Sample Domain XML Files

Two reference domain XML files are provided — one for the real-time guest VM and one for the non-RT housekeeping guest VM.

| File | VM Name | Purpose |
| - | - | - |
| `rt-vm.xml` | `rt-vm` | RT guest — vCPUs pinned to RT-isolated CPUs, SR-IOV VFs passed through |
| `hk-vm.xml` | `hk-vm` | Non-RT guest — vCPUs pinned to housekeeping CPUs, no device passthrough |

#### `rt-vm.xml` — Key Configuration

| Setting | Value | Purpose |
| - | - | - |
| vCPUs | 6 (pinned to CPUs 2–7) | Matches the RT-isolated CPU set from kernel cmdline |
| Memory | 4 GB (locked) | `<memoryBacking><locked/>` prevents swap-out and latency spikes |
| Emulator pin | CPU 0 | Keeps QEMU emulator thread on housekeeping CPU |
| RT scheduling | `SCHED_FIFO`, vCPUs 1–5 | Deterministic vCPU thread scheduling |
| SR-IOV VFs | 10 x Intel X550 VFs | Passed through via `<hostdev>` with `virtio-iommu-pci` |
| Guest cmdline | `isolcpus=1-5 rcu_nocbs=1-5 irqaffinity=0` | RT tuning applied inside the guest |

> [accordion] View full rt-vm.xml
>
> ```xml theme={null}
> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
>   <name>rt-vm</name>
>   <memory unit='KiB'>4194304</memory>
>   <currentMemory unit='KiB'>4194304</currentMemory>
>   <memoryBacking>
>     <locked/>
>   </memoryBacking>
>   <vcpu placement='static'>6</vcpu>
>   <os>
>     <type arch='aarch64' machine='virt-6.2'>hvm</type>
>     <kernel>/var/rtvm/Image</kernel>
>     <cmdline>console=hvc0 root=/dev/vda rcupdate.rcu_expedited=1 rcu_nocbs=1-5 isolcpus=1-5 irqaffinity=0 skew_tick=1</cmdline>
>     <boot dev='hd'/>
>   </os>
>   <features>
>     <gic version='3'/>
>   </features>
>   <cpu mode='host-passthrough' check='none'/>
>   <devices>
>     <emulator>/usr/bin/qemu-system-aarch64</emulator>
>     <disk type="file" device="disk">
>       <driver name="qemu" type="raw"/>
>       <source file="/var/rtvm/rootfs.img"/>
>       <target dev="vda" bus="virtio"/>
>     </disk>
>     <interface type='network'>
>       <source network='default' bridge='virbr0'/>
>       <model type='virtio'/>
>     </interface>
>     <console type='pty'>
>       <target type='virtio' port='0'/>
>     </console>
>     <!-- VF 1–10: Intel X550 SR-IOV virtual functions passed through -->
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source><address domain='0x0001' bus='0x02' slot='0x10' function='0x0'/></source>
>     </hostdev>
>     <!-- ... VF 2–10 follow the same pattern ... -->
>     <filesystem type='mount' accessmode='mapped' fmode='644' dmode='755'>
>       <source dir='/mnt/vmshare'/>
>       <target dir='hostshare'/>
>     </filesystem>
>   </devices>
>   <cputune>
>     <vcpupin vcpu='0' cpuset='2'/>
>     <vcpupin vcpu='1' cpuset='3'/>
>     <vcpupin vcpu='2' cpuset='4'/>
>     <vcpupin vcpu='3' cpuset='5'/>
>     <vcpupin vcpu='4' cpuset='6'/>
>     <vcpupin vcpu='5' cpuset='7'/>
>     <realtime>
>       <period>1000000</period>
>       <runtime>950000</runtime>
>     </realtime>
>     <vcpusched vcpus='1-5' scheduler='fifo' priority='1'/>
>     <emulatorpin cpuset='0'/>
>   </cputune>
>   <qemu:commandline>
>     <qemu:arg value="-device"/>
>     <qemu:arg value="virtio-iommu-pci,bus=pcie.0,addr=0x3"/>
>   </qemu:commandline>
> </domain>
> ```

#### `hk-vm.xml` — Key Configuration

| Setting | Value | Purpose |
| - | - | - |
| vCPUs | 2 (pinned to CPUs 0–1) | Housekeeping CPUs, separate from RT-isolated set |
| Memory | 2 GB | General-purpose, no memory locking needed |
| No `<hostdev>` / SR-IOV | — | Non-RT workload, no direct device passthrough |
| No `<realtime>` / `<vcpusched>` | — | Default CFS scheduler |

> [accordion] View full hk-vm.xml
>
> ```xml theme={null}
> <domain type='kvm'>
>   <name>hk-vm</name>
>   <memory unit='KiB'>2097152</memory>
>   <currentMemory unit='KiB'>2097152</currentMemory>
>   <vcpu placement='static'>2</vcpu>
>   <os>
>     <type arch='aarch64' machine='virt-6.2'>hvm</type>
>     <kernel>/var/hkvm/Image</kernel>
>     <cmdline>console=hvc0 root=/dev/vda</cmdline>
>     <boot dev='hd'/>
>   </os>
>   <features>
>     <gic version='3'/>
>   </features>
>   <cpu mode='host-passthrough' check='none'/>
>   <devices>
>     <emulator>/usr/bin/qemu-system-aarch64</emulator>
>     <disk type="file" device="disk">
>       <driver name="qemu" type="raw"/>
>       <source file="/var/hkvm/rootfs.img"/>
>       <target dev="vda" bus="virtio"/>
>     </disk>
>     <interface type='network'>
>       <source network='default' bridge='virbr0'/>
>       <model type='virtio'/>
>     </interface>
>     <console type='pty'>
>       <target type='virtio' port='0'/>
>     </console>
>   </devices>
>   <cputune>
>     <vcpupin vcpu='0' cpuset='0'/>
>     <vcpupin vcpu='1' cpuset='1'/>
>     <emulatorpin cpuset='0'/>
>   </cputune>
> </domain>
> ```

***

> [card] [NIC & SR-IOV](https://dragonwingdocs-staging.qualcomm.com/Suites/Industrial-Edge-Suite/rt-with-kvm-nic-sriov.md)
>
> Intel X550 kernel config and IOMMU-map setup

> [card] [RT with KVM](https://dragonwingdocs-staging.qualcomm.com/Suites/Industrial-Edge-Suite/rt-with-kvm.md)
>
> Back to overview
