> ## Documentation Index
> Fetch the complete documentation index at: https://dragonwingdocs-staging.qualcomm.com/llms.txt
> Use this file to discover all available pages before exploring further.

# LLMs/VLMs using GenieX

> Run GenieX CLI on a Qualcomm Dragonwing IQ9 device, pull the Gemma 4 E4B bundle, and run text, image, and audio inference on the Hexagon NPU.

GenieX is the Qualcomm AI Hub command line and serving layer for generative models. On
a Dragonwing ARM64 Linux host it runs models natively against the device NPU through
the Qualcomm driver libraries.

This page takes you from a freshly imaged IQ9 board to Gemma 4 E4B answering prompts on
the NPU. Expect around 15 minutes of setup, most of it spent downloading the \~5 GB model
bundle.

You will:

1. Pull the Gemma 4 E4B bundle from AI Hub.
2. Run text, image, and audio inference, and optionally serve an OpenAI-compatible API.

<Warning>
  Every step on this page runs **directly on the IQ9 device**, not on a development host.
</Warning>

Open a session on the board:

```bash theme={null}
ssh root@<device-ip>
```

## Prerequisites

| Requirement | Value |
| - | - |
| Architecture | Linux ARM64 (`aarch64`) |
| Operating system | Linux image for Dragonwing IQ-9075 EVK, or any device with a Qualcomm BSP |
| Chipset | Dragonwing IoT chipset — IQ-9075 / QCS9075 in this blog |
| Free storage | ≥ 12 GB |
| RAM | ≥ 12 GB |
| Network | Outbound HTTPS to `qaihub-public-assets.s3.us-west-2.amazonaws.com` |

Confirm the architecture and available space before continuing:

```bash theme={null}
uname -m      # expected: aarch64
df -h /       # confirm at least 12 GB available
```

<Note>
  If the board is not yet brought up, complete device setup first — see the
  [IQ-9075 EVK setup guide](https://dragonwingdocs.qualcomm.com/Linux/devices/iq9075-evk/set-up-the-device).
  This is only applicable to IQ9 and please refer [GenieX](https://aihub.qualcomm.com/geniex) page for other platforms and select the supported model.
</Note>

### Install GenieX

<Steps>
  <Step title="Run the installer">
    If `HOME` is unset — common in minimal containers, export it
    first:

    ```bash theme={null}
    export HOME=/root
    ```

    ```bash theme={null}
    curl -fsSL https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-geniex/install.sh | sh
    ```

    The script downloads the latest stable release and verifies its SHA256 checksum.
  </Step>
</Steps>

### Verify GenieX

```bash theme={null}
geniex --help
```

Add `--log` to any command to raise the log level. The flag is equivalent to the
`GENIEX_LOG` environment variable and takes precedence over it.

| Level | Emits |
| - | - |
| `none` | nothing (default) |
| `error` | errors only |
| `warn` | warnings and errors |
| `info` | informational messages and above |
| `debug` | debug messages and above |
| `trace` | everything |

## Download the model

```bash theme={null}
geniex pull google/gemma-4-E4B-it-qat-q4_0-gguf
```

This transfers roughly 5 GB, so allow several minutes on a typical link. If the transfer
is interrupted, re-run the same command — it resumes rather than starting over.

The general syntax is:

```bash theme={null}
geniex pull <model-name>[:<precision>]
```

| Flag | Purpose |
| - | - |
| `--model-hub` | Model source: `aihub`, `hf`, or `localfs`. Auto-detected when omitted. |
| `--local-path` | Register a bundle already present on disk. |

For GGUF models the CLI prompts for a precision when more than one is published. Choose
**`Q4_0`** for IQ9 — it is the quantization-aware-trained build and gives the best
accuracy per byte on the NPU. Pin it inline to skip the prompt in scripts:

```bash theme={null}
geniex pull google/gemma-4-E4B-it-qat-q4_0-gguf:Q4_0
```

Confirm the result:

```bash theme={null}
geniex list
```

<Note>
  `pull` copies files into the GenieX cache. After a successful `--local-path` pull you can
  delete the source directory rather than keeping two copies of a \~5 GB model.
</Note>

### Bundle contents

GenieX manages the GGUF bundle for you. It contains the quantized `*.gguf` weights, an
`mmproj-*.gguf` multimodal projector for image and audio input, tokenizer metadata
embedded in the GGUF container, and a manifest recording precision, runtime, and compute
defaults.

A **Genie/QAIRT** bundle — the `w4a16` build, or output from the Jupyter path — is
instead explicit, and must contain:

| File | Purpose |
| - | - |
| `genie_config.json` | Backend selection, model and tokenizer paths, context and token limits |
| `htp_backend_ext_config.json` | HTP/NPU backend settings (SoC ID, DSP architecture, performance mode) |
| `tokenizer.json` | Tokenizer vocabulary and merge rules |
| `*.bin` | Prompt-processor and token-generator context binaries |

<Note>
  Sample Genie configurations are published in the
  [AI Hub Apps repository](https://github.com/qualcomm/ai-hub-apps/tree/main/tutorials/llm_on_genie/configs/genie);
  for field definitions see the
  [Genie dialog JSON reference](https://docs.qualcomm.com/doc/80-63442-10/topic/json.html#genie-dialog-json-config-string).
</Note>

## Run inference

Start an interactive chat session:

```bash theme={null}
geniex infer google/gemma-4-E4B-it-qat-q4_0-gguf
```

The model loads onto the NPU and you get a prompt. Type a message and press Enter.

Pass a single prompt and exit instead — useful for scripts and smoke tests:

```bash theme={null}
geniex infer google/gemma-4-E4B-it-qat-q4_0-gguf \
  -p "Summarize what a Hexagon NPU does in three sentences."
```

### Common flags

| Flag | Effect |
| - | - |
| `-p "<prompt>"` | Run one prompt and exit instead of opening a session. |
| `--think` | Show intermediate reasoning before the answer. |
| `--think=false` | Answer directly. Recommended in production. |
| `--compute npu` | Run on the Hexagon NPU. This is the default. |
| `--compute cpu` / `--compute gpu` | Run on CPU or GPU. Valid for GGUF builds only; useful for A/B comparison or isolating an NPU driver problem. |
| `--log <level>` | Raise the log level for diagnostics. |

### Multimodal prompts

The `q4_0` bundle ships an audio-capable projector, so a single prompt can carry both an
image and an audio clip. Download two sample files into your home directory:

```bash theme={null}
cd ~
curl -L -o jfk.wav https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav
curl -L -o landmark.jpg "https://images.pexels.com/photos/402028/pexels-photo-402028.jpeg?w=1024"
```

Reference them by absolute path in the prompt:

```bash theme={null}
geniex infer google/gemma-4-E4B-it-qat-q4_0-gguf \
  -p "Describe the image and transcribe the audio. Image: $HOME/landmark.jpg Audio: $HOME/jfk.wav"
```

Expected output:

```
**Image Description:**
This is a scenic, panoramic photograph that features a traditional Japanese temple ...

**Audio Transcription:**
And so my fellow Americans, ask not what your country can do for you, ask what you
can do for your country.
```

<Note>
  Always use **absolute** paths for image and audio inputs. Relative paths resolve against
  the process working directory and are a frequent source of "file not found" errors.
</Note>

In an interactive session, `/mic` records a clip instead of loading one from disk;
`Ctrl-C` stops recording and transcribes it. This requires SoX on your `PATH`.

<Warning>
  QAIRT models report `audio: false`. Passing one an audio file fails with
  `GenieXError(-201201): Multimodal generation failed`. Use the GGUF build for audio.
</Warning>

### Serve an OpenAI-compatible endpoint

For application integration, run the local server instead of the interactive CLI:

```bash theme={null}
geniex serve
```

The server prints the address and port it is listening on at startup. Substitute those
values below:

```bash theme={null}
curl http://<device-ip>:<port>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-E4B-it-qat-q4_0-gguf",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What is the capital of Italy?"}
    ],
    "stream": false
  }'
```

Any OpenAI-compatible client or framework works, including LangChain and Open WebUI. Set
the per-request `reasoning_format` field to move a thinking model's chain-of-thought out
of `message.content` into `message.reasoning_content`, which keeps the rendered answer
clean while preserving the reasoning for logs.

<Note>
  Test with `curl` from the device itself first. If the endpoint works locally but not
  remotely, the server is bound to loopback or a firewall is blocking the port.
</Note>

## Performance and best practices

On IQ9 the reference figures are a 4096-token context, \~660 tokens/s prefill, and
\~17.9 tokens/s decode. Because decode is roughly 37× slower than prefill, output length
dominates response time far more than prompt length.

* **Reuse the loaded model.** Model load is the largest fixed cost — use `geniex serve`
  or a long-lived interactive session rather than invoking `geniex infer` per request.
* **Cap output length.** Ask for "three bullet points" rather than "explain in detail"
  when latency matters.
* **Keep prompts short.** Prefill is fast, but prompt tokens still consume context that
  generation needs.
* **Disable thinking mode** (`--think=false`) unless you consume the reasoning trace; it
  can multiply the number of generated tokens.
* **Right-size the context.** KV-cache memory scales with context length, so do not
  configure a large window for a task that never needs it.
* **Watch thermals.** Sustained generation raises SoC temperature and triggers
  throttling; measure steady-state throughput, not just the first request.
* **Keep storage headroom.** Provision roughly double the \~5 GB bundle size to allow for
  upgrades.

## Verify the result

| # | Check | Command | Expected |
| - | - | - | - |
| 1 | Correct architecture | `uname -m` | `aarch64` |
| 2 | Drivers installed | `apt list --installed qcom-adreno1 qcom-fastrpc1 libqnn1` | All three present |
| 3 | CLI works | `geniex --help` | Usage text, no library errors |
| 4 | Model in cache | `geniex list` | Gemma 4 E4B listed |
| 5 | Inference returns text | `geniex infer google/gemma-4-E4B-it-qat-q4_0-gguf -p "Reply with OK."` | A short coherent reply within seconds of model load |
| 6 | NPU backend selected | `geniex --log debug infer google/gemma-4-E4B-it-qat-q4_0-gguf -p "Hi"` | NPU backend in the log, not a CPU fallback |

If any check fails, see Troubleshooting.

## Next step

The published bundle is now running on the NPU. If it meets your accuracy, context
length, and licensing requirements, you are done — move on to integrating it with your
application.

If you need a custom quantization recipe, a longer context, or your own checkpoint,
continue to the Jupyter notebook path. Otherwise go to Troubleshooting and next steps.