# Compare engines on one model

## Goal

Measure what the engine choice costs and buys for one model, holding
board, toolchain, transport, counters and iterations fixed. You will learn
how to run the same recipe three times, how `hpx compare` reports the
result, and why it withholds per-layer deltas across engines.

## Hardware and wiring

An Apollo510 EVB on its USB J-Link port; no power instrument. The board,
its default RTT transport and the J-Link are described on
[Boards and targets](https://ambiqai.github.io/helia-profiler/guide/boards/) and
[Capture transports](https://ambiqai.github.io/helia-profiler/guide/transports/).

## Software prerequisites

hpx with the `aot` extra for heliaAOT (`pip install 'helia-profiler[aot]'`),
the GNU Arm toolchain, CMake and Ninja, SEGGER J-Link software
([Install](https://ambiqai.github.io/helia-profiler/getting-started/install/)). `hpx doctor` must pass.

## Config file

Start from these three checked-in configurations. They share the model,
board and toolchain; align their counter selections before a controlled
comparison:

| Engine | Config |
|---|---|
| heliaRT | [`examples/quickstart/hpx_rt.yml`](https://github.com/AmbiqAI/helia-profiler/blob/main/examples/quickstart/hpx_rt.yml) |
| heliaAOT | [`examples/quickstart/hpx_aot.yml`](https://github.com/AmbiqAI/helia-profiler/blob/main/examples/quickstart/hpx_aot.yml) |
| TFLM | [`examples/quickstart/hpx_tflm_baseline.yml`](https://github.com/AmbiqAI/helia-profiler/blob/main/examples/quickstart/hpx_tflm_baseline.yml) |

The heliaRT and TFLM configs also ask for the `memory` counter group;
`hpx_aot.yml` asks for `cpu` only, so add `memory: default` to it before
comparing per-layer columns, or expect them to differ.

## Run

From the repository root, pass the model path explicitly to override each config’s relative path:

```bash
hpx profile examples/quickstart/kws_model.tflite --config examples/quickstart/hpx_rt.yml
hpx profile examples/quickstart/kws_model.tflite --config examples/quickstart/hpx_aot.yml
hpx profile examples/quickstart/kws_model.tflite --config examples/quickstart/hpx_tflm_baseline.yml
hpx compare results/results_rt results/results_aot --output-dir results/rt-vs-aot
hpx compare results/results_rt results/results_tflm_baseline
```

## Expected output

The comparison below is the real `hpx compare` output for the heliaRT and
heliaAOT bundles named under Validated on, run on the host against the
shipped bundles (paths shortened; the Config table and the per-region
memory rows are elided, marked with `…`):

```
  Total cycles     2,060,239 cyc → 2,054,228 cyc  -6,011 cyc (-0.3%)
  Avg inference    21,717 us → 21,649 us  -68 us (-0.3%)

╭─ Warnings ─────────────────────────────────────────────────────────╮
│    - Per-layer deltas omitted because operation sequences differ.  │
│    - Comparison dimension differs: engine.                         │
│    - Comparison dimension differs: engine_version.                 │
│    - Comparison dimension differs: architecture_flags.             │
│    - Some run-level metrics are missing in one run:                │
│  memory.arena_size, memory.allocated_arena, memory.model_size      │
│    - AOT memory placement artifacts are present in only one run;   │
│  placement diffs may be partial.                                   │
╰────────────────────────────────────────────────────────────────────╯

 …

 Metric                         Baseline      Candidate              Change
 Total cycles                  2.06M cyc      2.05M cyc   -6.0k cyc (-0.3%)
 Layers                               13             13           0 (+0.0%)
 Binary .text                     279k B         151k B    -127k B (-45.7%)
 Binary .data                     155k B         2.8k B    -152k B (-98.2%)
 Binary .bss                       86k B         185k B     99k B (+114.9%)
 Binary total                     885k B         741k B    -144k B (-16.3%)
 …
```

The three single-engine runs, from their `summary.json`:

| Engine | `total_cycles` (per-layer sum) | `latency.device_clean_infer_avg_cycles` | `binary.total` bytes | `binary.text` bytes |
|---|---|---|---|---|
| heliaRT 1.20.0 | 2,060,239 | 2,051,232 | 885,056 | 278,596 |
| heliaAOT 0.20.0 | 2,054,228 | 2,044,512 | 740,984 | 151,160 |
| TFLM (CMSIS-NN backend) | 2,118,339 | 2,109,120 | 1,114,392 | 507,932 |

## How to read it

- Clean cycles are close for this captured model and configuration. These
  are comparisons of complete engine stacks; they do not isolate dispatch
  overhead or predict the difference on another model.
- `binary.text` is the GNU size tool's code plus read-only-data total for
  the complete profiling image, not the literal ELF `.text` section or
  model-only code. It is 45.7% smaller for this AOT image than this heliaRT
  image. `binary.total` includes RAM categories and is not deployment flash
  usage. The console labels above are preserved from the captured output;
  [Output files](https://ambiqai.github.io/helia-profiler/guide/output/) defines the metrics.
- `hpx compare` omits per-layer deltas when the operation sequences differ
  (heliaAOT reports the model's original operator indices, the
  interpreters report the runtime's), and it lists `engine`,
  `engine_version` and `architecture_flags` as informative dimensions so
  the report says what changed rather than hiding it. The rules are on
  [Analysis and comparison](https://ambiqai.github.io/helia-profiler/guide/analysis-comparison/).
- Quote `latency.device_clean_infer_avg_cycles` as the inference cost;
  `total_cycles` is the instrumented per-layer sum
  ([Latency and per-layer](https://ambiqai.github.io/helia-profiler/guide/latency/)).

## Variations

- ExecuTorch: the same nightly run profiled the KWS model as a `.pte`
  through `nsx-executorch` (14 layers, clean window 2,148,976 cycles on
  the same board); the bundle
  `apollo510_evb-kws-executorch-ns-arm-none-eabi-gcc-rtt-auto` sits beside
  the others. The export path is on [Engines](https://ambiqai.github.io/helia-profiler/guide/engines/).
- TFLM reference kernels: copy `hpx_tflm_baseline.yml` and set
  `engine.backend: reference` to measure the un-optimised interpreter.
- A different model: keep the three configs and change `model.path` and
  `model.arena_size` in each; `hpx analyze model.tflite --engine helia-aot --compare`
  shows operator coverage before you build.
- Placement: control the arena and constant locations across the runs,
  using each engine's placement model. Confirm actual placement in the
  result rather than assuming identical flags produce identical layouts
  ([Memory placement](https://ambiqai.github.io/helia-profiler/guide/memory/)).

## Validated on

| | |
|---|---|
| Board | `apollo510_evb` (Apollo510, Cortex-M55, `lp` clock tier) |
| Run | nightly hardware validation [run 35078330526](https://github.com/AmbiqAI/helia-profiler/actions/runs/35078330526), 2026-09-16, suite `complete`, hpx 0.1.6 at source commit `0714893` |
| Cases | `apollo510_evb-kws-rt-ns-arm-none-eabi-gcc-rtt-auto`, `apollo510_evb-kws-aot-ns-arm-none-eabi-gcc-rtt-auto`, `apollo510_evb-kws-tflm-arm-arm-none-eabi-gcc-rtt-auto`, all `valid` |
| nsx modules | baseline `hpx-neuralspotx-0.8.1-2026-09`: `nsx-helia-rt` helia-rt-v1.20.0, `nsx-cmsis-nn` aaeb145, `nsx-tflite-micro` 7afcf2b, `arm-cmsis-nn` 6d21a6f, `nsx-ambiq-sdk` a9f4ec2, `nsx-pmu-armv8m` 5725c06 (each bundle's `nsx.lock`) |
| Toolchain | arm-none-eabi-gcc 15.2.1 (Arm GNU Toolchain 15.2.Rel1), CMake 4.1.2 |
| Differences from the configs above | the nightly cases use `arena_size: 32768`, `pmu_counters: {cpu: default}`, `iterations: 3`, `warmup: 1`, `output.format: csv`; the quickstart configs use a 131,072-byte arena and, for heliaRT and TFLM, the `memory` group too |
| Bundles | [`examples/results/hardware-validation-2026-09-16/`](https://github.com/AmbiqAI/helia-profiler/tree/main/examples/results/hardware-validation-2026-09-16) |

The `hpx compare` output above was produced on the host from those
bundles on 2026-09-21 with hpx 0.1.6; the compare step needs no hardware.