Compare engines on one model

Goal

Measure what the engine choice costs and buys for one model, holding board, toolchain, transport, counters and iterations fixed. You will learn how to run the same recipe three times, how hpx compare reports the result, and why it withholds per-layer deltas across engines.

Hardware and wiring

An Apollo510 EVB on its USB J-Link port; no power instrument. The board, its default RTT transport and the J-Link are described on Boards and targets and Capture transports.

Software prerequisites

hpx with the aot extra for heliaAOT (pip install 'helia-profiler[aot]'), the GNU Arm toolchain, CMake and Ninja, SEGGER J-Link software (Install). hpx doctor must pass.

Config file

Start from these three checked-in configurations. They share the model, board and toolchain; align their counter selections before a controlled comparison:

Engine	Config
heliaRT	`examples/quickstart/hpx_rt.yml`
heliaAOT	`examples/quickstart/hpx_aot.yml`
TFLM	`examples/quickstart/hpx_tflm_baseline.yml`

The heliaRT and TFLM configs also ask for the memory counter group; hpx_aot.yml asks for cpu only, so add memory: default to it before comparing per-layer columns, or expect them to differ.

Run

From the repository root, pass the model path explicitly to override each config’s relative path:

hpx profile examples/quickstart/kws_model.tflite --config examples/quickstart/hpx_rt.yml
hpx profile examples/quickstart/kws_model.tflite --config examples/quickstart/hpx_aot.yml
hpx profile examples/quickstart/kws_model.tflite --config examples/quickstart/hpx_tflm_baseline.yml
hpx compare results/results_rt results/results_aot --output-dir results/rt-vs-aot
hpx compare results/results_rt results/results_tflm_baseline

Expected output

The comparison below is the real hpx compare output for the heliaRT and heliaAOT bundles named under Validated on, run on the host against the shipped bundles (paths shortened; the Config table and the per-region memory rows are elided, marked with …):

  Total cycles     2,060,239 cyc → 2,054,228 cyc  -6,011 cyc (-0.3%)
  Avg inference    21,717 us → 21,649 us  -68 us (-0.3%)

╭─ Warnings ─────────────────────────────────────────────────────────╮
│    - Per-layer deltas omitted because operation sequences differ.  │
│    - Comparison dimension differs: engine.                         │
│    - Comparison dimension differs: engine_version.                 │
│    - Comparison dimension differs: architecture_flags.             │
│    - Some run-level metrics are missing in one run:                │
│  memory.arena_size, memory.allocated_arena, memory.model_size      │
│    - AOT memory placement artifacts are present in only one run;   │
│  placement diffs may be partial.                                   │
╰────────────────────────────────────────────────────────────────────╯

 …

 Metric                         Baseline      Candidate              Change
 Total cycles                  2.06M cyc      2.05M cyc   -6.0k cyc (-0.3%)
 Layers                               13             13           0 (+0.0%)
 Binary .text                     279k B         151k B    -127k B (-45.7%)
 Binary .data                     155k B         2.8k B    -152k B (-98.2%)
 Binary .bss                       86k B         185k B     99k B (+114.9%)
 Binary total                     885k B         741k B    -144k B (-16.3%)
 …

The three single-engine runs, from their summary.json:

Engine	`total_cycles` (per-layer sum)	`latency.device_clean_infer_avg_cycles`	`binary.total` bytes	`binary.text` bytes
heliaRT 1.20.0	2,060,239	2,051,232	885,056	278,596
heliaAOT 0.20.0	2,054,228	2,044,512	740,984	151,160
TFLM (CMSIS-NN backend)	2,118,339	2,109,120	1,114,392	507,932

How to read it

Clean cycles are close for this captured model and configuration. These are comparisons of complete engine stacks; they do not isolate dispatch overhead or predict the difference on another model.
binary.text is the GNU size tool’s code plus read-only-data total for the complete profiling image, not the literal ELF .text section or model-only code. It is 45.7% smaller for this AOT image than this heliaRT image. binary.total includes RAM categories and is not deployment flash usage. The console labels above are preserved from the captured output; Output files defines the metrics.
hpx compare omits per-layer deltas when the operation sequences differ (heliaAOT reports the model’s original operator indices, the interpreters report the runtime’s), and it lists engine, engine_version and architecture_flags as informative dimensions so the report says what changed rather than hiding it. The rules are on Analysis and comparison.
Quote latency.device_clean_infer_avg_cycles as the inference cost; total_cycles is the instrumented per-layer sum (Latency and per-layer).

Variations

ExecuTorch: the same nightly run profiled the KWS model as a .pte through nsx-executorch (14 layers, clean window 2,148,976 cycles on the same board); the bundle apollo510_evb-kws-executorch-ns-arm-none-eabi-gcc-rtt-auto sits beside the others. The export path is on Engines.
TFLM reference kernels: copy hpx_tflm_baseline.yml and set engine.backend: reference to measure the un-optimised interpreter.
A different model: keep the three configs and change model.path and model.arena_size in each; hpx analyze model.tflite --engine helia-aot --compare shows operator coverage before you build.
Placement: control the arena and constant locations across the runs, using each engine’s placement model. Confirm actual placement in the result rather than assuming identical flags produce identical layouts (Memory placement).

Validated on


Board	`apollo510_evb` (Apollo510, Cortex-M55, `lp` clock tier)
Run	nightly hardware validation run 35078330526, 2026-09-16, suite `complete`, hpx 0.1.6 at source commit `0714893`
Cases	`apollo510_evb-kws-rt-ns-arm-none-eabi-gcc-rtt-auto`, `apollo510_evb-kws-aot-ns-arm-none-eabi-gcc-rtt-auto`, `apollo510_evb-kws-tflm-arm-arm-none-eabi-gcc-rtt-auto`, all `valid`
nsx modules	baseline `hpx-neuralspotx-0.8.1-2026-09`: `nsx-helia-rt` helia-rt-v1.20.0, `nsx-cmsis-nn` aaeb145, `nsx-tflite-micro` 7afcf2b, `arm-cmsis-nn` 6d21a6f, `nsx-ambiq-sdk` a9f4ec2, `nsx-pmu-armv8m` 5725c06 (each bundle’s `nsx.lock`)
Toolchain	arm-none-eabi-gcc 15.2.1 (Arm GNU Toolchain 15.2.Rel1), CMake 4.1.2
Differences from the configs above	the nightly cases use `arena_size: 32768`, `pmu_counters: {cpu: default}`, `iterations: 3`, `warmup: 1`, `output.format: csv`; the quickstart configs use a 131,072-byte arena and, for heliaRT and TFLM, the `memory` group too
Bundles	`examples/results/hardware-validation-2026-09-16/`

The hpx compare output above was produced on the host from those bundles on 2026-09-21 with hpx 0.1.6; the compare step needs no hardware.