Batch runs from YAML

Goal

Profile many combinations in one command and keep a manifest that says what ran, on what, and whether it passed. You will learn the difference between a config loop and hpx validate, how to list a matrix before running it, and what the run leaves behind.

Hardware and wiring

One or more boards on J-Link, each with a known probe serial when more than one is attached (hpx probes list). No power instrument unless the cases enable it.

Software prerequisites

hpx with the extras the engines need (aot for heliaAOT), every toolchain the matrix names, and the MLPerf Tiny fixture models the validation matrix registers (they ship in the repository under tests/fixtures/mlperf_tiny/).

Config file

hpx validate builds its cases from the registry, not from a file: models kws, vww, ic, ad; engines, toolchains, transports and memory presets from its flags. A config loop uses the checked-in files under configs/mlperf_tiny/, for example configs/mlperf_tiny/kws_rt_power.yaml, each of which pins a board, engine, toolchain, placement and power setup.

Run

List before running:

hpx validate --list --suite smoke --boards apollo510_evb
hpx validate --list --suite complete --boards apollo510_evb

Run a suite:

hpx validate --suite models-rt --boards apollo510_evb --output-dir results/validation --junit-xml results/validation/junit.xml

Or loop over configs:

for cfg in configs/mlperf_tiny/kws_rt_power_ap510.yaml configs/mlperf_tiny/vww_rt_power_ap510.yaml; do
  hpx profile --config "$cfg" --fail-on-invalid || echo "FAILED $cfg"
done

Expected output

--list for the smoke suite, run on the host on 2026-09-21:

Registered models: ad, ic, kws, vww
Registered boards: apollo330mP_evb, apollo3p_evb, apollo4l_blue_evb, apollo4p_blue_kxr_evb, apollo510_evb, apollo510dL_evb

1 case(s) would run:

  apollo510_evb-kws-rt-ns-arm-none-eabi-gcc-rtt-auto    helia-rt/ns     arm-none-eabi-gcc   rtt      auto

A completed suite writes one bundle directory per case plus validation_manifest.json, validation_report.md and junit.xml. The nightly complete suite on 2026-09-16 recorded, for apollo510_evb:

- total: 32
- pass: 32
- fail: 0
- skip: 0

| Case | Status | Duration (s) | Toolchain | Interface | Memory | Layers | Cycles |
| apollo510_evb-kws-rt-ns-arm-none-eabi-gcc-rtt-auto | pass | 13.7 | arm-none-eabi-gcc | rtt | auto | 13 | 2051232 |
| apollo510_evb-kws-rt-ns-atfe-rtt-auto              | pass | 15.2 | atfe              | rtt | auto | 13 | 1938240 |
| apollo510_evb-kws-aot-ns-arm-none-eabi-gcc-rtt-auto | pass | 32.4 | arm-none-eabi-gcc | rtt | auto | 13 | 2044512 |

How to read it

• A case id encodes the matrix point: board, model, engine, CMSIS-NN provider (ns or arm), toolchain, transport and memory preset. The Cycles column of the report is the clean-window average.
• validation_manifest.json records the hpx version, the source commit, the suite flags and, per case, the identity, provenance (model hash, compiler version, system clock, engine version) and health issues. It is the record a “validated on” claim points at.
• The loop form gives you full control of each config and --fail-on-invalid makes an invalid run exit 3; the suite form gives you the manifest and JUnit output CI can consume. Use the loop for your own models and the suite for the registered ones.
• A case whose built image does not fit the board’s flashable MRAM is a skip with the note image exceeds flashable MRAM: hpx profile stops when the link overflows MRAM, or after the build when the image exceeds the app extent, with exit 4, before flashing, and the case counts as skipped in the report and JUnit. On apollo330mP_evb and apollo510dL_evb the application MRAM is 1552 KB; radio firmware occupies the top of the 2 MB bank.
• Each case’s bundle is an ordinary result bundle, so Re-analyse saved results applies to any of them.

Variations

• Several boards at once: --boards apollo510_evb,apollo330mP_evb --jlink-serials <serial>,<serial>.
• Your own models: --model-paths a.tflite,b.tflite registers them for one run with one --model-arena-size; --models-file models.yaml gives each model its own ID and arena (see Models file).
• Power: --power on --power-boards apollo510_evb --power-serials <joulescope serial> turns the power stages on for the named boards (Power measurement).
• Repeats for statistics: --repeat 3 runs every case three times with attempt in the case id.
• The full option list is the generated hpx validate reference.

Models file

--models-file registers your own .tflite models with the validation matrix. The YAML holds a models mapping (or is that mapping) keyed by model ID; select the IDs with --models as usual.

models:
  mlperf-tiny-kws-a8w8:
    path: models/kws_a8w8.tflite
    arena_size: 32768
    name: KWS Reference (a8w8)
    family: mlperf-tiny-kws
    precision: a8w8
    source: zoo://mlperf-tiny-kws/a8w8@0124dea005e876b9589537180c0b1f111d115699

Key	Required	Meaning
path	yes	The .tflite file; a relative path resolves from the YAML file’s directory.
arena_size	no	Tensor arena in bytes; default 524288.
name	no	Display name; default the ID.
description	no	Free text for reports.
comparison_group	no	Recorded in case identity to group models for comparison; default the ID.
family	no	The model these precisions share, for example mlperf-tiny-kws.
precision	no	The precision label, for example a8w8 or fp32.
source	no	Where the file came from, for example a zoo:// reference.

An ID may not reuse a built-in model ID (kws, vww, ic, ad). family, precision and source reach each case’s model object in validation_manifest.json but not its case ID. Custom models have no ExecuTorch PTE contract, so the matrix enumerates no ExecuTorch cases for them. A model with a FLOAT16 tensor lists its LiteRT cases as skips, because upstream TFLM refuses such models. A model whose kernels read FLOAT16 also lists its helia-rt and helia-aot cases as skips on boards without an MVE (Helium) core, which have no fp16 kernels.

Validated on

The nightly hardware validation workflow is this recipe: on 2026-09-16 run 35078330526 ran hpx validate --suite complete at source commit 0714893 with hpx 0.1.6 on apollo510_evb (32 of 32 passed), apollo330mP_evb (32), apollo3p_evb (24 of 24) and apollo4l_blue_evb (12 of 24, 12 skipped because its NSX board module does not declare the ATfE toolchain), with the nsx modules recorded in each bundle’s nsx.lock. Nine of those bundles ship under examples/results/hardware-validation-2026-09-16/. The --list output above was produced on the host on 2026-09-21. The config loop over configs/mlperf_tiny/ is not yet validated on this page: those configs pin a bench-specific jlink_serial.