Inference and layer timing

Read the timing numbers correctly. A bundle carries three kinds of time and cycle counts with different boundaries; quoting the wrong one can lead to a misleading comparison.

Inputs and output checking

Ordinary hpx profile measures execution time and counters; it does not feed representative samples, export output tensor values, or compare outputs with a reference. The profiling harness initializes input storage with zero bytes: heliaRT/LiteRT clear the first input tensor, heliaAOT clears every input buffer, and ExecuTorch clears its configured input buffer. Zero bytes need not represent real-valued zero for a quantized tensor with a nonzero zero point.

Treat these results as timings for that input setup. Data-dependent kernels and stateful models can behave differently on application data; clearing input buffers is not a promise that the runtime resets internal state between calls. For caller-supplied inputs and output readback, use the run API, from the command line or from Python, rather than inferring numerical correctness from a successful profile.

Run API from the command line

The run API builds a model once for fixed input bytes and captures from that build later, on this host or another, comparing the outputs with a reference. Four commands walk it:

hpx runtimes prepare helia-rt   # heliaRT and LiteRT link a prepared runtime
hpx freeze model.tflite --engine helia-rt --backend helia --toolchain atfe \
    --cpu-clock lp --arena-size 131072 --arena-location sram \
    --weights-location mram -o plan.json
hpx build plan.json --out runs/model --input input.bin
hpx validate build runs/model
hpx run runs/model --board apollo510_evb --jlink-serial 1160002255 --tolerance 1

• hpx freeze takes a subset of the options hpx profile takes: the config file (-c), the model, --engine, --engine-config, --arena-size, --arena-location, --weights-location, --core-override, --board, --toolchain, --cpu-clock, --iterations and --warmup, plus --backend for engine.backend. It writes the resolved plan. A run build needs ATfE, the lp clock, an SRAM arena of explicit size and MRAM weights.
• hpx build copies the model and one file per model input (--input, repeated in model order; zero bytes when omitted) into runs/model/inputs, compiles under runs/model/work, and saves runs/model/build.json. The directory must be new or empty, so a failed build never touches an earlier one; a --work-dir must sit inside it, since the saved build records its files relative to the run directory. --lifecycle and --timing-scope choose how the build runs and what its timed span covers.
• hpx validate build re-verifies every pinned file and the plan without hardware. The saved build names the run directory and the HPX cache as roots. The run directory can move to another host as a unit, but a heliaRT or LiteRT build links a runtime prepared under the HPX cache: on the new host, run hpx runtimes prepare for the same release, or set HPX_CACHE_DIR to a cache that holds it.
• hpx run flashes the build, captures once into a fresh evidence-<N> directory, or a new --evidence-dir inside the run directory, and saves the run there as result.json (A saved run result). You vouch for the physical board behind the probe, as hpx profile assumes: --board names the board wired to the probe --jlink-serial names. A --board other than the build’s board is refused before any device is touched; HPX cannot check the physical board itself. Outputs are compared with host LiteRT by default (the analysis extra); --reference files or --expect none change that.

hpx validate build and hpx run take the run directory or its build.json. Each command prints JSON: the build’s inputs and identity, or a run’s outcome, the authority that vouched for the board, the saved result’s path, its identity, per-tensor verification and timing. A non-finite error is written as null; identical and mismatched_elements still say what differed. hpx run exits 1 when the capture fails and 3 when the outputs do not match.

Prerequisites

A result bundle and Output files for the field names.

Whole-model timing and per-layer detail come from separate measurement passes.

Which number

I want	Read	Boundary	Shaped by
The cost of one inference	latency.device_clean_infer_avg_cycles (cycles), latency.device_clean_infer_avg_us (microseconds)	the clean window: warm, back to back, no per-layer instrumentation; aggregated over device_clean_infer_count inferences	profiling.iterations, warmup, window_mode, window_target_ms; the clock tier target.clock.cpu
Where the cycles go	profile_results.csv rows, summary.top_layers	the instrumented passes: each layer bracketed by counter reads, aggregated over the measured iterations by profiling.aggregation	profiling.pmu_counters, per_layer, iterations, warmup
How long the profiled passes took on the device	latency.device_profiled_infer_avg_us, _total_us, _count	the instrumented passes, on the device clock	as above
How long the whole capture took	latency.capture_duration_s, hpx_start_latency_s, protocol_duration_s, boot_phases_s	host-side: boot, handshake, transport, everything; never an inference number	target.transport, timeouts
Energy per inference	power.energy_per_inference_j	the power scope’s window (Windows)	power.*

Steps

1. Quote the clean window

device_clean_infer_avg_cycles and device_clean_infer_avg_us describe the same window: the firmware’s clean loop with the gate high and no per-layer reads. That is the number to publish. summary.total_cycles is a different quantity: the sum of the instrumented per-layer rows, which includes a counter read around every operator and changes when profiling.pmu_counters changes. The terminal prints both, as Total cycles and Clean E2E cycles with the percentage difference against the per-layer sum; that gap is the instrumentation overhead (plus input restoration on heliaAOT, where the label says includes input refill).

2. Use per-layer rows to explain, not to total

Each profile_results.csv row is one operator over iterations instrumented inferences, aggregated by median (default), mean or trimmed after structurally invalid samples are rejected. top_layers in the summary is the same data ranked, with pct against total_cycles. Sort by cycles, then look at the same rows’ cache and MVE columns to see why a layer is hot; the derived metrics are the ratios to compute.

3. Convert cycles to time honestly

Cycles transfer between clock tiers far better than microseconds do, but not perfectly: memory latency is fixed in time, so a faster core waits more cycles on MRAM or PSRAM (Boards and targets). The clock tier the run used is in run_metadata.json (platform). When you convert or transfer a figure, say which tier.

4. Watch the boundaries a comparison crosses

• iterations and aggregation change the per-layer statistics but not the clean window’s meaning; window_mode, window_target_ms and window_min/window_max change how many inferences the clean window ran.
• On the Cortex-M4F families the clean window’s cycle counter can stall if the debugger detaches; latency.device_clean_stalled_iters and partial_iters say so, and the codes on Verifying a capture explain the checks.
• clean_workload names what the clean window ran (AOT input restoration is inside it from schema v6); a comparison across different workloads is not a latency comparison.

5. Check the window facts before comparing routes

summary.identity.run states how a time was taken, and a fixed-input fixture or a run-API capture states the same fields in its RunIdentity. Compare times only when these agree:

Field	Ordinary profile	Fixture and run API
timing_scope	the clean window: restore_and_invoke on heliaAOT, else invoke_only	the caller’s FixtureMethod: invoke_only brackets each invocation, restore_and_invoke one span around the timed loop, input restoration included
timing_clock, timing_clock_hz	stimer at 32,768 Hz for the busy_loop probe; otherwise cpu cycles at the CPU clock for ExecuTorch (its execute-cycle count) on every SoC, stimer on Apollo5, and cpu (DWT) cycles elsewhere	stimer at 32,768 Hz; a single invocation under about 30.5 µs can fail as TIMING_BOUND
timing_statistic	mean per invocation	total ticks over the timed invocations; divide by iterations and the rate for a mean
lifecycle, instance_boundary	carry/boot; ExecuTorch single/invocation: every call reloads the program, so each invocation is its own instance	the build’s lifecycle: carry runs the plan’s counts, single one timed invocation and no warmups per boot
first_invoke_initializer, first_invoke_window	whether the first invocation runs a one-time initializer, and whether it is a warmup (always, before the clean window) or measured (ExecuTorch)	the same answer per engine; measured under single or with no warmups, so a CALL_ONCE lands in the timed span

LiteRT and heliaRT run a CALL_ONCE subgraph inside the first Invoke(); heliaAOT runs it in model_init(), before any timed span. Engine initialization (AllocateTensors(), model_init()) is never timed. Clock rates are the configured ones, not measured.

Expected output

A number you can label: “N cycles per inference, clean window, average over M completed inferences, board B at tier T”, with the per-layer table explaining it.

What next

Analysis and comparison to hold two of those numbers against each other with the rules that keep the comparison fair.