Performance and accuracy options

heliaAOT’s defaults keep each knob’s default numerics; FP16 convolutions pack their weights by default, which changes only their FP16 rounding (see Accumulation). The optimization settings let a conversion trade accuracy or memory for speed where your model allows it. Every conversion writes the choice it made for each operator to <prefix>_plan.json, explains it in <prefix>_report.json, and prints the alternatives that apply under Optimization hints in the results, with the operators they would affect, their share of the model’s MACs, the measured gain and the line of YAML that sets them. Hints are facts only; they never change the conversion.

Vocabulary

• Setting: anything below that you set: the goal, the approximation gate, each knob, and constant placement.
• Knob: a per-operator mechanism with a closed set of values, plus auto.
• Exact and approximate: every knob value is classified. An approximate value changes numerics relative to the default value of its knob; auto picks one only when allow_approximate is true.
• Goal: what auto favors: latency, balanced, size or accuracy.
• Plan: the fully resolved choice of every knob for every operator, written to <prefix>_plan.json.
• Report: the facts behind the plan and its hints, written to <prefix>_report.json.
• auto: a static, documented table in heliaAOT that resolves a knob from the goal. It never measures or searches; searching for the best plan on a target happens outside the compiler.

The settings

Setting	Set with	Default	Values	Measured	Accuracy cost
Goal	optimization.goal	balanced	latency, balanced, size, accuracy	-	-
Approximation gate	optimization.allow_approximate	false	false, true	-	-
Accumulation	optimization.accumulation, operators[].optimization.accumulation	auto	auto, precise (the default accumulation), fast (approximate)	FP16 fully connected under fast: not yet measured (AmbiqAI/helia-aot#583). FP16 depthwise under fast: not yet measured. FP16 convolutions pack at every setting: 3.22x on an FP16 pointwise layer (1x256, 32 to 32 channels), Apollo510	FP16 fully connected under fast: not yet measured. FP16 convolutions, packed at every setting: max |error| 0.0156 against 0.0059 standard on that layer
Kernel	optimization.kernel, operators[].optimization.kernel	auto	auto, specialized (shape-specialized direct entries where they apply), generic (the general entries)	ns-cmsis-nn 7.37.0 kernel-level, Apollo510: 1.9-2.4x on first-layer convolutions with 1-3 input channels, 1.23x on 3x3 convolutions with 16 channels, 1.7-2.0x on 3x3 depthwise layers. ns-cmsis-nn 7.39.3 kernel-level, Apollo510, ATfE: 1.3-2.1x on 1x1 layers over 8 to 16 input channels	Exact: identical output under ns-cmsis-nn’s default rounding
Constant placement	memory.tensors[].attributes.constant_destination_memory	unset: constants are read where the planner places them	a memory, e.g. DTCM	DTCM instead of SRAM, whole model on Apollo510 at 96 MHz: AD 2.70x, sleep FP16 1.28x, sleep FP32 1.22x; KWS, VWW, IC and TCN within 1%	Exact; costs a DTCM copy of the weights and an init-time copy

The FP16 pointwise and constant placement measurements used heliaAOT 0.23.0 and ns-cmsis-nn 7.36.0 on an Apollo510 EVB (Cortex-M55). Constant placement is a record in heliaAOT’s measurement data, with its target, CPU, versions and source; accumulation has no record until FP16 fully connected is measured, so its hint states the facts without a measured gain. The report carries only the records measured on the conversion’s target and CPU; on any other target, even one with the same CPU, the hints state the facts without a measured gain. The kernel figures are ns-cmsis-nn’s own kernel-level measurements, not heliaAOT conversions, so they are not in the measurement data and no report quotes them yet.

optimization.budget and optimization.plan are reserved for an optimizer that searches plans on a target and for reproducing a searched plan; setting either is an error today. Further knobs (weight layout, activation LUTs, code specialization, placement) will join the same section.

How auto resolves

Knob	latency with allow_approximate: true	latency	balanced	size	accuracy
accumulation	fast	precise	precise	precise	precise
kernel	specialized	specialized	specialized	specialized for depthwise; generic otherwise	specialized

auto never picks an approximate value unless allow_approximate is true. A value you set explicitly, for the model or an operator, is used as given: setting accumulation: fast is your consent to its numerics, and the plan records it as approximate.

Accumulation

accumulation applies to FP16 fully connected and FP16 depthwise layers.

FP16 depthwise layers call arm_depthwise_conv_f16 under precise. From ns-cmsis-nn 7.47.0, the minimum, that entry accumulates every tap in FP32 from the bias and rounds once on MVE, as it already did without MVE; earlier releases summed in FP16 lanes on MVE and folded into FP32 only past 32 taps. fast calls arm_depthwise_conv_f16_acc16, which keeps FP16 lanes on MVE: on 3x3 and 5x5 windows it gives what the default entry gave before 7.47.0. Its latency on target is not yet measured, so no FP16 depthwise gain is assumed. Without MVE both entries accumulate in FP32, so fast changes nothing there and the report marks the knob not applicable. Both entries take the same scratch.

For FP16 fully connected layers, eligible packed layers under precise call arm_nn_mat_mult_nt_n_packed_f16_precise: each output uses four-tap FP16 partial sums folded into FP32, with bias added in FP32 before the final FP16 rounding and clamp. This changes summation order, so it does not promise bit equality or uniformly smaller error, and FP16 partial overflow remains possible.

Multirow packing needs constant weights, MVE FP16, at least four rows and eight outputs. Batch1 precise FC packs unshared constants outside TCM only when the output count is a multiple of eight, preserving the allocated byte size. Other layers keep standard weights, whose eight vector lanes split each output’s reduction. Batch1 fast keeps standard weights and this eight-lane reduction. When the multirow rule selects packed weights, fast uses the legacy packed entry, folding a chain of up to 32 FP16 taps into FP32. No FC latency improvement has been established for the new precise route. The report records the selected entry and weight layout; it marks accumulation inapplicable where changing it has no effect.

FP16 1x1 convolutions and conv1d (a 1 x k filter over one input row) pack wherever packing pays off, at every accumulation setting, and the report marks the knob not applicable on them; an accumulation setting on a convolution has no effect. An operators rule that sets fast, or precise on a layer that packs, warns, naming the layers and their weight layout. Because both layouts fold into FP32, packing changes their FP16 rounding only: the gap is largest for short reductions (on an FP16 pointwise layer with 32 input channels the worst error against the reference measured 0.0156 packed and 0.0059 standard; on a conv1d with K = 224 the two were equal, ns-cmsis-nn 7.38.1, Cortex-M55 emulation), and across seven FP16 models with pointwise layers, every packed worst output error was at most 0.02 or equal to the standard layout’s. On Apollo510, packed FP16 weights ran 3.22x faster on a pointwise layer (1x256, 32 to 32 channels), and packed FP16 conv1d ran 33% (1xN) and 17% (dilated) faster than standard on a sleep model. See packed float weights for the layout and the accuracy analysis. FP32 weights pack where it pays off regardless of this knob (1x1, fully connected and dilated conv1d), and keep FP32 accumulation. FP16 depthwise layers follow the knob as Accumulation describes (see also depthwise accumulation).

Let auto choose for latency:

optimization:
  goal: latency
  allow_approximate: true

Or set the knob for the whole model, and override it per operator type or id:

optimization:
  accumulation: fast
operators:
  - type: FULLY_CONNECTED
    id: "12"
    optimization:
      accumulation: precise

accumulation replaces the fp16_accuracy operator attribute of an earlier development build; that name was never released, so it has no alias.

Kernel

kernel applies to int8 convolutions, depthwise convolutions and fully connected layers on MVE targets, and int8 bilinear resize on all targets. With specialized, a layer calls the ns-cmsis-nn direct entry whose shape rule it meets:

• arm_resize_bilinear_s8_pow2 for half-pixel bilinear resize with integer power-of-two scales on both spatial axes and a scale product at most 2²¹. The general choice calls arm_resize_bilinear_s8. Both are scratch-free and retain exact rational coordinates with one final signed rounding step; MVE builds vectorize the specialized entry. INT8 bilinear resize requires a CORE build defining ARM_NN_HAS_RESIZE_BILINEAR_S8; older builds stop at compilation;
• arm_convolve_1x1_s8_short_k for 1x1 convolutions with stride 1 over 1 to 16 input channels and at least 8 pixels;
• arm_convolve_s8_small_cin for 1 to 3 input channels;
• arm_convolve_s8_3x3_c16_s1 for 3x3 kernels over 16 channels with stride 1;
• arm_depthwise_conv_s8_opt_3x3_c64_s1 and arm_depthwise_conv_s8_opt_3x3 for 3x3 depthwise layers;
• arm_depthwise_conv_s8_opt_planar where the planar rule holds;
• arm_depthwise_conv_s8_opt_channelwise for every other layer the optimized depthwise kernel takes;
• arm_fully_connected_per_channel_packed_s8 for fully connected layers with constant weights, a weight zero point of 0 and right-shift-only requantization. One weight stream, packed at conversion, holds the weights, kernel sums and requantization parameters; it is ceil(units / 4) blocks, each four rows of the depth rounded up to 16 bytes plus 48 bytes of parameters, so layers with few units or a short depth store more than their weights. memory.tensors rules on the layer’s weights place the stream; a rule on the stream’s own id (<op>_packed) is not applied.

With generic, no layer calls a direct entry: a layer that would take one calls arm_convolve_1x1_s8_fast, arm_convolve_s8, arm_depthwise_conv_s8_opt, arm_fully_connected_s8 or arm_fully_connected_per_channel_s8 instead. The other shape-specific kernels (arm_convolve_1x1_s8_fast for 1x1 layers outside the short-K rule, arm_convolve_1x1_s8, 1xN, and arm_depthwise_conv_3x3_s8 on DSP targets) apply with either value. Both values give identical output with ns-cmsis-nn’s default rounding; a build that defines CMSIS_NN_USE_SINGLE_ROUNDING can saturate differently on a 1x1 layer at extreme requantization shifts. Each layer calls one entry directly, with no run-time dispatch.

generic is not always the smaller module. It avoids linking several direct entries next to the general ones, but arm_depthwise_conv_s8_opt carries both its planar and channel paths, and a 3x3 depthwise layer needs a larger scratch buffer with it. Code size with specialized against generic, linked for the Cortex-M55 with unused sections removed, measured for heliaAOT 0.25.0, before the 1x1 short-K entry, which adds about 1.9 KB to a model that also keeps arm_convolve_1x1_s8_fast:

• the MLPerf Tiny keyword-spotting model is 1928 B smaller, with 1120 B less arena;
• the visual wake words model is 6520 B larger;
• the image classification model is 2652 B larger.

goal: size picks specialized for depthwise layers and generic otherwise; measure your model if the difference matters. The plan records the entry each convolution and depthwise layer, and each fully connected layer that reads a packed stream, calls under kernel.entry, and the report repeats it under entry.

Constant placement

Constants are read-only, so their cold copy stays in non-volatile memory, set with memory. constant_destination_memory adds a runtime copy in a faster memory, which the generated <prefix>_model_init fills once. Staging weights into DTCM removes the SRAM weight-read stalls of weight-bound layers such as fully connected and conv1d; activation-bound models change little. DTCM is shared with your application’s stack and data, so the report states how much of it the module leaves free, within memory.constraints. See memory placement.

memory:
  tensors:
    - type: constant
      attributes: {memory: MRAM, constant_destination_memory: DTCM}

The plan and the report

<prefix>_plan.json sits next to the module and holds the choices. It carries a schema_version, the heliaAOT version and the ns-cmsis-nn version the module requires, the model it was made from (file name, sha256 and subgraph), the target (name, cpu, capabilities, FP16 support), the goal and approximation gate, and one entry per operator under operators. A target’s capabilities are what the release that wrote the plan knew of it; a later release can remove one from a target as well as add one.

Each operator has a stable id, "<subgraph>:<output tensor name>", taken from the name the model gives the operator’s first output tensor, so it stays the same when other operators are added or removed. Its position in execution order is kept in index, and its AIR node id in node_id. Repeated names get #1, #2 in operator order, and an operator without a name falls back to "<subgraph>:<op_type>@<position>"; id_source says which applied. A tensor name can itself contain : (for example 0:StatefulPartitionedCall_1:0), so a parser splits these ids on their first colon only. An id with id_source: node_id (see below) is the opaque AIR node id and is not split. operators[].id rules match the stable id as well as the node id, so a plan’s ids can be written back as rules. A stable id that would also select another operator (another operator’s node id, or a stable id spelled twice) is not matched; the plan then uses the node id as id, with id_source: node_id, and the conversion warns.

For each knob of an operator the plan records the value used, the value requested, resolved_by (explicit, or auto_table when auto chose it from the goal; auto_heuristic and plan are reserved), whether the value is approximate where it applies, and the source: kind is operator (with rule, the position, type and id of the winning operators entry), model (set under optimization, even to auto), or default (plan is reserved). For layers that choose among candidate ns-cmsis-nn entries (convolution, depthwise, transpose convolution, packed int8 fully connected and precise packed FP16 fully connected), kernel.entry names the one chosen. Average and max pooling also record their selected int8, int16, FP16 or FP32 entry. Pooling, convolution, depthwise, transpose convolution, fully connected, batch matmul and SVDF use kernel.provider to identify the built-in CORE planning rules. Their single-entry paths are recorded explicitly, including FP16/FP32 and ordinary fully connected entries. The provider field is optional and absent from older plans. It is an opaque string: compare it with another plan’s, but do not parse it. Other unmigrated single-entry layers still have a null kernel entry. Selection follows the rules and knobs above; a plan cannot set it yet. Under fixed, it records choices that change numerics or speed but are not knobs yet, with what set them (attribute, default, heuristic, knob or placement; plan is reserved): RSQRT lut_mode (INT16 only), HARD_SWISH use_lut and compat_variant (INT8), and the float weight layout of 1x1 convolutions, conv1d and fully connected layers (weight_layout: set by the knob for fast packed FP16 fully connected, by placement for eligible batch-one FP32 or precise FP16 fully connected, and by a heuristic otherwise, including precise multirow FC and FP16 convolutions).

<prefix>_report.json explains the plan and does not change it: each operator’s MACs, the alternatives of each knob and whether they apply on this target (and why not), the measurements for this target and CPU keyed by setting, the constant placement facts, and the hints the results print. Its content may change between releases; the choices are in the plan.

Both files carry a schema_version. Adding an optional field, a knob, a knob value or an enum member keeps it; renaming or removing a field, or changing what one means, increments it. OptimizationPlan and OptimizationReport in helia_aot.optimization read both files back as typed models by that rule: they ignore a field they do not know, keep a knob, knob value or member they do not know as the string written, and reject only a file with another schema_version. A script that uses them reads the files of older and newer releases with the same schema_version. Plans and reports from heliaAOT 0.30 and earlier name the target’s CPU core; they still read, as cpu.

The JSON schemas are exported with the documentation data (plan-schema.json, report-schema.json). Under its json_schema key, each holds a schema that lists every field, knob, value and member the release that exported it writes, and allows no other field (additionalProperties: false) except in two free-form objects: each entry of the plan’s reserved tensors list and a report hint’s facts. It belongs to the heliaAOT release that wrote a file (generator.helia_aot): validate a file against that release’s schema’s json_schema object rather than pinning one. A newer file need not validate against an older schema. Some values and fields are reserved and empty today: plan as a source, optimization.knobs for model-level choices and tensors for per-tensor choices. Compatibility says what a release promises about each file.

Knobs are set only under optimization, for the model or in an operators rule; a knob name under attributes is rejected. An operators rule naming a type or an id that sets a knob where it has no effect warns, naming the operators and the reason; a rule asking for the knob’s default value warns only where the operator does not run as that value describes. optimization.budget and optimization.plan are YAML-only while they are reserved.

A hint appears only when its alternative applies and a single setting would select it. A knob hint lists the operators where the knob applies that do not run the value auto picks for goal: latency with allow_approximate: true (for accumulation, operators running precise where the fast alternative applies), leaving out operators an operators rule pins. The placement hint needs constants outside TCM that all come from one cold memory and fit in the free DTCM. Share of MACs counts convolution, depthwise and fully connected layers; it is not a share of run time unless you profile the model.

The depthwise size policy retains applicable direct entries instead of linking the optimized dispatcher and its alternative implementations. This is a per-operator policy, not a guarantee of the smallest whole-model binary. Explicit kernel: generic or kernel: specialized still overrides auto.

For operators with a recorded kernel entry, kernel.local_scratch_bytes in the plan and the report’s local-scratch column read the actual resolved local tensors. They exclude output tensors, constant weight sums and arena alignment padding; they are not the model’s total scratch-arena requirement. Older plans can omit this field, and operators without a recorded entry remain explicitly unrecorded.

Generated CORE calls

The optimization report’s kernel_calls records the CORE functions every operator’s generated source calls, including batch matmul’s operand transposes, SVDF’s kernel-sum call for runtime feature weights and the shared ADD/MUL helpers. An operator that calls no CORE function, such as DILATE or an Ethos-U command stream, records none. A call compiled only under ARM_MATH_MVEI, such as REVERSE_V2’s row copy, is recorded on every target. Each calls entry contains a symbol and operator_ids matching the plan’s stable operator IDs. Symbols are sorted; attribution follows execution order. Initialization calls are included when emitted; offline precomputation does not count as a generated call.

These records use the same selected symbols as code generation. They list direct generated calls, including calls inside shared generated helpers, not the functions those CORE entries call internally or the final linked image. unrecorded_operator_ids explicitly lists operators without call coverage; a partial list must not be interpreted as the module’s complete dependencies. Older reports have no kernel_calls field (read as null), which differs from an empty, covered call list. The generated HTML report displays the same records and coverage gaps.

This records existing decisions without changing selection, scratch sizing, or numerical behavior. It does not supply an upstream kernel manifest or infer requantization modes from multipliers and shifts.