Performance and accuracy options

heliaAOT’s defaults keep each knob’s default numerics; FP16 convolutions pack their weights by default, which changes only their FP16 rounding (see Accumulation). The optimization settings let a conversion trade accuracy or memory for speed where your model allows it.

Every conversion writes the choice it made for each operator to <prefix>_plan.json, explains it in <prefix>_report.json, and prints the alternatives that apply under Optimization hints in the results, with the operators they would affect, their share of the model’s MACs, the measured gain and the line of YAML that sets them. Hints are facts only; they never change the conversion. Plans and reports describes both files.

Vocabulary

Term	Meaning
Setting	Anything below that you set: the goal, the approximation gate, each knob, and constant placement.
Knob	A per-operator mechanism with a closed set of values, plus auto.
Exact and approximate	Every knob value is classified. An approximate value changes numerics relative to the default value of its knob; auto picks one only when allow_approximate is true.
Goal	What auto favors: latency, balanced, size or accuracy.
Plan	The fully resolved choice of every knob for every operator, written to <prefix>_plan.json.
Report	The facts behind the plan and its hints, written to <prefix>_report.json.
auto	A static, documented table in heliaAOT that resolves a knob from the goal. It never measures or searches; searching for the best plan on a target happens outside the compiler.

The settings

Setting	Set with	Default	Values	Measured	Accuracy cost
Goal	optimization.goal	balanced	latency, balanced, size, accuracy	-	-
Approximation gate	optimization.allow_approximate	false	false, true	-	-
Accumulation	optimization.accumulation, operators[].optimization.accumulation	auto	auto, precise (the default accumulation), fast (approximate)	FP16 fully connected under fast: not yet measured (AmbiqAI/helia-aot#583). FP16 depthwise under fast: not yet measured. FP16 convolutions pack at every setting: 3.22x on an FP16 pointwise layer (1x256, 32 to 32 channels), Apollo510	FP16 fully connected under fast: not yet measured. FP16 convolutions, packed at every setting: max |error| 0.0156 against 0.0059 standard on that layer
Kernel	optimization.kernel, operators[].optimization.kernel	auto	auto, specialized (shape-specialized direct entries where they apply), generic (the general entries)	ns-cmsis-nn 7.37.0 kernel-level, Apollo510: 1.9-2.4x on first-layer convolutions with 1-3 input channels, 1.23x on 3x3 convolutions with 16 channels, 1.7-2.0x on 3x3 depthwise layers. ns-cmsis-nn 7.39.3 kernel-level, Apollo510, ATfE: 1.3-2.1x on 1x1 layers over 8 to 16 input channels	Exact: identical output under ns-cmsis-nn’s default rounding
Constant placement	memory.tensors[].attributes.constant_destination_memory	unset: constants are read where the planner places them	a memory, e.g. DTCM	DTCM instead of SRAM, whole model on Apollo510 at 96 MHz: AD 2.70x, sleep FP16 1.28x, sleep FP32 1.22x; KWS, VWW, IC and TCN within 1%	Exact; costs a DTCM copy of the weights and an init-time copy

• The FP16 pointwise and constant placement measurements used heliaAOT 0.23.0 and ns-cmsis-nn 7.36.0 on an Apollo510 EVB (Cortex-M55). Constant placement is a record in heliaAOT’s measurement data, with its target, CPU, versions and source; accumulation has no record until FP16 fully connected is measured, so its hint states the facts without a measured gain.
• The report carries only the records measured on the conversion’s target and CPU; on any other target, even one with the same CPU, the hints state the facts without a measured gain. The kernel figures are ns-cmsis-nn’s own kernel-level measurements, not heliaAOT conversions, so they are not in the measurement data and no report quotes them yet.
• optimization.plan reads a plan as input (experimental): its schedule section is applied, so a field heliaAOT does not know there is rejected, and every other section must describe the conversion the configuration produces; see Plans and reports. optimization.budget is reserved for an optimizer that searches plans on a target; setting it is an error today. Further knobs (weight layout, activation LUTs, code specialization, placement) will join the same section.

How auto resolves

Knob	latency with allow_approximate: true	latency	balanced	size	accuracy
accumulation	fast	precise	precise	precise	precise
kernel	specialized	specialized	specialized	specialized for depthwise; generic otherwise	specialized

auto never picks an approximate value unless allow_approximate is true. A value you set explicitly, for the model or an operator, is used as given: setting accumulation: fast is your consent to its numerics, and the plan records it as approximate.

Let auto choose for latency:

optimization:
  goal: latency
  allow_approximate: true

Or set a knob for the whole model, and override it per operator type or id:

optimization:
  accumulation: fast
operators:
  - type: FULLY_CONNECTED
    id: "12"
    optimization:
      accumulation: precise

Knobs are set only under optimization, for the model or in an operators rule; a knob name under attributes is rejected. optimization.budget is YAML-only while it is reserved.

An operators rule naming a type or an id that sets a knob where it has no effect warns, naming the operators and the reason; a rule asking for the knob’s default value warns only where the operator does not run as that value describes.

Accumulation

accumulation applies to FP16 fully connected and FP16 depthwise layers.

Layer	precise	fast
FP16 depthwise	arm_depthwise_conv_f16. From ns-cmsis-nn 7.47.0, the minimum, it accumulates every tap in FP32 from the bias and rounds once on MVE, as it already did without MVE; earlier releases summed in FP16 lanes on MVE and folded into FP32 only past 32 taps.	arm_depthwise_conv_f16_acc16, which keeps FP16 lanes on MVE: on 3x3 and 5x5 windows it gives what the default entry gave before 7.47.0. Its latency on target is not measured, so no gain is assumed.
FP16 depthwise without MVE	FP32 accumulation. Both depthwise entries take the same scratch.	The same; the report marks the knob not applicable.
FP16 fully connected, eligible multirow layers (constant weights, MVE FP16, at least four rows and eight outputs)	Packed weights and arm_nn_mat_mult_nt_n_packed_f16_precise: each output uses four-tap FP16 partial sums folded into FP32, with the bias added in FP32 before the final FP16 rounding and clamp. This changes summation order, so it promises neither bit equality nor uniformly smaller error, and FP16 partial overflow remains possible.	The legacy packed entry, folding a chain of up to 32 FP16 taps into FP32.
FP16 fully connected, eligible batch-one layers	Packed unshared constants outside TCM when the output count is a multiple of eight, preserving the allocated byte size, on the precise entry.	Standard weights, whose eight vector lanes split each output’s reduction.
FP16 fully connected, other layers	Standard weights and the eight-lane reduction.	The same.

• No FC latency improvement has been established for the precise route.
• The report records the selected entry and weight layout, and marks accumulation inapplicable where changing it has no effect.
• FP16 1x1 convolutions and conv1d (a 1 x k filter over one input row) pack wherever packing pays off, at every accumulation setting, and the report marks the knob not applicable on them; an accumulation setting on a convolution has no effect.
• An operators rule that sets fast, or precise on a layer that packs, warns, naming the layers and their weight layout.
• Both layouts fold into FP32, so packing changes their FP16 rounding only, and the gap is largest for short reductions: on an FP16 pointwise layer with 32 input channels the worst error against the reference measured 0.0156 packed and 0.0059 standard; on a conv1d with K = 224 the two were equal (ns-cmsis-nn 7.38.1, Cortex-M55 emulation).
• Across seven FP16 models with pointwise layers, every packed worst output error was at most 0.02 or equal to the standard layout’s.
• On Apollo510, packed FP16 weights ran 3.22x faster on a pointwise layer (1x256, 32 to 32 channels), and packed FP16 conv1d ran 33% (1xN) and 17% (dilated) faster than standard on a sleep model. See packed float weights for the layout and the accuracy analysis.
• FP32 weights pack where it pays off regardless of this knob (1x1, fully connected and dilated conv1d), and keep FP32 accumulation. FP16 depthwise layers follow the knob as above; see also depthwise accumulation.
• accumulation replaces the fp16_accuracy operator attribute of an earlier development build; that name was never released, so it has no alias.

Kernel

kernel applies to int8 convolutions, depthwise convolutions and fully connected layers on MVE targets, and int8 bilinear resize on all targets. Each layer calls one entry directly, with no run-time dispatch. With specialized, a layer calls the ns-cmsis-nn direct entry whose shape rule it meets; with generic, it calls the general entry instead.

Layer	specialized entry and its rule	generic entry
Bilinear resize, int8	arm_resize_bilinear_s8_pow2 for half-pixel resize with integer power-of-two scales on both spatial axes and a scale product at most 2²¹. Both entries are scratch-free and keep exact rational coordinates with one final signed rounding step; MVE builds vectorize the specialized one. Needs a CORE build defining ARM_NN_HAS_RESIZE_BILINEAR_S8; older builds stop at compilation.	arm_resize_bilinear_s8
1x1 convolution, stride 1, 1 to 16 input channels, at least 8 pixels	arm_convolve_1x1_s8_short_k	arm_convolve_1x1_s8_fast
Convolution with 1 to 3 input channels (1x1 and 1xN layers keep their own entries)	arm_convolve_s8_small_cin	arm_convolve_s8
3x3 convolution over 16 channels, stride 1	arm_convolve_s8_3x3_c16_s1	arm_convolve_s8
3x3 depthwise	arm_depthwise_conv_s8_opt_3x3_c64_s1, arm_depthwise_conv_s8_opt_3x3	arm_depthwise_conv_s8_opt
Depthwise where the planar rule holds	arm_depthwise_conv_s8_opt_planar	arm_depthwise_conv_s8_opt
Every other depthwise the optimized kernel takes	arm_depthwise_conv_s8_opt_channelwise	arm_depthwise_conv_s8_opt
Fully connected with constant weights, a weight zero point of 0 and right-shift-only requantization	arm_fully_connected_per_channel_packed_s8. One weight stream, packed at conversion, holds the weights, kernel sums and requantization parameters: ceil(units / 4) blocks, each four rows of the depth rounded up to 16 bytes plus 48 bytes of parameters, so layers with few units or a short depth store more than their weights. memory.tensors rules on the layer’s weights place the stream; a rule on the stream’s own id (<op>_packed) is not applied.	arm_fully_connected_s8 or arm_fully_connected_per_channel_s8

• The other shape-specific kernels (arm_convolve_1x1_s8_fast for 1x1 layers outside the short-K rule, arm_convolve_1x1_s8, 1xN, and arm_depthwise_conv_3x3_s8 on DSP targets) apply with either value.
• Both values give identical output with ns-cmsis-nn’s default rounding; a build that defines CMSIS_NN_USE_SINGLE_ROUNDING can saturate differently on a 1x1 layer at extreme requantization shifts.
• generic is not always the smaller module. It avoids linking several direct entries next to the general ones, but arm_depthwise_conv_s8_opt carries both its planar and channel paths, and a 3x3 depthwise layer needs a larger scratch buffer with it.
• goal: size picks specialized for depthwise layers and generic otherwise; an explicit kernel: generic or kernel: specialized overrides auto. The depthwise size policy keeps applicable direct entries instead of linking the optimized dispatcher and its alternatives; it is a per-operator policy, not a guarantee of the smallest whole-model binary. Measure your model if the difference matters.

Code size with specialized against generic, linked for the Cortex-M55 with unused sections removed, measured for heliaAOT 0.25.0 before the 1x1 short-K entry, which adds about 1.9 KB to a model that also keeps arm_convolve_1x1_s8_fast:

Model	specialized against generic
MLPerf Tiny keyword spotting	1928 B smaller, 1120 B less arena
Visual wake words	6520 B larger
Image classification	2652 B larger

The plan records the entry each convolution and depthwise layer, and each fully connected layer that reads a packed stream, calls under kernel.entry, and the report repeats it under entry. kernel.candidates lists every entry whose rules match the layer on the target, so a plan diff shows which entries a release added or removed, and entry which one the knobs selected; see Plans and reports.

Constant placement

Constants are read-only, so their cold copy stays in non-volatile memory, set with memory. constant_destination_memory adds a runtime copy in a faster memory, which the generated <prefix>_model_init fills once.

memory:
  tensors:
    - type: constant
      attributes: {memory: MRAM, constant_destination_memory: DTCM}

Staging weights into DTCM removes the SRAM weight-read stalls of weight-bound layers such as fully connected and conv1d; activation-bound models change little. DTCM is shared with your application’s stack and data, so the report states how much of it the module leaves free, within memory.constraints. See memory placement.

Reading the result

Every knob’s value, what requested it and the entry each layer calls are in <prefix>_plan.json; the alternatives, measurements and hints are in <prefix>_report.json. Plans and reports describes both files, their schema versions, the generated CORE call records and the FP16 overflow thresholds.