Precision

heliaAOT compiles the tensor types and quantization parameters already present in the LiteRT file. Training, calibration and precision changes happen upstream. This does not promise bit-identical interpreter output: validate the generated module with known inputs and a predetermined numerical acceptance criterion.

Check three things before choosing a path: the model tensor types, operator coverage and the target/library build requirements. Precision affects memory use and kernel selection, but the complete model must be measured to compare size or speed.

For quantized compute paths, A8W8 means 8-bit activations and 8-bit weights; A16W8 means 16-bit activations and 8-bit weights. These labels describe a combination of tensor roles. The int8 and int16 types below identify individual tensors; biases, accumulators and state can use other types.

The four paths

Path	What the model carries	Where it runs	Reach for it when
int8	Quantized data tensors; weights and metadata have their own constraints	Every registered target, where each operator accepts the model	Start with the supplied KWS example or an already validated quantized model
int16	16-bit activations, for the operators that declare an int16 kernel	Every registered target, per operator	int8 calibration will not hold the dynamic range
FP16	Half-precision data tensors; index, mask and other metadata retain their required types	Native compute needs `supports_fp16`; storage/copy paths can differ	The model needs half-precision computation and each operator/target supports it
FP32	`float32` data tensors, with separate index, mask and state requirements	Targets with a hardware floating-point unit	You need reference numerics, or you are bringing a model up before quantizing it

Wider tensors increase storage per element. Total flash, arena use and cycles also depend on the model, selected kernels and target; precision alone does not predict their ordering. Check each operator’s tensor and target requirements.

int8 is the default path

The first-deployment example uses int8, and every registered platform has integer kernel paths. Check the operator catalog for the particular graph. An int8 element takes fewer bytes than an int16 or float32 element, but total constants, scratch and code also depend on shapes, operator choices and workspace requirements. Compare the generated memory reports rather than predicting whole-module size from precision alone.

The compiler reads quantization parameters from the flatbuffer and embeds them in generated code. Kernels can still rescale intermediate values during inference. Calibration happens upstream; check the exported scales and zero points when investigating a numerical mismatch.

Two int8 details show up in the catalog’s Restrictions column often enough to be worth knowing before you export:

Several operators require a specific zero point. LOGISTIC wants an int8 output at zero-point -128, for example, because the kernel assumes it.
Weights on the recurrent path have to be symmetric, which means zero-point 0.

Check the exported tensor parameters rather than assuming the exporter selected the required form. Conversion validates operator-specific restrictions.

int8 GELU runs as a 256-entry lookup table built at conversion time. The table follows TFLite’s float32 formula for both the exact and the tanh-approximate form, with correctly rounded erfcf and tanhf. LiteRT builds the same table with its host’s math library, which is not always correctly rounded. A LiteRT build can therefore differ from the generated module by one code at an input where its erfcf or tanhf result is one unit in the last place off.

Requantization rounds differently too. Generated modules requantize int8 accumulators with heliaCORE’s requantization, arm_nn_requantize or, on MVE builds, arm_requantize_mve, which both round twice, as gemmlowp does, unless ns-cmsis-nn is built with CMSIS_NN_USE_SINGLE_ROUNDING. LiteRT’s reference FULLY_CONNECTED rounds once, its reference DEPTHWISE_CONV_2D rounds twice like heliaCORE, and its default resolver, which applies XNNPACK and produces the automatic golden data, rounds half to even. Where the rules disagree, a requantization differs by one code, and differences can compound across layers: an accumulator of 3 at a multiplier of 0.125 gives 1 in the module and 0 in LiteRT’s reference fully connected kernel, and on one int8 average pool, depthwise and fully connected chain the module differed from LiteRT’s reference kernels at two of eight outputs, by one code, and from its default resolver at six, by up to two codes. Allow for this when you choose a tolerance.

int16 where the operator has an int16 kernel

int16 widens the activations while keeping integer arithmetic, which is the usual answer when a model has a layer whose range int8 cannot hold. It is not a whole-model switch that heliaAOT applies: it is a property of the tensors in the model you hand it. Check whether each operator accepts int16 data tensors; that may select a kernel, inline code or a copy route.

The Data types column of the operator catalog is that list. Check every operator your model uses before you export an int16 version of it, and read the Restrictions column in the same row: int16 brings its own conditions, such as operators that accept int16 only with a zero point of 0, and operators whose output type is constrained by the input type.

The catalog summarizes data types; index, mask and recurrent state tensors have separate roles. In particular, an int16 LSTM cell state does not establish an int16-input/output LSTM path.

Where the decision is enforced

Operators validate types and other conditions per node, using declarations or per-instance checks. A dtype match alone does not establish support for its shape, quantization, attributes or target. Mixed precision also requires valid conversion boundaries: DEQUANTIZE accepts int8, int16 or float16 input and produces float32; QUANTIZE accepts float32 to int8/int16, requantizes within the same integer width, or requantizes int8 to uint8 and uint8 to int8. All generated modules require the module-wide CORE minimum stated in the installation guide. These catalog types are not interchangeable input/output pairs.

A declared float API or C type dependency requires the corresponding library switch below. Operators using native FP16 kernels also check the platform’s FP16 capability. Byte copies and storage conversions have different needs: EXPAND_DIMS copies bytes, while FP16-to-FP32 DEQUANTIZE runs on every target: it calls ns-cmsis-nn’s arm_nn_dequantize_f16_bits_f32, which converts with MVE, the FPU or integer code, whichever the build has. Finite values and infinities convert exactly; a NaN comes back as a NaN, and its sign and payload depend on the path and build, as ns-cmsis-nn documents for arm_dequantize_f16_bits_f32.

How heliaAOT works describes the Resolve stage these checks belong to, and Troubleshooting lists the error classes they raise.

Float support

The float16 and float32 dtypes in the operator catalog are part of the support matrix, like the integer ones. As on every path, the kernel a layer calls and its numerical results are not promised.

heliaAOT can compile a model whose tensors are float16 or float32 and emit float calls to heliaCORE, the Ambiq fork of CMSIS-NN, instead of quantized paths where supported. Some operations use inline arithmetic, lookup loops, byte copies or alias no-ops instead. A library call can itself select a vector, DSP or scalar implementation depending on the shape, target and build flags; float support is not a count of optimized kernels.

float32 GELU calls heliaCORE’s exact arm_nn_gelu_f32. For finite inputs each result is within 2⁻²¹·|x·Φ(x)| + 2⁻²⁴ of x·Φ(x); the exact bits depend on the target’s math library. float16 GELU calls arm_nn_gelu_f16, which needs an FP16-capable (Cortex-M55-class) target. It evaluates in float32, with the same math-library dependence, and rounds once, so with FZ16 clear each finite result is within 2⁻¹⁰·|x·Φ(x)| + 2⁻²⁴ of x·Φ(x); with FZ16 set, a result whose |x·Φ(x)| is below 2⁻¹⁴ may instead be a zero of the same sign. The tanh-approximate form has no float kernel, so conversion rejects it in float16 and float32. Re-export with approximate=false, or quantize GELU to int8, which supports both forms.

What the hardware has to provide

Format	Required core feature	Where to check
FP32	Hardware single-precision FPU	Targets
FP16	Armv8.1-M MVE-FP, half-precision vector floating point	The `FP16` capability in the Targets reference

The platform gate

Emitting _f16 kernels for a core without MVE-FP would either fail to build or misbehave silently, so heliaAOT refuses at conversion time. Operators that call the shared native-FP16 support check reject a float16 tensor when the platform’s supports_fp16 check fails, naming the platform and core. This gate does not apply universally to FP16 storage or byte movement. The platform’s supports_fp16 property accepts a compatible explicit FP16 capability and also infers support from a normalized cortex-m55 CPU when that capability is absent. Omitting that capability from a custom Cortex-M55 definition therefore does not disable every native half-precision path. Extra capability fields on registered target names are ignored; use the custom-target mechanism when changing hardware declarations.

FP32 has no such gate: any target with a floating-point unit can run it.

What is and is not a float model

LiteRT’s built-in FP16 conversion is a storage optimization. It keeps weight constants as FP16 but inserts a dequantize in front of each layer and leaves activations and the model’s inputs and outputs in FP32.

A model produced that way does not exercise the FP16 kernels. heliaAOT sees the dequantize followed by an FP32 kernel, which is exactly what it emits. For weightless operators there is nothing to convert at all, so they arrive as plain FP32.

To exercise native FP16 computation, the relevant operator’s data inputs, weights, biases and outputs must meet its FP16 requirements. Metadata/index tensors can retain other types, and a mixed graph can have valid conversion boundaries. Inspect the actual tensor roles and emitted path rather than classifying the entire model from its filename.

Packed float weights

Float 1x1 convolutions (arm_convolve_1x1_f16/f32), float fully connected layers, FP16 conv1d (arm_convolve_1_x_n_f16, and arm_convolve_f16 for dilated filters and for the unpadded unit-stride width-3 and width-5 filters it runs on dedicated kernels) and dilated FP32 conv1d (arm_convolve_f32) can store their weights in ns-cmsis-nn’s NT_N_PACKED layout, [ceil(C_OUT / L)][K][L] with K the filter length times the input channels and L = 8 for FP16 and 4 for FP32, so each output channel is a vector lane. Undilated FP32 conv1d keeps the standard layout, where packing did not pay off. On Apollo510 a pointwise layer (width 256, 32 channels) measured 3.22x faster at FP16 and 2.19x at FP32 packed.

When it applies. FP32 and FP16 convolutions (1x1 and conv1d) pack by default. Eligible FP16 fully connected packs under precise using the four-tap folding direct entry; fast retains the legacy entry (see Performance and accuracy options). Planning-time packing applies only on an MVE target, with at least 4 rows sharing them (output positions, or batch rows for fully connected) and at least one full lane group of outputs. A packed FP16 1xN layer gets 8 rows of scratch, so the kernel can tile several output rows at once. A batch-1 FP32 or precise FP16 fully connected layer takes its layout from where the plan puts its weights: packed outside TCM and standard from DTCM or ITCM. FP32 packed streams measured about twice as fast from SRAM and MRAM; the precise FP16 route has no timing qualification. PSRAM and DRAM were not measured. A staged constant counts where the kernel reads it, its runtime memory. That applies when its output count is a multiple of 4 for FP32 or 8 for precise FP16, so both layouts have the same size; otherwise a batch1 layer keeps standard weights. Fast FP16 batch1 behavior is unchanged.
Accuracy. On the standard layout the 8 vector lanes split each output’s reduction of K products, so each lane sums about K/8 of them in FP16. On the packed layout a lane is an output, so it sums all K in one FP16 chain. On the legacy packed and standard paths, every ns-cmsis-nn release a module builds against folds each lane’s FP16 sum into FP32 every 32 taps (ARM_NN_F16_ACC_BLOCK) on both layouts (see Accumulation), so the gap is largest for short reductions: on an FP16 pointwise layer with 32 input channels the worst error against the reference measured 0.0156 packed and 0.0059 standard, and on a conv1d with K = 224 they were equal (Cortex-M55 emulation). Across seven FP16 models with pointwise layers, packing the convolutions changed each model’s worst output error by FP16 rounding only: every packed worst error was at most 0.02 or equal to the standard layout’s, so FP16 convolutions pack at every accumulation setting. Precise packed FP16 fully connected uses four-tap partials and FP32 bias through arm_nn_mat_mult_nt_n_packed_f16_precise. Legacy fast FC and convolution accumulation are unchanged; no universal numerical or FC speed improvement is claimed. auto resolves to precise unless you allow approximate values. FP32 keeps FP32 accumulation on both layouts.
Storage. Output channels are padded to a multiple of L, so a packed layer stores up to 7 (FP16) or 3 (FP32) padding rows of weights.
Tensor ids and placement. Weights that only one operator reads are packed in place and keep their tensor id. Weights shared with other operators get one packed copy, <id>_packed (or <id>_packed_N when that id is taken); the original stays for any reader that does not pack and is dropped when none is left. memory.tensors rules name the original id and also place the copy; a rule on the _packed id itself is not applied. Weights that are a model input or a variable are written at run time and keep the standard layout.

Depthwise accumulation

FP16 DEPTHWISE_CONV_2D follows accumulation. Under precise, the default, it calls arm_depthwise_conv_f16. From CORE 7.47.0, the minimum, that entry accumulates every tap in FP32 on MVE targets; earlier releases folded an output’s FP16 sum into FP32 only when its window had more than 32 taps (ARM_NN_F16_ACC_BLOCK), so a 3x3 or 5x5 layer accumulated in FP16 throughout. Under fast, FP16 depthwise layers call arm_depthwise_conv_f16_acc16, which keeps FP16 lanes on MVE (see Accumulation). Allow for FP16 lanes when you choose a tolerance.

The library build switch

Start with the generated module’s dependency requirements and enable the needed precisions on the kernel library before adding the generated module. A compiler flag on application code alone cannot add missing library objects.

The float kernel objects exist only if ns-cmsis-nn itself was built with them, so the switch belongs on the library, not on the generated module.

Build system	Switch
CMake and NSX	`ARM_NN_ENABLE_F32`, `ARM_NN_ENABLE_F16`, set before adding the ns-cmsis-nn subdirectory
Zephyr	`CONFIG_NS_CMSIS_NN_ENABLE_F32`, `CONFIG_NS_CMSIS_NN_ENABLE_F16` in the application’s `prj.conf`

On the standalone CMake and NSX backends the module does not define these itself. It inherits them from the kernel library’s usage interface and asks the library what it was built with, through a capability query that every supported ns-cmsis-nn release (7.47.0 and later) provides. A linked kernel target without that query fails configure with a hint to load the library’s normal CMake entry point. The library’s answer overrides a conflicting cache variable.

Only the precisions the operators actually declare are required. An FP16-only module can have F16 on and F32 off; an FP32-only module does not need F16. A module with no float kernel or float type dependency at all - an FP32 reshape, a metadata-only shape query - needs neither. Half-precision utilities that bind the half-precision type still require F16 even when they only move bytes.

On standalone CMake a parent may bring its own kernel target instead. The module then links nothing and defines only the switches its operators require. Set a required switch on, or leave it unset to take that default; setting a required switch off fails configure.

Validating against LiteRT

Automatic golden data comes from LiteRT on the conversion host. Known differences include the int8 rounding and GELU differences above and the following. The repository’s end-to-end cases (tests/e2e/cases/) record each measured difference in a comment beside the case’s tolerance. Where, at some input, the kernel is more than half a code further from exact math than LiteRT, the case also names the issue that accepts it under test.kernel_error_allowance.

int8 ADD, SUB, MUL and LEAKY_RELU. On the end-to-end cases the module matches LiteRT’s builtin kernels at every input. LiteRT’s default resolver rounds some outputs one code differently (for MUL, on x86_64 hosts), so those cases allow one code.
int8 UNIDIRECTIONAL_SEQUENCE_LSTM. An int8 LSTM can differ from LiteRT’s reference kernels by one code: on 27 of 720 output elements over 30 random inputs to one TF-quantized model, and at some output on 9 of 14 calibrator-produced models. The difference is bounded at one code. It is not the rounding of the effective scales, and its cause in the gate arithmetic is not yet attributed. The end-to-end LSTM cases allow one code (#823).
int8 and int16 MEAN. The module folds the division by the element count into the requantization multiplier, where LiteRT requantizes and then divides. Both round twice, at different steps, so each is up to 0.75 code from exact math on a single layer and the two can differ by one code at an output. In a model the differences compound: in three HeartKit models with six squeeze-and-excite MEAN layers, the first divergence from LiteRT was a MEAN, and the model outputs differed from those of LiteRT’s reference kernels by 1.3% to 5% relative, and by up to 6.7% on one model against LiteRT’s default resolver.
int16 FULLY_CONNECTED. heliaCORE’s int16 fully connected kernels (arm_fully_connected_s16, and arm_nn_vec_mat_mult_t_per_ch_s16 for per-channel weights) requantize with the output multiplier reduced from Q31 to Q15 (REDUCE_MULTIPLIER); LiteRT’s int16 fully connected kernel keeps the full Q31 multiplier and rounds once. The module is therefore not bit-exact with LiteRT: a layer’s outputs differ by up to one code, on 187 of 1280 outputs of a single 64-to-32 layer. The difference grows through the layers after it, most through a LOGISTIC or TANH whose input scale is much larger than its output scale: at a ratio of about 25, one input code becomes about 25 output codes. Over int16 recurrent models the largest output differences from LiteRT measured 4 to 79 codes, at most about 0.25% of full scale, and each exceeded the default tolerance of 1. int16 CONV_2D and DEPTHWISE_CONV_2D are bit-exact with LiteRT, whose int16 convolutions reduce the multiplier the same way.
int16 SOFTMAX. The module builds the exponential and reciprocal lookup tables heliaCORE’s int16 softmax reads, as TFLM does; LiteRT computes its int16 softmax from tables of its own. Over 50 random full-range inputs to a single layer, the outputs differed from LiteRT’s by up to 9 codes at 10 classes and up to 63 codes at 64 classes (input scale 0.001 or more). The module is the closer of the two to exact softmax in every configuration measured: at 64 classes, 62 codes from exact against LiteRT’s 125.
int16 TANH and LOGISTIC. When the log2 of the input scale is within 0.001 of −12 (both operators) or −11 (TANH only), LiteRT switches to its gemmlowp kernel and the module uses shift-only scaling, computing as if the scale were exactly that power of two. Over every int16 input of the end-to-end models with such a scale, the outputs differ by up to 12 codes for TANH and 7 for LOGISTIC. Against exact math the module is off by up to 7.9 and 4.2 codes and LiteRT by up to 17.1 and 9.5 (#501).
int16 SQRT. The kernel approximates the square root in float32 and truncates, so where the exact result is within 0.01 of an integer the output can be one code from LiteRT’s (#567).
int8-state SVDF. LiteRT’s int8-state SVDF can be far from exact SVDF math: by up to 255 codes, and up to 234 on the current models of heliaAOT’s two int8-state end-to-end cases, with its default and reference kernels alike. Its int16-state SVDF matches exact math. Those two cases are therefore checked against exact SVDF math on the dequantized parameters, with a tolerance of 1, not against LiteRT. Supply golden data from a trusted reference for such a model.
Float REDUCE_MAX and REDUCE_MIN. These follow TFLM, and the operator catalog restriction says where LiteRT differs on NaN and infinities. Two further differences are in the bits only. Of equal values the module keeps the first, so the maximum of [+0, -0] is +0, where LiteRT’s default kernels keep the last, -0; a numeric comparison treats them as equal. On an x86_64 host LiteRT’s default resolver flushes subnormal values to zero, so the maximum of [1.4e-45, 0] is 0 there and 1.4e-45 in the module.

The measurements above are from LiteRT 2.1.6 and host builds of the module against heliaCORE 7.47.0. To check whether one of these explains a mismatch on your model:

Convert with LiteRT golden data and keep the reported maximum difference per output; do not raise test.tolerance yet.
Find the first operator whose output differs. Run LiteRT with experimental_preserve_all_tensors=True to read every intermediate tensor. On the module side, the context’s callback, which module.schedule: table (the default) calls as each operator finishes running, is where a host build can capture that operator’s output. Its op argument indexes ConversionResult.operators (with result_views set), whose source_indices name the model-file operator to compare against.
If that operator is one of those listed here, at its dtype, and its own difference is within the bound given for it, its rounding is a candidate explanation, not proof of the cause of every difference after it. Before blaming propagated rounding for the final mismatch, or accepting a larger tolerance, check the operators downstream: feed each one the module’s own input to it (LiteRT’s preserved tensors give you the same input on the LiteRT side), and confirm that each, given matching inputs, agrees with LiteRT within its own bound. A downstream operator that differs with matching inputs is a separate difference. For int16 FULLY_CONNECTED, SOFTMAX and MEAN the model-level difference can be much larger than the per-layer one, so set the tolerance from your application’s accuracy requirement against an exact or trusted float reference, not from LiteRT’s outputs alone.
If the first divergent operator is not listed, or exceeds its bound, report it with the model and configuration.

Checking a model before you convert

Coverage is per operator, per element type and per target, not per model. The operator catalog is the table: every operator heliaAOT lowers, its dtype summary, declared float API or type dependency, and the oldest library that float path builds against. The module-wide kernel library floor is on the Versions page.

The conversion itself checks whether each remaining node has an applicable lowering. Then build and validate the generated module with the intended library and target. Successful conversion alone does not establish optimized execution or numerical correctness. Use the output verification workflow with the intended library, compiler and execution environment. Keep the acceptance threshold fixed while investigating a mismatch; increasing it to fit observed errors is not validation.