Precision

heliaAOT compiles the tensor types and quantization parameters already present in the LiteRT file. Training, calibration and precision changes happen upstream. This does not promise bit-identical interpreter output: validate the generated module with known inputs and a predetermined numerical acceptance criterion.

Check three things before choosing a path: the model tensor types, operator coverage and the target/library build requirements. Precision affects memory use and kernel selection, but the complete model must be measured to compare size or speed.

For quantized compute paths, A8W8 means 8-bit activations and 8-bit weights; A16W8 means 16-bit activations and 8-bit weights. These labels describe a combination of tensor roles. The int8 and int16 types below identify individual tensors; biases, accumulators and state can use other types.

The four paths

Path	What the model carries	Where it runs	Reach for it when
int8	Quantized data tensors; weights and metadata have their own constraints	Every registered target, where each operator accepts the model	Start with the supplied KWS example or an already validated quantized model
int16	16-bit activations, for the operators that declare an int16 kernel	Every registered target, per operator	int8 calibration will not hold the dynamic range
FP16	Half-precision data tensors; index, mask and other metadata retain their required types	Native compute needs supports_fp16; storage/copy paths can differ	The model needs half-precision computation and each operator/target supports it
FP32	float32 data tensors, with separate index, mask and state requirements	Targets with a hardware floating-point unit	You need reference numerics, or you are bringing a model up before quantizing it

Wider tensors increase storage per element. Total flash, arena use and cycles also depend on the model, selected kernels and target; precision alone does not predict their ordering. Check each operator’s tensor and target requirements.

int8 is the default path

The first-deployment example uses int8, and every registered platform has integer kernel paths. Check the operator catalog for the particular graph.

• An int8 element takes fewer bytes than an int16 or float32 element, but total constants, scratch and code also depend on shapes, operator choices and workspace requirements. Compare the generated memory reports rather than predicting whole-module size from precision alone.
• The compiler reads quantization parameters from the flatbuffer and embeds them in generated code. Kernels can still rescale intermediate values during inference. Calibration happens upstream; check the exported scales and zero points when investigating a numerical mismatch.
• Several operators require a specific zero point: LOGISTIC wants an int8 output at zero point -128, because the kernel assumes it. Weights on the recurrent path have to be symmetric, which means zero point 0. Check the exported tensor parameters rather than assuming the exporter selected the required form; conversion validates operator-specific restrictions.

Two int8 details show up in the catalog’s Restrictions column often enough to be worth knowing before you export; the zero-point bullets above are theirs. Two int8 numerics also differ from LiteRT:

Operator	Module	LiteRT
int8 GELU	A 256-entry lookup table built at conversion from TFLite’s float32 formula, exact and tanh-approximate, with correctly rounded erfcf and tanhf.	The same table from the host’s math library, which is not always correctly rounded, so a LiteRT build can differ by one code at an input where its erfcf or tanhf is one unit in the last place off.
int8 requantization	heliaCORE’s arm_nn_requantize, or arm_requantize_mve on MVE builds, which round twice as gemmlowp does, unless ns-cmsis-nn is built with CMSIS_NN_USE_SINGLE_ROUNDING.	The reference FULLY_CONNECTED rounds once, the reference DEPTHWISE_CONV_2D rounds twice like heliaCORE, and the default resolver, which applies XNNPACK and produces the automatic golden data, rounds half to even.

Where the rounding rules disagree, a requantization differs by one code, and differences can compound across layers. Allow for this when you choose a tolerance.

• An accumulator of 3 at a multiplier of 0.125 gives 1 in the module and 0 in LiteRT’s reference fully connected kernel.
• On one int8 average pool, depthwise and fully connected chain the module differed from LiteRT’s reference kernels at two of eight outputs, by one code, and from its default resolver at six, by up to two codes.

int16 where the operator has an int16 kernel

int16 widens the activations while keeping integer arithmetic, which is the usual answer when a model has a layer whose range int8 cannot hold. It is not a whole-model switch that heliaAOT applies: it is a property of the tensors in the model you hand it.

• The Data types column of the operator catalog says which operators accept int16 data tensors; that may select a kernel, inline code or a copy route. Check every operator your model uses before you export an int16 version of it.
• Read the Restrictions column in the same row: int16 brings its own conditions, such as operators that accept int16 only with a zero point of 0, and operators whose output type is constrained by the input type.
• The catalog summarizes data types; index, mask and recurrent state tensors have separate roles. An int16 LSTM cell state does not establish an int16-input/output LSTM path.

Where the decision is enforced

Operators validate types and other conditions per node, using declarations or per-instance checks. A dtype match alone does not establish support for its shape, quantization, attributes or target.

• Mixed precision needs valid conversion boundaries: DEQUANTIZE accepts int8, int16 or float16 input and produces float32; QUANTIZE accepts float32 to int8 or int16, requantizes within the same integer width, or requantizes int8 to uint8 and uint8 to int8. These catalog types are not interchangeable input/output pairs.
• All generated modules require the module-wide CORE minimum stated in the installation guide.
• A declared float API or C type dependency requires the corresponding library switch below. Operators using native FP16 kernels also check the platform’s FP16 capability.
• Byte copies and storage conversions have different needs: EXPAND_DIMS copies bytes, while FP16-to-FP32 DEQUANTIZE runs on every target through ns-cmsis-nn’s arm_nn_dequantize_f16_bits_f32, which converts with MVE, the FPU or integer code, whichever the build has. Finite values and infinities convert exactly; a NaN comes back as a NaN, and its sign and payload depend on the path and build, as ns-cmsis-nn documents for arm_dequantize_f16_bits_f32.

How heliaAOT works describes the Resolve stage these checks belong to, and Troubleshooting lists the error classes they raise.

Float support

The float16 and float32 dtypes in the operator catalog are part of the support matrix, like the integer ones. As on every path, the kernel a layer calls and its numerical results are not promised.

heliaAOT can compile a model whose tensors are float16 or float32 and emit float calls to heliaCORE, the Ambiq fork of CMSIS-NN, instead of quantized paths where supported.

• Some operations use inline arithmetic, lookup loops, byte copies or alias no-ops instead.
• A library call can itself select a vector, DSP or scalar implementation depending on the shape, target and build flags; float support is not a count of optimized kernels.

Float GELU calls heliaCORE’s entries, exact in float32; the exact bits depend on the target’s math library.

Path	Entry	Bound for finite inputs
float32	arm_nn_gelu_f32	Within 2⁻²¹·|x·Φ(x)| + 2⁻²⁴ of x·Φ(x).
float16	arm_nn_gelu_f16, which needs an FP16-capable (Cortex-M55-class) target; evaluates in float32 and rounds once	So, with FZ16 clear, within 2⁻¹⁰·|x·Φ(x)| + 2⁻²⁴ of x·Φ(x); with FZ16 set, a result whose |x·Φ(x)| is below 2⁻¹⁴ may instead be a zero of the same sign.

The tanh-approximate form has no float kernel, so conversion rejects it in float16 and float32. Re-export with approximate=false, or quantize GELU to int8, which supports both forms.

What the hardware has to provide

Format	Required core feature	Where to check
FP32	Hardware single-precision FPU	Targets
FP16	Armv8.1-M MVE-FP, half-precision vector floating point	The FP16 capability in the Targets reference

The platform gate

Emitting _f16 kernels for a core without MVE-FP would either fail to build or misbehave silently, so heliaAOT refuses at conversion time: operators that call the shared native-FP16 support check reject a float16 tensor when the platform’s supports_fp16 check fails, naming the platform and core.

• The gate does not apply universally to FP16 storage or byte movement.
• supports_fp16 accepts a compatible explicit FP16 capability and also infers support from a normalized cortex-m55 CPU when that capability is absent, so omitting it from a custom Cortex-M55 definition does not disable every native half-precision path.
• Extra capability fields on registered target names are ignored; use the custom-target mechanism when changing hardware declarations.

FP32 has no such gate: any target with a floating-point unit can run it.

What is and is not a float model

LiteRT’s built-in FP16 conversion is a storage optimization. It keeps weight constants as FP16 but inserts a dequantize in front of each layer and leaves activations and the model’s inputs and outputs in FP32.

A model produced that way does not exercise the FP16 kernels. heliaAOT sees the dequantize followed by an FP32 kernel, which is exactly what it emits. For weightless operators there is nothing to convert at all, so they arrive as plain FP32.

To exercise native FP16 computation, the relevant operator’s data inputs, weights, biases and outputs must meet its FP16 requirements. Metadata/index tensors can retain other types, and a mixed graph can have valid conversion boundaries. Inspect the actual tensor roles and emitted path rather than classifying the entire model from its filename.

Note

A CMSIS-NN kernel is typed by a single dtype suffix. There is no kernel that mixes FP16 weights with FP32 activations; that combination exists only as the LiteRT storage trick above and has to be resolved before a kernel runs.

Packed float weights

Some float layers can store their weights in ns-cmsis-nn’s NT_N_PACKED layout, [ceil(C_OUT / L)][K][L] with K the filter length times the input channels and L = 8 for FP16 and 4 for FP32, so each output channel is a vector lane. On Apollo510 a pointwise layer (width 256, 32 channels) measured 3.22x faster at FP16 and 2.19x at FP32 packed.

• Float 1x1 convolutions (arm_convolve_1x1_f16/f32) and float fully connected layers.
• FP16 conv1d: arm_convolve_1_x_n_f16, and arm_convolve_f16 for dilated filters and for the unpadded unit-stride width-3 and width-5 filters it runs on dedicated kernels.
• Dilated FP32 conv1d (arm_convolve_f32). Undilated FP32 conv1d keeps the standard layout, where packing did not pay off.

Aspect	Rule
When it applies	FP32 and FP16 convolutions (1x1 and conv1d) pack by default. Eligible FP16 fully connected packs under precise using the four-tap folding direct entry; fast retains the legacy entry (see Performance and accuracy options). Planning-time packing applies only on an MVE target, with at least 4 rows sharing the weights (output positions, or batch rows for fully connected) and at least one full lane group of outputs. A packed FP16 1xN layer gets 8 rows of scratch, so the kernel can tile several output rows at once.
Batch-one fully connected	A batch-1 FP32 or precise FP16 fully connected layer takes its layout from where the plan puts its weights: packed outside TCM and standard from DTCM or ITCM, when its output count is a multiple of 4 for FP32 or 8 for precise FP16, so both layouts have the same size; otherwise it keeps standard weights. A staged constant counts where the kernel reads it, its runtime memory. FP32 packed streams measured about twice as fast from SRAM and MRAM; the precise FP16 route has no timing qualification; PSRAM and DRAM were not measured. Fast FP16 batch-1 behavior is unchanged.
Accuracy	On the standard layout the 8 vector lanes split each output’s reduction of K products, so each lane sums about K/8 of them in FP16; on the packed layout a lane is an output, so it sums all K in one FP16 chain. On the legacy packed and standard paths every ns-cmsis-nn release a module builds against folds each lane’s FP16 sum into FP32 every 32 taps (ARM_NN_F16_ACC_BLOCK) on both layouts (see Accumulation), so the gap is largest for short reductions: on an FP16 pointwise layer with 32 input channels the worst error against the reference measured 0.0156 packed and 0.0059 standard, and on a conv1d with K = 224 they were equal (Cortex-M55 emulation). Across seven FP16 models with pointwise layers, packing the convolutions changed each model’s worst output error by FP16 rounding only: every packed worst error was at most 0.02 or equal to the standard layout’s, so FP16 convolutions pack at every accumulation setting. Precise packed FP16 fully connected uses four-tap partials and FP32 bias through arm_nn_mat_mult_nt_n_packed_f16_precise; legacy fast FC and convolution accumulation are unchanged, and no universal numerical or FC speed improvement is claimed. auto resolves to precise unless you allow approximate values. FP32 keeps FP32 accumulation on both layouts.
Storage	Output channels are padded to a multiple of L, so a packed layer stores up to 7 (FP16) or 3 (FP32) padding rows of weights.
Tensor ids and placement	Weights that only one operator reads are packed in place and keep their tensor id. Weights shared with other operators get one packed copy, <id>_packed (or <id>_packed_N when that id is taken); the original stays for any reader that does not pack and is dropped when none is left. memory.tensors rules name the original id and also place the copy; a rule on the _packed id itself is not applied. Weights that are a model input or a variable are written at run time and keep the standard layout.

Depthwise accumulation

FP16 DEPTHWISE_CONV_2D follows accumulation (see Accumulation). Allow for FP16 lanes when you choose a tolerance.

• Under precise, the default, it calls arm_depthwise_conv_f16. From CORE 7.47.0, the minimum, that entry accumulates every tap in FP32 on MVE targets; earlier releases folded an output’s FP16 sum into FP32 only when its window had more than 32 taps (ARM_NN_F16_ACC_BLOCK), so a 3x3 or 5x5 layer accumulated in FP16 throughout.
• Under fast, FP16 depthwise layers call arm_depthwise_conv_f16_acc16, which keeps FP16 lanes on MVE.

The library build switch

Start with the generated module’s dependency requirements and enable the needed precisions on the kernel library before adding the generated module. A compiler flag on application code alone cannot add missing library objects.

The float kernel objects exist only if ns-cmsis-nn itself was built with them, so the switch belongs on the library, not on the generated module.

Build system	Switch
CMake and NSX	ARM_NN_ENABLE_F32, ARM_NN_ENABLE_F16, set before adding the ns-cmsis-nn subdirectory
Zephyr	CONFIG_NS_CMSIS_NN_ENABLE_F32, CONFIG_NS_CMSIS_NN_ENABLE_F16 in the application’s prj.conf

On the standalone CMake and NSX backends the module does not define these itself. It inherits them from the kernel library’s usage interface and asks the library what it was built with, through a capability query that every supported ns-cmsis-nn release (7.47.0 and later) provides. A linked kernel target without that query fails configure with a hint to load the library’s normal CMake entry point. The library’s answer overrides a conflicting cache variable.

Only the precisions the operators actually declare are required.

• An FP16-only module can have F16 on and F32 off; an FP32-only module does not need F16. A module with no float kernel or float type dependency at all (an FP32 reshape, a metadata-only shape query) needs neither.
• Half-precision utilities that bind the half-precision type still require F16 even when they only move bytes.
• On standalone CMake a parent may bring its own kernel target instead. The module then links nothing and defines only the switches its operators require. Set a required switch on, or leave it unset to take that default; setting a required switch off fails configure.

Caution

Zephyr checks F32 for every module with a float dependency, including FP16-only modules, and checks F16 as well when required. That backend has not adopted independent precision selection.

ns-cmsis-nn also rejects the removed NSX float-enable switch names at configure time. Use ARM_NN_ENABLE_F32 and ARM_NN_ENABLE_F16; the Zephyr symbols are unchanged.

Validating against LiteRT

Automatic golden data comes from LiteRT on the conversion host. Known differences include the int8 rounding and GELU differences above and the ones below; the measurements in the table are from LiteRT 2.1.6 and host builds of the module against heliaCORE 7.47.0.

The repository’s end-to-end cases (tests/e2e/cases/) record each measured difference in a comment beside the case’s tolerance; where, at some input, the kernel is more than half a code further from exact math than LiteRT, the case also names the issue that accepts it under test.kernel_error_allowance.

Operator	Difference	Bound and reference
int8 ADD, SUB, MUL, LEAKY_RELU	The module matches LiteRT’s builtin kernels at every input of the end-to-end cases. LiteRT’s default resolver rounds some outputs one code differently (for MUL, on x86_64 hosts).	One code; the cases allow it.
int8 UNIDIRECTIONAL_SEQUENCE_LSTM	Differs from LiteRT’s reference kernels on 27 of 720 output elements over 30 random inputs to one TF-quantized model, and at some output on 9 of 14 calibrator-produced models. Not the rounding of the effective scales; the cause in the gate arithmetic is not yet attributed.	One code; the cases allow it (#823).
int8 and int16 MEAN	The module folds the division by the element count into the requantization multiplier, where LiteRT requantizes and then divides. Both round twice, at different steps. In three HeartKit models with six squeeze-and-excite MEAN layers, the first divergence from LiteRT was a MEAN, and the model outputs differed from LiteRT’s reference kernels by 1.3% to 5% relative, and by up to 6.7% on one model against the default resolver.	Each up to 0.75 code from exact math on a single layer; the two can differ by one code at an output, compounding through a model.
int16 FULLY_CONNECTED	heliaCORE’s arm_fully_connected_s16, and arm_nn_vec_mat_mult_t_per_ch_s16 for per-channel weights, requantize with the output multiplier reduced from Q31 to Q15 (REDUCE_MULTIPLIER); LiteRT keeps the full Q31 multiplier and rounds once, so the module is not bit-exact with LiteRT. The difference grows through later layers, most through a LOGISTIC or TANH whose input scale is much larger than its output scale: at a ratio of about 25, one input code becomes about 25 output codes. int16 CONV_2D and DEPTHWISE_CONV_2D are bit-exact with LiteRT, whose int16 convolutions reduce the multiplier the same way.	Up to one code per layer (187 of 1280 outputs of a single 64-to-32 layer); over int16 recurrent models the largest output differences measured 4 to 79 codes, at most about 0.25% of full scale, each above the default tolerance of 1.
int16 SOFTMAX	The module builds the exponential and reciprocal lookup tables heliaCORE’s int16 softmax reads, as TFLM does; LiteRT computes from tables of its own. The module is the closer of the two to exact softmax in every configuration measured: at 64 classes, 62 codes from exact against LiteRT’s 125.	Over 50 random full-range inputs to a single layer, up to 9 codes at 10 classes and 63 at 64 classes (input scale 0.001 or more).
int16 TANH, LOGISTIC	When the log2 of the input scale is within 0.001 of −12 (both) or −11 (TANH only), LiteRT switches to its gemmlowp kernel and the module uses shift-only scaling, computing as if the scale were exactly that power of two.	Over every such int16 input of the end-to-end models, up to 12 codes for TANH and 7 for LOGISTIC; against exact math the module is off by up to 7.9 and 4.2 codes and LiteRT by up to 17.1 and 9.5 (#501).
int16 SQRT	The kernel approximates the square root in float32 and truncates.	Can differ from LiteRT by one code where the exact result is within 0.01 of an integer (#567).
int8-state SVDF	LiteRT’s int8-state SVDF can be far from exact SVDF math, with its default and reference kernels alike; its int16-state SVDF matches exact math. heliaAOT’s two int8-state end-to-end cases are therefore checked against exact SVDF math on the dequantized parameters, not against LiteRT.	LiteRT up to 255 codes from exact, and up to 234 on the current models of those cases; the cases use a tolerance of 1 against exact math. Supply golden data from a trusted reference for such a model.
Float REDUCE_MAX, REDUCE_MIN	These follow TFLM, and the operator catalog restriction says where LiteRT differs on NaN and infinities. Two further differences are in the bits only: of equal values the module keeps the first, so the maximum of [+0, -0] is +0, where LiteRT’s default kernels keep the last, -0, and a numeric comparison treats them as equal; on an x86_64 host LiteRT’s default resolver flushes subnormal values to zero, so the maximum of [1.4e-45, 0] is 0 there and 1.4e-45 in the module.	NaN and infinities per the catalog; the two further differences are bits only.

To check whether one of these explains a mismatch on your model:

1. Convert with LiteRT golden data and keep the reported maximum difference per output; do not raise test.tolerance yet.
2. Find the first operator whose output differs. Run LiteRT with experimental_preserve_all_tensors=True to read every intermediate tensor. On the module side, the context’s callback, which module.schedule: table (the default) calls as each operator finishes running, is where a host build can capture that operator’s output. Its op argument indexes ConversionResult.operators (with result_views set), whose source_indices name the model-file operator to compare against.
3. If that operator is one of those listed here, at its dtype, and its own difference is within the bound given for it, its rounding is a candidate explanation, not proof of the cause of every difference after it.
   • Before blaming propagated rounding for the final mismatch, or accepting a larger tolerance, check the operators downstream: feed each one the module’s own input to it (LiteRT’s preserved tensors give you the same input on the LiteRT side), and confirm that each, given matching inputs, agrees with LiteRT within its own bound.
   • A downstream operator that differs with matching inputs is a separate difference.
4. For int16 FULLY_CONNECTED, SOFTMAX and MEAN the model-level difference can be much larger than the per-layer one, so set the tolerance from your application’s accuracy requirement against an exact or trusted float reference, not from LiteRT’s outputs alone.
5. If the first divergent operator is not listed, or exceeds its bound, report it with the model and configuration.

Checking a model before you convert

Coverage is per operator, per element type and per target, not per model. The operator catalog is the table: every operator heliaAOT lowers, its dtype summary, declared float API or type dependency, and the oldest library that float path builds against. The module-wide kernel library floor is on the Versions page.

The conversion itself checks whether each remaining node has an applicable lowering. Then build and validate the generated module with the intended library and target, following the output verification workflow with the intended library, compiler and execution environment.

• Successful conversion alone does not establish optimized execution or numerical correctness.
• Keep the acceptance threshold fixed while investigating a mismatch; increasing it to fit observed errors is not validation.