Tensor inputs, outputs and numerical verification

Opt-in tensor capture binds exact input bytes into the runner, retains outputs for each measured invocation and optionally compares them with reference bytes. It supports LiteRT, heliaRT and heliaAOT on the existing FVP paths. The default run still fills inputs with zeros and retains compact output signatures.

Capture outputs

Add this to an ordinary unfenced FVP config:

defaults:
  fvp:
    enabled: true
    tensor_io: {}

An empty policy explicitly uses zero input bytes and captures output bytes. Inputs are refilled before every warmup and measured invocation; persistent model variables still follow the acquisition state policy. Output serialization occurs after counters and CPU timers stop. UART transport and host wall time grow with captured bytes; they are not added to the measured inference latency.

helx run -c experiment.yaml
helx show results/my-run

Capture requires concrete positive tensor dimensions (scalars are supported). A model’s shape signature does not request runtime resizing: HELX binds the concrete stored shape. Storage types are bool, int8, uint8, int16, int32, int64, float16 and float32, using little-endian bytes. Boolean bytes must be zero or one. Inputs plus reference data are limited to 16 MiB per request; total measured output bytes are also limited to 16 MiB. Explicit tensor policies currently exclude bus tracing, fencing and arbitrary AOT conversion overrides.

Supply exact inputs from Python

model_tensors reads model metadata without invoking the model. Supply every input in model-index order, with matching name, dtype, shape and byte length. TensorData validates byte storage; TensorLiteral records it as explicit hexadecimal in YAML/JSON so a saved request contains its own input bytes.

from pathlib import Path
from helia_estimator.api import (
    EstimateRequest, TensorData, TensorIOPolicy, TensorLiteral,
    estimate, load_request, model_tensors,
)

request = load_request("experiment.yaml")
inputs, outputs = model_tensors("model.tflite")
values = tuple(
    TensorLiteral.from_tensor(TensorData(spec, Path(f"input-{spec.index}.bin").read_bytes()))
    for spec in inputs
)
config = request.config
config.defaults.fvp.tensor_io = TensorIOPolicy(inputs=values)
outcome = estimate(EstimateRequest(config, base_dir=request.base_dir))

The same request works with freeze → build → run_build. Tensor policy, input bytes and reference policy are bound to the build. Changing them requires a new build; replay does not silently read replacement files or compile another image. A runner reports the compiled tensor-request digest, which is checked before publishing its results. This digest identifies the captured request; it is not a cross-product semantic input identity.

Read and compare retained outputs

A numerical reference must be identified explicitly. ReferenceOrigin records whether it came from files, a host LiteRT run or another producer; the description records the caller’s identifying context. HELX does not infer that a reference is independent or correct. Reference generation remains the caller’s job.

from pathlib import Path
from helia_estimator.api import (
    ComparisonPolicy, NaNPolicy, ReferenceOrigin, ReferenceOutputs,
    TensorData, TensorTolerance, Tolerance,
    load_result, output_tensors, verify_outputs,
)

result = load_result("results/my-run")
actual = output_tensors(result, sample_index=0)
reference = ReferenceOutputs(
    tensors=tuple(
        TensorData(t.spec, Path(f"reference-{t.spec.index}.bin").read_bytes())
        for t in actual
    ),
    origin=ReferenceOrigin.FILES,
    description="Original model, independent reference tool/version and matching initialization/input sequence",
)
policy = ComparisonPolicy(
    default=Tolerance(absolute=0, relative=0),
    per_tensor=(TensorTolerance(0, Tolerance(absolute=1, relative=0)),),
    nan=NaNPolicy.NEVER_MATCH,
)
check = verify_outputs(result, 0, reference, policy)
print(check.matched, check.tensors)

The comparison is abs(actual-reference) <= absolute + relative*abs(reference) in stored-value units. Quantized values are compared as their stored integers; HELX does not silently dequantize them. Exact integer comparisons avoid conversion to float, including int64 values above 2**53. The default tolerance is zero. Per-tensor overrides identify output ordinals. Matching signed infinities match; NaNs mismatch by default, or match only when both positions are NaN under MATCH_BOTH. Shape, dtype, name and output count must agree.

To perform and retain comparison during execution, set TensorIOPolicy.references to one TensorReference.from_reference(reference) per measured invocation, and set its comparison policy. Stateful references must follow the same initialization, warmup, input and carry sequence; do not repeat one reference unless that is the intended model behavior.

Result and failure semantics

• tensors records whether inputs were explicit or implicit zeros, effective input artifacts, output specifications, references, policy and the runner’s request digest. Build-only results can carry these inputs.
• measurements.samples[i].output_tensors names the retained binary output files.
• measurements.samples[i].verification records the reference index, match flag, mismatches, nonfinite-pair counts and exact maximum finite absolute error.
• load_result verifies artifact hashes, tensor storage and runner signatures, reconstructs the canonical tensor request and checks its compiled digest, then recomputes retained numerical verdicts. It rejects inconsistent evidence.
• A numerical mismatch retains a complete, inspectable bundle. Python execution returns that result with matched=False. CLI run/replay/run-build exits 1 with an actionable message and the bundle path. show can inspect mismatching results.
• Without reference outputs, capture has no numerical verdict. Neither output signatures nor manifest integrity establish numerical correctness.

The portable draft keeps tensor evidence and indexed verification in HELX’s extension. BENCH decides how to admit or compare such results; HELX does not assign suite-level correctness or silicon qualification.

Portable input identity and assessment

The public offline reader exposes the actual stored input identity separately from the build’s tensor request digest:

from helia_estimator.api import input_identity, load_result, observations

result = load_result("results/my-run")
identity = input_identity(result)  # Frozen InputIdentity; no model execution.
print(identity.value, identity.unavailable_reason)
view = observations(result)
print(view.workload.input_identity)
assert view.extensions is not None
for sample in view.extensions["helia-estimator"]["tensor_binding"]["samples"]:
    print(sample["sample_index"], sample["status"], sample["reference_index"])

The namespaced tensor_binding contains ordered input and output artifact IDs, SHA-256, byte sizes, tensor metadata, explicit/implicit input intent, reference origin/description, tolerance/NaN policy and each sample’s recomputed comparison. Statuses are match, mismatch and not_compared. A mismatch does not rewrite execution outcome; no reference means no numerical pass. comparison_units is stored_values; independent quantization scale/zero-point metadata is currently unavailable and explicitly labeled. Model qualification and cross-producer comparison eligibility remain separate from these recorded assessments.

Input storage identity v1

The format is helx.inputs/v1:sha256:<64 lowercase hex characters>. Its preimage is UTF-8 JSON with sorted keys, compact separators, non-ASCII text preserved and no nonfinite numbers:

{"inputs":[{"dtype":"int8","index":0,"name":"input","sha256":"<raw input byte digest>","shape":[2]}],"schema":"helx.inputs/v1","storage":"little-endian"}

The inputs list is in contiguous model-index order. Each element binds name, index, dtype, shape and the SHA-256 of its exact little-endian bytes. The frozen InputIdentity.canonical_json exposes the exact preimage for independent checks. Paths, references, tolerance policy, sample count and explicit/implicit intent are excluded: identical effective zero bytes have the same storage identity whether supplied explicitly or filled automatically. Input intent remains a separate recorded fact. Changed tensor metadata or bytes changes the identity.

This is a producer-specific storage identity, not the compiled request digest, a raw file digest or a claim of equivalent quantized semantics. HPX’s input hash has its own kind/basis. Consumers must inspect those definitions before matching across producers; deleting prefixes or equating digest tails is not supported. No captured inputs yields a null identity with a reason, even if the runner’s default input behavior is known.

Portable export is helx results export results/my-run --format json --output view.json; Python callers can use render_result(result) or export_result(result, path). The public schema remains BENCH-owned 1.0-draft; mapping helx-schema4-draft4 binds the new evidence through the existing workload field and the HELX namespaced extension.