Skip to content
heliaPROFILER
Reference
HELIA

expect

Checking a run’s outputs: reference outputs from host LiteRT or caller files, compared tensor by tensor, with a verdict only when a tolerance is given.

Every name on this page is imported from helia_profiler.

API tier: experimental

Generated from the src/helia_profiler tree f38e378c801d34438affb680257217e0b58d2cd1.

Machine-readable model

  • ToleranceclassLargest accepted absolute error per element, in each tensor's stored units.
  • ReferenceOriginclassWhere reference outputs came from.
  • ReferenceOutputsclassReference bytes for every output, in model order, and their origin.
  • ExpectclassWhat a run's outputs are compared against.
  • TensorComparisonclassComparison facts for one output tensor.
  • VerificationclassPer-tensor comparison facts, the reference they used and an optional verdict.
  • host_referencefunctionRun a LiteRT model on the host and return its output bytes in outputs order.
  • compare_outputsfunctionCompare observed output bytes with reference outputs, tensor by tensor.
  • compare_fixture_outputsfunctionCompare a fixture capture's outputs with expect.
class

Tolerance

Python

Largest accepted absolute error per element, in each tensor's stored units.

src/helia_profiler/runs/expect.py:45

Tolerance(max_abs_error: float = 0.0) -> None

dataclass

Largest accepted absolute error per element, in each tensor’s stored units.

Integer tensors count quantization steps; float tensors count values. One tolerance applies to every output tensor.

API tier: experimental

class

ReferenceOrigin

Python

Where reference outputs came from.

src/helia_profiler/runs/expect.py:60

ReferenceOrigin(
    kind: ReferenceKind,
    litert_version: str | None = None,
    resolver: str | None = None,
    invocations: int | None = None,
) -> None

dataclass

Where reference outputs came from.

A host reference records the LiteRT package version, its op resolver and how many times the model was invoked on the inputs.

API tier: experimental

class

Expect

Python

What a run's outputs are compared against.

src/helia_profiler/runs/expect.py:82

Expect(
    source: ReferenceKind,
    files: tuple[FixtureFile | Path, ...] = (),
    resolver: LiteRtResolver = 'BUILTIN_REF',
    tolerance: Tolerance | None = None,
) -> None

dataclass

What a run’s outputs are compared against.

Build with :meth:host_litert, :meth:from_build or :meth:from_files.

API tier: experimental

method

host_litert

Python

Run the model with host LiteRT on the run's input bytes (LiteRT models only).

src/helia_profiler/runs/expect.py:94

host_litert(*, resolver: LiteRtResolver = 'BUILTIN_REF', tolerance: Tolerance | None = None) -> Expect

classmethod

Run the model with host LiteRT on the run’s input bytes (LiteRT models only).

method

from_files

Python

Compare with caller bytes, one file per output in model order.

src/helia_profiler/runs/expect.py:107

from_files(*files: FixtureFile | str | Path = (), tolerance: Tolerance | None = None) -> Expect

classmethod

Compare with caller bytes, one file per output in model order.

class

TensorComparison

Python

Comparison facts for one output tensor.

src/helia_profiler/runs/expect.py:118

TensorComparison(
    name: str,
    index: int,
    dtype: str,
    shape: tuple[int, ...],
    reference_sha256: str,
    observed_sha256: str,
    elements: int,
    mismatched_elements: int,
    max_abs_error: float,
    max_abs_error_real: float | None,
    within_tolerance: bool | None,
) -> None

dataclass

Comparison facts for one output tensor.

max_abs_error is in stored units; max_abs_error_real is the same error after dequantization, None for an unquantized tensor. A NaN on one side only counts as a mismatch with infinite error. within_tolerance is None when no tolerance was given.

API tier: experimental

class

Verification

Python

Per-tensor comparison facts, the reference they used and an optional verdict.

src/helia_profiler/runs/expect.py:145

Verification(reference: ReferenceOutputs, tensors: tuple[TensorComparison, ...], tolerance: Tolerance | None) -> None

dataclass

Per-tensor comparison facts, the reference they used and an optional verdict.

API tier: experimental

attribute

matched

Python

Whether every tensor is within tolerance; None when no tolerance was given.

src/helia_profiler/runs/expect.py:154

matched: bool | None

Whether every tensor is within tolerance; None when no tolerance was given.

function

host_reference

Python

Run a LiteRT model on the host and return its output bytes in outputs order.

src/helia_profiler/runs/expect.py:161

host_reference(
    model: bytes,
    inputs: Sequence[tuple[FixtureTensor, bytes]],
    outputs: Sequence[FixtureTensor],
    *,
    resolver: LiteRtResolver = 'BUILTIN_REF',
    invocations: int = 1,
) -> ReferenceOutputs

Run a LiteRT model on the host and return its output bytes in outputs order.

inputs must cover every model input. They are restored before every invocation, as the fixture firmware does, so internal state carries across invocations while inputs do not.

API tier: experimental

function

compare_outputs

Python

Compare observed output bytes with reference outputs, tensor by tensor.

src/helia_profiler/runs/expect.py:235

compare_outputs(
    tensors: Sequence[FixtureTensor],
    reference: ReferenceOutputs,
    observed: Sequence[bytes],
    tolerance: Tolerance | None = None,
) -> Verification

Compare observed output bytes with reference outputs, tensor by tensor.

API tier: experimental

function

compare_fixture_outputs

Python

Compare a fixture capture's outputs with expect.

src/helia_profiler/runs/expect.py:304

compare_fixture_outputs(build: FixtureBuild, capture: FixtureCaptureResult, expect: Expect) -> Verification

Compare a fixture capture’s outputs with expect.

A host reference runs the build’s model on its pinned input bytes as many times as the firmware invokes it (warmups, then timed iterations).

API tier: experimental

Machine-readable: expect.json · expect.md, complete and lossless.