evaluation

Comparing two runs and judging one: the comparability rules that decide whether a comparison is meaningful, and the metric policies that turn numbers into a verdict.

Every name on this page is imported from helia_profiler.

API tier: experimental

Generated from the src/helia_profiler tree fe269596788d6ab1604cf433559e96b5f48c1d13.

Machine-readable model

MetricDirectionclassPreferred candidate direction for one metric.
MissingMetricPolicyclassVerdict when a selected metric is unavailable.
ComparabilityIssueclassOne machine-readable compatibility decision.
RunEvaluationclassAuthoritative validity and structured issues for one completed run.
VerdictStatusclassRegression policy outcome.
ComparabilityAssessmentclassWhether run-level and per-layer deltas may be computed.
MetricPolicyclassTolerance and availability policy for one named comparison metric.
ComparisonProfileclassOpen v1 profile selecting deterministic metric regression policies.
CompareResultclassFull comparison between two profile runs.
assess_comparabilityfunctionCompare identity, validity, topology, and intentional run dimensions.
MetricVerdictclassVerdict and evidence for one selected metric.
evaluate_runfunctionEvaluate captured results without mutating pipeline state.
ComparisonVerdictclassDeterministic verdict for one result pair and profile.
evaluate_comparison_profilefunctionEvaluate existing metric deltas against one versioned profile.
ComparabilitySeverityclassEffect of one comparability issue on comparison output.

MetricDirection()

Preferred candidate direction for one metric.

API tier: experimental

SMALLER = 'smaller'

LARGER = 'larger'

EQUAL = 'equal'

MissingMetricPolicy()

Verdict when a selected metric is unavailable.

API tier: experimental

FAIL = 'fail'

WARN = 'warn'

IGNORE = 'ignore'

ComparabilityIssue(code: str, severity: ComparabilitySeverity, message: str, context: dict[str, Any] = dict()) -> None

dataclass

One machine-readable compatibility decision.

API tier: experimental

code: str

severity: ComparabilitySeverity

message: str

context: dict[str, Any] = field(default_factory=dict)

RunEvaluation(
    validity: ResultValidity,
    issues: tuple[ResultIssue, ...] = (),
    gate_arbitration: GateArbitration | None = None,
) -> None

dataclass

Authoritative validity and structured issues for one completed run.

API tier: experimental

validity: ResultValidity

issues: tuple[ResultIssue, ...] = ()

gate_arbitration: GateArbitration | None = None

VerdictStatus()

Regression policy outcome.

API tier: experimental

PASS = 'pass'

WARN = 'warn'

FAIL = 'fail'

SKIP = 'skip'

ComparabilityAssessment(issues: tuple[ComparabilityIssue, ...] = ()) -> None

dataclass

Whether run-level and per-layer deltas may be computed.

API tier: experimental

issues: tuple[ComparabilityIssue, ...] = ()

run_metrics_comparable: bool

layers_comparable: bool

power_metrics_comparable: bool

memory_metrics_comparable: bool

Per-region used/free rows are gated on the link family (#206).

metric_group_comparable(group: str) -> bool

Whether one metric group’s rows may be computed (#206): no METRIC_BLOCKING issue carrying that group’s tag.

MetricPolicy(
    direction: MetricDirection,
    unit: str,
    max_regression_pct: float | None = None,
    max_regression_abs: float | None = None,
    missing: MissingMetricPolicy | None = None,
    extra: dict[str, Any] = dict(),
) -> None

dataclass

Tolerance and availability policy for one named comparison metric.

API tier: experimental

direction: MetricDirection

unit: str

max_regression_pct: float | None = None

max_regression_abs: float | None = None

missing: MissingMetricPolicy | None = None

extra: dict[str, Any] = field(default_factory=dict, repr=False)

from_dict(data: dict[str, Any]) -> Self

classmethod

to_dict() -> dict[str, Any]

ComparisonProfile(
    schema: str,
    schema_version: int,
    metrics: dict[str, MetricPolicy],
    missing: MissingMetricPolicy | None = None,
    required_dimensions: tuple[str, ...] = (),
    name: str | None = None,
    extra: dict[str, Any] = dict(),
) -> None

dataclass

Open v1 profile selecting deterministic metric regression policies.

API tier: experimental

schema: str

schema_version: int

metrics: dict[str, MetricPolicy]

missing: MissingMetricPolicy | None = None

required_dimensions: tuple[str, ...] = ()

name: str | None = None

extra: dict[str, Any] = field(default_factory=dict, repr=False)

from_dict(data: dict[str, Any]) -> Self

classmethod

load(path: str | Path) -> Self

classmethod

to_dict() -> dict[str, Any]

CompareResult(
    baseline: RunArtifacts,
    candidate: RunArtifacts,
    config_rows: list[ConfigDiffRow],
    metrics: list[MetricDiff],
    layer_rows: list[LayerDiffRow],
    warnings: list[str] = list(),
    comparability: ComparabilityAssessment = ComparabilityAssessment(),
    verdict: ComparisonVerdict | None = None,
) -> None

dataclass

Full comparison between two profile runs.

API tier: experimental

baseline: RunArtifacts

candidate: RunArtifacts

config_rows: list[ConfigDiffRow]

metrics: list[MetricDiff]

layer_rows: list[LayerDiffRow]

warnings: list[str] = field(default_factory=list)

comparability: ComparabilityAssessment = field(default_factory=ComparabilityAssessment)

verdict: ComparisonVerdict | None = None

assess_comparability(baseline: RunArtifacts, candidate: RunArtifacts) -> ComparabilityAssessment

Compare identity, validity, topology, and intentional run dimensions.

API tier: experimental

MetricVerdict(
    metric: str,
    status: VerdictStatus,
    message: str,
    baseline: float | None = None,
    candidate: float | None = None,
    regression: float | None = None,
    allowed_regression: float | None = None,
    unit: str = '',
) -> None

dataclass

Verdict and evidence for one selected metric.

API tier: experimental

metric: str

status: VerdictStatus

message: str

baseline: float | None = None

candidate: float | None = None

regression: float | None = None

allowed_regression: float | None = None

unit: str = ''

evaluate_run(ctx: PipelineContext) -> RunEvaluation

Evaluate captured results without mutating pipeline state.

API tier: experimental

ComparisonVerdict(
    status: VerdictStatus,
    metrics: tuple[MetricVerdict, ...],
    dimension_mismatches: tuple[str, ...] = (),
    profile_name: str | None = None,
    profile_schema: str = COMPARISON_PROFILE_SCHEMA,
    profile_schema_version: int = COMPARISON_PROFILE_SCHEMA_VERSION,
    profile_sha256: str = '',
) -> None

dataclass

Deterministic verdict for one result pair and profile.

API tier: experimental

status: VerdictStatus

metrics: tuple[MetricVerdict, ...]

dimension_mismatches: tuple[str, ...] = ()

profile_name: str | None = None

profile_schema: str = COMPARISON_PROFILE_SCHEMA

profile_schema_version: int = COMPARISON_PROFILE_SCHEMA_VERSION

profile_sha256: str = ''

evaluate_comparison_profile(result: CompareResult, profile: ComparisonProfile) -> ComparisonVerdict

Evaluate existing metric deltas against one versioned profile.

API tier: experimental

ComparabilitySeverity()

Effect of one comparability issue on comparison output.

Defined here so the comparability registry can bind severity to code without importing from evaluation; evaluation.comparability re-exports it, which remains the canonical public import path.

API tier: experimental

BLOCKING = 'blocking'

LAYER_BLOCKING = 'layer_blocking'

METRIC_BLOCKING = 'metric_blocking'

INFORMATIVE = 'informative'

Machine-readable: evaluation.json · evaluation.md, complete and lossless.