Skip to content

Add a model to release testing

Release-model coverage starts with an immutable model/golden pair in helia-model-zoo. Adding a model to that repository does not automatically add it to the helia-aot release gate. The model-zoo change must merge first, then a helia-aot change deliberately promotes the new corpus commit and selects the model for execution.

Before changing helia-aot

Confirm that the merged model-zoo commit contains:

  • a stable ID in corpus-manifest-v1.json;
  • hydrated .tflite and .npz artifacts whose SHA-256 values match the manifest;
  • all required input_N and output_N arrays with signature-compatible shapes and dtypes; and
  • a model card documenting provenance and the applicable upstream license.

Follow the model-zoo's docs/how-to/add-release-model.md guide to prepare and validate that change. Record the exact commit SHA after it merges.

Promote the corpus commit

Update HELIA_MODEL_ZOO_SHA in .github/workflows/release-model-e2e.yml to the merged model-zoo commit. This is an intentional manual promotion: do not use a branch or tag that can move.

Keep the corpus SHA update and its release case in the same helia-aot pull request. That ensures CI evaluates the case against the manifest version that introduced its ID.

Add the release case

Create tests/e2e/cases/release-models/<model>.yaml. Reference only the stable manifest ID; do not repeat model or golden paths:

id: model_example_int8
description: Example int8 model
platforms: [apollo510_evb]
model:
  source: corpus
  corpus_id: example-int8
test:
  tolerance: 10

Omit platforms when the case should run on both supported platforms. An explicit list must overlap the release workflow matrix; collection rejects a release case that no workflow leg can execute. Add memory-placement rules or a longer fvp_timeout_seconds only when the model requires them. A manifest-backed case must not set model.path or test.golden_data; the manifest is the only source of the artifact pair. Every case in release-models/ must use model.source: corpus — collection rejects any other source, because only manifest-backed models carry the guardrails (no skip_build, no skip_verification) that make a pass mean "executed on target and compared against golden".

Do not add a second default/prebuilt case for the same model. Release cases are excluded from ordinary E2E discovery and run through the dedicated release workflow.

If a known model cannot currently run, keep its case as <model>.yaml.disabled with a short explanation. A disabled file documents future work but is not release coverage.

Qualify and merge

  1. Run focused unit tests for corpus resolution and provenance.
  2. Run the release-model workflow manually with the helia-aot commit under review.
  3. Confirm every enabled case passes on each configured platform and inspect the retained provenance and FVP evidence. The job's own Enforce release-model qualification step re-reads that record and fails unless every declared case/platform pair is passed and the recorded model-zoo revision matches HELIA_MODEL_ZOO_SHA, so an all-skipped run or corpus substitution cannot report green.
  4. Confirm a failure prevents the ci-gate aggregate check from passing.
  5. Merge only after the pinned corpus SHA and case selection have been reviewed together.

Repository configuration prerequisites

The gate only blocks publication when repository settings enforce it:

  • The main branch ruleset must list ci-gate as a required status check. This single aggregate requires ordinary lint and E2E on every PR, plus release-model E2E on Release Please PRs. Without the ruleset entry, workflow results are advisory and one approving review is sufficient to merge.
  • The Release Please PR is opened by github-actions[bot], which trips the repository's "require approval to run workflows" policy. Its CI runs are therefore created in the action_required state: the PR page shows "N workflows awaiting approval" with an Approve workflows to run button, and gh pr checks reports nothing until a maintainer clicks it.

This is deliberate and load-bearing. Release-model E2E is expensive (two platforms under FVP), and holding the run means it executes when you are actually releasing rather than on every merge to main. A maintainer must approve the workflows before the Release Please PR can merge — until then the required ci-gate check never reports and the main ruleset keeps the PR blocked. Approval binds to a specific commit, so if Release Please pushes a new commit afterwards, approve the fresh run as well.

Interim coverage between releases

Release cases are excluded from ordinary per-PR E2E discovery, so a codegen regression in a release model would otherwise surface only at release time. .github/workflows/release-model-e2e.yml therefore also runs weekly on a schedule against the default branch head. Run it manually (workflow_dispatch) with an explicit helia_aot_ref when a change needs qualification sooner.

A failed scheduled run files (or comments on) a single open issue labelled release-model-e2e, including the run link and the per-platform verdicts read back from reports/provenance-*.json. The report names every expected matrix platform and marks missing evidence as a failure, so a cancelled or early-failed leg cannot disappear behind a passing leg. Passing runs create no noise, and re-runs update the existing issue instead of opening a new one — close it once the corpus qualifies again. When more than one open issue carries the label, the oldest is treated as canonical so the original thread keeps accumulating history.

Evidence is uploaded per attempt (release-e2e-<platform>-<attempt>). Because Re-run failed jobs re-uploads only the legs that failed, the reporter downloads every attempt and keeps the newest record per platform — a leg that passed on the first attempt is still reported as passing after a partial re-run.

If rendering the detailed report itself fails, the issue is still filed with a fallback body pointing at the run, and the reporter job then fails so the fault is visible rather than silently suppressing the week's only signal.

Publication gating

Release Please creates the GitHub Release as a draft ("draft": true in release-please-config.json). GitHub does not create the git tag for a draft release, so until qualification succeeds there is nothing consumable: no Latest release, no auto-attached source tarball/zipball, and no pip install git+https://github.com/AmbiqAI/helia-aot@<tag>.

.github/workflows/release.yml promotes the draft in its publish-release job, which runs only after release-model-e2e qualifies both platforms and the binary and wheel assets are attached. Promoting is what finally creates the git tag, so the job verifies the tag appeared before succeeding.

This is deliberate: a binary build failure on any runner OS blocks promotion, and therefore blocks PyPI and the docs site too. A Release that goes public missing assets is worse than one that is late, and recovery is manual either way. docs is gated behind promotion for the same reason — a published site is itself a consumable artifact and must not advertise a version whose Release may never go public.

Release Please runs in two invocations, and the split matters. The release-please job uses skip-github-pull-request: true and only creates the draft Release; the release-pr job runs afterwards with skip-github-release: true and opens the next release PR. Release Please ignores any release whose git tag does not exist, so while an unpromoted draft is outstanding it would see "no previous release" even though .release-please-manifest.json already claims the new version — and would replay the entire commit history into the next changelog. release-pr therefore refuses to run until the manifest version has a real tag, and records why in the run summary when it skips.

If the release is not published — whether qualification failed or promotion failed downstream of it — the draft is retained rather than deleted, so the generated release notes stay available for inspection and retry; the report-unpublished-release job records that state in the run summary. Promote manually with gh release edit <tag> --draft=false once the underlying failure is fixed, or use Re-run failed jobs. A full re-run of the workflow will not work: release-please finds no pending release PR and everything skips.