ETHOS_U_STREAM_SPLIT contract

Owner of the rewrite and this contract: helia-npu (helia_npu.fencing). Consumers: helx’s local FVP runner (kernel in helia_estimator/local_fvp_runtime/src/stream_split.cpp), heliaRT (in-tree registration factory when fenced models become a supported format; app-side AddCustom until then), heliaAOT 1.3.0 and later. All three engines support HELX fenced per-segment profiling. Contract version: 1 (custom-op version 1). Any change below bumps it.

The custom-op identifier replaces the earlier STREAM_SPLIT name. Regenerate fenced models from their original unfenced input before using this runner; the old identifier is not registered as an alias. Version 1 describes the unchanged identity/copy semantics of the new identifier. Python constant names and /stream_split tensor-name suffixes remain unchanged.

Purpose

Vela ends the current Ethos-U command stream at a custom op it cannot fuse. ETHOS_U_STREAM_SPLIT is an identity op inserted between two original operators so that the graph is compiled as N NPU command streams with a CPU handoff at each fence. It exists to make a boundary, not to compute.

Wire format (TFLite flatbuffer)

Item	Value
OperatorCode.customCode	"ETHOS_U_STREAM_SPLIT" (exact, case-sensitive; consumers may canonicalize case on lookup)
OperatorCode.builtinCode / deprecatedBuiltinCode	CUSTOM (32)
OperatorCode.version	1
inputs	N ≥ 1: all tensors produced at/before after_op and consumed later or returned by the graph
outputs	N outputs, paired by index with the inputs; first output is the sidecar join key; additional outputs have cut-specific names
builtinOptionsType	NONE (0); builtinOptions absent
customOptions	absent (no flexbuffer)
output tensor	same shape, type, and a deep copy of quantization (scale, zero point, per-axis fields) as the input; isVariable = false; new empty buffer
rewiring	every later operator that consumed the intercepted tensor now consumes the new tensor; if the intercepted tensor was a graph output, the graph output is rewired to the new tensor
placement	inserted immediately after operator after_op; fences are applied in descending order so original indices stay valid during the rewrite
existing operator code	a ETHOS_U_STREAM_SPLIT entry already in the model is reused only if its effective builtin code is CUSTOM (the larger of builtin_code and deprecated_builtin_code, as TFLite’s GetBuiltinCode reads the v3 and v3a schemas, so 32/32, 32/0 and 0/32 qualify, while a non-CUSTOM builtin_code with a CUSTOM deprecated field does not, since Vela reads builtin_code first) and its version is 1; a reused entry is passed through as written, not rewritten to 32/32; any other same-name entry is rejected before output is written, never reused or duplicated
subgraph	0 only; a model with more than one subgraph is rejected by _primary_subgraph before anything is written

Supported dense storage types: int8, uint8, int16, int32, int64, float16, float32 and bool. Pairs may use different types and shapes; each output must match its corresponding input. Byte counts come from tensor metadata, not a fixed element size. Packed INT4, sparse, string/resource and variable tensors are outside this contract. FP16 copying requires no floating-point arithmetic.

The rewrite carries residual paths and all live outputs, not just the selected operator’s first output. Constants and graph inputs are already available and are not copied. This establishes data dependencies across the cut; it does not impose a global order on unrelated branches with no crossing data dependencies. This is the initial contract for the renamed operator, including multi-tensor support.

Boundary semantics

• Identity of contents per pair: output bytes equal input bytes; dtype, shape, quantization unchanged. byte length = flat size × element size and must match on both sides; a kernel must fail Prepare on any count, type, shape or byte-length mismatch rather than truncate.
• Copy, not alias, by default: the output is a distinct buffer with a distinct lifetime. Exact alias (out == in) is permitted and is a no-op. Outputs overlapping another pair’s inputs or outputs, and partial overlap within a pair, are rejected before copying so no destination clobbers data still needed by another pair.
• Execution boundary: the op must remain a scheduling point between the NPU region that produces the input and the NPU region that consumes the output. A compiler must not eliminate it as a reshape or no-op even when aliasing would be legal.
• What it does not promise: no cache maintenance, no coherency, no resumable or cooperative scheduling. Those belong to the runtime’s Ethos-U driver handoff and are qualified on the target, not by this op.

Identity carried with a fenced model

RewriteResult and the <stem>.fenced_boundaries.json sidecar record: source_model, source_sha256 (of the model the fences were cut into), subgraph, num_segments, boundary_after_ops (original operator indices, authoritative), and per segment op_range_start, op_range_end, tensor_pairs (actual input/output names in the final fenced graph), end_tensor_name (the first intercepted tensor in the original source model) and fence_tensor_name (<name>/stream_split). plan_fences attaches the same output_tensor_name, opcode, segment_index to each Fence and records model_sha256 when given the model. A consumer joins across recompilation by (source_sha256, after_op) first and tensor names second; compiled and runtime operator indices are different coordinates and are not carried here.

A plan is bound to the model it was cut for. fence_model (via check_plan_binding) rejects, before writing, a FencePlan whose model_sha256 (when the planner recorded one) or operator count does not match the model it is applied to or whose subgraph is not 0, and any Fence carrying identity whose output_tensor_name or opcode is not what sits at after_op in that model, or whose after_op follows the last operator (FenceBindingError). A plan cut without the model has no hash and is bound by count and names only. The unbound path is positions alone: plan.after_ops, a list of ints, or a FenceConfig (fence_config_for is the deliberate reduction of a plan to positions) given to rewrite_model_with_fences; that is the caller asserting the coordinates apply.

Cost planning caveat

plan_fences budgets on Vela’s estimate attributed to original operators. FenceCoverage reports the denominator (sum of op_cycles over all Vela rows), the matched part, and the unmatched rows. A cycle-based policy raises when coverage is below min_coverage (default 100%) or when there are no rows at all; lowering the threshold records uncertainty, it does not create a latency bound. An operator larger than the budget is isolated, not split.