Memory planners

The default planner is hill_climb. It searches for a smaller scratch layout and keeps the greedy layout unless the search’s is smaller in total and no larger in any memory, so no scratch arena is larger than greedy’s. Choose another planner when you need a specific layout, such as the one an earlier release produced (memory.planner: greedy). Keep the model, platform, placement rules and budgets fixed while comparing planners; a lower footprint alone does not establish faster inference.

Compare a candidate

This block spells out the default planner and its options; add it to a configuration that already converts, and change planner to greedy or greedy_by_size (which take no planner_options) for the other side of the comparison:

memory:
  planner: hill_climb
  planner_options:
    iterations: 300
    seed: 0
    max_stall_iterations: 50
  dump_residency_json: true

Convert to a separate output directory for each planner. Compare scratch used, peak_live and fragmentation in the residency reports, then compare the memory bank of each tensor. A successful candidate fits the same constraints and passes the same numerical validation as the baseline. Measure on your target before choosing a layout for latency.

These are the hill-climb defaults: at most 300 order perturbations, deterministic seed 0, and a stop after 50 consecutive perturbations that find no improvement. iterations: 0 skips the search and compares the greedy-by-size seed layout with greedy’s. A positive max_stall_iterations stops after that many consecutive attempts without improvement, and null runs all iterations. The limit counts attempts rather than time, so a conversion produces the same layout on any host.

More iterations cost conversion time, not inference time: there is no runtime planning. Measured on a shared Linux host for apollo510_evb, whole conversions of the repository’s models took, with greedy and then with the default: 0.017 and 0.027 seconds for examples/first-model/kws_ref, 0.015 and 0.020 seconds for tests/e2e/models/cnn_s_quantized, and 0.007 and 0.010 seconds for tests/e2e/models/con2d_stateful_int16x8. Without the stall limit kws_ref took 0.074 seconds. The cost grows with the number of scratch tensors: planning a synthetic chain of 2,000 scratch tensors took 0.12 seconds with greedy, 3.3 seconds with the default and 18.8 seconds without the stall limit.

The planner

memory.planner names the algorithm that turns tensor lifetimes into arena offsets. Three are registered: hill_climb is the default, greedy remains available, and greedy_by_size is experimental.

Value	What it does
greedy	First fit in lifetime-start order over a shared buffer per writable memory: a tensor takes the first gap large enough, and gaps left by tensors whose lifetime has ended are reused.
greedy_by_size	Experimental. Packs offsets largest-first, which usually lowers the scratch high-water mark on models whose tensors differ a lot in size.
hill_climb	The default. A local search over greedy-by-size placement orders, whose layout is kept only when its scratch arena in every memory is no larger than greedy’s and their total is strictly smaller; otherwise greedy’s layout is used. Tuned through memory.planner_options (iterations, seed, max_stall_iterations).

memory.planner_options is a JSON object validated against the selected planner’s option schema, so a key the planner does not know is a configuration error, not a silent no-op. Every planner honours the same placement rules: per-tensor memory pins, constant_destination_memory routing and per-memory max_size budgets. The outcomes can still differ, because unpinned scratch spills to the next writable memory when the current one fills, and a different visit order can move a tensor between memories, which can change access latency as well as footprint. Hill-climb minimizes the total high-water mark, breaking ties by per-bank usage in the preferred memory order. It then plans the model with greedy as well and compares the two arena by arena. It keeps its own layout only when its scratch arena in every memory is no larger than greedy’s and the total across memories is strictly smaller; otherwise, including on a tie, it uses greedy’s layout, so no arena grows and no scratch moves into a memory greedy left alone. Individual tensors can still move between offsets and memories. When only one of the two layouts fits the budgets, that one is used, and when neither fits, the conversion fails with the search’s error.

Every planner is deterministic. The greedy planner walks the operators in the order the model gives them, and hill-climb’s search is seeded, so a conversion is reproducible: the same model, target and rules produce the same offsets and the same plan hash. Each planner fills the three arena families independently. Scratch arenas are reused by liveness, persistent arenas are bump allocated and never reclaimed, and constant arenas are keyed by the memory the kernels read them from.

Alignment is resolved per tensor as the largest of three floors: the platform’s min_alignment, the natural alignment of the element type, and any stronger hint the operator asked for, which is how dot-product kernels and Ethos-U command streams get the boundaries they need.

Planners are a registered extension point rather than a fixed list. A planner is a subclass of AirMemoryPlanner with a NAME, registered through register_memory_planner, and the emitter reaches it through the structural protocol in helia_aot.memory.backend, so a planner published in another package does not have to import the code generator to be usable. The published memory API reference covers the planning data contracts; the planner implementation and registration internals are not part of that public API manifest.

Next

Set placement rules or inspect residency reports.