Hyperloom

Hyperloom v1.1.4

This is a patch release on top of 1.1.3. Every command now runs behind one hyperloom <command> entry point. A session lost with its container can be resumed elsewhere, and a failing LLM backend can no longer burn hours of a long run. Hyperloom now measures GPU power and scans server logs for runtime findings, and the validated SGLang stack moves to 0.5.21.

Sessions recorded by 1.1.3 resume on this build. How Hyperloom is launched, installed and configured does change, and so do several report fields; see Breaking changes and the upgrade guide before upgrading. The per-change history lives in docs/release-notes.md.

Highlights

1. One hyperloom command, and no flags that do nothing

hyperloom <command> (or python -m hyperloom <command>) is now the only entry point. setup, check, optimize, recover, quantize, session ... and multi-node ... replace about 50 console scripts and python -m paths, and hyperloom --help lists them all (#1736). Flags that were ignored or only advisory now fail at startup instead of being accepted (#1736, #1739, #1786). On resume, preflight checks the session's recorded framework instead of the SGLang default (#1753), and every --extra-env pin is exported so all readers resolve it alike (#1797, #1818).

2. A session lost with its container can be resumed

hyperloom recover --session-dir S --confirm-scope-gone OWNER_SCOPE --confirmation-reason TEXT cancels the unfinished tasks of a vanished execution environment and releases their execution and GPU leases, so optimize --resume-from can continue the session elsewhere. Owner scopes now include the start time of the namespace's PID 1, so a new container on the same node is not mistaken for the lost one. A session resumed in SWEEP re-dispatches its lost concurrency sweep, and a crashed integrate_patch whose tree went with the container rolls back while an accepted env-only candidate is replayed into the stack (#1874, carrying #1859, #1870, #1871 and #1879). A local Ray head without the serving_slot resource is restarted with it when that is safe, and unrecoverable baseline failures end the session early (#1816).

3. Long runs stop burning hours on a broken LLM loop

Prompt sections that grew with the session are now bounded; in the reported run this cut a 1.6M-character prompt that failed with Prompt is too long to about 109K (#1735). A failing LLM backend is retried with exponential backoff from 2 s up to 5 min, and an hour without a successful turn stops the session with stop_reason=backend_unhealthy while CLOSE still publishes the best validated stack (#1737). Every Claude SDK turn is bounded by INFERENCE_OPTIMIZER_CLAUDE_TURN_TIMEOUT_SEC (default 1500 s) and its CLI is killed past the bound (#1754). emit_intent is no longer silently hidden from the model, and 50 consecutive turns without an intent also end the session as backend_unhealthy (#1805).

4. Specialists run beside the loop and draw on earlier sessions

The dispatcher no longer waits on running CPU specialists before the tick advances, and the top untested specialist proposals are benched automatically as one explore grid (#1742). Each workspace runs a local Experience KB service: measured FRAMEWORK_AGENT attempts are published as Experiences and read back into prompts, with optional sync against a shared global KB (#1732). Operators can define their own specialist with --custom-specialist-prompt-file and --custom-specialist-description (#1804). The installer owns the single Claude CLI on PATH, and specialists always run as CLI subprocesses (#1775, #1786).

5. Runtime findings from server logs, and verdicts that say why

Every measured server log is scanned for runtime findings such as unknown vLLM enviro

Hyperloom v1.1.3

This is a patch release on top of 1.1.2. It adds targeted, repository-wide KernelForge fusion discovery, makes ATOM installable on bare metal, tightens AgentX grading, and hardens GPU ownership, phase transitions, recovery and reporting.

Most 1.1.2 sessions resume on this build. A pre-1.1.3 AgentX session does not carry the median-interactivity anchor this release requires and should not be resumed. The command line, environment contract and several Python and report fields also move; see Breaking changes before upgrading.

The full per-change history lives in docs/release-notes.md.

Highlights

1. KernelForge fusion gains targeted, repository-wide discovery

forge-fuse --fuse-kernel anchors a fusion on an exact traced kernel and derives its neighbours from the trace; --repo-scope lets discovery and authoring inspect the whole framework repository when the chain is not in the resolved model file (#1678). Discovery avoids folding work into a tuned library GEMM's epilogue, validates harness alignment, and checks that a keeper reduces launch count. DeepSeek-V4 harnesses also derive the output groups from o_groups, with --attn-tp recording the attention shard independently of serving TP (#1709).

2. ATOM becomes a first-class setup and optimization target

Bare-metal setup accepts --install-framework atom, installs AITER and ATOM from source, and refuses environments where ATOM would interfere with SGLang or vLLM plugins (#1694). The shared 12-hour demos now cover ATOM in Docker and bare-metal modes. User-selected execution and kernel integration are preserved (#1615), and an unset KERNEL_OPT_BACKEND_ORDER resolves to GEAK for ATOM as it does for every other framework (#1693).

3. AgentX grading uses median interactivity with explicit guards

An AgentX KEEP now requires a 3% gain on e2e_norm_intvty_p50, while P90 interactivity, output throughput, measurement-window duration and request errors must hold (#1652). SBD V6 publishes and renders the objective, guards, comparability inputs and submission validity, so the report shows the axes on which the verdict was actually taken (#1687).

4. GPU ownership and phase transitions fail closed

KERNEL_AGENT work holds a GPU lane until its physical execution settles, and phase transitions wait for a GPU barrier (#1654). Ended holders are reclaimed, Ray waits are bounded without making a still-owned device available, and shutdown no longer waits on its own caller (#1595, #1682). Stack validation stops when it cannot undo its patches instead of continuing from an unknown tree (#1677).

5. Workflow evidence becomes replayable

Fresh sessions publish a versioned workflow-evaluation contract with replayable transition inputs, dispatch provenance and separate process and business outcomes (#1666, #1688). A per-process trajectory ledger records LLM requests, tool events and lifecycle transitions and projects them onto Langfuse spans (#1697). AIPerf KV-cache telemetry is recovered from the endpoint that actually serves it (#1698).

6. AgentX and KernelForge use the remote knowledge store consistently

AgentX gains its own remote Recipe identity, selection metric, evidence and warm-replay path (#1597). Experience rows are stored in the shape their producer writes (#1561), while KernelForge remote experience storage converges on KB Store and refuses the GBrain configuration that previously ran as a silent no-op (#1614).

7. Measurement and publication stop inventing success

Graph-capture fallbacks that changed the measured workload are removed (#1609). Fusion, GEMM tuning and the critic now report unavailable checks, missing measurements and outages as such rather than as PASS, 1.00x or ACCEPT (#1628, #1629, #1635, #1636, #1639). A failed JIT rebuild stops the loop instead of benchmarking a stale binary (#1634), and CLOSE

Hyperloom v1.1.2

This is a patch release on top of 1.1.1. The session record resumes across the upgrade: ensure_schema migrates a 1.1.1 coordinator.db on the way in.

ROCm 10 becomes a validated stack, bring-up gets a phase of its own, and the automatic supervision layer is retired in favour of explicit resume and recovery. ATOM is supported end to end, KernelForge gains an assembly backend and stops losing work a killed campaign had already proven, and the Session Breakdown now says which axis a session was graded on.

The command line, the environment contract, the installation prerequisites and several report fields do move. The ones that need a plan before upgrading are listed under Breaking changes. CHANGELOG.md is retired (#1589); the full per-change list now lives in docs/release-notes.md.

Highlights

1. ROCm 10 is a validated stack

Docker runs SGLang 0.5.20 and vLLM on ROCm 10 user space, bare-metal vLLM builds from source on a ROCm 10 host, and rocprof-compute is installed and found on TheRock's pip-packaged ROCm (#1549, #1593, #1605). A prebuilt SGLang image whose torch fails under rocprofv3 is marked profiling-unavailable instead of hanging the round. SGLang >= 0.5.18 profiles through TraceLens kernel_shape_tool rather than git-applied roofline patches (#1486, #1593). The six-leg ROCm 10 run also surfaced scheduler stalls that are fixed here: KERNEL entry hooks no longer block the coordinator tick, conc-sweep boot rounds are no longer counted as measured variants, and a trace with zero GPU events stops repeat roofline scheduling with a recorded reason (#1605).

2. ENABLEMENT is the sixth phase of the loop

Bring-up leaves FRAMEWORK_AGENT and sits between PRELUDE and FRAMEWORK_AGENT with its own entry and exit, phase_history rows and report section (#1491). It gets no budget share: it is bounded by the wall clock and ENABLEMENT_MAX_ATTEMPTS, and runnability is decided from the measurement rather than a log scan. A KEEP now ships an ordered replay recipe (setup, build, patch) with a sufficient / insufficient verdict under enablement.recipe in session_breakdown.json (#1414), and a KB recipe carrying code overlays can be replayed into a pip-installed framework (#1563).

3. Automatic supervision is retired; limits and recovery are explicit

The Robustness agent, the runtime recover action and the Monitor/Supervisor auto-resume are removed (#1547). Each benchmark spawn has a hard deadline (INFERENCE_OPTIMIZER_BENCHMARK_TIMEOUT_SEC, default 7800) plus a silence limit (INFERENCE_OPTIMIZER_BENCHMARK_SILENCE_TIMEOUT_SEC, default 600) armed only once a ready marker is seen. Resume refuses ownership it cannot verify instead of guessing, and recover-session --confirm-stopped releases a stopped task's residue under the session lock. Inline MCP actions no longer block the coordinator's event loop (#1577).

4. ATOM is supported end to end

Its recovery workers are reaped, --framework atom defaults the kernel backend to forge, and it is published as supported (#1416). Detailed profiling adds --mark-trace and ATOM's annotation switches only when the installed build supports them (#1509).

5. KernelForge gains an assembly backend and keeps more of its work

forge-loop --kernel-backend assembly optimizes compiler-emitted AMDGPU assembly behind a numerical contract (#1426). Warm starts fall back to a fuzzy KB match across framework versions and GPUs (#1512). A lane's patch is merged against the KEEPs that landed before it instead of being dropped on stale context (#1528), Controller patches integrate in priority order with clean patch hygiene (#1583), and a Controller KEEP reaches the stack ledger in every benchmark mode (#1522). A killed campaign's proven fusions are salvaged instead of reported as REVERT (#1533, #1536), its leftovers no longer fail the next task

Hyperloom v1.1.1

This is a patch release on top of 1.1.0. The session record does not move, so a session recorded by 1.1.0 resumes on this build.

Most of it is the Recipe knowledge base telling the truth: the prior work a session had actually earned was not reaching the model at all — an exact warm-start hit printed "first session for this workload/hw", and the lessons and pitfalls sections rendered (none) no matter how much a prior session had learned. The other corrections are in agent-backend selection, which three places answered differently, and in the supervisor watchdog, whose restarts are now resumable and bounded.

The command line and the environment contract do move: one optimizer option, one console script and eight environment variables that 1.1.0 accepted are gone, one variable's accepted values narrow, and one option is added. The ones that need a plan before upgrading are listed under Breaking changes; the full per-change list is in CHANGELOG.md.

Highlights

1. The warm-start block, the KB's lessons, and its pitfalls reach the model again

All three read field shapes no writer produces. An exact hit carrying a full config printed (no recipe text — first session for this workload/hw), and sections 5b and 5c rendered (none) regardless of what a prior session had recorded (#1466, #1464). The block now renders warm_start_context, the view recipe_kb_t0 already persists on every anchor, and a borrowed config is labelled with the model it came from. No recorded knowledge was lost; until now none of it was being shown. A losing operator's progress is also banked rather than discarded, and a warm start is picked by measurement (#1448).

2. One rule picks the agent backend across both packages

A configured credential first, then an installed SDK, with Claude ahead of Codex. Four places answered this and three disagreed, so forge-loop on an OpenAI-only host could resolve to Claude and then fail to authenticate (#1472). The Robustness Agent's RCA engine follows the same precedence instead of checking the OpenAI side unconditionally. One consequence worth naming: a deployment whose runtime is logged in by other means is no longer refused up front — forge-fuse dropped its --agent-backend auto usage error and forge-fusion dropped the llm_provider_unconfigured result, leaving the real authentication failure to the preflight that can see it.

Forge also stops naming its own settings: it reads Hyperloom's environment contract (CLAUDE_MODEL / CODEX_MODEL) and nothing above it, and reasoning effort falls back to the project-wide HYPERLOOM_REASONING_EFFORT (#1442). One reasoning-effort ladder now covers both surfaces — see Breaking changes.

3. Supervisor watchdog restarts are resumable and bounded

A wedged coordinator takes SIGHUP and keeps its interrupted phase segment rather than recording a session outcome; three restart attempts, counted durably before the signal goes out, make the wedge terminal (#1462). Separately, the robustness monitor now reads the real stop-reason vocabulary — it had been importing a module that does not exist and silently falling back to a subset missing 16 terminal reasons, so a finished session could be relaunched. Long executor subprocesses also move off the event loop (#1471).

4. --extend-hours grants a resumed session more budget

Elapsed time is summed forward across every leg and never reset, so this is the only way to continue a run that has already spent its budget. The grant is applied on --resume-from and recorded in the session state with its reason; it defaults to 0.0, so an invocation that does not pass it behaves as before.

5. The collective lane becomes part of the rewrite controller

A communication operator and a dense operator are the same rewrite job; the only thing separating them was the measurement harness, which belon