halogen-flash-server

halogen-flash-server

halogen™ is the fastest way to run Qwen3.8-Flash-Next on AMD Strix Halo, and it is built for agents: deep into a long session, each turn starts answering in about 1.6 seconds.

Every kernel is written for this one GPU and this one model family. No general-purpose runtime, no portability layer, no fallback path. That is why it can do things a general engine cannot, and why it runs on exactly one piece of silicon.

0.18.0, the default configuration	halogen-flash
decode	up to 94 tok/s on repetitive output · 73 writing code · 54 prose
prefill: 8,192 / 32,768 / 131,072-token prompt	1,821 / 1,740 / 1,675 tok/s
an agent turn, 30,000 to 96,000 tokens of context	4.2 s (first token in 1.6 s)
the first turn of a 34,000-token agent prompt, cold	21 s
decode: editing code / an agent's tool call / six other languages	81 / 71 / 52 tok/s

What each row is. The agent turns are three sessions of a coding agent fixing a bug in a Python repository: each turn sends the whole conversation back with the model's last tool call and a tool result of 800 to 3,000 tokens appended, and the time runs from the request to the last byte of the answer, on the client's clock (the median of 115 turns, about 200 tokens of answer each). Decode is measured on the kind of text each row names. The best case is five short prompts whose answer repeats itself ("print the word red ten times" and four like it), shown so it can be compared like for like with other figures of that kind. Prefill is the image's own sweep, on a synthetic prompt; real text through the chat template runs about 8 to 12% slower, which is why the cold first turn takes 21 s for 34,000 tokens.

All of it is the published 0.18.0 image at its defaults (the v2 checkpoint, the prompt cache on, greedy, thinking off) on an AMD Ryzen AI Max+ 395 with 128 GB, at about 85 W of sustained package power. Measured has every number for both checkpoints. The prefill rows, and a decode mean over ten prompt shapes, reproduce with one command from the image (how); the agent turns and the decode by kind of text are measured from the client side, over the same OpenAI-compatible API you use. Prefill barely decays with depth: 131,072 tokens run at 96% of the 32,768 rate.

Precision: 4.16 bpw across the trunk, the experts and the draft head of the default checkpoint (v2), or 5.25 bpw across all 179.55B parameters with the FP8 n-gram lookup table counted in. Both are measured from the files rather than quoted from a format name. docs/QUANT.md gives the breakdown by tensor family, so it can be checked with arithmetic rather than taken on trust.

At temperature 0, output is byte-identical to serial greedy decode. Speculation here is a pure speed optimization, verified on every release, not a quality trade. Since 0.6.0 there are two draft sources, the model's own draft head and the request's own text (prompt lookup), and the guarantee covers both.

0.20.0: document parsing on the NPU with your own OvisOCR3, the image on ROCm 10.1, faster prefill, and a fix for strict tool calls under concurrency. A numeric release: Flash's outputs can differ from 0.19.1's in the last bits, and the quality gates are unchanged (CHANGELOG).

Since 0.7.0 the engine also opens a llama.cpp GGUF of this model directly: point it at the file you already have (unsloth's UD-IQ4_XS, say) and it runs on these kernels, with the same speculation and the same identity guarantee, and no conversion step. See Bring your own GGUF for which files, and for the numbers.

There is a Discord for questions, for numbers from your own box, and for comparing setups. Bugs and regressions go to issues, where the changelog can credit them. See Community.

---

Documentation

The full documentation is the docs/ tree, one page per task. It is also hosted with search at peonist-ai.github.io/halogen-flash-server. For an AI agent: llms.txt maps every page in one fetch, and AGENTS.md is the short playbook for deploying, driving or debugging the server.

Quickstart	first run on a Strix Halo box, weights included
Give it a machine of its own	what the server holds; recipes if you must share it
API reference	every endpoint, conventions, status codes
Clients	Codex (Responses API) · Claude Code (Messages API) · agent harnesses · sessions: keep, fork, join
Requests	sampling · images · token budgets · as a classifier
Configuration	cache modes, the KV pool, disk cache, 1M context, attention budget, API key, request records
Checkpoints	precision · bring your own GGUF · measuring a checkpoint · QUANT.md
Numbers	measured · quality: what is measured, and what is not
NPU	small models on the NPU · host details
Troubleshooting	will not start, hangs at pool reservation, crawls on long prompts, Fedora/SELinux, the host settings measured on
Reference	every flag · classify endpoint · what this release is not

Quickstart

mkdir -p ~/halogen-models
podman run --rm -p 8731:8731 \
  --device /dev/kfd --device /dev/dri --group-add keep-groups \
  --ipc=host --ulimit memlock=-1:-1 \
  -e HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next \
  -v ~/halogen-models:/models \
  ghcr.io/peonist-ai/halogen-flash-server:0.20.0

The weights come from peonist-ai/halogen-qwen3.8-flash-next on Hugging Face; each NPU model's repository is listed on the NPU page.

One condition first: set the BIOS carve-out (UMA frame buffer) to its minimum, not Auto: this engine wants host RAM, not dedicated VRAM. The Quickstart covers that, Docker, the compose file, and upgrading from an older checkpoint.

Flint preview

A separate image serves RecursiveSI's Flint (40B mixture-of-experts, about 2B active, Apache-2.0) as a preview; :latest above is unchanged. On first start it downloads the int4 weights (about 21 GB) into ~/halogen-models and checks them against the tested revision:

podman run --rm -p 8731:8731 \
  --device /dev/kfd --device /dev/dri --group-add keep-groups \
  --ipc=host --ulimit memlock=-1:-1 \
  -e HALOGEN_SERVE_MODEL=flint \
  -v ~/halogen-models:/models \
  ghcr.io/peonist-ai/halogen-flash-server:0.18.14-flint-preview

It is a preview: prefill slows past roughly 80-90k tokens of context, and speculative decoding is opt-in (HALOGEN_FLINT_SPEC=1) and greedy-only. Speeds, options and limits: Flint preview.

---

Community

• Discord for questions, setup help, numbers from your own machine, and comparisons with other runtimes. If you have run a benchmark against this server, post the command and the curve. A result from a box we do not own is worth more to us than one from ours.
• Issues for bugs and regressions. CONTRIBUTING.md says what to include. The changelog credits the report that drove each fix.
• Security issues go to the maintainers directly, not to Discord or a public issue.

---

License

The engine is distributed under the terms in LICENSE.md. Third-party components and their licenses are listed in THIRD-PARTY-NOTICES.md. Model weights are licensed separately by their original authors.

"halogen" and "Peonist" are trademarks of Peonist, LLC (U.S. application pending). TRADEMARKS.md says how the names may be used; referring to the project, running it, and publishing numbers about it need no permission.