Xolotl

Built from Strata, and grown a long way past it. Niko1221's engine, server and core kernels are still at the heart of it (merged at 0.1.44.1), but Xolotl adds more than 70,000 lines on top: its own DeepSeek-V4.1, V4-Flash and GLM-5.3-Flash engines, the Qwen3.8-27B dense engine (grown from Mirko Perrone's port), the LFM3-Omni engine, fast mode, Xolotl Direct, and the quantisation, expert-tiering and serving work behind every model here.

Latest

• 8 Nov - Short agent turns (a tool result on a long conversation) are faster on DeepSeek-V4.1 (4.40 -> 3.83 s for 161 tokens) and GLM-5.3-Flash (5.47 -> 4.63 s for 246 tokens): the CPU computes some of the experts while the card runs the rest, and the CPU's i-quant kernels now decode an expert once for all its tokens (-32 % at 4-8 tokens). GLM-5.3-Flash decode is 28.5-29 tok/s since the RAM upgrade. Default on; numbers in the CHANGELOG.
• 8 Nov - Qwen3.8-27B decode: MTP now stays on with thinking off (52 -> 111 tok/s), one-pass verify attention (+7-10 %) and an n-gram draft chain (--ngram-chain: 137 -> 173 tok/s when rewriting a file). Same text, bit for bit.
• 8 Nov - Gauntlet v3 is the main table: every model on the same 6 hard agentic tasks. Our DeepSeek-V4.1 and Qwen3.8-Flash-Next builds score at or above their own full-precision APIs on it.
• 8 Nov - The Qwen3.8-27B dense engine is xolotl-dense now (built on Mirko Perrone's dense port); strata-dense stays as a symlink, so nothing breaks.
• 7 Nov - DeepSeek-V4.1 fast mode (opt-in): 41 tok/s on real agent tasks against 32.6, the same Gauntlet score.
• 7 Nov - Xolotl Direct (Linux, experimental): the NVMe drive writes experts straight into a GeForce card's VRAM, 3.1 ms per expert instead of 11.
• 7 Nov - DeepSeek-V4.1 loads in 44 s instead of 96, same output.
• 6 Nov - +8 % decode on RTX 40 cards, +13-17 % prefill on IQ1_M layers, +11 % DeepSeek-V4.1 long-prompt prefill (ideas from Basalt). All on by default and bit-exact.
• 2-3 Nov - LFM3-Omni-2B natively (text, pictures, audio and video in, speech out), and as the ears and voice of Qwen3.8-Flash-Next on one 4090.

Everything else, with PR numbers and who did it: docs/xolotl/CHANGELOG.md.

Fast mode (DeepSeek-V4.1)

DeepSeek-V4.1's decode speed is set by how many expert bytes cross the RAM bus and the NVMe drive per token. Fast mode keeps a 2-bit (Q2_K) copy of the two biggest expert formats (about half of those bytes) in a ~40 GB sidecar file and reads that instead. The model file is not changed; turning it off is unsetting one variable.

DeepSeek-V4.1 V4, RTX 4090	Decode, agent tasks (tok/s)	Decode, 8K prompt (tok/s)	Gauntlet score	Next-token agreement
normal	32.6	26.1	4.99	87.0 %
fast mode	41.1 (+26 %)	33.0 (+27 %)	4.99	86.2 %

build-xolotl/ds41_generate -m V41-XQ-1017-v4.gguf --build-shadow /nvme/V4.fast.x41s   # once, ~30 min CPU (or download v4-fast/ from the model's HF page)
XOLOTL_SHADOW=$(cat /nvme/V4.fast.x41s.list)  <your usual serve command>

It is lossy, so it is opt-in: on the agent tasks it cost nothing measurable, but it changes about one token in a hundred, so keep normal mode where every token counts. Details: docs/XOLOTL_FAST.md. The idea is compress-then-correct from PROMETHEVS (Ryan Flavell).

What one PC can do

One RTX 4090 (24 GB) · Ryzen 7 5700X · 110 GB DDR4 · one NVMe drive. Every number is measured on that machine, through the real server, while the model works through the Gauntlet v3-small: 6 hard agentic coding tasks graded by hidden tests, 10 turns each, thinking on (32K-token budget), each model as it is served. Decode is the generation speed during those tasks.

Model	Size	Score (of 6)	Decode (tok/s)	Prefill (tok/s)	Context	API score
DeepSeek-V4.1-Flash-1017 V4	552B MoE	4.99	32.6	289-409	512K	4.49
DeepSeek-V4.1 V4, fast mode	552B MoE	4.99	41.1	258-439	512K	-
GLM-5.3-Flash	320B MoE	to come	28.5-29	312-450	512K	5.87
Qwen3.8-Flash-Next XQ 45 GB	125B MoE	4.49	157	3,800-4,070	200K	3.90
Qwen3.8-27B	27B dense	5.50	105	1,420-1,630	200K	4.47

Score (of 6) = each task's progress summed (1 per solve, partial credit from the hidden tests), averaged over the runs. API score = each model's own full-precision API through OpenRouter (DeepSeek-V4.1-Flash-1017, GLM-5.3-Flash, Qwen3.8-Flash, Qwen3.8-27B), the same tasks and turns, reasoning on (8 Nov). Qwen3.8-27B: one run on 8 Nov after that day's decode fixes (MTP with thinking off, one-pass verify attention, --ngram-chain); before them it scored 4.47 twice at 89 tok/s. GLM-5.3-Flash: no Gauntlet run yet (score to come); its decode is from an agent replay on 8 Nov (a 25K-token conversation with tool-output turns, Maya 512K as served; since the RAM upgrade every expert sits in RAM, 23 tok/s before it), prefill 454 tok/s on a fresh 25K prompt. Prefill = a fresh 6K-17K-token prompt (tools/throughput.py), DeepSeek-V4.1's with its dense half at Q8_0. In the Gauntlet DeepSeek-V4.1 runs with its dense half at Q6_K, as it is served. More numbers, the Q8_0 references, the speed log and the older HumanEval table (every model 19-21/21 there, so it no longer separates them): docs/xolotl/BENCHMARKS.md.

Smaller cards. The engine fits the work to the card, and what doesn't fit lives in RAM and on the SSD, so RAM matters as much as VRAM. Community runs on 12 GB cards: DeepSeek-V4.1 at 8-10 tok/s (RTX 3060, 46 GB RAM), Qwen3.8-Flash-Next at 39-48 tok/s (RTX 3060 / 4070). The runs, and what your machine needs: docs/WILL_IT_RUN.md.

Quick start

Linux (or Windows 11 for DeepSeek-V4.1, docs/WINDOWS.md), an NVIDIA GPU, CUDA 12.x/13.x, and as much RAM as you can.

git clone https://github.com/Indras-Mirror/Xolotl.git xolotl && cd xolotl
bash tools/xolotl/fetch-llama.sh   # llama.cpp at the pinned commit + tools/xolotl/ggml/xolotl-ggml.patch -> third_party/llama.cpp
cmake -S . -B build-xolotl -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON -DSTRATA_GGML_CUDA=ON \
  -DSTRATA_MMQ_KQUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=89 -DSTRATA_GGML_DIR=$PWD/third_party/llama.cpp -DSTRATA_DSV41_FQ_OP=ON \
  -DSTRATA_DSV41_HC_MIX=ON
ninja -C build-xolotl      # strata, xolotl-dense, glm_generate, ds41_generate, ds4_generate

Get a model:

• DeepSeek-V4.1-Flash: the 1017 builds of the October checkpoint (V4 recommended, with v4/rank-1017-v4.bin), or the earlier GSQ build (152 GiB, mixed ternary / 2-bit / 3-bit). Both cards have the download and Engram steps. Optional for V4: the fast-mode sidecar (~40 GB), built once with ds41_generate --build-shadow.
• Qwen3.8-Flash-Next: XQ 45 GB (the two .gguf shards plus pack-qfn45/).
• Qwen3.8-27B: any dense Qwen3.8-27B GGUF. python tools/dense_setup.py --download UD-Q4_K_M --build --arch 89 fetches one and writes its run config; add --mtp --ngram-chain to its engine args for the fast (still exact) decode.

Then serve it (edit the model paths in the DeepSeek and GLM configs first):

python3 tools/v41/serve/serve_ds41.py --engine strata --config tools/v41/serve/strata-ds41.json --port 8150           # DeepSeek-V4.1-Flash
XOLOTL_SHADOW=$(cat /nvme/V4.fast.x41s.list) python3 tools/v41/serve/serve_ds41.py ...                                # the same in fast mode (V4)
python3 tools/glm/serve/serve_glm.py  --engine strata --config tools/glm/serve/strata-glm-maya-abl-512k.json --port 8144  # GLM-5.3-Flash
./scripts/xol-qfn.sh --serve-only --model <shard 1 .gguf> --pack <pack-qfn45>                                       # Qwen3.8-Flash-Next (port 8099)
python3 -m serve.server --engine strata --config <your-27b.json> --port 8090                                          # Qwen3.8-27B (exe: build-xolotl/xolotl-dense)

Then point any OpenAI- or Anthropic-compatible client at http://127.0.0.1:<port>/v1.

How it works, in one paragraph

A mixture-of-experts model only uses a few experts per token. Xolotl keeps every expert somewhere - the hottest in VRAM, tens of GiB in a pinned RAM arena the GPU reads by DMA, the rest on NVMe read at the drive's ceiling - and moves the ones each token's router asks for, a layer ahead. Each model adds its own tricks: DeepSeek-V4.1 gets a calibrated mixed-precision quantisation, speculative decoding with exact sampling, a bit-exact 4-bit KV cache for 512K context, an arena that adapts to the conversation, an opt-in fast mode that reads a 2-bit copy of its biggest experts, and Xolotl Direct, which lets the NVMe drive write experts straight into VRAM; GLM-5.3-Flash gets its abliteration LoRA on every path and soft pruning; the Qwen models get MTP speculative decoding. Read the full write-up.

Read more

Doc	What's in it
How Xolotl works	the three memory tiers, per-model engines, the server, the agent-loop work
Will it run on my machine?	GPU, RAM and disk needs for DeepSeek-V4.1
Benchmarks	every measured number, exactness checks, how to measure your own PC
Changelog	everything that landed, newest first, with the PR numbers and who did it
DeepSeek-V4.1-Flash engine	GSQ quantisation, DSpark, the fp4 KV cache, Engram
DeepSeek-V4.1 fast mode	the 2-bit expert sidecar: build, serve, switches, what it costs and what we tried
DeepSeek-V4.1 on Windows	native MSVC + CUDA build, configs, RTX 50 notes
GLM-5.3-Flash engine	KDA + MLA attention, the LoRA on routed experts, soft pruning
LFM3-Omni engine	pictures, audio, video and speech on a 2B model, park/unpark beside a big one
Qwen3.8-27B dense howto	dense vs MoE paths, binaries, configs, MTP (by Mirko Perrone, also in Italian)
Setting up with an agent	the build-and-serve steps for an AI agent (or a person) setting Xolotl up
Qwen3.8 / upstream Strata	the Strata engine this grew from
History	earlier session notes, kept as written
Brand kit	logo, banner, palette

Credits

Built on Strata by Niko1221 and ggml / llama.cpp, with the Qwen3.8-27B dense port by perronemirko. Model weights by DeepSeek, Zhipu AI, Qwen and Liquid AI, with quants and fine-tunes by peasantsmith, ukisai, d0xin and gcsa-abliterix. Thanks to every contributor: NektoDron, Xiaomi2008, AlexeyMatskevich, paul123637-lab, kelheor, exye137, penink-WH, cpacsm75, jesdga95 (Basalt) and Ryan Flavell (PROMETHEVS). Who did what: CREDITS.md.

Xolotl is Quetzalcoatl's dark twin, who carries the dead up from the underworld - as the engine carries experts up from RAM and NVMe. Research code from one developer, measured on one PC; reports from other hardware welcome. MIT.

Support

If Xolotl's been useful to you and you want to help support what I do, you can shout me a coffee on Ko-fi. Every bit helps keep this going :)

Write-ups, benchmarks and news about Xolotl: indrasmirror.au.