kyojin

Kyojin 1.8.14

• CUDA prefill: tuned dense GEMMs (#169). The large fp16 GEMMs of the reconstruct (prefill) path pick a cuBLASLt configuration on the live operands. A configuration is kept only if its output is bit-identical to cublasGemmEx (no split-K) and it is at least 3 % faster; choices are cached per cuBLASLt version and GPU in ~/.cache/exllamav3/cuda_gemm_tune.txt (EXL3_CUDA_GEMM_TUNE_FILE). On by default; EXL3_CUDA_GEMM_TUNE=0 turns it off. The first start on a new GPU tunes during warm-up (about 2-3 minutes once); later starts read the cache.
• Strix Halo MoE prefill (#166). The sigmoid-gated shared expert is summed inside the grouped expert reduction instead of a separate pass. Output is bit-identical.

Served gate (Strix Halo box for the GLM and Flash-Next cells, RTX 3090 box for the RTX cells; 1 warm-up; RTX cells are the median of 3 reps):

• RTX 3090 Qwen27 prefill with the tuner, A/B on one build (median of 3 per arm): +2.5 % at 5K, +1.7 % at 8K, +1.6 % at 32K.
• Flash-Next prefill A/B for #166 (6 reps per arm): +0.76 % at 8K, +0.04 % at 32K.
• Greedy output is byte-identical to 1.8.13 on 8 of 8 prompts for Flash-Next and for GLM-5.3.
• The RTX decode cells are single-stream with speculative decoding; their per-step time (31.8 ms vs 31.2 ms) and draft acceptance match 1.8.13, and the gap is within the usual per-rep spread of these cells.

Kyojin 1.8.13

• Relaxed acceptance of argmax drafts for sampled requests: EXL3_SPEC_TYPICAL=1 (default off) accepts a draft token when the target gives it enough probability, instead of requiring a sampled match (#164). On GLM-5.3 with T=1 Hermes turns, decode went from 34.4 to 39.9 tok/s (+15.9 %) on one Strix Halo machine and from 34.7 to 40.7 tok/s (+17 %) on another. Tool calls (32 requests per arm) and GSM8K (100) are unchanged within noise.

Served gate (Strix Halo box for the GLM and Flash-Next cells, RTX 3090 box for the RTX cells; 1 warm-up, one rep per cell; greedy, so the new flag does not change these numbers):

No cell is more than 5 % below the reference. Qwen27 RTX 39K decode: the gate and a recheck read 73-75 tok/s on 1.8.13; a same-machine A/B on freshly started lanes then gave 83.1 on 1.8.13 and 84.7 on 1.8.12 (2 reps each, within noise). The cause of the earlier low readings is not identified yet.

Kyojin 1.8.12

• Fixes the hang on start-up with draft2s packs (issue #125): the pack is loaded on the host and then copied, and the draft head is built before warm-up (#161).
• GLM serve stops a job that runs past its token budget or stalls, so an abandoned request no longer keeps the GPU busy (#162).
• Native fdot2 GEMV for gfx1201 (RDNA 4) and a fix for ROCm block-graph capture (#160).
• The skinny hgemm grid is sized for one-column row counts (#154). New GEMV test harness (#155).

Served gate (Strix Halo box for the GLM and Flash-Next cells, RTX 3090 box for the RTX cells; 1 warm-up, one rep per cell):

No cell is more than 5 % below the reference. Flash-Next 128K decode A/B on the same machine: 1.8.12 54.7, 1.8.11 56.2 (2 reps each, within noise).

Kyojin 1.8.11

• Disk-first recurrent cache: the RAM cache for recurrent checkpoints can be capped at 0. Evicted checkpoints go to the prefix disk tier. Strix and RTX lanes use 2 to 5 GiB less RAM. A 61K-token turn-2 resume from disk costs the same on Strix and is faster on RTX.
• One llama.cpp-style timings object on GLM, Qwen, MiMo and DS4.1, with prompt_per_second and predicted_per_second. Progressive timings at most once per second while streaming. One speed line per request in the server log.
• Includes the no_tool_call request field.

Served gate (Strix Halo box for the GLM and Flash-Next cells, RTX 3090 box for the RTX cells; 1 warm-up, medians when a cell needed more reps):

No cell is more than 5 % below the 1.8.10 reference.