xinfer

---

✨ Why xInfer?

	Feature	Details
0️⃣	Zero Python dependencies	Pure Rust backend — no PyTorch, no CUDA Python bindings
⚡	Fast	Native Flash Attention, FlashInfer, CUDA Graphs, continuous batching, prefix caching, PD disaggregation. Up to 197 tok/s decode for 30B+ models on consumer GPUs
🪶	Tiny footprint	Core scheduling + attention logic in < 5 000 lines of Rust
🌍	Cross-platform	CUDA (Linux/Windows), Metal (macOS). Same binary, same API
🏭	Production-ready	OpenAI/Anthropic-compatible APIs, built-in ChatGPT-style Web UI, MCP tool calling, structured outputs, embedding + tokenizer endpoints, multi-token prediction (MTP)
🗜️	Aggressive KV compression	TurboQuant (2–4 bit KV cache) extends context up to 4.3× with minimal quality loss. Run 30B+ MoE models with millions of context on single 24/32 GB GPUs
🔥	V100 + NVFP4	First-ever NVFP4 + low-bit KV cache on V100 — no hardware FP4 needed, coherent output on legacy GPUs
🐍	Lightweight Python bindings	Optional PyO3 wheel when you need a Python entry point

---

📽️ Demo

---

🚀 Quick Start

📦 Install

Option 1 — Install DEB or Python package

curl -sSL https://guoqingbao.github.io/xinfer/install.sh | bash

Option 2 — npm

npm install -g xinfer-ai

---

▶️ Run

Using HuggingFace Model ID:

xinfer --m Qwen/Qwen3.6-27B-FP8 --kvcache-dtype turbo4 --ui-server

Using local model path:

xinfer --m /home/Qwen3.6-35B-A3B --d 0,1 --ui-server

Python usage:

# python3 -m xinfer.chat
python3 -m xinfer.server --m Qwen/Qwen3.6-27B-FP8 --kvcache-dtype turbo4 --ui-server

Speculative decoding (MTP & DFlash2)

Built-in MTP (model must include MTP heads):

xinfer --m Qwen/Qwen3.5-35B-A3B --d 0,1 --ui-server --num-speculative-tokens 3

External DFlash2 draft model:

xinfer --m Qwen/Qwen3.8-... --d 0,1 --ui-server \
  --draft-model <dflash2-draft-id-or-path> --num-speculative-tokens 7

See docs/speculative_decoding.md for details.

🧩 MoE Expert Offloading

MoE expert offloading is supported on CUDA only. Run a large MoE model with expert weights offloaded to host RAM. The value is a fraction: 0.2 keeps 20% of each layer's experts on the GPU; auto selects the cache size automatically. Tensor parallelism requires NCCL.

XINFER_MOE_OFFLOAD=0.2 xinfer \
  --m Qwen/Qwen3.5-35B-A3B-FP8 --d 0,1 --ui-server

See the MoE expert offloading guide for configuration and TP details.

Tip: Open http://IP:8001 for the built-in chat UI, or use http://IP:8000/v1/ as your API Base URL.

---

🗜️ KV Cache Compression

Add --kvcache-dtype to compress KV cache and extend context length:

Flag (--kvcache-dtype)	Compression	Quality	GPU Requirement
(default)	1× (BF16)	Baseline	All
fp8	2×	Near-lossless	SM70+ / Apple M1
turbo8	2.6×	79–100% throughput	SM70+ / Apple M1
turbo4	3.7×	Best balance	SM70+ / Apple M1
turbo3	4.7×	Max compression	SM70+

📈 Performance

Tested on V100-32G, A100-40G, Hopper-80G and RTX 5090

Model	Format	Size	Decoding Speed (without MTP)
Ministral-3-3B (Multimodal)	ISQ (BF16→Q4K)	3B	193.67 tokens/s
Qwen3-VL-8B-Instruct (Multimodal)	Q8_0	8B	112.51 tokens/s
Llama-3.1-8B	ISQ (BF16→Q4K)	8B	133.10 tokens/s
DeepSeek-R1-0528-Qwen3-8B	Q4_K_M	8B	161.60 tokens/s
GLM-4-9B-0414	Q4_K_M	9B	92.10 tokens/s
QwQ-32B	Q4_K_M	32B	55.65 tokens/s
Qwen3-30B-A3B	NVFP4	30B (MoE)	197.29 tokens/s (RTX 5090)
Qwen3-30B-A3B	NVFP4	30B (MoE)	72.86 tokens/s (V100, Software FP4)
Qwen3-Next-80B	AWQ	80B (MoE)	90 tokens/s (Hopper)
Qwen3.5/3.6/3.8 27B (Multimodal)	Q4_K_M/IQ4	27B (Dense)	61.9 tokens/s
Qwen3.5/3.6/3.8 27B	FP8	27B (Dense)	52.2 tokens/s (Hopper)
Qwen3.6-35B-A3B (Multimodal)	FP8	35B (MoE)	120 tokens/s (Hopper)
GLM4.7 Flash	NVFP4	30B (MoE)	79 tokens/s (Hopper, Software FP4)
Gemma4-31B	ISQ (BF16→Q4K)	31B (Dense)	55 tokens/s (Hopper)
Gemma4-26B-A4B	NVFP4	26B (MoE)	137.23 tokens/s (RTX 5090)
MiniMax-M2.5	NVFP4	229B (MoE)	64.50 tokens/s (Hopper, Software FP4, TP=2)

Model	Batch Size	Output Tokens	Time (s)	Throughput (tokens/s)
Qwen3-0.6B (BF16)	128	63488	83.13s	763.73
Qwen3-0.6B (BF16)	32	15872	23.53s	674.43
Qwen3-0.6B (BF16)	1	456	9.23s	49.42
Qwen3-4B (Q4_K_M)	1	1683	52.62s	31.98
Qwen3-8B (Q2_K)	1	1300	80.88s	16.07
Qwen3.5-4B (Q3_K_M)	1	1592	69.04s	23.06
Qwen3.5-2B (NVFP4)	1	1883	60.76s	30.99
Qwen3.5-2B (NVFP4)	2	3942	81.96s	48.10

Full benchmarks →

---

🧠 Supported Models

• ✅ LLaMa (LLaMa2, LLaMa3, LLaMa4, IQuest-Coder)
• ✅ Qwen (Qwen2, Qwen3)
• ✅ Qwen2/Qwen3 MoE
• ✅ Qwen3 Next
• ✅ Qwen3.5/3.6/3.8 Dense/MoE (27B, 35B, 122B, 397B, Multimodal model)
• ✅ Qwen3.8-Next/Qwen4
• ✅ Mistral v1, v2
• ✅ Mistral-3-VL Reasoning (3B, 8B, 14B, Multimodal model)
• ✅ GLM4 (0414)
• ✅ GLM4 MoE (4.6/4.7)
• ✅ GLM4.7 Flash
• ✅ GLM 5.2 (DeepSeek V3.2 DSA architecture)
• ✅ DeepSeek V3/R1/V3.2
• ✅ DeepSeek V4 (support 2-bit ISQ via --isq w2)
• ✅ Phi3 / Phi4 (Phi-3, Phi-4, Phi-4-mini, etc.)
• ✅ Gemma3/Gemma4 (Multimodal model)
• ✅ Qwen3-VL (Dense, Multimodal model)
• ✅ MiroThinker-v1.5 (30B, 235B)

Formats: Safetensors (BF16, FP8-blockwise, GPTQ, AWQ, MXFP4, NVFP4) | GGUF (all quant types) | ISQ (on-the-fly quantization, including 2-bit W2 for DeepSeek V4 MoE)

DeepSeek V4 Flash + 2-bit ISQ: --isq w2 re-packs MXFP4 routed experts to 2-bit on load. With 2× GPU tensor parallel (--d 0,1), model weights use about 46 GB per GPU (46 × 2 GB total).

---

TurboQuant KV Cache — Run 30B+ Models on Consumer GPUs

TurboQuant compresses KV cache to 2–4 bits via Walsh-Hadamard transform rotation + per-head absmax quantization. Max context tokens with turbo4:

Model	KV budget	BF16	turbo4	Gain
Qwen3.6-35B-A3B (NVFP4)	7 GB (24 GB GPU)	700k	2.7M	3.9×
	15 GB (32 GB GPU)	1.5M	5.8M	3.9×
Qwen3.6-27B (FP8)	7 GB	112k	434k	3.9×
	15 GB	240k	930k	3.9×
Qwen3-30B-A3B (Q4_K_M)	7 GB	74k	281k	3.8×
	15 GB	160k	602k	3.8×
Gemma4-26B-A4B (NVFP4)	7 GB	32k	125k	3.9×
	15 GB	70k	271k	3.9×

Hybrid models (Qwen3.6) have fewer full attention layers, making TurboQuant especially effective. MLA models (DeepSeek, GLM4.7 Flash) use fp8 instead. The KV budget in the table is the theoretical maximum; actual usage can only utilize up to 90% of the KV budget (--kv-fraction 0.9), leaving room for runtime and batching buffers.

# 35B MoE on single 24/32 GB GPU
xinfer --m unsloth/Qwen3.6-35B-A3B-NVFP4 --kvcache-dtype turbo4

# Production precision
xinfer --m Qwen/Qwen3.6-35B-A3B-FP8 --kvcache-dtype fp8

# 27B Dense + turbo4
xinfer --m Qwen/Qwen3.6-27B-FP8 --kvcache-dtype turbo4

# 26B Gemma4 (local model, occupy more kvcache with --kv-fraction)
xinfer --w /data/gemma-4-26B-A4B-it --ui-server --port 9000 --kv-fraction 0.8

# 30B MoE GGUF + turbo4
xinfer --m unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF \
  --f Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --kvcache-dtype turbo4

# Metal/MacOS
xinfer --m unsloth/Qwen3.5-4B-GGUF --f Qwen3.5-4B-Q4_K_M.gguf

---

📘 Usage

For Python installaion, running model with python3 -m xinfer.server

For Docker builds, refer to Run xInfer in Docker →

Running Models

Tip: By default, xInfer starts an OpenAI-compatible API server at http://localhost:8000. Add --ui-server to also launch the built-in ChatGPT-style Web UI at http://localhost:8001.

# FP8 model (sm90+ with cutlass) + web UI
xinfer --m Qwen/Qwen3.6-27B-FP8 --ui-server

# Unquantized Safetensors (multi-GPU)
xinfer --d 0,1 --m Qwen/Qwen3-30B-A3B-Instruct-2507 --kvcache-dtype fp8

# ISQ on-the-fly quantization
xinfer --m Qwen/Qwen3.6-35B-A3B --isq q4k

# DeepSeek V4 Flash (2-bit ISQ MoE, ~46 GB weights/GPU on 2× TP)
xinfer --d 0,1 --m /path/DeepSeek-V4-Flash --isq w2 --kv-fraction 0.7 --ui-server

# NVFP4 model
xinfer --m unsloth/Qwen3.6-27B-NVFP4

# MXFP4
xinfer --m olka-fi/Qwen3.5-4B-MXFP4

# GGUF model (4-bit KvCache)
xinfer --m unsloth/Qwen3.5-27B-GGUF --f Qwen3.5-27B-Q4_K_M.gguf --kvcache-dtype turbo4

# Multi-shard 5.2 GGUF (auto-detected from HF subfolder)
xinfer --d 0,1,2,3 --m unsloth/Qwen3.5-122B-A10B-GGUF --f Q3_K_M --kvcache-dtype fp8

# FP8 on Metal
xinfer --m Qwen/Qwen3.5-4B-FP8

# Gemma4 26B (NVFP4)
xinfer --m unsloth/gemma-4-26b-a4b-it-NVFP4

# MLA model (GLM4.7 Flash)
xinfer --m GadflyII/GLM-4.7-Flash-NVFP4

# Interactive CLI chat
xinfer --i --m unsloth/Qwen3.5-27B-GGUF --f Qwen3.5-27B-Q4_K_M.gguf

# Faster GDN prefill on Hopper with slight precision loss
SM90_LOWER_PRECISION_GDN_PREFILL=1 xinfer --m Qwen/Qwen3.5-35B-A3B-FP8

# AWQ (pack-quantized)
xinfer --m cyankiwi/Qwen3-Coder-Next-AWQ-4bit

# MultiNode: GLM 5.2 (DeepSeek V3.2 architecture, FP8)
# Master node
xinfer --d 0,1,2,3,4,5,6,7 --m zai-org/GLM-5.2-FP8 --num-nodes 2 --node-rank 0 --master-addr 192.168.xxx.xxx
# Other node(s)
xinfer --d 0,1,2,3,4,5,6,7 --m zai-org/GLM-5.2-FP8 --num-nodes 2 --node-rank 1 --master-addr 192.168.xxx.xxx

# ISQ Q4K + FP8 KV cache
xinfer --m Qwen/Qwen3.6-35B-A3B --isq q4k --kvcache-dtype fp8

# ISQ Q4K + TurboQuant KV cache
xinfer --m Qwen/Qwen3.6-35B-A3B --isq q4k --kvcache-dtype turbo4

# Metal ISQ
xinfer --m /path/Qwen3-4B --isq q6k

# DeepSeek V4 Flash: 2-bit W2 ISQ for routed MoE experts (~46 GB weights/GPU, 2× TP)
xinfer --d 0,1 --m /path/DeepSeek-V4-Flash --isq w2 --kv-fraction 0.7

# Multi-GPU GGUF (model ID + subfolder)
xinfer --d 0,1,2,3 --m unsloth/Qwen3.5-122B-A10B-GGUF --f Q3_K_M

# Remote GGUF — single-file model
xinfer --m unsloth/Qwen3.5-27B-GGUF --f Qwen3.5-27B-Q4_K_M.gguf

# Local GGUF file (direct path)
xinfer --d 0,1 --m /path/Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf

# Local GGUF folder
xinfer --d 0,1,2,3 --m /path/to/Qwen3.5-122B-A10B-GGUF/Q3_K_M

# turbo4: 4-bit K+V — 3.7× compression, best tradeoff
xinfer --m Qwen/Qwen3.6-27B-FP8 --kvcache-dtype turbo4

# turbo3: 3-bit K + 4-bit V — 4.7× compression
xinfer --m Qwen/Qwen3.6-27B-FP8 --kvcache-dtype turbo3

# turbo8: FP8 K + 4-bit V — 2.6× compression, highest quality
xinfer --m Qwen/Qwen3.6-27B-FP8 --kvcache-dtype turbo8

# 35B MoE (NVFP4 + turbo4) — fits on single 24 GB GPU
xinfer --m unsloth/Qwen3.6-35B-A3B-NVFP4 --kvcache-dtype turbo4

# 30B MoE (GGUF Q4_K_M + turbo4) — consumer GPU
xinfer --m unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF \
  --f Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --kvcache-dtype turbo4

# Upload images via built-in Chat UI or send image_url in API requests

# Qwen3.6 35B MoE (FP8, multimodal)
xinfer --m Qwen/Qwen3.6-35B-A3B-FP8 --ui-server

# Qwen3-VL 8B (GGUF)
xinfer --m unsloth/Qwen3-VL-8B-Instruct-GGUF --f Qwen3-VL-8B-Instruct-Q8_0.gguf --ui-server

# Gemma4 26B MoE (NVFP4, multimodal)
xinfer --m unsloth/gemma-4-26b-a4b-it-NVFP4 --ui-server

# Mistral-3 VL 3B (BF16, multimodal)
xinfer --m mistralai/Ministral-3-3B --ui-server

---

📘 Build from source code

Option 1 — Cargo

# Prerequisites: Rust compiler, CUDA Toolkit (optional) or Metal Xcode command line tool
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
sudo apt-get install -y git build-essential

export XINFER_REPO="https://github.com/guoqingbao/xinfer"
# MacOS/Metal: replace features to `metal`
# SM_70/SM_75 (e.g., V100): remove `flashinfer` and `cutlass` features
cargo install --git $XINFER_REPO xinfer --features cuda,nccl,flashinfer,cutlass

Option 2 — Docker

# Turing/V100 (sm_70/sm_75): remove `flashinfer` and `cutlass` features
./build_docker.sh "cuda,nccl,flashinfer,cutlass"

See Docker guide →

pip install maturin maturin[patchelf]

# FlashInfer backend (SM80+)
./build.sh --release --features cuda,nccl,flashinfer,cutlass,python

# Flash Attention backend
./build.sh --release --features cuda,nccl,flashattn,cutlass,python

# macOS Metal
maturin build --release --features metal,python

# Install
pip install target/wheels/xinfer*.whl --force-reinstall

See more Python examples →

---

🌐 Multi-Node Tensor Parallelism

Distribute inference across multiple machines using TCP-based NCCL bootstrap. No MPI required.

# Node 0 (master, 192.168.1.100): scheduler + API
xinfer --d 0,1,2,3 --m /data/DeepSeek-R1/ \
  --num-nodes 2 --node-rank 0 --master-addr 192.168.1.100 --ui-server

# Node 1 (worker, 192.168.1.101): forward-only daemon
xinfer --d 0,1,2,3 --m /data/DeepSeek-R1/ \
  --num-nodes 2 --node-rank 1 --master-addr 192.168.1.100

All nodes must have model weights locally and be TCP-reachable on --master-port (default 29500). See Get Started for details.

---

🔀 Prefill-Decode Disaggregation

Split prefill (prompt processing) and decode (token generation) across GPUs or machines. Eliminates decode stalls during long-context prefilling. PD Server and PD Client must use same KvCache type (--kvcache-dtype). API request(s) must send to PD Client and the PD Server only process internal prefill requests sent from PD Client.

Mode	Config	Use Case
Local IPC	(default, no flag)	Same machine, CUDA
File IPC	--pd-url file:///path	Docker containers, shared volume
Remote TCP	--pd-url tcp://host:port	Different machines

Local IPC (multirank)

# PD Server (prefill GPU, default port 7000)
xinfer --d 0,1 --m Qwen/Qwen3-30B-A3B-Instruct-2507 --pd-server

# PD Client (decode GPU + API)
xinfer --d 2,3 --m /path/Qwen3-30B-A3B-Instruct-2507 --isq q4k --ui-server --port 8000 --pd-client

Multinode (tcp mode)

# Server machine (192.168.1.100)
target/release/xinfer --d 0,1 --m Qwen/... --pd-server --pd-url tcp://0.0.0.0:8100

# Client machine
target/release/xinfer --d 0,1 --m /path/... --pd-client --pd-url tcp://192.168.1.100:8100 --ui-server --port 8000

Metal/macOS requires --pd-url (no LocalIPC support).

mkdir -p /tmp/pd-sockets

# Server container
docker run --gpus '"device=0,1"' -v /tmp/pd-sockets:/sockets ...
target/release/xinfer --d 0,1 --m Qwen/... --pd-server --pd-url file:///sockets

# Client container
docker run --gpus '"device=2,3"' -v /tmp/pd-sockets:/sockets ...
target/release/xinfer --d 0,1 --m /path/... --pd-client --pd-url file:///sockets --ui-server --port 8000

---

🔌 MCP Tool Calling

xinfer --m unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF \
  --f Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --ui-server --mcp-config ./mcp.json

MCP documentation →

---

🔌 Structured Outputs & Guided Decoding

Constraint-based generation via llguidance — JSON Schema, regex, Lark grammars, choice lists.

# Enable structured outputs and tool grammar
xinfer --m Qwen/Qwen3.6-27B-FP8 --ui-server --enable-tool-grammar

Structured outputs documentation →

---

📚 Documentation

Guide	Description
Get Started	Build, run, and configure
Docker	Container builds and deployment
Performance	Full benchmark tables
Prefix Cache	Automatic KV cache reuse
Multimodal	Vision-language models
Embedding	Text embedding API
Tokenizer API	Tokenize / detokenize endpoints
Tool Parsing	Tool call detection and parsing
MCP Integration	Model Context Protocol
Guided Decoding	Structured outputs
Rust Crate	Use as a library
Add a Model	Port a new architecture (AI-assisted)
Test a Model	Validate model quality (AI-assisted)
CUDA Precision Probes	Isolate CUDA kernel precision problems

Using Agents under xInfer backend: xbot · OpenCode · Kilo Code · Claude Code · Goose

---

⚙️ CLI Reference

Flag	Description
--m	Model source: HuggingFace model ID, local Safetensors directory, local GGUF file, or local folder containing GGUF files
--w	Legacy alias for local Safetensors directory; prefer --m <local_dir>
--f	Local GGUF file when used alone; remote GGUF filename when paired with --m <model_id>
--d	Device IDs (e.g. --d 0,1)
--ui-server	API server + built-in ChatGPT-style web UI
--server	API server. Bare: 0.0.0.0:8000; or host[:port], file:///path, socket:///path, unix:///path
--i	Interactive CLI chat
--isq	On-the-fly quantization: q2k, q3k, q4k, q5k, q6k, q8_0, w2 / moe_w2
--kvcache-dtype	KV cache quantization: fp8, turbo8, turbo4, turbo3
--max-num-seqs	Max concurrent requests (default: 32, macOS: 8)
--max-tokens	Max tokens per response (default: 16384)
--kv-fraction	GPU memory fraction for KV cache
--cpu-mem-fold	CPU swap memory ratio (default: 0.5)
--pd-server	Run as PD prefill server
--pd-client	Run as PD decode client
--pd-url	PD connection URL (tcp://, http://, file://)
--disable-prefix-cache	Disable prefix caching
--prefix-cache-max-tokens	Cap prefix cache size
--prefill-chunk-size	Cap prefill chunk size (default: CUDA 8K, Metal: 4k)
--disable-cuda-graph	Disable CUDA graph capture
--yarn-scaling-factor	YARN RoPE context extension factor
--temperature	Sampling temperature (0–1)
--top-k / --top-p	Top-k / nucleus sampling
--presence-penalty	Penalize repeated tokens (−2 to 2)
--frequency-penalty	Penalize frequent tokens (−2 to 2)
--mcp-config	MCP servers JSON config
--mcp-command / --mcp-args	Single MCP server command + args
--num-speculative-tokens	Speculative draft tokens per decode step. Enables built-in MTP on models with MTP heads (e.g. Qwen3.5). Example: --num-speculative-tokens 3
--draft-model	External DFlash2 draft model (HuggingFace id or local directory). When set, uses DFlash2 instead of built-in MTP.

Environment Variables

Variable	Description
XINFER_NVFP4_FORCE_LUT=1	Force software NVFP4 decode to use the LUT-based dequantization path (higher precision) instead of hardware FP4 intrinsics on Blackwell (SM100+). Useful when decode precision matters more than peak throughput.
XINFER_ENABLE_FLASHMLA	Enables DeepSeek V4 sparse MLA acceleration when set to 1 or true. It is disabled by default (unset), so the precision-preserving custom BF16 sparse-attention kernel is used.
XINFER_SSE_BUFFER_SIZE=256	Size of the bounded SSE streaming buffer per client connection (default: 256). Increase for slow network proxies or high-throughput models.
SM90_LOWER_PRECISION_GDN_PREFILL=1	Enable the FlashInfer SM90 persistent kernel for GatedDeltaNet (GDN) prefill on Hopper GPUs (SM90). Delivers faster prefill speedup for Qwen3.5/3.6, with a slight precision trade-off.

Example (Blackwell with high-precision NVFP4 decode):

XINFER_NVFP4_FORCE_LUT=1 xinfer --m nvidia/Qwen3-30B-A3B-FP4 --ui-server

Example (Hopper with faster GDN prefill):

SM90_LOWER_PRECISION_GDN_PREFILL=1 xinfer --m Qwen/Qwen3.5-35B-A3B-FP8 --ui-server

---

🛠️ Roadmap

• Batched inference (Metal)
• GGUF format support
• FlashAttention (CUDA)
• CUDA Graph
• OpenAI-compatible API (streaming support)
• Continuous batching
• Multi-gpu inference (Safetensors, GPTQ, AWQ, GGUF)
• Multi-node tensor parallelism (TCP-based NCCL bootstrap, no MPI required)
• Speedup prompt processing on Metal/macOS
• Chunked Prefill
• Prefix cache (available on CUDA when prefix-cache enabled)
• Model loading from hugginface hub
• Model loading from ModelScope (China)
• Prefix cache for Metal/macOS
• FP8 KV Cache (CUDA, all backends including FlashInfer on SM80+)
• FP8 KV Cache (Metal)
• FP8 KV Cache (with FlashInfer, SM80+)
• TurboQuant KV Cache (2-4 bit compression with WHT rotation)
• FP8 Models (CUDA: MoE, Dense; Metal: Dense)
• GLM 5.2 (DeepSeek V3.2 + DSA)
• DeepSeek V4 (Flash) + 2-bit W2 ISQ MoE (~46 GB weights/GPU × 2)
• Additional model support (Kimi K2, etc.)
• CPU KV Cache Offloading
• Prefill-decode Disaggregation (CUDA)
• Prefill-decode Disaggregation (Metal)
• Built-in ChatGPT-like Web Server
• Embedding API
• Tokenize/Detokenize API
• MCP Integration & Tool Calling
• Prefix Caching
• Claude/Anthropic-compatible API Server
• Support CUDA 13
• Support FlashInfer backend
• Support DeepGEMM backend (Hopper)
• MXFP4/NVFP4 Model Support
• Support Turboquant (4-bit, 3-bit) KvCache
• TentorRT-LLM
• Multi-token Prediction (MTP) & DFlash2 speculative decoding

---

📚 References

• Candle-vLLM
• Python nano-vllm

Star History

Like this project? Give it a ⭐ and contribute!