sglang

sgl-projectApache-2.0活跃维护有专家点评

高性能大模型与多模态模型推理服务框架，通过优化注意力机制和调度策略提升吞吐与延迟表现，支持 Llama、DeepSeek、MoE 等主流模型部署。

SGLang is a high-performance serving framework for large language models and multimodal models.

v0.5.23发布于 2026-11-02 22:14 UTC38,678star +128 / 7 天10,334fork4.52 条点评同步于 2026-11-10 17:20 UTC

推理服务框架大模型推理多模态模型注意力优化 MoE

对比 GitHub 官网 attention blackwell cuda deepseek diffusion

专家点评

上手难度：进阶更新于 2026-11-10

前缀复用做得最好的推理框架，RAG 与多轮对话场景吞吐明显占优

适用场景

RAG 服务：同一份长 system prompt / 检索文档被反复复用，RadixAttention 能省掉大量 prefill
结构化输出：原生支持 JSON schema / 正则约束解码，比在 vLLM 上外挂快很多
多模态推理：对 VLM 的批处理支持比多数框架成熟

踩坑

现象：吞吐反而不如 vLLM。原因：请求之间没有共享前缀（每个 prompt 都不同），RadixAttention 用不上。解法：这种场景回 vLLM
现象：显存占用高于预期。原因：Radix cache 会保留历史前缀。解法：调小 --mem-fraction-static 或关闭 prefix cache

怎么上手

编辑整理

原文

pip install "sglang[all]"
python -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct \
  --port 30000 --tp 1 --mem-fraction-static 0.85

硬件要求编辑整理

- 推理：与 vLLM 同量级；7B FP16 约 16G，70B 4bit 约 40G - 推荐 H100/A100；也支持 AMD ROCm 与部分昇腾（社区分支） - 社区版与 SGLang Cloud 功能有差异，部署前先对版本

下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼

面向大模型与多模态模型的高性能推理服务框架，专为智能体负载和大规模部署优化

高性能大模型与多模态模型推理服务框架，通过优化注意力机制和调度策略提升吞吐与延迟表现，支持 Llama、DeepSeek、MoE 等主流模型部署。

核心能力

内置 SGLang Diffusion 图像与视频生成引擎
支持 LLM、视觉语言模型与扩散模型推理
覆盖智能体负载、大规模服务与 RL rollout 场景
提供 Docker 镜像与 uv 安装两种上手方式
生态含投机解码、音频服务与集群编排集成

容易踩的坑

uv 安装需加 --prerelease=allow 才能装到预发布版本；源码开发需在容器内从仓库根目录执行 pip install -e "python"；具体模型的启动命令需按 Cookbook 中对应硬件选择，README 未直接给出。

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

Highlights

1007 PRs from 257 contributors.

New models

Model	Type	Cookbook
Muse Spark Open	LLM / VLM	link
Beam-1.0	LLM / VLM	link
MiniMax-M3.1-Flash	LLM / VLM	link
MiniCPM-V 4.7	LLM / VLM	link
Kev	LLM / VLM	link
Cosmos3-Nano-Sim (Bimanual, Transfer, Auto)	Diffusion	link
FastH3 Omni	Diffusion	link
FLUX.3 Image Klein	Diffusion	link
HY-Nomas	Diffusion	link
MOSS-Image	Diffusion	link

What's new

DeepSeek-V4.1 on GB300 gets up to 1.5x more prefill throughput under load and about 28% lower first-token latency on 32K to 64K prompts (#44403, #45059, #45107).
SGLang low-latency mode: the opt-in NVIDIA megakernel and AMD monokernel plugins run DeepSeek-V4.1-Flash decode as fused persistent kernels, for up to 44% faster single-user decode on GB300 and 75% on MI350X (#44945, #45450, #44353, #44621).
MoE models such as DeepSeek-V4.1-Flash, GLM-5.3-Flash and gpt-oss reach the first token 30% to 58% sooner on 8K prompts on H200 (#43363).
Paged Experts (opt-in) runs MoE models larger than GPU memory on a single consumer GPU, up to 30x faster than CPU offload (#42557, #45787).
SGLang now runs on Google TPU7x through SGLang-TorchTPU, a PyTorch-native plugin for TPU.
You can now serve LoRA adapters on quantized MoE models (FP8 and NVFP4) with a new opt-in LoRA engine (#45892).
DeepSeek-style MLA models running decode context parallelism can now keep their prefix cache in HiCache file-based L3 storage (#41187).

To upgrade

uv pip install --prerelease=allow sglang==0.5.23

Platform	Docker
NVIDIA (CUDA 13)	`docker pull lmsysorg/sglang:v0.5.23`
AMD MI35x	`docker pull lmsysorg/sglang:v0.5.23-rocm10-mi35x`
AMD MI30x	`docker pull lmsysorg/sglang:v0.5.23-rocm10-mi30x`
Intel GPU	`docker pull lmsysorg/sglang:v0.5.23-xpu`
Intel CPU	`docker pull lmsysorg/sglang:v0.5.23-xeon`
Google TPU7x (TorchTPU)	`docker pull lmsysorg/sglang:v0.1.0-tpu`

Full Release Notes

Speculative Decoding

[Feat] speculative: Seed rejection-sampling draft proposals for deterministic inference: #33395
[Fix] observability: Correct speculative FPM timing and decode lengths: #38711
[Perf] spec: Let the FlashAttention backend capture the draft-extend CUDA graph (about 1.5x decode throughput at batch size 1 with a single-layer MoE drafter): [

v0.5.222026-10-19 · 2026-10-19 02:55 UTC

查看更新说明

Highlights

950 PRs from 245 contributors.

New models

Model	Type	Cookbook
Clef / Clef Flash	LLM / VLM	link
PPLX-Decider v1 / v1.1	LLM / VLM	link
EmbeddingGemma 2	LLM / VLM	link
LLaDA2.2	LLM / VLM
HunyuanImage-3.0-Instruct	Diffusion	link
Kandinsky 6 / Kandinsky 6 SR	Diffusion	link
Wan-Animate-2	Diffusion	link
SANA-Video 2.0	Diffusion	link
LLaDA-Image / LLaDA-Image-Turbo	Diffusion

What's new

DeepSeek-V4.1 gets up to 2.3x more throughput per GPU on H200, and now runs on AMD MI350X / MI355X with up to 1.9x throughput from DSpark (#41251, #41308).
Qwen3.8-Flash-Next decodes 12% faster on B200, and GLM-5.3-Flash with DFlash 10% faster on B300 (#42175, #42489).
MiMo-V2.6-Pro gets 1.85x decode throughput with DFlash on B300 (#41886).
Models load 10x faster with the new InstantTensor loader (#40453).
PD disaggregation gets up to 27% more throughput and half the p95 TTFT on long-prompt workloads (#40703).
HiSparse now supports EAGLE and MTP for long-context serving with the KV cache in CPU memory (#37771).
GLM-5.2 and GLM-5.3 can now combine decode context parallelism with MTP, with up to 2.1x faster decoding on GLM-5.3 (#43735).
SGLang now runs on Apple Silicon Macs with the standard PyTorch backend, alongside MLX (#36780).
MiniMax-H3 and Qwen-Image 2.1 now run in ComfyUI with SGLang Diffusion, which also loads community GGUF checkpoints and LoRAs (#43504, #43581, #42556, #39523, #42174).
Important fix: Hybrid linear models could resume from wrong mamba states when speculative decoding and structured output were used together. Thanks to @Wirick and Baseten for finding it (#43029).

To upgrade

SGLang now requires Python 3.11 or newer.

uv pip install --prerelease=allow sglang==0.5.22

Platform	Docker
NVIDIA (CUDA 13)	`docker pull lmsysorg/sglang:v0.5.22`
AMD MI35x	`docker pull lmsysorg/sglang:v0.5.22-rocm10-mi35x`
AMD MI30x	`docker pull lmsysorg/sglang:v0.5.22-rocm10-mi30x`
Intel GPU	`docker pull lmsysorg/sglang:v0.5.22-xpu`
Intel CPU	`docker pull lmsysorg/sglang:v0.5.22-xeon`

Full Release Notes

Speculative Decoding

[RL][Spec] Introduce top-p mask capture for spec and add DFlash/DSpark impl: #34201
[HiSparse] Add speculative decoding support with multi-step KV cache swapping: #37771
[Kimi-K3][DCP][DSpark] fix eager mode nonetype crash due to no extend_prefix_lens

v0.5.212026-10-02 · 2026-10-02 01:09 UTC

查看更新说明

Highlights

779 PRs from 227 contributors.

New models

Model	Type	Cookbook
DeepSeek-V4.1 Flash	LLM / VLM	link
GigaChat 3.5	LLM / VLM	link
IQuest-Q1	LLM / VLM	link
MiMo-V2.6 / MiMo-V2.6-Pro	LLM / VLM	link
Ling-3.0-flash-VL	LLM / VLM	link
DiffusionGemma	Diffusion	link
Qwen-Image 2.1	Diffusion	link
Anima Base v1.0	Diffusion	link
Ming-Image 0.1 Design / Design-Layer	Diffusion	link
FLUX 3 Action	Diffusion	link

Key features

PD instances can switch between prefill and decode on the fly, no restart needed (#28403).
The prefix cache now runs on a Rust core by default (#39627).
DeepSeek-V4.1 gets 22% faster first token on long prompts (#40352).
Kimi K3 gets 20.6% higher prefill throughput in PD serving (#40045).
GLM-5.3-Flash now runs on AMD MI355X, with FP8 / MXFP4 MoE and MTP speculative decoding (#38546, #39341, #39778).
More accurate results under PP, DP attention, and CP, with layer communication now handled by SGLang (guide, #41557).
New Decisions API (/v1/decisions) turns an LLM / VLM into a low-latency classifier and scorer (docs, #41208).
Score API (/v1/score) scores all candidates in one request (#38965, #41188).
Run MiniMax-H3 with SGLang Diffusion inside ComfyUI (#35990).

To upgrade

uv pip install --prerelease=allow sglang==0.5.21

Platform	Docker
NVIDIA (CUDA 13)	`docker pull lmsysorg/sglang:v0.5.21`
AMD MI35x	`docker pull lmsysorg/sglang:v0.5.21-rocm10-mi35x`
AMD MI30x	`docker pull lmsysorg/sglang:v0.5.21-rocm10-mi30x`
Intel GPU	`docker pull lmsysorg/sglang:v0.5.21-xpu`
Intel CPU	`docker pull lmsysorg/sglang:v0.5.21-xeon`

Full Release Notes

Speculative Decoding

Pipeline parallelism x speculative decoding (EAGLE/MTP) compatibility: #30775
Support XQA backend for SpecDec verify: #32269
[Spec] Windowed draft-decode attention for built-in EAGLE / MTP drafts: [#32673](https://github.com/sgl-project/sglang/pull/32

v0.5.202026-09-18 · 2026-09-18 22:41 UTC

查看更新说明

Highlights

713 PRs from 237 contributors.

New models in this release (see the cookbook for all supported models):

Model	Type	PRs	Cookbook
GLM-5.3-Flash	Autoregressive	#36507, #38621	link
Hy4-Preview	Autoregressive	#36805	link
Qwen3.8-Flash-Next	Autoregressive	#37500	link
K2 Horizon	Autoregressive	#37654, #38033	link
Nanbeige4.2	Autoregressive	#32151
SenseNova-U1.5-8B-MoT	Diffusion	#36606	link
FastH3 (4-step MiniMax-H3 distill)	Diffusion	#37480	link
VDN-H3 (hybrid-attention MiniMax-H3 distill)	Diffusion	#37903	link

Sampling masks for RL rollouts. With return_sampling_mask, each decode step returns the exact token support the sampler drew from and the log-probability of the sampled token under it, so a trainer can replay the rollout without reconstructing top-k or top-p (#36630). Masks now run under overlap scheduling: on Qwen3-8B, decode throughput is 17% higher at batch 1 and 52% higher at batch 64 than the previous implementation. Capacity is set by --sampling-mask-max-tokens (default 4096) (#36631). DisallowedTokensLogitsProcessor is supported alongside masks (#38279).

Unified radix tree. Branching-point caching for the SWA component keeps the sliding-window state at the point where requests fork from a shared prefix, so branches reuse it instead of recomputing. On DeepSeek-V4-Flash with a shared system prompt, token hit rate rises from 43.8% to 60.8% and mean TTFT falls from 1.57 s to 1.07 s (#34565). An opt-in external linker lets the tree address a shared global memory pool through Mooncake or UMBP (#37381).

DSpark under PD with decode context parallelism. A DCP1 prefill can now transfer its DSpark draft KV to a DCP-N decode, so hybrid models such as Kimi-Linear run DSpark in disaggregated, context-parallel serving. Verified on 8x B300 over NIXL and Mooncake up to 256K input (#37709).

Responses API storage is opt-in. /v1/responses no longer retains results in memory unless the server starts with --enable-response-store. Without it, retrieval, previous_response_id chaining, and background requests return 400; PD deployments cannot enable it (#39122).

SGLang Simulator. A CPU-only simulator runs the real scheduler, radix cache, and hierarchical cache with a latency predictor in place of the model forward. Against measured serving traces it predicts TTFT within about 6% on most traces (up to 10% on the longest 32K to 128K ones) and prefix reuse within 0.05 percentage points,

v0.5.192026-09-05 · 2026-09-05 02:27 UTC

查看更新说明

Highlights

786 PRs from 214 contributors.

New models in this release (see the cookbook for all supported models):

Model	Type	PRs	Cookbook
Qwen3.8 (2.4T-A95B)	Autoregressive	#35758 ⭐	link
Qwen3.8-27B	Autoregressive	#34859	link
dots3.note	Autoregressive	#33829 ⭐	link
Ling-3.0-flash	Autoregressive	#33561	link
Ling-3.0-tiny	Autoregressive	#33561	link
Spark2.5	Autoregressive	#35963 ⭐
MiniCPM-SALA	Autoregressive	#30360
Granite 4.2	Autoregressive	#36286	link
LongCat-Image-Edit & Edit-Turbo	Diffusion	#35829

Cookbook updates:

GLM-5.3 deployment guide: link
PaddleOCR-VL deployment guide: link
Kimi-K3 on Ascend A3: #35508
Kimi-K2.7-Code-MXFP4 on MI355X: #36246
Qwen3.5 MXFP4 on MI355X with an FP8 KV cache or a HiCache host-memory tier: #35445, #36245
MiniMax-H3 on a 24 GB GPU or DGX Spark, with a consumer-GPU tuning guide: #36169, #35816
Ling-3.0-flash on DGX Spark: #36364
Qwen3.8-27B on RTX 5090, RTX PRO 6000, and DGX Spark, re-measured: #35825

Beam search. SGLang can now do beam search. Pass beam_width in your request and you get back the n best sequences instead of a single sample. It works out of the box next to regular requests, though it does not yet mix with speculative decoding, disaggregation, DP attention, or HiCache (#31626).

DeepEP v2. DeepEP's new ElasticBuffer engine is available as --moe-a2a-backend deepep_v2 for DeepSeek-V3/V4 and Qwen3-MoE in FP8. Its buffers have a fixed size, so decode can run under CUDA graphs even across nodes. Performance is on par with the classic backend (#35634, #34923).

LayerNorm sequence parallelism. With --enable-layernorm-sp, each tensor-parallel rank normalizes only its own share of the prefill tokens instead of all of them. That takes 3.5% off Qwen3-8B prefill on H100 and 5.6% on B200, and the saving grows with the TP degree. Dense Qwen3 models only for now (#30915).

W4A8 MoE on Hopper. If you serve MXFP4 experts on Hopper, you can now quantize the activations to FP8 as well with --flashinfer-mxfp4-moe-precision fp8. DeepSeek-V4-Flash gains about 12% output throughput with no change in GSM8K accuracy. Needs FlashInfer 0.6.18 ([#34967](h

用户点评

登录用户的真实使用记录，可按版本和硬件筛选

4.5

2 人评分

用户实测硬件

48G 单卡 ×1

共 2 条

erin
2026-11-09
erin 对 sglang 的评价：前缀缓存和 RadixAttention 确实好用，vLLM 的很多坑这里没有。
dave
2026-11-0948G 单卡
团队从 vLLM 迁移过来的，多轮对话场景前缀缓存效果比 vLLM 好不少，主要用 --enable-prefix-caching。代价是模型支持面窄一些：新出的模型往往要等一周适配。如果只用 Llama 和 DeepSeek 系列，体验很好。

README 原文

未翻译

在 GitHub 打开

SGLang: Fast inference for LLMs and multimodal models

SGLang is an open-source inference framework for large language, vision-language, and diffusion models, optimized for agentic workloads, large-scale serving, and RL rollouts. SGLang Diffusion is its built-in image and video generation engine, included in this repository and the sglang Python package.

👋 Get started below, or meet the community at SGLang Events, including meetups, developer meetings, workshops, and office hours.

SGLang Summit 2026: Meet the builders of open AI Our flagship annual gathering · November 12–13 · San Francisco

Go deep on inference, training, and AI hardware through technical talks, live demos, and hands-on workshops. Bring your deployment challenges and meet the engineers and researchers building the stack.

Get Started

Pull the Docker image, which includes SGLang and its dependencies:

docker pull lmsysorg/sglang:latest

Alternatively, install SGLang in an activated Python environment with uv:

uv pip install --prerelease=allow sglang

Next, launch your model:

Quickstart: Run your first model and send a request.
Cookbook: Choose your model and hardware to get a ready-to-run launch command.

Supported Hardware

SGLang supports a wide range of GPUs, TPUs, NPUs, CPUs, and Apple Silicon platforms.

Platform	Representative hardware
NVIDIA	B200/B300/GB200/GB300; H100/H200/H800/H20; A100; RTX 30/40/50 series, RTX 6000 Ada / PRO 6000; DGX Spark, Jetson Orin
AMD	Instinct MI300X, MI325X, MI350X, MI355X
Google TPU	v6e, v7; SGLang-TorchTPU / SGL-JAX
Intel (GPU / CPU)	Arc / Arc Pro B-Series GPUs, Xeon CPUs
Apple Silicon	Macs via Metal / MLX
Huawei Ascend	A2, A3, 950PR/DT NPUs
Moore Threads	MTT S5000 GPUs

Integrations in progress: AWS Trainium, Alibaba T-Head PPU, Cambricon MLU, Qualcomm QAIC, MetaX, Hygon HCU/DCU, Iluvatar CoreX, and more.

SGLang-TorchTPU brings the SGLang serving engine to Google TPUs through PyTorch, torch_tpu, and Pallas kernels. It supports LLM and vision-language serving on TPU7x (Ironwood), including tensor and expert parallelism across hosts. SGL-JAX provides the JAX-based TPU serving path.

See the Cookbook and platform guides for model compatibility and setup.

SGL Ecosystem

Area	Projects	Purpose
Diffusion	SGLang Diffusion	Built into SGLang for image and video generation with diffusion models.
Audio	SGLang Omni	Audio model serving for text-to-speech (TTS) and automatic speech recognition (ASR).
Speculative Decoding	SpecForge	Train draft models for speculative decoding and deploy them with SGLang.
RL and Post-Training	Miles, slime, AReaL, Tunix, verl	Training frameworks that integrate SGLang for rollout generation.
Deployment and Orchestration	SMG, RBG, llm-d, Ray Serve, NVIDIA Dynamo	Deploy and scale SGLang inference services with routing, load balancing, and cluster orchestration.
Education	Mini-SGLang, zero-to-sglang, DeepLearning.AI course	Learn inference engine design and efficient text and image generation through code and hands-on courses.

Development and Contributing

Contributions are welcome, from bug fixes and documentation to model support and performance improvements.

Development setup

Start from the lmsysorg/sglang:dev Docker image, which provides development tools and most dependencies. Clone or mount your SGLang checkout inside the container, then install it in editable mode from the repository root so tests use your local Python changes:

pip install -e "python"

In an activated virtual environment, you can use uv pip install --prerelease=allow -e "python" instead. See the development guide for container setup and testing.

Contribute

Fork the repository and create a branch for your changes. For larger changes, discuss your proposal in a GitHub issue or on Slack.
Make your changes, run the relevant tests, and add regression coverage for fixes or new behavior. Run pre-commit run --all-files before submitting.
Open a pull request describing the change and how you tested it. Include benchmarks or accuracy evaluations when relevant.

See the contributor guide for formatting, testing, and pull request instructions. Documentation contributors can start with the docs guide.

Community and Sponsorship

SGLang is hosted by LMSYS, a non-profit open-source organization.

Community discussions: Join Slack for technical questions and development discussions.
Events: Find meetups, workshops, and office hours on SGLang Events.
Updates: Follow X and LinkedIn for project updates, and the LMSYS Blog for release announcements and technical articles.
Project resources: Explore the documentation, Cookbook, roadmap, release notes, issue tracker, and contributor guide.
Contact Us: For enterprise adoption and deployment, technical consulting, sponsorship, or partnership inquiries, please contact [email protected].
Contributor sponsorship: Long-term active SGLang contributors are eligible for coding agent sponsorship, including Cursor, Claude Code, or OpenAI Codex. To apply, email [email protected] with links to your key commits or pull requests.

Trusted by Industry and Research

SGLang serves production workloads across AI labs, cloud platforms, enterprises, and universities.

Acknowledgment

We learned the design and reused code from the following projects: Guidance, vLLM, LightLLM, FlashInfer, Outlines, and LMQL.

专家点评

适用场景

踩坑

同类对比

怎么上手

项目说明

核心能力

容易踩的坑

最近发版

Highlights

New models

What's new

To upgrade

Full Release Notes

Speculative Decoding

Highlights

New models

What's new

To upgrade

Full Release Notes

Speculative Decoding

Highlights

New models

Key features

To upgrade

Full Release Notes

Speculative Decoding

Highlights

Highlights

用户点评

README 原文

SGLang: Fast inference for LLMs and multimodal models

Get Started

Supported Hardware

SGL Ecosystem

Development and Contributing

Development setup

Contribute

Community and Sponsorship

Trusted by Industry and Research

Acknowledgment

相关项目