vllm

vllm-projectApache-2.0活跃维护有专家点评

面向大语言模型的高吞吐、低显存推理与服务引擎，通过 PagedAttention 等优化显著提升并发吞吐并降低显存占用，适合部署在线推理服务。

A high-throughput and memory-efficient inference and serving engine for LLMs

v0.33.0rc3发布于 2026-11-02 10:10 UTC96,680star +187 / 7 天24,640fork4.52 条点评同步于 2026-11-10 17:17 UTC

推理引擎大模型推理推理服务高吞吐低显存

对比 GitHub 官网 amd blackwell cuda deepseek deepseek-v3

专家点评

上手难度：进阶更新于 2026-11-10

单卡也能压榨出高吞吐的 LLM 推理引擎，PagedAttention 基本成了行业标准

适用场景

在线服务：需要高并发、低首 token 延迟的对话/补全 API
多卡部署：张量并行 + 连续批处理，单节点 8 卡吞吐远超 HuggingFace pipeline
量化推理：原生支持 AWQ / GPTQ / FP8，24G 卡能跑 32B 级量化模型

踩坑

现象：启动即 CUDA OOM。原因：--max-model-len 默认取模型最大长度，KV cache 预留过大。解法：显式设小，并用 --gpu-memory-utilization 0.9 留余量
现象：多卡吞吐不如单卡。原因：PCIe 机型未开 P2P 或 TP 过大。解法：减小 TP、换 NVLink 机型
现象：长上下文首 token 很慢。原因：chunked prefill 默认关闭。解法：加 --enable-chunked-prefill

怎么上手

编辑整理

原文

pip install vllm
# 单卡
vllm serve Qwen/Qwen2.5-7B-Instruct --max-model-len 8192
# 多卡张量并行（TP 数要能整除注意力头数）
vllm serve meta-llama/Llama-3.1-70B-Instruct --tensor-parallel-size 4

硬件要求编辑整理

- 推理：7B FP16 约需 16G 显存；70B AWQ 约需 48G（2×24G 或 1×48G） - 推荐：A100/H100 + CUDA 12.x；RTX 4090 可用，但无 P2P，多卡效率打折 - 不支持 Apple Silicon / 纯 CPU（有 CPU 后端但性能很弱）

下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼

高吞吐、省显存的 LLM 推理与服务引擎，适合把模型跑成线上 API

面向大语言模型的高吞吐、低显存推理与服务引擎，通过 PagedAttention 等优化显著提升并发吞吐并降低显存占用，适合部署在线推理服务。

核心能力

用 PagedAttention 管理注意力 KV 显存
连续批处理、分块预填充与前缀缓存
支持 FP8/INT4/GPTQ/AWQ 等多种量化
提供 OpenAI 兼容 API 服务与流式输出
支持张量/流水线/专家等多种并行推理

容易踩的坑

安装后需额外执行 vllm download-kernels 预编译内核以加快启动；README 未说明默认端口等细节，需查文档；从源码构建需按官方安装文档操作。

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

v0.32.0

Highlights

This release features 814 commits from 363 contributors (121 new)!

Structured decisions and generic parsers: the new POST /v1/systemone endpoint answers choice, noul and score questions over a structured state (#59299), is served by default (#60677, #60696), and also runs on Cloudflare/clef-omni (#61064), LiquidAI/d1-3B (#60676) and DiffusionGemma (#60466). The new hf tool and reasoning parser (--tool-call-parser hf, --reasoning-parser hf) reads the checkpoint's response_template, so models that ship it need no model-specific parser (#58604, #58602), in both the Python and Rust frontends (#59005).
Low-precision KV cache everywhere: NVFP4 KV cache now runs on SM8x and SM12x (#46963), SM90 (#60620) and with XQA decode on SM12x (#60452), and on MiniMax-M3 sparse attention on SM100 (#59300); GLM-5.x defaults to an FP8 KV cache on SM100, halving KV memory per rank with 2.3-5.5% more throughput at a 7-11% TTFT cost in the PR's benchmark (#60140, #60281); fp8_ds_mla on SM90 gives GLM-5.3-Flash up to 31% more output throughput (#59246); ROCm gains the 4-bit ultraquant_4bit KV cache with about 2x the density of FP8 (#57057).
Model Runner V2: a GPU n-gram speculator with variable-length drafts (#40704), Elastic EP (#53934), pipeline parallelism with prefill context parallelism for MLA models (#59139), a stock torch.compile mode (#59079), and MRV2 is now the default on ROCm with the model-specific MRV1 exceptions removed (#56360).
Faster startup and serving: vllm download-kernels installs FlashInfer's precompiled kernels (#58765); vllm preload --preload-autotune keeps the FlashInfer autotune table warm across restarts (#60085); a shared-memory tensor arena broadcasts large multimodal tensors to workers with one copy (#51207); tokenizers and processors load from the local snapshot (#60825); and with --api-server-count above 1 each connection goes to the least-loaded API worker, for 43-85% more req/s on short requests (Qwen3-0.6B, 32 input / 1 output tokens) in the PR's benchmark (#60723).
Model support: new EmbeddingGemma 2 omni-modal embeddings (#60254), MiniCPM-V 4.7 (#58674) and Nemotron 3.5 ASR (#59827); video input in the Transformers modeling backend (#57441); GLM-5.3 sparse MLA index conversions reused across layers (3.5-3.9x on that kernel, #59464) and fused multi-step draft decode (13.3% at concurrency 1, #57443); HiSparse per-request residency caching for up to 58% more throughput on GLM-5.3-NVFP4 on B200 with long, fully GPU-resident contexts at high concurrency (#60083).
Security: fixes for GHSA-4hhp-h66f-j5j7 (chat template resource exhaustion, #50300), GHSA-q97c-p368-75c4 (tokenizer max_token_id, #59491) and GHSA-frvg-qw88-hww9 (P2P ZMQ socket exhaustion, #51504); CORS no longer allows every origin by default (#61009); untrusted images are limited to raster formats (#60022); shared-memory multimodal cache handles are signed (#59357); and JSON schemas nested deeper than 64 levels are rejected (#60036).
Breaking changes: Python 3.10 dropped (#60402); default CORS origins restricted (#61009); transformers>=5.16.1,<5.20.0 and tokenizers>=0.23.2 (#57387, #59796); torch.compile sequence parallelism and AsyncTP passes removed (#60517); cascade attention removed (#61126); mamba_cache_mode="all" removed (#58997); max_tokens no longer reserved from the context window (#57035); Responses API reasoning stream events renamed (#59652).

Release Artifacts

Python Wheels

Platform	Install
PyPI (CUDA 13.0)	`pip install vllm`
PyPI (CUDA 13.0, uv)	`uv pip install vllm --torch-backend=auto`
ROCm	`pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.32.0/rocm723`
XPU	`uv pip install vllm --extra-index-url https://wheels.vllm.ai/0.32.0/xpu --extra-index-url https://download.pytorch.org/whl/xpu --index-strategy unsafe-best-match`

Docker Images

| Plat

v0.31.02026-10-05 · 2026-10-05 06:44 UTC
查看更新说明
v0.31.0

Highlights

This release features 717 commits from 307 contributors (96 new)!
- DeepSeek-V4.1-Flash performance: FlashMLA mega attention with the V4.1 NVFP4 compressed KV cache is now the SM100 default (#56935); DeepGEMM sparse MQA logits for the indexer (#56254) and Mega-Gate fusing the gate GEMM with expert selection (#56266); decoder boundaries fuse the TP all-reduce, mHC input preparation (#57643) and the MoE finalize (#58586); a fused small-batch WO-A with inverse RoPE and MXFP8 quant on SM100/SM103 (#58634); the MXFP8 wo_b GEMM fused with the sequence-parallel reduce-scatter (#57428); Engram wkv sharded across TP ranks (#58678) and Engram host tables shared across co-located DP replicas by default (#57651); encoder CUDA graphs for the vision tower (#56625); and SWA bounded replay that keeps the sliding-window KV out of prefix caching (#56227).
- Fast restart: the new vllm preload CLI launches the weight-cache daemon that keeps post-quantized weights resident in GPU memory across engine restarts (#56680), now with data parallelism (#57386), MTP draft models (#57312), a /health endpoint (#58552) and a readiness wait (#58370). Experimental initialized-engine snapshots (vllm snapshot create/restore) use CRIU to restore a fully initialized TP1 engine (#51360).
- Model Runner V2 and speculative decoding: draft-model speculative decoding (#43091) and custom logits processors (#56497) on Model Runner V2; the new LiLiCorr drafter (#57934); async scheduling for DFlash (#58065) with the context K/V precompute captured in the draft CUDA graph (#57632); DSpark adaptive verification for Gemma4 (#57263) and variable-length decode for Kimi-K3 (#52988); a DCP target with a non-DCP DSpark draft (#56723); and MoE memory now counted during MRV2 profiling, avoiding OOMs on WideEP deployments (#57270, #58411).
- Large scale serving: MoonEP balanced EP all2all backend via --all2all-backend moonep (#52101), prefill context parallelism with data parallelism (#57075), a low-SM multimem reduce-scatter for SM100/SM103 (#55072), DeepEPv2 with sequence parallelism (#57210) and EPLB with shared-expert overlap (#57236), a sharding-aware NCCL M2N weight-transfer backend for RL (#51520), and KV offloading back-pressure detection (#50045).
- Scheduling controls: --max-num-active-seqs caps RUNNING admission independently of max_num_seqs (#56758), --long-prefill-token-threshold now adapts to the number of waiting prefills instead of chunking a lone request (#57951, #58459), the waiting queue was reworked so requests already holding KV blocks are scheduled first (#58947), and the KV connector + MTP deadlock under KV pressure was fixed (#57104).
- HiSparse hardening: MTP verification rows resolved with a union residency kernel (#59235), MTP acceptance collapse under FULL graphs fixed (#59309), no GPU pages without host backing (#59036), a chunked-prefill preemption livelock fixed (#59494), KV cache sized from the groups HiSparse allocates (#59450), and host prefix publication and GPU prefix adoption fixes (#59007, #59282).
- Security: per-request mm_processor_kwargs and media_io_kwargs are rejected unless --trust-request-mm-kwargs is set (#58830); prefix-cache extra keys are tagged by source so a LoRA name and a cache_salt can no longer collide (#51899), and the LoRA path is part of the block hash (#59335); stale multimodal receiver-cache entries can no longer replace fresh payloads (#57833).
- Breaking changes: per-request multimodal kwargs gated (#58830); tokenizer_mode="slow" removed (#58545); --enable-mamba-fine-grained-prefix-cache renamed to --enable-mamba-shared-prefix-checkpoint (#57382); online quantization through quantization="fp8" replaced by the fp8_per_tensor shorthand (#53585) and Quark silent online quantization removed (#51800); the AllSpark INT8 W8A16 backend removed (#58001); --enforce-eager now also disables JIT kernel warmup (#58197); XPU gra
v0.30.02026-09-22 · 2026-09-22 05:20 UTC
查看更新说明
v0.30.0

Highlights

This release features 762 commits from 315 contributors (104 new)!
- New models: DeepSeek-V4.1-Flash (#56214, #56228, #56208) with the whole KV stored in MXFP8 through the FlashMLA V4.1 record on SM100 (#56893), DeepGEMM Mega-mHC (#56962), and async Engram prefetch with Engram DP sharding (#56512); DeepSeek-V4-Flash-Vision-Exp (#54566), also on ROCm (#55107) and with LoRA (#55897); GLM-5.3-Flash (#53906) with EPLB (#55119); K2-Horizon (#55063); Cohere Compass (#54774); Bailing V3 VL (#55921); Nanbeige4.2 via the Transformers backend (#56071); and a DeepSeek-V4 CPU backend with AVX512/AMX sparse MLA, indexer, mHC and compressor kernels (#55355).
- Fast Start: a persistent per-GPU weight-cache daemon holds post-quantized, TP-sharded weights in GPU memory so restarting engines map them over CUDA IPC with --load-format ipc_cache instead of reloading from disk (#54921), now covering FP4 checkpoints (#55465) and multi-node TP (#55468).
- Watermarking: Gumbel-max watermarked generation and detection with a keyed PRF, per-request opt-out and an example detection endpoint (#54053); dual-key Gumbel-max makes it compatible with speculative decoding (#56122); the Rust frontend forwards the per-request controls (#56338).
- HiSparse: a host-resident tier for sparse-MLA decode that spills KV pages to pinned host memory under GPU pressure and serves top-k misses from a per-request GPU hot buffer, enabled through HiSparseConnector (#53781), with Prometheus counters (#56061), a host cache shared across TP ranks (#56629), and the attention config inferred from the connector (#57041).
- Model Runner V2: dual-batch overlap in eager mode (#50945) and with FULL CUDA graphs for microbatched steps (#51700); MTP (#46994) and EAGLE3/DFlash/DSpark (#50514) speculative decoding under pipeline parallelism; adaptive verification for every draft-model speculator through an online acceptance estimator (#52228); gc frozen during graph capture, cutting capture from 12s to 2s and engine init from 28.9s to 8.2s on H200 (#54646); --return-sampling-mask compacted on GPU, fixing an about 2x RL step-time regression (#54901).
- Qwen3.8-Flash-Next performance: separate prefill and decode QSA indexer kernels (#54513), fused PLE kernels (#54517), FP8 indexer cache (#54890), padded-index skipping in sparse GQA (#54873), fused PLE residual and QSA output gate (#55309), UVA PLE offload and Engram tensor parallelism via --engram-config (#54371), and torch.compile removed from the NVIDIA implementation so FP8 fits on a single GB300 (#55272).
- Kimi K3 performance: native CUDA AttnRes default on SM100 (#54261), KDA mixed-batch gather/scatter removed (5.2-7.7% E2E throughput, #56159), grouped FP8 MLA cache insertion (4-6x kernel speedup at small batch, #55356), DSV3 low-latency GEMM on strided tensors (12-81% kernel speedup, #54565), overlapped TP8 KDA projections (#54697), FlashInfer KDA kernels (#55364), internal prefix checkpoints with partial prefix caching and speculative decoding (#53614), and symmetric DCP disaggregation for hybrid Mamba models (#55531).
- Large scale serving: PCP+DCP on sparse-MLA models (#56157), PCP with single-module MTP and replicated DSpark (#56107) and decode-only FULL CUDA graphs (#53867), Elastic EP reusing CUDA graphs across reconfiguration (#54985), an opt-in FlashInfer PCIe IPC all-reduce for NVLink-less boxes (#53576), DeepEP v2 async finalize overlapping shared experts with combine (#52781), Mooncake Store heterogeneous TP sharing (#53129), a KVCR secondary-tier adapter (#53624), and encoder-cache sharing over NIXL (#47941) and Mooncake (#41567).
- Quantization: targeted online quantization through quantization_config.targets (#51285) and on partially pre-quantized checkpoints from any quant method (#51392), W4A16 DSA with the nvfp4_fp8_ds_mla KV cache (#51724), FlashInfer CuTeDSL NVFP4 W4A16 default over Marlin on SM100/103 (#53014), NVFP4 in the torch
v0.29.02026-09-09 · 2026-09-09 08:54 UTC
查看更新说明
v0.29.0

Highlights

This release features 594 commits from 277 contributors (91 new)!
- Model Runner V2 is now the default for all models (#53183), completing the rollout that began with pooling models (#48290). MRV2 also gained CUDA graph memory profiling for KV cache auto-sizing (#53306), batch-sharded sampling that cuts per-step logits memory by 1/TP (#50465), prompt embeds (#42963), extract_hidden_states speculation (#49811), padded FULL cudagraph dispatch for uniform decode under spec decode (#53407), and DP-sync skipping before EAGLE/MTP draft prefill (#53694). MRV1 remains in use for a few ROCm models and features MRV2 does not yet support.
- New models: Hy4-preview, Tencent's 770B/49B-active MoE with Gated DeepSeek Sparse Attention and native MTP (#54160); Qwen3.8-Flash-Next with BF16/FP8/NVFP4 and MTP (#53896); GraniteSWA and GraniteMoeSWA (#52706); NemotronH_Omni_Reasoning_V3 with MTP (#52929, #53121); Kimi K3 NVFP4 checkpoints (#53132).
- Kimi-K3 and DeepSeek V4 performance: fused MXFP4 top-k finalization in the K3 latent tail (about 5% E2E latency, #53152), K3 Mamba metadata preparation in one Triton launch (6.6-7.6x kernel speedup, #52388), tuned Hopper low-latency GEMM (#54088) now also dispatched on SM100 (#53534) and used for eh_proj (12.9-25.2% kernel speedup, #53942), GEMM-RS extended to GEMM-AR (#53053), MLA gate merged into the QKV-A projection (#54015), K3 DCP with DSpark (#52188) and DCP partial prefix cache hits (#50493); DeepSeek V4 shared experts fused into MegaMoE (#53040), adaptive top-k width re-landed (#52823), a native SwiGLU clamp kernel for Humming MoE (#53685), and an opt-in FlashInfer moe_ep expert backend (#49636).
- Speculative decoding: per-request acceptance stats in OpenAI API responses via --per-request-spec-decode-metrics (#48915), adaptive verification extended to logprobs (#52242), SM100 sparse MLA for GLM-5.2 (#52783) and DeepSeek V4 on SM90 (#52795), Qwen3-Omni DSpark drafts (#52560), PLaMo3 EAGLE-3/DFlash (#54239), and DFlash2 loading from speculators format (#53797).
- RL weight sync: a new sharded_rdt P2P backend where each worker pulls only its TP/EP slice over NIXL or Ray Direct Transport (#43375), rank-local IPC weight updates (#52497), sparse checkpoint-coordinate updates through native weight loaders (#50723, #53751), and routed expert loading for gpt-oss (#52209).
- Mamba prefix caching: internal prefill checkpoints deliver a 9%-25% TTFT improvement (#52789); prefix_cache_retention_interval is now a CLI argument defaulting to 0 (#52216), with dense retention automatically restored for hybrid models using EAGLE/MTP (#55760, #55861).
- New defaults: FlashInfer all-reduce enabled by default for TP CUDA groups, opt out with VLLM_ALLREDUCE_USE_FLASHINFER=0 (#52998); prefix-cache NONE_HASH is deterministic by default so distributed KV cache users no longer need to pin PYTHONHASHSEED (#51875); new --max-num-queued-reqs / --max-num-queued-tokens admission-control flags (#49445).
- Breaking changes: ten deprecated model architectures removed (#53608); FlexOlmo, Olmo3 and Hunyuan V1/VL migrated to the Transformers modeling backend (#53615); PyAV video decoder backend removed (#54231); python -m vllm.entrypoints.openai.api_server deprecated in favor of vllm serve (#52131); VLLM_TEST_FORCE_FP8_MARLIN (#52182) and VLLM_ROCM_USE_AITER_FP4_ASM_GEMM (#53141) removed.
:warning: Model Runner V1 Deprecation

Now that Model Runner V2 is used by default, we are considering Model Runner V1 deprecated and are targeting v0.32 for its removal. We do not intend to accept any more MRV1-specific improvements or optimizations.

Some features are not yet supported in MRV2 but we are planning for these gaps to be closed within the next 2-3 weeks. These include sequence parallelism, dual-batch overlap, elastic expert parallellism, custom logits processors and cer

用户点评

登录用户的真实使用记录，可按版本和硬件筛选

4.5

2 人评分

用户实测硬件

24G 单卡 ×1

共 2 条

carol
2026-11-09@ v0.33.0rc324G 单卡
vLLM 在 24G 单卡上跑 7B 模型很稳，连续压测 48 小时没有 OOM。PagedAttention 对长上下文帮助明显，同样显存能多容纳大概一倍并发。注意 --gpu-memory-utilization 默认 0.9 偏激进，和别的进程共享 GPU 时建议降到 0.85。
dave
2026-11-09
dave 对 vllm 的评价：功能全、社区活跃，缺点是依赖比较重，镜像两三个 G。

README 原文

未翻译

在 GitHub 打开

🔥 We have built a vLLM website to help you get started with vLLM. Please visit vllm.ai to learn more. For events, please visit vllm.ai/events to join us.

About

vLLM is a fast and easy-to-use library for LLM inference and serving.

Originally developed in the Sky Computing Lab at UC Berkeley, vLLM has grown into one of the most active open-source AI projects built and maintained by a diverse community of many dozens of academic institutions and companies from over 2000 contributors.

vLLM is fast with:

State-of-the-art serving throughput
Efficient management of attention key and value memory with PagedAttention
Continuous batching of incoming requests, chunked prefill, prefix caching
Fast and flexible model execution with piecewise and full CUDA/HIP graphs
Quantization: FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, compressed-tensors, ModelOpt, TorchAO, and more
Optimized attention kernels including FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA, and Triton
Optimized GEMM/MoE kernels for various precisions using CUTLASS, TRTLLM-GEN, CuTeDSL
Speculative decoding including n-gram, suffix, EAGLE, DFlash
Automatic kernel generation and graph-level transformations using torch.compile
Disaggregated prefill, decode, and encode

vLLM is flexible and easy to use with:

Seamless integration with popular Hugging Face models
High-throughput serving with various decoding algorithms, including parallel sampling, beam search, and more
Tensor, pipeline, data, expert, and context parallelism for distributed inference
Streaming outputs
Generation of structured outputs using xgrammar or guidance
Tool calling and reasoning parsers
OpenAI-compatible API server, plus Anthropic Messages API and gRPC support
Efficient multi-LoRA support for dense and MoE layers
Support for NVIDIA GPUs, AMD GPUs, Intel GPUs, and x86/ARM/PowerPC CPUs. Additionally, diverse hardware plugins such as Google TPUs, Intel Gaudi, IBM Spyre, Huawei Ascend, Rebellions NPU, Apple Silicon, MetaX GPU, Iluvatar CoreX GPU, and more.

vLLM seamlessly supports 200+ model architectures on Hugging Face, including:

Decoder-only LLMs (e.g., Llama, Qwen, Gemma)
Mixture-of-Expert LLMs (e.g., Mixtral, DeepSeek-V3, Qwen-MoE, GPT-OSS)
Hybrid attention and state-space models (e.g., Mamba, Qwen3.5)
Multi-modal models (e.g., LLaVA, Qwen-VL, Pixtral)
Embedding and retrieval models (e.g., E5-Mistral, GTE, ColBERT)
Reward and classification models (e.g., Qwen-Math)

Find the full list of supported models here.

Getting Started

Install vLLM with uv (recommended) or pip:

uv pip install vllm
vllm download-kernels  # precompiled kernels for faster startup

Or build from source for development.

Visit our documentation to learn more.

Contributing

We welcome and value any contributions and collaborations. Please check out Contributing to vLLM for how to get involved.

Citation

If you use vLLM for your research, please cite our paper:

@inproceedings{kwon2023efficient,
  title={Efficient Memory Management for Large Language Model Serving with PagedAttention},
  author={Woosuk Kwon and Zhuohan Li and Siyuan Zhuang and Ying Sheng and Lianmin Zheng and Cody Hao Yu and Joseph E. Gonzalez and Hao Zhang and Ion Stoica},
  booktitle={Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles},
  year={2023}
}

Contact Us

For technical questions and feature requests, please use GitHub Issues
For discussing with fellow users, please use the vLLM Forum
For coordinating contributions and development, please use Slack
For security disclosures, please use GitHub's Security Advisories feature
For collaborations and partnerships, please contact us at [email protected]

Media Kit

If you wish to use vLLM's logo, please refer to our media kit repo

vllm

专家点评

适用场景

踩坑

同类对比

怎么上手

项目说明

核心能力

容易踩的坑

最近发版

v0.33.0

Highlights

v0.32.0

Highlights

Release Artifacts

Python Wheels

Docker Images

v0.31.0

Highlights

v0.30.0

Highlights

v0.29.0

Highlights

:warning: Model Runner V1 Deprecation

用户点评

README 原文

About

Getting Started

Contributing

Citation

Contact Us

Media Kit

专家点评

适用场景

踩坑

同类对比

怎么上手

项目说明

核心能力

容易踩的坑

最近发版

v0.33.0

Highlights

v0.32.0

Highlights

Release Artifacts

Python Wheels

Docker Images

v0.31.0

Highlights

v0.30.0

Highlights

v0.29.0

Highlights

:warning: Model Runner V1 Deprecation

用户点评

README 原文

About

Getting Started

Contributing

Citation

Contact Us

Media Kit

相关项目