dynamo

ai-dynamo活跃维护

面向数据中心的分布式推理服务框架，用 Rust 编写，支持 vLLM、SGLang、TensorRT-LLM 等后端，解决大规模 LLM 与扩散模型在生产环境下的解耦部署和请求路由问题。

A Datacenter Scale Distributed Inference Serving Framework

v1.6.2发布于 2026-11-05 22:11 UTC8,464star +11 / 7 天1,838fork同步于 2026-11-10 11:31 UTC

分布式推理推理服务框架解耦部署请求路由 Kubernetes

对比 GitHub 官网 diffusion disaggregated-serving kubernetes llm-inference omni

怎么上手

由 README 提炼，执行前请确认

原文

docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.6.0
uv pip install --prerelease=allow "ai-dynamo[vllm]"

硬件要求按 README 说明

README 未说明具体显存要求；容器方案需 NVIDIA GPU（docker run --gpus all），官方提供 vLLM/SGLang/TensorRT-LLM 的 NGC 运行时镜像。

需要 CUDA有 Docker 镜像

下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼

没人工点评？写下你的经验

在 vLLM/SGLang/TensorRT-LLM 之上做多节点推理编排，把单机引擎变成数据中心级推理系统

核心能力

分离式 serving 与智能路由协同调度多节点推理
多级 KV 缓存复用跨节点请求的上下文
按 SLA 自动扩缩容，K8s 单清单零配置部署
Rust 写核心保性能，Python 做扩展
兼容 vLLM、SGLang、TensorRT-LLM 三种后端

容易踩的坑

PyPI 安装需加 --prerelease=allow，且要先装 uv；TensorRT-LLM 后端需用 pip 并加 --extra-index-url https://pypi.nvidia.com；启动 frontend 和 worker 都要显式指定 --discovery-backend file，否则服务发现不通。

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

Release Notes

Dynamo v1.7.0-kimi-k3.1-pro-dev.1 is an experimental snapshot build with early Kimi-K3.1-Pro support on the Dynamo vLLM and SGLang backends. It is not recommended for production and is not a QA-gated release. APIs, behavior, and defaults may change before stable support. Use it for evaluation, testing, and early feedback only.

Summary

Dynamo v1.7.0-kimi-k3.1-pro-dev.1 adds serving for moonshotai/Kimi-K3.1-Pro on Dynamo vLLM and SGLang, including Vera Rubin (VR200). The published containers are built on public Kimi-specific vLLM and SGLang base images pinned by digest (CUDA 13.0 on amd64, CUDA 13.4 on arm64 for Rubin) and carry backend compatibility, multimodal, and request-validation fixes for Kimi serving (#18124).

Kubernetes recipes cover vLLM aggregated and disaggregated profiles on GB300 (with multimodal overlays for image and video input) and VR200, a vLLM aggregated profile on B200, and experimental SGLang aggregated and disaggregated profiles on GB300, with DSpark speculative decoding, shared model-cache manifests, and an AIPerf AgentX benchmark under recipes/kimi-k3.1-pro (#18330, cherry-pick of #18083; #18346, cherry-pick of #18255).

Release Branch: release/1.7.0-kimi-k3.1-pro-dev.1, tip 229be5cff1156859b0e3e43b598ee800b2f81ccc (2026-11-06) Base: main at 45e30e07a4f4cef37ab7bfa739f7bee0c89b3057 (2026-10-27) vLLM: vllm/vllm-openai:kimi-k31 base (CUDA 13.0), kimi-k31-cu134 on arm64 (CUDA 13.4), NIXL v1.5.0 SGLang: lmsysorg/sglang:dev-kimi-k31-pro base (CUDA 13.0), dev-kimi-k31-pro-cu134 on arm64 (CUDA 13.4), NIXL v1.5.0

Container Images

Backend	Arch	Image
vLLM (CUDA 13.0 / 13.4)	multi-arch (`amd64` + `arm64`)	`nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.7.0-kimi-k3.1-pro-dev.1`
SGLang (CUDA 13.0 / 13.4)	multi-arch (`amd64` + `arm64`)	`nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.7.0-kimi-k3.1-pro-dev.1`

Backend Versions

Backend	Source	CUDA	Python	Notes
vLLM	Dynamo-built runtime on the public Kimi vLLM images (`kimi-k31` for `amd64`, `kimi-k31-cu134` for `arm64`), pinned by digest	13.0 (`amd64`), 13.4 (`arm64`)	3.12	NIXL v1.5.0 (required for Kimi strided descriptors); vLLM-Omni disabled to preserve the model image's Transformers stack
SGLang	Dynamo-built runtime on the public Kimi SGLang images (`dev-kimi-k31-pro`, `dev-kimi-k31-pro-cu134` for `arm64`), pinned by digest	13.0 (`amd64`), 13.4 (`arm64`)	3.12	NIXL v1.5.0

Published frontend crates: renderer 8.2.2-kimi-k3.1-pro-dev.1, tokenizers 3.1.4-kimi-k3.1-pro-dev.1, protocols 8.5.0, legacy parsers 10.2.2.

Models

Model	Precision	GPU	Notes
`moonshotai/Kimi-K3.1-Pro`	see model card	GB300, VR200, B200	target model

About Kimi-K3.1-Pro

Kimi-K3.1-Pro is a Moonshot AI model published on Hugging Face as moonshotai/Kimi-K3.1-Pro. This build serves it through Dynamo's vLLM and SGLang backends, with image and video input through the GB300 vLLM multimodal overlays and DSpark speculative decoding on the B200 and VR200 recipes.

Full Changelog

vLLM Backend

Kimi-K3.1-Pro runtime stack: Moved the vLLM runtime to public Kimi vLLM base images pinned by digest, with per-architecture selection (CUDA 13.0 on amd64, CUDA 13.4 on arm64 for Rubin) and NIXL v1.5.0 for Kimi's strided-descriptor KV transfer ([#18124](https://github.com/ai-dyna

v1.6.2Dynamo v1.6.2最新正式版2026-11-05 · 2026-11-05 22:11 UTC
Dynamo v1.6.2 - Release Notes

Summary

Dynamo v1.6.2 is a patch release on top of v1.6.1. It refreshes the OS and Python packages in the runtime images, moves the SGLang image to CPython 3.12.16, returns HTTP 400 instead of HTTP 500 when a TensorRT-LLM prompt exceeds the engine token budget, and adds an opt-in garbage-collection freeze for SGLang workers.

One change can affect existing workflows. See Dependency Changes.

Base Branch: release/1.6.2

Dependency Changes
- Runtime OS Package Refresh: Upgraded libgnutls30t64 in the Frontend, vLLM, and sidecar images (#18064) and libbrotli1 in the Frontend, SGLang, TensorRT-LLM, and vLLM images (#17913) to the current Ubuntu patch releases.
- SGLang Image CPython 3.12.16: Moved the SGLang runtime image from Ubuntu's Python 3.12.3 to a uv-installed CPython 3.12.16, which bundles OpenSSL 3.5.9 (#18081). The Intel XPU image is unchanged.
Note: cuda-gdb in the SGLang image no longer supports Python scripting, because its Python front end links the Ubuntu libpython3.12 that this change removes.
- Planner Python 3.13.18: Moved the Planner image to python:3.13.18-slim for the build and the distroless Python 3.13-v4.1.10 base for the runtime (#17915). The build now fails if the runtime base bundles an older pip.
- Mooncake 0.3.14: Raised the Mooncake transfer engine floor from 0.3.13.post1 to 0.3.14 in the SGLang and vLLM images (#18091). The TensorRT-LLM image does not ship Mooncake.
- PyNvVideoCodec 2.2.4: Upgraded PyNvVideoCodec from 2.2.3 to 2.2.4 in the SGLang, TensorRT-LLM, and vLLM images (#17960), a patch release with decoder fixes.
- pyasn1 0.6.5 Floor: Floored pyasn1 at >=0.6.5,<0.7 in the SGLang and vLLM images (#17925), which otherwise kept the 0.6.4 copy from their upstream base.
- SGLang diffusers 0.40.0: Pinned diffusers at 0.40.0 in the SGLang image (#17926), matching the TensorRT-LLM and vLLM images. The Intel XPU build keeps SGLang's own pin.
Bug Fixes
- Oversized Prompt Rejection on TensorRT-LLM: A prompt longer than the engine's max_num_tokens now returns HTTP 400 invalid_request_error instead of HTTP 500 api_error, and the worker keeps running (#16193). This fixes the v1.6.0 known issue for non-streaming /v1/messages.
- SGLang Garbage-Collection Stalls: Added an opt-in --freeze-gc-after-init flag (DYN_SGL_FREEZE_GC_AFTER_INIT) that freezes garbage collection after startup on SGLang decode and aggregated workers, which reduces streaming stalls in long-running workers (#15203, #16761). It is off by default. Check memory use against a control worker before enabling it widely.
Build and Test
- Release Version Update: Set the package versions, Helm charts, and image references across the repository to 1.6.2 (#18157).
- Kubernetes 1.36 Warning Filter: Scoped the pytest warning filter so the kr8s supported-version warning on Kubernetes 1.36 no longer fails the deploy tests (#17285).
Key Dependencies

Backend runtime versions are unchanged from v1.6.1:

Dynamo SGLang TensorRT-LLM vLLM NIXL UCX
v1.6.2 v0.5.21 v1.3.0rc29 v0.31.0 v1.5.0 (Frontend, SGLang) / v1.3.1 (TensorRT-LLM) / v1.4.1 (vLLM) v1.21.0

CUDA Variants

SGLang TensorRT-LLM vLLM
13.0 13.1 13.0

Dynamo Ecosystem

AIPerf AISimulate Grove ModelExpress
v0.13.0 v0.13.0 v0.1.0-alpha.14 v0.7.0

Known Issues
- Carried Forward from v1.6.0: The known issues documented in the v1.6.0 release notes still apply, except the TensorRT-LLM HTTP 500 on oversized non-streaming /v1/messages reque

Dynamo	SGLang	TensorRT-LLM	vLLM	NIXL	UCX
v1.6.2	`v0.5.21`	`v1.3.0rc29`	`v0.31.0`	`v1.5.0` (Frontend, SGLang) / `v1.3.1` (TensorRT-LLM) / `v1.4.1` (vLLM)	`v1.21.0`

SGLang	TensorRT-LLM	vLLM
13.0	13.1	13.0

AIPerf	AISimulate	Grove	ModelExpress
`v0.13.0`	`v0.13.0`	`v0.1.0-alpha.14`	`v0.7.0`

v1.6.0-deepseek-v4.1-flash-dev.2-post.1Dynamo v1.6.0-deepseek-v4.1-flash-dev.2-post.1pre-release2026-11-05 · 2026-11-05 00:49 UTC

查看更新说明

Release Notes

Dynamo v1.6.0-deepseek-v4.1-flash-dev.2-post.1 is an experimental snapshot build with early DeepSeek-V4.1-Flash support on the Dynamo vLLM backend. It is not recommended for production and is not a QA-gated release. APIs, behavior, and defaults may change before stable support. Use it for evaluation, testing, and early feedback only.

Summary

Dynamo v1.6.0-deepseek-v4.1-flash-dev.2-post.1 is a follow-up to v1.6.0-deepseek-v4.1-flash-dev.2 for serving deepseek-ai/DeepSeek-V4.1-Flash on Dynamo vLLM. The published container returns to the pinned multi-arch vLLM nightly base (vllm/vllm-openai:nightly-ac9126e58aa7bbab1856ba6593ba4d5003fea516, CUDA 13.0.2) instead of the v0.31.0 base used by dev.2, and keeps the NIXL transfer patches at connector protocol 14. The image also includes a fix so prefill health probes no longer request a KV handoff, which stops the canary from leaking KV leases on prefill workers (#18076). vLLM Omni stays disabled.

The Kubernetes recipes now pin this image across all six vLLM profiles (aggregated and 1P1D disaggregated on B200, GB200, and H200). The H200 aggregated profile now runs 2 x TP4 on eight GPUs instead of 4 x TP4 on 16, so every vLLM profile uses eight GPUs, and its measured agentic results are refreshed (concurrency 56: 315.45 output tok/s/GPU, 50.63 user tok/s p50, TTFT p50 153.68 ms, TTFT p90 3,453.68 ms). The recipe changes landed on the release branch through #18156 (from #18141); related issue: #18153.

Release Branch: release/1.6.0-deepseek-v4.1-flash-dev.2-post.1, tip 90addf7c921ee14f05a8976e54118a5b98ad50df (2026-11-05) Follows: v1.6.0-deepseek-v4.1-flash-dev.2 (a201d5bd608e67a3c1def7eebace3201e6de0000) vLLM: nightly ac9126e58aa7bbab1856ba6593ba4d5003fea516 base (2026-10-01, CUDA 13.0.2), NIXL v1.3.2 (connector protocol 14)

Container Images

Backend	Arch	Image
vLLM (CUDA 13.0.2)	multi-arch (`amd64` + `arm64`)	`nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.6.0-deepseek-v4.1-flash-dev.2-post.1`

Backend Versions

Backend	Source	CUDA	Python	Notes
vLLM	Dynamo-built runtime on `vllm/vllm-openai:nightly-ac9126e58aa7bbab1856ba6593ba4d5003fea516` (pinned by digest), with NIXL telemetry, region-dedup, and piecewise-prefix patches	13.0.2	3.12	NIXL v1.3.2; connector protocol 14; vLLM Omni disabled

Models

Model	Precision	GPU	Notes
`deepseek-ai/DeepSeek-V4.1-Flash`	MXFP4 experts; B200/GB200 sparse-indexer KV and sparse-indexer logits; H200 CUTLASS MoE	8x B200, 8x GB200, 8x H200 (aggregated and disaggregated)	`deepseek_v41` reasoning and tool-call parsers; DSpark-3; up to 1,048,576-token context; text serving

About DeepSeek-V4.1-Flash

DeepSeek-V4.1-Flash is a DeepSeek model published on Hugging Face as deepseek-ai/DeepSeek-V4.1-Flash. This build serves it through Dynamo's vLLM backend with expert parallelism, expert load balancing, KV-aware routing, and DSpark speculative decoding with three draft tokens.

Full Changelog

vLLM Backend

Pinned nightly base: Returned the Dynamo vLLM runtime to the pinned multi-arch vLLM nightly base, together with the matching vllm-rs wrapper, patch set, and instrumented scheduler, replacing the v0.31.0 base used by dev.2 (#18076).
Prefill health probes: Prefill health probes no longer request a KV handoff, so the canary no longer leaks KV leases on prefill workers (#18076).

Recipes

Image pin: Updated

v1.7.0-mistral-large-4-dev.1Dynamo v1.7.0-mistral-large-4-dev.1pre-release2026-11-02 · 2026-11-02 21:08 UTC

查看更新说明

Release Notes

Dynamo v1.7.0-mistral-large-4-dev.1 is an experimental snapshot build with early Mistral Large 4 support on the Dynamo vLLM and SGLang backends. It is not recommended for production and is not a QA-gated release. APIs, behavior, and defaults may change before stable support. Use it for evaluation, testing, and early feedback only.

Summary

Dynamo v1.7.0-mistral-large-4-dev.1 adds native serving for Mistral Large 4 on Dynamo vLLM and SGLang (#17659). The Rust frontend now builds model cards from Mistral-native checkpoints (params.json and tekken.json), renders prompts with the native Mistral renderer, and parses [THINK] reasoning and [TOOL_CALLS] tool calls. The published containers are built on CUDA 13.4 engine images.

Kubernetes recipes are provided for vLLM only: aggregated and disaggregated (prefill/decode) profiles on GB300 and Vera Rubin, with DSpark speculative decoding, KV-aware routing, CPU KV offload, a shared model-cache PVC/download job, and an agentic-trace perf job under recipes/mistral-large-4 (#17868, cherry-picked from #17595). The SGLang container is also published, without recipes.

Release Branch: release/1.7.0-mistral-large-4-dev.1, tip 01be4d70c0ee6da09a9e78c7a3a05b2ef2f01e17 (2026-11-02) Base Commit: 649bc1469578aeeecc30099ce706185e917479d1 on main (2026-10-19) vLLM: vllm/vllm-openai:cu134-nightly-83c078d955cd6e0c63d3b12abe669c8de936a370 base (CUDA 13.4), mistral-common 1.13.0 SGLang: SGLang build 07abe5f63a933cef0f88f4cd7d41e5d9b4712709 (CUDA 13.4)

Container Images

Backend	Arch	Image	Digest
vLLM (CUDA 13.4)	`linux/arm64`	`nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.7.0-mistral-large-4-dev.1`	`sha256:2329c7a65b8707c5413cdea7a7305beabd0f1ba5c60f1946c9d4244d06d6f7f2`
SGLang (CUDA 13.4)	`linux/arm64`	`nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.7.0-mistral-large-4-dev.1`	`sha256:80c226f454d2df9189b5c61bc630973bebd8ff7ef80c772114394b74a39b292c`

Images are published for arm64 only (GB300 and Vera Rubin).

Backend Versions

Backend	Source	CUDA	Python	Notes
vLLM	Dynamo-built runtime on the vLLM `cu134-nightly-83c078d955cd6e0c63d3b12abe669c8de936a370` image	13.4.1	3.12	NCCL 2.32.3; `mistral-common` 1.13.0; custom FlashInfer with Vera Rubin MoE and sparse-indexer kernels
SGLang	Dynamo-built runtime on SGLang build `07abe5f63a933cef0f88f4cd7d41e5d9b4712709`	13.4.1	3.12	NCCL 2.31.2; cuDNN 9.26.0

Models

Model	Precision	GPU	Notes
`mistralai/Mistral-Large-4-1T-A52B-NVFP4`	NVFP4 weights; FP8 KV (GB300), NVFP4 KV (Vera Rubin)	GB300, Vera Rubin	Served as `mistral-large-4`; `mistral` tool-call and reasoning parsers; DSpark speculative decoding (3 draft tokens)

About Mistral Large 4

Mistral Large 4 is a 1,051B-parameter mixture-of-experts model (54B active) with a 1M-token context. This build serves the NVFP4 checkpoint through Dynamo's vLLM and SGLang backends. Mistral-native checkpoints ship params.json and tekken.json, which Dynamo now uses directly for tokenization and prompt rendering.

Full Changelog

Frontend

Native Mistral serving: Tekken model cards, native Mistral prompt rendering, unified reasoning/tool-call parsing ([THINK], [TOOL_CALLS]NAME[ARGS]{json}), and Mistral tool-call grammars (#17659).
Session affinity: Recipes use KV-aware routing with hard session affinity; cli

v1.6.1Dynamo v1.6.12026-10-27 · 2026-10-27 20:24 UTC
查看更新说明
Dynamo v1.6.1 - Release Notes

Summary

Dynamo v1.6.1 is a patch release on top of v1.6.0. It refreshes every trust store Dynamo ships, in the container images and in the ai-dynamo-runtime and kvbm wheels, so the updated bundles no longer include the emSign ECC Root CA - G3 root. It moves every image to Pillow 13, refreshes the image toolchain, and fixes crates.io publishing for the KVBM crates.

One change can affect existing deployments. See Security.

Base Branch: release/1.6.1

Security
- emSign ECC Root Removal: Updated the system CA bundles in every container image and the Rust TLS roots compiled into the ai-dynamo-runtime and kvbm wheels, which now use rustls-webpki 0.103.16 and webpki-roots 1.0.10 (#17108). gRPC clients in the images now verify against the refreshed system CA bundle instead of the roots bundled with grpcio.
Breaking: The updated bundles drop the emSign ECC Root CA - G3 root, so a TLS connection to an endpoint whose certificate chains to that root now fails verification.

Dependency Changes
- Pillow 13 Upgrade: Upgraded Pillow to the 13.0 line (pillow>=13.0.0,<14) in every runtime image (#17160). The Frontend and Planner images apply a uv override to stay on Pillow 13, because AIPerf v0.13.0 pins pillow~=12.3.0.
- Image Toolchain Refresh: Upgraded uv to 0.13.5, pip to 26.2.2, and certifi to 2026.10.19 in the images, and rebuilt the bundled NATS and etcd v3.5.35 with Go 1.27.3 (#17108). The Operator image moved to Go 1.27.3 on the distroless Go v4.1.7 base, and the Planner image moved to the distroless Python 3.13 v4.1.9 base.
Bug Fixes
- KVBM Crate Publishing: Restored the package description that kvbm-kernels, kvbm-logical, and kvbm-physical lost in v1.6.0 (#17141). crates.io rejects a crate without one, so these three crates and kvbm-engine, which depends on them, were not published for v1.6.0. All four are on crates.io at 1.6.1.
Documentation
- Kubernetes Image Tag Commands: Fixed the sed commands in the Intel XPU quickstart and the Dynamo Snapshot guide so they replace any image tag instead of only the literal 1.6.1 (#17220). Before the fix, a manifest that carried another tag, such as the my-tag placeholder in v1.6.0, kept it, and the deployment pointed at an image that could not be pulled.
Build and Test
- Release Version Update: Set the package versions, Helm charts, and image references across the repository to 1.6.1 (#17152).
- Test Dependency Pin: Pinned pandas below 3 in the test requirements, matching main, so the Profiler integration tests pass with AISimulate v0.13.0 (#17171).
Key Dependencies

Backend runtime versions are unchanged from v1.6.0:

Dynamo SGLang TensorRT-LLM vLLM NIXL UCX
v1.6.1 v0.5.21 v1.3.0rc29 v0.31.0 v1.5.0 (Frontend, SGLang) / v1.3.1 (TensorRT-LLM) / v1.4.1 (vLLM) v1.21.0

CUDA Variants

SGLang TensorRT-LLM vLLM
13.0 13.1 13.0

Dynamo Ecosystem

AIPerf AISimulate Grove ModelExpress
v0.13.0 v0.13.0 v0.1.0-alpha.14 v0.7.0

Known Issues
- Carried Forward from v1.6.0: The known issues documented in the v1.6.0 release notes still apply.
Full Changelog: https://github.com/ai-dynamo/dynamo/compare/v1.6.0...v1.6.1

Dynamo	SGLang	TensorRT-LLM	vLLM	NIXL	UCX
v1.6.1	`v0.5.21`	`v1.3.0rc29`	`v0.31.0`	`v1.5.0` (Frontend, SGLang) / `v1.3.1` (TensorRT-LLM) / `v1.4.1` (vLLM)	`v1.21.0`

SGLang	TensorRT-LLM	vLLM
13.0	13.1	13.0

AIPerf	AISimulate	Grove	ModelExpress
`v0.13.0`	`v0.13.0`	`v0.1.0-alpha.14`	`v0.7.0`

用户点评

登录用户的真实使用记录，可按版本和硬件筛选

共 0 条

还没有人点评。第一个说出真实体验的人最有价值。

README 原文

未翻译

在 GitHub 打开

Dynamo

The open-source, datacenter-scale inference stack. Dynamo is the orchestration layer above inference engines — it doesn't replace SGLang, TensorRT-LLM, or vLLM, it turns them into a coordinated multi-node inference system. Disaggregated serving, intelligent routing, multi-tier KV caching, and automatic scaling work together to maximize throughput and minimize latency for LLM, reasoning, multimodal, and video generation workloads.

Built in Rust for performance, Python for extensibility.

Community Events

Date	Event	Location
Thu, Nov 12, 2026	SIG Frontend — Biweekly Sync	–
Thu, Nov 12, 2026	SIG Engines — Biweekly Sync	–
Thu, Nov 19, 2026	SIG Router	–
Thu, Nov 5, 2026	~~Dynamo SIG RL~~	–
Thu, Nov 5, 2026	~~SIG Router~~	–

Events are updated automatically. Subscribe to our public calendar.

When to use Dynamo

You're serving LLMs across multiple GPUs or nodes and need to coordinate them
You want KV-aware routing to avoid redundant prefill computation
You need to independently scale prefill and decode (disaggregated serving)
You want automatic scaling that meets latency SLAs at minimum total cost of ownership (TCO)
You need fast cold-starts when spinning up new replicas

If you're running a single model on a single GPU, your inference engine alone is probably sufficient.

Feature support at a glance:

	SGLang	TensorRT-LLM	vLLM
Disaggregated Serving	✅	✅	✅
KV-Aware Routing	✅	✅	✅
SLA-Based Planner	✅	✅	✅
KVBM	🚧	✅	✅
Multimodal	✅	✅	✅
Tool Calling	✅	✅	✅

Full Feature Matrix → — LoRA, request migration, speculative decoding, and feature interactions.

Key Results

Result	Context
7x higher throughput per GPU	DeepSeek R1 on GB200 NVL72 w/ Dynamo vs B200 without (InferenceX)
7x faster model startup	ModelExpress weight streaming (DeepSeek-V3 on H200)
2x faster time to first token	KV-aware routing, Qwen3-Coder 480B (Baseten benchmark)
80% fewer SLA breaches	Planner autoscaling at 5% lower TCO (Alibaba APSARA 2025 @ 2:50:00)
750x higher throughput	DeepSeek-R1 on GB300 NVL72 (InferenceXv2)

What Dynamo Does

Most inference engines optimize a single GPU or a single node. Dynamo is the orchestration layer above them — it turns a cluster of GPUs into a coordinated inference system.

Architecture Deep Dive →

Core Capabilities

Capability	What it does	Why it matters
Disaggregated Prefill/Decode	Separates prefill and decode into independently scalable GPU pools	Maximizes GPU utilization; each phase runs on hardware tuned for its workload
KV-Aware Routing	Routes requests based on worker load and KV cache overlap	Eliminates redundant prefill computation — 2x faster TTFT
KV Block Manager (KVBM)	Offloads KV cache across GPU → CPU → SSD → remote storage	Extends effective context length beyond GPU memory
ModelExpress	Streams model weights GPU-to-GPU via NIXL/NVLink	7x faster cold-start for new replicas
Planner	SLA-driven autoscaler that profiles workloads and right-sizes pools	Meets latency targets at minimum total cost of ownership (TCO)
Grove	K8s operator for topology-aware gang scheduling (NVL72)	Places workloads optimally across racks, hosts, and NUMA nodes
AISimulate	Predicts serving behavior and searches deployment configurations offline	Finds a strong serving configuration without bringing up a GPU cluster
Fault Tolerance	Canary health checks + in-flight request migration	Workers fail; user requests don't

New in 1.0

Zero-config deploy (DGDR) (beta): Specify model, HW, and SLA in one YAML — AISimulate performance models estimate candidate configurations, Planner optimizes the topology, and Dynamo deploys
Agentic inference: Per-request hints for priority, expected output length, and speculative prefill, plus session metadata for tracing and SGLang subagent KV isolation. LangChain + NeMo Agent Toolkit integrations
Multimodal E/P/D: Disaggregated encode/prefill/decode with embedding cache — 30% faster TTFT on image workloads
Video generation: Native FastVideo + SGLang Diffusion support — real-time 1080p on single B200
K8s Inference Gateway plugin: KV-aware routing inside the standard Kubernetes gateway
Storage-tier KV offload: S3/Azure blob support + global KV events for cluster-wide cache visibility

Request Routing Topologies

Dynamo can expose traffic through two Kubernetes request routing topologies. Both expose an OpenAI-compatible API and support the same backends, disaggregated serving, and KV-aware routing.

Topology	What it is	When to use
Dynamo-native Frontend routing	The Dynamo Frontend serves HTTP and the integrated Dynamo Router makes worker-selection decisions. No external gateway is required.	Local development, single-cluster deployments, and environments where Dynamo should own the request entry point end to end.
Gateway API routing with GAIE	A Kubernetes Gateway API Inference Extension gateway calls the Dynamo Endpoint Picker Plugin (EPP) before forwarding to the selected worker's Frontend sidecar in `--router-mode direct`.	Kubernetes platforms that standardize on Gateway API, or deployments where gateway-level policy, auth, rate limiting, and observability should sit at the cluster edge.

Request flow for the Dynamo-native path is client → Frontend → Router → workers. Request flow for the Gateway API path is client → Gateway → EPP → Frontend sidecar (direct) → workers.

See the Gateway API Inference Extension (GAIE) guide for the Gateway API setup, supported features, and configuration.

Quick Start

This repo ships agent skills: if you work with an AI coding agent (Claude Code, Codex, Cursor), clone the repo and ask it to deploy, troubleshoot, benchmark, or optimize a Dynamo deployment. The skills activate automatically; no setup required.

Option A: Container (fastest)

Choose either vLLM or SGLang:

vLLM

# Pull a prebuilt container
docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.6.0

# Inside the container — start frontend and worker
python3 -m dynamo.frontend --http-port 8000 --discovery-backend file > /dev/null 2>&1 &
python3 -m dynamo.vllm --model Qwen/Qwen3-0.6B --discovery-backend file &

# Once the worker is ready, send a request
curl -s localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen3-0.6B",
  "messages": [{"role": "user", "content": "Hello!"}],
  "max_tokens": 100
}' | jq

SGLang

# Pull a prebuilt container
docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.6.0

# Inside the container — start frontend and worker
python3 -m dynamo.frontend --http-port 8000 --discovery-backend file > /dev/null 2>&1 &
python3 -m dynamo.sglang --model-path Qwen/Qwen3-0.6B --discovery-backend file &

# Once the worker is ready, send a request
curl -s localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen3-0.6B",
  "messages": [{"role": "user", "content": "Hello!"}],
  "max_tokens": 100
}' | jq

Also available: tensorrtllm-runtime:1.6.0.

Option B: Install from PyPI

Install uv (curl -LsSf https://astral.sh/uv/install.sh | sh), then choose your backend:

vLLM

uv pip install --prerelease=allow "ai-dynamo[vllm]"

SGLang

uv pip install --prerelease=allow "ai-dynamo[sglang]"

Note: TensorRT-LLM requires pip with --extra-index-url https://pypi.nvidia.com. See the install guide for TRT-LLM-specific instructions.

Then start the frontend and the matching worker as shown above. See the full installation guide for system dependencies and backend-specific notes.

Option C: Kubernetes (recommended)

For production multi-node clusters, install the Dynamo Platform and deploy with a single manifest:

# Zero-config deploy: specify model + SLA, Dynamo handles the rest
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata:
  name: my-model
spec:
  model: Qwen/Qwen3-0.6B
  backend: vllm
  sla:
    ttft: 200.0   # ms
    itl: 20.0     # ms
  autoApply: true

Pre-built recipes for common models:

Model	Framework	Mode	Recipe
Qwen3-32B-FP8	TensorRT-LLM	Aggregated	View
DeepSeek-R1	SGLang	Disaggregated	View
Kimi-K3	vLLM	Aggregated	View

See recipes/ for the full list. Cloud-specific guides: AWS EKS · Google GKE · Azure AKS · Amazon ECS

Building from Source

For contributors who want to build and develop locally. See the full build guide for details.

# Install system deps (Ubuntu 24.04)
sudo apt install -y build-essential libhwloc-dev libudev-dev pkg-config libclang-dev protobuf-compiler python3-dev cmake

# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh && source $HOME/.cargo/env

# Create venv and build
uv venv dynamo && source dynamo/bin/activate
uv pip install pip 'maturin[patchelf]'
cd lib/bindings/python && maturin develop --uv && cd $PROJECT_ROOT
uv pip install -e lib/gpu_memory_service
uv pip install -e .

VSCode/Cursor users: see the .devcontainer for a pre-configured dev environment.

Community and Contributing

Dynamo is built in the open with an OSS-first development model. We welcome contributions of all kinds.

Contribution Guide — How to contribute code, docs, and recipes
Design Proposals — RFCs for major features, tracked as dep:* labeled GitHub issues
Office Hours — Biweekly calls
Community Meetings (Youtube) – Weekly (Tue 9:00 AM PT) development community meetings
Slack — Chat with the team and community
Dynamo Day Recordings — Deep dives from production users

Latest News

Dynamo provides comprehensive benchmarking tools:

Benchmarking Guide – Compare deployment topologies using AIPerf
SLA-Driven Deployments – Optimize deployments to meet SLA requirements

Frontend OpenAPI Specification

The OpenAI-compatible frontend exposes an OpenAPI 3 spec at /openapi.json. To generate without running the server:

cargo run -p dynamo-llm --bin generate-frontend-openapi

This writes to docs/reference/api/openapi.json.

Service Discovery and Messaging

Dynamo uses TCP for inter-component communication. On Kubernetes, native resources (CRDs + EndpointSlices) handle service discovery. External services are optional for most deployments:

Deployment	etcd	NATS	Notes
Local Development	❌ Not required	❌ Not required	Pass `--discovery-backend file`; vLLM also needs `--kv-events-config '{"enable_kv_cache_events": false}'`
Kubernetes	❌ Not required	❌ Not required	K8s-native discovery; TCP request plane

Note: KV-aware routing does not require NATS. Enable KV events when you need event-backed cache-state tracking, or use --no-router-kv-events for prediction-based routing without external event infrastructure.

For Slurm or other distributed deployments that choose etcd or NATS JetStream-backed modes:

etcd can be run directly as ./etcd.
nats needs JetStream enabled: nats-server -js.

To quickly setup both: docker compose -f dev/docker-compose.yml up -d

More News

Reference

Support Matrix — Hardware, OS, CUDA, and backend versions
Feature Matrix — Detailed backend compatibility
Release Artifacts — Containers, wheels, Helm charts
Service Discovery — K8s-native vs etcd vs file-based discovery
Benchmarking Guide — Compare deployment topologies with AIPerf

怎么上手

项目说明

核心能力

容易踩的坑

最近发版

Release Notes

Summary

Container Images

Backend Versions

Models

About Kimi-K3.1-Pro

Full Changelog

vLLM Backend

Dynamo v1.6.2 - Release Notes

Summary

Dependency Changes

Bug Fixes

Build and Test

Key Dependencies

Known Issues

Release Notes

Summary

Container Images

Backend Versions

Models

About DeepSeek-V4.1-Flash

Full Changelog

vLLM Backend

Recipes

Release Notes

Summary

Container Images

Backend Versions

Models

About Mistral Large 4

Full Changelog

Frontend

Dynamo v1.6.1 - Release Notes

Summary

Security

Dependency Changes

Bug Fixes

Documentation

Build and Test

Key Dependencies

Known Issues

用户点评

README 原文

Dynamo

Community Events

When to use Dynamo

Key Results

What Dynamo Does

Core Capabilities

New in 1.0

Request Routing Topologies

Quick Start

Option A: Container (fastest)

vLLM

SGLang

Option B: Install from PyPI

vLLM

SGLang

Option C: Kubernetes (recommended)

Building from Source

Community and Contributing

Latest News

Frontend OpenAPI Specification

Service Discovery and Messaging

More News

Reference

相关项目