登录

vllm-omni

vllm-projectApache-2.0活跃维护

面向全模态模型的高效推理框架，支持图像、视频、音频生成与多模态模型服务，基于 PyTorch 提供统一的推理与部署能力。

A framework for efficient model inference with omni-modality models

v0.32.0发布于 2026-10-28 10:10 UTC7,675star +68 / 7 天2,089fork同步于 2026-11-10 23:33 UTC
全模态推理多模态生成模型服务图像生成视频生成
对比 GitHub 官网audio-generationdiffusionimage-generationinferencemodel-serving

怎么上手

由 README 提炼，执行前请确认
原文 
README 未给出安装命令，仅指向文档 Installation 与 Quickstart 页面，并提供基于 main 分支构建的 nightly CUDA Docker 镜像（vllm/vllm-omni）。
硬件要求按 README 说明

README 未说明具体显存要求；发布记录提到支持 CUDA、ROCm、MUSA、NPU、XPU 等多种后端，并在 NVIDIA Blackwell 与 Ascend 950 上做过实时推理。

需要 CUDA支持 ROCm / AMD昇腾 / Ascend
下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼
没人工点评？写下你的经验

vLLM 官方扩展，把文本、图像、音频、视频和动作模型统一做推理与在线服务

面向全模态模型的高效推理框架，支持图像、视频、音频生成与多模态模型服务，基于 PyTorch 提供统一的推理与部署能力。

核心能力

  • 复用 vLLM 的高效 KV 缓存管理支撑自回归推理
  • 用流水线阶段重叠执行提升整体吞吐
  • 基于 OmniConnector 做阶段间完全解耦与动态资源分配
  • 支持张量、流水线、数据和专家并行分布式推理
  • 提供 OpenAI 兼容 API 服务与流式输出

容易踩的坑

版本与上游 vLLM 版本线对齐，需按对应 release 说明安装以免不兼容；模型需从 Hugging Face 单独获取，README 未内置权重；安装步骤需查阅在线文档，README 本身不含可直接执行的命令。

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

最近发版

5 个版本v0.32.0
全部 release
  1. v0.33.0rc1vLLM-Omni v0.33.0rc1pre-release2026-11-04 · 2026-11-04 08:48 UTC
    查看更新说明

    First release candidate for vLLM-Omni 0.33.0.

    Aligns vLLM-Omni with final upstream vLLM 0.33.0, including runtime integration, dependency updates, installation instructions, and upgrade-validation fixes.

    Installation

    The release candidate wheel is available on PyPI:

    pip install vllm-omni==0.33.0rc1

    Docker images are still building; follow the publication build above for their status.

  2. v0.32.0最新正式版2026-10-28 · 2026-10-28 10:10 UTC

    Highlights

    vLLM-Omni v0.32.0 aligns with vLLM 0.32.0 and brings multi-node execution, expanded realtime serving, new image and audio models, and release-branch stability fixes since v0.30.0.

    Runtime and realtime serving

    • Add optional Ray-based multi-node inference, session-aware replica placement, and multi-frontend full-duplex serving. (#7938, #8913)
    • Add server VAD for turn-based /v1/realtime and RL weight-transfer support. (#8600, #7686)
    • Improve request failure isolation, repeated-stage-failure handling, and FlashInfer compatibility with torch.compile(fullgraph=True). (#9470, #9275, #9397)

    Models and generation

    • Add YuE2-3B text-to-music, Qwen-Image-2.1 / 2.1-Turbo, Kandinsky6, and MOSS-Image. (#7886, #8099, #8699, #8537, #8926)
    • Add TaoMate-H3 realtime AR-diffusion streaming and an LTX 2.5 single-stage text-to-audio pipeline. (#8268, #6489)
    • Add optional single-stage Qwen3-TTS with in-Talker streaming codec decoding; improve PersonaPlex batched execution and CUDA graphs. (#8259, #8441)
    • Add CUDA FA3 FP8 diffusion attention and online FP8 quantization for MammothModa2 DiT. (#8965, #7221)

    Compatibility

    • Requires Python >=3.11,<3.14 and the matching vLLM 0.32.0 release. (#9066)
    • Output-type names require canonical spellings: text, image, audio, latent, and token_ids. Legacy aliases and malformed values are rejected. (#7843)

    Release stabilization

    This release includes curated fixes on release/v0.32.0, including the idle MoE LoRA-context repair for HunyuanImage-3. (#9515)

    Source commit: 5917730a4b02aedb44d374173cdc1b10484ccd52.

    Release artifacts

    Install the matching CUDA 13.0 packages in a fresh environment:

    uv venv --python 3.12
source .venv/bin/activate
uv pip install "vllm==0.32.0" "vllm-omni==0.32.0" --torch-backend=cu130

    Or pull the CUDA container:

    docker pull vllm/vllm-omni:v0.32.0

    Full changelog: https://github.com/vllm-project/vllm-omni/compare/v0.30.0...v0.32.0

  3. v0.32.0rc1pre-release2026-10-19 · 2026-10-19 16:10 UTC
    查看更新说明

    Highlights

    vLLM-Omni v0.32.0rc1 includes 297 merged changes since v0.31.0rc1. This prerelease (not GA) aligns with / is built on vLLM v0.32.0, expands multimodal and realtime serving, and improves diffusion and speech performance.

    Core Architecture & Runtime

    • Align vLLM-Omni with vLLM v0.32.0 (runtime integration surfaces, dependencies, container/docs defaults, and Python support >=3.11,<3.14). (#9066)
    • Add session-aware replica placement and multi-frontend full-duplex serving, plus server VAD on turn-based /v1/realtime. (#8913, #8600)
    • Add RL weight-transfer support for training handoff into Omni serving. (#7686)

    Model Support & Realtime Serving

    • Add MOSS-Image text-to-image with SLG. (#8926)
    • Add TaoMate-H3 realtime streaming on AR-Diffusion (USP2/USP4). (#8268)
    • Add LTX 2.5 text-to-audio one-stage pipeline. (#6489)
    • Add JoyAI speech input on unified duplex sessions. (#8931)
    • Add YuE2 music generation and cover workflows to ComfyUI. (#8881)

    Speech & Audio Performance

    • Improve PersonaPlex with batched execution, CUDA graphs, and memory optimization across Stage 0 / Depformer / Stage 1. (#8441)
    • Continue MiniCPM-o / Qwen3-TTS / CosyVoice3 throughput and streaming fixes landed since v0.31.0rc1 (see full changelog).

    Diffusion, Image & Video

    • Add CUDA FA3 FP8 diffusion attention. (#8965)
    • Enable online FP8 quantization for MammothModa2 DiT. (#7221)

    Compatibility & Packaging

    • Python support for this cut: >=3.11,<3.14, with dependency, container reference, and install-guidance updates from the v0.32.0 alignment. (#9066)
    • This tag is a release candidate, not a GA release.

    Validation & Known Issues

    • The release is pinned to da572cddb38383deae9caef33b8d490e292ef071 on branch release/v0.32.0rc1 (includes the vLLM v0.32.0 alignment from #9066).
    • This note does not claim a fully green multi-platform nightly or day-one support for every new model. Please file issues against this tip SHA.

    Release Artifacts

    • GitHub pre-release tag: v0.32.0rc1. PyPI wheel and container images will be published by the omni-release pipeline after this tag; install links will be updated once those artifacts are available.
    • Until then, install from the tagged source: pip install git+https://github.com/vllm-project/[email protected]

    What's Changed

  4. v0.31.0rc1pre-release2026-10-06 · 2026-10-06 03:25 UTC
    查看更新说明

    Highlights

    vLLM-Omni v0.31.0rc1 includes 110 merged changes since v0.30.0. This prerelease aligns with vLLM 0.31.0, adds Ray-based multi-node inference, expands music generation and realtime serving, and improves speech and multimodal inference performance.

    Core Architecture & Runtime

    • Align vLLM-Omni with vLLM 0.31.0, including CUDA/ROCm base images and native auxiliary-output integration. Update routed-expert tests to cover the supported V2 runner path. (#8459, #8500)
    • Add an optional Ray executor for multi-node inference, with cluster-wide stage GPU allocation and environment forwarding. (#7938)
    • Extend shared execution for mixed V1/Model Runner V2 pipelines, add byte backpressure to chunk transfers, and bound duplex output delivery and cancellation. (#8184, #7916, #7644)
    • Add an independent token_ids output type and enforce canonical output-type names. (#7843)

    Model Support & Realtime Serving

    • Add YuE2-3B text-to-music through /v1/audio/speech, with lyrics, style instructions and optional ABC-score conditioning. Add async single-stage synthesis and CUDA graph acceleration. (#7886, #8407)
    • Add SheetSage2 preprocessing and an optional YuE2 score-conditioned request export. This is an official-backend preprocessing utility, rather than a native vLLM model or a new serving endpoint. (#8371)
    • Add SenseNova-U1-A3B-MoT, restore PersonaPlex serving on the unified full-duplex framework, and extend OpenAI-compatible realtime serving with Qwen3-Omni and incremental video prefill. (#8172, #7695, #7285, #5894)
    • Add AuK online serving and DiT CUDA graphs, alongside encoder, codec and attention optimizations. (#7469, #7881, #8300, #8328, #8305)

    Speech & Audio Performance

    • Add opt-in single-stage Qwen3-TTS on one CUDA GPU, with streaming codec decoding inside the Talker; the existing two-stage pipeline remains available. Improve CUDA throughput and first-audio latency. (#8259, #8163)
    • Improve MiniCPM-o 4.5 with block-aligned duplex KV sliding windows, whole-Euler Code2Wav CUDA graphs, turn-mode MRv2 and batched audio inference. (#7821, #8007, #8222)
    • Improve CosyVoice3 with a GPU-resident F0 predictor, pure-PyTorch vocoder, opt-in packed Flow/batched HiFT execution, bounded streaming graphs and device-dependent defaults. Restore Fish Speech/CosyVoice3 audio quality at high concurrency. (#7518, #8224, #8420, #8433, #8422)
    • Improve MOSS-TTS streaming codec ramps, seeded batch generation and MRv2 serving. Add Higgs Audio v3 MRv2 streaming and Ascend VoxCPM2 LocDiT NPUGraph acceleration. (#8124, #7922, #8213, #8226, #7275)

    Diffusion, Image & Video

    • Add MiniMax-H3 latent super-resolution, high-resolution refinement and res_multistep sampling; avoid unnecessarily enlarging reference inputs. (#8322, #8378, #8253)
    • Add Boogu-Image request-level batching, SenseNova online FP8, and bounded long-video restoration with SeedVR2. (#7227, #7955, #8102)
    • Improve MammothModa2 with TeaCache, continuous batching, fused QK normalization/RoPE and text-to-image serving fixes. (#5357, #7954, #7969, #7199, #7293, #7482)
    • Add HSDP pre-sharded loading, prototype FA4 diffusion attention execution contracts, and fix layerwise-offload shutdown, Cosmos3 FP32 sampling state and washed-out LingBot Video output. (#7948, #7379, #8453, #7592, #7037)

    Compatibility & Packaging

    • Output-type names now require canonical spellings: text, image, audio, latent and token_ids. Legacy aliases and malformed values are rejected; existing +/, combinations and None defaults remain supported. (#7843)
    • Unsupported async-chunk configurations are disabled with a warning, and async chunk defaults depend on pipeline support. (#5099)
    • Explicitly package runtime deployment files, chat templates, kernel sources and model assets, with source-archive/wheel checks. Fix remaining P
  5. v0.30.02026-09-25 · 2026-09-25 18:20 UTC
    查看更新说明

    Highlights

    This release features 504 merged changes from 183 contributors, including 94 new contributors.

    vLLM-Omni v0.30.0 is led by three major advances: 1) a unified full-duplex serving framework, 2) native cross-stage KV and multimodal payload transfer, and 3) interactive world-model and streaming video generation, 4) realtime minimax-h3 4-step inference in both Nvidia Blackwell and Ascend NPU 950 series. The release rebases onto vLLM 0.30.0, expands model support across speech, image, audiovisual generation, and VLA, and improves batching, memory efficiency, and deployment scalability.

    Key Improvements

    • Unified full-duplex serving around engine-owned sessions, separating duplex orchestration from turn-based execution and introducing shared model-plugin interfaces. MiniCPM-o 4.5 uses the new framework, with AURA adding multimodal interaction and multi-turn history. (#7413, #7633)
    • Extended scheduler-managed diffusion KV into cross-stage transfer and reuse, adding native Mooncake AR-to-DiT handoff, KV prefetch, and HunyuanImage3 cross-request paged prefix caching. Added NIXL transport for complete AR/encoder-to-diffusion conditioning payloads. (#7166, #7637, #7719, #6264)
    • Advanced interactive world-model and video execution, with LingBot World stepwise generation and mid-stream camera control, session-owned streaming VAE decode, and bounded worker-side video encoding and asynchronous chunk transfer. (#6844, #7198, #6533, #7048, #7406)

    Core Architecture & Runtime

    • Rebased the project onto vLLM 0.30.0, including corresponding Ascend NPU integration updates. (#7820, #8027)
    • Introduced DuplexOmni, DuplexOmniEngine, and DuplexOrchestrator, moving session ownership and lifecycle management into the engine. Eligible models can serve duplex sessions and Chat Completions using the same engine and weights. (#7413)
    • Added parallel stage initialization with device-aware admission and locking, allowing independent stages and replicas to initialize concurrently. Centralized Omni configuration resolution and retained structured configuration through runtime startup. (#5224, #7292, #5140, #6849)
    • Added NIXL Omni connectors and complete conditioning-payload transfer, including text, visual/audio latents, and layout metadata for disaggregated MiniMax H3 execution. (#6093, #6264)
    • Added scheduler-level pause_generation(mode="keep") for request-level diffusion execution, preserving queued work and acknowledging device synchronization before returning. (#7685)

    Model Support

    • Added MAGI-2 Preview with native text/image-to-video-and-audio generation, synchronized stereo audio, distributed execution, and Cache-DiT integration. (#5918)
    • Added Ming-Image-0.1-Design and Design-Layer, supporting text-to-image generation, single-reference image editing, and RGBA layer decomposition. (#8021)
    • Added Anima native text-to-image inference from local single-file checkpoints, and support for HunyuanImage-3.0-Instruct-Distil. (#4083, #4048)
    • Expanded SenseNova-U1.5-8B-MoT support with its distilled eight-step LoRA, and added the Boogu-Image Turbo native DMD execution path. (#6516, #6699)
    • Added Tencent AuK and AuK-Flash for instruction-driven speech generation and editing, Audio8 TTS Preview 0.6B, and Breeze-TTS-2 with a two-stage streaming AR/codec pipeline. (#7385, #6157, #7084)
    • Added an experimental JoyAI-VL-Interaction native multi-stage speech pipeline and an experimental Mage-VL duplex adapter and reference-backed offline integration. (#5352, #6537)
    • Added π0.5 (Pi0.5), extending vision-language-action model coverage. (#6950)

    Audio, Speech & Realtime Serving

    • Moved the shared MiniCPM-o 4.5 and PersonaPlex duplex infrastructure out of the experimental namespace, and integrated AURA with the unified runtime

用户点评

登录用户的真实使用记录，可按版本和硬件筛选
共 0 条

    还没有人点评。第一个说出真实体验的人最有价值。

    README 原文

    未翻译
    在 GitHub 打开

    Latest News 🔥

    • [2026/10] We released 0.32.0, aligned with vLLM 0.32.0, featuring optional Ray-based multi-node inference, session-aware full-duplex serving, server VAD for turn-based realtime requests, and expanded image, music, and speech generation with YuE2-3B, Qwen-Image-2.1, Kandinsky6, MOSS-Image, and TaoMate-H3. See the release notes for compatibility changes and installation instructions.
    • [2026/10] We released the vLLM-Omni Technical Report, describing a unified serving runtime for text, audio, image, video, and action generation.
    • [2026/09] We released 0.30.0, rebased onto vLLM 0.30.0, featuring a unified full-duplex serving framework around engine-owned sessions for MiniCPM-o 4.5 and AURA, native cross-stage KV and multimodal payload transfer (Mooncake AR-to-DiT handoff and NIXL connectors), interactive world-model serving with LingBot World, and realtime MiniMax H3 Turbo inference on NVIDIA Blackwell and Ascend 950.
    • [2026/08] We released 0.28.0, featuring production-ready MiniMax H3 serving on GPU and NPU, a unified AR/DiT paged KV cache runtime, and enhanced realtime full-duplex serving for the MiniCPM-o series.
    • [2026/08] VeRL-Omni v0.2.0 is released: faster diffusion RL powered by vLLM-Omni (request-level/step-wise batching with FA3), rebuilt Qwen3-Omni multimodal training (DPO & GSPO), plus LTX-2.3, Qwen-Image-Edit support and more. See the release notes.
    • [2026/08] We released 0.26.0 - aligned with the vLLM 0.26 release line, featuring MiniMax H3 joint video/audio generation, an experimental full-duplex realtime runtime for MiniCPM-o 4.5, distributed layerwise diffusion offload, and broader model, hardware, streaming, TTS, and quantization support.
    • [2026/07] We released 0.24.0 - aligned with the vLLM 0.24 release line, expanding production-ready coverage across TTS, speech, diffusion, image/video generation, and robot-policy serving, with major Omni stage runtime refactoring, diffusion request-level batching, async output materialization, quantization/cache/memory improvements, and broad CUDA/ROCm/XPU/NPU support.
    • [2026/06] Stable releases 0.14.0, 0.16.0, 0.18.0, 0.20.0, and 0.22.0 aligned with every even-numbered upstream vLLM minor version, expanding omni and world-model serving, TTS, diffusion, distributed execution, quantization, RL integration through VeRL-Omni, and CUDA/ROCm/MUSA/NPU/XPU support.
    • [2026/03] Check out our first public project deepdive at the vLLM Hong Kong Meetup!
    • [2025/11] vLLM community officially released vllm-project/vllm-omni in order to support omni-modality models serving.

    About

    vLLM was originally designed to support large language models for text-based autoregressive generation tasks. vLLM-Omni is a framework that extends its support for omni-modality model inference and serving:

    • Omni-modality: Text, image, audio, video, and action data processing
    • Non-autoregressive Architectures: extend the AR support of vLLM to Diffusion Transformers (DiT) and other parallel generation models
    • Heterogeneous outputs: from traditional text generation to multimodal and action outputs

    vLLM-Omni is fast with:

    • State-of-the-art AR support by leveraging efficient KV cache management from vLLM
    • Pipelined stage execution overlapping for high throughput performance
    • Fully disaggregation based on OmniConnector and dynamic resource allocation across stages

    vLLM-Omni is flexible and easy to use with:

    • Heterogeneous pipeline abstraction to manage complex model workflows
    • Seamless integration with popular Hugging Face models
    • Tensor, pipeline, data and expert parallelism support for distributed inference
    • Streaming outputs
    • OpenAI-compatible API server
    • Full-duplex realtime serving with streaming audio input and output

    vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including:

    • Omni-modality models (e.g. Qwen3-Omni, MiniCPM-o 4.5, Cosmos3, HunyuanImage, BAGEL)
    • TTS models (e.g. Qwen3-TTS, Tencent AuK, Breeze-TTS-2, CosyVoice3)
    • Diffusion models — image, video, and audio generation (e.g. MiniMax H3, LingBot World, MAGI-2, LTX-2.5, Wan2.2)
    • Robot-policy and action models (e.g. π0.5, GR00T-N1.7, DreamZero-DROID, InternVLA-A1)

    Getting Started

    Visit our documentation to learn more.

    Contributing

    We welcome and value any contributions and collaborations. Please check out Contributing to vLLM-Omni for how to get involved.

    Citation

    If you use vLLM-Omni for your research, please cite our technical report and original paper:

    @article{vllmomni2026technicalreport,
  title={vLLM-Omni Technical Report: A Unified Serving Runtime for Omni-Modality Generation},
  author={{vLLM-Omni Team}},
  journal={arXiv preprint arXiv:2610.09307},
  year={2026}
}

@article{yin2026vllmomni,
  title={vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models},
  author={Peiqi Yin, Jiangyun Zhu, Han Gao, Chenguang Zheng, Yongxiang Huang, Taichang Zhou, Ruirui Yang, Weizhi Liu, Weiqing Chen, Canlin Guo, Didan Deng, Zifeng Mo, Cong Wang, James Cheng, Roger Wang, Hongsheng Liu},
  journal={arXiv preprint arXiv:2602.02204},
  year={2026}
}

    Join the Community

    Feel free to ask questions, provide feedbacks and discuss with fellow users of vLLM-Omni in #sig-omni slack channel at slack.vllm.ai or vLLM user forum at discuss.vllm.ai.

    Star History

    License

    Apache License 2.0, as found in the LICENSE file.

    相关项目

    同分类里 star 最高的项目
    对比全部 →
    stable-diffusion-webui
    AUTOMATIC1111·Python

    基于 Gradio 的 Stable Diffusion 网页界面，把文生图、图生图、局部重绘等能力做成可视化操作面板，不用写代码就能跑图像生成。

    图像生成+4图像生成进阶
    165287+4131kv1.10.12026-03-02 07:00 UTC
    ComfyUI
    Comfy-Org·Python

    基于节点图的扩散模型图形界面与后端，支持本地快速推理和 API 调用，方便搭建和调试图像生成工作流。

    扩散模型+4图像生成进阶
    146180+69417kv0.45.02026-11-10 22:53 UTC
    OpenMontage
    OpenMontageInc·Python

    把 Claude、Cursor 等 AI 编程助手变成完整视频制作工作室，内置 12 条制作流水线和 100+ 工具，覆盖脚本、配音、图像生成到剪辑全流程。

    视频制作+4图像生成硬核
    79416+76211k2026-11-09 06:23 UTC
    OpenMontage
    calesthio·Python

    把 Claude、Cursor 等 AI 编程助手变成完整的视频制作工作室，内置 12 条制作流水线和 100+ 工具，覆盖从生成到剪辑的全流程。

    视频制作+4图像生成硬核
    7786710k2026-11-03 17:42 UTC