登录

xllm

xLLM-AIApache-2.0活跃维护

面向 LLM、VLM、DiT 和 REC 模型的高性能推理引擎，针对多种 AI 加速卡做优化，适合在异构硬件上部署大模型服务。

A high-performance inference engine for LLM, VLM and DiT models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.

v0.12.0发布于 2026-10-31 12:34 UTC1,619star +1 / 7 天331fork同步于 2026-11-10 11:47 UTC
推理引擎大模型推理多硬件加速异构部署C++
对比 GitHub 官网deepseekglminferenceinference-enginelarge-language-models

怎么上手

由 README 提炼，执行前请确认
原文 
README 未给出安装命令，仅提供 Docker 镜像 quay.io/repository/jd_xllm/xllm-ai?tab=tags 与文档链接 https://docs.xllm-ai.com/en/getting_started/quick_start/
硬件要求按 README 说明

README 列出支持的硬件：昇腾 NPU（A2/A3，需 HDK Driver 25.2.0+）、寒武纪 MLU、摩尔线程 GPU（S5000）、海光 DCU（BW1000）、沐曦 MACA（MXC500）、天数智芯 GPU（BI150）。README 未给出具体显存要求，也未说明是否支持纯 CPU 运行。

支持 Apple Silicon昇腾 / Ascend
下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼
没人工点评？写下你的经验

面向国产 AI 加速卡优化的 LLM/VLM/DiT 推理引擎，主打企业级高吞吐低延迟部署

面向 LLM、VLM、DiT 和 REC 模型的高性能推理引擎，针对多种 AI 加速卡做优化，适合在异构硬件上部署大模型服务。

核心能力

  • 服务层与引擎层解耦，调度与计算分离
  • 基于 Mooncake 构建混合 KV 缓存，支持全局管理与智能卸载预取
  • 深度适配昇腾、寒武纪、摩尔线程等国产加速卡
  • 支持 LLM、VLM 与 DiT 多类模型推理
  • 基于 brpc 构建高性能 HTTP 服务

容易踩的坑

README 未提供安装步骤，需按官方文档 Quick Start 操作；昇腾 NPU 需 HDK Driver 25.2.0 及以上版本；模型需按各模型 Deployment 文档单独准备与部署

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

最近发版

5 个版本v0.12.0
全部 release
  1. v0.12.0最新正式版2026-10-31 · 2026-10-31 12:34 UTC

    Release xllm 0.12.0

    Major Features and Improvements

    Highlights

    • GLM-5.3 and GLM-5.3-Flash: expand NPU inference with Python models, KDA attention, compressed KPool caches, MTP and DFlash2 decoding, and additional parallel serving configurations.
    • Qwen3.8: add Python multimodal inference and MTP, DFlash2, and DSpark decoding; improve Qwen prefill with FlashComm1, MMRS, and fused GDN operators.
    • Elastic virtual KV cache: grow and reclaim physical backing on demand, admit requests against live memory capacity, and support co-located MTP with separate KV-cache and weight VMM controls.
    • Serving compatibility: extend OpenAI-compatible sampling controls and improve OpenAI and Anthropic request validation, streaming, usage reporting, and tool handling.
    • Developer tooling: introduce the standalone xllm-kernel package, expand Ascend online profiling, and improve NPU wheel packaging and build workflows.

    Model Support

    NPU
    • Add GLM-5.3-Flash Python inference with eager execution and ACLGraph decode, KDA kernels, compressed KPool cache management, and fused compression and indexer operators.
    • Add GLM-5.3 Python MTP and DFlash2 support, GLM-5.3-Flash MTP and DFlash2 support, and direct loading of Flash MTP weights from native checkpoints.
    • Extend GLM serving with prefill context parallelism, decode context parallelism, expert parallelism, and GLM-5.3 MTP in disaggregated prefill/decode (PD) deployments, according to model-specific configuration support.
    • Add Qwen3.8 Python image and video inference, plus MTP, DFlash2, and DSpark decoding. Extend Qwen3.5 Python execution with data parallelism, MoE expert parallelism, and decode context parallelism.
    • Extend the DeepSeek-V4 Python implementation with CSA/HCA attention, MTP and DSpark models, checkpoint weight aliases, and context/data parallel execution paths.
    • Add Eagle3 support for Kimi-K2.5 and vision-language models, and support Eagle3.1 drafts exported in the speculators format.
    • Extend GLM-5.2 with INT8 indexer caching, INT8 SFA KV caching on the C++ path, packed C8 KV layouts, MegaMoE execution, and attention data-parallel weight sharding.
    MLU and CUDA
    • Add native GLM5-Next inference on MLU with KDA and KPool layers; extend DCP indexer and cache-transfer support and optimize GLM DCP kernels.
    • Add compressed-tensors W8A8 checkpoint loading on MLU.
    • Add Phi-3 mini 4K inference on CUDA with native fused checkpoint loading and tensor-parallel weight handling.
    Multimodal and Diffusion
    • Add MiniMax-H3 inference with video and audio autoencoders, media processors, and diffusion scheduling.
    • Add JoyAI ASR speech recognition with audio preprocessing and Python model execution.
    • Improve DiT request serialization, media transport, and scheduler wakeups; preserve audio outputs across independent worker routes.

    Speculative Decoding

    • Add throughput-aware MTP draft-step selection using startup calibration and observed prefix acceptance.
    • Add asynchronous adaptive target verification for DSpark and MTP, including Draft5/7 configurations and greedy, random, or mixed target sampling on supported execution paths.
    • Extend Python DSpark verification to recurrent targets with packed fused GDN kernels, and improve sliding-window attention handling for block drafts.
    • Expand prewarmed target/draft batch and KV-capacity coverage, support no-padding Python ACLGraph decode, and admit only execution shapes covered by warmup.
    • Improve MTP target sampling semantics for minimum output length, seeded generation, bad-word constraints, and token penalties in supported LLM configurations.
    • Correct speculative acceptance accounting to use actual verification results and exclude resampled tokens and warmup batches.

    KV Cache and Distributed Serving

    • Add independent enable_vmm_kvcache and enable_vmm_weights controls on NPU. Elastic VMM grows physical pages on demand and releases idle backing while preserving execution hea
  2. v0.11.02026-09-16 · 2026-09-16 15:25 UTC
    查看更新说明

    Release xllm 0.11.0

    Major Features and Improvements

    Highlights

    • DeepSeek-V4 end-to-end hardening on NPU: PD disaggregation with prefill context parallelism, FlashComm1 sequence parallelism with schedule overlap, host/device prefix cache spanning SWA + C4 + C128 units, DSpark decoding, and EPLB integrated with the fused MoE / MC2 path. MLU gained its own DeepSeek-V4 attention, MoE, PD transfer, and decode-path optimizations.
    • DeepSeek-V3.2 and GLM-5.2 move onto the Python model stack: expert-parallel (EP) support for DeepSeek-V3.2, DeepSeek-V3.2 MTP under ACL graph, and GLM-5.2 EPLv2 with shared-expert overlap, on top of existing W8A8 PyTorch adaptation, DSA top-k sharing, cache elision, and Mooncake PD.
    • Qwen3.5 / Qwen3.6 / Qwen3.8 round out with a CUDA Python model implementation, optional FIA (fused-infer-attention) decode, Qwen3.8-27B chat templates, context-parallel sequence sharding, NPU Python ACL-graph data-parallel decode, and a new Qwen3-VL-dense Python backend with ViT + deepstack + mRoPE — alongside the existing MegaMoE kernel, MegaChunkGDN fused operator, FlashComm1+MMRS fusion, and heterogeneous PD disaggregation.
    • Speculative decoding grows an adaptive layer: adaptive speculative decoding for DFlash and DSpark on NPU, extended to graph mode and per-sequence validate pruning, plus JSON-object constrained decoding integrated with MTP verification.
    • KV cache gets a new sharding and offload story: layerwise split KV cache sharding (shard persistent KV by static layer owner so PD cache ids stay aligned across prefill/decode/spec-verify), asynchronous host KV cache transfers, Mooncake Store as a host-cache backend, and host KV offload wired through PD and MTP.
    • Context parallelism lands on MLU: foundational decode context-parallel (DCP) support with dedicated attention-merge and decode-context kernels for MLU DeepSeek-V2/GLM-5.2.
    • Multimodal / diffusion: Flux2 text encoder, DiT, and VAE on NPU; Wan2.2 pipeline with distilled I2V, laser attention, RainFusion sparse attention, fused RoPE/norm operators, and single-NPU rolling weight load; QwenImageEditPlus, joy-image-edit-plus, and DFlash block-diffusion speculative decoding.
    • Multi-hardware breadth: MUSA (Moore Threads) gains a graph executor, FP8/MoE/sampling kernels, Qwen3.5 dense-model layers, standalone builds, and a dedicated runtime-integration refactor; DCU adds DeepSeek-V2/V3 FP8 W8A8, MiniMax-M2.7 Channel FP8, MiMo-MTP, Qwen3.5, Flux image generation, Mooncake PD, and PD-OOC.
    • Developer experience: the Python executor tree is now linted and formatted with ruff + codespell, and platform-specific typing/dispatch code was consolidated into a single torch-based module.
    • A large wave of MTP / PD / SWA correctness fixes stabilizes speculative decoding and disaggregated serving under sustained, high-concurrency load — see Bugfix below.

    Model Support

    NPU
    • Support DeepSeek-V4 PD disaggregation, prefill context parallelism, FlashComm1 sequence parallelism with schedule overlap, and host/device prefix cache (SWA + C4 + C128).
    • Support DeepSeek-V4 DSpark decoding and integrate DeepSeek-V4 EPLB with NPU fused MoE and MC2.
    • Support DeepSeek-V3.2 and GLM-5.2 W8A8 PyTorch adaptation, GLM-5.2 DSA top-k sharing, cache elision, and Mooncake PD; add expert-parallel (EP) support for the DeepSeek-V3.2 Python model and DeepSeek-V3.2 Python MTP under ACL graph; enable GLM-5.2 EPLv2 and shared-expert overlap.
    • Support KIMI-K25 W4A8 with ACL graph.
    • Support Qwen3.5 / Qwen3.6 with MegaMoE kernel, MegaChunkGDN fused operator, FlashComm1 + MMRS fusion, gated-delta layers, and heterogeneous PD disaggregation; add a Qwen3.5 Python CUDA-parity model implementation, optional FIA decode, context-parallel sequence sharding, and NPU Python ACL-graph data-parallel decode for dense Qwen3.
    • Support Qwen3.8-27B chat templates and add Qwen3-VL-dense to the Python model backend with Vi
  3. v0.10.12026-07-14 · 2026-07-14 07:20 UTC
    查看更新说明

    Release xllm 0.10.1

    Release Notes

    Highlights

    • Improve MTP correctness and stability during asynchronous execution.
    • Harden request cancellation, KV transfer completion, and disaggregated PD scheduling.
    • Improve OpenAI-compatible tool-call handling and DeepSeek streaming behavior.

    Bugfix

    • Preserve MTP inputs and persistent graph parameters across asynchronous execution.
    • Fix request cancellation races and guard KV transfer completion callbacks.
    • Fix chunked-prefill scheduling races in disaggregated PD deployments.
    • Handle OpenAI named tool choices and incomplete DeepSeek streaming payloads correctly.
    • Correct model version handling in the model-list API.
  4. v0.10.02026-07-01 · 2026-07-01 11:32 UTC
    查看更新说明

    Release xllm 0.10.0

    Major Features and Improvements

    Model Support

    NPU
    • Support DeepSeek-V4 model.
    • Support KIMI-K25 model.
    • Support MiniMax-M2.7 model.
    • Support JoyAI-LLM-Flash model.
    • Support QwenImageEditPlus model.
    • Support Qwen3-VL video inference.
    CUDA
    • Support Qwen3-MoE model.
    • Support MiMo-7B-Base model.
    • Support LongCat-AudioDiT model.
    MLU
    • Support DeepSeek-V4 model.
    • Support Qwen3.5 model.
    • Support OxygenVLM model.
    • Support Flux model.

    Feature

    • Support CANN 9.0.0 toolkit and torch_npu 2.9.0 for NPU devices.
    • Support DCU backend.
    • Support Torch 2.10.0 + CUDA 13.0 builds.
    • Update the MLU container to version 26.04.
    • Support xLLM service Anthropic PD protocol and improve Anthropic tool-call compatibility.
    • Support online profiling.
    • Support Python-based startup for online and offline services.
    • Support importing and exporting xLLM server startup flags through JSON config files.
    • Support cached token usage in responses and best_of_n in normal and disaggregated PD modes.
    • Support pause and resume for fully async RL.
    • Support offline speculative decoding and graph mode in offline inference.
    • Support multi-priority scheduling for PD disaggregation and prefix-cache-aware PD chunk budgeting.
    • Support namespace prefixes and optional authentication for etcd keys.
    • Add typed brpc handlers for API service endpoints and improve xLLM server startup routing.
    • Add encoder cache and request-transfer parallelization for multimodal models.
    • Support prefix cache for multimodal models and embedding interfaces for generate VLM models.
    • Add REC enhancements, including OneRec XAttention on NPU, REC XAttention for Qwen3-MoE on CUDA, constrained top-k sampling, beam-search num_return_sequences, extended item information, logprobs, and multi-item outputs.
    • Add Wan2.2 text encoder, scheduler, TP parallelism, VAE, and /v1/video/generation.
    • Improve KV cache internals with hybrid attention block manager, separated transfer logic, llmdatadist tensor registration, and centralized NPU allocation paths.
    • Add persistent Triton NPU runtime binary cache and improve xllm_ops incremental build and CI cache behavior.
    • Improve CUDA shared-memory tensor handling and add CUDA block-copy kernel support.
    • Add more manual model loaders and support parsing dtype fields from model config.

    Bugfix

    • Fix DeepSeek-V3.2 graph failures, prefix-cache starvation, and MLA option propagation in speculative paths.
    • Fix Qwen3.5 / Qwen3.6 decode reshape, TileLang gating, causal-conv1d tiling with padded batches, and repeated weight adjustment.
    • Fix NPU int4 MoE / torch communication initialization, prompt/context length limits, xattention accuracy, and offline TP initialization when selecting NPU kernel backend.
    • Fix multimodal build breaks, header paths, encoder-cache linking, QwenImageEdit precision, and preprocessing accuracy.
    • Fix MLU layer tests, MLU build issues, Mooncake PD paths, and local bind address handling.
    • Fix CUDA and ILU build errors, chunked-prefill CUDA failures, and pipeline LongCat Image Edit compile failures.
    • Fix service and API reliability issues including HTTP content types, console script metadata, brpc callback blocking, request cancellation/lifecycle shutdown, and model name extraction.
    • Fix REC XAttention fallback, REC tokenizer forwarding, OneRec input copies, ND format preservation, and beam helper exposure for non-NPU builds.
    • Fix build, CI, submodule cleanup, and release packaging issues around Triton NPU runtime assets and custom xllm math operator install paths.

    Release Images

    x86 image

    quay.io/jd_xllm/xllm-ai:xllm-0.10.0-release-hb-rc2-x86

    ARM a2 device image

    quay.io/jd_xllm/xllm-ai:xllm-0.10.0-release-hb-rc2-arm

    ARM a3 device image

    quay.io/jd_xllm/xllm-ai:xllm-0.10.0-release-hc-rc2-arm

  5. v0.9.12026-04-14 · 2026-04-14 11:47 UTC
    查看更新说明

    Highlights

    Highlights

    • Improve overall runtime stability and robustness for long-sequence and multi-scenario serving.
    • Refine distributed runtime behavior and compatibility under production deployments.
    • Enhance maintainability with incremental bug fixes and cleanup.

    Bugfix

    • Fix ACL graph related stability issues in DeepSeek-3.2 scenarios.
    • Improve CP/MTP compatibility behavior in edge cases.
    • Fix etcd auth and service registration related reliability issues.
    • Address memory rollback and worker lifecycle corner-case failures.

用户点评

登录用户的真实使用记录，可按版本和硬件筛选
共 0 条

    还没有人点评。第一个说出真实体验的人最有价值。

    README 原文

    未翻译
    在 GitHub 打开

    English | 中文

    Document Docker License report Ask DeepWiki

    News

    • 2026-04-24: 🎉 We day-0 support the DeepSeek-V4 model, please refer to the Deployment Document for deployment.
    • 2026-02-12: 🎉 We day-0 support high-performance inference for the GLM-5 model, please refer to the Deployment Document for deployment.
    • 2025-12-21: 🎉 We day-0 support high-performance inference for the GLM-4.7 model.
    • 2025-12-08: 🎉 We day-0 support high-performance inference for the GLM-4.6V model.
    • 2025-12-05: 🎉 We now support high-performance inference for the GLM-4.5/GLM-4.6 series models.
    • 2025-12-05: 🎉 We now support high-performance inference for the VLM-R1 model.
    • 2025-12-05: 🎉 We build hybrid KV cache management based on Mooncake, supporting global KV cache management with intelligent offloading and prefetching.
    • 2025-10-16: 🎉 We recently have released our xLLM Technical Report on arXiv, providing comprehensive technical blueprints and implementation insights.

    Overview

    xLLM is an efficient LLM inference framework, specifically optimized for Chinese AI accelerators, enabling enterprise-grade deployment with enhanced efficiency and reduced cost.

    Highlights

    • Top-tier Performance: Delivers high-throughput, low-latency inference through many advanced features.
    • Mainstream Hardware Support: Purpose-built and deeply optimized for Chinese AI accelerators.
    • Service-Engine Decoupled Architecture: Service layer handles scheduling and availability; engine layer handles computation.
    • Enterprise-grade Deployment: Battle-tested at scale across JD.com's core retail business.

    Hardware Support

    HardwareAbbreviationExampleRemark
    Ascend NPUNPUA2, A3HDK Driver 25.2.0 +
    Cambricon MLUMLUMLU
    Moore Threads GPUMUSAS5000
    Hygon DCUDCUBW1000
    MetaX MACAMACAMXC500
    Iluvatar CoreX GPUILUBI150

    Getting Started

    Community & Support

    Acknowledgment

    This project was made possible thanks to the following open-source projects:

    • ScaleLLM - xLLM draws inspiration from ScaleLLM's graph construction method and references its runtime execution.
    • Mooncake - Build xLLM hybrid KV cache management based on Mooncake.
    • brpc - Build high-performance http service based on brpc.
    • tokenizers-cpp - Build C++ tokenizer based on tokenizers-cpp.
    • safetensors - xLLM relies on the C binding safetensors capability.
    • Partial JSON Parser - Implement xLLM's C++ JSON parser with insights from Python and Go implementations.
    • concurrentqueue - A fast multi-producer, multi-consumer lock-free concurrent queue for C++11.

    Thanks to the following collaborating university laboratories:

    • THU-MIG (School of Software, BNRist, Tsinghua University)
    • USTC-Cloudlab (Cloud Computing Lab, University of Science and Technology of China)
    • Beihang-HiPO (Beihang HiPO research group)
    • PKU-DS-LAB (Data Structure Laboratory, Peking University)
    • PKU-NetSys-LAB (NetSys Lab, Peking University)
    • TJU-TANKLab (TANK Lab, Tianjin University)

    Thanks to all the following developers who have contributed to xLLM.

    Citation

    If you think this repository is helpful to you, welcome to cite us:

    @article{liu2025xllm,
  title={xLLM Technical Report},
  author={xLLM team},
  journal={arXiv preprint arXiv:2510.14686},
  year={2025}
}

    相关项目

    同分类里 star 最高的项目
    对比全部 →
    ollama
    ollama·Go

    本地一键跑起 Kimi、DeepSeek、Qwen、Gemma 等大模型，省去环境配置和依赖折腾，适合快速做推理和原型验证。

    本地推理+3大模型推理服务入门
    186510+32019kv0.40.152026-11-10 01:49 UTC
    llama.cpp
    ggml-org·C++

    用 C/C++ 实现的大模型推理引擎，能在普通 CPU 和消费级显卡上跑 LLaMA 等模型，支持量化以降低显存占用。

    推理引擎+4大模型推理服务入门
    133898+30125kb122332026-11-10 10:59 UTC
    vllm
    vllm-project·Python

    面向大语言模型的高吞吐、低显存推理与服务引擎，通过 PagedAttention 等优化显著提升并发吞吐并降低显存占用，适合部署在线推理服务。

    推理引擎+4大模型推理服务进阶
    96670+17725kv0.33.0rc32026-11-10 11:13 UTC
    jan
    janhq·Rust

    在本地电脑上离线运行大模型的桌面客户端，是 ChatGPT 的开源替代品，无需联网即可对话，数据不出本机。

    本地推理+4大模型推理服务
    45651+1093.2kv0.8.82026-11-10 09:10 UTC