登录

inference

xorbitsaiApache-2.0活跃维护

一行代码就能把 GPT 换成任意开源大模型，统一推理 API 支持云端、本地和笔记本部署，覆盖语言、语音和多模态模型。

Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.

v4.2.0发布于 2026-11-07 13:07 UTC9,650star +3 / 7 天898fork同步于 2026-11-10 11:30 UTC
推理服务大模型部署多模态推理统一API本地推理
对比 GitHub 官网artificial-intelligencediffusersgemmaglmglm-5-3

怎么上手

由 README 提炼，执行前请确认
原文 
pip install xinference
xinference-local
硬件要求按 README 说明

README 未给出具体显存数值；Docker 方式面向 Nvidia GPU 用户，需预先装好 Docker 与 CUDA，K8s 部署也要求集群具备 GPU 支持；README 提到可在笔记本上运行，但未说明 CPU 运行细节。

需要 CUDA有 Docker 镜像
下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼
没人工点评？写下你的经验

一条命令部署语言、语音与多模态模型的统一推理服务，换模型只改一行代码

一行代码就能把 GPT 换成任意开源大模型，统一推理 API 支持云端、本地和笔记本部署，覆盖语言、语音和多模态模型。

核心能力

  • 一条命令启动本地推理服务并自带 Web UI
  • 统一 API 服务 LLM、语音识别与多模态模型
  • 内置大量开源模型，也支持自定义模型接入
  • 多引擎 Xavier 支持异构 P/D 与运行时重配置
  • 自动批处理合并并发请求以提升吞吐

容易踩的坑

Docker 启动需加 --gpus all 并挂载 /data 目录，否则模型缓存无法持久化；K8s 安装需集群已具备 GPU 支持；README 未说明默认端口以外的网络配置，Docker 示例映射的是 9997 端口。

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

最近发版

5 个版本v4.2.0
全部 release
  1. v4.2.0最新正式版2026-11-07 · 2026-11-07 13:07 UTC

    What's new in 4.2.0 (2026-11-07)

    Highlights

    • Xavier prefix cache sharing. Reuse compatible prefixes across replicas running vLLM, SGLang, TRT-LLM, or MLX. Xavier expands LMCache storage support with Mooncake Store and can prioritize retaining KV prefixes while agents wait for tool results, reducing repeated prefill when they resume. (#5917, #5905, #6018)

    • Software updates and rolling model reloads. Administrators can check for and install Xinference updates in the Web UI, upgrading Workers, the Supervisor, and the API in order. Supported vLLM and SGLang deployments can retain GPU weights and reload one replica at a time while in-flight requests finish on the previous engine instance. (#5873, #5944)

    • Start model benchmarks from the Web UI. Run tests at different concurrency levels against a running chat model, compare throughput, time to first token (TTFT), and time per output token (TPOT) across runs, and export results as JSON. (#5904)

    • Native reasoning controls and request priority across LLM engines. The OpenAI- and Anthropic-compatible APIs expose native reasoning effort for supported models. The OpenAI-compatible API also supports a thinking budget on compatible runtimes, and the Web UI shows controls when available. Chat and text completion requests use a common priority parameter across all supported LLM engines, subject to engine version and configuration. (#6001, #6019)

    • Real-time Qwen3 audio. Serve concurrent real-time speech recognition with Qwen3-ASR and streaming speech synthesis with Qwen3-TTS CustomVoice through vLLM. (#5993)

    These are the changes in inference v4.2.0.

    New features

  2. v4.1.02026-10-25 · 2026-10-25 07:33 UTC
    查看更新说明

    What's new in 4.1.0 (2026-10-25)

    These are the changes in inference v4.1.0.

    Highlights

    • TRT-LLM is now an inference backend. Run supported models through Xinference's unified serving API with TensorRT-LLM, including chat, generation and streaming.

    • Xavier now supports TRT-LLM. Extend prefix KV cache sharing to TRT-LLM and use native or Xavier prefill/decode separation. Compatible models can hand off between TRT-LLM, vLLM, SGLang and MLX, including NVIDIA–Apple silicon deployments.

    • LMCache disk storage for Xavier. Extend prefix KV cache storage beyond memory with an optional LMCache backend for vLLM, SGLang and TRT-LLM.

    • Live upgrades that preserve in-flight inference (experimental). Upgrade supported model engines and replace Worker, API and Supervisor processes on POSIX, with Web UI controls for local and distributed deployments. Existing requests and streams finish before the old serving processes retire.

    • Consistent context-length controls. Configure model context limits across inference engines and adjust them through the Web UI slider, with validation that follows the effective model configuration.

    • Expanded model support. Add DeepSeek-V4.1-Flash-1017, GLM-5.3 and Qwen3.8-Flash-Next, alongside new multimodal, image, speech and OCR models such as MiniCPM-V-4.7, FLUX 3, Ming-3.0 TTS and Xiaomi-OCR-0.

    Security

    Xinference 4.1.0 is the first patched release for the following published advisories. Users of affected releases should upgrade to 4.1.0 or later.

    • Critical — CosyVoice model loading: require explicit operator trust before loading custom sources or explicit local paths, preventing untrusted HyperPyYAML configuration from reaching the loader without that opt-in. GHSA-2gq8-2q98-6w29 · #5765.
    • High — Initial administrator setup: require an operator setup token before creating the first administrator, preventing unauthenticated callers from claiming an uninitialized server. Affects versions >= 3.0.0, < 4.1.0. GHSA-v43j-w7xm-2vwh · #5703.
    • Medium — Auto-registration paths and Flexible launchers: confine model-configuration reads to approved roots and require an operator allowlist for additional launchers. The reported impacts are partial configuration disclosure/filesystem probing and arbitrary module import. GHSA-3jw7-r8gq-5ggj · #5766.
    • Medium — Chat-completion error disclosure: return client-safe HTTP/SSE errors without exposing raw worker addresses, process identifiers or other unexpected backend diagnostics. GHSA-h97g-6mc6-66hm · #5767.

    Thanks to Paul White (Su1ph3r), Cairn Security for reporting the initial-administrator issue and verifying the API-level remediation.

    Upgrade notes: For fresh instances, use the setup token in the API server startup log to create the initial administrator. Custom CosyVoice sources require worker-side XINFERENCE_TRUST_REMOTE_CODE=1 and must still be trusted. Additional registration roots and Flexible launchers require XINFERENCE_MODEL_REGISTER_ROOTS and XINFERENCE_FLEXIBLE_LAUNCHERS, respectively; see the advisories for scope and configuration.

    Additional security improvements in this release:

    • Validate shared-store API keys against current state. #5770.
    • Integrate Xoscar actor transport authentication. This is opt-in: configure the same nonempty XOSCAR_AUTH_KEY on every actor peer; upgrading a
  3. v4.0.12026-10-16 · 2026-10-16 11:38 UTC
    查看更新说明

    What's new in 4.0.1 (2026-10-16)

    This patch release is based on v4.0.0 and backports the actor transport authentication integration from #5696.

    Security

    • Require Xoscar 0.12.0 or later in the Python package and CPU image dependencies. Xoscar 0.12.0 introduces opt-in HMAC-SHA256 authentication of actor messages before Python object deserialization, mitigating the issue described in CVE-2026-49950.
    • Pass the deployment's XOSCAR_AUTH_KEY through the standalone and distributed Docker Compose configurations, including all supervisor and worker services.
    • Add deployment guidance and translations for all nine existing documentation locales, plus integration coverage for local/distributed communication, authentication rejection, and model-subprocess recovery.

    Actor transport authentication is disabled unless a nonempty XOSCAR_AUTH_KEY is configured. Upgrading alone does not enable this protection. HTTP API authentication does not protect actor ports.

    Upgrade and enable authentication

    1. Upgrade every Xinference server to 4.0.1, or use the corresponding updated container image. Ensure any direct Xoscar actor clients also use Xoscar 0.12.0 or later.

    2. Generate one random secret per deployment, for example by running the following command once, and distribute it through your secret-management system:

      python -c "import secrets; print(secrets.token_hex(32))"

    3. Set the same generated value before starting xinference-local, xinference-supervisor, every xinference-worker, and any direct actor clients:

      export XOSCAR_AUTH_KEY='<same-random-deployment-secret>'

      For the supplied Compose configurations, set this value in .env or export it before running Compose. Child processes, including model virtual environments, inherit the key. HTTP/REST clients do not need this secret.

    4. Coordinate a restart of all actor peers. Keyed and unkeyed peers cannot communicate; enabling or rotating the key requires a coordinated rollout, with no automatic downgrade or mixed-mode rolling upgrade.

    Deployment administrators must continue restricting actor ports to intended cluster peers. Anyone holding the shared key is a trusted peer with Python execution privileges. Authentication does not encrypt traffic or prevent replay of captured messages. Authenticated GPU transfers stage device buffers in host memory for hashing, adding CPU and host-memory overhead.

    See the Xinference environment configuration and Xoscar actor transport authentication documentation for details.

    Full Changelog: https://github.com/xorbitsai/inference/compare/v4.0.0...v4.0.1

  4. v4.0.02026-10-10 · 2026-10-10 01:53 UTC
    查看更新说明

    What's new in 4.0.0 (2026-10-10)

    Highlights

    Xinference 4.0 brings Xavier to more inference engines, enables heterogeneous prefill/decode deployments across engines and hardware, and lets you reconfigure running models without reloading their weights.

    • Xavier across vLLM, SGLang, and MLX. Xavier now supports all three engines, extending prefix KV cache sharing across compatible replicas. Reuse previously computed context across requests and replicas to avoid repeated prefill work.

    • Cross-engine and cross-hardware P/D separation. Run prefill and decode with different inference engines, including vLLM ↔ SGLang and bidirectional NVIDIA ↔ Apple silicon deployments. Choose engines and hardware independently for each role through Xinference’s unified serving interface.

    • Reload engines without reloading model weights. Opt-in GPU weight caching lets vLLM and SGLang retain weights while rebuilding the inference engine. Adjust supported concurrency, context, memory, and CUDA graph settings while preserving the model UID.

    • Distributed vLLM V1 inference. Combine tensor and pipeline parallelism across Xinference workers through the xoscar executor, with improved pipeline scheduling and failure handling.

    • One-command installation and service management. Install and start Xinference on Linux, macOS, and Windows in an isolated Python environment. Optional system services support background operation and automatic startup, with rollback for failed managed service upgrades.

    • OpenAI Responses API support. Serve Responses-based clients through the new stateless /v1/responses endpoint, including streaming and tool calls. Cache-hit token reporting across supported engines makes prompt reuse more visible.

    New features

  5. v3.5.02026-09-25 · 2026-09-25 19:35 UTC
    查看更新说明

    What's new in 3.5.0 (2026-09-25)

    These are the changes in inference v3.5.0.

    Security

    Xinference 3.5.0 fixes a custom-model registration vulnerability that allowed a caller to forge built-in provenance and bypass repository-code execution restrictions. See GHSA-jv65-rccw-j2j6 for affected versions and authentication requirements.

    Upgrade every supervisor and worker to 3.5.0 or a later supported release. Also review existing persisted custom-model registrations for an unexpected is_builtin: true value: the patch sanitizes new registration requests but does not rewrite registrations saved before the upgrade. The advisory includes cleanup guidance.

    Thanks to Paul White (Su1ph3r), Cairn Security for reporting and validating this issue, and to Jan Kahmen (kah-ja), turingpoint for the independent report and patch.

    New features

用户点评

登录用户的真实使用记录，可按版本和硬件筛选
共 0 条

    还没有人点评。第一个说出真实体验的人最有价值。

    README 原文

    未翻译
    在 GitHub 打开

    Xorbits Inference: Model Serving Made Easy 🤖

    PyPI Latest Release License Build Status Docker Pulls Discord Telegram Twitter

    Xorbits Inference(Xinference) is a powerful and versatile library designed to serve language, speech recognition, and multimodal models. With Xorbits Inference, you can effortlessly deploy and serve your or state-of-the-art built-in models using just a single command. Whether you are a researcher, developer, or data scientist, Xorbits Inference empowers you to unleash the full potential of cutting-edge AI models.

    🔥 Hot Topics

    Framework Enhancements

    • Xinference 4.0.0 is available with multi-engine Xavier, heterogeneous P/D, and runtime model reconfiguration: Release Notes
    • Agent-native Serving: Xinference integrates with Xagent to enable dynamic planning, tool use, and autonomous multi-step reasoning — moving beyond static pipelines.
    • Auto batch: Multiple concurrent requests are automatically batched, significantly improving throughput: #4197
    • Xllamacpp: New llama.cpp Python binding, maintained by Xinference team, supports continuous batching and is more production-ready.: #2997
    • Distributed inference: running models across workers: #2877
    • VLLM enhancement: Shared KV cache across multiple replicas: #2732

    New Models

    Integrations

    • Xagent: an enterprise agent platform for building and running AI agents with planning, memory, and tool use — not limited to rigid workflows.
    • Dify: an LLMOps platform that enables developers (and even non-developers) to quickly build useful applications based on large language models, ensuring they are visual, operable, and improvable.
    • FastGPT: a knowledge-based platform built on the LLM, offers out-of-the-box data processing and model invocation capabilities, allows for workflow orchestration through Flow visualization.
    • RAGFlow: is an open-source RAG engine based on deep document understanding.
    • MaxKB: MaxKB = Max Knowledge Brain, it is a powerful and easy-to-use AI assistant that integrates Retrieval-Augmented Generation (RAG) pipelines, supports robust workflows, and provides advanced MCP tool-use capabilities.

    Key Features

    🌟 Model Serving Made Easy: Simplify the process of serving large language, speech recognition, and multimodal models. You can set up and deploy your models for experimentation and production with a single command.

    ⚡️ State-of-the-Art Models: Experiment with cutting-edge built-in models using a single command. Inference provides access to state-of-the-art open-source models!

    🖥 Heterogeneous Hardware Utilization: Make the most of your hardware resources with ggml. Xorbits Inference intelligently utilizes heterogeneous hardware, including GPUs and CPUs, to accelerate your model inference tasks.

    ⚙️ Flexible API and Interfaces: Offer multiple interfaces for interacting with your models, supporting OpenAI compatible RESTful API (including Function Calling API), RPC, CLI and WebUI for seamless model management and interaction.

    🌐 Distributed Deployment: Excel in distributed deployment scenarios, allowing the seamless distribution of model inference across multiple devices or machines.

    🔌 Built-in Integration with Third-Party Libraries: Xorbits Inference seamlessly integrates with popular third-party libraries including LangChain, LlamaIndex, Dify, and Chatbox.

    Why Xinference

    FeatureXinferenceFastChatOpenLLMRayLLM
    OpenAI-Compatible RESTful API✅✅✅✅
    vLLM Integrations✅✅✅✅
    More Inference Engines (GGML, TensorRT)✅❌✅✅
    More Platforms (CPU, Metal)✅✅❌❌
    Multi-node Cluster Deployment✅❌❌✅
    Image Models (Text-to-Image)✅✅❌❌
    Text Embedding Models✅❌❌❌
    Multimodal Models✅❌❌❌
    Audio Models✅❌❌❌
    More OpenAI Functionalities (Function Calling)✅❌❌❌

    Using Xinference

    • Self-hosting Xinference Community Edition Quickly get Xinference running in your environment with this starter guide. Use our documentation for further references and more in-depth instructions.

    • Xinference for enterprise / organizations Xinference Enterprise is the commercial edition for teams running models in production. Deploy it in your own cloud or data centre, or choose a managed dedicated deployment with Xinference Cloud. The hosted Model API gives you access to supported models through an OpenAI-compatible API without managing inference infrastructure. Learn more at xinference.co. For enterprise enquiries, email us.

    Staying Ahead

    Star Xinference on GitHub and be instantly notified of new releases.

    star-us

    Getting Started

    Quick Start

    pip

    pip install xinference
xinference-local

    One-command installer

    Linux/macOS:

    curl -fsSL https://raw.githubusercontent.com/xorbitsai/inference/main/scripts/install.sh | sh

    Windows PowerShell:

    irm https://raw.githubusercontent.com/xorbitsai/inference/main/scripts/install.ps1 | iex

    Installation guide · Usage guide

    Docker

    Nvidia GPU users can start Xinference server using Xinference Docker Image. Prior to executing the installation command, ensure that both Docker and CUDA are set up on your system.

    docker run --name xinference -d -p 9997:9997 -e XINFERENCE_HOME=/data -v </on/your/host>:/data --gpus all xprobe/xinference:latest xinference-local -H 0.0.0.0

    K8s via helm

    Ensure that you have GPU support in your Kubernetes cluster, then install as follows.

    # add repo
helm repo add xinference https://xorbitsai.github.io/xinference-helm-charts

# update indexes and query xinference versions
helm repo update xinference
helm search repo xinference/xinference --devel --versions

# install xinference
helm install xinference xinference/xinference -n xinference --version 0.0.1-v<xinference_release_version>

    For more customized installation methods on K8s, please refer to the documentation.

    web UI

    Getting involved

    PlatformPurpose
    Github IssuesReporting bugs and filing feature requests.
    DiscordCollaborating with other Xinference users.
    TelegramChatting with other Xinference users.
    TwitterStaying up-to-date on new features.

    Citation

    If this work is helpful, please kindly cite as:

    @inproceedings{lu2024xinference,
    title = "Xinference: Making Large Model Serving Easy",
    author = "Lu, Weizheng and Xiong, Lingfeng and Zhang, Feng and Qin, Xuye and Chen, Yueguo",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.emnlp-demo.30",
    pages = "291--300",
}

    Contributors

    Star History

    Star History Chart

    相关项目

    同分类里 star 最高的项目
    对比全部 →
    ollama
    ollama·Go

    本地一键跑起 Kimi、DeepSeek、Qwen、Gemma 等大模型，省去环境配置和依赖折腾，适合快速做推理和原型验证。

    本地推理+3大模型推理服务入门
    186510+32019kv0.40.152026-11-10 01:49 UTC
    llama.cpp
    ggml-org·C++

    用 C/C++ 实现的大模型推理引擎，能在普通 CPU 和消费级显卡上跑 LLaMA 等模型，支持量化以降低显存占用。

    推理引擎+4大模型推理服务入门
    133898+30125kb122332026-11-10 10:59 UTC
    vllm
    vllm-project·Python

    面向大语言模型的高吞吐、低显存推理与服务引擎，通过 PagedAttention 等优化显著提升并发吞吐并降低显存占用，适合部署在线推理服务。

    推理引擎+4大模型推理服务进阶
    96670+17725kv0.33.0rc32026-11-10 11:13 UTC
    jan
    janhq·Rust

    在本地电脑上离线运行大模型的桌面客户端，是 ChatGPT 的开源替代品，无需联网即可对话，数据不出本机。

    本地推理+4大模型推理服务
    45651+1093.2kv0.8.82026-11-10 09:10 UTC