登录

lmdeploy

InternLMApache-2.0活跃维护

LMDeploy 是面向大模型的高效推理与服务工具包，支持模型压缩、部署和在线服务，内置 TurboMind 推理引擎和 CUDA 算子，可加速 Llama、InternLM 等模型的推理。

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

v0.18.0发布于 2026-09-28 07:04 UTC8,136star774fork同步于 2026-11-10 05:31 UTC
大模型推理推理引擎模型压缩部署服务TurboMind
对比 GitHub 官网codellamacuda-kernelsdeepspeedfastertransformerinternlm

怎么上手

由 README 提炼，执行前请确认
原文 
conda create -n lmdeploy python=3.12 -y
conda activate lmdeploy
pip install lmdeploy
硬件要求按 README 说明

README 未给出具体显存要求；预编译 wheel 自 v0.13.0 起基于 CUDA 12.8 构建，覆盖 GeForce RTX 50 系列；TurboMind 支持 NVIDIA GPU（MXFP4 自 V100 起），PyTorch 引擎支持华为 Ascend。

支持 Apple Silicon需要 CUDA昇腾 / Ascend多卡 / 张量并行
下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼
没人工点评？写下你的经验

压缩、部署并服务大模型的一站式工具包，主打高吞吐推理与量化

LMDeploy 是面向大模型的高效推理与服务工具包，支持模型压缩、部署和在线服务，内置 TurboMind 推理引擎和 CUDA 算子，可加速 Llama、InternLM 等模型的推理。

核心能力

  • 用持续批处理与分块 KV 缓存提升请求吞吐
  • 支持权重与 KV 的 4bit 量化推理
  • 提供 TurboMind 与纯 Python 的 PyTorch 双推理引擎
  • 支持张量并行与多机多卡推理服务
  • 提供 OpenAI 兼容的 LLM/VLM 服务接口

容易踩的坑

默认从 HuggingFace 下载模型，用 ModelScope 需装 modelscope 并设 LMDEPLOY_USE_MODELSCOPE=True；建议在 conda 环境用 Python 3.10–3.13 安装；预编译 wheel 基于 CUDA 12.8，其他 CUDA 版本需自行处理。

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

最近发版

5 个版本v0.18.0
全部 release
  1. v0.18.0最新正式版2026-09-28 · 2026-09-28 07:04 UTC

    What's Changed

    🚀 Features

    💥 Improvements

    🐞 Bug fixes

  2. v0.17.02026-09-01 · 2026-09-01 07:13 UTC
    查看更新说明

    What's Changed

    🚀 Features

    💥 Improvements

    🐞 Bug fixes

    🌐 Other

    New Contributors

    Full Changelog: https://github.com/InternLM/lmdeploy/compare/v0.16.0...v0.17.0

  3. v0.16.02026-08-19 · 2026-08-19 04:43 UTC
    查看更新说明

    What's Changed

    🚀 Features

    💥 Improvements

    🐞 Bug fixes

    📚 Documentations

    🌐 Other

  4. v0.15.02026-07-31 · 2026-07-31 13:00 UTC
    查看更新说明

    What's Changed

    🚀 Features

    💥 Improvements

    🐞 Bug fixes

  5. v0.14.02026-06-24 · 2026-06-24 04:36 UTC
    查看更新说明

    What's Changed

    🚀 Features

    💥 Improvements

    🐞 Bug fixes

用户点评

登录用户的真实使用记录，可按版本和硬件筛选
共 0 条

    还没有人点评。第一个说出真实体验的人最有价值。

    README 原文

    未翻译
    在 GitHub 打开

    PyPI PyPI - Downloads license issue resolution open issues

    📘Documentation | 🛠️Quick Start | 🤔Reporting Issues

    English | 简体中文 | 日本語

    Latest News 🎉

    • [2025/09] TurboMind supports MXFP4 on NVIDIA GPUs starting from V100, achieving 1.5x the performance of vLLM on H800 for openai gpt-oss models!
    • [2025/06] Comprehensive inference optimization for FP8 MoE Models
    • [2025/06] DeepSeek PD Disaggregation deployment is now supported through integration with DLSlime and Mooncake. Huge thanks to both teams!
    • [2025/04] Enhance DeepSeek inference performance by integrating deepseek-ai techniques: FlashMLA, DeepGemm, DeepEP, MicroBatch and eplb
    • [2025/01] Support DeepSeek V3 and R1
    • [2024/11] Support Mono-InternVL with PyTorch engine
    • [2024/10] PyTorchEngine supports graph mode on ascend platform, doubling the inference speed
    • [2024/09] LMDeploy PyTorchEngine adds support for Huawei Ascend. See supported models here
    • [2024/09] LMDeploy PyTorchEngine achieves 1.3x faster on Llama3-8B inference by introducing CUDA graph
    • [2024/08] LMDeploy is integrated into modelscope/swift as the default accelerator for VLMs inference
    • [2024/07] Support Llama3.1 8B, 70B and its TOOLS CALLING
    • [2024/07] Support InternVL2 full-series models, InternLM-XComposer2.5 and function call of InternLM2.5
    • [2024/06] PyTorch engine support DeepSeek-V2 and several VLMs, such as CogVLM2, Mini-InternVL, LlaVA-Next
    • [2024/05] Balance vision model when deploying VLMs with multiple GPUs
    • [2024/05] Support 4-bit weight-only quantization and inference on VLMs, such as InternVL v1.5, LLaVa, InternLMXComposer2
    • [2024/04] Support Llama3 and more VLMs, such as InternVL v1.1, v1.2, MiniGemini, InternLMXComposer2.
    • [2024/04] TurboMind adds online int8/int4 KV cache quantization and inference for all supported devices. Refer here for detailed guide
    • [2024/04] TurboMind latest upgrade boosts GQA, rocketing the internlm2-20b model inference to 16+ RPS, about 1.8x faster than vLLM.
    • [2024/04] Support Qwen1.5-MOE and dbrx.
    • [2024/03] Support DeepSeek-VL offline inference pipeline and serving.
    • [2024/03] Support VLM offline inference pipeline and serving.
    • [2024/02] Support Qwen 1.5, Gemma, Mistral, Mixtral, Deepseek-MOE and so on.
    • [2024/01] OpenAOE seamless integration with LMDeploy Serving Service.
    • [2024/01] Support for multi-model, multi-machine, multi-card inference services. For usage instructions, please refer to here
    • [2024/01] Support PyTorch inference engine, developed entirely in Python, helping to lower the barriers for developers and enable rapid experimentation with new features and technologies.
    • [2023/12] Turbomind supports multimodal input.
    • [2023/11] Turbomind supports loading hf model directly. Click here for details.
    • [2023/11] TurboMind major upgrades, including: Paged Attention, faster attention kernels without sequence length limitation, 2x faster KV8 kernels, Split-K decoding (Flash Decoding), and W4A16 inference for sm_75
    • [2023/09] TurboMind supports Qwen-14B
    • [2023/09] TurboMind supports InternLM-20B
    • [2023/09] TurboMind supports all features of Code Llama: code completion, infilling, chat / instruct, and python specialist. Click here for deployment guide
    • [2023/09] TurboMind supports Baichuan2-7B
    • [2023/08] TurboMind supports flash-attention2.
    • [2023/08] TurboMind supports Qwen-7B, dynamic NTK-RoPE scaling and dynamic logN scaling
    • [2023/08] TurboMind supports Windows (tp=1)
    • [2023/08] TurboMind supports 4-bit inference, 2.4x faster than FP16, the fastest open-source implementation. Check this guide for detailed info
    • [2023/08] LMDeploy has launched on the HuggingFace Hub, providing ready-to-use 4-bit models.
    • [2023/08] LMDeploy supports 4-bit quantization using the AWQ algorithm.
    • [2023/07] TurboMind supports Llama-2 70B with GQA.
    • [2023/07] TurboMind supports Llama-2 7B/13B.
    • [2023/07] TurboMind supports tensor-parallel inference of InternLM.

    Introduction

    LMDeploy is a toolkit for compressing, deploying, and serving LLM, developed by the MMRazor and MMDeploy teams. It has the following core features:

    • Efficient Inference: LMDeploy delivers up to 1.8x higher request throughput than vLLM, by introducing key features like persistent batch(a.k.a. continuous batching), blocked KV cache, dynamic split&fuse, tensor parallelism, high-performance CUDA kernels and so on.

    • Effective Quantization: LMDeploy supports weight-only and k/v quantization, and the 4-bit inference performance is 2.4x higher than FP16. The quantization quality has been confirmed via OpenCompass evaluation.

    • Effortless Distribution Server: Leveraging the request distribution service, LMDeploy facilitates an easy and efficient deployment of multi-model services across multiple machines and cards.

    • Excellent Compatibility: LMDeploy supports KV Cache Quant, AWQ and Automatic Prefix Caching to be used simultaneously.

    Performance

    v0 1 0-benchmark

    Supported Models

    LMDeploy has developed two inference engines - TurboMind and PyTorch, each with a different focus. The former strives for ultimate optimization of inference performance, while the latter, developed purely in Python, aims to decrease the barriers for developers.

    They differ in the types of supported models and the inference data type. Please refer to this table for each engine's capability and choose the proper one that best fits your actual needs.

    Quick Start Open In Colab

    Installation

    It is recommended to install lmdeploy using pip in a conda environment (python 3.10 - 3.13):

    conda create -n lmdeploy python=3.12 -y
conda activate lmdeploy
pip install lmdeploy

    Starting from v0.13.0, the default prebuilt wheels published on PyPI are built against CUDA 12.8, so pip install lmdeploy is sufficient for typical setups including GeForce RTX 50 series.

    Offline Batch Inference

    import lmdeploy
with lmdeploy.pipeline("internlm/internlm3-8b-instruct") as pipe:
    response = pipe(["Hi, pls intro yourself", "Shanghai is"])
    print(response)

    [!NOTE] By default, LMDeploy downloads model from HuggingFace. If you would like to use models from ModelScope, please install ModelScope by pip install modelscope and set the environment variable:

    export LMDEPLOY_USE_MODELSCOPE=True

    If you would like to use models from openMind Hub, please install openMind Hub by pip install openmind_hub and set the environment variable:

    export LMDEPLOY_USE_OPENMIND_HUB=True

    For more information about inference pipeline, please refer to here.

    Tutorials

    Please review getting_started section for the basic usage of LMDeploy.

    For detailed user guides and advanced guides, please refer to our tutorials:

    Third-party projects

    • Deploying LLMs offline on the NVIDIA Jetson platform by LMDeploy: LMDeploy-Jetson

    • Example project for deploying LLMs using LMDeploy and BentoML: BentoLMDeploy

    Contributing

    We appreciate all contributions to LMDeploy. Please refer to CONTRIBUTING.md for the contributing guideline.

    Acknowledgement

    Citation

    @misc{2023lmdeploy,
    title={LMDeploy: A Toolkit for Compressing, Deploying, and Serving LLM},
    author={LMDeploy Contributors},
    howpublished = {\url{https://github.com/InternLM/lmdeploy}},
    year={2023}
}

    @article{zhang2025lmdeploy,
  title={LMDeploy Accelerates Mixed-Precision LLM Inference with TurboMind},
  author={Zhang, Li and Jiang, Youhe and He, Guoliang and Chen, Xin and Lv, Han and Yao, Qian and Ma, Ningsheng and Fu, Fangcheng and Chen, Kai},
  journal={arXiv preprint arXiv:2508.15601},
  year={2025}
}

    License

    This project is released under the Apache 2.0 license.

    相关项目

    同分类里 star 最高的项目
    对比全部 →
    ollama
    ollama·Go

    本地一键跑起 Kimi、DeepSeek、Qwen、Gemma 等大模型，省去环境配置和依赖折腾，适合快速做推理和原型验证。

    本地推理+3大模型推理服务入门
    186472+28219kv0.40.152026-11-10 01:49 UTC
    llama.cpp
    ggml-org·C++

    用 C/C++ 实现的大模型推理引擎，能在普通 CPU 和消费级显卡上跑 LLaMA 等模型，支持量化以降低显存占用。

    推理引擎+4大模型推理服务入门
    133865+26825kb122332026-11-10 01:44 UTC
    vllm
    vllm-project·Python

    面向大语言模型的高吞吐、低显存推理与服务引擎，通过 PagedAttention 等优化显著提升并发吞吐并降低显存占用，适合部署在线推理服务。

    推理引擎+4大模型推理服务进阶
    96643+15025kv0.33.0rc32026-11-10 05:17 UTC
    jan
    janhq·Rust

    在本地电脑上离线运行大模型的桌面客户端，是 ChatGPT 的开源替代品，无需联网即可对话，数据不出本机。

    本地推理+4大模型推理服务
    45644+1023.2kv0.8.82026-11-10 04:20 UTC