rtp-llm

alibabaApache-2.0活跃维护

阿里开源的高性能大模型推理引擎，支持 GPT、Llama 等多种模型，面向各类应用提供低延迟、高吞吐的在线推理服务。

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

v0.2.0发布于 2025-10-31 07:54 UTC1,394star301fork同步于 2026-11-10 17:50 UTC

推理引擎大模型推理模型服务高吞吐低延迟

对比 GitHub gpt inference llama llm llm-serving

怎么上手

由 README 提炼，执行前请确认

原文

README 未给出安装命令，仅链接到官方安装文档 https://rtp-llm.ai/build/en/start/install.html

硬件要求按 README 说明

README 未给出具体显存要求；支持 NVIDIA GPU 并对 V100 做了专门优化，Qwen 系列与 bert embedding 模型可在倚天 ARM CPU 上运行，AMD ROCm 与 Intel CPU 支持在开发中。

需要 CUDA支持 ROCm / AMD多卡 / 张量并行

下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼

没人工点评？写下你的经验

阿里开源的高性能 LLM 推理加速引擎，支撑淘宝、天猫等业务的大模型在线服务

阿里开源的高性能大模型推理引擎，支持 GPT、Llama 等多种模型，面向各类应用提供低延迟、高吞吐的在线推理服务。

核心能力

用 PagedAttention、FlashAttention、FlashDecoding 等 CUDA 内核加速
加载时自动做 WeightOnly INT8 量化，支持 GPTQ/AWQ 的 INT4
支持 Prefill/Decode 分离与多机多卡张量并行
单实例部署多个 LoRA 服务，支持多模态图文输入
提供上下文前缀缓存、系统提示缓存与投机解码

容易踩的坑

安装与快速上手步骤不在 README 内，需按官网文档操作；项目基于 FasterTransformer 并集成 TensorRT-LLM 内核，编译依赖可能较重；使用 INT4 量化需准备 GPTQ 或 AWQ 格式的权重。

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

最近发版

5 个版本v0.2.0

全部 release

v0.2.0最新正式版2025-10-31 · 2025-10-31 07:54 UTC

Overview

RTP-LLM First Release Version:0.2.0(2025.09)

Features

Framkework Advanced Feature

PD Disaggregation && PD Entrance Transpose
Attention Support more Backend: XQA, FlashInfer
Speculative Decoding
EPLB
MicroBatch & Overlapping
MTP
DeepEP
LoadBalance
3FS
FP8 KVCache
REUSE KV CACHE
Quantization
MultiLoRA
Attention FFN Disaggregation
Frontend/Backend Disaggregation

New Models

Model Family (Variants)	Example HuggingFace Identifier	Description	Support CardType
DeepSeek (v1, v2, v3/R1)	`deepseek-ai/DeepSeek-R1`	Series of advanced reasoning-optimized models (including a 671B MoE) trained with reinforcement learning; top performance on complex reasoning, math, and code tasks. RTP-LLM provides Deepseek v3/R1 model-specific optimizations	NV ✅ AMD ✅
Kimi (Kimi-K2)	`moonshotai/Kimi-K2-Instruct`	Moonshot's MoE LLMs with 1 trillion parameters, exceptional on agentic intellegence	NV ✅ AMD ✅
Qwen (v1, v1.5, v2, v2.5, v3, QWQ, Qwen3-Coder)	`Qwen/Qwen3-235B-A22B`	Series of advanced reasoning-optimized models, Significantly improved performance on reasoning tasks, including logical reasoning, mathematics, science, coding, and academic benchmarks that typically require human expertise — achieving state-of-the-art results among open-source thinking models.Markedly better general capabilities, such as instruction following, tool usage, text generation, and alignment with human preferences.Enhanced 256K long-context understanding capabilities.	NV ✅ AMD ✅
QwenVL (VL2, VL2.5, VL3)	`Qwen/Qwen2-VL-2B`	Series of advanced Vision-language model series based on Qwen2.5/Qwen3	NV ✅ AMD ❌
Llama	`meta-llama/Llama-4-Scout-17B-16E-Instruct`	Meta’s open LLM series, spanning 7B to 400B parameters (Llama 2, 3, and new Llama 4) with well-recognized performance.	NV ✅ AMD ✅

Bug Fixs

P/D Disaggregation dead lock casuse by request cancel/failed before remote running
Raw Request stream stop_words cause fake hang

Question of omission

In 3fs Case need more MEM or set FRONTEND_SERVER_COUNT=1 to reduce frontend_server mem usage in P/D when Use Frontend Disaggregation.
too many dynamic lora need more reserver_runtime_mem_mb
AMD not support MoE models
MoE model without shared_experter cannot use enable-layer-micro-batch
P/D Disaggregation with EPLB and MTP step > 1 may cause Prefill Hang
Embedding of VL Model is not ok cause by position id is wrong

Performance

Compatibility

Package

Docker Image

CardType	image	tag
CUDA-SM9x	ali-han

v0.1.132024-04-30 · 2024-04-30 06:38 UTC
查看更新说明
feat
- support gte-Qwen1.5-7B-instruct
- support Qwen1.5-MoE
fix
- fix V100 performance
- fix MULTI_TASK_PROMPT and MULTI_TASK_PROMPT_STR env
- fix starcode-7b load failed
- fix llava renderer sep
- fix split_k_factor
v0.1.122024-04-21 · 2024-04-21 11:08 UTC
查看更新说明
feature:
- 支持新模型llama3/code-qwen2/cohere bug fix:
- bloom weight加载错误
- temperature不生效
v0.1.112024-04-12 · 2024-04-12 09:50 UTC
查看更新说明
fix
- int4 tp issue
v0.1.102024-04-07 · 2024-04-07 15:05 UTC
查看更新说明
feat
- sp support TP
- suport tie_word_embeddings option in hf config.json
- update transformers version to 4.39.3
refactor
- add log for weight load: lora apply success / miss weight
fix
- lora support one q/k/v weight is miss
docs
- add Quantization docs

用户点评

登录用户的真实使用记录，可按版本和硬件筛选

共 0 条

还没有人点评。第一个说出真实体验的人最有价值。

README 原文

未翻译

在 GitHub 打开

| Documentation | Contact Us |

News

[2025/09] 🔥 RTP-LLM 0.2.0 release with enhanced performance and new features
[2025/01] 🚀 RTP-LLM now supports Prefill/Decode separation with detailed technical report
[2025/01] 🌟 Qwen series model and bert embedding model now supported on Yitian ARM CPU
[2024/06] 🔄 Major refactor: Scheduling and batching framework rewritten in C++, complete GPU memory management, and new Device backend
[2024/06] 🏗️ Multi-hardware support in development: AMD ROCm, Intel CPU and ARM CPU support coming soon

About

RTP-LLM is a Large Language Model (LLM) inference acceleration engine developed by Alibaba's Foundation Model Inference Team. It is widely used within Alibaba Group, supporting LLM service across multiple business units including Taobao, Tmall, Idlefish, Cainiao, Amap, Ele.me, AE, and Lazada.

RTP-LLM is a sub-project of the havenask project.

Key Features

🏢 Production Proven

Trusted and deployed across numerous LLM scenarios:

Taobao Wenwen
Alibaba's international AI platform, Aidge
OpenSearch LLM Smart Q&A Edition
Large Language Model based Long-tail Query Rewriting in Taobao Search

⚡ High Performance

Utilizes high-performance CUDA kernels, including PagedAttention, FlashAttention, FlashDecoding, etc.
Implements WeightOnly INT8 Quantization with automatic quantization at load time
Support WeightOnly INT4 Quantization with GPTQ and AWQ
Adaptive KVCache Quantization
Detailed optimization of dynamic batching overhead at the framework level
Specially optimized for the V100 GPU

🔧 Flexibility and Ease of Use

Seamless integration with the HuggingFace models, supporting multiple weight formats such as SafeTensors, Pytorch, and Megatron
Deploys multiple LoRA services with a single model instance
Handles multimodal inputs (combining images and text)
Enables multi-machine/multi-GPU tensor parallelism
Supports P-tuning models

🚀 Advanced Acceleration Techniques

Loads pruned irregular models
Contextual Prefix Cache for multi-turn dialogues
System Prompt Cache
Speculative Decoding

Getting Started

Benchmark and Performance

Learn more about RTP-LLM's performance in our benchmark reports:

Performance Benchmark Tool

Acknowledgments

Our project is mainly based on FasterTransformer, and on this basis, we have integrated some kernel implementations from TensorRT-LLM. We also draw inspiration from vllm, transformers, llava, and qwen-vl. We thank these projects for their inspiration and help.

Citation

If you find RTP-LLM useful in your research or project, please consider citing:

@Misc{rtp-llm,
  author       = {Alibaba},
  title        = {RTP-LLM: A High-Performance LLM Inference Engine},
  howpublished = {\url{https://github.com/alibaba/rtp-llm}},
  year         = {2025},
}

rtp-llm

怎么上手

项目说明