需先自行安装 CUDA 12.3 与 mamba 环境，README 未给出 pip 直接安装方式；这是研究原型，功能不如 vLLM 完整，不适合直接当生产服务用；复现实验结果需按 osdi-experiments 各子目录的 readme 单独操作。
git clone [email protected]:microsoft/sarathi-serve.git
mamba create -p ./env python=3.11
pip install -e . 硬件要求按 README 说明
README 说明已在 H100 和 A100 GPU 上配合 CUDA 12.3 测试；未提及 CPU 运行或其他 GPU 厂商支持。
需要 CUDA
微软开源的 LLM 推理服务引擎，主打高吞吐与低延迟的权衡
面向大语言模型的低延迟、高吞吐推理服务引擎，基于 PyTorch 和 Transformer 架构，适合部署 Llama 等模型对外提供在线推理服务。
核心能力
- 基于 OSDI'24 论文的 chunked-prefill 调度实现
- fork 自 vLLM，保留最核心的推理特性
- 提供 osdi-experiments 目录复现论文各图结果
- 面向研究迭代，代码库精简便于改动
容易踩的坑
内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README
暂无 release 数据（可能没有发版，或尚未同步）。
用户点评
登录用户的真实使用记录，可按版本和硬件筛选共 0 条
还没有人点评。第一个说出真实体验的人最有价值。
Sarathi-Serve
Sarathi-Serve is a high througput and low-latency LLM serving framework. Please refer to our OSDI'24 paper for more details.
Setup
Setup CUDA
Sarathi-Serve has been tested with CUDA 12.3 on H100 and A100 GPUs.
Clone repository
git clone [email protected]:microsoft/sarathi-serve.git
Create mamba environment
Setup mamba if you don't already have it,
wget https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Linux-x86_64.sh
bash Mambaforge-Linux-x86_64.sh # follow the instructions from there
Create a Python 3.10 environment,
mamba create -p ./env python=3.11
Install Sarathi-Serve
pip install -e .
Reproducing Results
Refer to readmes in individual folders corresponding to each figure in osdi-experiments.
Citation
If you use our work, please consider citing our paper:
@article{agrawal2024taming,
title={Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve},
author={Agrawal, Amey and Kedia, Nitin and Panwar, Ashish and Mohan, Jayashree and Kwatra, Nipun and Gulavani, Bhargav S and Tumanov, Alexey and Ramjee, Ramachandran},
journal={Proceedings of 18th USENIX Symposium on Operating Systems Design and Implementation, 2024, Santa Clara},
year={2024}
}
Acknowledgment
This repository originally started as a fork of the vLLM project. Sarathi-Serve is a research prototype and does not have complete feature parity with open-source vLLM. We have only retained the most critical features and adopted the codebase for faster research iterations.