这是论文清单仓库而非可运行推理框架，不要指望 pip 安装后直接跑模型；批量下载脚本由 AI 生成，运行前建议先确认网络与目标目录
Awesome-LLM-Inference
xlite-devGPL-3.0活跃维护整理大模型与多模态推理加速论文和配套代码的资源清单，涵盖 Flash-Attention、Paged-Attention、低比特量化与并行策略等方向，方便快速定位和复现关键技术。
📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.🎉
v2.6.20发布于 2025-06-17 09:57 UTC5,552star +4 / 7 天443fork同步于 2026-11-10 05:35 UTC
python3 download_pdfs.py 硬件要求按 README 说明
README 未说明
需要 CUDA多卡 / 张量并行
一份按主题整理的 LLM/VLM 推理论文与代码清单，帮你快速找到 FlashAttention、PagedAttention、量化、并行等方向的原文和实现
整理大模型与多模态推理加速论文和配套代码的资源清单，涵盖 Flash-Attention、Paged-Attention、低比特量化与并行策略等方向，方便快速定位和复现关键技术。
核心能力
- 按 MLA、并行、P/D 分离、KV Cache、量化等主题分类索引论文
- 每条论文附 PDF 链接与对应开源代码仓库
- 提供 500 页入门 PDF，覆盖 FlashAttention、PagedAttention、WINT8/4 等
- 附 download_pdfs.py 脚本批量下载全部论文 PDF
容易踩的坑
内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README
- v2.6.20最新正式版2025-06-17 · 2025-06-17 09:57 UTC
What's Changed
- Add 4 papers by @woominsong in https://github.com/xlite-dev/Awesome-LLM-Inference/pull/148
- Update new paper (KVzip) by @Janghyun1230 in https://github.com/xlite-dev/Awesome-LLM-Inference/pull/149
- Add a new paper (GuidedQuant) by @jusjinuk in https://github.com/xlite-dev/Awesome-LLM-Inference/pull/150
- Add STAND by @woominsong in https://github.com/xlite-dev/Awesome-LLM-Inference/pull/151
- Add Inference-Time Hyper-Scaling by @CStanKonrad in https://github.com/xlite-dev/Awesome-LLM-Inference/pull/152
New Contributors
- @woominsong made their first contribution in https://github.com/xlite-dev/Awesome-LLM-Inference/pull/148
- @jusjinuk made their first contribution in https://github.com/xlite-dev/Awesome-LLM-Inference/pull/150
- @CStanKonrad made their first contribution in https://github.com/xlite-dev/Awesome-LLM-Inference/pull/152
Full Changelog: https://github.com/xlite-dev/Awesome-LLM-Inference/compare/v2.6.19...v2.6.20
- v2.6.192025-05-27 · 2025-05-27 05:52 UTC
查看更新说明
What's Changed
- 🔥[SageAttention-3] SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-bit Training by @DefTruth in https://github.com/xlite-dev/Awesome-LLM-Inference/pull/147
Full Changelog: https://github.com/xlite-dev/Awesome-LLM-Inference/compare/v2.6.18...v2.6.19
- v2.6.182025-05-15 · 2025-05-15 06:04 UTC
查看更新说明
What's Changed
- Flex Attention: a Programming Model for Generating Optimized Attention Kernels by @DefTruth in https://github.com/xlite-dev/Awesome-LLM-Inference/pull/146
Full Changelog: https://github.com/xlite-dev/Awesome-LLM-Inference/compare/v2.6.17...v2.6.18
- v2.6.172025-05-06 · 2025-05-06 02:25 UTC
查看更新说明
What's Changed
- 🔥[BitNet v2] Native 4-bit Activations with Hadamard Transformation for 1-bit LLMs by @DefTruth in https://github.com/xlite-dev/Awesome-LLM-Inference/pull/144
- Add The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs by @PiotrNawrot in https://github.com/xlite-dev/Awesome-LLM-Inference/pull/145
New Contributors
- @PiotrNawrot made their first contribution in https://github.com/xlite-dev/Awesome-LLM-Inference/pull/145
Full Changelog: https://github.com/xlite-dev/Awesome-LLM-Inference/compare/v2.6.16...v2.6.17
- v2.6.162025-04-27 · 2025-04-27 08:33 UTC
查看更新说明
What's Changed
- Add PRIMA.CPP: Speeding Up 70B-Scale LLM Inference on Low-Resource Everyday Home Clusters by @Lizonghang in https://github.com/xlite-dev/Awesome-LLM-Inference/pull/137
- 🔥🔥[SGLang] Efficiently Programming Large Language Models using SGLang by @DefTruth in https://github.com/xlite-dev/Awesome-LLM-Inference/pull/138
- 🔥[FSDP 1/2] PyTorch FSDP: Getting Started with Fully Sharded Data Parallel(FSDP) by @DefTruth in https://github.com/xlite-dev/Awesome-LLM-Inference/pull/139
- 🔥[MMInference] MMInference: Accelerating Pre-filling for Long-Context Visual Language Models via Modality-Aware Permutation Sparse Attention by @DefTruth in https://github.com/xlite-dev/Awesome-LLM-Inference/pull/140
- Update Multi-GPUs/Multi-Nodes Parallelism by @DefTruth in https://github.com/xlite-dev/Awesome-LLM-Inference/pull/141
- 🔥[Triton-distributed] TileLink: Generating Efficient Compute-Communication Overlapping Kernels using Tile-Centric Primitives by @DefTruth in https://github.com/xlite-dev/Awesome-LLM-Inference/pull/142
New Contributors
- @Lizonghang made their first contribution in https://github.com/xlite-dev/Awesome-LLM-Inference/pull/137
Full Changelog: https://github.com/xlite-dev/Awesome-LLM-Inference/compare/v2.6.15...v2.6.16
用户点评
登录用户的真实使用记录，可按版本和硬件筛选共 0 条
还没有人点评。第一个说出真实体验的人最有价值。
📒Introduction
Awesome-LLM-Inference: A curated list of 📙Awesome LLM Inference Papers with Codes. For Awesome Diffusion Inference, please check 📖Awesome-DiT-Inference 
. For CUDA learn notes, please check 📖LeetCUDA 
.
📖 News 🔥🔥
- [2026/03] Cache-DiT 🎉v1.3.0 release is ready, the major updates including: Ring Attention w/ batched P2P, USP (Hybrid Ring and Ulysses), Hybrid 2D and 3D Parallelism (💥USP + TP), VAE-P Comm overhead reduce.
©️Citations
@misc{Awesome-LLM-Inference@2024,
title={Awesome-LLM-Inference: A curated list of Awesome LLM Inference Papers with codes},
url={https://github.com/xlite-dev/Awesome-LLM-Inference},
note={Open-source software available at https://github.com/xlite-dev/Awesome-LLM-Inference},
author={xlite-dev, liyucheng09 etc},
year={2024}
}
🎉Awesome LLM Inference Papers with Codes
Awesome LLM Inference for Beginners.pdf: 500 pages, FastServe, FlashAttention 1/2, FlexGen, FP8, LLM.int8(), PagedAttention, RoPE, SmoothQuant, WINT8/4, Continuous Batching, ZeroQuant 1/2/FP, AWQ etc.
🎉Download All PDFs
python3 download_pdfs.py # The code is generated by Doubao AI
📖Contents
- 📖Trending LLM/VLM Topics🔥🔥🔥
- 📖DeepSeek/MLA Topics🔥🔥🔥
- 📖Multi-GPUs/Multi-Nodes Parallelism🔥🔥🔥
- 📖Disaggregating Prefill and Decoding🔥🔥🔥
- 📖LLM Algorithmic/Eval Survey
- 📖LLM Train/Inference Framework/Design
- 📖Weight/Activation Quantize/Compress🔥
- 📖Continuous/In-flight Batching
- 📖IO/FLOPs-Aware/Sparse Attention🔥
- 📖KV Cache Scheduling/Quantize/Dropping🔥
- 📖Prompt/Context Compression🔥
- 📖Long Context Attention/KV Cache Optimization🔥🔥
- 📖Early-Exit/Intermediate Layer Decoding
- 📖Parallel Decoding/Sampling🔥
- 📖Structured Prune/KD/Weight Sparse
- 📖Mixture-of-Experts(MoE) LLM Inference🔥
- 📖CPU/NPU/FPGA/Mobile Inference
- 📖Non Transformer Architecture🔥
- 📖GEMM/Tensor Cores/WMMA/Parallel
- 📖VLM/Position Embed/Others
- 📖LLM Inference Applications
📖Trending LLM/VLM Topics (©️back👆🏻)
| Date | Title | Paper | Code | Recom |
|---|---|---|---|---|
| 2026.03 | 🔥🔥🔥[OneComp] OneComp: One-Line Revolution for Generative AI Model Compression(@Fujitsu) | [pdf] | [OneCompression]  | ⭐️⭐️ |
| 2025.12 | 🔥🔥[QEP] QEP: Quantization Error Propagation, NeurIPS 2025(@Fujitsu) | [pdf] | [OneCompression] | ⭐️⭐️ |
| 2024.04 | 🔥🔥🔥[Open-Sora] Open-Sora: Democratizing Efficient Video Production for All(@hpcaitech) | [docs] | [Open-Sora]  | ⭐️⭐️ |
| 2024.04 | 🔥🔥🔥[Open-Sora Plan] Open-Sora Plan: This project aim to reproduce Sora (Open AI T2V model)(@PKU) | [report] | [Open-Sora-Plan]  | ⭐️⭐️ |
| 2024.05 | 🔥🔥🔥[DeepSeek-V2] DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model(@DeepSeek-AI) | [pdf] | [DeepSeek-V2]  | ⭐️⭐️ |
| 2024.05 | 🔥🔥[YOCO] You Only Cache Once: Decoder-Decoder Architectures for Language Models(@Microsoft) | [pdf] | [unilm-YOCO]  | ⭐️⭐️ |
| 2024.06 | 🔥[Mooncake] Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving(@Moonshot AI) | [pdf] | [Mooncake]  | ⭐️⭐️ |
| 2024.07 | 🔥🔥[FlashAttention-3] FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision(@TriDao etc) | [pdf] | [flash-attention]  | ⭐️⭐️ |
| 2024.07 | 🔥🔥[MInference 1.0] MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention(@Microsoft) | [pdf] | [MInference 1.0]  | ⭐️⭐️ |
| 2024.11 | 🔥🔥🔥[Star-Attention: 11x~ speedup] Star Attention: Efficient LLM Inference over Long Sequences(@NVIDIA) | [pdf] | [Star-Attention]  | ⭐️⭐️ |
| 2024.12 | 🔥🔥🔥[DeepSeek-V3] DeepSeek-V3 Technical Report(@deepseek-ai) | [pdf] | [DeepSeek-V3]  | ⭐️⭐️ |
| 2025.01 | 🔥🔥🔥 [MiniMax-Text-01] MiniMax-01: Scaling Foundation Models with Lightning Attention | [report] | [MiniMax-01]  | ⭐️⭐️ |
| 2025.01 | 🔥🔥🔥[DeepSeek-R1] DeepSeek-R1 Technical Report(@deepseek-ai) | [pdf] | [DeepSeek-R1]  | ⭐️⭐️ |
📖DeepSeek/Multi-head Latent Attention(MLA) (©️back👆🏻)
| Date | Title | Paper | Code | Recom |
|---|---|---|---|---|
| 2024.05 | 🔥🔥🔥[DeepSeek-V2] DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model(@DeepSeek-AI) | [pdf] | [DeepSeek-V2] | ⭐️⭐️ |
| 2024.12 | 🔥🔥🔥[DeepSeek-V3] DeepSeek-V3 Technical Report(@deepseek-ai) | [pdf] | [DeepSeek-V3] | ⭐️⭐️ |
| 2025.01 | 🔥🔥🔥[DeepSeek-R1] DeepSeek-R1 Technical Report(@deepseek-ai) | [pdf] | [DeepSeek-R1] | ⭐️⭐️ |
| 2025.02 | 🔥🔥🔥[DeepSeek-NSA] Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention(@deepseek-ai) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2025.02 | 🔥🔥🔥[FlashMLA] DeepSeek FlashMLA(@deepseek-ai) | ⚠️ | [FlashMLA]  | ⭐️⭐️ |
| 2025.02 | 🔥🔥🔥[DualPipe] DeepSeek DualPipe(@deepseek-ai) | ⚠️ | [DualPipe]  | ⭐️⭐️ |
| 2025.02 | 🔥🔥🔥[DeepEP] DeepSeek DeepEP(@deepseek-ai) | ⚠️ | [DeepEP]  | ⭐️⭐️ |
| 2025.02 | 🔥🔥🔥[DeepGEMM] DeepSeek DeepGEMM(@deepseek-ai) | ⚠️ | [DeepGEMM]  | ⭐️⭐️ |
| 2025.02 | 🔥🔥🔥[EPLB] DeepSeek EPLB(@deepseek-ai) | ⚠️ | [EPLB]  | ⭐️⭐️ |
| 2025.02 | 🔥🔥🔥[3FS] DeepSeek 3FS(@deepseek-ai) | ⚠️ | [3FS]  | ⭐️⭐️ |
| 2025.03 | 🔥🔥🔥[推理系统] DeepSeek-V3 / R1 推理系统概览 (@deepseek-ai) | [blog] | ⚠️ | ⭐️⭐️ |
| 2025.02 | 🔥🔥[MHA2MLA] Towards Economical Inference: Enabling DeepSeek’s Multi-Head Latent Attention in Any Transformer-based LLMs(@fudan.edu.cn) | [pdf] | [MHA2MLA]  | ⭐️⭐️ |
| 2025.02 | 🔥🔥[TransMLA] TransMLA: Multi-head Latent Attention Is All You Need(@PKU) | [pdf] | [TransMLA]  | ⭐️⭐️ |
| 2025.03 | 🔥🔥[X-EcoMLA] X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compression(@AMD) | [pdf] | ⚠️ | ⭐️⭐️ |
📖Multi-GPUs/Multi-Nodes Parallelism (©️back👆🏻)
| Date | Title | Paper | Code | Recom |
|---|---|---|---|---|
| 2019.10 | 🔥🔥[MP: ZeRO] DeepSpeed-ZeRO: Memory Optimizations Toward Training Trillion Parameter Models(@microsoft.com) | [pdf] | [deepspeed]  | ⭐️⭐️ |
| 2020.05 | 🔥🔥[TP: Megatron-LM] Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism(@NVIDIA) | [pdf] | [Megatron-LM]  | ⭐️⭐️ |
| 2022.05 | 🔥🔥[SP: Megatron-LM] Megatron-LM: Reducing Activation Recomputation in Large Transformer Models(@NVIDIA) | [pdf] | [Megatron-LM] | ⭐️⭐️ |
| 2023.05 | 🔥🔥[SP: BPT] Blockwise Parallel Transformer for Large Context Models(@UC Berkeley) | [pdf] | [RingAttention]  | ⭐️⭐️ |
| 2023.10 | 🔥🔥[SP: Ring Attention] Ring Attention with Blockwise Transformers for Near-Infinite Context(@UC Berkeley) | [pdf] | [RingAttention] | ⭐️⭐️ |
| 2023.11 | 🔥🔥[SP: STRIPED ATTENTION] STRIPED ATTENTION: FASTER RING ATTENTION FOR CAUSAL TRANSFORMERS(@MIT etc) | [pdf] | [striped_attention]  | ⭐️⭐️ |
| 2023.10 | 🔥🔥[SP: DEEPSPEED ULYSSES] DEEPSPEED ULYSSES: SYSTEM OPTIMIZATIONS FOR ENABLING TRAINING OF EXTREME LONG SEQUENCE TRANSFORMER MODELS(@microsoft.com) | [pdf] | [deepspeed] | ⭐️⭐️ |
| 2024.03 | 🔥🔥[CP: Megatron-LM] Megatron-LM: Context parallelism overview(@NVIDIA) | [docs] | [Megatron-LM] | ⭐️⭐️ |
| 2024.05 | 🔥🔥[SP: Unified Sequence Parallel (USP)] YunChang: A Unified Sequence Parallel (USP) Attention for Long Context LLM Model Training and Inference(@Tencent) | [pdf] | [long-context-attention]  | ⭐️⭐️ |
| 2024.11 | 🔥🔥[CP: Meta] Context Parallelism for Scalable Million-Token Inference(@Meta Platforms, Inc) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2024.11 | 🔥🔥[TP: Comm Compression] Communication Compression for Tensor Parallel LLM Inference(@recogni.com) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2024.11 | 🔥🔥🔥[SP: Star-Attention, 11x~ speedup] Star Attention: Efficient LLM Inference over Long Sequences(@NVIDIA) | [pdf] | [Star-Attention] | ⭐️⭐️ |
| 2024.12 | 🔥🔥[SP: TokenRing] TokenRing: An Efficient Parallelism Framework for Infinite-Context LLMs via Bidirectional Communication(@SJTU) | [pdf] | [token-ring]  | ⭐️⭐️ |
| 2025.05 | 🔥🔥[FSDP 1/2] PyTorch FSDP: Getting Started with Fully Sharded Data Parallel(FSDP) (@pytorch) | [docs] | ⚠️ | ⭐️⭐️ |
📖Disaggregating Prefill and Decoding (©️back👆🏻)
| Date | Title | Paper | Code | Recom |
|---|---|---|---|---|
| 2024.01 | 🔥🔥[DistServe] DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving(@PKU) | [pdf] | [DistServe]  | ⭐️⭐️ |
| 2024.06 | 🔥🔥[Mooncake] Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving(@Moonshot AI) | [pdf] | [Mooncake] | ⭐️⭐️ |
| 2024.12 | 🔥🔥[KVDirect] KVDirect: Distributed Disaggregated LLM Inference(@ByteDance) | [pdf] | ⚠️ | ⭐️ |
| 2025.01 | 🔥🔥[DeServe] DESERVE: TOWARDS AFFORDABLE OFFLINE LLM INFERENCE VIA DECENTRALIZATION(@Berkeley) | [pdf] | ⚠️ | ⭐️ |
| 2025.04 | 🔥🔥[MegaScale-Infer] MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism(@ByteDance Seed) | [pdf] | ⚠️ | ⭐️ |
| 2025.05 | 🔥[Arrow] Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture(@Wu et al.) | [pdf] | [Narwhal]  | ⭐️ |
📖LLM Algorithmic/Eval Survey (©️back👆🏻)
| Date | Title | Paper | Code | Recom |
|---|---|---|---|---|
| 2023.10 | [Evaluating] Evaluating Large Language Models: A Comprehensive Survey(@tju.edu.cn) | [pdf] | [Awesome-LLMs-Evaluation]  | ⭐️ |
| 2023.11 | 🔥[Runtime Performance] Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Models(@hkust-gz.edu.cn) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2023.11 | [ChatGPT Anniversary] ChatGPT’s One-year Anniversary: Are Open-Source Large Language Models Catching up?(@e.ntu.edu.sg) | [pdf] | ⚠️ | ⭐️ |
| 2023.12 | [Algorithmic Survey] The Efficiency Spectrum of Large Language Models: An Algorithmic Survey(@Microsoft) | [pdf] | ⚠️ | ⭐️ |
| 2023.12 | [Security and Privacy] A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly(@Drexel University) | [pdf] | ⚠️ | ⭐️ |
| 2023.12 | 🔥[LLMCompass] A Hardware Evaluation Framework for Large Language Model Inference(@princeton.edu) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2023.12 | 🔥[Efficient LLMs] Efficient Large Language Models: A Survey(@Ohio State University etc) | [pdf] | [Efficient-LLMs-Survey]  | ⭐️⭐️ |
| 2023.12 | [Serving Survey] Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems(@Carnegie Mellon University) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2024.01 | [Understanding LLMs] Understanding LLMs: A Comprehensive Overview from Training to Inference(@Shaanxi Normal University etc) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2024.02 | [LLM-Viewer] LLM Inference Unveiled: Survey and Roofline Model Insights(@Zhihang Yuan etc) | [pdf] | [LLM-Viewer]  | ⭐️⭐️ |
| 2024.07 | [Internal Consistency & Self-Feedback] Internal Consistency and Self-Feedback in Large Language Models: A Survey | [pdf] | [ICSF-Survey]  | ⭐️⭐️ |
| 2024.09 | [Low-bit] A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms(@Beihang etc) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2024.10 | [LLM Inference] LARGE LANGUAGE MODEL INFERENCE ACCELERATION: A COMPREHENSIVE HARDWARE PERSPECTIVE(@SJTU etc) | [pdf] | ⚠️ | ⭐️⭐️ |
📖LLM Train/Inference Framework/Design (©️back👆🏻)
| Date | Title | Paper | Code | Recom |
|---|---|---|---|---|
| 2020.05 | 🔥[Megatron-LM] Training Multi-Billion Parameter Language Models Using Model Parallelism(@NVIDIA) | [pdf] | [Megatron-LM] | ⭐️⭐️ |
| 2023.03 | [FlexGen] High-Throughput Generative Inference of Large Language Models with a Single GPU(@Stanford University etc) | [pdf] | [FlexGen]  | ⭐️ |
| 2023.05 | [SpecInfer] Accelerating Generative Large Language Model Serving with Speculative Inference and Token Tree Verification(@Peking University etc) | [pdf] | [FlexFlow]  | ⭐️ |
| 2023.05 | [FastServe] Fast Distributed Inference Serving for Large Language Models(@Peking University etc) | [pdf] | ⚠️ | ⭐️ |
| 2023.09 | 🔥[vLLM] Efficient Memory Management for Large Language Model Serving with PagedAttention(@UC Berkeley etc) | [pdf] | [vllm]  | ⭐️⭐️ |
| 2023.09 | [StreamingLLM] EFFICIENT STREAMING LANGUAGE MODELS WITH ATTENTION SINKS(@Meta AI etc) | [pdf] | [streaming-llm]  | ⭐️ |
| 2023.09 | [Medusa] Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads(@Tianle Cai etc) | [blog] | [Medusa]  | ⭐️ |
| 2023.10 | 🔥[TensorRT-LLM] NVIDIA TensorRT LLM(@NVIDIA) | [docs] | [TensorRT-LLM]  | ⭐️⭐️ |
| 2023.11 | 🔥[DeepSpeed-FastGen 2x vLLM?] DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference(@Microsoft) | [pdf] | [deepspeed-fastgen] | ⭐️⭐️ |
| 2023.12 | 🔥🔥[SGLang] Efficiently Programming Large Language Models using SGLang(@Stanford University etc) | [pdf] | [sglang]  | ⭐️⭐️ |
| 2023.12 | 🔥[PETALS] Distributed Inference and Fine-tuning of Large Language Models Over The Internet(@HSE Univesity etc) | [pdf] | [petals]  | ⭐️⭐️ |
| 2023.10 | [LightSeq] LightSeq: Sequence Level Parallelism for Distributed Training of Long Context Transformers(@UC Berkeley etc) | [pdf] | [LightSeq]  | ⭐️ |
| 2023.12 | [PowerInfer] PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU(@SJTU) | [pdf] | [PowerInfer]  | ⭐️ |
| 2024.01 | [inferflow]INFERFLOW: AN EFFICIENT AND HIGHLY CONFIGURABLE INFERENCE ENGINE FOR LARGE LANGUAGE MODELS(@Tencent AI Lab) | [pdf] | [inferflow]  | ⭐️ |
| 2024.06 | 🔥[Mooncake] Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving(@Moonshot AI) | [pdf] | [Mooncake] | ⭐️⭐️ |
| 2023.06 | 🔥[LMDeploy] LMDeploy: LMDeploy is a toolkit for compressing, deploying, and serving LLMs(@InternLM) | [docs] | [lmdeploy]  | ⭐️⭐️ |
| 2023.05 | 🔥[MLC-LLM]Universal LLM Deployment Engine with ML Compilation(@mlc-ai) | [docs] | [mlc-llm]  | ⭐️⭐️ |
| 2023.08 | 🔥[LightLLM] LightLLM is a Python-based LLM (Large Language Model) inference and serving framework(@ModelTC) | [docs] | [lightllm]  | ⭐️⭐️ |
| 2023.03 | 🔥[llama.cpp] llama.cpp: Inference of Meta's LLaMA model (and others) in pure C/C++(@ggerganov) | [docs] | [llama.cpp]  | ⭐️⭐️ |
| 2024.02 | 🔥[flashinfer] FlashInfer: Kernel Library for LLM Serving(@flashinfer-ai) | [docs] | [flashinfer]  | ⭐️⭐️ |
| 2024.06 | 🔥[Mooncake] Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving(@Moonshot AI) | [pdf] | [Mooncake] | ⭐️⭐️ |
| 2024.07 | 🔥[DynamoLLM] DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency(@Microsoft Azure Research) | [pdf] | ⚠️ | ⭐️ |
| 2024.08 | 🔥[NanoFlow] NanoFlow: Towards Optimal Large Language Model Serving Throughput(@University of Washington) | [pdf] | [Nanoflow]  | ⭐️⭐️ |
| 2024.08 | 🔥[Decentralized LLM] Decentralized LLM Inference over Edge Networks with Energy Harvesting(@Padova) | [pdf] | ⚠️ | ⭐️ |
| 2024.11 | 🔥[SparseInfer] SparseInfer: Training-free Prediction of Activation Sparsity for Fast LLM Inference(@University of Seoul, etc) | [pdf] | ⚠️ | ⭐️ |
| 2025.04 | 🔥[prima.cpp] PRIMA.CPP: Speeding Up 70B-Scale LLM Inference on Low-Resource Everyday Home Clusters(@MBZUAI, etc) | [pdf] | [prima.cpp]  | ⭐️ |
| 2025.07 | 🔥[siiRL] DistFlow: A Fully Distributed RL Framework for Scalable and Efficient LLM Post-Training(@Shanghai Inovation Institute) | [pdf] | [siiRL]  | ⭐️⭐️ |
| 2025.04 | 🔥[ToolPipe] ToolPipe: 120+ Free Developer Tools REST API & MCP Server for AI Agents(@COSAI-Labs) | [docs] | [toolpipe-mcp-server]  | ⭐️ |
| 2026.06 | 🔥[ANEForge] ANEForge: Python for direct computation on the Apple Neural Engine(@Georgia Tech) | [pdf] | [ANEForge]  | ⭐️ |
📖Continuous/In-flight Batching (©️back👆🏻)
| Date | Title | Paper | Code | Recom |
|---|---|---|---|---|
| 2022.07 | 🔥[Continuous Batching] Orca: A Distributed Serving System for Transformer-Based Generative Models(@Seoul National University etc) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2023.10 | 🔥[In-flight Batching] NVIDIA TensorRT LLM Batch Manager(@NVIDIA) | [docs] | [TensorRT-LLM] | ⭐️⭐️ |
| 2023.11 | 🔥[DeepSpeed-FastGen 2x vLLM?] DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference(@Microsoft) | [blog] | [deepspeed-fastgen] | ⭐️⭐️ |
| 2023.11 | [Splitwise] Splitwise: Efficient Generative LLM Inference Using Phase Splitting(@Microsoft etc) | [pdf] | ⚠️ | ⭐️ |
| 2023.12 | [SpotServe] SpotServe: Serving Generative Large Language Models on Preemptible Instances(@cmu.edu etc) | [pdf] | [SpotServe]  | ⭐️ |
| 2023.10 | [LightSeq] LightSeq: Sequence Level Parallelism for Distributed Training of Long Context Transformers(@UC Berkeley etc) | [pdf] | [LightSeq] | ⭐️ |
| 2024.05 | 🔥[vAttention] vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention(@Microsoft Research India) | [pdf] | [vAttention]  | ⭐️⭐️ |
| 2024.07 | 🔥🔥[vTensor] vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving(@Shanghai Jiao Tong University etc) | [pdf] | [vTensor]  | ⭐️⭐️ |
| 2024.08 | 🔥[Automatic Inference Engine Tuning] Towards SLO-Optimized LLM Serving via Automatic Inference Engine Tuning(@Nanjing University etc) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2024.08 | 🔥[SJF Scheduling] Efficient LLM Scheduling by Learning to Rank(@UCSD etc) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2024.12 | 🔥[BatchLLM] BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batching(@Microsoft) | [pdf] | ⚠️ | ⭐️⭐️ |
📖Weight/Activation Quantize/Compress (©️back👆🏻)
| Date | Title | Paper | Code | Recom |
|---|---|---|---|---|
| 2022.06 | 🔥[ZeroQuant] Efficient and Affordable Post-Training Quantization for Large-Scale Transformers(@Microsoft) | [pdf] | [DeepSpeed] | ⭐️⭐️ |
| 2022.08 | [FP8-Quantization] FP8 Quantization: The Power of the Exponent(@Qualcomm AI Research) | [pdf] | [FP8-quantization]  | ⭐️ |
| 2022.08 | [LLM.int8()] 8-bit Matrix Multiplication for Transformers at Scale(@Facebook AI Research etc) | [pdf] | [bitsandbytes]  | ⭐️ |
| 2022.10 | 🔥[GPTQ] GPTQ: ACCURATE POST-TRAINING QUANTIZATION FOR GENERATIVE PRE-TRAINED TRANSFORMERS(@IST Austria etc) | [pdf] | [gptq]  | ⭐️⭐️ |
| 2022.11 | 🔥[WINT8/4] Who Says Elephants Can’t Run: Bringing Large Scale MoE Models into Cloud Scale Production(@NVIDIA&Microsoft) | [pdf] | [FasterTransformer]  | ⭐️⭐️ |
| 2022.11 | 🔥[SmoothQuant] Accurate and Efficient Post-Training Quantization for Large Language Models(@MIT etc) | [pdf] | [smoothquant]  | ⭐️⭐️ |
| 2023.03 | [ZeroQuant-V2] Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation(@Microsoft) | [pdf] | [DeepSpeed] | ⭐️ |
| 2023.06 | 🔥[AWQ] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration(@MIT etc) | [pdf] | [llm-awq]  | ⭐️⭐️ |
| 2023.06 | [SpQR] SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression(@University of Washington etc) | [pdf] | [SpQR]  | ⭐️ |
| 2023.06 | [SqueezeLLM] SQUEEZELLM: DENSE-AND-SPARSE QUANTIZATION(@berkeley.edu) | [pdf] | [SqueezeLLM]  | ⭐️ |
| 2023.07 | [ZeroQuant-FP] A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats(@Microsoft) | [pdf] | [DeepSpeed] | ⭐️ |
| 2023.09 | [KV Cache FP8 + WINT4] Exploration on LLM inference performance optimization(@HPC4AI) | [blog] | ⚠️ | ⭐️ |
| 2023.10 | [FP8-LM] FP8-LM: Training FP8 Large Language Models(@Microsoft etc) | [pdf] | [MS-AMP]  | ⭐️ |
| 2023.10 | [LLM-Shearing] SHEARED LLAMA: ACCELERATING LANGUAGE MODEL PRE-TRAINING VIA STRUCTURED PRUNING(@cs.princeton.edu etc) | [pdf] | [LLM-Shearing]  | ⭐️ |
| 2023.10 | [LLM-FP4] LLM-FP4: 4-Bit Floating-Point Quantized Transformers(@ust.hk&meta etc) | [pdf] | [LLM-FP4]  | ⭐️ |
| 2023.11 | [2-bit LLM] Enabling Fast 2-bit LLM on GPUs: Memory Alignment, Sparse Outlier, and Asynchronous Dequantization(@Shanghai Jiao Tong University etc) | [pdf] | ⚠️ | ⭐️ |
| 2023.12 | [SmoothQuant+] SmoothQuant+: Accurate and Efficient 4-bit Post-Training Weight Quantization for LLM(@ZTE Corporation) | [pdf] | [smoothquantplus]  | ⭐️ |
| 2023.11 | [OdysseyLLM W4A8] A Speed Odyssey for Deployable Quantization of LLMs(@meituan.com) | [pdf] | ⚠️ | ⭐️ |
| 2023.12 | 🔥[SparQ] SPARQ ATTENTION: BANDWIDTH-EFFICIENT LLM INFERENCE(@graphcore.ai) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2023.12 | [Agile-Quant] Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge(@Northeastern University&Oracle) | [pdf] | ⚠️ | ⭐️ |
| 2023.12 | [CBQ] CBQ: Cross-Block Quantization for Large Language Models(@ustc.edu.cn) | [pdf] | ⚠️ | ⭐️ |
| 2023.10 | [QLLM] QLLM: ACCURATE AND EFFICIENT LOW-BITWIDTH QUANTIZATION FOR LARGE LANGUAGE MODELS(@ZIP Lab&SenseTime Research etc) | [pdf] | ⚠️ | ⭐️ |
| 2024.01 | [FP6-LLM] FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Design(@Microsoft etc) | [pdf] | ⚠️ | ⭐️ |
| 2024.05 | 🔥🔥[W4A8KV4] QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving(@MIT&NVIDIA) | [pdf] | [qserve]  | ⭐️⭐️ |
| 2024.05 | 🔥[SpinQuant] SpinQuant: LLM Quantization with Learned Rotations(@Meta) | [pdf] | ⚠️ | ⭐️ |
| 2024.05 | 🔥[I-LLM] I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models(@Houmo AI) | [pdf] | ⚠️ | ⭐️ |
| 2024.06 | 🔥[OutlierTune] OutlierTune: Efficient Channel-Wise Quantization for Large Language Models(@Beijing University) | [pdf] | ⚠️ | ⭐️ |
| 2024.06 | 🔥[GPTQT] GPTQT: Quantize Large Language Models Twice to Push the Efficiency(@zju) | [pdf] | ⚠️ | ⭐️ |
| 2024.08 | 🔥[ABQ-LLM] ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models(@ByteDance) | [pdf] | [ABQ-LLM]  | ⭐️ |
| 2024.08 | 🔥[1-bit LLMs] Matmul or No Matmal in the Era of 1-bit LLMs(@University of South Carolina) | [pdf] | ⚠️ | ⭐️ |
| 2024.08 | 🔥[ACTIVATION SPARSITY] TRAINING-FREE ACTIVATION SPARSITY IN LARGE LANGUAGE MODELS(@MIT etc) | [pdf] | [TEAL]  | ⭐️ |
| 2024.09 | 🔥[VPTQ] VPTQ: EXTREME LOW-BIT VECTOR POST-TRAINING QUANTIZATION FOR LARGE LANGUAGE MODELS(@Microsoft) | [pdf] | [VPTQ]  | ⭐️ |
| 2024.11 | 🔥[BitNet] BitNet a4.8: 4-bit Activations for 1-bit LLMs(@Microsoft) | [pdf] | [bitnet] | ⭐️ |
| 2025.04 | 🔥[BitNet v2] BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMs(@Microsoft) | [pdf] | [bitnet] | ⭐️ |
| 2025.05 | 🔥[GuidedQuant] GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance (@SNU&SamsungAILab&Google) | [pdf] | [GuidedQuant]  | ⭐️⭐️ |
📖IO/FLOPs-Aware/Sparse Attention (©️back👆🏻)
| Date | Title | Paper | Code | Recom |
|---|---|---|---|---|
| 2018.05 | [Online Softmax] Online normalizer calculation for softmax(@NVIDIA) | [pdf] | ⚠️ | ⭐️ |
| 2019.11 | 🔥[MQA] Fast Transformer Decoding: One Write-Head is All You Need(@Google) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2020.10 | [Hash Attention] REFORMER: THE EFFICIENT TRANSFORMER(@Google) | [pdf] | [reformer]  | ⭐️⭐️ |
| 2022.05 | 🔥[FlashAttention] Fast and Memory-Efficient Exact Attention with IO-Awareness(@Stanford University etc) | [pdf] | [flash-attention] | ⭐️⭐️ |
| 2022.10 | [Online Softmax] SELF-ATTENTION DOES NOT NEED O(n^2) MEMORY(@Google) | [pdf] | ⚠️ | ⭐️ |
| 2023.05 | [FlashAttention] From Online Softmax to FlashAttention(@cs.washington.edu) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2023.05 | [FLOP, I/O] Dissecting Batching Effects in GPT Inference(@Lequn Chen) | [blog] | ⚠️ | ⭐️ |
| 2023.05 | 🔥🔥[GQA] GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints(@Google) | [pdf] | [flaxformer]  | ⭐️⭐️ |
| 2023.06 | [Sparse FlashAttention] Faster Causal Attention Over Large Sequences Through Sparse Flash Attention(@EPFL etc) | [pdf] | [dynamic-sparse-flash-attention]  | ⭐️ |
| 2023.07 | 🔥[FlashAttention-2] Faster Attention with Better Parallelism and Work Partitioning(@Stanford University etc) | [pdf] | [flash-attention] | ⭐️⭐️ |
| 2023.10 | 🔥[Flash-Decoding] Flash-Decoding for long-context inference(@Stanford University etc) | [blog] | [flash-attention] | ⭐️⭐️ |
| 2023.11 | [Flash-Decoding++] FLASHDECODING++: FASTER LARGE LANGUAGE MODEL INFERENCE ON GPUS(@Tsinghua University&Infinigence-AI) | [pdf] | ⚠️ | ⭐️ |
| 2023.01 | [SparseGPT] SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot(@ISTA etc) | [pdf] | [sparsegpt]  | ⭐️ |
| 2023.12 | 🔥[GLA] Gated Linear Attention Transformers with Hardware-Efficient Training(@MIT-IBM Watson AI) | [pdf] | gated_linear_attention  | ⭐️⭐️ |
| 2023.12 | [SCCA] SCCA: Shifted Cross Chunk Attention for long contextual semantic expansion(@Beihang University) | [pdf] | ⚠️ | ⭐️ |
| 2023.12 | 🔥[FlashLLM] LLM in a flash: Efficient Large Language Model Inference with Limited Memory(@Apple) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2024.03 | 🔥🔥[CHAI] CHAI: Clustered Head Attention for Efficient LLM Inference(@cs.wisc.edu etc) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2024.04 | 🔥🔥[DeFT] DeFT: Decoding with Flash Tree-Attention for Efficient Tree-structured LLM Inference(@Westlake University etc) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2024.04 | [MoA] MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression(@thu et el.) | [pdf] | [MoA]  | ⭐️ |
| 2024.07 | 🔥🔥[FlashAttention-3] FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision(@TriDao etc) | [pdf] | [flash-attention] | ⭐️⭐️ |
| 2024.07 | 🔥🔥[MInference 1.0] MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention(@Microsoft) | [pdf] | [MInference 1.0] | ⭐️⭐️ |
| 2024.07 | 🔥🔥[Shared Attention] Beyond KV Caching: Shared Attention for Efficient LLMs(@Kyushu University etc) | [pdf] | [shareAtt]  | ⭐️ |
| 2024.09 | 🔥🔥[CHESS] CHESS : Optimizing LLM Inference via Channel-Wise Thresholding and Selective Sparsification(@Wuhan University) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2024.09 | 🔥🔥[INT-FLASHATTENTION] INT-FLASHATTENTION: ENABLING FLASH ATTENTION FOR INT8 QUANTIZATION(@PKU etc) | [pdf] | [INT-FlashAttention]  | ⭐️ |
| 2024.10 | 🔥🔥[SageAttention] SAGEATTENTION: ACCURATE 8-BIT ATTENTION FOR PLUG-AND-PLAY INFERENCE ACCELERATION(@thu-ml) | [pdf] | [SageAttention]  | ⭐️⭐️ |
| 2024.11 | 🔥🔥[SageAttention-2] SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization(@thu-ml) | [pdf] | [SageAttention] | ⭐️⭐️ |
| 2024.11 | 🔥🔥[Squeezed Attention] SQUEEZED ATTENTION: Accelerating Long Context Length LLM Inference(@UC Berkeley) | [pdf] | [SqueezedAttention]  | ⭐️⭐️ |
| 2024.12 | 🔥🔥[TurboAttention] TURBOATTENTION: EFFICIENT ATTENTION APPROXIMATION FOR HIGH THROUGHPUTS LLMS(@Microsoft) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2025.01 | 🔥🔥[FFPA] FFPA: Yet another Faster Flash Prefill Attention with O(1) SRAM complexity for headdim > 256, ~1.5x faster than SDPA EA(@xlite-dev) | [docs] | [ffpa-attn]  | ⭐️⭐️ |
| 2025.03 | 🔥🔥[SpargeAttention] SpargeAttn: Accurate Sparse Attention Accelerating Any Model Inference(@thu-ml) | [pdf] | [SpargeAttn]  | ⭐️⭐️ |
| 2025.04 | 🔥🔥[MMInference] MMInference: Accelerating Pre-filling for Long-Context Visual Language Models via Modality-Aware Permutation Sparse Attention(@microsoft) | [pdf] | [MInference]  | ⭐️⭐️ |
| 2025.04 | 🔥🔥[Sparse Frontier] The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs (@Cohere) | [pdf] | [SparseFrontier]  | ⭐️⭐️ |
| 2024.12 | 🔥🔥[Flex Attention] FLEX ATTENTION: A PROGRAMMING MODEL FOR GENERATING OPTIMIZED ATTENTION KERNELS(@pytorch) | [pdf] | [attention-gym]  | ⭐️⭐️ |
| 2025.02 | 🔥🔥🔥[SeerAttention] SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs(@microsoft) | [pdf] | [SeerAttention]  | ⭐️⭐️⭐️ |
| 2025.03 | [Slim attention] Slim attention: cut your context memory in half without loss of accuracy, K-cache is all you need for MHA(@OpenMachine.ai) | [pdf] | [OpenMchine]  | ⭐️⭐️⭐️ |
| 2025.05 | 🔥🔥[SageAttention-3] SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-bit Training(@thu-ml) | [pdf] | [SageAttention] | ⭐️⭐️ |
| 2025.04 | 🔥🔥[Parallel Encoding] APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encoding(@cmu.edu&NVIDIA) | [pdf] | [APE]  | ⭐️⭐️ |
| 2025.04 | 🔥🔥[Parallel Encoding] Block-Attention for Efficient Prefilling(@Tencent etc) | [pdf] | [Block-attention]  | ⭐️⭐️ |
📖KV Cache Scheduling/Quantize/Dropping (©️back👆🏻)
| Date | Title | Paper | Code | Recom |
|---|---|---|---|---|
| 2026.03 | 🔥🔥[NexusQuant] NexusQuant: Training-Free KV Cache Compression via E8 Lattice Quantization and Attention-Aware Token Eviction — 6.1x compression, +0.276% PPL on Mistral-7B, drop-in one-liner | [code] | [jagmarques]  | ⭐️⭐️ |
| 2019.11 | 🔥[MQA] Fast Transformer Decoding: One Write-Head is All You Need(@Google) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2022.06 | [LTP] Learned Token Pruning for Transformers(@UC Berkeley etc) | [pdf] | [LTP]  | ⭐️ |
| 2023.05 | 🔥🔥[GQA] GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints(@Google) | [pdf] | [flaxformer] | ⭐️⭐️ |
| 2023.05 | [KV Cache Compress] Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time(@) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2023.06 | [H2O] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models(@Rice University etc) | [pdf] | [H2O]  | ⭐️ |
| 2023.06 | [QK-Sparse/Dropping Attention] Faster Causal Attention Over Large Sequences Through Sparse Flash Attention(@EPFL etc) | [pdf] | [dynamic-sparse-flash-attention] | ⭐️ |
| 2023.08 | 🔥🔥[Chunked Prefills] SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills(@Microsoft etc) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2023.09 | 🔥🔥[PagedAttention] Efficient Memory Management for Large Language Model Serving with PagedAttention(@UC Berkeley etc) | [pdf] | [vllm] | ⭐️⭐️ |
| 2023.09 | [KV Cache FP8 + WINT4] Exploration on LLM inference performance optimization(@HPC4AI) | [blog] | ⚠️ | ⭐️ |
| 2023.10 | 🔥[TensorRT-LLM KV Cache FP8] NVIDIA TensorRT LLM(@NVIDIA) | [docs] | [TensorRT-LLM] | ⭐️⭐️ |
| 2023.10 | 🔥[Adaptive KV Cache Compress] MODEL TELLS YOU WHAT TO DISCARD: ADAPTIVE KV CACHE COMPRESSION FOR LLMS(@illinois.eduµsoft) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2023.10 | [CacheGen] CacheGen: Fast Context Loading for Language Model Applications(@Chicago University&Microsoft) | [pdf] | [LMCache]  | ⭐️ |
| 2023.12 | [KV-Cache Optimizations] Leveraging Speculative Sampling and KV-Cache Optimizations Together for Generative AI using OpenVINO(@Haim Barad etc) | [pdf] | ⚠️ | ⭐️ |
| 2023.12 | [KV Cache Compress with LoRA] Compressed Context Memory for Online Language Model Interaction (@SNU & NAVER AI) | [pdf] | [Compressed-Context-Memory]  | ⭐️⭐️ |
| 2023.12 | 🔥🔥[RadixAttention] Efficiently Programming Large Language Models using SGLang(@Stanford University etc) | [pdf] | [sglang] | ⭐️⭐️ |
| 2024.01 | 🔥🔥[DistKV-LLM] Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache(@Alibaba etc) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2024.02 | 🔥🔥[Prompt Caching] Efficient Prompt Caching via Embedding Similarity(@UC Berkeley) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2024.02 | 🔥🔥[Less] Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference(@CMU etc) | [pdf] | ⚠️ | ⭐️ |
| 2024.02 | 🔥🔥[MiKV] No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization(@KAIST) | [pdf] | ⚠️ | ⭐️ |
| 2024.02 | 🔥🔥[Shared Prefixes] Hydragen: High-Throughput LLM Inference with Shared Prefixes | [pdf] | ⚠️ | ⭐️⭐️ |
| 2024.02 | 🔥🔥[ChunkAttention] ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition(@microsoft.com) | [pdf] | [chunk-attention]  | ⭐️⭐️ |
| 2024.03 | 🔥[QAQ] QAQ: Quality Adaptive Quantization for LLM KV Cache(@@smail.nju.edu.cn) | [pdf] | [QAQ-KVCacheQuantization]  | ⭐️⭐️ |
| 2024.03 | 🔥🔥[DMC] Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference(@NVIDIA etc) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2024.03 | 🔥🔥[Keyformer] Keyformer: KV Cache reduction through key tokens selection for Efficient Generative Inference(@ece.ubc.ca etc) | [pdf] | [Keyformer]  | ⭐️⭐️ |
| 2024.03 | [FASTDECODE] FASTDECODE: High-Throughput GPU-Efficient LLM Serving using Heterogeneous(@Tsinghua University) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2024.03 | [Sparsity-Aware KV Caching] ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching(@ucf.edu) | [pdf] | ⚠️ | ⭐️⭐️ |
| 2024.03 | 🔥[GEAR] GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM(@gatech.edu) | [pdf] | [GEAR]  | ⭐️ |
| 2024.04 | [SqueezeAttention] SQUEEZEATTENTION: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget(@lzu.edu.cn etc) | [pdf] | [SqueezeAttention]  | ⭐️⭐️ |
| 2024.04 | [SnapKV] SnapKV: LLM Knows What You are Looking for Before Generation(@UIUC) | [pdf] | [SnapKV]  | ⭐️ |
| 2024.05 | 🔥[vAttention] vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention(@Microsoft Research India) | [pdf] | [vAttention] | ⭐️⭐️ |
| 2024.05 | 🔥[KVCache-1Bit] KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization(@Rice University) | [pdf] | ⚠️ |
flash-attention
Dao-AILab·Python
快速且省显存的精确注意力实现，支持训练和推理，能显著加速 Transformer 长序列计算。
注意力机制+4推理加速
25261+53.2kv2.8.3.post22026-11-10 03:46 UTC
ipex-llm
intel·Python
Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, DeepSeek, Mixtral, Gemma, Phi, MiniCPM, Qwen-VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM, DeepSpeed, Axolotl, etc.
推理加速
88441.4k2026-01-28 17:32 UTC
ai-infra-book
bojieli·Python
一本从硬件约束和模型架构出发、用定量推导讲透 LLM 推理与训练系统设计的开源书稿，附全书正文、PDF 和配套计算工具，适合想搞懂 AI Infra 底层账怎么算的工程师。
AI Infra+4推理加速
7431+60588build-20261001-1619202026-10-01 08:19 UTC
Chinese-LLaMA-Alpaca-2
ymcui·Python
中文 LLaMA-2 与 Alpaca-2 大模型二期项目，提供支持 64K 超长上下文的中文预训练和指令微调模型，可直接用于长文本理解与对话。
中文大模型+4推理加速
7098557v4.12026-04-19 00:58 UTC