Awesome-KV-Cache-Optimization

[ACL Anthology] [arXiv] [Website] [LinkedIn] [机器之心]

This repository accompanies our survey paper (ACL 2026 Findings):

Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
*Jiantong Jiang1, Peiyu Yang1*, Rui Zhang2, Feng Liu1
1The University of Melbourne, 2Huazhong University of Science and Technology

---

This repository maintains a continuously updated collection of system-aware, serving-time, KV-centric optimization methods that improve system metrics without retraining or architecture modification, a scope we call sKis. We organize the literature using the system behavior-oriented taxonomy introduced in our survey:
🔷 Temporal — when is KV cache accessed or computed?
🔷 Spatial — where is KV cache placed and migrated?
🔷 Structural — how is KV cache represented and managed?

📚 The taxonomy serves as a stable organizing framework, while the paper collection below is continuously updated as new work appears.
🧠 Grounded in this taxonomy, we analyze cross-behavior co-design affinity and behavior–objective effects, revealing overlooked regions and concrete open challenges.

News

• 2026-10: 🔥 63 new papers were added.
• 2026-09: 🧰 Added a new Tools, Simulators & Benchmarking Resources section. Many thanks to @Naloam for the contribution!
• 2026-08: 🌐 Launched our new website with an interactive paper explorer. Huge thanks to @ryan-utopia for making this possible!
• 2026-06: 🔥 51 new papers were added.
• 2026-04: 🎉 Our survey was accepted to ACL 2026.
• 2026-01: 📰 Our survey was featured by 机器之心.
• 2025-11: ⭐ 100-star milestone! Thanks to the amazing support from the community!
• 2025-10: 📝 The survey was available on TechRxiv.

Contributing

The repository is still under active development and will be updated regularly.

🙋 If you would like to include your paper in this survey and repository, please feel free to submit a pull request. You can generate the markdown row for each paper by filling in the first part of generate.py and running python generate.py. Alternatively, you can open an issue with the paper's title and a brief summary highlighting its key techniques. You can also contact us via email.

🙋🏻‍♀️ Please let us know if you find any metadata, venue, classification, or link errors, or have suggestions for the repository. We greatly appreciate your feedback!

🌟 If you find this resource helpful for your work, please consider giving us a star and citing our research.

---

Quick Index

• Temporal — Execution & Scheduling
  • KV-Centric Scheduling (KVS)
  • Pipelining & Overlapping (OVLP)
  • Hardware-Aware Execution (HAE)
• Spatial — Placement & Migration
  • Memory Hierarchy KV Orchestration (MHO)
  • Compute Device KV Orchestration (CDO)
• Structural — Representation & Retention
  • KV Cache Compression (KVCC)
  • KV Cache Retention Management (KVRM)
• Insights from the ACL 2026 Survey
  • Cross-behavior Co-design Affinity
  • Behavior-objective Effects
• Tools, Simulators & Benchmarking Resources

---

Temporal — Execution & Scheduling

These methods act on when KV data is executed, computed, or scheduled to improve latency and throughput. We divide these methods into three categories: KV-centric scheduling, pipelining & overlapping, and hardware-aware execution.

KV-Centric Scheduling

KV-centric scheduling methods explicitly integrate KV characteristics into runtime scheduling decisions.

Paper	Type	Code
TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications [Link] Zhuohang Bian, Feiyang Wu, Zhuoran Li, Teng Ma, Youwei Zhuo	Agent-aware KV scheduling under tool-call stalls; Also belongs to memory hierarchy KV orchestration
Does Execution Require Target KV Fidelity? A Mixed-Fidelity KV Runtime for LLM Serving [Link] Jiantong Jiang, Yue Yang, Peiyu Yang, Feng Liu	Mixed-fidelity KV execution and runtime scheduling
Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches [Link] Vivek Kalyanarangan	Token-level attention compute scheduling; per-query, per-channel read depth over a bit-plane K store; Also belongs to KV cache compression, quantization (structural)	fathom
CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Serving [Link] Huang Cheng	Prefix-affinity request routing with load-aware cache locality
Simple Is Better: Multiplication May Be All You Need for LLM Request Scheduling [Link] Dingyan Zhang, Jinbo Han, Kaixi Zhang, Xingda Wei, Sijie Shen, Chenguang Fang, Wenyuan Yu, Jingren Zhou, Rong Chen	KV reuse- and load-aware request scheduling via multiplicative routing score
Strata: Hierarchical Context Caching for Long Context Language Model Serving [Link] Zhiqiang Xie, Ziyi Xu, Mark Zhao, Yuwei An, Vikram Sharma Mailthody, Scott Mahlke, Michael Garland, Christos Kozyrakis	Cache-loading-aware request scheduling; Also belongs to Memory Hierarchy KV Orchestration (spatial)
Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding [Link] WenHung Lee, Jian-Jia Chen, Xiaolin Lin, Pei-Shuo Wang, Chi-Chih Chang, Chun-Che Yang, Wei-Chen Wang, Hanrui Wang, Ning-Chi Huang, Grace Li Zhang, Kai-Chiang Wu	Draft- and target-guided token-level KV selection for sparse speculative verification
RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference [Link] Wenhao Li, Jinhao Dong, Hailin Zhang, Wenhang Shi, Wei Lu, Xiaoyong Du	Binary-quantized proxy scoring for adaptive Top-p KV retrieval in sparse attention	RaBitQCache
Zipage: Maintain High Request Concurrency for LLM Reasoning through Compressed PagedAttention [Link] Mengqi Liao, Lu Wang, Chaoyun Zhang, Bo Qiao, Si Qin, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Huaiyu Wan	KV block- and compression-state-aware hybrid request scheduling; Also belongs to KV cache retention management (structural)	Zipage
AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System [Link] Fengyao Bai, Hongbin Zhang, Zhitao Chen, Jiangsu Du, Zhiguang Chen, Yutong Lu	KV-length-aware request batching and batch-level scheduling
CoDec: Prefix-Shared Decoding Kernel for LLMs [Link] Zhibin Wang, Rui Ning, Chao Fang, Zhonghui Zhang, Xi Lin, Shaobo Ma, Mo Zhou, Xue Li, Zhongfeng Wang, Chengying Huan, Rong Gu, Kun Yang, Guihai Chen, Sheng Zhong, Chen Tian	Shared KV access scheduling across requests	codec
HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems [Link] Yuhang Li, Rong Gu, Chengying Huan, Zhibin Wang, Renjie Yao, Chen Tian, Guihai Chen	Hotness-aware KV cache scheduling for prefix sharing; Also belongs to KV cache retention management
PRISM: Fast Online LLM Serving via Scheduling-Memory Co-design [Link] Xingyu Qu, Tianhao Lin, Yiqi Li, Zhiyu Chen, Sheng Wang	KV reuse-aware request scheduling; Also belongs to KV cache retention management (structural)
Bidaw: Enhancing Key-Value Caching for Interactive LLM Serving via Bidirectional Computation–Storage Awareness [Link] Shipeng Hu, Guangyan Zhang, Yuqi Zhou, Yaya Wei, Ziyan Zhong, Jike Chen	KV-loading-aware request scheduling; Also belongs to Memory hierarchy KV orchestration (spatial)
Fast Inference for Augmented Large Language Models [Link] Rana Shahout, Cong Liang, Shiji Xin, Qianru Lao, Yong Cui, Minlan Yu, Michael Mitzenmacher	KV usage-aware request-level scheduling
TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection [Link] Wei Wu, Zhuoshi Pan, Kun Fu, Chao Wang, Liyi Chen, Yunchu Bai, Tianfu Wang, Zheng Wang, Hui Xiong	Token-level attention compute scheduling	TokenSelect
TinyServe: Query-Aware Cache Selection for Efficient LLM Serving [Link] Dong Liu, Yanxuan Yu	Query-aware page-level KV selection for sparse attention computation
RefreshKV: Updating Small KV Cache During Long-form Generation [Link] Fangyuan Xu, Tanya Goyal, Eunsol Choi	Token-level attention compute scheduling	RefreshKV
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression [Link] Payman Behnam, Yaosheng Fu, Ritchie Zhao, Po-An Tsai, Zhiding Yu, Alexey Tumanov	Token-level attention compute scheduling	RocketKV
FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving [Link] Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, Luis Ceze	Kernel-level workload scheduling across CUDA thread blocks; Also belongs to allocation & reuse (structural)	FlashInfer 🌟
Preble: Efficient Distributed Prompt Scheduling for LLM Serving [Link] Vikranth Srivatsa, Zijian He, Reyna Abhyankar, Dongming Li, Yiying Zhang	KV reuse-aware request-level scheduling	Preble
Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot [Link] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu	KV reuse-aware request-level scheduling; Also belongs to HW-aware execution	Mooncake 🌟
Loki: Low-rank Keys for Efficient Sparse Attention [Link] Prajwal Singhania, Siddharth Singh, Shwai He, Soheil Feizi, Abhinav Bhatele	Token-level attention compute scheduling	Loki
SGLang: Efficient Execution of Structured Language Model Programs [Link] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, Ying Sheng	KV reuse-aware request-level scheduling; Also belongs to allocation & reuse (structural)	SGLang 🌟
LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism [Link] Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, Xin Jin	KV usage-aware request-level scheduling	LoongServe
LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management [Link] Yi Xiong, Hao Wu, Changxu Shao, Ziqing Wang, Rui Zhang, Yuhong Guo, Junping Zhao, Ke Zhang, Zhenxuan Pan	KV usage-aware request-level scheduling; Also belongs to memory hierarchy KV orchestration (spatial)
InferCept: Efficient Intercept Support for Augmented Large Language Model Inference [Link] Reyna Abhyankar, Zijian He, Vikranth Srivatsa, Hao Zhang, Yiying Zhang	KV-state-aware request scheduling over preserve, swap, and recompute decisions; Also belongs to memory hierarchy KV orchestration (spatial)	InferCept
SparQ Attention: Bandwidth-Efficient LLM Inference [Link] Luka Ribar, Ivan Chelombiev, Luke Hudlass-Galley, Charlie Blake, Carlo Luschi, Douglas Orr	Token-level attention compute scheduling	SparQ Attention
QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference [Link] Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, Song Han	Token-level attention compute scheduling	Quest
MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving [Link] Jiangfei Duan, Runyu Lu, Haojie Duanmu, Xiuhong Li, Xingcheng Zhang, Dahua Lin, Ion Stoica, Hao Zhang	KV usage-aware request-level scheduling; Also belongs to HW-aware execution	MuxServe
Inference without interference: Disaggregate LLM inference for mixed downstream workloads [Link] Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, Yizhou Shan	KV usage-aware request-level scheduling; Also belongs to HW-aware execution

Pipelining & Overlapping

Pipelining and overlapping methods hide latency by concurrently executing KV-related compute, communication, and I/O. Though often embedded in the broader systems, we highlight the following methods where pipelining and overlapping forms the core technical contribution.

Paper	Type	Code
DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch [Link] Zuning Liang, Zhiyi Yao, Qi Chen, Yuedong Xu, Hao Dai, Zhiqiang Ding, Tongkai Yang, Jinlong Hou, Yuan Cheng	KV prefetch (CPU→GPU) ↔ GPU decode compute; Also belongs to memory hierarchy KV orchestration (spatial)
ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference [Link] Qiuyang Zhang, Kai Zhou, Ding Tang, Kai Lu, Cheng Li, Zhenyu Yang, Peng Xu, Jiguang Wan	Layer-ahead CPU attention precompute ↔ GPU decode; Also belongs to HW-aware execution
ECHO: Efficient KV Cache Offloading with Lossless Prefetching for Serving Native Sparse Attention LLMs [Link] Guangda Liu, Wenhao Chen, Chengwei Li, Zhenyu Ning, Jing Lin, Yiwu Yao, Quan Chen, Shixuan Sun, Jieru Zhao, Minyi Guo	KV prefetch ↔ sparse-attention indexer computation; Also belongs to memory hierarchy KV orchestration (spatial)	ECHO
RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference [Link] Wenhao Li, Jinhao Dong, Hailin Zhang, Wenhang Shi, Wei Lu, Xiaoyong Du	Asynchronous proxy-index construction and attention execution with lazy updates; Also belongs to KV-centric scheduling (temporal)	RaBitQCache
ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reuse [Link] Yu Zhu, Aditya Dhakal, Yunming Xiao, Dejan Milojicic, Gustavo Alonso	Object-storage KV retrieval ↔ GPU layerwise consumption; Also belongs to memory hierarchy KV orchestration (spatial)
DroidSpeak: KV Cache Sharing Across Fine-tuned Model Variants [Link] Yuhan Liu, Yuyang Huang, Jiayi Yao, Shaoting Feng, Zhuohan Gu, Kuntai Du, Hanchen Li, Yihua Cheng, Junchen Jiang, Shan Lu, Madan Musuvathi, Esha Choukse	Critical-layer KV recomputation ↔ remote reused-KV loading; Also belongs to compute device KV orchestration (spatial)
High Throughput and Low Latency LLM Serving via Adaptive KV Caching [Link] Wenyan Chen, Chengzhi Lu, Huanle Xu, Kejiang Ye, ChengZhong Xu	Uncached-token KV recomputation ↔ decoding/communication via fine-grained adaptive caching; Also belongs to KV cache retention management (structural)
CXL-SpecKV: A Disaggregated FPGA Speculative KV-Cache for Datacenter LLM Serving [Link] Dong Liu, Yanxuan Yu	GPU compute ↔ GPU KV prefetch (CXL memory→L2); Also belongs to memory hierarchy KV orchestration (spatial)	CXL-SpecKV
SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning [Link] Lingkun Long, Rubing Yang, Yushi Huang, Desheng Hui, Ao Zhou, Jianlei Yang	KV prefetch/offload (CPU↔GPU) ↔ FFN & QKV generation; Also belongs to memory hierarchy KV orchestration (spatial)
Accelerating LLM Inference Throughput via Asynchronous KV Cache Prefetching [Link] Yanhao Dong, Yubo Miao, Weinan Li, Xiao Zheng, Chao Wang, Jiesheng Wu, Feng Lyu	GPU KV prefetch (HBM→L2) ↔ GPU attention compute; Also belongs to memory hierarchy KV orchestration (spatial)
KVPR: Efficient LLM inference with i/o-aware KV cache partial recomputation [Link] Chaoyi Jiang, Lei Gao, Hossein Entezari Zarch, Murali Annavaram	GPU KV recompute ↔ KV transfer (CPU↔GPU)	KVPR
NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference [Link] Xuanlin Jiang, Yang Zhou, Shiyi Cao, Ion Stoica, Minlan Yu	CPU attention compute ↔ GPU linear ops; Also belongs to HW-aware execution	NEO
CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion [Link] Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, Junchen Jiang	Selective KV recomputation ↔ cached-KV retrieval for non-prefix RAG chunks; Also belongs to KV cache retention management (structural)	CacheBlend
Fast State Restoration in LLM Serving with HCache [Link] Shiwei Gao, Youmin Chen, Jiwu Shu	KV state restoration compute ↔ host-storage I/O via bubble-free scheduling
PRESERVE: Prefetching Model Weights and KV-Cache in Distributed LLM Serving [Link] Ahmet Caner Yüzügüler, Jiawei Zhuang, Lukas Cavigelli	GPU KV prefetch (HBM→L2) ↔ GPU collective communication; Also belongs to memory hierarchy KV orchestration (spatial)
Improving Throughput-Oriented LLM Inference with CPU Computations [Link] Daon Park, Bernhard Egger	CPU MHSA compute ↔ FFN data transfer (CPU→GPU); Also belongs to HW-aware execution	Heterogen
FastDecode: High-Throughput LLM Serving through Disaggregating Attention Computation [Link] Jiaao He, Kezhao Huang, Jidong Zhai	CPU R-part compute ↔ GPU S-part compute; Also belongs to HW-aware execution
Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention [Link] Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, Pengfei Zuo	KV load/store (CPU↔GPU) ↔ GPU compute; Also belongs to memory hierarchy KV orchestration (spatial)

Hardware-aware Execution

Hardware-aware execution methods adapt KV cache-related operations to the underlying heterogeneous hardware.

Disaggregated Inference

Disaggregated inference separates heterogeneous computation in LLM inference and maps them to distinct hardware resources to reduce interference and improve utilization.

Paper	Type	Code
ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference [Link] Qiuyang Zhang, Kai Zhou, Ding Tang, Kai Lu, Cheng Li, Zhenyu Yang, Peng Xu, Jiguang Wan	CPU/GPU collaborative attention; Also belongs to pipelining & overlapping
HillInfer: Efficient Long-Context LLM Inference on the Edge with Hierarchical KV Eviction using SmartSSD [Link] He Sun, Shinan Liu, Li Li, Mingjun Xiao	SmartSSD-assisted KV offloading; Also belongs to memory hierarchy KV orchestration (spatial)
Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot [Link] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu	PD disaggregation; Also belongs to KV-centric scheduling	Mooncake 🌟
DéjàVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving [Link] Foteini Strati, Sara Mcallister, Amar Phanishayee, Jakub Tarnawski, Ana Klimovic	Decoupling prefill and decode to different GPUs; Also belongs to memory hierarchy KV orchestration (spatial)	DéjàVu
MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving [Link] Jiangfei Duan, Runyu Lu, Haojie Duanmu, Xiuhong Li, Xingcheng Zhang, Dahua Lin, Ion Stoica, Hao Zhang	Colocating PD jobs of multiple LLMs within each GPU via SM partitioning; Also belongs to KV-centric scheduling	MuxServe
DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving [Link] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang	Decoupling prefill and decode to different GPUs; Also belongs to compute device KV orchestration (spatial)	DistServe
Splitwise: Efficient generative LLM inference using phase splitting [Link] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, Ricardo Bianchini	Decoupling prefill and decode to different GPUs; Also belongs to compute device KV orchestration (spatial)	Splitwise (integrated into vLLM)
Inference without interference: Disaggregate LLM inference for mixed downstream workloads [Link] Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, Yizhou Shan	Decoupling prefill and decode to different GPUs; Also belongs to KV-centric scheduling
Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache [Link] Bin Lin, Chen Zhang, Tao Peng, Hanyu Zhao, Wencong Xiao, Minmin Sun, Anmin Liu, Zhipeng Zhang, Lanbo Li, Xiafei Qiu, Shen Li, Zhigang Ji, Tao Xie, Yong Li, Wei Lin	Disaggregating at the operator level; Also belongs to compute device KV orchestration (spatial)

Compute Offloading

Compute offloading relocates partial compute to auxiliary devices to reduce GPU bottlenecks, utilizing hardware heterogeneity and workload features.

Paper	Type	Code
Turbo: Efficiently Serving Long-Context Large Language Models with In-Network Aggregation [Link] Ying Wan, Yuchen Xu, Chuwen Zhang, Yingsheng Huang, Yong Feng, Wenquan Xu, Jialin Li, Mingwei Xu, Wenfei Wu, Congcong Miao	In-network query broadcast and attention aggregation offloaded to programmable switches/NICs; Also belongs to compute device KV orchestration (spatial)	Turbo
A Cost-Effective Near-Storage Processing Solution for Offline Inference of Long-Context LLMs [Link] Hongsun Jang, Jaeyong Song, Changmin Shin, Si Ung Noh, Jaewon Jung, Jisung Park, Jinho Lee	Near-storage attention offloading to SmartSSD accelerators; Also belongs to compute device KV orchestration (spatial)	HILOS
NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference [Link] Xuanlin Jiang, Yang Zhou, Shiyi Cao, Ion Stoica, Minlan Yu	CPU offloading (attention and KV caches); Also belongs to pipelining & overlapping	NEO
MagicPIG: LSH Sampling for Efficient LLM Generation [Link] Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia, Beidi Chen	CPU offloading (attention and retrieval)	MagicPIG
PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing System [Link] Yintao He, Haiyu Mao, Christina Giannoula, Mohammad Sadrosadati, Juan Gómez-Luna, Huawei Li, Xiaowei Li, Ying Wang, Onur Mutlu	PIM-based offloading
Improving Throughput-Oriented LLM Inference with CPU Computations [Link] Daon Park, Bernhard Egger	CPU offloading with dynamic GPU-CPU division; Also belongs to pipelining & overlapping	Heterogen
TwinPilots: A New Computing Paradigm for GPU-CPU Parallel LLM Inference [Link] Chengye Yu, Tianyu Wang, Zili Shao, Linjie Zhu, Xu Zhou, Song Jiang	CPU offloading
InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference [Link] Xiurui Pan, Endian Li, Qiao Li, Shengwen Liang, Yizhou Shan, Ke Zhou, Yingwei Luo, Xiaolin Wang, Jie Zhang	CSD Offloading; Also belongs to compute device KV orchestration (spatial)
FastDecode: High-Throughput LLM Serving through Disaggregating Attention Computation [Link] Jiaao He, Kezhao Huang, Jidong Zhai	CPU offloading (attention and KV caches); Also belongs to pipelining & overlapping
AttAcc! Unleashing the power of PIM for batched transformer-based generative model inference [Link] Jaehyun Park, Jaewan Choi, Kwanhee Kyung, Michael Jaemin Kim, Yongsuk Kwon, Nam Sung Kim, Jung Ho Ahn	PIM-based offloading; Also belongs to compute device KV orchestration (spatial)	AttAcc

Hardware-Specialized Execution

[Newly added] These methods specialize KV-related kernels, data layouts, or dedicated execution engines to exploit hardware-native compute primitives and access patterns.

Paper	Type	Code
PAT: Accelerating LLM Decoding via Prefix-Aware Attention with Resource Efficient Multi-Tile Kernel [Link] Jinjun Yi, Zhixin Zhao, Yitao Hu, Ke Yan, Weiwei Sun, Hao Wang, Laiping Zhao, Yuhao Zhang, Wenxin Li, Keqiu Li	Prefix-aware multi-tile attention kernel that co-schedules shared-prefix queries to reduce redundant KV reads	PAT
V-Rex: Real-Time Streaming Video LLM Acceleration via Dynamic KV Cache Retrieval [Link] Donghyuk Kim, Sejeong Yang, Wonjin Shin, Joo-Young Kim	SW-HW co-designed dynamic KV retrieval engine for streaming video LLM inference; Also belongs to memory hierarchy KV orchestration
BitDecoding: Unlocking Tensor Cores for Long-Context LLMs with Low-Bit KV Cache [Link] Dayou Du, Shijie Cao, Jianyi Cheng, Luo Mai, Ting Cao, Mao Yang	CUDA/Tensor-Core cooperative execution for efficient low-bit KV cache decoding	BitDecoding

---

Spatial — Placement & Migration

These works optimize where KV data is stored or transferred to balance memory and I/O pressure. We divide these methods into two categories: memory hierarchy KV orchestration, and compute device KV orchestration.

Memory Hierarchy KV Orchestration

Memory hierarchy KV orchestration methods distribute KV caches across memory hierarchies.

Cross-device Memory Hierarchy

These methods manage KV caches across fast but limited GPU HBM, and larger but slower alternatives like CPU DRAM or SSD.

Paper	Type	Code
TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications [Link] Zhuohang Bian, Feiyang Wu, Zhuoran Li, Teng Ma, Youwei Zhuo	Proactive KV offload/upload for tool-call stalls; Also belongs to KV-centric scheduling (temporal)
ORBITFLOW: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration [Link] Xinyue Ma, Heelim Hong, Taegeon Um, Jongseop Lee, Seoyeong Choy, Woo-Yeon Lee, Myeongjae Jeon	Fine-grained adaptive GPU–CPU KV placement reconfiguration under latency SLOs
DualPath: Accelerating Agentic LLM Inference by Harvesting Disaggregated KV-Cache Storage I/O [Link] Yongtong Wu, Shaoyuan Chen, Rilin Huang, Yixuan Tan, Yinmin Zhong, Mingxing Zhang, Xin Jin, Panpan Huang	Dual-path KV loading from external storage; Also belongs to compute device KV orchestration
DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch [Link] Zuning Liang, Zhiyi Yao, Qi Chen, Yuedong Xu, Hao Dai, Zhiqiang Ding, Tongkai Yang, Jinlong Hou, Yuan Cheng	Predictive host-to-GPU KV prefetching; Also belongs to pipelining & overlapping (temporal)
A CXL Memory Rack for Multi-Turn LLM Serving [Link] Hakbeom Jang, Inho Song, Hoshik Kim, Sam H. Noh, Jongryool Kim	CXL-hybrid-memory KV tiering with request-level prefix prefetching
ECHO: Efficient KV Cache Offloading with Lossless Prefetching for Serving Native Sparse Attention LLMs [Link] Guangda Liu, Wenhao Chen, Chengwei Li, Zhenyu Ning, Jing Lin, Yiwu Yao, Quan Chen, Shixuan Sun, Jieru Zhao, Minyi Guo	Dynamic host–GPU KV offloading with lossless prefetching; Also belongs to pipelining & overlapping (temporal)	ECHO
No Buffer, No Bottleneck: Efficient Zero-Copy KV Cache Offloading for Long-Context LLMs [Link] Shutian Luo, Haiying Shen	Zero-copy CPU-resident KV offloading over NVLink-C2C	DirectKV
Strata: Hierarchical Context Caching for Long Context Language Model Serving [Link] Zhiqiang Xie, Ziyi Xu, Mark Zhao, Yuwei An, Vikram Sharma Mailthody, Scott Mahlke, Michael Garland, Christos Kozyrakis	Hierarchical KV caching across GPU, CPU, and SSD with cache-aware request scheduling; Also belongs to KV-centric scheduling (temporal)
ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMs [Link] Yanlin Qi, Xinhang Chen, Huiqiang Jiang, Qitong Wang, Botao Peng, Themis Palpanas	GPU-native top-k KV retrieval from CPU-offloaded caches	ParisKV
HeteroCache: A Dynamic Retrieval Approach to Heterogeneous KV Cache Compression for Long-Context LLM Inference [Link] Zhiyuan Shi, Qibo Qiu, Feng Xue, Zhonglin Jiang, Li Yu, Jian Jiang, Xiaofei He, Wenxiao Wang	Asynchronous hierarchical KV retrieval with head heterogeneity
DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference [Link] Bodon Jeong, Hongsu Byun, Youngjae Kim, Weikuan Yu, Kyungkeun Lee, Jihoon Yang, Sungyong Park	NVMe-direct KV offloading on memory-constrained edge devices
KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference [Link] Jian Lin, Jiazhi Mi, Zicong Hong, Haodong Wang, Qianli Liu, Haoyue Zhang, Peng Li, Song Guo	CPU/DRAM/SSD multi-tier KV placement and migration
Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management [Link] Xinjun Yang, Qingda Hu, Junru Li, Feifei Li, Yicong Zhu, Yuqi Zhou, Qiuru Lin, Jian Dai, Yang Kong, Jiayu Zhang, Guoqiang Xu, Qiang Liu	System cost-driven decision
FlexiCache: Leveraging Temporal Stability of Attention Heads for Efficient KV Cache Management [Link] Nazmul Takbir, HamidReza Alikhani Koshkak, Nikil Dutt, Sangeetha Abdu Jyothi	Head-stability-aware GPU–CPU KV placement with periodic top-page promotion	FlexiCache
ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reuse [Link] Yu Zhu, Aditya Dhakal, Yunming Xiao, Dejan Milojicic, Gustavo Alonso	Layerwise KV placement and retrieval from object storage; Also belongs to pipelining & overlapping (temporal)
Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving [Link] Shi Qiu, Yifan Hu, Xintao Wang, Wenhao Zhu, Jianqin Yan, Hao Chen, Kaiqiang Xu, Kai Chen, Yiming Zhang	SSD-backed KV placement with GPU-friendly retrieval layout
SYMPHONY: Enabling Compute-Memory Disaggregation in LLM Serving Systems [Link] Saurabh Agarwal, Bodun Hu, Anyong Mao, Aditya Akella, Shivaram Venkataraman	Disaggregated KV cache storage with advisory prefetching and cooperative GPU-memory management
FastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference [Link] Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, Xin Jin	System cost-driven decision; Also belongs to compute device KV orchestration	FastServe
AtlasKV: Augmenting LLMs with Billion-Scale Knowledge Graphs in 20GB VRAM [Link] Haoyu Huang, Hong Ting Tsang, Jiaxin Bai, Xi Peng, Gong Zhang, Yangqiu Song	Importance-aware methods	AtlasKV
FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference [Link] Guangda Liu, Chengwei Li, Zhenyu Ning, Jing Lin, Yiwu Yao, Danning Ke, Minyi Guo, Jieru Zhao	Speculative top-k KV retrieval	FreeKV
TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference [Link] Gradwell Dzikanyanga, Weihao Yang, Hao Huang, Donglei Wu, Shihao Wang, Wen Xia, Sanjeeb K C	Temporal-tiered KV placement with heterogeneous precision/retention
Bidaw: Enhancing Key-Value Caching for Interactive LLM Serving via Bidirectional Computation–Storage Awareness [Link] Shipeng Hu, Guangyan Zhang, Yuqi Zhou, Yaya Wei, Ziyan Zhong, Jike Chen	Host-memory/SSD KV caching; Also belongs to KV-centric scheduling (temporal)
SolidAttention: Low-Latency SSD-based Serving on Memory-Constrained PCs [Link] Xinrui Zheng, Dongliang Wei, Jianxiang Gao, Yixin Song, Zeyu Mi, Haibo Chen	SSD-resident KV management with sparse selective access and speculative prefetching
CXL-SpecKV: A Disaggregated FPGA Speculative KV-Cache for Datacenter LLM Serving [Link] Dong Liu, Yanxuan Yu	System cost-driven decision; Also belongs to pipelining & overlapping (temporal)	CXL-SpecKV
HillInfer: Efficient Long-Context LLM Inference on the Edge with Hierarchical KV Eviction using SmartSSD [Link] He Sun, Shinan Liu, Li Li, Mingjun Xiao	SmartSSD-backed hierarchical KV offloading; Also belongs to HW-aware execution (temporal)
V-Rex: Real-Time Streaming Video LLM Acceleration via Dynamic KV Cache Retrieval [Link] Donghyuk Kim, Sejeong Yang, Wonjin Shin, Joo-Young Kim	Hierarchical KV cache management with selective retrieval for streaming video contexts; Also belongs to HW-aware execution
SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning [Link] Lingkun Long, Rubing Yang, Yushi Huang, Desheng Hui, Ao Zhou, Jianlei Yang	Importance-aware methods; Also belongs to pipelining & overlapping (temporal)
TraCT: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scale [Link] Dongha Yoon, Younghoon Min, Hoshik Kim, Sam H. Noh, Jongryool Kim	System cost-driven decision; Also belongs to compute device KV orchestration
HiFC: High-efficiency Flash-based KV Cache Swapping for Scaling LLM Inference [[Link](https://proceedings.neu