0.1.0 尚未发布，需从仓库源码自行构建 wheel，且引擎需用固定版本（vLLM 0.33.0、SGLang 0.5.23）。vLLM 需设 VLLM_USE_V2_MODEL_RUNNER=0 并加 --enforce-eager；SGLang 需关闭 decode/prefill 的 cuda graph。验证外部命中要重启引擎后重放同一 prompt，同进程重复只会命中原生 GPU 缓存。
OrbitKV
OrbitInfraApache-2.0活跃维护面向 vLLM 和 SGLang 的 KV cache 复用层，用 Rust 实现，把前缀缓存扩展到 DRAM、SSD 和跨主机，降低大模型推理的重复计算和显存压力。
OrbitKV is a KV cache for vLLM and SGLang.
暂无版本279star +5 / 7 天18fork同步于 2026-11-10 12:03 UTC
PYO3_PYTHON=python3.11 ./scripts/build-wheel.sh --release
uv pip install --python /path/to/engine-env/bin/python /absolute/path/to/the-built-wheel.whl
orbitkv-cache-manager --addr 127.0.0.1:50055 --http-addr 127.0.0.1:9091 --pool-size 8gb 硬件要求按 README 说明
README 未给出具体显存或 GPU 型号要求，仅说明需 Linux 与兼容的 CUDA 13 构建/运行环境；当前 GPU、RDMA、GDS 路径均标注为未认证（unqualified），CPU/TCP 与 io_uring 有 CI 证据。
需要 CUDA
给 vLLM 和 SGLang 用的分布式 KV 缓存，把可复用前缀存到 DRAM、SSD 和别的机器上
面向 vLLM 和 SGLang 的 KV cache 复用层，用 Rust 实现，把前缀缓存扩展到 DRAM、SSD 和跨主机，降低大模型推理的重复计算和显存压力。
核心能力
- 独立 Cache Manager 在引擎重启后仍保留 DRAM 中的可复用前缀
- 可选 SSD 承接溢出缓存，但 Manager 重启后不保证持久
- 通过有界清单索引和源授权在多个副本间共享缓存容量
- Rust 管理缓存标识、放置、租约与有界准入
- OrbiTE 统一负责 GPU↔DRAM、SSD 与对端读取的字节搬运
容易踩的坑
内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README
暂无 release 数据（可能没有发版，或尚未同步）。
用户点评
登录用户的真实使用记录，可按版本和硬件筛选共 0 条
还没有人点评。第一个说出真实体验的人最有价值。
Updates
- 2026-11-07: OrbiTE becomes the sole transfer engine; hardware re-qualification remains open.
About
OrbitKV extends the KV cache of vLLM and SGLang beyond GPU memory. It is for serving workloads that reuse documents, system prompts and conversation prefixes. An independent Cache Manager keeps reusable state in DRAM and optional SSD across engine restarts; inference engines keep scheduling and GPU allocation.
Rust owns cache identity, placement, leases and bounded admission. OrbiTE is the only byte-transfer engine for local GPU↔DRAM copies, SSD and peer reads. Current GPU, RDMA, GDS and installed profiles are unqualified; CPU/TCP and io_uring controls are CI evidence. Historical TENT results do not qualify OrbiTE. See support status and evidence.
Key features
- Reuse beyond GPU memory: retain compatible prefixes in DRAM and optional SSD while the Manager stays alive. SSD is not durable across Manager restarts.
- Share cache capacity: attach local engines to a Manager or discover peer replicas through bounded inventory indexes and authoritative source grants.
- Keep ownership explicit: hold sources, destinations and registrations until physical completion; a timeout is never permission to reuse their bytes.
- Describe recovery requirements: compile supported prefix, window and checkpoint layouts into complete recovery boundaries. Engine guards still apply.
Features and limits cover optional codecs and retention policies. Live prefill/decode uses the inference engine's native transport.
Architecture
Implemented ownership boundaries; hardware paths remain unqualified. See architecture and transfer contracts.
Getting started
OrbitKV 0.1.0 is unreleased. On Linux with a compatible CUDA 13 build/runtime environment, build a complete wheel from the repository root and install it into your chosen engine environment:
PYO3_PYTHON=python3.11 ./scripts/build-wheel.sh --release
uv pip install --python /path/to/engine-env/bin/python /absolute/path/to/the-built-wheel.whl
Use separate environments for the pinned engines and the same wheel for client and Manager. The quickstart supplies prerequisites, engine installation and hit checks. With the chosen environment activated, start a Manager:
orbitkv-cache-manager --addr 127.0.0.1:50055 --http-addr 127.0.0.1:9091 --pool-size 8gb
Then choose one engine in another terminal with the same environment activated. These are unqualified hardware recipes, not benchmark results.
vLLM (pinned release and limits):
VLLM_USE_V2_MODEL_RUNNER=0 vllm serve /path/to/immutable-model \
--enforce-eager --enable-prefix-caching \
--kv-transfer-config '{"kv_connector":"OrbitKVConnector","kv_role":"kv_both","kv_connector_module_path":"orbitkv.vllm"}'
SGLang (pinned release and limits):
ORBITKV_SGLANG_ENDPOINT=unix:///tmp/orbitkv-50055.sock \
sglang serve --model-path /path/to/immutable-model --page-size 64 \
--cuda-graph-backend-decode disabled --cuda-graph-backend-prefill disabled \
--enable-unified-cache-external-linker --radix-cache-backend orbitkv
Verify an external hit: save a
multi-block prompt, keep the Manager alive, restart the engine and repeat it.
Check that orbitkv_load_bytes_total increases; a same-process repeat can hit
only native GPU cache. Manager options explain the example budgets.
The documentation covers engine integration, storage, distributed cache, deployment, observability and internals. The support matrix selects vLLM 0.33.0 (V1, one attention group) and SGLang 0.5.23 (Full/SWA external linker); recurrent external recovery remains rejected. See model recipes and limits.
Contributing
Follow the contributor guide and the completion plan. For the data flow, read transport → engine-local restore → distributed cache → adapter contracts.
License
Apache 2.0. Bundled dependencies retain their own licenses and notices.
flash-attention
Dao-AILab·Python
快速且省显存的精确注意力实现，支持训练和推理，能显著加速 Transformer 长序列计算。
注意力机制+4推理加速
25265+93.2kv2.8.3.post22026-11-10 03:46 UTC
ipex-llm
intel·Python
Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, DeepSeek, Mixtral, Gemma, Phi, MiniCPM, Qwen-VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM, DeepSpeed, Axolotl, etc.
推理加速
88441.4k2026-01-28 17:32 UTC
ai-infra-book
bojieli·Python
一本从硬件约束和模型架构出发、用定量推导讲透 LLM 推理与训练系统设计的开源书稿，附全书正文、PDF 和配套计算工具，适合想搞懂 AI Infra 底层账怎么算的工程师。
AI Infra+4推理加速
7443+72592build-20261001-1619202026-10-01 08:19 UTC
Chinese-LLaMA-Alpaca-2
ymcui·Python
中文 LLaMA-2 与 Alpaca-2 大模型二期项目，提供支持 64K 超长上下文的中文预训练和指令微调模型，可直接用于长文本理解与对话。
中文大模型+4推理加速
7098557v4.12026-04-19 00:58 UTC