标签：vLLM 生态

把 VAD、ASR、标点、说话人分离串成一条流水线的中文语音识别工具包，顶替自己拼 Whisper + pyannote 的胶水层

Meta 官方 Llama 示例库，用 notebook 顶替你自己从零搭的推理、微调、RAG 样板代码

把 Android 模拟器封装成可编排的容器，顶替 Jenkins/CI 里那台常年开着的真机或本地 AVD

给编码 Agent 挂载 98 个 AI 研究技能包，把 Claude Code 变成能跑训练、评测、写论文的研究工作台

不是软件，是逐行拆 vLLM 和 Go 运行时的中文源码笔记库，用来在动手前把实现读明白

把 vLLM 这类引擎的 KV 缓存抽成独立进程，跨实例复用 prefill 结果，专治长上下文首 token 慢

用 Ray 把训练和 vLLM 推理拆成两个集群，在单机多卡上跑 PPO/GRPO 这类在线 RL 后训练

把 vLLM、llama.cpp、TTS 和 embedding 服务收进一个 OpenAI 兼容 API，换模型改一行

把 vLLM/SGLang/TensorRT-LLM 的单机推理引擎编排成多节点集群，顶替你自己写的路由层和 K8s 部署脚本

把预填充和解码拆到不同机器上跑，用 RDMA 在实例间搬运 KV 缓存，替代单机 PD 合一的推理部署

在 agent harness 和多个模型后端之间做策略路由的 Go 网关，顶替手写的模型选择逻辑

本地多模型推理的统一入口，按请求里的 model 字段热切换上游服务

在 K8s 上把模型推理做成标准 CRD，顶替手写 Deployment 加自建推理网关那一层

把散落的 GPU 机器收成一个可调度的推理集群，顶替手写 docker run 加 vLLM 启动脚本那套

用 YAML 把检索、重排、生成串成 MCP 服务，替代手写胶水代码搭 RAG 流程

在 Apple Silicon 上从零手写 Qwen3 推理栈的课程，用来替换「读 vLLM 源码但跑不起来」的学习路径

一份按主题索引的 LLM/VLM 推理论文与实现清单，用来替代你自己在 arXiv 和 GitHub 之间来回翻找

把票据表格转成经 JSON Schema 校验的结构化结果，顶替自建的 OCR 加 LLM 抽取流水线

Cascading runtime for AI agents. Optimize cost, latency, quality, and policy decisions inside the agent loop.

High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle

AI 基础知识 - GPU 架构、CUDA 编程、大模型基础及AI Agent 相关知识。

RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.

Community maintained hardware plugin for vLLM on Huawei Ascend

Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

Open Source AI Accelerator Research Platform Standard / 开源推理研究平台

Community maintained hardware plugin for vLLM on Apple Silicon

Control panel for VLLM, Sglang, llama.cpp, exllamav3

Open decision models from 0.8B to 27B: a calibrated probability for every option in one pass, and a short thought only when unsure. Engine for vLLM, llama.cpp and MLX.

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率，量化感知训练的精度

High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.

Turn any LLM into a Jev-style decision model: typed decisions, real probabilities, no training. (continue updating, welcome any issue and PR request)

Open-source AI computer builds, from 2-GPU desktops to 8-GPU servers: CAD, BOM, BIOS and build guides. Get the case kit with PCIe risers, or ask us for a free fit check.

Multimodal RL training framework for diffusion & omni models

AI Inference Operator for Kubernetes. The easiest way to serve ML models in production. Supports VLMs, LLMs, embeddings, and speech-to-text.

LLM model quantization (compression) toolkit with HW acceleration support for Nvidia (CUDA), AMD (ROCm), Intel GPU (XPU), Huawei Ascent (NPU), and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.

Official Inference Code of EAGLE-4

Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

A 10-week, 30-minutes-a-day roadmap for LLM inference serving and optimization. vLLM, SGLang, quantization, speculative decoding, benchmarking.

UniRL is a Framework for Unified Multimodal Model Reinforcement Learning

Spark-x2.6 open model series. Pushing the Limits of Agentic Capabilities in On-Device Models

LLM notes, including model inference, transformer model structure, and llm framework code analysis notes.

An local, offline (after initial setup), portable OCR software that can process images and PDF files, using DeepSeek-OCR-2 AI (running directly on your machine).

Halo is an open-source framework built by White Circle for training large language and multimodal models

End-to-end documentation to set up your own local & fully private LLM server on Debian. Equipped with chat, web search, RAG, model management, MCP servers, image generation, and TTS.

AI-powered data analysis workbench built for transparency, reproducibility and user control. Inspect the code and evidence behind results, verify calculations through replay, and choose from a wide range of models, including local open-weight models.

Make Discord your LLM frontend

Frontier-class open models on a free Kaggle TPU v5e-8: GLM-5.3-Flash 320B MoE (~165 tok/s, our own JAX engine) and Qwen3.8-27B bf16 (~130 tok/s), 262k context, prefix caching. Works with Claude Code, Codex, opencode and pi.

[EMNLP 2024 & AAAI 2026] A powerful toolkit for compressing large models including LLMs, VLMs, and video generative models.

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2