标签：vLLM 生态

Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.

Welcome to the Llama Cookbook! This is your go to guide for Building with Llama: Getting started with Inference, Fine-Tuning, RAG. We also show you how to solve end to end problems using Llama model family and using them on various provider services

Android in docker solution with noVNC supported, video recording, mcp server and AI-agent

Comprehensive open-source library of AI research and engineering skills for any AI model. Package the skills and your claude code/codex/gemini agent will be an AI research agent with full horsepower. Maintained by Orchestra Research.

✍🏻 Source Code Deep Dives, System Design & Engineering Blogs | Halfrost-Field 冰霜之地：源码解析、系统设计与工程实践笔记

LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

An Easy-to-use, Scalable and High-performance Agentic RL Framework based on Ray (PPO & DAPO & REINFORCE++ & VLM & TIS & vLLM & Ray & Async RL)

Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.

A Datacenter Scale Distributed Inference Serving Framework

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

An open, programmable decision layer for models and compute.

Reliable model swapping for any local AI server. Chosen by thousands of users. Batteries included. If we don't have it, you probably don't need it.

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

A Low-Code MCP Framework for Building Complex and Innovative RAG Pipelines

learn LLM inference system on Apple Silicon for systems engineers: build a tiny vLLM + Qwen

📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.🎉

Structured data extraction, instruction calling and agentic workflows with ML, LLM and Vision LLM

Cascading runtime for AI agents. Optimize cost, latency, quality, and policy decisions inside the agent loop.

High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle

AI 基础知识 - GPU 架构、CUDA 编程、大模型基础及AI Agent 相关知识。

RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.

Community maintained hardware plugin for vLLM on Huawei Ascend

Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

Open Source AI Accelerator Research Platform Standard / 开源推理研究平台

Community maintained hardware plugin for vLLM on Apple Silicon

Control panel for VLLM, Sglang, llama.cpp, exllamav3

Open decision models from 0.8B to 27B: a calibrated probability for every option in one pass, and a short thought only when unsure. Engine for vLLM, llama.cpp and MLX.

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率，量化感知训练的精度

High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.

Turn any LLM into a Jev-style decision model: typed decisions, real probabilities, no training. (continue updating, welcome any issue and PR request)

Open-source AI computer builds, from 2-GPU desktops to 8-GPU servers: CAD, BOM, BIOS and build guides. Get the case kit with PCIe risers, or ask us for a free fit check.

Multimodal RL training framework for diffusion & omni models

AI Inference Operator for Kubernetes. The easiest way to serve ML models in production. Supports VLMs, LLMs, embeddings, and speech-to-text.

LLM model quantization (compression) toolkit with HW acceleration support for Nvidia (CUDA), AMD (ROCm), Intel GPU (XPU), Huawei Ascent (NPU), and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.

Official Inference Code of EAGLE-4

Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

A 10-week, 30-minutes-a-day roadmap for LLM inference serving and optimization. vLLM, SGLang, quantization, speculative decoding, benchmarking.

UniRL is a Framework for Unified Multimodal Model Reinforcement Learning

Spark-x2.6 open model series. Pushing the Limits of Agentic Capabilities in On-Device Models

LLM notes, including model inference, transformer model structure, and llm framework code analysis notes.

An local, offline (after initial setup), portable OCR software that can process images and PDF files, using DeepSeek-OCR-2 AI (running directly on your machine).

Halo is an open-source framework built by White Circle for training large language and multimodal models

End-to-end documentation to set up your own local & fully private LLM server on Debian. Equipped with chat, web search, RAG, model management, MCP servers, image generation, and TTS.

AI-powered data analysis workbench built for transparency, reproducibility and user control. Inspect the code and evidence behind results, verify calculations through replay, and choose from a wide range of models, including local open-weight models.

Make Discord your LLM frontend

Frontier-class open models on a free Kaggle TPU v5e-8: GLM-5.3-Flash 320B MoE (~165 tok/s, our own JAX engine) and Qwen3.8-27B bf16 (~130 tok/s), 262k context, prefix caching. Works with Claude Code, Codex, opencode and pi.

[EMNLP 2024 & AAAI 2026] A powerful toolkit for compressing large models including LLMs, VLMs, and video generative models.

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2