大模型推理服务

Python package for LLM compression

[ICML 2024] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache

大模型/LLM推理和部署理论与实践

📝 Automatically annotate papers using LLMs

Station is our open-source runtime that lets teams deploy agents on their own infrastructure with full control.

🧱 Experimental library for building inference engines

TopicGPT: A Prompt-Based Framework for Topic Modeling [NAACL'24]

Framework to bring LLM applications to production

[Fully open] [Encoder-free MLLM] Vision as LoRA

The missing bridge between your ML models and your AI agents.

LightRFT: Light, Efficient, Omni-modal & Reward-model Driven Reinforcement Fine-Tuning Framework

EVE Series: Encoder-Free Vision-Language Models from BAAI

GitHub Action that reviews pull request diffs with an LLM and posts the comments on the PR

Walk through a diff the way a senior engineer reads it. A read-only TUI that turns a git change into a guided tour.

A tiny yet powerful LLM inference system tailored for researching purpose. vLLM-equivalent performance with only 2k lines of code (2% of vLLM).

Fast DuckDB-powered TypeScript library for tabular analysis, geospatial analysis, network graph analysis, vector similarity search, AI, Google Sheets, and data visualization.

Setup and run a local LLM and Chatbot using consumer grade hardware.

Blazing-fast LLM inference in pure Rust. No PyTorch and Python runtime.

High-performance lightweight proxy and load balancer for LLM infrastructure. Intelligent routing, automatic failover and unified model discovery across local and remote inference backends.

Versatile Almost Local, Eventually Reasonable Assistant 🔫

AI-powered offensive security testing using autonomous agents, directly in your terminal.

Retrieve, aggregate, filter, evaluate, rewrite and serve RSS feeds using Large Language Models for fun, research and learning purposes

On-device LLM Inference Powered by X-Bit Quantization

Qwen3.5-122B-A10B on DGX Spark: 28.3 → 51 tok/s (+80%)

☸️ Easy, advanced inference platform for large language models on Kubernetes. 🌟 Star to support our work!

Examples and guides for using the VLM Run API

Implementation for FP8/INT8 Rollout for RL training without performence drop.

verl for a single consumer GPU. PPO, GRPO and on-policy distillation on NVIDIA GPUs.

SkillsVote: Lifecycle Governance of Agent Skills from Collection, Recommendation to Evolution

The production framework for Predictive and Generative AI. Serve any model as an API in one line, with OpenAI/Anthropic/Ollama-compatible endpoints, a built-in chat UI, and native MCP.

Boot your PC straight into an LLM. Rust, UEFI-resident, no operating system underneath.

Seamlessly integrate state-of-the-art transformer models into robotics stacks

Multimodal RAG to search and interact locally with technical documents of any kind

Berth is a lightweight GPU resource manager written in Go. It controls which applications get access to your GPU, when they get it, and how that access is handed off.

NVIDIA AITune is an inference toolkit designed for tuning and deploying Deep Learning models with a focus on NVIDIA GPUs.

Automated alignment adjustment for LLMs — direct steering, LoRA, and MoE expert-granular abliteration, optimized via multi-objective Optuna TPE.

A hands-on course on modern LLM architectures, training, and inference, with step-by-step PyTorch implementations and runnable notebooks.

扫描版 PDF 的书签自动生成工具，可以根据 PDF 目录页内容，为 PDF 设置可跳转的书签。

A PyTorch native library for training speculative decoding models

One API for every LLM: 170+ providers through one client, with a Rust core and native bindings for every major language.

[ICML-2026] Official implementation of "SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience"

CacheRoute is an innovative LLM scheduling scheme dedicated to enabling flexible KV cache reuse across LLM systems, improving task performance and system efficiency.

Shrinkflation detector for Codex

Simple LLM service identification - translate IP:Port to Ollama, vLLM, LiteLLM, or 60+ other AI services in seconds

Easy installation and usage of Rockchip's NPUs found in RK3588 and similar SoCs

Translate entire books with AI you already have.

🌱 A little course on Reinforcement Learning Environments for evaluating and training Language Models

Foundation models based medical image analysis

[ICLR 2026] An official implementation of "CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning"