登录

sonar

dphnAIAGPL-3.0活跃维护

面向大规模部署的 LLM 推理引擎，支持 CUDA、ROCm、TPU、Inferentia 等多种硬件后端，并提供 LoRA 与投机解码等加速能力。

Large-scale LLM inference engine

v0.25.0发布于 2026-11-08 16:00 UTC1,907star +2 / 7 天213fork同步于 2026-11-10 11:46 UTC
推理引擎大模型推理投机解码LoRA多硬件后端
对比 GitHub 官网api-restcudainference-engineinferentiaintel

怎么上手

由 README 提炼，执行前请确认
原文 
curl -fsSL https://sonar.dphn.ai/install.sh | bash
sonar serve Qwen/Qwen3-0.6B --served-model-name qwen3
硬件要求按 README 说明

安装脚本支持 Linux x86-64（NVIDIA CUDA、AMD ROCm 或 CPU）、Linux Arm64（NVIDIA CUDA 或 CPU）及 Apple silicon macOS（Metal）；完整指南另含 Intel XPU、Google TPU。README 未给出具体显存要求。

支持 Apple Silicon需要 CUDA支持 ROCm / AMD
下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼
没人工点评？写下你的经验

基于 vLLM 扩展的大规模 LLM 推理引擎，支持多模态与分布式生产部署

面向大规模部署的 LLM 推理引擎，支持 CUDA、ROCm、TPU、Inferentia 等多种硬件后端，并提供 LoRA 与投机解码等加速能力。

核心能力

  • 连续批处理与分页 KV 缓存管理
  • 默认开启前缀缓存，支持张量/流水线/数据/专家并行
  • 无需 Ray 集群即可多节点多进程部署
  • 支持 MTP、EAGLE、n-gram 等多种投机解码
  • 提供 OpenAI、Anthropic、Kobold 等兼容 API

容易踩的坑

服务默认监听 127.0.0.1:2242，需确认端口占用与外部访问配置；各特性依赖模型、设备、数据类型和量化方式，部署前要查模型矩阵和量化矩阵；源码构建需 Python 3.13 并指定 torch 后端（如 cu130）。

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

最近发版

5 个版本v0.25.0
全部 release
  1. v0.25.0最新正式版2026-11-08 · 2026-11-08 16:00 UTC

    What's Changed

    • fix(ci): serialize CUDA builds on the shared runner (#1799) (416649116) by @AlpinDale
    • feat(ci): add ARM64 CUDA wheel builds (#1798) (5b0ff3aff) by @AlpinDale
    • chore(ci): remove unused Buildkite infrastructure (8af3878ff) by AlpinDale
    • fix(docs): preserve code formatting in generated kernel reference (076845665) by AlpinDale
    • feat!: rename Aphrodite Engine to Sonar (49c386154) by AlpinDale
    • refactor: separate duplicate-file moves to preserve history (805ca6d6c) by AlpinDale
    • refactor: move aphrodite package paths to sonar (e0334ee4b) by AlpinDale
    • feat: add support for lossless quantization of BF16 to NF12 (#1796) (c824d853a) by @AlpinDale
    • chore(sync): sync to upstream b8c04dd3b (#1795) (180c058c1) by @AlpinDale
    • perf(sm120): optimize DeepSeek-V4.1 inference (#1788) (113398b9e) by @AlpinDale
    • fix(cpu): keep the triton-cpu URL dependency out of wheel metadata (#1792) (03fbc02b7) by @AlpinDale
    • feat(cli): restore aphrodite tokenizer tokenizer-only server (#1791) (07449f1b5) by @AlpinDale
    • fix(ci): pin ROCm base image and fix empty Metal reconcile (#1790) (634bc6216) by @AlpinDale
    • chore(sync): port vLLM through 255f14599 (#1787) (11c991b7d) by @AlpinDale

    Full Changelog: https://github.com/dphnAI/sonar/compare/v0.24.1...v0.25.0

  2. v0.24.12026-09-11 · 2026-09-11 20:40 UTC
    查看更新说明

    What's Changed

    • chore: change aphrodite ascii art to sonar (#1785) (82b0cd8c2) by @lucyknada

    Full Changelog: https://github.com/dphnAI/sonar/compare/v0.24.0...v0.24.1

  3. v0.24.02026-09-08 · 2026-09-08 15:35 UTC
    查看更新说明

    What's Changed

    • fix(docs): align Sharp dependency with Astro (dc4e975c5) by AlpinDale
    • fix(ci): use managed Python for CPU wheel builds (#1780) (1055453c8) by @AlpinDale
    • feat: add per-request phrase suppression with rollback (#1778) (73b46abb3) by @AlpinDale
    • fix: restore hybrid DCP handling and repair sync test fixtures (de21128fa) by AlpinDale
    • [sync] [Bugfix] Defer adaptive verification until after kernel warmup (#55455) (3bef3dc0c) by AlpinDale
    • [sync] [CI] Align extraction test with canonical auxiliary layer order (#55457) (f56b885a4) by AlpinDale
    • [sync] [Frontend] Expose multimodal metadata for disaggregated prefill (#54659) (01daa08e0) by AlpinDale
    • [sync] [Bugfix][KV Offload] Stop offloading the final sampled token's KV slot (#54288) (d33f08f8f) by AlpinDale
    • [sync] [Bugfix][KV Offload] Fix SWA store reachability during chunked prefill (#54362) (78f10e1f3) by AlpinDale
    • [sync] [Bugfix] Fall back to T1 when ARC cannot reclaim enough entries from T2 (#55461) (ed99e6ec7) by AlpinDale
    • [sync] [Bugfix][KV Offload] Skip cleaned-up async lookup batches (#55075) (b9fa30c3b) by AlpinDale
    • [sync] [Perf] Use SDPA for BLIP-2 Q-Former attention (#55285) (a02103572) by AlpinDale
    • [sync] [Kimi K3] Support internal prefix checkpoints with partial prefix caching and spec-decoding (#53614) (5909fa1d1) by AlpinDale
    • [sync] Validate scale-out multimodal data before engine handoff (#51898) (6e56e9cca) by AlpinDale
    • [sync] [Kernel] Add fused MoE tuned config for E=256,N=512 on NVIDIA A100 80GB PCIe (#55511) (5c91ebd3f) by AlpinDale
    • [sync] [CPU] [Feat] Add native AMX-FP8 attention impl for Diamond Rapids (#49410) (bef33149b) by AlpinDale
    • [sync] [Core] Sync DP state on the first step of a wave (#52957) (cbc932146) by AlpinDale
    • [sync] [Bugfix][Multimodal] Bound renderer warmup to the prefill token budget (#55448) (069b86e11) by AlpinDale
    • [sync] [Kernel][HY V4] Add Triton iHC pre/post fallback (#55059) (4794ffa76) by AlpinDale
    • [sync] [Bugfix][Kernel] Build fused GDN MTP decode for SM110 (#53835) (f87b9b5a8) by AlpinDale
    • [sync] [Bugfix][Qwen4Exp] fix state index strides in fused PLE conv (#55375) (f6ebe7238) by AlpinDale
    • [sync] [Kernel] Fall back from persiste
  4. v0.23.02026-07-31 · 2026-07-31 05:08 UTC
    查看更新说明

    What's Changed

    • fix(release): apply manylinux tag before PyPI upload (795396a3a) by AlpinDale
    • fix(release): attach all wheels to GitHub releases (#1749) (ba857e9b9) by @AlpinDale
    • fix(build): package native artifacts in CUDA wheels (#1748) (d23fda737) by @AlpinDale
    • fix(moe): support Kimi K3 DeepGEMM gather shape (#1747) (a4a09745f) by @AlpinDale
    • fix(ci): allow execution in ROCm build cache (6586b2889) by AlpinDale
    • fix(ci): isolate wheel builder storage (a646aa2c4) by AlpinDale
    • fix(ci): pin ROCm backfill build driver (c1279d75a) by AlpinDale
    • fix(ci): persist incremental ROCm build state (4ac0eb0bb) by AlpinDale
    • docs: correct installer platform list (2be97a6fc) by AlpinDale
    • docs: list automatic installer platforms (4f8f6aa52) by AlpinDale
    • docs: update stale vLLM references (d41a3ee16) by AlpinDale
    • refactor: remove vLLM compatibility names (aae7ceb48) by AlpinDale
    • fix(sync): complete repository namespace port (db6c97167) by AlpinDale
    • [sync] [ROCm]Migrating Deepseek V3.2 to vllm/models/deepseek_v32/ (#47207) (1a6353405) by AlpinDale
    • [sync] [MyPy][1/N] Fix mypy errors in some tests/ directories and enforce follow-imports=silent (#49570) (e1f8de1f6) by AlpinDale
    • [sync] docs(security): document Ray cluster trust model and env var propagation (#50397) (6dbfb4973) by AlpinDale
    • [sync] [CPU] Bump up CPU kernels to latest version (#50387) (da866b35c) by AlpinDale
    • [sync] [New model] Kimi K3 (#50000) (9c06da9f0) by AlpinDale
    • [sync] [Test][ROCm] Account for gfx950 FP8 RMSNorm rounding (#49839) (898ae52a7) by AlpinDale
    • [sync] [CI] Stabilize speculator memory teardown (#50284) (d01367531) by AlpinDale
    • [sync] [CI][ROCm] Stabilize LLM GC teardown check (#50340) (eeb1d2f8c) by AlpinDale
    • [sync] [Frontend] Lazily initialize chat media connectors (#49914) (c1c9538a9) by AlpinDale
    • [sync] [ROCm] Add AITER FP8 ViT encoder attention (#49937) (a28f50621) by AlpinDale
    • [sync] advance vllm-sha to 0028fc8d8 ([580346f91](https://github.com/dphnAI/sonar/commit/580346f9163f110caaf5949498b61cde
  5. v0.22.02026-07-19 · 2026-07-19 20:37 UTC
    查看更新说明

    What's Changed

用户点评

登录用户的真实使用记录，可按版本和硬件筛选
共 0 条

    还没有人点评。第一个说出真实体验的人最有价值。

    README 原文

    未翻译
    在 GitHub 打开

    Sonar

    Sonar is an inference engine for Hugging Face-compatible language and multimodal models. It provides continuous batching, paged KV-cache management, optimized kernels, quantization, speculative decoding, and distributed serving.

    Sonar is based on vLLM. It includes additional model and quantization formats, sampling methods, kernels, platforms, and deployment features. It serves production workloads for the Dolphin Inference Network and PygmalionAI.

    Documentation

    Read the documentation at sonar.dphn.ai.

    The model, quantization, and server-argument references are generated from the current source tree.

    Install

    Run the automatic installer:

    curl -fsSL https://sonar.dphn.ai/install.sh | bash

    The installer supports Linux x86-64 (NVIDIA CUDA, AMD ROCm, or CPU), Linux Arm64 (NVIDIA CUDA or CPU), and Apple silicon macOS (Metal).

    Use the complete installation guide for AMD ROCm, Intel XPU, CPU, Apple silicon, Google TPU, Docker, WSL 2, source builds, and nightly wheels.

    Serve a model

    sonar serve Qwen/Qwen3-0.6B \
  --served-model-name qwen3

    The server listens on http://127.0.0.1:2242 by default. It provides OpenAI-compatible APIs, health checks, metrics, and an OpenAPI schema.

    See the OpenAI-compatible API guide for streaming, embeddings, tool calls, reasoning output, and Sonar request parameters.

    Key features

    • Continuous batching and paged KV-cache management
    • Prefix caching enabled by default
    • Tensor, pipeline, data, and expert parallelism
    • Multi-node multiprocessing without a Ray cluster
    • Prefill/decode disaggregation through NIXL and other KV connectors
    • Quantized weights and FP8 KV cache
    • Speculative decoding with MTP, EAGLE, DSpark, DFlash, n-gram, and other methods
    • Structured output, reasoning parsers, and automatic tool calling
    • Image, audio, and video model support
    • LoRA adapter serving
    • Prometheus metrics and health endpoints
    • OpenAI, Anthropic, pooling, scoring, reranking, transcription, and Kobold APIs

    Support depends on the model, device, data type, and quantization method. Check the generated model matrix and quantization matrix before deployment.

    Read the optimization guide before you tune scheduler, cache, compilation, quantization, or speculative decoding settings.

    Development

    git clone https://github.com/dphnAI/sonar.git
cd sonar
uv venv --python 3.13 --seed --prompt sonar
source .venv/bin/activate
SONAR_USE_PRECOMPILED=1 \
  uv pip install --editable . --torch-backend=cu130

    相关项目

    同分类里 star 最高的项目
    对比全部 →
    unsloth
    unslothai·Python

    本地跑和训练大模型、扩散模型的图形界面，支持 GGUF、MLX 及 Qwen、DeepSeek、Gemma、FLUX 等主流模型，省去命令行配置的麻烦。

    本地推理+4参数高效微调
    79276+1277.4kv0.1.916-beta2026-11-10 11:18 UTC
    annotated_deep_learning_paper_implementations
    labmlai·Python

    用 PyTorch 逐行复现 60 多篇深度学习经典论文，代码旁配注释讲解，涵盖 Transformer、优化器、GAN、强化学习等，适合边读论文边对照实现学习。

    论文复现+4参数高效微调
    67742+276.8k2026-11-02 05:57 UTC
    airllm
    lyogavin·Jupyter Notebook

    单张 4GB 显存显卡就能跑 70B 甚至 2.8T 参数大模型的分层推理方案，通过逐层加载权重把显存需求压到极低，适合本地低配设备部署大模型。

    大模型推理+4参数高效微调
    42086+1874.3kv4.0.02026-11-10 06:14 UTC
    self-llm
    datawhalechina·Jupyter Notebook

    面向中文用户的 Linux 环境大模型实战教程，手把手教你在本地微调（全参数/LoRA）和部署国内外开源 LLM 与多模态模型。

    大模型微调+4参数高效微调
    32744+143.2k2026-10-29 10:59 UTC