登录

shimmy

Michael-A-KuykendallApache-2.0活跃维护 有专家点评

纯 Rust 实现的 WebGPU 推理引擎，原生支持 GGUF 模型，兼容 OpenAI API，无需 Python 和 llama.cpp，单个二进制文件即可在任意 GPU 上跑本地大模型。

⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.

v2.6.6发布于 2026-11-07 20:08 UTC6,015star +3 / 7 天586fork同步于 2026-11-11 05:33 UTC
推理引擎WebGPUGGUF本地部署OpenAI兼容
对比 GitHubapi-servercommand-line-tooldeveloper-toolsggufhuggingface

专家点评

上手难度：进阶更新于 2026-11-11

纯 Rust 单二进制的本地 GGUF 推理服务器，用来顶替 llama.cpp server 或 Ollama 那一层

适用场景

  • 内网或离线环境部署 OpenAI 兼容端点：一个二进制丢进去就能起服务，不用带 Python 运行时和 CUDA 工具链，比 llama.cpp server 少一层构建依赖
  • 已有客户端代码指向 OpenAI API、想切到本地模型做联调：改 base_url 到 11435 即可，不用改 SDK
  • 边缘设备或容器镜像体积敏感的场景：没有 Python 解释器和 torch 依赖，镜像层数能压下来
  • 想用 Rust 直接嵌推理能力而不是外挂进程：库形态比 llama.cpp 的 C++ FFI 绑定更顺手

踩坑

  • 模型覆盖范围窄：README 只验证了 Qwen3-0.6B/1.7B 和 Llama 3.2 3B 的 Q4_K_M 等有限组合，Gemma2、MoE、多模态执行器未启用。拿一个非验证过的 GGUF 上去，很可能加载失败或输出异常，且没有 llama.cpp 那样的广泛模型兼容兜底
  • 预编译产物和安装路径有坑：issue 里反复出现 crates.io 缺模板文件、从 Git 安装依赖冲突、GitHub 预编译包不含 GPU 支持这几类问题。想省事直接下 release 二进制的人会撞上 GPU 不可用，走 cargo install 的人会撞上打包问题
  • 平台验证不均：完整 logits 的 GPU 验证只在 NVIDIA 经 WSL 的 Dozen/Vulkan 路径做过，Metal/M4 未完整验证，issue 里也有 MLX 上 smoke test 挂掉的记录。macOS 上跑要有心理准备

同类对比

llama.cpp 的 llama-server 模型兼容面和量化格式支持更广，是默认选择；要开箱即用的模型管理和拉取体验选 Ollama。只有在明确不想引入 Python/C++ 构建链、且模型在验证列表内时，shimmy 的单二进制才划算

怎么上手

由 README 提炼，执行前请确认
原文 
cargo install shimmy --version 2.6.6 --locked
shimmy --ctx 512 --kv-quant f32 serve --model-path /path/to/model.gguf --bind 127.0.0.1:11435
硬件要求按 README 说明

需要可用的 Rust 工具链和 GPU 驱动；完整 logits 的 GPU 验证仅在 NVIDIA 经 WSL 的 Dozen/Vulkan 路径执行，Metal/M4 等平台未做完整验证。README 未给出显存要求。

支持 Apple Silicon多卡 / 张量并行
下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼

纯 Rust 的本地 GGUF 推理服务器，提供 OpenAI 兼容 API，无需 Python 和 llama.cpp

纯 Rust 实现的 WebGPU 推理引擎，原生支持 GGUF 模型，兼容 OpenAI API，无需 Python 和 llama.cpp，单个二进制文件即可在任意 GPU 上跑本地大模型。

核心能力

  • 用 Rust 与 WGSL 实现 GPU 推理，单二进制部署
  • 原生加载 GGUF 模型，无需 llama.cpp 转换
  • 提供 OpenAI 兼容的模型列表、JSON 对话与文本补全接口
  • 对话支持 SSE 流式输出，文本流式请求会被拒绝
  • 非回环绑定需设置 SHIMMY_API_KEY，浏览器来源需配置允许列表

容易踩的坑

需自行准备 GGUF 模型文件，README 未提供自动下载；默认监听 11435 而非 11434，客户端需改地址；仅验证了 Qwen3-0.6B/1.7B 与 Llama 3.2 3B 的 Q4_K_M 等有限组合，Gemma2、MoE、多模态等执行器未启用；fast profile 不提供推理。

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

最近发版

5 个版本v2.6.6
全部 release
  1. v2.6.6shimmy 2.6.6最新正式版2026-11-07 · 2026-11-07 20:08 UTC

    Shimmy 2.6.6 — verified native distribution

    Download the platform .tar.gz or Windows .zip bundle for the default Airframe engine with complete licensing materials. The raw executable assets are also provided. SHA256SUMS covers the four raw binaries; BUNDLE-SHA256SUMS covers the four self-contained archives; RELEASE-SHA256SUMS covers all companion assets. Windows users should use Invoke-WebRequest and Expand-Archive as described in the Windows guide, avoiding the PowerShell curl alias.

    Executed build and install checks

    Linux x86-64, Linux ARM64, Windows x86-64 and macOS ARM64 were built and tested natively from unchanged tag v2.6.6/source21c7d28381352e1cafc55e84ba941b48d2b12d3e, using Rust1.99.1 and the locked default Airframe dependency graph. Native default/fast unit tests, final default-engine compilation, copy-install/PATH version/help checks, architecture headers, hashes and actual compiled-dependency license coverage passed in one manually triggered build run. The fast profile is not the distributed inference engine.

    Correctness and limits

    Retains the existing dense Llama/Qwen3 correctness foundation: checked GGUF admission, architecture-aware state/reset contracts, Q/K normalization ownership and tensor-window span corrections, and typed server errors. See 2.6.6 changelog, exact model matrix and certification gates. Full-logit limits remain0.1 maximum absolute/0.03 RMS; no broad family or performance guarantee is introduced.

    A separate native Apple Paravirtual Metal diagnostic executed actual GPU arithmetic and the exact Qwen3-0.6B fixture: reporter/omitted-stream requests returned JSON, correct2+2=4 and nonzero/accounted usage. Its initial job failed an overly strict bare4 string assertion; the preserved responses pass corrected local arithmetic validation without a GPU rerun. Original diagnostic run and METAL-DIAGNOSTIC.json preserve that distinction. This is not full-vector or physical M4 Max certification.

    NOT_RUN: full-vector Metal/M4 and other unexecuted native GPU configurations, DGX Spark inference, INT4 KV, experimental kernels and unlisted model/context/mode combinations. Qwen3.5/hybrid recurrent, Gemma/MoE/MLA/SSM/multimodal executors are not enabled by this dense release. WSL destructor and active-generation device-loss investigations remain open. No M2/M4 speedup or resolution of every issue218 configuration is claimed.

    Dependency security disposition

    The frozen applicable security corrections remain in the locked build. The three disclosed Quinn vulnerability coordinates are absent from all four final compiler graphs. Paste1.0.15's RUSTSEC-2024-0436 unmaintained-package warning is active on macOS; it remains disclosed and unchanged, not suppressed or called inactive there. This is not a claim of complete security clearance.

    Distribution scope

    This release supplies the four verified native binary bundles. The versioned ghcr.io/michael-a-kuykendall/shimmy:2.6.6 image is published for Linux x86-64/ARM64 after native startup, licensing and API-authentication tests; Docker latest is unchanged. Container instructions.

    The custom Homebrew tap and Scoop bucket install the exact licensed release archives with verified checksums. Real Homebrew install/test passed on Linux x86-64 and macOS ARM64; Linux ARM64 Homebrew execution is NOT_RUN. Real

  2. v2.6.5Shimmy v2.6.52026-10-19 · 2026-10-19 00:47 UTC
    查看更新说明

    [2.6.5] — 2026-10-18 — Airframe 0.4.4

    Fixed

    • Return exact prompt/completion token counts and truthful stop/length reasons in OpenAI/native JSON responses and OpenAI streaming responses.
    • Support stream_options.include_usage; report streaming generation failures as errors instead of successful completion.
    • Render HTTP chat and completion prompts once before Airframe generation, preserving ordered conversation turns and explicit raw prompts.
    • Return a JSON chat completion when the OpenAI stream field is omitted.
    • Add weight-free protocol regressions for prompt routing and streaming.
    • Restore default-feature invariant-test imports and exercise those tests in CI.
    • Restore the missing v2.6.4 release notes required by binary publication.
    • Correct response-cache byte accounting on overwrite and eviction accounting on clear (#217). Thanks to @aiqubits for the diagnosis and fix proposal (#219).

    Security

    • Require SHIMMY_API_KEY for non-loopback HTTP binds and protect inference, management, diagnostics and WebSocket routes when a key is configured.
    • Replace wildcard CORS with explicit SHIMMY_ALLOWED_ORIGINS.
    • Require deployment credentials, keep Compose host ports on loopback, and use a Kubernetes ClusterIP service with Secret-backed authentication (#221).

    Changed

    • Use Airframe 0.4.4 and airframe-observe 0.1.3.
    • Test on native Linux x86-64/ARM64, Windows x86-64 and macOS ARM64 runners.
    • Add Linux ARM64 release binaries and publish SHA256 checksums for all assets.
    • Refresh Docker images for the current Airframe build and Vulkan runtime.
    • Resolve shimmytok 0.8.4 in the release lockfile.
    • Correct quickstart commands and add scripts/smoke_text.py for model-backed endpoint verification.
  3. v2.6.42026-08-30 · 2026-08-30 22:48 UTC
    查看更新说明

    Full Changelog: https://github.com/Michael-A-Kuykendall/shimmy/compare/v2.6.3...v2.6.4

  4. v2.6.3Shimmy v2.6.32026-08-29 · 2026-08-29 00:05 UTC
    查看更新说明

    [2.6.3] — 2026-08-28 — Intel Arc Binding-Limit Fix

    Fixed

    • Integrated GPU model loading. Shimmy now uses Airframe 0.4.2, which sizes bindless model chunks from the adapter's actual storage-buffer binding limit instead of rejecting integrated GPUs with a limit below the default 2 GB chunk size. This resolves #214.

    Changed

    • Updated the Airframe dependency to 0.4.2.
  5. v2.6.2Shimmy v2.6.22026-08-28 · 2026-08-28 23:04 UTC
    查看更新说明

    [2.6.2] — 2026-08-28 — OpenAPI and Swagger Documentation

    Added

    • OpenAPI and Swagger UI documentation. Running servers now expose the complete HTTP API contract at GET /openapi.json and an interactive Swagger UI at GET /docs, including OpenAI-compatible, native, Ollama-compatible, Anthropic-compatible, diagnostics, and WebSocket routes. See docs/OPENAPI.md. Closes #153.

用户点评

登录用户的真实使用记录，可按版本和硬件筛选
共 0 条

    还没有人点评。第一个说出真实体验的人最有价值。

    README 原文

    未翻译
    在 GitHub 打开

    Shimmy — Local Inference, OpenAI-Compatible

    🔒 Local GPU inference — Rust/WGSL application 🚀

    License: Apache-2.0 Security CI Crates.io Downloads Rust GitHub Stars 💝 Sponsor this project Trans rights LGBTQ+ friendly

    Shimmy 2.6.6

    Shimmy is a local inference server and CLI; Airframe 0.4.5 is its engine. This is a coordinated dense-text correctness, dependency-security and documentation release, not a new architecture or performance release. See CHANGELOG.

    Default dependency licensing: Shimmy → Airframe → libfse. Shimmy is Apache-2.0; Airframe and airframe-observe are MIT; ShimmyTok is MIT OR Apache-2.0. The libfse 0.1.4 additional Apache-2.0 grant permits commercial use, modification, redistribution and incorporation into products without separate commercial permission. The original crates.io archive and metadata remain unchanged. Preserve component and third-party notices; see stack licensing.

    Install and run

    Verified native 2.6.6 archives and checksum instructions are available separately from crates.io, including the Windows executable and complete licensing bundles.

    Install the versioned crate with a Rust toolchain:

    cargo install shimmy --version 2.6.6 --locked
shimmy --version
shimmy --ctx 512 --kv-quant f32 serve --model-path /path/to/model.gguf --bind 127.0.0.1:11435

    Use a verified admitted fixture in place of the path. A working Rust toolchain and GPU driver are required. The fast profile does not provide inference. The separately verified native downloads, Homebrew and Scoop installers and versioned GHCR image are now available for 2.6.6. A crates.io upload does not create those channels automatically. Docker latest is unchanged; npm is not included. Verify the exact installed version.

    See quickstart for model hashes, listing, requests and network access. Unavailable translations are not linked as current guides.

    Correctness and support

    The exact fixture matrix covers selected Qwen3-0.6B/Qwen3-1.7B and Llama 3.2 3B Q4_K_M cases, with independent FP32 full logits, maximum absolute 0.1/RMS 0.03 gates and matching measured greedy decisions. These are bounded model/context/mode results, not universal family support.

    F32 KV and reference kernels define the validated path. Parsing metadata, tokenizers or quantization is not certification. Gemma2, MoE, MLA, recurrent, state-space and multimodal executors are not enabled. INT4/TurboShimmy and experimental kernels remain excluded and rejected.

    Executed full-logit GPU evidence is NVIDIA through WSL Dozen/Vulkan. NOT_RUN for full-logit certification: Metal/M4, other unexecuted native GPU platforms, extended-context Llama, alternate prefill chunks and unlisted combinations. WSL device-destruction and active-generation device-loss investigations remain open. No universal determinism, startup, multi-GPU, memory or speed guarantee is made.

    Server contract

    Selected OpenAI-compatible endpoints provide model listing, JSON chat/text completions and explicit chat-only SSE. Chat defaults to JSON; text streaming rejects. Usage, stop/finish metadata and typed errors are part of the tested contract. This is not every OpenAI endpoint/SDK feature.

    Loopback can run without a key. Non-loopback binds require SHIMMY_API_KEY; browser origins require SHIMMY_ALLOWED_ORIGINS. With a key, routes except GET /health require credentials; origin checks still apply to health. Use TLS and appropriate network access controls remotely. Lazy loading is the default; optional selected-model warm-up shifts cold work before readiness without eliminating it or guaranteeing stability.

    Documentation

    Shimmy source is Apache-2.0. Airframe runtime and Observe are MIT; ShimmyTok is dual-licensed. libfse 0.1.4 is covered by the additional Apache-2.0 grant, permitting commercial use without separate permission. Preserve applicable component, dependency and model terms; see stack licensing.

    Support

    This project is independently maintained. Component licensing terms described above apply. Sponsorship helps fund ongoing development, certification, compatibility work, and releases.

    Sponsor on GitHub · Buy Me a Coffee

    Any amount is appreciated. Suggested tiers:

    • $5/month — Thanks
    • $15/month — Supporter
    • $50/month — Backer
    • Custom — Email for invoicing

    License & Philosophy

    Shimmy server code: Apache-2.0 — see LICENSE. The default dependency stack permits commercial use under its component terms, including the libfse additional grant. See stack licensing; original published archives remain unchanged.

    Promise: This will never become a paid product.

    Shimmy is infrastructure: it should be invisible. Reliability through comprehensive validation and property-based testing.

    Maintainer: Michael A. Kuykendall · Mission: Making local model inference simple and reliable

    Support

    This project is a safe space. Trans rights are human rights.

    If you or someone you love needs support:

    相关项目

    同分类里 star 最高的项目
    对比全部 →
    headroom
    headroomlabs-ai·Python

    在工具输出、日志、文件和 RAG 片段进入 LLM 前先做压缩，编码 Agent 省约 20% token，JSON 省 60-95%，答案不变。提供库、代理和 MCP 服务器三种用法。

    上下文压缩+4量化进阶
    79325+2806.2kv0.44.02026-11-11 04:46 UTC
    llmfit
    AlexsJones·Rust

    一条命令就能从数百个模型和供应商里筛出适合你硬件配置的本地大模型，省去逐个试跑和查显存的麻烦。

    本地推理+4量化入门
    38904+1772.5kv1.1.202026-11-10 10:03 UTC
    PaddleNLP
    PaddlePaddle·Python

    基于飞桨的 NLP 与大模型开发库，提供丰富模型库，覆盖 BERT、ERNIE、LLaMA 等，支持分布式训练、压缩和文档智能等任务。

    大模型开发库+4量化硬核
    12987+13.0krl-v1.0.02026-05-23 02:54 UTC
    bitsandbytes
    bitsandbytes-foundation·Python

    让大模型在普通显卡上跑起来的 PyTorch 量化库，支持 8-bit 和 4-bit 权重压缩，也是 QLoRA 微调背后的核心依赖。

    量化+4量化进阶
    85379450.50.22026-11-02 14:36 UTC