登录

vllm-mlx

waybarriosApache-2.0活跃维护

在 Apple Silicon 上跑大模型的高性能推理服务，兼容 OpenAI 和 Anthropic 接口，支持 MLX 原生加速、连续批处理、多模态和 MCP 工具调用。

High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.

v0.6.0发布于 2026-10-20 05:21 UTC1,651star +1 / 7 天237fork同步于 2026-11-10 23:48 UTC
推理服务Apple SiliconMLX连续批处理多模态
对比 GitHub 官网anthropicanthropic-apiapple-siliconclaude-codecontinuous-batching

怎么上手

由 README 提炼，执行前请确认
原文 
python -m pip install 'vllm-mlx==0.4.1'
vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --host 127.0.0.1 --port 8000
硬件要求按 README 说明

仅支持 Apple Silicon（M1、M2、M3、M4、M5），通过 MLX 使用 Metal 内核和统一内存；README 未给出最低内存要求，性能表中 30B MoE 4bit 模型约占用 18 GB。

支持 Apple Silicon需要 CUDA
下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼
没人工点评？写下你的经验

在 Apple Silicon 上跑 vLLM 风格推理服务，一个进程同时提供 OpenAI 和 Anthropic 兼容 API

在 Apple Silicon 上跑大模型的高性能推理服务，兼容 OpenAI 和 Anthropic 接口，支持 MLX 原生加速、连续批处理、多模态和 MCP 工具调用。

核心能力

  • 单进程同时暴露 OpenAI /v1/* 与 Anthropic /v1/messages
  • 用连续批处理、分页 KV 缓存和前缀缓存提升并发吞吐
  • 支持 SSD 分层 KV 缓存，可把前缀缓存写入磁盘
  • 支持文本、图像、视频、音频多模态与原生 TTS/STT
  • 内置 19 种工具调用解析器并支持 JSON Schema 结构化输出

容易踩的坑

首次启动需下载并加载模型，耗时取决于机器和网络；默认引擎不含连续批处理，需加 --continuous-batching 重启才启用；OpenAI SDK 不在服务端包内，客户端环境需单独安装 openai。

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

最近发版

5 个版本v0.6.0
全部 release
  1. v0.6.0最新正式版2026-10-20 · 2026-10-20 05:21 UTC

    vllm-mlx 0.6.0 adds media-aware SpecPrefill, broader assistant-drafter loading, and more reliable multimodal requests, tool calls, caching, and model cleanup for Apple Silicon.

    Highlights

    • Media-aware SpecPrefill: use sparse prefill with supported VLMs on batched media routes. Assistant drafters now load through architecture-aware loaders, with safer registered MTP drafter resolution and validation.
    • More reliable multimodal APIs: preserve Anthropic image blocks, reject invalid multimodal inputs atomically, and honor temperature and top-p sampling for image requests.
    • Tokenizer and reasoning fixes: return the text tokenizer from SimpleEngine for MLLM models, restoring token counting, thinking budgets, and JSON output. Detect template-opened thinking markers and honor the model's complete EOS set on text routes.
    • Tool-call streaming fixes: dispatch GLM-4.7 native tool calls automatically, preserve schema-aware arguments and Qwen argument whitespace, and send each streamed tool call once.
    • Better operational tooling: add capacity-envelope benchmark artifacts and client-acceptance coverage for seven coding agents, with clearer request, cache, and engine-state reporting.

    Reliability improvements

    • Release MLLM model memory during registry idle unload, including self-referential generators created by chunked prefill.
    • Keep prefix-cache I/O on the owning thread and materialize batch cache metadata after membership changes to prevent lazy-graph handle retention.
    • Make hybrid prefix-cache replay owned and exact, isolate disk caches by model revision and quantization, and harden rotating-cache trim and store paths.
    • Surface generation failures as API errors instead of empty completions, and detect and reap dead registry engines.
    • Preserve authoritative MLLM prompt-token usage and per-request speculative-decoding counters; finish HTTP metrics when streams are exhausted.
    • Resolve MLLM versus LLM routing from model configuration before engine selection and load processors for MLX Whisper checkpoints correctly.

    Upgrade notes

    • Runtime inference requires Apple Silicon and Python 3.10 or later.
    • The supported mlx-lm range is now >=0.31.3,<0.32.0; newer cache and generation APIs require further compatibility work.
    • MLX_BUFFER_CACHE_LIMIT accepts human-readable sizes and uses a device-scaled fallback when no valid override is set.
    • Models with hybrid or rotating caches receive additional replay, ownership, and sequential-prefix regression coverage.

    Install

    python -m pip install --upgrade 'vllm-mlx==0.6.0'

    Optional Harmony support:

    python -m pip install --upgrade 'vllm-mlx[harmony]==0.6.0'

    What's Changed

  2. v0.5.02026-09-17 · 2026-09-17 21:48 UTC
    查看更新说明

    vllm-mlx 0.5.0 adds reusable conversation caching, broader model and parser support, and more reliable streaming and resource management for Apple Silicon.

    Highlights

    • SimpleEngine prefix-trie caching: reuse growing conversation prefixes with configurable entry and memory limits. This release also fixes a completion race so a normally finished stream stores its cache entry reliably.
    • DeepSeek-V4-Flash support: native prompt encoding, reasoning extraction, and DSML tool calls, including streamed and multi-turn tool use.
    • MLLM assistant drafting: support configured assistant drafters with continuous batching, plus MTP routing and mlx-lm compatibility fixes.
    • More reliable API responses: preserve assistant text alongside tool calls, retain replayed reasoning, forward reasoning effort, and normalize Anthropic system messages consistently.
    • Operational controls: configure embedding input limits and overflow handling, override registry memory budgets, and report registry engine state and MLLM cache memory more accurately.

    Reliability improvements

    • Keep model loading and generation on the owning thread in SimpleEngine, BatchedEngine, and VLM text routes.
    • Improve prefix-cache ownership, snapshot isolation, non-trimmable cache reuse, and rotating-cache guards.
    • Apply --max-kv-size limits and honor --prefill-step-size with continuous batching.
    • Preserve streamed tokens when --stream-interval is greater than one; avoid leaking split reasoning markers and isolate parser state between requests.
    • Add Step3p5 parsing and recognize additional Llama 3.1/3.3 tool-call formats.
    • Clear generator-owned vision caches correctly, release idle model memory, and close delegated streams and SSD caches during shutdown.

    Upgrade notes

    • Runtime inference requires Apple Silicon and Python 3.10 or later.
    • Structured-output enforcement now rejects requests when the requested constraints cannot be enforced, instead of silently allowing unconstrained output. The package includes llguidance>=1.7.6.
    • Prefix-trie caching is opt-in with --prefix-trie-cache. Use --prefix-trie-cache-size and --prefix-trie-cache-memory-mb to control its capacity.
    • Embedding operators can set --embedding-max-length and choose --embedding-overflow-policy truncate or error.

    Install

    python -m pip install --upgrade 'vllm-mlx==0.5.0'

    Optional Harmony support:

    python -m pip install --upgrade 'vllm-mlx[harmony]==0.5.0'

    What's Changed

  3. v0.4.12026-08-12 · 2026-08-12 05:13 UTC
    查看更新说明

    vllm-mlx 0.4.1 collects the latest scheduler, streaming, MTP, parser, memory, and model compatibility improvements merged since 0.4.0.

    Highlights

    • Preserve active streams and correct finish reasons across SimpleEngine streaming paths.
    • Support native mlx-lm chunked prefill while keeping MLLM media generation on its owner thread.
    • Improve MTP compatibility for Qwen models, sampled concurrent MLLM decoding, and native status counters.
    • Add request-local Poolside v1 parsing and complete GPT-OSS harmony tool-call responses.

    Improvements

    Streaming and scheduling

    • Keep active streaming responses alive through request cleanup.
    • Preserve model-provided finish reasons and emit stop for natural completion.
    • Support the current native mlx-lm chunked-prefill API.
    • Keep incremental MLLM media generation on the model owner thread.

    MTP and model compatibility

    • Support sampled concurrent MLLM decoding and standard Qwen MTP shard prefixes.
    • Pass Qwen pre-norm hidden states to MTP checkpoints.
    • Expose cumulative native MTP counters without resetting them when sampling changes replace the batch generator.
    • Select text-model families by prefix and require mlx-vlm 0.6.5 for corrected Qwen 3.5 weight loading.

    Tool calling and parsing

    • Add request-local Poolside v1 reasoning and tool parsers.
    • Preserve complete GPT-OSS harmony tool calls through final responses.
    • Parse the Mistral [ARGS] format used by Ministral 3 and Devstral Small 2.

    Memory and API behavior

    • Release MLX embedding buffers after each batch.
    • Reconcile registry budgets with Metal allocation limits.
    • Resolve embedding and rerank truncation from each model's configuration instead of a fixed length.

    Install

    pip install vllm-mlx==0.4.1

    Harmony support remains available as an optional extra.

    pip install "vllm-mlx[harmony]==0.4.1"

    Changes

    • Poolside v1 request-local parsers (#644)
    • MLX embedding buffer cleanup (#667)
    • Active streaming response preservation (#666)
    • Sampled concurrent MLLM MTP decoding (#662)
    • mlx-vlm 0.6.5 compatibility floor (#633, #675)
    • Prefix-cache guard regression coverage (#655)
    • Standard Qwen MTP shard prefixes (#664)
    • Mistral [ARGS] tool-call parsing (#631)
    • Complete GPT-OSS harmony tool-call responses (#562)
    • Registry and Metal memory budget reconciliation (#696)
    • Model-aware embedding and rerank truncation (#626)
    • Streaming finish-reason preservation (#681, #629)
    • Prefix-based text-model family selection (#686)
    • Qwen pre-norm MTP hidden states (#660)
    • Native mlx-lm chunked prefill support (#648)
    • MLLM owner-thread media streaming (#551)
    • Native MTP status counters with cumulative replacement-safe state (#656)

    Thanks to @Thump604, @SvaRgoS, @mabaeyens, @CBribiescas, @danmackinlay, @brandy975, @janhilgard, and @mvmories for the contributions included in this release.

    Full changelog https://github.com/waybarrios/vllm-mlx/compare/v0.4.0...v0.4.1

  4. v0.4.02026-06-28 · 2026-06-28 20:20 UTC
    查看更新说明

    vllm-mlx 0.4.0 finalizes the 0.4.0 line (promoting 0.4.0rc1). It brings harmony/GPT-OSS prompt rendering, omni audio-from-video fusion, a tiered SSD KV cache that now covers multimodal models, configurable Metal memory, and a long list of tool-calling, reasoning, and MLLM correctness fixes.

    Highlights

    • GPT-OSS / harmony prompt rendering through the official openai-harmony renderer, so prior assistant tool_calls survive multi-turn agentic conversations instead of being flattened to bracket text. Optional, gated behind --enable-auto-tool-choice + --tool-call-parser harmony|gpt-oss and the vllm-mlx[harmony] extra, with a clean fallback to apply_chat_template when the package is absent (#581).
    • Omni audio-from-video fusion: omni models (e.g. Nemotron-Omni, Qwen2.5-Omni) auto-extract the audio track from a video_url and fuse it into the forward pass; caller-provided audio always wins, non-omni models are untouched (#591).
    • SSD cold tier for multimodal: --ssd-cache-dir now wires the SSD prefix-cache tier onto the MLLM/VLM path, not just text models (#618).
    • Configurable Metal buffer cache: MLX_BUFFER_CACHE_LIMIT env var, with a device-scaled default instead of a hardcoded 32 GB cap, fixing OOM on <=32 GB Macs (#620).
    • Multi-slot system-KV cache with hit-ratio counters, extended to the pure-LLM stream_chat path (#541, #523).
    • logit_bias support on the chat endpoint (#596).

    Constrained decoding & tool calling

    • Validate OpenAI tool function names (#582)
    • Bound constrained JSON whitespace progress (#547)
    • Route constrained non-stream chat through the stream path (#548)
    • Qwen3-XML: parse bare <function=> without a <tool_call> wrapper (#597)
    • Qwen tool parser: extract mixed XML + function calls, strip truncated markup (#583)
    • Gemma 4: parse paren / fenced / unfenced tool_code fallback forms (#623)
    • Gemma 4: route the streaming tool marker out of the reasoning stream (#625)
    • Fail invalid response_format output before returning success (#539)
    • Correct _suffix off-by-one and strip backslashes before non-ASCII in JSON output (#525)
    • Fix dangling think before tool calls in templates (#494)

    Reasoning

    • Mistral / Ministral [THINK] reasoning parser (#599)
    • Respect enable_thinking=false from chat_template_kwargs (#537)
    • No-final-content watchdog metadata for thinking routes (#512)

    MLLM / multimodal

    • Keep assistant tool_calls and tool messages on the MLLM chat path so agent continuations differ from the fresh prompt (#611)
    • Honor chat_template_kwargs (e.g. enable_thinking) on the --mllm path (#600)
    • Preserve MLLM content part order (#536)
    • Detect MLLM via config.json with a substring matcher fallback (#520)
    • Wire MLLM assistant drafters for Gemma 4 MTP, expose MTP draft counters/stats, clarify bypass counter semantics (#507, #504, #473, #527)
    • Gemma 4 text-model dispatch and build fixes: realize private lazy arrays on the build thread, eval() the derived TextModel for the fast kernel, honor per-layer quantization overrides, match mlx-vlm 0.5.0 attention signature (#614, #606, #607, #595, #564)
    • Qwen 3.5/3.6: accept the current mlx-vlm attention kwargs in the batch patch (#616)
    • Require mlx-vlm 0.6.2 for Step 3.7 Flash (#589)

    KV cache & memory

    • Preserve original bfloat16 dtype across quantized SSD spill; snapshot KV on the producer thread; spill native QuantizedKVCache and handle bf16 buffers (#612, #563, #605)
    • Fix hybrid cache snapshot aliasing (#576)
    • System-prompt canonicalization for OpenAI paths (#528)
    • Fix sampling defaults and short prefix-cache reuse (#424)

    Streaming & concurrency

    • Preserve the VLLM_MLX_SIMPLE_ENGINE_LOCK_ADMISSION=wait opt-in; fail fast on an occupied SimpleEngine route (#615, #540)
    • Trim Python overhead in decode, streaming, and constrained-decoding hot paths (#526)
    • Thread the SpecPrefill backbone policy (#579)

    Server / API

    • Use the request model in registry responses (#54
  5. v0.4.0rc1pre-release2026-05-21 · 2026-05-21 19:54 UTC
    查看更新说明

    This release candidate adds Gemma 4 MTP support via MLLM assistant drafters, extends the system-prompt KV cache to the pure-LLM streaming path, and trims Python overhead across decode, streaming, and constrained-decoding hot paths. It also lands the bench-serve workload runner refactor and several reasoning and tool-calling correctness fixes.

    This is a release candidate. Install with pip install vllm-mlx==0.4.0rc1 --pre.

    Highlights

    • Gemma 4 MTP support wired through the MLLM assistant drafter path (#507), with counters and effective draft stats exposed for inspection (#473, #504).
    • The system-prompt KV cache now covers the pure-LLM stream_chat path, not only the chat-completion routes (#523).
    • Python overhead in decode, streaming, and constrained-decoding hot paths reduced (#526).
    • bench-serve workload runner split into focused helpers across four PRs (#515, #517, #518, #519).

    MLLM and multimodal

    • Wire MLLM assistant drafters for Gemma 4 MTP (#507)
    • Expose effective MLLM MTP draft stats (#473)
    • Expose MLLM MTP draft counters (#504)
    • Detect MLLM via config.json with a substring matcher as fallback (#520)

    Reasoning and tool calling

    • Fix dangling think tags before tool calls in chat templates (#494)
    • Respect enable_thinking=false from chat_template_kwargs (#537)
    • Correct _suffix off-by-one and strip backslashes before non-ASCII in JSON output (#525)
    • Fix sampling defaults and short prefix-cache reuse (#424)
    • Fix unexpected keyword argument 'mtp' when enable-mtp is set (#503)

    KV cache and memory

    • Extend the system-prompt KV cache to the pure-LLM stream_chat path (#523)

    Performance

    • Trim Python overhead in decode, streaming, and constrained-decoding hot paths (#526)

    Server and API

    • Use the request model in registry responses (#549)

    bench-serve

    • Split load_workload and harden the workload runner with regression tests (#515)
    • Split validate_quality_checks into per-check helpers (#517)
    • Split run_workload_case into request, transport, and record helpers (#518)
    • Split summarize_workload_results into per-case helper (#519)

    Tooling and docs

    • Re-enable lifecycle tests and wire them into CI (#532)
    • Add missing sampling defaults to lifecycle CLI fixtures (#558)
    • Add structured issue templates and refresh PR template (#559)
    • Add M5 to the Apple Silicon support list (#557)
    • Document PR merge readiness checklist (#492)

    Install

    pip install vllm-mlx==0.4.0rc1 --pre

    Full diff: https://github.com/waybarrios/vllm-mlx/compare/v0.3.0...v0.4.0rc1

用户点评

登录用户的真实使用记录，可按版本和硬件筛选
共 0 条

    还没有人点评。第一个说出真实体验的人最有价值。

    README 原文

    未翻译
    在 GitHub 打开

    vllm-mlx

    Continuous batching + OpenAI + Anthropic APIs in one server. Native Apple Silicon inference.

    Read this in other languages: English · Español · Français · 中文

    PyPI version PyPI Downloads Python 3.10+ License Apple Silicon GitHub stars

    What is vllm-mlx?

    A vLLM-style inference server for Apple Silicon Macs. Unlike Ollama or mlx-lm used directly, it ships continuous batching, paged KV cache, prefix caching, and SSD-tiered cache, and exposes both OpenAI /v1/* and Anthropic /v1/messages from a single process. Run LLMs, vision models, audio, and embeddings on Metal with unified memory, no conversion step.

    Quick start

    New users: follow the isolated release install and first-response check. The example below assumes that environment is activated. Initial model download/loading time depends on the machine and connection.

    The version below is this walkthrough's pinned reference release, not an automatically updated latest version.

    python -m pip install 'vllm-mlx==0.4.1'
vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --host 127.0.0.1 --port 8000

    After the first response, restart with --continuous-batching to try continuous batching and its cache options. The minimal command above uses the default engine without that flag.

    OpenAI SDK:

    Install openai in your Python client environment first; it is not installed by the server package. See the client setup.

    from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
r = client.chat.completions.create(model="mlx-community/Llama-3.2-3B-Instruct-4bit", messages=[{"role": "user", "content": "Hi!"}])
print(r.choices[0].message.content)

    Anthropic SDK / Claude Code:

    export ANTHROPIC_BASE_URL=http://localhost:8000
export ANTHROPIC_API_KEY=not-needed
claude

    Coding CLI compatibility

    Validated with OpenCode, pi, Codex, Claude Code, GitHub Copilot CLI, Cline CLI, and OpenClaw's embedded agent. All seven completed a streamed tool interaction and an exact file edit in one local run with Qwen3.8-27B-4bit on September 19, 2026. Results apply to the tested client versions and settings.

    See the validated CLI matrix and setup guide for versions, API transports, reproduction commands, and coverage limits.

    Features

    APIs

    • OpenAI-compatible: /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/rerank, /v1/responses
    • Anthropic-compatible: /v1/messages (streaming, tool use, system prompts)
    • MCP Tool Calling: 19 parsers (OpenAI, Anthropic, Gemini, Qwen, DeepSeek, Gemma, and more)
    • Structured output: JSON Schema via response_format (lm-format-enforcer)

    Throughput & memory

    • Continuous batching: high throughput for concurrent requests
    • Paged KV cache: memory-efficient with prefix sharing
    • SSD-tiered KV cache: spill prefix cache to disk for long-context agents (--ssd-cache-dir)
    • Warm prompts: preload popular prefixes at startup (--warm-prompts) for 1.3-2.25x TTFT
    • Prefix cache: trie-based, shared across requests

    Multimodal

    • Text + image + video + audio from one server
    • Vision models: Gemma 3, Gemma 4, Qwen3-VL, Pixtral, Llama vision
    • Audio input in chat (audio_url content blocks)
    • Native TTS: 11 voices, 15+ languages (Kokoro, Chatterbox, VibeVoice, VoxCPM)
    • STT: Whisper family with RTF up to 197x on M4 Max

    Reasoning & advanced

    • Reasoning extraction: Qwen3 and DeepSeek-R1 parsers (--reasoning-parser)
    • MoE expert reduction: --moe-top-k for +7-16% on Qwen3-30B-A3B
    • Speculative decoding: --enable-mtp for supported models
    • Sparse prefill: attention-based --specprefill for supported model/draft combinations

    Observability

    • Prometheus metrics: /metrics endpoint with --enable-metrics
    • Built-in benchmarker: vllm-mlx bench-serve for prompt sweeps with CSV/JSON output

    Native GPU acceleration

    • Apple Silicon only (M1, M2, M3, M4, M5) with Metal kernels via MLX
    • Unified memory, no model conversion

    Performance

    LLM decode (M4 Max, 128 GB, greedy, single stream):

    ModelTok/sMemory
    Qwen3-0.6B-8bit417.90.7 GB
    Llama-3.2-3B-Instruct-4bit205.61.8 GB
    Qwen3-30B-A3B-4bit127.7~18 GB

    Audio speech-to-text (M4 Max, RTF = real-time factor):

    ModelRTFUse case
    whisper-tiny197xReal-time / low latency
    whisper-large-v3-turbo55xQuality + speed
    whisper-large-v324xHighest accuracy

    See docs/benchmarks/ for continuous-batching results, KV-cache quantization (4-bit / 8-bit / fp16), and MoE top-k sweeps.

    Examples

    Anthropic API (Claude Code, OpenCode)

    vllm-mlx serve mlx-community/Qwen3-8B-4bit --port 8000
export ANTHROPIC_BASE_URL=http://localhost:8000
export ANTHROPIC_API_KEY=not-needed
claude

    Reasoning models (Qwen3, DeepSeek-R1)

    vllm-mlx serve mlx-community/Qwen3-8B-4bit --reasoning-parser qwen3

    r = client.chat.completions.create(
    model="mlx-community/Qwen3-8B-4bit",
    messages=[{"role": "user", "content": "What is 17 * 23?"}],
)
print("Thinking:", r.choices[0].message.reasoning)
print("Answer:",   r.choices[0].message.content)

    Multimodal (image + text)

    vllm-mlx serve mlx-community/Qwen3-VL-4B-Instruct-3bit --port 8000

    r = client.chat.completions.create(
    model="mlx-community/Qwen3-VL-4B-Instruct-3bit",
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "What is in this image?"},
        {"type": "image_url", "image_url": {"url": "https://example.com/cat.jpg"}},
    ]}],
)

    Structured output (JSON Schema)

    With the text server from the quick start:

    r = client.chat.completions.create(
    model="mlx-community/Llama-3.2-3B-Instruct-4bit",
    messages=[{"role": "user", "content": "List 3 colors."}],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "schema": {"type": "object", "properties": {"colors": {"type": "array", "items": {"type": "string"}}}}
        },
    },
)

    Reranking (/v1/rerank)

    curl http://localhost:8000/v1/rerank -H 'Content-Type: application/json' -d '{
  "model": "default",
  "query": "apple silicon inference",
  "documents": ["MLX is Apples framework", "Metal kernels on M-series", "CUDA on NVIDIA"]
}'

    The built-in MLX reranker forward path supports standard BERT/XLM-RoBERTa sequence-classification weights with gelu, gelu_new/gelu_fast, relu, or silu/swish hidden_act values. Other activations fail explicitly so custom reranker architectures can add a dedicated adapter instead of silently using the wrong activation.

    Embeddings

    vllm-mlx serve <llm-model> --embedding-model mlx-community/all-MiniLM-L6-v2-4bit

    emb = client.embeddings.create(model="mlx-community/all-MiniLM-L6-v2-4bit", input=["Hello", "World"])

    Audio (TTS / STT)

    pip install vllm-mlx[audio]
brew install espeak-ng        # macOS, needed for non-English TTS

python examples/tts_example.py "Hello, how are you?" --play
python examples/tts_multilingual.py "Hola mundo" --lang es --play

    Built-in benchmarking

    vllm-mlx bench-serve --url http://localhost:8000 --concurrency 5 --prompts prompts.txt --output results.csv

# Product-style workload with quality checks and metrics deltas
vllm-mlx bench-serve --url http://localhost:8000 --workload workload.json --repetitions 5 --output results.json

# Append workload rows into SQLite for longitudinal comparisons
vllm-mlx bench-serve --url http://localhost:8000 --workload workload.json --repetitions 5 --format sqlite --output bench.db

    Model acquisition and conversion

    # Inspect repo metadata, file sizes, config, and rough fit before downloading weights
vllm-mlx model inspect mlx-community/Llama-3.2-3B-Instruct-4bit

# Acquire with resumable Hugging Face transfer and write a local artifact manifest
vllm-mlx model acquire mlx-community/Llama-3.2-3B-Instruct-4bit --target-dir ./models/llama-3b-4bit

# Wrap mlx-lm conversion and record the exact recipe in the converted artifact
vllm-mlx model convert meta-llama/Llama-3.2-3B-Instruct --output ./models/llama-3b-mlx-q4 --quantize --q-bits 4 --q-group-size 64 --q-mode affine

    Prometheus metrics

    vllm-mlx serve <model> --enable-metrics
curl http://localhost:8000/metrics

    Installation

    Use the release walkthrough for an isolated environment and a pinned server version. If you already manage CLI tools with uv, its isolated equivalent is:

    uv tool install 'vllm-mlx==0.4.1'

    Keep development checkouts separate. See the Installation Guide for optional extras and Audio Guide for audio setup.

    Documentation

    Browse the complete documentation at vllm-mlx.is-a.dev.

    Architecture

    ┌─────────────────────────────────────────────────────────────────────────┐
│                           vllm-mlx Server                               │
│   OpenAI /v1/*  ·  Anthropic /v1/messages  ·  /v1/rerank  ·  /metrics   │
└─────────────────────────────────────────────────────────────────────────┘
                                   │
                                   ▼
┌─────────────────────────────────────────────────────────────────────────┐
│  Continuous batching · Paged KV cache · Prefix cache · SSD tiering      │
└─────────────────────────────────────────────────────────────────────────┘
                                   │
        ┌─────────────┬────────────┴────────────┬─────────────┐
        ▼             ▼                         ▼             ▼
┌───────────────┐ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐
│    mlx-lm     │ │   mlx-vlm     │ │   mlx-audio   │ │mlx-embeddings │
│    (LLMs)     │ │  (Vision)     │ │  (TTS + STT)  │ │ (Embeddings)  │
└───────────────┘ └───────────────┘ └───────────────┘ └───────────────┘
                                   │
                                   ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                   MLX · Metal kernels · Unified memory                  │
└─────────────────────────────────────────────────────────────────────────┘

    Contributing

    Bug fixes, perf work, docs, and benchmarks on different Apple Silicon chips all welcome. See the Contributing Guide.

    License

    Apache 2.0. See LICENSE.

    Citation

    @software{vllm_mlx2025,
  author = {Barrios, Wayner},
  title  = {vllm-mlx: Apple Silicon MLX Backend for vLLM},
  year   = {2025},
  url    = {https://github.com/waybarrios/vllm-mlx},
  note   = {Native GPU-accelerated LLM and vision-language model inference on Apple Silicon}
}

    Acknowledgments

    • MLX. Apple's ML framework.
    • mlx-lm. LLM inference library.
    • mlx-vlm. Vision-language models.
    • mlx-audio. Text-to-Speech and Speech-to-Text.
    • mlx-embeddings. Text embeddings.
    • Rapid-MLX. Community fork of vllm-mlx.
    • vLLM. High-throughput LLM serving. vllm-mlx is inspired by vLLM and adopts its continuous-batching and paged KV-cache design for Apple Silicon via MLX.

    Star history

    Star History Chart

    If vllm-mlx helped you, please star the repo. It helps more Apple Silicon devs find it.

    相关项目

    同分类里 star 最高的项目
    对比全部 →
    VoiceStudio
    debpalash·Python

    本地运行的 ElevenLabs 开源替代品，支持声音克隆、语音设计、视频配音、听写、转写和有声书制作，覆盖 646 种语言。

    语音克隆+4语音推理进阶
    107215+409112kv0.5.8-beta.42026-11-10 23:08 UTC
    voicebox
    jamiepine·Python

    开源 AI 语音工作室，支持声音克隆、语音听写和语音生成，基于 Qwen3-TTS 与 Whisper，可在 CUDA 或 MLX 上本地运行。

    语音克隆+4语音推理硬核
    60390+6377.6kv0.5.02026-10-07 01:17 UTC
    whisper.cpp
    ggml-org·C++

    OpenAI Whisper 语音识别模型的 C/C++ 移植版，无需 Python 和 GPU 依赖，在普通 CPU 上就能跑语音转文字。

    语音识别+4语音推理进阶
    55073+866.3kv1.9.52026-11-10 12:09 UTC
    faster-whisper
    SYSTRAN·Python

    基于 CTranslate2 重实现的 Whisper 语音转文字推理引擎，通过量化与高效推理大幅提速并降低显存占用，适合本地部署转录。

    语音识别+4语音推理进阶
    26655+832.2kv1.2.12026-11-10 10:13 UTC