登录

speech-swift

soniqoApache-2.0活跃维护

面向 Apple Silicon 的本地语音工具箱，用 MLX 和 CoreML 在 iOS/macOS 上跑语音识别、语音合成、语音转换、VAD 和说话人分离，全程端侧推理。

AI speech toolkit for Apple Silicon — ASR, TTS, speech-to-speech, VAD, and diarization powered by MLX and CoreML

v0.0.29发布于 2026-10-25 08:19 UTC1,269star +3 / 7 天189fork同步于 2026-11-10 23:50 UTC
语音识别语音合成说话人分离端侧推理Apple Silicon
对比 GitHub 官网apple-siliconasrcoremliosmacos

怎么上手

由 README 提炼，执行前请确认
原文 
在 Package.swift 中添加：.package(url: "https://github.com/soniqo/speech-swift", branch: "main")，并按需引入 .product(name: "ParakeetStreamingASR", package: "speech-swift")
硬件要求按 README 说明

仅支持 Apple Silicon（Mac 与 iOS），部分模型走 CoreML 的 Neural Engine，README 未给出具体显存或内存数值

支持 Apple Silicon
下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼
没人工点评？写下你的经验

面向 Apple Silicon 的本地语音工具箱，用 MLX 和 CoreML 在 Mac 和 iOS 上运行 ASR、TTS、VAD 等模型

面向 Apple Silicon 的本地语音工具箱，用 MLX 和 CoreML 在 iOS/macOS 上跑语音识别、语音合成、语音转换、VAD 和说话人分离，全程端侧推理。

核心能力

  • 按模型拆分为独立 SPM 库，可按需引入不付多余体积
  • 用 MLX Swift 与 CoreML 在设备本地推理，无需云端和 API Key
  • 支持流式识别并输出中间结果与句末检测
  • 覆盖 ASR、TTS、说话人分离、VAD、语音增强等能力组
  • 提供 SpeechUI 的 SwiftUI 转写视图与流式适配器

容易踩的坑

模型需通过 fromPretrained 从 HuggingFace 下载，首次运行要联网；各模型是独立 SPM 产品，用哪个引哪个，漏引会编译失败；部分 TTS 模型（如 F5-TTS、Higgs TTS 3）为研究或非商用许可，商用前需确认

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

最近发版

5 个版本v0.0.29
全部 release
  1. v0.0.29最新正式版2026-10-25 · 2026-10-25 08:19 UTC

    New models and runtimes

    • German Kokoro voices: Martin and Victoria, with German text normalization and phonemization. Kokoro also gains sentence-aware streaming chunks, a hard chunk budget and automatic warm-up on load.
    • Breeze TTS 2: native MLX English/Chinese synthesis, voice design, reference cloning and direction, and streamed 24 kHz audio.
    • Pocket TTS: native CoreML English/Alba synthesis with incremental 80 ms PCM frames.
    • MOSS-TTS-Nano: native CoreML 48 kHz stereo synthesis and reference voice cloning.
    • Voxtral TTS: native MLX synthesis with 20 preset voices in nine languages. Experimental Fish Audio Drama 3 support adds native buffered and streaming synthesis.
    • Kyutai STT: native MLX transcription, streaming sessions and CLI support.
    • GLiNER2.5-Decide classification/entity extraction and experimental Clef-flash local text decisions.
    • Gemma 4 12B local chat and tool calling, with repeated-prefix reuse and reduced prompt-prefill work.

    Memory and reliability

    • Bound reusable MLX buffers across model runtimes with coordinated lifetime budgets (#537). The shared default is min(2 GiB, physical RAM / 4); existing model-specific budgets and lower caller limits are preserved. This limits spare buffers rather than total process memory.
    • Fix repeated-request cache growth in Qwen3-ASR, MOSS and VoxCPM2. VoxCPM2's measured process footprint stayed around 13 GB instead of reaching 33.9 GB over eight varying-length lines, with identical audio checksums.
    • Coordinate temporary Chatterbox and Kyutai cache budgets with loaded models, including disabled caching and unload in either order.
    • Fix model-loading graph retention and add shared runtime ownership regression coverage.
    • Preserve typed download failures and expose retry events. Harden local-path validation and correct versioned speaker-model caches.

    APIs and service

    • Offline native inference service with explicitly staged manifests, native tools and result history, incremental PCM16 output, request cancellation and source provenance.
    • JSON Schema response formats, ASR word timestamps and a Nemotron 3 diarization streaming session.
    • Nemotron multilingual language guidance and corrected Gemma tool serialization and failure diagnostics.

    Validation

    PR and main CI passed, along with all 13 nightly E2E groups on the release commit. Nine focused MLX lifecycle/output checks passed locally, including bit-identical seeded OmniVoice synthesis across cache budgets and VoxCPM2 bf16 synthesis.

    SwiftPM users can select from: "0.0.29". The release workflow publishes the macOS arm64 CLI/server archive and updates the Homebrew formula after packaging succeeds.

    Full changelog: https://github.com/soniqo/speech-swift/compare/v0.0.28...v0.0.29

  2. v0.0.282026-09-24 · 2026-09-24 08:46 UTC
    查看更新说明

    Highlights

    • Add NVIDIA Nemotron 3 diarization with Core ML INT8 and MLX INT8 backends for Apple Silicon.
    • Add Qwen3-TTS 1.7B Core ML runtime and export support.
    • Improve Qwen3-ASR cancellation and Core ML embedding handling.
    • Improve model downloads with background transfers and live progress reporting.
    • Fix Metal build recovery and pin compatible Metal targets.

    Install or upgrade with brew install soniqo/tap/speech or brew update && brew upgrade soniqo/tap/speech after the Homebrew formula updates.

  3. v0.0.272026-09-02 · 2026-09-02 17:56 UTC
    查看更新说明

    Adds Smart Turn v3.2 end-of-turn detection.

    A VAD pause is no longer the end of the turn. SmartTurnModel (Core ML, 8M parameters, 23 languages) listens to the last eight seconds of the user's speech and returns the probability that they have finished. StreamingVADProcessor and VoicePipeline can consult it on every confirmed pause: a finished sentence ends the turn immediately, a mid-sentence pause keeps the agent listening, speech that resumes continues the same turn, and a configurable silence cap (maxSilenceDuration / turnCompletionMaxSilence, 2 s) ends the turn regardless. Eager STT respects the veto, and a classifier that throws fails open so a broken model never stalls a conversation.

    • SmartTurnModel.fromPretrained() loads aufklarer/Smart-Turn-v3.2-CoreML (17 MB); the compiled bundle embeds the Whisper log-mel front-end, so callers pass raw 16 kHz audio. One window takes about 3.5 ms on Apple Silicon; on 1,000 upstream test clips the bundle matches the upstream fp32 model exactly (92.9 %).
    • TurnCompletionProvider protocol in AudioCommon, mirroring speech-core's sc_turn_completion_vtable_t.
    • StreamingVADProcessor(model:config:turnCompletion:turnCompletionConfig:) and init(provider:) over any StreamingVADProvider.
    • VoicePipeline.setTurnCompletion(_:) with PipelineConfig.turnCompletionThreshold and turnCompletionMaxSilence; SpeechCore.xcframework updated to speech-core v0.0.14, which adds the underlying C API.
    • CLI: speech turn <wav> prints the completion probability; speech vad-stream --smart-turn merges segments across mid-sentence pauses.
    • SpeechDemo gets a Smart Turn toggle with per-pause probabilities; iOSEchoDemo attaches the model by default.

    Docs: https://soniqo.audio/guides/turn

  4. v0.0.262026-08-17 · 2026-08-17 08:00 UTC
    查看更新说明

    Fixes two defects in the full-duplex VoiceChat CLI.

    --no-aec captured no audio and hung. The playback graph was attached after engine.start(), an ordering only AUVoiceProcessingIO tolerates because it is a combined I/O unit that pulls audio regardless. With voice processing disabled the input tap never fired, so the session reported frames 0 and never exited, since frame-driven limits like --max-seconds only advance once frames arrive. The playback graph is now attached before starting when AEC is off; the enabled path is unchanged.

    Tool calls were unbounded per user turn. A checkpoint that fell into a repetition pattern could issue dozens of successful calls in a row — one observed session searched reminders for "8 PM", "9 PM", and onward past "36 PM" before crashing. Repeat protection was write-only and the function-start gate arms only on failures, so nothing bounded successful reads. Every tool is now limited to five calls per accepted user turn, resetting on fresh acoustic activity, and returns a structured call_budget_exhausted result beyond it.

  5. v0.0.252026-08-16 · 2026-08-16 09:22 UTC
    查看更新说明

    Highlights

    • Adds speech voice-chat, a local full-duplex Soniqo VoiceChat demo for Nemotron VoiceChat 11B INT5 with native MCP tool calling, Apple Reminders example configuration, human-readable real-time metrics, and optional debug timelines.
    • Includes the VoiceChat turn-taking, overload recovery, asynchronous tool execution, audio-continuity, and regression coverage developed for the live CLI demo.
    • Pins matching stable MLX dependencies and hardens clean release/Homebrew builds against moving upstream branches and stale SwiftPM caches.

    The v0.0.24 binary packaging attempt was superseded by this release.

    What's Changed

用户点评

登录用户的真实使用记录，可按版本和硬件筛选
共 0 条

    还没有人点评。第一个说出真实体验的人最有价值。

    README 原文

    未翻译
    在 GitHub 打开

    Speech Swift

    AI speech models for Apple Silicon, powered by MLX Swift and CoreML.

    📖 Read in: English · 中文 · 日本語 · 한국어 · Español · Deutsch · Français · हिन्दी · Português · Русский · العربية · Tiếng Việt · Türkçe · ไทย

    On-device speech recognition, synthesis, and understanding for Mac and iOS. Runs locally on Apple Silicon — no cloud, no API keys, no data leaves your device.

    📚 Full Documentation → · 🤗 HuggingFace Models · 📝 Blog · 💬 Discord

    Use cases: Voice Agents · Transcription · Speech Generation

    Built with Speech Swift

    16 public repositories with verifiable Speech Swift package references.

    AnythingLLM · Palmier Pro · Anarlog · ClawdHome · Jabber · Ora · VoxFlow · LokalBot · Voicey · HushType · DexDictate macOS · Watchtower · Wishper App · FriSpeak · Scribe · VoicePen

    Capability groups: STT / ASR · Alignment · TTS · LLMs & translation · Speech-to-speech · Enhancement/restoration · Source separation · Music/audio generation · Wake word, VAD, diarization & speaker identity

    STT / ASR

    • Qwen3-ASR — Speech-to-text (automatic speech recognition, 52 languages, MLX + CoreML)
    • WhisperASR — Whisper Large-v3 Turbo speech-to-text via native CoreML runtime (ANE, multilingual)
    • MOSS Transcribe Diarize — Native CoreML/MLX offline transcription with model-generated speaker labels and timestamps (128K MLX context; INT5/INT8)
    • Parakeet TDT — Speech-to-text via CoreML (Neural Engine, NVIDIA FastConformer + TDT decoder, 25 languages)
    • Omnilingual ASR — Speech-to-text (Meta wav2vec2 + CTC, 1,672 languages across 32 scripts, CoreML 300M + MLX 300M/1B/3B/7B) — 0.28 RTF on iPhone 16 Pro
    • Cohere Transcribe 2B — Native MLX speech-to-text (14 languages, FP16/INT5/INT8)
    • Kyutai STT — Native MLX streaming speech recognition (1B English/French; 2.6B English configuration; 24 kHz Mimi; FP16/INT4/INT8 exports) · INT8 (default) · INT4 · FP16
    • Voxtral Mini 3B — Native MLX speech-to-text (8 languages, FP16/INT5/INT8) — 0.074 RTF on M5 Pro
    • Streaming Dictation — Real-time dictation with partials and end-of-utterance detection (Parakeet-EOU-120M) — 0.04 RTF on iPhone 16 Pro
    • Nemotron Streaming (Multilingual) — Low-latency streaming ASR with native punctuation and capitalization (NVIDIA Nemotron-3.5-ASR-Streaming-0.6B, CoreML + MLX, 40 language-locales)
    • Nemotron Streaming (English) — Low-latency streaming ASR with native punctuation and capitalization (NVIDIA Nemotron-Speech-Streaming-0.6B, CoreML, English-only, smaller and faster than the multilingual variant)

    Alignment

    TTS / Speech Generation

    • Qwen3-TTS — Text-to-speech (highest quality, streaming, custom speakers, 10 languages)
    • CosyVoice TTS — Streaming TTS with voice cloning, multi-speaker dialogue, emotion tags (9 languages)
    • VoxCPM2 — 48 kHz studio-quality TTS with voice cloning + instruction-driven voice design (2B, MLX bf16/int8, 30 languages)
    • IndexTTS2 — Native MLX voice cloning from a reference voice (IndexTeam IndexTTS-2, 1.5B-class fp16 bundle, speaker/emotion/pause controls)
    • F5-TTS — Zero-shot voice cloning from a short reference clip + transcript (SWivid F5-TTS v1 Base, DiT flow matching + Vocos, MLX fp16, 24 kHz, English + Mandarin; non-commercial license)
    • Higgs TTS 3 — Conversational TTS with zero-shot voice cloning and inline emotion/style/SFX/prosody tags (Boson Higgs TTS 3, Qwen3-4B backbone, MLX bf16, 24 kHz, 100+ languages; research/non-commercial license)
    • Breeze TTS 2 — English/Chinese TTS with voice design, reference cloning and direction, and streamed audio (MLX BF16/8-bit, 24 kHz; research/non-commercial license)
    • FireRedTTS3-Instruct — Experimental English/Chinese voice design from a script and instruction; native MLX FP32, 24 kHz, complete-audio output. Requires a compatible exported bundle.
    • MOSS-VoiceGenerator — Experimental voice design from a script and instruction; native MLX BF16 with an FP32 codec, 24 kHz streaming. Requires a compatible exported bundle.
    • Kokoro TTS — On-device TTS (82M, CoreML/Neural Engine, 54 voices, iOS-ready, 10 languages + two German voice packs) — 0.08 RTF on iPhone 16 Pro
    • VibeVoice TTS — Long-form / multi-speaker TTS (Microsoft VibeVoice Realtime-0.5B + 1.5B, MLX, up to 90-min podcast/audiobook synthesis, EN/ZH)
    • Magpie TTS — Multilingual TTS (NVIDIA Magpie-TTS Multilingual 357M, MLX INT8 411 MB or CoreML INT8 342 MB, 9 languages, 5 baked speakers, streaming on MLX)
    • MOSS-TTS-Nano — Streaming TTS and voice cloning without a transcript (CoreML, 48 kHz stereo, 80 ms chunks, 367.6 MB compiled graphs; macOS 15+ / iOS 18+)
    • Supertonic TTS — On-device flow-matching TTS (Supertone Supertonic-3 99M, CoreML/Neural Engine, 31 languages, 10 voices, G2P-free, 44.1 kHz) — 0.15 RTF on iPhone 16 Pro
    • Chatterbox TTS — Multilingual TTS with zero-shot voice cloning (Resemble AI Chatterbox Multilingual, MLX fp16 ~1.3 GB, 23 runtime languages; Hebrew requires niqqud, MIT) plus Chatterbox Flash Core ML text-to-waveform from precomputed reference conditioning.
    • OmniVoice TTS — Non-autoregressive diffusion TTS with zero-shot voice cloning (k2-fsa OmniVoice, Qwen3 backbone, MLX fp16 default / int8 available, 600+ languages, Apache-2.0)
    • Indic-Mio — Hindi/Indic TTS with inline emotion markers and optional reference-voice cloning (MLX, 24 kHz)
    • CSM (Conversational Speech Model) — Conversational text-to-speech with zero-shot voice cloning from a reference clip + transcript (Sesame CSM-1B, Llama-1B backbone + Mimi codec, MLX int8/fp16, 24 kHz, English; Apache-2.0)

    LLMs & Translation

    • Qwen3Chat — On-device LLM chat (Qwen3.5-0.8B MLX/CoreML plus dense Qwen3 4B and Gemma 4 E2B/E4B MLX backends, streaming tokens)
    • FunctionGemma — On-device LLM for structured function / tool calls (Gemma 3 270M, CoreML 8-bit palettized, Neural Engine, ~252 tok/s on M5 Pro · 128 tok/s on iPhone 16 Pro)
    • MADLAD-400 — Many-to-many translation across 400+ languages (3B, MLX INT4 + INT8, T5 v1.1, Apache 2.0)
    • GLiNER2.5-Decide - Local schema-conditioned text classification and entity spans (DeBERTa-v3-large encoder, MLX, INT8/FP16/FP32; development preview, Apache 2.0)
    • Clef-flash — Local decisions from text and allowed answers (9B, MLX 4-bit; development preview, Apache 2.0)

    Speech-to-Speech & Voice Agents

    • Hibiki Zero-3B — Streaming speech-to-speech translation (FR/ES/PT/DE → EN, MLX INT4 + INT8, Kyutai Moshi/Mimi stack, CC-BY-4.0)
    • PersonaPlex — Full-duplex speech-to-speech (7B, audio in → audio out, 18 voice presets)
    • VoiceChat 11B — Native MLX duplex speech-to-speech with continuous speech input, text/function channels, direct EAR-TTS speech output, and a neural codec (INT5/INT8)
    • Audio2Face-3D — Speech-driven facial animation for avatars (NVIDIA Audio2Face-3D v2.3 Mark, 301 facial coefficients, MLX)

    Enhancement, Separation & Audio Generation

    • DeepFilterNet3 — Real-time noise suppression (2.1M params, 48 kHz) on Core ML (Neural Engine) or MLX (GPU, fp32). Long-form audio above the 60 s single-shot cap is auto-chunked with crossfade — see enhanceChunked(...)
    • LocalVQE v1.4-AEC — Streaming acoustic echo cancellation from separate, synchronized microphone and playback-reference streams (Core ML + native adaptive filter, 16 kHz, 16 ms algorithmic latency)
    • Source Separation — Music source separation via HTDemucs (Demucs v4) + Open-Unmix (UMX-HQ / UMX-L, 4 stems: vocals/drums/bass/other, 44.1 kHz stereo)
    • MAGNeT — Text-to-music generation (Meta MAGNeT Small 300M / Medium 1.5B, MLX INT8, 30 s clips at 32 kHz mono, masked parallel decoding)
    • Stable Audio 3 — Text-to-audio/music generation (Stable Audio 3 Medium, MLX INT8/INT4, 44.1 kHz stereo, variable length)
    • FlashSR — Audio super-resolution (FlashSR ICASSP 2025, MLX, 48 kHz mono, 1-step distilled diffusion, INT4 363 MB / INT8 720 MB)

    Turn Detection, Diarization & Speaker Identity

    • Wake-word — On-device keyword spotting (KWS Zipformer 3M, CoreML, 26× real-time, configurable keyword list)
    • VAD — Voice activity detection (Silero streaming, Pyannote offline, FireRedVAD 100+ languages)
    • Smart Turn — End-of-turn detection from audio (Pipecat Smart Turn v3.2, CoreML, 8M, 23 languages; confirms Silero VAD pauses so a mid-sentence pause does not end the turn)
    • Speaker Diarization — Who spoke when (Pyannote pipeline, Sortformer end-to-end on Neural Engine) — now with an incremental streaming session (stable speaker IDs, updates every 480 ms)
    • Speaker Embeddings — WeSpeaker ResNet34 (256-dim), ReDimNet2-B6 named identity (192-dim), CAM++ (192-dim)

    Papers: Qwen3-ASR (Alibaba) · Qwen3-TTS (Alibaba) · Omnilingual ASR (Meta) · Parakeet TDT (NVIDIA) · CosyVoice 3 (Alibaba) · Kokoro (StyleTTS 2) · PersonaPlex (NVIDIA) · Mimi (Kyutai) · Hibiki (Kyutai) · Sortformer (NVIDIA)

    On-device iPhone 16 Pro CoreML benchmarks (RTF, tokens/s, peak memory): docs/benchmarks/ios-coreml.md.

    News

    Quick start

    Add the package to your Package.swift:

    .package(url: "https://github.com/soniqo/speech-swift", branch: "main")

    Import only the modules you need — every model is its own SPM library, so you don't pay for what you don't use:

    .product(name: "ParakeetStreamingASR", package: "speech-swift"),
.product(name: "SpeechUI",             package: "speech-swift"),  // optional SwiftUI views

    Transcribe an audio buffer in 3 lines:

    import ParakeetStreamingASR

let model = try await ParakeetStreamingASRModel.fromPretrained()
let text = try model.transcribeAudio(audioSamples, sampleRate: 16000)

    Live streaming with partials:

    for await partial in model.transcribeStream(audio: samples, sampleRate: 16000) {
    print(partial.isFinal ? "FINAL: \(partial.text)" : "... \(partial.text)")
}

    SwiftUI dictation view in ~10 lines:

    import SwiftUI
import ParakeetStreamingASR
import SpeechUI

@MainActor
struct DictateView: View {
    @State private var store = TranscriptionStore()

    var body: some View {
        TranscriptionView(finals: store.finalLines, currentPartial: store.currentPartial)
            .task {
                let model = try? await ParakeetStreamingASRModel.fromPretrained()
                guard let model else { return }
                for await p in model.transcribeStream(audio: samples, sampleRate: 16000) {
                    store.apply(text: p.text, isFinal: p.isFinal)
                }
            }
    }
}

    SpeechUI ships only TranscriptionView (finals + partials) and TranscriptionStore (streaming ASR adapter). Use AVFoundation for audio visualization and playback.

    Available SPM products: Qwen3ASR, WhisperASR, MossTranscribe, Qwen3TTS, Qwen3TTSCoreML, ParakeetASR, ParakeetStreamingASR, NemotronStreamingASR, OmnilingualASR, CohereTranscribeASR, KyutaiSTT, VoxtralASR, KokoroTTS, SupertonicTTS, VibeVoiceTTS, CosyVoiceTTS, VoxCPM2TTS, IndexTTS2TTS, F5TTS, HiggsTTS, BreezeTTS, FireRedTTS3TTS, MossVoiceGeneratorTTS, ChatterboxTTS, OmniVoiceTTS, IndicMioTTS, FishAudioTTS, MagpieTTS, MagpieTTSCoreML, MAGNeTMusicGen, StableAudio3MusicGen, FlashSR, PersonaPlex, VoiceChat, CSM, Audio2Face3D, HibikiTranslate, MADLADTranslation, GLiNER, Clef, SpeechVAD, SpeechLanguageID, SpeechWakeWord, SpeechEnhancement, SpeechRestoration, SourceSeparation, Qwen3Chat, FunctionGemma, SpeechCore, SpeechUI, AudioCommon.

    Models

    Compact view below. Full model catalogue with sizes, quantisations, download URLs, and memory tables → soniqo.audio/architecture.

    ModelTaskBackendsSizesLanguages
    Qwen3-ASRSpeech → TextMLX, CoreML (hybrid)0.6B, 1.7B52
    WhisperASRSpeech → TextCoreML (ANE)Large-v3 TurboMulti
    MOSS Transcribe DiarizeSpeech → Text + speaker timestampsCoreML / MLX0.9B (MLX INT5/INT8; CoreML INT8/FP16)Multi
    Parakeet TDTSpeech → TextCoreML (ANE)0.6B25 European
    Parakeet EOUSpeech → Text (streaming)CoreML (ANE)120M25 European
    Nemotron Streaming (Multilingual)Speech → Text (streaming, punctuated)CoreML (ANE), MLX0.6B40
    Nemotron Streaming (English)Speech → Text (streaming, punctuated)CoreML (ANE)0.6BEN
    Omnilingual ASRSpeech → TextCoreML (ANE), MLX300M / 1B / 3B / 7B1,672
    Cohere Transcribe 2BSpeech → TextMLX2B (FP16 / INT5 / INT8)14
    Kyutai STTSpeech → Text (streaming)MLX1B (FP16 / INT4 / INT8)EN/FR
    Voxtral Mini 3BSpeech → TextMLX3B (FP16 / INT5 / INT8)8
    Qwen3-ForcedAlignerAudio + Text → TimestampsMLX, CoreML0.6BMulti
    Qwen3-TTSText → SpeechMLX, CoreML0.6B, 1.7B10
    CosyVoice3Text → SpeechMLX0.5B9
    VoxCPM2Text → Speech (48 kHz, voice design + cloning)MLX2B (bf16/int8)30
    IndexTTS2Text → Speech (zero-shot voice cloning)MLX1.5B-class (fp16)EN/ZH
    F5-TTSText → Speech (zero-shot voice cloning)MLX336M (fp16)EN/ZH
    Higgs TTS 3Text → Speech (conversational, zero-shot voice cloning)MLX4B (bf16)100+
    Breeze TTS 2Text → SpeechMLXBF16 / INT8EN/ZH
    FireRedTTS3-InstructText → SpeechMLXFP32EN/ZH
    MOSS-VoiceGeneratorText → SpeechMLXBF16 + FP32 codecEN/ZH
    Kokoro-82MText → SpeechCoreML (ANE)82M10
    Supertonic-3Text → Speech (44.1 kHz, flow-matching, G2P-free)CoreML (ANE)99M31
    VibeVoice Realtime-0.5BText → Speech (long-form, multi-speaker)MLX0.5BEN/ZH
    VibeVoice 1.5BText → Speech (up to 90-min podcast)MLX1.5BEN/ZH
    Magpie-TTS MultilingualText → Speech (5 baked speakers, streaming)MLX / CoreML357M (MLX INT8, CoreML INT8)9 (CoreML excludes JA)
    Chatterbox MultilingualText → Speech (zero-shot cloning)MLX0.8B (fp16)23 (HE requires niqqud)
    Chatterbox FlashText → Speech (voice cloning with external reference conditioning)CoreML + MLX conditioning bridge0.8B (fp16 Core ML)EN
    OmniVoiceText → Speech (NAR diffusion, zero-shot cloning)MLX0.8B (fp16 default / int8)600+
    Indic-MioText → Speech (Hindi/Indic, emotion tags, voice cloning)MLXfp16Hindi / Indic
    Fish Audio S2 ProText → Speech (zero-shot cloning, explicit style markers)MLX0.5B-class (fp16)Multilingual
    CSMText → Speech (conversational, zero-shot voice cloning)MLX1B (int8 / fp16)EN
    Qwen3.5 ChatText → Text (LLM)MLX, CoreML0.8BMulti
    Qwen3 Dense ChatText → Text (LLM)MLX4BMulti
    Gemma 4 ChatText → Text (LLM)MLXE2B / E4B (4-bit)Multi
    FunctionGemmaText → Tool calls (LLM)CoreML270MEN-tuned
    MADLAD-400Text → Text (Translation)MLX3B400+
    GLiNER2.5-DecideText → Labels + spans (classification, entity extraction)MLX340MEN
    Clef-flashText → Probabilities for allowed answersMLX9B (4-bit)EN-tested
    Hibiki Zero-3BSpeech → Speech (Translation)MLX3BFR/ES/PT/DE → EN
    PersonaPlexSpeech → SpeechMLX7BEN
    VoiceChat 11BSpeech → Speech + TextMLX11B (INT5 / INT8)EN
    Audio2Face-3DSpeech → Facial animationMLXv2.3 MarkAgnostic
    Silero VADVoice Activity DetectionMLX, CoreML309KAgnostic
    Smart Turn v3.2End-of-Turn DetectionCoreML8M23 languages
    p99 turn-1-miniStreaming turn completionCoreML5.3MEnglish
    VAP (otoSpeech)Two-channel speech forecastsCoreML5.8MEnglish
    KWS ZipformerAudio → Wake wordCoreML (ANE)3MEN/custom keywords
    PyannoteVAD + DiarizationMLX1.5MAgnostic
    Pyannote Community-1Diarization + speaker embeddingsCoreML (ANE) + Swift VBx8.35MAgnostic
    SortformerDiarization (E2E), incremental streamingCoreML (ANE)117MAgnostic
    Nemotron 3Diarization (E2E, up to 8 speakers)CoreML / MLX (INT8)100MAgnostic
    Ultra-Sortformer 8spkDiarization (E2E, up to 8 speakers, experimental)CoreML (ANE)117MAgnostic
    DeepFilterNet3Speech EnhancementCoreML / MLX2.1MAgnostic
    LocalVQE v1.4-AECAcoustic Echo CancellationCoreML + C++200K + 2,742Agnostic
    SidonSpeech Restoration (denoise + dereverb, 48 kHz)CoreMLw2v-BERT 2.0 + DAC (fp16/int8)Agnostic
    HTDemucs (Demucs v4)Source SeparationMLX168MAgnostic
    Open-UnmixSource SeparationMLX8.6MAgnostic
    MAGNeTText → Music (30s @ 32 kHz)MLX300M / 1.5B (int4/int8)EN prompts
    Stable Audio 3Text → Music/audio (44.1 kHz stereo)MLXMedium 1.4B (int4/int8)EN prompts
    FlashSRAudio super-resolution (48 kHz)MLX363 MB / 720 MB (int4/int8)Agnostic
    WeSpeakerSpeaker EmbeddingMLX, CoreML6.6MAgnostic
    SpeechBrain ECAPA VoxLingua107Language IdentificationMLX, CoreML21.25M107 languages
    ReDimNet2-B6Named Voice IdentityCoreML12.3MAgnostic

    Installation

    Homebrew

    Requires native ARM Homebrew (/opt/homebrew). Rosetta/x86_64 Homebrew is not supported.

    brew install speech

    Then:

    speech transcribe recording.wav
speech transcribe recording.wav --engine cohere
speech transcribe recording.wav --engine voxtral
speech transcribe recording.wav --engine moss
speech transcribe meeting.wav --engine moss --backend mlx
speech speak "Hello world"
speech csm "Nice to meet you" --ref-audio voice.wav --ref-text "reference transcript"
speech translate "Hello, how are you?" --to es
speech gliner classify "Remind me to call Dad at six PM." --labels create_reminder,send_message,other
speech clef decide request.json
speech respond --input question.wav --transcript
speech voice-chat
speech-server --port 8080            # local HTTP / WebSocket server (OpenAI-compatible /v1/realtime + /v1/audio/transcriptions)

    Full CLI reference →

    Swift Package Manager

    dependencies: [
    .package(url: "https://github.com/soniqo/speech-swift", branch: "main")
]

    Import only what you need — every model is its own SPM target:

    import Qwen3ASR             // Speech recognition (MLX)
import WhisperASR           // Whisper Large-v3 Turbo (CoreML)
import MossTranscribe       // MOSS transcription with timestamps + speaker labels (CoreML + MLX)
import ParakeetASR          // Speech recognition (CoreML, batch)
import ParakeetStreamingASR // Streaming dictation with partials + EOU
import NemotronStreamingASR // Multilingual streaming ASR with native punctuation (0.6B, 40 langs)
import OmnilingualASR       // 1,672 languages (CoreML + MLX)
import CohereTranscribeASR  // Cohere Transcribe 2B (MLX, 14 languages)
import VoxtralASR           // Voxtral Mini 3B (MLX, 8 languages)
import Qwen3TTS             // Text-to-speech
import CosyVoiceTTS         // Text-to-speech with voice cloning
import VoxCPM2TTS           // 48 kHz TTS with voice cloning + voice design (2B)
import IndexTTS2TTS         // Native MLX voice cloning from reference audio
import F5TTS                // Zero-shot voice cloning (DiT flow matching + Vocos)
import HiggsTTS             // Conversational TTS + cloning (Qwen3 backbone, control tags)
import FireRedTTS3TTS       // FireRedTTS3-Instruct (experimental, MLX FP32)
import MossVoiceGeneratorTTS // MOSS-VoiceGenerator (experimental, MLX BF16 + FP32 codec)
import CSM                  // Conversational Speech Model — text→audio + voice cloning (Sesame CSM-1B, MLX)
import KokoroTTS            // Text-to-speech (iOS-ready)
import VibeVoiceTTS         // Long-form / multi-speaker TTS (EN/ZH)
import MagpieTTS            // Multilingual TTS (NVIDIA Magpie 357M, MLX, 9 langs)
import MagpieTTSCoreML      // Magpie CoreML backend (hybrid CoreML + MLX, 8 langs)
import FishAudioTTS         // Experimental Fish Audio S2 Pro runtime with voice cloning
import IndicMioTTS          // Hindi/Indic TTS with emotion markers
import Qwen3Chat            // On-device LLM chat
import FunctionGemma    // On-device tool-call LLM
import Clef                // Local decisions from text
import MADLADTranslation    // Many-to-many translation across 400+ languages
import HibikiTranslate      // Streaming speech-to-speech translation (FR/ES/PT/DE → EN)
import PersonaPlex          // Full-duplex speech-to-speech
import SpeechVAD            // VAD + speaker diarization + embeddings
import SpeechWakeWord       // Wake-word / keyword spotting
import SpeechEnhancement    // Noise suppression
import SpeechRestoration    // Speech restoration — denoise + dereverb (Sidon, CoreML, 48 kHz)
import SourceSeparation     // Music source separation (Open-Unmix, 4 stems)
import StableAudio3MusicGen // Text-to-audio/music generation (Stable Audio 3)
import SpeechUI             // SwiftUI components for streaming transcripts
import AudioCommon          // Shared protocols and utilities

    Requirements

    • Swift 6+, Xcode 16+ (with Metal Toolchain)
    • macOS 15+ (Sequoia) or iOS 18+, Apple Silicon (M1/M2/M3/M4)

    The macOS 15 / iOS 18 minimum comes from MLState — Apple's persistent ANE state API used by the CoreML pipelines (Qwen3-ASR, Qwen3-Chat, Qwen3-TTS) to keep KV caches resident on the Neural Engine across token steps.

    Build from source

    git clone https://github.com/soniqo/speech-swift
cd speech-swift
make build

    make build compiles the Swift package and the MLX Metal shader library. The Metal library is required for GPU inference — without it you'll see Failed to load the default metallib at runtime. make debug for debug builds, make test for the test suite.

    Full build and install guide →

    Demo apps

    • DictateDemo (docs) — macOS menu-bar streaming dictation with live partials, VAD-driven end-of-utterance detection, and one-click copy. Runs as a background agent (Parakeet-EOU-120M + Silero VAD).
    • iOSEchoDemo — iOS echo demo (Parakeet ASR + Kokoro TTS). Device and simulator.
    • PersonaPlexDemo — Conversational voice assistant with mic input, VAD, and multi-turn context. macOS. RTF ~0.94 on M2 Max (faster than real-time).
    • Soniqo VoiceChat CLI (guide) — Native full-duplex Nemotron 11B terminal assistant with live RNN-T captions, model-driven turn-taking, adaptive EAR-TTS detail, and optional MCP tools. The included Apple Reminders configuration exposes only create, list, and update operations.
    • SpeechDemo — Dictation and TTS synthesis in a tabbed interface. macOS.

    Run the VoiceChat reminders demo after a release build:

    ./.build/release/speech voice-chat \
  --model /path/to/voicechat-mlx-int5 \
  --mcp-config Examples/VoiceChatMCP/apple-reminders.json

    Add --debug-timeline for phrase, generated-pronunciation-end, and model-decoded tool lifecycle timestamps. It can reveal tool arguments, so keep it out of shared logs. See each app's README or the linked VoiceChat guide for build and runtime details.

    Code examples

    The snippets below show the minimal path for each domain. Every section links to a full guide on soniqo.audio with configuration options, multiple backends, streaming patterns, and CLI recipes.

    Speech-to-Text — full guide →

    import Qwen3ASR

let model = try await Qwen3ASRModel.fromPretrained()
let text = model.transcribe(audio: audioSamples, sampleRate: 16000)

    Alternative backends: WhisperASR (Whisper Large-v3 Turbo, native CoreML), Parakeet TDT (CoreML, 32× realtime), Omnilingual ASR (1,672 languages, CoreML or MLX), Streaming dictation (live partials).

    Forced Alignment — full guide →

    import Qwen3ASR

let aligner = try await Qwen3ForcedAligner.fromPretrained()
let aligned = aligner.align(
    audio: audioSamples,
    text: "Can you guarantee that the replacement part will be shipped tomorrow?",
    sampleRate: 24000
)
for word in aligned {
    print("[\(word.startTime)s - \(word.endTime)s] \(word.text)")
}

    Text-to-Speech — full guide →

    import Qwen3TTS
import AudioCommon

let model = try await Qwen3TTSModel.fromPretrained()
let audio = model.synthesize(text: "Hello world", language: "english")
try WAVWriter.write(samples: audio, sampleRate: 24000, to: outputURL)

    Alternative TTS engines: CosyVoice3 (streaming + voice cloning + emotion tags), Kokoro-82M (iOS-ready, 54 voices), VibeVoice (long-form podcast / multi-speaker, EN/ZH), Fish Audio S2 Pro (experimental zero-shot cloning + bracket style markers), Voice cloning.

    Speech-to-Speech — full guide →

    import PersonaPlex

let model = try await PersonaPlexModel.fromPretrained()
let responseAudio = model.respond(userAudio: userSamples)
// 24 kHz mono Float32 output ready for playback

    LLM Chat — full guide →

    import Qwen3Chat
import FunctionGemma

let chat = try await Qwen35MLXChat.fromPretrained()
chat.chat(messages: [(.user, "Explain MLX in one sentence")]) { token, isFinal in
    print(token, terminator: "")
}

    Local decisions with Clef-flash

    Give Clef some text and a list of answers. It scores each answer so your app can choose the next step. Supports choices, true/false questions, and ordered scores. Add the Clef SPM product to use the library.

    import Clef

let model = try await Clef.fromPretrained()
let result = try model.decide(
    state: "Please turn the kitchen lights on.",
    questions: [
        ClefQuestion(id: "action", instructions: "Which action was requested?",
            kind: .choice(["on": "Turn lights on", "off": "Turn lights off", "other": "Other request"]))
    ])
print(result.decisions[0].selectedOption)
print(result.decisions[0].probabilities)

    Requires Apple Silicon and macOS 15+. The first load downloads about 5.3 GB; runtime memory is additional. Keep one instance loaded and use it serially.

    Model · Usage · Measured latency and memory

    Translation — full guide →

    import MADLADTranslation

let translator = try await MADLADTranslator.fromPretrained()
let es = try translator.translate("Hello, how are you?", to: "es")
// → "Hola, ¿cómo estás?"

    Speech Translation — full guide →

    import HibikiTranslate
import AudioCommon

let model = try await HibikiTranslateModel.fromPretrained()
let pcm = try AudioFileLoader.load(url: input, targetSampleRate: 24000)
let (englishAudio, textTokens) = model.translate(
    sourceAudio: pcm, sourceLanguage: .fr
)
// Hibiki Zero-3B — FR/ES/PT/DE → EN, on-device, streaming Mimi codec

    Voice Activity Detection — full guide →

    import SpeechVAD

let vad = try await SileroVADModel.fromPretrained()
let segments = vad.detectSpeech(audio: samples, sampleRate: 16000)
for s in segments { print("\(s.startTime)s → \(s.endTime)s") }

    Speaker Diarization — full guide →

    import SpeechVAD

let diarizer = try await DiarizationPipeline.fromPretrained()
let segments = diarizer.diarize(audio: samples, sampleRate: 16000)
for s in segments { print("Speaker \(s.speakerId): \(s.startTime)s - \(s.endTime)s") }

    Speech Enhancement — full guide →

    import SpeechEnhancement

let denoiser = try await SpeechEnhancer.fromPretrained()             // CoreML (Neural Engine)
// let denoiser = try await SpeechEnhancer.fromPretrained(engine: .mlx)  // MLX (GPU, fp32)
let clean = try denoiser.enhance(audio: noisySamples, sampleRate: 48000)

    Acoustic Echo Cancellation — full guide →

    import SpeechEnhancement

let aec = try await LocalVQEEchoCanceller.fromPretrained()
let cleanMicrophone = try aec.processFrame(
    microphone: microphoneFrame,
    reference: playbackReferenceFrame
)

    Speech Restoration — full guide →

    Joint denoise and dereverb with Sidon (w2v-BERT 2.0 predictor + DAC vocoder, Core ML). Unlike a generic noise suppressor, Sidon is trained to preserve speaker identity, so it is well suited to cleaning a noisy or reverberant voice-cloning reference before TTS. Input is 16 kHz; output is 48 kHz mono.

    import SpeechRestoration

let restorer = try await SpeechRestorer.fromPretrained()          // .fp16 (default) or .int8
let clean = try restorer.restore(audio: noisySamples, sampleRate: 16000)  // → 48 kHz

    From the CLI:

    speech restore noisy.wav -o clean.wav            # denoise + dereverb, 48 kHz output
speech restore noisy.wav --variant int8          # smaller, lower peak RAM

# Clean a voice-cloning reference before TTS (opt-in; preserves speaker identity):
speech speak "Hello" --engine voxcpm2 --voice-sample ref.wav --clean-reference

    Voice Pipeline (ASR → LLM → TTS) — full guide →

    import SpeechCore

let pipeline = VoicePipeline(
    stt: parakeetASR,
    tts: qwen3TTS,
    vad: sileroVAD,
    config: .init(mode: .voicePipeline),
    onEvent: { event in print(event) }
)
pipeline.start()
pipeline.pushAudio(micSamples)

    VoicePipeline is the real-time voice-agent state machine (powered by speech-core) with VAD-driven turn detection, interruption handling, and eager STT. It connects any SpeechRecognitionModel + SpeechGenerationModel + StreamingVADProvider.

    HTTP API server

    speech-server --port 8080

    Exposes every model via HTTP REST + WebSocket endpoints, including OpenAI-compatible APIs: a Realtime WebSocket at /v1/realtime and a transcription REST endpoint at /v1/audio/transcriptions. Realtime sessions use explicit input_audio_buffer.commit by default; send session.update with turn_detection.type set to server_vad for Silero-based automatic speech start/end detection, prefix padding, and bounded turn length. See Sources/AudioServer/ and Shared Protocols.

    Papers

    Architecture

    speech-swift is split into one SPM target per model so consumers only pay for what they import. Shared infrastructure lives in AudioCommon (protocols, audio I/O, HuggingFace downloader, SentencePieceModel) and MLXCommon (weight loading, QuantizedLinear helpers, SDPA multi-head attention helper).

    Full architecture diagram with backends, memory tables, and module map → soniqo.audio/architecture · API reference → soniqo.audio/api · Benchmarks → soniqo.audio/benchmarks

    Local docs (repo):

    Cache configuration

    Model weights download from HuggingFace on first use and cache to ~/Library/Caches/qwen3-speech/. Override with QWEN3_CACHE_DIR (CLI) or cacheDir: (Swift API). All fromPretrained() entry points also accept offlineMode: true to skip network when weights are already cached.

    Users in mainland China (or anywhere huggingface.co is slow/blocked) can fetch from a mirror by setting HF_ENDPOINT, e.g. export HF_ENDPOINT=https://hf-mirror.com.

    See docs/inference/cache-and-offline.md for full details including sandboxed iOS container paths.

    MLX Metal library

    If you see Failed to load the default metallib at runtime, the Metal shader library is missing. Run make build or ./scripts/build_mlx_metallib.sh release after a manual swift build. If the Metal Toolchain is missing, install it first:

    xcodebuild -downloadComponent MetalToolchain

    Testing

    make test                            # full suite (unit + E2E with model downloads)
swift test --skip E2E                # unit only (CI-safe, no downloads)
swift test --filter Qwen3ASRTests    # specific module

    E2E test classes use the E2E prefix so CI can filter them out with --skip E2E. See CLAUDE.md for the full testing convention.

    Contributing

    PRs welcome — bug fixes, new model integrations, documentation. Fork, create a feature branch, make build && make test, open a PR against main.

    License

    Apache 2.0

    相关项目

    同分类里 star 最高的项目
    对比全部 →
    VoiceStudio
    debpalash·Python

    本地运行的 ElevenLabs 开源替代品，支持声音克隆、语音设计、视频配音、听写、转写和有声书制作，覆盖 646 种语言。

    语音克隆+4语音推理
    107215+409112kv0.5.72026-11-10 23:08 UTC
    voicebox
    jamiepine·Python

    开源 AI 语音工作室，支持声音克隆、语音听写和语音生成，基于 Qwen3-TTS 与 Whisper，可在 CUDA 或 MLX 上本地运行。

    语音克隆+4语音推理
    60390+6377.6kv0.5.02026-10-07 01:17 UTC
    whisper.cpp
    ggml-org·C++

    OpenAI Whisper 语音识别模型的 C/C++ 移植版，无需 Python 和 GPU 依赖，在普通 CPU 上就能跑语音转文字。

    语音识别+4语音推理
    55073+866.3kv1.9.52026-11-10 12:09 UTC
    faster-whisper
    SYSTRAN·Python

    基于 CTranslate2 重实现的 Whisper 语音转文字推理引擎，通过量化与高效推理大幅提速并降低显存占用，适合本地部署转录。

    语音识别+4语音推理
    26655+832.2kv1.2.12026-11-10 10:13 UTC