qvac

tethertoApache-2.0活跃维护

开源本地 AI SDK，无需云端和 API Key 即可在设备端跑 GGUF 模型，集成 RAG、图像音乐视频生成、语音转写和 P2P 推理，覆盖桌面与移动端。

Open-source local AI SDK - run AI on-device with no cloud, no API keys. Supports GGUF, RAG, image, music, and video generation, speech-to-text, P2P inference, and more. Cross-platform: Linux, macOS, Windows, Android, iOS.

sdk-v0.23.0发布于 2026-11-02 12:57 UTC759star +8 / 7 天156fork同步于 2026-11-10 11:56 UTC

端侧推理本地大模型 GGUF RAG P2P推理

对比 GitHub 官网 ai-sdk decentralized-ai llama llm-inference local-ai

怎么上手

由 README 提炼，执行前请确认

原文

npm i @qvac/sdk
pip install tetherto-qvac-sdk -f https://github.com/tetherto/qvac/releases/expanded_assets/sdk-v<version>
npm install -g @qvac/cli

硬件要求按 README 说明

README 未给出具体显存或 GPU 型号要求，仅说明针对消费级硬件与嵌入式系统优化，并覆盖 Linux、macOS、Windows、Android、iOS；部分能力（如音乐生成）标注为桌面端。

需要 CUDA

下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼

没人工点评？写下你的经验

开源本地 AI SDK，无需云端和 API Key，在设备端跑 LLM、语音、视觉与生成模型

开源本地 AI SDK，无需云端和 API Key 即可在设备端跑 GGUF 模型，集成 RAG、图像音乐视频生成、语音转写和 P2P 推理，覆盖桌面与移动端。

核心能力

提供 JS/TS 与 Python SDK 统一调用多种本地 AI 能力
内置 OpenAI 兼容 HTTP 服务，可接入 OpenCode 等工具
支持 GGUF 模型加载与流式 token 输出
覆盖文本、嵌入、RAG、LoRA 微调与多模态推理
支持图像、视频、音乐生成及语音转写与合成

容易踩的坑

首次运行会先下载模型，需保证网络与磁盘空间；Python 包安装需把替换为实际发布版本并指定 releases 源；HTTP 服务默认绑定 127.0.0.1:11434，需在配置中声明要 serve 的模型。

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

Package	0.22.0 floor	0.23.0 floor
`@qvac/inference`	`^0.22.0`	`^0.23.0`
`@qvac/asr-ggml`	`^0.10.0`	`^0.11.1`
`@qvac/audiogen-ggml`	`^0.8.0`	`^0.9.0`
`@qvac/bci-whispercpp`

用户点评

登录用户的真实使用记录，可按版本和硬件筛选

共 0 条

还没有人点评。第一个说出真实体验的人最有价值。

README 原文

未翻译

在 GitHub 打开

QVAC lets you run a comprehensive range of AI workloads locally using open models across Linux, macOS, Windows, Android, and iOS.

QVAC provides:

SDK for building local-first AI applications and systems in JavaScript/TypeScript and Python.
HTTP server for using QVAC as a local model provider. Its OpenAI-compatible API lets you connect AI tools such as OpenCode and OpenClaw, or any other compatible tool.

Why QVAC

Local-first: run AI offline with inference optimized for commodity hardware, from consumer apps and embedded systems to on-premises deployments.
Privacy and control: keep data local, own the AI system. No cloud or third-party APIs required.
One SDK, all of AI: a comprehensive range of AI capabilities through one interface.
Cross-platform: one codebase for Linux, macOS, Windows, Android, and iOS, using JavaScript/TypeScript or Python.
Peer-to-peer: fetch AI models directly between peers and build unstoppable internet systems, like BitTorrent or IPFS, but for AI.

Quickstart

Load a model and run inference locally in a few steps. Pick your path.

Run your first example using the JS/TS SDK.

Create the examples workspace:

mkdir qvac-examples
cd qvac-examples
npm init -y && npm pkg set type=module

Install the SDK:

npm i @qvac/sdk

Create qvac.config.json to enable client and server logs during the run:

{
  "loggerConsoleOutput": true,
  "loggerLevel": "info"
}

Create the quickstart.js script:

import { loadModel, LLAMA_3_2_1B_INST_Q4_0, completion, unloadModel } from '@qvac/sdk';
try {
  const modelId = await loadModel({
    modelSrc: LLAMA_3_2_1B_INST_Q4_0,
    onProgress: (p) => {
      const mb = (n) => (n / 1e6).toFixed(1);
      const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`;
      process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
      if (p.percentage >= 100) process.stderr.write('\n');
    },
  });
  const history = [{ role: 'user', content: 'Explain quantum computing in one sentence' }];
  const result = completion({ modelId, history, stream: true });
  for await (const token of result.tokenStream) {
    process.stdout.write(token);
  }
  await unloadModel({ modelId });
} catch (error) {
  console.error('✖', error);
  process.exit(1);
}

Run the quickstart script:

QVAC_CONFIG_PATH=./qvac.config.json node quickstart.js

You'll see the model download first. Then QVAC will stream the response tokens and print them to the terminal.

Run your first example using the Python SDK.

Create the examples workspace:

mkdir qvac-examples-py
cd qvac-examples-py
python -m venv .venv
source .venv/bin/activate

Install the package (self-contained — bundles the QVAC worker and Bare runtime, no Node.js required):

# Replace <version> with the release you want, e.g. sdk-v0.17.0:
pip install tetherto-qvac-sdk \
  -f https://github.com/tetherto/qvac/releases/expanded_assets/sdk-v<version>

Create the quickstart.py script:

import asyncio
import sys

from tetherto.qvac_sdk import Client, completion, load_model, unload_model
from tetherto.qvac_sdk.models import LLAMA_3_2_1B_INST_Q4_0


def print_progress(p):
    line = f"▸ Downloading {p.percentage:.0f}% ({p.downloaded / 1e6:.1f}/{p.total / 1e6:.1f} MB)"
    print(line, end="\r" if sys.stderr.isatty() else "\n", file=sys.stderr)
    if p.percentage >= 100:
        print(file=sys.stderr)


async def main():
    async with Client() as client:
        t = client.transport
        try:
            model_id = await load_model(
                t, model_src=LLAMA_3_2_1B_INST_Q4_0, on_progress=print_progress
            )
            run = completion(
                t,
                model_id=model_id,
                history=[
                    {"role": "user", "content": "Explain quantum computing in one sentence"},
                ],
            )
            async for event in run.events:
                if event.type == "contentDelta":
                    sys.stdout.write(event.text)
                    sys.stdout.flush()
            print()
            await unload_model(t, model_id)
        except Exception as error:
            print(f"✖ {error}", file=sys.stderr)
            return 1
    return 0


if __name__ == "__main__":
    sys.exit(asyncio.run(main()))

Run the quickstart script:

python quickstart.py

You'll see the model download first. Then QVAC will stream the response tokens and print them to the terminal.

Launch the server with the CLI, then use QVAC as model provider for OpenAI-compatible tools like OpenCode and OpenClaw.

Install the CLI globally (this also installs @qvac/sdk as a transitive dependency):

npm install -g @qvac/cli

Create the examples workspace:

mkdir qvac-server
cd qvac-server

Create the qvac.config.json declaring one model to serve:

{
  "serve": {
    "models": {
      "my-llm": {
        "model": "QWEN3_600M_INST_Q4",
        "default": true,
        "config": { "ctx_size": 8192 }
      }
    }
  }
}

Start the server (bound to 127.0.0.1:11434 by default):

qvac serve openai

The model downloads on first start and is preloaded into memory. You'll see progress in the server output.

From another terminal, hit it with any OpenAI-compatible client. A minimal curl:

curl http://localhost:11434/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "my-llm",
    "messages": [{"role": "user", "content": "Explain quantum computing in one sentence"}]
  }'

The response comes back as a single JSON payload with the model's answer. Add "stream": true to the body to get an SSE stream instead.

Point your AI tool at the server: open its model provider settings and add a new OpenAI-compatible provider with base URL http://localhost:11434/v1, any string as the API key, and my-llm as the model name.

[!IMPORTANT] Setup varies by tool, and we ship dedicated plugins for some of them (like OpenCode and OpenClaw) that run the server for you. See Connect AI tools to QVAC for details.

⭐ If QVAC saves you from shipping yet another cloud dependency, give it a star, it helps other developers find the project!

AI capabilities

Task	Description
Text generation	LLM inference for text generation and chat via Fabric LLM.
Text embeddings	Vector embedding generation for semantic search, clustering, and retrieval.
RAG	Out-of-the-box retrieval-augmented generation workflow.
Fine-tuning	Adapting LLMs to domain-specific tasks via LoRA.
Multimodal	LLM inference over text, images, and other media in one context.
Image generation	Text-to-image and image-to-image generation via a Diffusion backend.
Video generation	Text-to-video and image-to-video generation via a Diffusion backend.
Music generation	Generate music from text, lyrics, and musical controls via ACE-Step or MiniMax-Music3 (desktop). The AudioGen addon exposes independent ACE-Step backend selection, opt-in WebGPU, Apple Core ML sidecars, and memory preflight for the selected placement. Native dependency versions are declared in the AudioGen manifest.
Transcription	Speech-to-text via a Whisper backend, NVIDIA Parakeet, TranscribePsy EU and EU-sub (full 25-language EU with CAFE adapters and language routing, or the eight-language EU-sub), or MOSS-Transcribe-Diarize (speaker-labelled transcripts with hotwords, fused CPU encoder attention and metadata-only memory fit via `assessFit`).
Speaker diarization	Whisper tinydiarize speaker turns, four-speaker Sortformer, up to eight-speaker Nemotron 3 Diarization, Parakeet text per speaker turn with a separate diarizer, and speaker-labelled MOSS-Transcribe-Diarize transcripts through `@qvac/asr-ggml`. The addon exposes native Parakeet backend selection and streaming token IDs, finality and event timing, delivering transcript chunks before their end-of-turn boundary.
Audio decoding	FFmpeg audio decoder with streaming PCM output and a default 64 MiB decoded-output limit.
Text-to-Speech	Speech synthesis with selective MOSS Delay text projection on Metal and single-reference MOSS-TTSD preparation, incremental Audio8 PCM streaming, speech-to-speech replies and text-to-sound-effects generation via a GGML backend, with memory preflight for MOSS-SoundEffect, LavaSR denoising on Android Mali GPUs with explicit Core ML denoiser/enhancer options on Apple, and Pocket TTS (CPU, Metal, CUDA, Vulkan and OpenCL through the addon). Native engine controls expose MOSS sampling/budgets, Supertonic precision/warm-up, Parler Core ML selection and CosyVoice token replay. Exact native dependency versions are declared in the package manifest. Chatterbox and CosyVoice3 Core ML sidecars are optional on Apple platforms. Re-export existing CosyVoice3 LM sidecars after upgrading.
Translation	Neural machine translation, via Fabric LLM and Bergamot.
BCI	Brain–computer interface transcription via a Whisper backend.
VLA	Vision-language-action for robot control via a GGML backend.
OCR	Extract text from images via ONNX Runtime or GGML backends. See OCR GPU selection (`main-gpu`) to select a GGML GPU by registry index or device class.
Image classification and embeddings	Classify images with MobileNet or generate MegaLoc descriptors for image retrieval via a GGML/Fabric backend.

Peer-to-peer

QVAC's built-in P2P capabilities let you build unstoppable internet systems without depending on centralized infrastructure:

Fetch models: download AI models directly from peers through a distributed model registry, removing the need for centralized model hosting and distribution. The registry also serves weightless descriptions of models, their settings and tensor list without the weights, so whether a model fits in memory can be assessed before it is downloaded.
Blind relays: route traffic through relay peers when devices cannot connect directly across NATs and firewalls, keeping the network connected without centralized infrastructure.

Resources

Explore and use QVAC:

Resource	Description
Docs	Comprehensive QVAC documentation.
Examples	Sample apps and PoCs built with QVAC SDK.
Local model provider	Use QVAC as a local model provider connected to your favorite AI tools.
QV.AC	Get to know our local AI assistant.
Support and community	We gather on Discord and Keet. Ask for help, give feedback, and discuss QVAC.
Blog	Tutorials, deep dives, engineering notes, and announcements.
Ecosystem	Discover the broader QVAC ecosystem.
Research	Papers, datasets, and models optimized for edge devices.
Our vision	Learn why Tether built QVAC.

Contributing

We welcome contributions! Feel free to open a pull request, report bugs, or share ideas through issues.

See CONTRIBUTING for details.

Banners and badges

Built something with QVAC? Add a badge to your README to show it and help others discover QVAC:

[![Built with QVAC](https://raw.githubusercontent.com/tetherto/qvac/refs/heads/main/docs/branding/qvac-badge-green-light.svg)](https://github.com/tetherto/qvac)

The full set of banners and light/dark and inline badge variants, with copy-paste snippets, lives in BADGES.md.

qvac

怎么上手

项目说明

核心能力

容易踩的坑

最近发版

Table of Contents

Kotlin client

Breaking Changes

Fabric 0.24 addon set

Table of Contents

Breaking Changes

Node version for `qvac bundle` / bare-pack

`assessModelFit` uses engine fitters

Per-load `modelFitPolicy`

Table of Contents

Breaking Changes

`assessModelFit` uses loadModel parameters

Flattened fit-probe projection

Catalog constant names

Table of Contents

New APIs

llama.cpp placement, MoE, fit, and image-token load fields

`assessModelFit` uses the engine fitter

Bug Fixes

Mobile Bare-kit host

用户点评

README 原文

Why QVAC

Quickstart

AI capabilities

Peer-to-peer

Resources

Contributing

Banners and badges

怎么上手

项目说明

核心能力

容易踩的坑

最近发版

Table of Contents

Kotlin client

Breaking Changes

Fabric 0.24 addon set

Table of Contents

Breaking Changes

Node version for qvac bundle / bare-pack

assessModelFit uses engine fitters

Per-load modelFitPolicy

Table of Contents

Breaking Changes

assessModelFit uses loadModel parameters

Flattened fit-probe projection

Catalog constant names

Table of Contents

New APIs

llama.cpp placement, MoE, fit, and image-token load fields

assessModelFit uses the engine fitter

Bug Fixes

Mobile Bare-kit host

用户点评

README 原文

Why QVAC

Quickstart

AI capabilities

Peer-to-peer

Resources

Contributing

Banners and badges

相关项目

Node version for `qvac bundle` / bare-pack

`assessModelFit` uses engine fitters

Per-load `modelFitPolicy`

`assessModelFit` uses loadModel parameters

`assessModelFit` uses the engine fitter