Linux 用 CUDA 13 编译内核，驱动低于 R580 会在启动时被拒绝；结构化输出和必需工具调用依赖发布包里的 lib/ 或自行用 xgrammar v0.2.8 构建 grammar；从源码构建需完整 Xcode 和 Metal Toolchain，仅装 Command Line Tools 会导致内核编译失败。
TensorFold
ashhartApache-2.0活跃维护用 Zig 写的跨平台 LLM 推理引擎，支持 Metal、CUDA 和 Vulkan，让同一套代码在苹果、NVIDIA 和通用 GPU 上跑大模型推理。
LLM Inference Engine for Metal, CUDA and Vulkan.
v1.2.5发布于 2026-11-08 17:02 UTC1,725star +32 / 7 天374fork同步于 2026-11-10 11:47 UTC
brew install ashhart/tensorfold/tensorfold
tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3
tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit --name local-model --context 8192 --no-thinking 硬件要求按 README 说明
Apple Silicon Mac 或 NVIDIA GPU；Linux 需 glibc 2.28+、NVIDIA 驱动 R580+（CUDA 13.0）并提供 libcuda.so.1，旧驱动启动时会被拒绝；README 未给出显存要求。
支持 Apple Silicon需要 CUDA多卡 / 张量并行
用 Zig 编写的本地 LLM 推理引擎，在 Apple Silicon 和 NVIDIA GPU 上直接运行 Metal/CUDA 内核，无需 Python 或 MLX
用 Zig 写的跨平台 LLM 推理引擎，支持 Metal、CUDA 和 Vulkan，让同一套代码在苹果、NVIDIA 和通用 GPU 上跑大模型推理。
核心能力
- 直接读取检查点并运行 Metal 或 CUDA 内核
- 提供 OpenAI 兼容的 chat/completions 等 HTTP 接口
- 用 --parallel 限制并发请求数，超出部分排队等待
- 支持本地检查点目录或已有 Hugging Face 缓存
- 附带本地 Web 应用用于选模型和对话
容易踩的坑
内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README
- v1.2.5TensorFold 1.2.5最新正式版2026-11-08 · 2026-11-08 17:02 UTC
TensorFold 1.2.5
1.2.5 brings LFM2.5-24B-A2B on CUDA, Nemotron Parse and LightOnOCR-3 on HIP, and Clef's joint schema head on CUDA. It also adds TensorFold Web, a local web app. Qwen3.8-27B's MLX 8-bit checkpoints get faster prompts on Macs, and a new opt-in keeps its keys and values in pool pages. HIP's opt-in gfx12 matrix tiles get faster with the same bytes. Settings are stricter: DeepSeek-V4.1's settings are refused by name when malformed, and an explicit thinking off that a chat template can't honour is refused.
- New:
- LFM2.5-24B-A2B on one NVIDIA GPU, from the MLX 4-bit export;
- NVIDIA's Nemotron 3.5 Parse on HIP;
- LightOnOCR-3 (0.8B, 1B, 4B) on HIP;
- Clef's joint schema head and
/v1/systemoneon CUDA; - TensorFold Web, opened with
tensorfold web; - the HIP qwen3_5 engine reads a DFlash2 drafter's config and weights, and taps its target layers;
- a CPU projection and RMS norm in the kernel contract.
- Speed:
- Qwen3.8-27B MLX 8-bit on an M5 Ultra: cold prompts are 18% faster and decode 7.6% faster.
- Flash Next BF16 cold prompts on a GB10 take about 4% less time, and under 1% less again on top.
- HIP on gfx12 with
--policy matrix=on: reranking and embedding take 2.9-4.8% less time on an R9700, with the same bytes. - Qwen 3.5 and 3.6 on HIP: sampled streams with an MTP head make 4-12% more tokens a second.
- Qwen3.8-27B prompt chunks on an M3 Pro: 2-3% faster served prefill, bit-equal.
--learnwith 100,000 kept states starts about 5 s sooner.
- Serving:
- An explicit thinking off that the template can't honour is a 400.
- A refused disk read tries one shorter kept state.
- Qwen 3.5 and 3.6 streams on CUDA take less memory.
- A request's stats list its prompt pass widths.
- The PDF render worker no longer crashes on macOS.
- New opt-ins: the Qwen CUDA history starts with drafts; Gemma 4's 2,048-row solo chunk; CUDA prompt-cache keep-others; Qwen3.8-27B Metal KV pages.
Where a speedup was checked for equal output, its line says so. Some changes alter settings, refusals, memory or startup by default; they're listed under What can change for you.
Headline numbers
Each number names its machine and its PR.
Release reads, each machine with 1.2.4 and 1.2.5 in alternating fresh servers (base, new, new, base, after an unscored warm-up):
-
Flash Next MLX 4-bit on a GB10, against 1.2.4: level, with the same tokens: every prompt and decode cell from 2K to 64K within 0.8%, and every sweep from one to eight streams within 1.1% apart from greedy chat at four streams (3.6% faster). The 64K long-context answer is right on both.
-
Qwen3.8-27B on an M5 Ultra, against 1.2.4: level, with the same tokens: every sweep within 0.4% and every prompt and decode cell from 2K to 64K within 1.2%, apart from one 1.2.5 run's first 2K prompt, which came up cold while the other 1.2.5 run matched both 1.2.4 runs.
-
Qwen3.8-27B on a GB10, against 1.2.4: level, with the same tokens: all 56 cells within -1% to +2%. The 64K long-context answer is right on both.
-
Nemotron 3.5 Lightning 30B-A3B on a GB10, against 1.2.4: level, with the same tokens: prompts within 1% and one stream level. Greedy chat at two to eight streams moved both ways (-2.8% to +1.5%), as one-prompt multi-stream cells do; decode after a 32K prompt resumed 10K tokens in was 2.5% lower in both 1.2.5 runs.
-
Qwen3.8-27B on an M3 Ultra, against 1.2.4: level, with the same tokens: every prompt and decode cell from 2K to 64K within 1.3%, and 2K prompts 2.1% faster in both 1.2.5 runs. Sweeps were within 0.4%, against one 1.2.4 run. The 64K long-context answer is right on both.
-
Qwen3.8-27B MLX 8-bit (groups of 64), on an M5 Ultra: cold prompts are 18% faster (649 to 768 tok/s at 2K) and decode 7.6% faster, with the same tokens and prompt state (#2518, fixes #2502).
-
Qwen3.8 Flash Next on CUDA, on a GB10 (NVFP4):
- BF16 prompt projections take a 128 × 128 tile on
- New:
- v1.2.4TensorFold 1.2.42026-11-07 · 2026-11-07 11:53 UTC
查看更新说明
TensorFold 1.2.4
1.2.4 brings Gemma 4 E4B on CUDA, Cloudflare's Clef-Flash on HIP, LFM3-1.7B's text decoder on Metal, and the HIP runtime on Windows. DeepSeek-V4.1 recalls a saved long conversation in seconds instead of minutes. Flash Next gets faster prompts and decode, sooner first tokens, kept prompt states with drafts at
--tp 2, faster SSD experts, and much faster overlapping clients on M5 Macs. Gemma 4 and Qwen 3.5 load faster cold on CUDA. HIP decode and rerank are faster. GLM on two or more GB10s prefetches its next decode weights into L2. DeepSeek-V4 and V4.1 prompts follow the release's recorded encoder outputs. An engine's refusals now reach the client as clear 400s.- New:
- Gemma 4 E4B (NVIDIA Nemotron 3.5 Bolt E4B) on CUDA;
- Clef-Flash's joint schema decisions on
/v1/systemoneon HIP; - DeepSeek-V4.x chat rendered with its own encoder;
- LFM3-1.7B text on Metal, from the official MLX checkpoints;
- the HIP runtime on Windows;
- Clef-Flash's BF16 Qwen3.5 backbone on CUDA, as a decision backbone.
- Speed:
- A 319K-token DeepSeek-V4.1 conversation restores from NVMe in 1.2 to 1.4 s instead of 155 to 178 s.
- A Flash Next chat resumed at
--tp 2with drafts replies in 1.5 s instead of 21.2 s. - Flash Next prompts at
--parallel 1on one CUDA GPU are 15-23% faster, and 8 streams decode 3-6% faster. - A new short Flash Next prompt's first token on CUDA comes 19% sooner.
- Two overlapping Flash Next clients on an M5 Ultra finish in 47 s instead of 127 s.
- Gemma 4 and Qwen 3.5 on CUDA load 28-31% faster cold.
- A lone
"draft": falseFlash Next reply on Metal is 17-18% faster. - Rerank on HIP is about 25% faster, and one-stream HIP decode up to 2.7% faster.
- Qwen 3.5 first tokens on a GB10 come 1 to 3% sooner.
- GLM on CUDA prefetches a decode window's next weights into L2 on two or more GB10s, with the same replies.
- Serving:
- An engine's refusal reaches the client as a 400 before any header.
- Flash Next on Metal enforces structured output at
--parallel 1too. - Unsupported penalties are refused instead of dropped.
- A refused prompt-cache restore tries a shorter kept state.
- DeepSeek-V4.1 tool calls stream sooner.
- Tool calls made just before the thinking close are returned.
- DeepSeek-V4 and V4.1 prompts match the release's recorded encoder outputs, and malformed tool requests are refused with a 400.
- An opt-in queue-wait histogram for native metrics.
/v1/modelslists what each model supports.- An opt-in drift gate for Living Weights lessons.
- Malformed
TF_,TENSORFOLD_andTENSORFOLD_PREFILL_*settings stop the server at startup, by name. - A kept prompt state on disk is reused only by the same build and checkpoint files.
Where a speedup was checked for equal output, its line says so. Some changes alter scheduling, memory, loading, prompts or startup by default. They're listed under What can change for you.
Headline numbers
Each number names its machine. Where replies were checked against the base, the line says how.
Release reads, each machine with 1.2.3 and 1.2.4 in alternating fresh servers (base, new, new, base, after an unscored warm-up):
- Flash Next MLX 4-bit on a GB10, against 1.2.3: prompts and one stream level, with the same tokens; two to eight streams up to 5.1% faster (eight streams 1.7% to 5.1%). The 64K long-context answer is right on both.
- Qwen3.8-27B on an M5 Ultra, against 1.2.3: level, within 1.2% in every sweep and on every prompt and decode cell from 2K to 64K, with the same tokens.
- Qwen3.8-27B on a GB10, against 1.2.3: level, within 1.0% in every sweep and on every prompt and decode cell from 2K to 64K, with the same tokens.
- Qwen3.8-27B on an M3 Ultra, against 1.2.3: level on every prompt and decode cell from 2K to 64K, with the same tokens. At 64K one of the two 1.2.4 runs was up to 2.5% lower and the other matched both 1.2.3 runs. Sweeps
- New:
- v1.2.3TensorFold 1.2.32026-11-05 · 2026-11-05 17:36 UTC
查看更新说明
TensorFold 1.2.3
1.2.3 is mostly fixes, memory and serving work, with a few new paths.
- HIP: dots.mocr, Qwen3-Embedding-8B and Qwen3-Reranker-8B are served natively, and Qwen 3.5 loads BF16 checkpoints through rocBLAS.
- Metal: the Qwen3.8-27B engine serves Qwen3.5-MoE checkpoints, including Ornith-1.5-122B on one M5 Ultra.
- PDF documents work as input with
--vision --vision-pdf. - Intel GPUs: LFM3-1.7B runs on Iris Xe.
- Kimi K3 and K3.1 are reachable through the native server on Metal, but only as groundwork. They have run on a synthetic graph, not on real weights; see below.
- Speed: Flash Next stops a long prompt from stalling a running reply on Macs without tensor units, and decodes faster on a GB10, at
--tp 2on two GB10s and on a Strix Halo. A GLM cold prompt at TP=3 gets its first token sooner. - Memory and correctness: a prompt cache you size with
--prompt-cache-gibnow gets that size on CUDA KV pages. Cached-token counts can't exceed the prompt. A few loaders and decoders refuse bad input they used to accept.
Where a speedup was checked for equal output, its line says so. Opt-in switches change nothing unless set. A few changes move replies, change what a request returns, or refuse a start that used to go ahead. They're listed under What can change for you.
Headline numbers
Each number names its machine. Where replies were checked against the base, the line says how.
- Flash Next NVFP4 on two GB10s at
--tp 2: plain decode is 21% to 23% faster, drafted decode 9% to 11% faster at one stream and 76% to 77% at two, and prompts about 6% faster. Each rank holds 2.62 GiB less of the model and a stream's KV at 65,536 tokens takes 40% less, with replies token-equal to one GPU (#1784). - Flash Next on an M3 Ultra,
--parallelabove one: a 1,024-token reply beside a 64K prompt finishes in 8 s instead of 48 s. The prompt's first token comes at 54 s instead of 90 s, with the same tokens (#2167). - Flash Next on an M3 Ultra, a lone reply at
--parallelabove one: it runs as fast as at--parallel 1. Sampled replies are 12% to 19% faster than before and greedy ones about 4%, with the same tokens (#2199). - Flash Next on an M3 Ultra with
--ssd-experts: about 11% less read from disk at 17 and 35 GiB pools, and a mixed workload took 8% less time at 35 GiB, with the same replies (#2209). - Flash Next FP4 on a GB10: one-stream decode up to 11.5% faster (chat +8.7%, code +6.3% at temperature 0), with every reply token-equal to the previous version (#1786).
- Flash Next on a Strix Halo (gfx1151): the first request after start decodes at 35.8 tok/s instead of 24.4, and a cold 7K-token prompt starts in 28.9 s instead of 48.4 s, with every reply token-equal (#1615).
- HIP on a gfx1151: Qwen3.8-27B at 16 streams goes from about 51 to 75 tok/s on code, with the same bytes. These are the contributor's own records, as we have no AMD GPU (#1694).
- GLM-5.3-Flash on three GB10s at TP=3, with
TF_GLM_ROCE_MULTI=1andTF_GLM_CHUNK_SOLO=4096: a cold prompt's first token comes about 5% sooner, with the same tokens (#2116). - GLM-5.3-Flash on two GB10s at TP=2 with
--drafter:TF_GLM_CHUNK_SOLO=4096now serves, and a cold prompt's first token comes 6.1% sooner at 32K and 7.0% sooner at 64K, with the same tokens, for 80 MiB more a rank (#2219). - GLM-5.3-Flash on an M3 Ultra: about 14% more throughput at four streams when drafts land often, with the same tokens. This was measured on 40 of the model's 45 layers, not the whole model (#2111).
- GLM-5.3-Flash
csf0_v1checkpoints on CUDA: about 13 GiB less across a TP=2 pair, with the same tokens (#2041). - Flash Next MLX 4-bit on a GB10, against 1.2.2: one stream is 3.5% to 6.6% faster and eight streams 1.1% to 3.8%, with prompts level and the same tokens. Decode after a long prompt is faster in 18 of 20 of our cells, by 2.3% to 14.8%; two cells where drafts land very ofte
- v1.2.2TensorFold 1.2.22026-11-03 · 2026-11-03 18:53 UTC
查看更新说明
TensorFold 1.2.2
1.2.2 adds new models on CUDA and HIP. Qwen3.8-Flash-Next serves NVIDIA's own NVFP4 checkpoint at
--tp 2. GLM-OCR runs natively on HIP, and Qwen3.5-9B runs on Metal on the 27B's engine. Qwen3.8-27B EXL3 checkpoints prefill 38% to 42% faster on a GB10, HIP decode is faster on Ryzen AI Max, and Flash Next reads its per-layer embeddings from an SSD much faster on Macs. CUDA and Metal loaders release the page cache once the weights are loaded. CUDA loaders read checkpoints ahead, so they load sooner.zig build -Dgpu=<gpu>builds a much smaller server for one GPU. Readers of safetensors and other checkpoint headers across the engine now refuse a malformed file instead of crashing on it.Where a speedup was checked for equal output, its line says so. Opt-in switches change nothing unless set. A few changes move replies or refuse a start that used to go ahead; they're listed under What can change for you.
Headline numbers
Where a number's replies were checked against its base, its line says how.
- Qwen3.8-27B EXL3 on a GB10: prompt prefill 38% to 42% faster from 2K to 64K tokens (494 to 705 tok/s at 2K), and decode 5% to 12% faster, with the same tokens (#1467).
- HIP decode on a gfx1151: 7.5% to 18.9% faster on Qwen3.6-35B-A3B and Qwen3.8-27B at 32K and 64K, with identical tokens (#1932).
- Ryzen AI Max (RDNA 3.5): Qwen3.6-35B-A3B chat decodes about 18% faster with drafts on, and Qwen3.8-27B about 5%, with the same tokens (#1855).
- Flash Next on an M5 Max with
--ple-on-ssd: decode 36% to 39% faster (75.8 to 105.1 tok/s on code), and a cold 2K prompt in a third of the time, with the same replies (#1462). - Flash Next on a gfx1151: the first cold prompt runs in 28.55 s instead of 48.44 s, and the page cache falls from 31.5 to 2.8 GiB (#1611).
- GLM on three GB10s at TP=3: cold prompts of 32K to 167K tokens finish 5% to 14% sooner, with identical replies (#1577).
- Flash Next on a GB10: a cold start drops from about 74 s to 48 s with the direct expert load (#1901). Cold prompts run about 4% to 5% faster, with identical tokens (#1764).
New models and backends
nvidia/Qwen3.8-Flash-Next-NVFP4on CUDA at--tp 2, on two GPUs, one rank a host (#1771, for #1754).- The BF16 attention, DeltaNet and shared-expert projections, the n-gram table (E4M3 rows with one BF16 scale) and the MTP head's E4M3 experts (128x128 block scales) load as stored.
- Each rank owns half the routed experts whole, so replies are one GPU's bytes.
- On two GB10s, drafted replies equal
"draft": false(6 of 6), and concurrent replies equal their solo runs in 8 of 8 cells.
- The CUDA server serves
--tp, with--rank,--masterand--master-port. Families that open without ranks refuse it by name, and--idle-unloadis refused with it.- The ranks compute and compare a cached digest of every checkpoint byte, so
TENSORFOLD_CHECKPOINT_SHA256is no longer required (it still overrides the digest) (#1771).
- The ranks compute and compare a cached digest of every checkpoint byte, so
- GLM-OCR on HIP: the publisher's BF16
zai-org/GLM-OCR, served natively. Its CogViT image encoder and GLM decoder run in Zig and HIP, with no Python or libtorch. Every reply was checked against the model card's Transformers examples (#1940, part of #1927).- Groundwork for the OCR models (#1929, #1930, #1931, #1933):
- a family can declare that its replies carry no reasoning block;
- the tokenizer supports NFKC and NFKD normalizers;
- HIP has a shared rocBLAS BF16 projection, loaded as an optional library;
- the prompt cache can count repeated keyed prompts as repeats.
- No model already served decodes differently.
- Groundwork for the OCR models (#1929, #1930, #1931, #1933):
- Qwen3.5-9B on Metal runs on the 27B's engine, instead of being refused as an unqualified shape (#1826, part of #503). The dense Qwen Metal loader checks config-derived projection geometry through shared structural checks, and keeps the 9B and 27B qualification policy (part of
- app-v1.0.0TensorFold for Mac 1.0.02026-11-02 · 2026-11-02 09:36 UTC
查看更新说明
The first stable TensorFold desktop release for Apple silicon Macs running macOS 13.3 or later. The app is signed with Developer ID and notarized by Apple.
Download TensorFold for Mac 1.0.0
- Ask TensorFold opens as a compact input bar, with explicit window selection, capture preview and available local text extraction.
- The main window closes into the background, with the TensorFold mesh logo in the menu bar.
- Connection health and the unified inbox show account state and provider errors. Google Workspace and Microsoft connections require your configured credentials and consent.
- Portuguese is included in the interface languages.
The app bundles engine 1.2.0. Engine 1.2.1 is available separately through Models → Details → Engine updates. Checking, downloading and activation are explicit actions. The app waits for local work to finish before switching engines, and you can return to the bundled engine without replacing the app or model files.
Windows and Linux builds remain previews. Cortheon remains disabled. OS automation and messaging depend on the operating system and configured services.
Validation includes 931 passing app test files, 259 native-helper checks, isolated packaged startup on both build and verification Macs, and a real-model engine 1.2.0 to 1.2.1 upgrade and rollback. The Studio independently matched all 121 compiled application files against the pinned source build.
The DMG, automatic-update ZIP and engine archive are accompanied by signed update metadata and SHA-256 checksums. This desktop tag is separate from the engine release tags. Source:
aa8611f46b374cf709bed812b1355de2b1199e84.The Mac desktop, shared Linux desktop tests and mobile checks passed in the release CI run. The Linux AppImage preview smoke test stopped because the runner lacks the required sandbox approval. No Linux installer is published with this Mac release.
用户点评
登录用户的真实使用记录，可按版本和硬件筛选共 0 条
还没有人点评。第一个说出真实体验的人最有价值。
TensorFold
TensorFold serves language models from a Zig binary on Apple Silicon and NVIDIA GPUs. The engine reads checkpoints, tokenizes requests and runs Metal or CUDA kernels directly. Serving needs no Python or MLX installation.
Building with a coding agent? Have it load the contributor skill before it writes any code.
Contributing with agents
Please use Agent Traffic Control before editing TensorFold so contributors and agents can avoid duplicate work and conflicting changes. Check the live agent radar before choosing an issue and follow CONTRIBUTING.md and the ATC contributor instructions.
Install or upgrade ATC to 0.7.0 or later, then run from an updated TensorFold clone:
npm install -g github:ashhart/ATC
atc check ISSUE --json
atc claim ISSUE --json
Replace ISSUE with the issue number and claim only after a complete check says the work is free.
TensorFold enables automatic public check-ins: a successful new claim starts an anonymous session on the radar and returns its session number.
Keep that number instead of starting a duplicate session.
For work already yours, start a session with atc start ISSUE --json.
Add --github-name to claim or start only if you want your verified GitHub name shown; --local keeps a new session on your machine.
The board shows the issue, activity state and last check-in; GitHub still associates the verification comment with its author.
Check that publication is public; unavailable means the public update failed even if the local claim or session succeeded.
Pulse only while the agent is working, before its five-minute check-in expires:
atc pulse ISSUE SESSION working --json
atc who path/to/file --json
atc overlap --json
atc land ISSUE SESSION --json
atc release ISSUE --json
Use reviewing or blocked when appropriate, land the session when it stops, and resolve overlaps before opening a pull request.
Before opening a pull request, have the agent run the pull request skill; agents that support skills can install it as tensorfold-pull-request.
Quick start
Start with Qwen3.5-2B, the small 4-bit checkpoint checked on Metal and a GB10. The commands below download its
pinned revision, then serve it as local-model. Model weights have their own licenses.
Reviewing or benchmarking TensorFold? Evaluating TensorFold covers pinning, signature checks,
a container and the tests that need no GPU.
Mac with Apple Silicon
Install with Homebrew, then start the server:
brew install ashhart/tensorfold/tensorfold
tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3
tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit --name local-model --context 8192 --no-thinking
Linux with an NVIDIA GPU
Use the release archive for your CPU: linux-aarch64 on a GB10, linux-x86_64 on an x86_64 host.
It needs glibc 2.28 or newer and NVIDIA driver R580 or newer (CUDA 13.0), providing libcuda.so.1. Keep the archive's directories together.
Its kernels are built with CUDA 13, so TensorFold refuses an older driver at startup; nvidia-smi shows the installed one.
See the CUDA runbook for runtime libraries and supported GPUs.
V=1.2.5
case "$(uname -m)" in
aarch64|arm64) PLATFORM=linux-aarch64 ;;
x86_64) PLATFORM=linux-x86_64 ;;
*) echo 'No release archive for this CPU' >&2; exit 1 ;;
esac
curl -fLO "https://github.com/ashhart/TensorFold/releases/download/v$V/tensorfold-$V-$PLATFORM.tar.gz"
curl -fLO "https://github.com/ashhart/TensorFold/releases/download/v$V/tensorfold-$V-$PLATFORM.tar.gz.sha256"
sha256sum -c "tensorfold-$V-$PLATFORM.tar.gz.sha256"
tar -xzf "tensorfold-$V-$PLATFORM.tar.gz"
export PATH="$PWD/tensorfold-$V-$PLATFORM/bin:$PATH"
tensorfold() { command tensorfold-native "$@"; }
tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3
tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit --name local-model --context 8192 --no-thinking
Send a request
Leave the server running. In another terminal, on either platform:
curl -fsS http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"local-model","messages":[{"role":"user","content":"Say hello in one sentence."}],"max_tokens":128,"temperature":0}'
The API also serves completions, Responses and Anthropic Messages. /v1/models lists the served name.
On Windows with an NVIDIA GPU, follow the native Windows build guide.
Use tensorfold --help, tensorfold capabilities --json and tensorfold serve REPO --help to inspect your binary.
You can also serve a complete local checkpoint directory or an existing Hugging Face cache.
tensorfold web [ARGS...] opens TensorFold Web, a small local web app for picking, running and chatting with models.
If it needs installing, an interactive release binary offers to install its matching version with uv after you say yes.
See the web launcher for manual installation and unattended use.
--parallel N caps active requests at N; extra requests wait for a free slot.
Omitting the flag uses --parallel auto, which asks for up to eight streams as memory and the engine allow.
An explicit number is refused if the engine cannot fit that many streams.
Read the engine's startup capacity line, such as 8 streams at once on CUDA, for the count it accepted.
With API authentication off, /health reports live.connections as active plus waiting requests and
live.waiting as the queued count. For example, 8 connections with 4 waiting means 4 active requests.
Completing or cancelling an active request frees its slot; cancelling a waiting request removes it from the queue.
Mac app
The TensorFold desktop workspace bundles the local engine, model downloads, hardware recommendations, live generation metrics, coding workers and document editing.
Download the Apple-silicon beta 9 DMG, or build from app/.
The desktop app launches the bundled tensorfold-native executable directly.
Mobile clients in app/mobile/ remain preview software.
TensorFold Starlight (enterprise)
TensorFold Starlight gives an organisation one governed OpenAI- and Anthropic-compatible endpoint for a fleet of TensorFold engines on Macs, Linux GPU servers and Windows machines.
It signs people in through the organisation's identity provider (OIDC or SAML, with SCIM provisioning), decides who may use which model and where it runs, places and supervises models on nodes, and accounts for every token.
It runs fully offline, or brokers approved frontier providers under the same identity, policy and budgets; the install guide covers Docker, Kubernetes and Linux services.
Starlight lives in its own repository, ashhart/tf-starlight: it pins the TensorFold release it is tested with and checks every flag its nodes pass to tensorfold serve against that release in CI.
Starlight is new: the gates that still need real GPU clusters or identity-provider tenants are listed in its qualification notes.
Living Weights (experimental)
Living Weights teaches a running model new facts in its own weights. Start a Nemotron 3.5 Lightning server with
--slide and tell it something: each fact that passes the lesson's checks is written into the checkpoint folder's
files. It is experimental: related facts can disturb each other, recall drops past a few dozen facts and with thinking
on, and learned facts cost some decode speed on a Mac. Known issues
has the details.
tensorfold pull TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit
hub="${HF_HUB_CACHE:-${HF_HOME:-$HOME/.cache/huggingface}/hub}/models--TensorFold--NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit"
src="$hub/snapshots/$(cat "$hub/refs/main")"
mkdir -p ~/models && cp -RL "$src" ~/models/nemotron-living # always teach a copy
(umask 077; openssl rand -hex 24 > ~/models/slide-key) # --slide starts only with a key
tensorfold serve ~/models/nemotron-living --slide --api-key-file ~/models/slide-key
Leave that server running and send a lesson from another terminal:
curl -N http://127.0.0.1:8080/v1/slide/learn -H "Authorization: Bearer $(cat ~/models/slide-key)" \
-H 'Content-Type: application/json' -d '{"text": "I like blue."}'
Send it a document instead and it finds the facts in it and learns them one by one. Open
http://127.0.0.1:8080/slide to watch in 3D: the facts, the neurons each one took in the model's layers, the shard
files that hold them, and the lesson running now.
Learning runs on Apple-silicon Macs and on an NVIDIA GB10 (DGX Spark), and a folder that learned on one serves on the
other. On an APFS volume, cp -cRL makes a clone that shares the original's data until lessons change it. It rewrites the folder's files:
before the first lesson the server keeps the originals in a hidden snapshot, and
tensorfold slide rollback ~/models/nemotron-living puts them back, undoing every lesson. The
Living Weights guide covers the key, teaching files and web pages, omp's /learn command,
the 3D view, undoing lessons, and what to expect.
Pick a model
These are the native release's serving paths. A recipe may also describe the older Python 0.6.6 engine;
use its native section and the current changes together.
Hardware below records the documented scope, not an unmeasured minimum. Context, parallel requests, prompt caches
and drafters need memory beyond the weights. tensorfold info REPO reports the checkpoint's memory floor.
Where no RAM capacity is published, inspect that floor before serving a large checkpoint.
CUDA release images include sm_80, sm_86, sm_120 and sm_121. The A100 (sm_80) has kernel-check evidence
(#846), but no served checkpoint on record, so it is not model-qualified.
Nemotron and dense Qwen have NVFP4/FP8 fallback code for sm_80 to sm_89 (#901, #1470); no such model run on sm_80 to sm_89 is recorded.
Run a table command after installation. On a Linux archive installation, use the tensorfold shell function from Quick start.
REPO@revision pins a pull; serve the repository name without the revision after pulling it.
Conversions are explicitly marked: their upstream weights need the linked conversion before native serving.
| Model / format | Download | Machine and memory scope | Native recipe / limits |
|---|---|---|---|
| dots.mocr, BF16 OCR | Use the local dots-studio/dots.mocr checkpoint. | Native HIP; AMD R9700 (gfx1201) checked, one GPU. | Native OCR recipe; full model-card demo matches HF tokens. No video, MTP head, CUDA, or Metal qualification. |
| Qwen3.5-2B, MLX 4-bit | tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3 | One Mac (M5 Max checked), or one GB10; no RAM minimum published. | 2B; CUDA release checks. Text only. |
| Qwen3.8-27B, MLX 4-bit | tensorfold pull TensorFold/Qwen3.8-27B-MLX-4bittensorfold pull z-lab/Qwen3.8-27B-DFlash2 | One Mac (M5 Max/M3 Ultra), or one GB10; a 32 GB Mac needs iogpu.wired_limit_mb raised (the startup refusal names the value). | 27B. Add --drafter z-lab/Qwen3.8-27B-DFlash2; without it decoding is plain. |
| Nemotron 3.5 Lightning 30B-A3B, MLX 4-bit | tensorfold pull TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit | One Mac (M1-M5), or one GB10; also checked on RTX 3090 / RTX PRO 6000. | Lightning. Includes MTP. |
| Lightning, mixed oQ4 / oQ6 / oQ8 | tensorfold pull TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-oQ4tensorfold pull TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-oQ6tensorfold pull TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-oQ8 | One M3 Ultra checked. | Width and MTP instructions. Metal only; copy the 4-bit checkpoint's MTP head to draft. |
| Lightning, NVIDIA NVFP4 | tensorfold pull nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 | One GB10 checked. Non-cluster fallback code exists for sm_80 to sm_89; no NVFP4/FP8 serving run is recorded on sm_80. | Lightning; CUDA changes. |
| Nemotron 3.5 Super VL 120B-A12B, NVFP4 | tensorfold pull nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4@82d93699acd3be9e4617c53360fe5b3f5dbeb1b3 | One or two GB10s; weights 70.8 GiB, 73.2 GiB with vision on one. | One/two-GB10 commands. Add --vision for images. |
| Nemotron 3.5 Bolt 74B-A9B | NVIDIA's checkpoint | Native CUDA; 24 GB checkpoint, checked on RTX PRO 6000. | Bolt/Omni video; release qualification. |
| Qwen3.8 Flash Next, MLX 6-bit | tensorfold pull TensorFold/Qwen3.8-Flash-Next-MLX-6bit-MTP | One M5 Ultra checked; RAM minimum not published. | Metal, runbook. |
| Qwen3.8 Flash Next, MLX 4-bit | tensorfold pull TensorFold/Qwen3.8-Flash-Next-MLX-4bit-MTP | One GB10; 73.3 GiB uploaded, plus mapped tables and caches. Metal has a native 4-bit path too (M3 Ultra checked). | Native CUDA. NVFP4/two-local-rank and vision paths are experimental. Metal's shared-session limits are in known issues. |
| GLM-5.3-Flash, MLX 4-bit | tensorfold pull TensorFold/GLM-5.3-Flash-MLX-4bit-MTP | Two M5 Ultras; single-Mac native serving is not qualified. | Paired Metal runbook. Do not use the Python recipe's single-Mac command. |
| GLM-5.3-Flash, EXL3/TR3 | tensorfold pull Mia-AiLab/GLM-5.3-Flash-EXL3-TR3-4bpw | Two CUDA ranks; the release checks the two-GB10 setup. | Release scope, checkpoint. Experimental checkpoint; use native rank options, not the Python installation. |
| GLM-5.3-Flash, NVFP4 | tensorfold pull nvidia/GLM-5.3-Flash-NVFP4@423acf37583782c51c142d145aef733d72943d93 | Two GB10s at TP=2: @Bizuayeu loaded the whole checkpoint on 1.1.0 in 302.1 s and served it. | Native CUDA; contributed run: one token hash per prompt across three runs and a passphrase found in 199,307 tokens; scored prompts and more than two ranks were not covered. |
| GLM-5.3-Flash, FP4-Local | tensorfold pull nvidia/GLM-5.3-Flash-FP4-Local@76348ddb2be2e35e39ecfc00e75dc07703f4f3d5 | Two GB10s at TP=2, or three at TP=3 over a ring of direct links: @Bizuayeu loaded it as stored and served it at both sizes. 165.3 GiB of weights, 13% less than NVFP4's. | Native CUDA, in 1.1.5 (#1050). NVIDIA's mixed-precision checkpoint: NVFP4 routed experts, FP8 for most of the rest. As close to BF16 as NVFP4 at its smaller size (receipt). |
| GLM-5.3-Flash, XSFP4 | tensorfold pull Bizuayeu/GLM-5.3-Flash-XSFP4@88736f916f8939607fde8a4b881dcae749f7551d | Two GB10s at TP=2, or three at TP=3 over a ring of direct links: @Bizuayeu served it at both sizes. 165.7 GiB of weights. | Native CUDA, in 1.1.5 through FP4-Local's path. FP4-Local with its routed experts refitted from the BF16 weights and a smoothing folded in, on the same kernels. Closer to BF16 than FP4-Local and NVFP4 by all three measures in its card. |
| GLM-5.3-Flash, Jammed XSFP4 | tensorfold pull Bizuayeu/GLM-5.3-Flash-Jammed-XSFP4@0a22bb46ecef3d0f5f96e41dc6011be12974fbf1 | Two GB10s at TP=2, or three at TP=3 over a ring of direct links: @Bizuayeu served it at both sizes. 160.2 GiB of weights; at TP=2 each GB10 holds 6.6 GiB less than at block 16. | Native CUDA, in 1.2.3 (#2041). Refitted from BF16 at block 32 with XSFP4's smoothing, each row's scales packed (csf0_v1, read only by this path), at XSFP4's speed. Its KL and top-1 are nearer BF16 than NVIDIA's two checkpoints', short of XSFP4's (its card). |
GLM-5.3, EXLR8 home | tensorfold pull kindlingai/glm-5.3-exlr8-k2-k3-k4@f459f5d0272a529403dd9b3fd32f771f4250a0d4 --exlr8 home --node 0/2 --dir ~/models/glm-5.3-exlr8-home | Two M5 Ultra Macs, 256 GB each; about 160 GB resident per Mac at 8k context / four streams. | Two-Mac recipe. Use --node 1/2 on the other Mac; Thunderbolt 5 and MCDMA required. No CUDA backend. |
| LFM2.5-24B-A2B, MLX 4 / 6 / 8-bit | tensorfold pull LiquidAI/LFM2.5-24B-A2B-MLX-4bit@667408d494bde191c2cb9dd4f4549827816b9275tensorfold pull LiquidAI/LFM2.5-24B-A2B-MLX-6bit@1deec53eb49571e21cdcbf82fd5839248f07ed88tensorfold pull LiquidAI/LFM2.5-24B-A2B-MLX-8bit@67ab417dd02d0c9545802a726df668620eae049d | One Mac (M4 Pro checked), or one NVIDIA GPU with the 4-bit export (GB10 checked). | Native recipe (CUDA), release history. Weights: LFM Open License v1.0. |
| Qwen3.6-35B-A3B, MLX with MTP | tensorfold pull TensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTP@81169a9bc511a27c1b4eedb77a2cd98ced431847 | One GB10; 20.9 GB checkpoint. | Qwen3.6. Native CUDA receipt here. Metal also registers this model type; the recipe's Metal commands describe Python 0.6.6. |
| Qwen3.6-35B-A3B, NVIDIA NVFP4 with MTP | tensorfold pull nvidia/Qwen3.6-35B-A3B-NVFP4@1355db6a052410cfd62085d94b58866fd0f2c3c5 | One GB10; 20.3 GiB on the device, 20.9 GiB with the drafter. | Native CUDA. Serve with --drafter naming the same checkpoint for MTP drafts. |
| Gemma 4 31B, MLX 4-bit | tensorfold pull mlx-community/gemma-4-31B-it-4bit@696d436c404745a59f30e4939a658162b0a9e57ftensorfold pull z-lab/gemma-4-31B-it-DFlash@eabd648301ce28583cc14757912e5e0f84e152e1 | One Mac (M5 Ultra checked), or one GB10 on CUDA. Pre-M5 Metal has a row path; its assistant and DFlash drafters require M5 tensor units. | Native 31B on Metal, CUDA qualification. Name the drafter with --drafter; images with --vision. |
| Gemma 4 12B, converted MLX 4-bit | hf download google/gemma-4-12B-it --revision 707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7 --local-dir ~/models/gemma-4-12b-hfhf download google/gemma-4-12B-it-assistant --revision 46d4c6f13f0ac0ad827b915669b8df9b81c64c51 --local-dir ~/models/gemma-4-12b-assistant | One Mac (M5 Ultra checked); converted text weights 6.3 GB, image weights 100 MB, plus assistant/caches. MLX 4-bit also served on one GB10. Pre-M5 Metal has a row path; the MTP assistant requires M5 tensor units. | Convert and serve. Upstream download needs hf; conversion needs Python, mlx and mlx-lm. Serving is native; CUDA qualification includes the MTP assistant. |
| Step-5-Preview, converted affine | hf download stepfun-ai/Step-5-Preview --local-dir ~/models/step5-preview-fp8 | One 256 GB Mac; text 166.9 GiB, MTP adds 2.5 GiB and vision 3.7 GiB. | Convert and serve. The documented 256 GB setup serves converted affine weights; converting the upstream FP8 download needs Python and mlx. |
| MiniMax-M3.1-Flash, community affine 3/4-bit | hf download BobClawblaw/MiniMax-M3.1-Flash-MLX-mix34 --revision 9524999057e0d277d26438606e033e2191fb465c --local-dir ~/models/MiniMax-M3.1-Flash-MLX-mix34hf download BobClawblaw/MiniMax-M3.1-Flash-MLX-mix256k --revision 916e7ca1044f1c3e7566163096243c940327224d --local-dir ~/models/MiniMax-M3.1-Flash-MLX-mix256k | One 256 GB M5 Ultra; 203.7 GiB (32,768-token window) or 189.2 GiB (147,456), plus caches. | MiniMax-M3.1-Flash. Set MINIMAX_LOAD_LIMIT_GB; images with --vision. Community checkpoints: lossy conversions of MiniMax's NVFP4 release (KL 0.244 / 0.295). MiniMax Community License. |
| Kolibri-1, block FP8 (research) | tensorfold pull Aleph-Alpha/Kolibri-1@e52eb4627d11516b0c01de49210ab5a4e4061444 | One NVIDIA GPU; GB10 measured; 74 GB checkpoint. | Kolibri. Text only; copy drafts, no EAGLE-3/neural drafter. |
| Liquid AI d1-3B, decisions | Not documented yet. | Native CUDA; RAM minimum not published. | API section below. /v1/systemone and /v1/decisions. |
| Fish Audio Drama 3, speech | Not documented yet. | Native Metal or CUDA; GB10 measured. | Release audio instructions, reference voices and dialogue speakers. /v1/audio/speech with sampling controls; checkpoint and codec.pth required. |
| Drama 3 pruned NVFP4, speech | hf download plotarmordev/drama-3-NVFP4-Pruned --revision 494db08481db952f834b2241bce63fb3dfa72554 --local-dir drama-3-NVFP4-Pruned | Native Metal; roughly 21 GiB checkpoint plus runtime state. Not a minimum-RAM or cross-backend quality claim. | Packed experts, BF16 dense/tower weights and FP32 accumulation; no conversion. Use serve drama-3-NVFP4-Pruned --cfg-gamma 1.5 --cfg-batched with an attested catalog voice. |
GLM-5.3: one machine or two?
The native release does not qualify GLM-5.3 on one machine. GLM-5.3 EXLR8 home needs two 256 GB M5 Ultra Macs;
its recipe gives both pull and serve commands. GLM-5.3-Flash is a different model:
its native Metal setup needs two M5 Ultras. Native CUDA has recorded TP=2 and TP=3 setups, including
FP4-Local and XSFP4 on two and three GB10s. The single-256-GB-Mac
GLM-5.3-Flash command in the older recipe is for Python 0.6.6, not this release.
For the NVFP4 path, see GLM-5.3-Flash on CUDA and what
the contributed run on two GB10s covered.
DeepSeek checkpoints are not served yet by any release binary: whole-model gate #827 is still open. MiMo's native backend is not in a release yet. Bonsai's ternary pack and Gemma 4 26B-A4B remain Python-only; they are not native download recommendations. Known issues lists current limits, including Flash Next shared-session cache/performance behavior and the 27B's drafter limits.
Exact decoding
Every accepted draft must equal the token the same native engine would produce with "draft": false.
A resumed request must equal fresh execution, and each concurrent stream must equal its solo run.
The comparison fixes the checkpoint, backend, settings and runtime.
Different quantizations and different backends can produce different outputs.
Flash Next on Metal uses shared target rounds with an explicit --parallel N above one, when that many sessions fit.
A lone greedy request there uses GPU-side rounds and hands over when a second request arrives.
--parallel auto (the default) and --parallel 1, learning and two-Mac serving use the single-reply driver.
FZ_BATCH_MTP_SERIAL=1 restores serial MTP drafting on the shared host.
The earlier MTP chain receipt records its own pinned commits.
Nemotron, GLM, the 2B model and Qwen3.8-27B use shared lane rounds.
On Metal, each of up to 128 streams in the 27B's shared rounds drafts its own DFlash2 tree.
Models on disk
tensorfold pull TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit
tensorfold models
tensorfold models --json
tensorfold info TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit
pull downloads a checkpoint into the Hugging Face cache and checks what it downloads against the Hub: large files by sha256, small files by their git blob sha1.
Large files come down in pieces over up to 16 connections, and an interrupted pull resumes with the pieces it lacks.
It refuses a checkpoint whose model_type has no native family, and fetches the 27B's DFlash2 drafter for --drafter. models lists the cached checkpoints with a model type it knows, and info shows one checkpoint's family, format, context and memory floor.
models --json prints the same list as one document for programs: schema 1, and each checkpoint's repo, family and model_type; family is a display title and model_type is the key to match on.
tensorfold serve REPO serves a cached checkpoint by its repository name.
Context compaction
Compaction is off by default, and with it off every reply is unchanged.
With --compact-at, a conversation that would overflow is compacted instead of refused.
The server keeps the system prompt and the most recent turns word for word, and never separates a tool call from its result.
The older turns become a memory note: goal, constraints, progress, decisions, next steps, exact names and numbers, and the files the tool calls read or changed.
Each later compaction updates the previous note instead of starting over. A client that sends back the compacted conversation keeps its note.
The reply carries the note and the compaction counts under tensorfold.compaction, and every reply reports context_window and context_used.
The same request gives the same compaction and the same reply. With --compact-memory, the stored note carries over between requests.
Cortheon mode
Off by default. --cortheon gives every chat request Cortheon's reasoning program and web research run by the server, so any OpenAI-compatible client gets it with no setup.
The model works through Cortheon's stages for its task type and can call cortheon_search and cortheon_fetch, which the server runs without the client seeing them.
A question about current facts, or a request to build with an outside library, is researched before the model starts.
A coding task that changed files is asked to run its tests before it finishes.
Web text is filtered for instructions aimed at the model, and reads reach public addresses only.
With the flag, search queries written from the conversation go to DuckDuckGo, or to Brave with BRAVE_SEARCH_API_KEY, and pages are fetched from this machine; without it, the server makes no searches and reads no pages.
See Cortheon mode.
Model discovery
GET /v1/models (also /models) advertises each served name and alias. When the engine reports a finite
context window, each entry includes equal context_length and max_model_len integer extensions. These
are the loaded engine's effective prompt-plus-reply token limit, including any startup memory fitting;
they are not the default reply limit. An unspecified engine window omits both fields. Clients should
still handle per-request memory refusals: an advertised context is not a guarantee that every workload fits.
A model that thinks also carries an OpenRouter-style reasoning: default_enabled, default_effort (only
with --reasoning-effort), and supported_efforts, the levels its template names, highest first, then none
when the template has a thinking switch. A template that names no level leaves supported_efforts out. These
describe chat and Responses requests.
Serve flags
The read-only /memory diagnostics report process footprint and lifetime peak,
available backend allocation counters, and the applied retained-prefix plan without changing memory policy.
The binary's capabilities --json response lists its supported flags and platform-specific values.
serve MODEL --help prints usage without loading a model.
| Flag | Meaning |
|---|---|
--host HOST, --port PORT | Listen address and port, default 127.0.0.1:8080. |
--profile stable | Explicitly preserve current defaults; profiles describes the initial switch registry and startup report. |
--name NAME | Model ID advertised to clients. |
--alias NAME | Another accepted model ID; repeat for multiple aliases. |
--api-key KEY | Require a bearer key; repeat for multiple keys. |
--api-key-file FILE | Read keys from a restricted-permission file; a key can be limited to some routes. |
--metrics-open | Allow /metrics, /v1/metrics and /v1/load without a key when API authentication is enabled. |
--dashboard | Enable the local /dashboard page and /stats endpoint. |
--max-inflight N, --max-inflight-per-key N | Refuse model requests past N in progress at once, overall or per key, with 429; 0, the default, is off. See admission. |
--requests-per-second N, --request-burst N | Refuse a key's model requests past N a second, after a burst; 0, the default, is off. |
--context N | Bound prompt plus reply tokens; the model's window and memory checks still apply. Qwen3.8-27B defaults to 32,768 and takes up to 262,144. |
--rope-yarn F | YaRN context extension with this factor over the checkpoint's trained window, for Qwen3.5 and 3.6 on a CUDA build; other builds refuse the flag. |
--loops N | Passes over a looped model's recurrent block, from 1; without it the checkpoint's own count runs. Other builds refuse the flag. |
--speed-up FILE | Rank and link settings for two-Mac Flash Next or GLM serving. |
--max-tokens N | Default reply limit, 4096; requests can override it. |
--temperature T | Sampling temperature; zero requests greedy decoding. |
--prompt-seed | Use native prompt-derived keys instead of the fresh OS fallback; request and configured default seeds still win. |
--top-p P, --top-k K, --min-p P | Sampling defaults. |
--thinking, --no-thinking | Choose whether the chat template opens a reasoning block. |
--reasoning-effort LEVEL | Default template effort: minimal, low, medium, high, xhigh or max. |
--thinking-budget N | Limit reasoning tokens where the engine supports closing the reasoning block. |
--thinking-reserve N | With thinking on, let reasoning use up to N tokens beyond a request's max_tokens, which then limits the answer (lane engines). |
--loop-guard | Close reasoning that repeats itself verbatim (a short cycle, or a long passage) where the engine supports it. |
--loop-guard-stop | As --loop-guard, but end the reply where the guard fires: the reasoning so far, no answer or tool calls, finish_reason content_filter and a reasoning_loop error (API). |
--no-drafts | Produce the plain reference through the same engine. |
--drafter DRAFTER | An external drafter, as a directory or a repo id in the Hugging Face cache, for models that take one: Qwen3.8-27B uses DFlash2 and Gemma 4 31B the drafter named in its recipe. Without one they decode plain. |
--drafter-bits 0|4 | Drafter weight precision where the model offers a choice: 4 bits (the default), or 0 to keep bf16. Qwen3.8-27B on CUDA takes 4 only. |
--record DIR | Write each stream's committed tokens, positions and tapped rows under DIR, for drafter training. CUDA Nemotron only; a single stream runs through the lane rounds, and recording stops below 25 GiB free disk. |
--keep-warm SECONDS | Keep the Metal GPU active while idle for this long after a request, default 900; zero disables it. |
--cfg-gamma GAMMA | Metal Drama speech: semantic guidance strength, finite 1–5; 0/default disables it. Only reference-conditioned requests are guided. Enabled replies report X-Cfg-Gamma; codec heads stay conditional. |
--cfg-allow-request-gamma | Metal Drama speech: allow a finite 1–5 X-Cfg-Gamma request header; invalid values return 400 before engine submission. Without this opt-in, the server's gamma is fixed. |
--cfg-force-guided | With an enabled Metal Drama guidance policy, exercise the unconditional state even at gamma 1. The semantic mix at 1 remains exact conditional-logit identity. |
--cfg-batched | With enabled Metal Drama guidance, submit both independent backbone states in one command buffer rather than separate submissions. Weights are shared; extra state is included in admission. Not vectorized batching. |
--silence-floor-dbfs DBFS | Drama speech only: explicit frame-power floor for the silence guard, with no default floor. Calibrate below quiet intended speech using public or built-in speech. |
--silence-seconds SECONDS | Drama speech only: stop after this continuous duration below the explicit floor, retaining audio and reporting silence. Unset by default: off; calibrate above intentional pauses. |
--silence-onset-s SECONDS | Drama speech only: with the silence guard on, stop once this many seconds run below its floor while the synthesis has voiced less than --silence-onset-voiced-s, retaining audio and reporting silence. Off unless given; must be below --silence-seconds. |
--silence-onset-voiced-s SECONDS | Drama speech only: the voiced seconds below which --silence-onset-s applies; required with it, with no default. Calibrate on public or built-in speech. |
--silence-mid-s SECONDS | Drama speech only: with the silence guard on, finish as silence when end of speech falls in a run this long below the floor, after the explicit voiced minimum. Whole replies carry the decimal X-Silence-Mid-Trips count, including zero. Must be below --silence-seconds and paired with --silence-mid-min-voiced-s. Unset by default: off. |
--silence-mid-min-voiced-s SECONDS | Required with --silence-mid-s: positive finite cumulative voiced duration before the mid rule applies, including equality. No default; calibrate with the other silence thresholds. |
--silence-redraws N | Drama speech only: draw a whole reply the silence guard ended again, up to N times, as seed salt + k or explicit request seed + k; N of 1 or more needs the guard. Unset by default: off, as is 0; streams are never redrawn. See speech redraws. |
--silence-redraw-budget-s SECONDS | Start a redraw only while elapsed time plus the last take's duration fits; a request's X-Deadline-S can lower it. No default budget. |
--silence-redraw-controls LIST | Redraws speak the input filtered to the listed controls, as --allowed-controls filters; an empty list removes every control. |
--short-reply-ratio R | Also redraw a whole reply that finished stop with less audio than R (above 0, below 1) of its text's expected length; needs --silence-redraws and --short-reply-chars-per-s. No default. |
--short-reply-chars-per-s CPS | Input characters per second of audio in complete replies of the served voice, setting the expected length; needs --short-reply-ratio. No default; calibrate on your own voice. |
--register-redraw-st ST | Drama speech only: draw a whole reply once more when a phrase after the first opens ST semitones above the other phrases' median opening; needs the two flags below. Unset by default: off, with no default for any of the three. See speech redraws. |
--register-redraw-high PERCENT | The share of that phrase's voiced frames that must lie above --register-high-hz. |
--register-high-hz HZ | The pitch line, 75 to 600 Hz, that marks a high register; calibrate it on your own voice. |
--parallel N|auto | Cap active requests at N and queue the rest. Default auto asks for up to 8 as memory and the engine allow; an explicit count that cannot fit is refused. Read the accepted count at startup. |
--prompt-cache-gib GIB | Retained-prefix budget, 16 GiB by default or less where memory is short; 0 disables it. |
--prompt-cache-over-cap | Permit an explicit prefix budget past what 70% of RAM leaves once the server loads. |
--learn | Keep shared prompt prefixes on disk for new conversations to resume; the models are below the table. |
--learn-dir DIR | Where --learn keeps them, ~/.cache/tensorfold/learned by default; implies --learn. |
--learn-gib GIB | Disk for learned prefixes on each Mac, 32 by default; the least recently used go first. Implies --learn. |
--learn-min-free-gib GIB | Free disk --learn leaves on each Mac, 4 by default; below it learning pauses and serving goes on. |
--learn-conversations | Keep every prompt state on disk too, each conversation's and the shared prefixes, in one store in place of --learn's, so a next turn after a restart, a crash or an eviction reads it back. Implies --learn; see Conversations on disk. |
--learn-daily-gib GIB | What --learn-conversations may write to disk a day, every model together, 32 by default; past it states stay in RAM only. |
--recall | Draft from a record of the token ids earlier requests read and wrote, so repeated text is verified in a few rounds; replies are unchanged. Off by default; see Recall for what stays on disk. |
--recall-dir DIR | The record's directory, one a tokenizer under ~/.cache/tensorfold/recall by default; implies --recall. |
--recall-gib GIB | Disk for the record, 1/16 GiB by default; the oldest ids go first. Implies --recall. |
--recall-read-only | Draft from the record without adding to it, as gates and benchmarks run; implies --recall. |
--snapshot-dir none | Keep prefix state in memory; none is the supported value. |
--max-snapshots 0 | Disable disk snapshots at startup; 0 is the supported value. |
--compact-at auto|FRACTION | Turn on context compaction (below). auto compacts when the prompt and reply would pass the window minus a reserve. |
--compact-keep TOKENS | Recent tokens kept word for word when compacting; the default is 20,000 or a quarter of the window, whichever is smaller. |
--compact-memory DIR | Also keep each conversation's memory note as a Markdown file in DIR. |
--cortheon | Cortheon's reasoning program and server-run web research for chat requests; see Cortheon mode. |
--no-update-check | Accepted compatibility switch; update the installed binary through its package manager. |
--backend auto | Use the backend compiled for the platform; mlx selects native Metal on macOS and cuda selects CUDA on Linux. |
--prompt-cache-gib applies to Nemotron, Flash Next, GLM, Qwen3.8-27B and LFM2.5-24B-A2B, and
--prompt-cache-over-cap to the same models but GLM.
--learn keeps shared prompt prefixes, such as a system prompt and its tools, on disk, so new conversations resume
them after a restart or an upgrade that computes the same bits. It covers GLM-5.3-Flash, Nemotron, and Flash Next on
one Mac; Qwen3.8-27B, Qwen3.5-2B, LFM2.5-24B-A2B, GLM-5.3 (EXLR8) and a CUDA server refuse it at startup.
Qwen3.8-27B on Metal also resumes a follow-up turn from the state its last reply left in the server's first slot,
when the prompt cache holds nothing as far: the turn reads only its new tokens, even on a Mac whose cache cannot keep
a long conversation. Where the cache cannot keep a whole context, the server also keeps that slot's DeltaNet states
at each prompt's history end, so a turn whose template drops the last reply's reasoning resumes there. Replies are
the same tokens as a fresh prompt pass; TF_QWEN27_SLOT_RESUME=0 turns both off.
Sampling defaults come from the checkpoint's generation_config.json, then serve flags and request fields override them.
--context 0 is family-specific; use a positive limit for GLM and inspect the capacity reported at startup.
The supported environment variables are listed in capabilities --json, including API keys, request logging and Hugging Face cache paths.
TF_Q27_PROFILE=N makes a native Metal Qwen3.8-27B server log a per-class GPU-time breakdown every N decode rounds (1 means 64) and add gpu_seconds to its replies; it is diagnostic only, and a profiled round runs slower.
Community benchmarks
tensorfold benchmark MODEL runs public code and chat fixtures on a cached model and saves a portable receipt;
--publish optionally submits it for review at tensorfold.dev. See benchmark commands and measurement definitions.
Build and contribute
Use the Zig version pinned in .zig-version, currently 0.17.0.
The .metal sources need Xcode's Metal compiler. Install full Xcode and its Metal Toolchain component
(xcodebuild -downloadComponent MetalToolchain); the Command Line Tools alone fail every kernel compile with
unable to find utility "metal".
zig build native -Dcpu=apple_m1
zig build test test-golden -Dcpu=apple_m1
Structured output and required tool calls need the release archive's lib/ or zig build grammar -Dxgrammar=DIR; prepare DIR with git clone --branch v0.2.8 --recurse-submodules https://github.com/mlc-ai/xgrammar (use the native build's same --prefix for grammar).
To build one stripped server for one GPU (zig build -Dgpu=spark -Dnvcc=..., about 69% smaller than the four-target server), see Build a server for one GPU.
On macOS, test also runs available Metal device checks. See CONTRIBUTING.md for the host-test subset.
The server is zig-out/native/bin/tensorfold-native.
Release archives include the native executable, runtime assets and license notices; packaging describes the qualified CUDA inputs.
For cache-aware performance receipts, see observed cache benchmark evidence.
The clients retain server usage and output fingerprints, and verify cold/reused states rather than inferring them.
Read CONTRIBUTING.md for exactness, precision and performance gates.
Coding agents should also load docs/recipes/SKILL.md; agents that support skills can install it as tensorfold-contributor.
TensorFold is Apache-2.0; see LICENSE, NOTICE and THIRD_PARTY_NOTICES.md.
Contributors
Thank you to everyone who has sent TensorFold a pull request, a measurement or a bug report:
@0mao0, @321sssrt-bit, @AbrahamCete, @aditya1503, @AdrianBinDC, @ajensenwaud, @akol1, @alenperic, @Alexbob0, @alexderz, @anvilsong, @Arminova, @b-ostrov, @BadAd84, @barelyworkingcode, @Benjamin-Wegener, @benthecarman, @benwilson, @BHCC2025, @Bizuayeu, @BlivionIaG, @BobClawblaw, @borodach23, @Boscoeuk, @boxabirds, @brandonmmusic-max, @bunnyfu, @CanadaApollo6, @CerebralCoding, @cesarswong, @chadhurley25075-png, @chaog992, @Charlie-Louis, @Chedrian07, @chris247474, @christrade215, @crescit, @cshintov, @cwschroeder, @DakotaTexas, @danyo1399, @Deesha08, @Defilan, @DevRico003, @di37, @diwakergupta, @drowzeys, @easthambuilds, @ecohash-co, @edurdias, @eleqtrizit, @ericlsimplifi, @EugeneClaw, @fche, @feni6, @florianbrede-ayet, @gbgbgbg, @GDACONSULT, @gecobattya, @gilby, @Gogo6969, @gprot42, @GraithSecurity, @grantoverton, @grearjake-star, @greatyingzi, @GustavBlack, @harrisonfriia, @haxudev, @heitke, @hichaiuse, @ivanfioravanti, @jasontitus, @javierbits, @jayleaton, @jeffpeng3, @jeidbugs404, @Jenpo, @jerryfane, @jetnet, @jishnuvenugopal, @jkuepker, @joeynyc, @johnymoo, @JordiPosthumus, @joshuaswarren, @JRaxworthy, @jrb-ai, @jschmied, @juliankang4, @justincasey, @kelchm, @kieranscotney, @kingjamez, @kky42, @laudney, @lcgutierrez, @LECYWZA, @lijian1999, @liumorrisclaw, @LXD-8, @m-naoki-m, @mapamalu, @mcclanahanaman, @MESevenJourney, @mgoldwasser, @MiaAI-Lab, @mikolaj92, @millaguie, @Mirrdhyn, @Moutonc, @MovieMaker93, @mrpmorris, @mrv777, @MV10, @NeoAiLabs, @Nipale-ai, @nmcbride, @nood-co1, @nullburn, @olexale, @omar16100, @optimisme, @otarkhan, @outcastofmusic, @paragontasx, @peacockesq, @philip-pentatonic, @plotarmordev, @pmeenan, @ppardus, @pulseandthread, @quigles1977, @rafafortes, @raymondkpwong, @robertpitt, @rootsbymenda, @RoscoeTT, @RS91Dev, @russellmm, @salmanarshad321, @samwang0041-star, @sanjaibalajee, @satindergrewal, @scottleimroth, @sethforprivacy, @sfxnz, @shantanugoel, @simon-lin88, @simonmd, @spenchey, @squarrier, @ss-cong, @Stephane-Eden, @styles01, @SvangenStudios, @swbiggart, @SxMShaDoW, @sxuff, @taussoe, @tfolkman, @ThinkOffApp, @ThomasHalwax, @Thotheris, @tinyapps, @tolewis, @tomByrer, @tonydehnke, @tournierjc, @tovereast, @tpischke, @urtho, @valacroix, @vcruz305, @vinicius-symetrix, @wojo, @xjqx2z, @yguanrong, @Yuepixel, @YvesLaRose, @ZackSample, @zweck.
docs/credits.md names whose Python engine work each part of the native engine carries.
Usable memory profiles explains how to record applied budgets, context, cache state and process peaks.