FastFlowLM

This release does four things:

• New models. The Decision 3.0 (d3) family from vLLM Semantic Router: five decision models from 0.8B to 27B, served through a new /v1/systemone endpoint. Also LFM3-1.7B and Nemotron 3.5 Bolt E4B.
• The prefix cache now covers almost every chat model. flm serve keeps several saved prompts at once, so agents that share a system prompt and tools, or switch between conversations, don't re-read the whole prompt on every request.
• More OpenAI API fields: logprobs / top_logprobs, seed, continue_final_message and repeat_last_n.
• max_tokens is exact. A reply used to be one token longer than max_tokens asked for.

It also fixes a gemma4-it:12b crash on repeated image or audio requests.

🆕 New Models

🧭 Decision 3.0 (d3): d3:lite, d3:nano, d3:mini, d3:turbo, d3:ultra

d3 is the decision model that vLLM Semantic Router calls for routing, guardrails and tool selection. It doesn't generate text. You send it a state (any JSON) and one or more questions, and it returns a probability for every option:

• choice: pick one of up to 255 options
• noul: no / yes
• score: 2 to 10 ordered levels

FLM serves the same HTTP API as the upstream d3 server, so vLLM Semantic Router can use FLM as its d3 backend without changes.

A decision model runs in server mode, on its own or next to a chat model:

flm serve --decision d3:lite                 # decision model only
flm serve qwen3.5:4b --decision d3:lite      # next to a chat model

* About 330 tokens, over HTTP, on an idle machine.

• Endpoints.
  • POST /v1/systemone (alias /v1/decisions) answers decision requests.
  • GET /v1/systemone/models lists the decision models this build can serve.
• Input limit: 32,768 tokens per question. A longer prompt gets max_length_exceeded for that question. Nothing is truncated.
• Accuracy. On the Decision Index public suite, every model stays close to the upstream BF16 model. Each model's numbers are on the benchmark page.
• d3:turbo and d3:ultra:
  • The state is computed once per request. Every question then reads it, so many questions about one state cost little more than one. That is 2.5× faster at 16 questions and up to 32× with a 4k-token state.
  • box and track questions locate objects in images and follow them through videos. assets put images in questions, options and instructions.
  • Videos need the FFmpeg shared libraries (FFmpeg 6, 7 or 8) on the machine. FLM doesn't ship FFmpeg. It loads it the first time a video arrives. Without it, encoded videos get a clear 422, and frame lists ({"frames": [...]}) still work. See the d3 page for install steps.
  • Memory. Plan for about 28 GB of free memory.
  • long_context: true is refused with a 422.

📖 d3 models · d3 benchmarks

💧 LFM3-1.7B: lfm3:1.7b

LiquidAI LFM3-1.7B runs fully on the NPU, prefill, all 24 layers and the LM head included.

• Thinking is on by default. You can turn it off.
• Supports tool calling.
• Context: up to 32k tokens.
• Uses LiquidAI's Q4_0 quantization-aware b

This release does three things:

• Reasoning, tool calls and structured output are handled one way for every model on /v1/chat/completions.
• tool_choice and response_format are enforced while the model generates, not just requested.
• Whisper transcription is more accurate, and /v1/audio/transcriptions now supports language, prompt and response_format.

It also fixes several flm serve bugs, including Lemonade failing to launch FLM on Linux.

✨ New Features

🧩 OpenAI API: Shared Chat Parsing and Constrained Decoding

PR #790

Every model writes its answer in its own markup. Qwen3 uses <think> and <tool_call>, Gemma 4 uses a thought channel and call:f{…}, GPT-OSS uses Harmony channels, and LFM2 writes Python calls. Until now each model had its own parser for this. In v1.0.10, one shared parser handles all of them, so /v1/chat/completions returns the same OpenAI-shaped message whatever the model:

• reasoning_content, content and tool_calls are split the same way for every model, streamed or not.
• The streamed deltas add up to exactly the non-streamed reply.
• Tool-call markup counts as a call only when the request includes tools. A reply that is only tool calls has content: null, as in the OpenAI API.

Constrained decoding. Before each token is sampled, FLM works out which tokens can still lead to valid output and rules out the rest. This uses xgrammar. The model can't misspell a tool name, give an argument the wrong type, leave out a required field, or wrap your JSON in a Markdown fence. You control it with the standard OpenAI request fields:

Bad values return a clear 400 error instead of being ignored. This covers a malformed tool_choice, an unknown function name, "required" with no tools, and a schema that can't be compiled. Small models that don't see tools in their prompt (Qwen3 up to 1.7B, Qwen3.5 0.8B) also return a 400 for a forced tool call, instead of making up a call to a tool they never saw.

When the model's thinking is off, it can no longer slip its answer into a reasoning block. It answers directly in the requested format.

Coming later: a grammar request field, so you can pass your own grammar for output that isn't JSON (for example, digits only or a fixed set of labels). The constrained-decoding engine in this release already supports this. Only the API field is still to come.

Other API changes:

• Streaming usage now follows OpenAI. A streamed response carries usage only when the request sets stream_options: {"include_usage": true}. In that case the usage comes in one final chunk before [DONE]. ⚠️ If your client read usage from streamed responses, add include_usage.
• null request fields are treated as unset. Some OpenAI clients send null for every option they leave out. Before, this could return a 400 or 500 error, for example for parallel_tool_calls: null or tool_choice: null.

New docs:

• Reasoning, Tool Calls & Streaming
• Structured Output & tool_choice

🎙️ Audio Transcription: /v1/audio/transcriptions

PR #794, by community contributor [@quine00](https://github.com

This is a maintenance release. It fixes flm validate on Linux, which wrongly reported "No NPU device found." in v1.0.8.

🐛 Bug Fixes

✅ flm validate Finds the NPU Again on Linux

PR #796

In v1.0.8, flm validate on Linux printed "No NPU device found." and exited with status 1, even on systems where the NPU worked fine and flm run and flm serve had no problems.

The cause: since v1.0.8, flm opens the NPU device at startup, before it runs any command. flm validate then tried to open the same /dev/accel/accel* device a second time to check it. The amdxdna driver allows only one open of a device per process, so the second open failed with EBUSY. The check skipped busy devices without saying anything, so it found no device at all.

What changed:

• flm validate now runs before flm opens the NPU, so its check gets the device.
• If a device node is busy, flm validate now reports it (device busy) instead of skipping it silently. The --json output marks it with "busy": true.

If you run flm validate in a setup script or CI job, upgrade to get the right result back.

📦 MSI Installer: Uninstall v1.0.8 First

The v1.0.8 MSI (flm-setup.msi) was published with the wrong version number. Windows registered it as version 8 instead of 1.0.8, so it shows up as "8" in Installed apps. Because 8 is higher than 1.0.9, the v1.0.9 MSI treats v1.0.8 as a newer install and stops with "A newer version of flm is already installed."

If you installed v1.0.8 with the MSI, uninstall FLM from Settings → Apps → Installed apps first, then run the v1.0.9 MSI. Your downloaded models are kept. The v1.0.9 MSI has the right version number, so later upgrades will work normally again.

This only affects the MSI. The portable .zip packages and the Linux packages are not affected.

🌟 Summary

Thanks for your support — see you in the next one! 🚀

This release adds minicpm-v-4.7:1b, a 1B vision-language model with toggleable thinking and tool calling. It also opens up the source of the Gemma 4 engine, so you can build it yourself and swap individual operators through new override hooks. And for the first time, the release ships HRX packages as a preview: builds of FastFlowLM that run on the HRX NPU runtime instead of XRT.

📦 New Model Support

👁️ MiniCPM-V-4.7-1B

FastFlowLM now supports minicpm-v-4.7:1b, a compact image-text-to-text model from OpenBMB, with toggleable thinking and tool calling.

• Tag: minicpm-v-4.7:1b
• Context: 32k by default, up to 256k

Run in CLI mode:

flm run minicpm-v-4.7:1b

Run in server mode:

flm serve minicpm-v-4.7:1b

🖼️ How It Sees Images

MiniCPM-V-4.7 handles images the same way the original MiniCPM-V does:

• The model always sees a downscaled view of the whole image. Large images are also split into up to 9 tiles of about 448 × 448 pixels each.
• Image TTFT stops growing at about 1080p. From that size up, every image uses the maximum 9 tiles, so a 4K image costs about the same as a 1080p one. Very fine detail in large images is downscaled.
• --img-pre-resize (-r) has no effect on this model, because the tiling already decides the input resolution.
• Video and audio input are not supported.

🎛️ MiniCPM-V does not ship its own sampling defaults. FLM starts from temperature 0.7, top_p 0.8, top_k 20 and presence penalty 1.5. These are the Qwen3.5 non-thinking defaults, because MiniCPM-V-4.7-1B's language model has the same architecture as Qwen3.5-0.8B.

For more details, see the model card and benchmark results.

🔓 The Gemma 4 Engine Is Now Open Source

PR #763

Until now, every model engine in FastFlowLM shipped only as a prebuilt library. In this release the source of the Gemma 4 E2B/E4B engine is public. It covers the text decode and prefill paths, the vision encoder and the audio encoder, plus the shared host code they depend on. You can now read exactly how each Gemma 4 operator is dispatched to the NPU, and build the engine yourself. The other engines, including the Gemma 4 flash and 12B variants, still ship prebuilt.

🛠️ Build It from Source

Source builds are opt-in. By default, FLM still links the prebuilt engine, so nothing changes unless you ask for it.

Linux:

cd src
cmake --preset linux-default -DFLM_BUILD_GEMMA4E=ON
cmake --build build

Windows (developer command prompt):

cd src
cmake --preset windows-default -DFLM_BUILD_GEMMA4E=ON
cmake --build build

The source-built library goes into the build tree and does not overwrite the prebuilt one. CMake caches the option, so to switch back, reconfigure with -DFLM_BUILD_GEMMA4E=OFF or use a fresh build directory.

Source builds work on Linux, Windows and the HRX backend. CI now builds and sweep-tests a source-built Gemma 4 package on both Linux and Windows.

🧩 Operator Override Hooks for IRON

Every Gemma 4 operator dispatch (52 sites across text, vision, audio and weight preload) is now wrapped in an FLM_OVERRIDE annotation. In a normal build the annotation expan