OlliteRT

Added

Speculative Decoding Support (Multi-Token Prediction, MTP)

• Per-model toggle — Inference Settings sheet exposes an MTP toggle on capable models, off by default
• Three-gate activation — allowlist capability, runtime Capabilities API verification, and the user toggle must all agree before MTP is enabled at Engine construction
• Benchmark integration — MTP toggle on the benchmark screen
• Import dialog support — user-imported MTP-capable models can declare and toggle the capability
• Observability — speculative_decoding field on /v1/models, ollitert_model_speculative_decoding_enabled Prometheus gauge, MTP pill on the Status screen, Home Assistant sensor config
• Gemma 4 E2B / E4B — model files updated to MTP-enabled variants, updatableModelFiles carries existing users forward
• Documentation — capability rows added to model tables, FAQ entry explaining what speculative decoding is, allowlist schema doc updated
• See What is speculative decoding / MTP for more information

[!NOTE] Speculative decoding requires to re-download already downloaded Gemma-4 models from previous version. Update button will automatically appear upon installing this version. Downloading new Gemma-4 models will automatically download latest version with speculative decoding support.

NPU / TPU Accelerator Support

• Pixel TPU support for imported models — NPU offered as a selectable accelerator at model import time
• Pixel device renaming — accelerator labels and descriptions show "TPU" on all Pixel devices via isPixelDevice()

KV Cache Reuse Across Conversation Turns

• Incremental dispatch — when consecutive chat-completion requests on the same model share system prompt and tools, only the new user turn is dispatched to the SDK so the LiteRT-LM KV cache is preserved (substantial latency reduction on long multi-turn conversations)
• Strict invalidation — cache is dropped on any inference error, on tool-use turns, on multimodal turns, on size mismatch, or when the system prompt or tool set changes
• Auditable decisions — verbose-debug logs every per-request request_incremental decision=EXTEND|RESET reason=… line for diagnostics

Anthropic Messages API (Experimental)

• /v1/messages endpoint — non-streaming and streaming, mirrors Anthropic's Messages API including message_start → content_block_* → message_delta → message_stop SSE sequence
• /v1/messages/count_tokens endpoint — works with no model loaded, uses the same character-based token estimator as the rest of the server
• x-api-key auth — accepted alongside Authorization: Bearer so Anthropic SDKs work without a server-side proxy
• Per-request thinking override — Anthropic thinking.type=enabled|disabled overrides the model default, silently downgraded when the model has no thinking capability
• ping heartbeat events — emitted every 10 s during prefill so SD

Added

Native Tool Calling (Schema Injection)

• Schema injection mode — tool schemas are registered directly with the LiteRT SDK for structured output, producing significantly more reliable tool calls than prompt-based parsing in 0.9.0-beta.1, see demo with Home Assistant Integration
• Real-time streaming with tools — stream tokens in real-time even when native tool calling is active
• Tool call indicator — log cards show a tool badge when the response contains tool calls
• Tools capability — models with native tool calling support now show tools capability chip in Model Cards
• Tools filter — filter the Models screen to show only models that support tool calling

[!NOTE] Native tool calling is currently supported only by Gemma-4 models

Multi-Repository Model Sources

Model sources are independently-hosted model lists — similar to how F-Droid repositories work for apps. Each source provides a set of models that OlliteRT refreshes automatically for new models. Once added, new models appear on the Models screen as they're published — no app update required, as long as the bundled SDK supports them. See SDK Compatibility.

• Multiple model sources — add, enable, disable, and remove custom model sources independently in Model Sources screen
• Add sources from file or URL — the + button now offers both a local JSON file picker and URL input
• Repository detail screen — tap any source to see all models it provides, including incompatible ones (greyed out with reason)
• Incompatible model visibility — sources show "N models · M hidden" when some models require a newer app version
• Model deduplication — models with same name across sources are shown only once — Official Source takes priority

Cross-Channel Update Notifications

• Cross-channel awareness — get notified about beta/dev releases while on stable (or vice versa)
• Per-channel notification channels — separate Android notification channels for beta and dev releases
• Dismiss per version — dismiss a notification for a specific version without disabling all notifications

Model Behaviour Settings Card

• Dedicated card — thinking token limit, thread pool, inference timeout, idle unload, and tool calling settings grouped together
• Schema injection toggle — enable/disable native tool calling per preference
• Reject when busy — optional - return error 503 instead of queueing when inference is running
• Transcription enabled by default — force transcription prompt is now on by default, with an editable prompt to guide how the model processes audio input

[!NOTE] The transcription prompt guides the model's behavior but does not guarantee accurate transcription — results depend on model capability and audio quality. Used by clients like Open WebUI and Home Assistant when sending audio input. Home Assistant requires a custom integration — see Voice Transcription (STT) for setup.

Monitoring & Metrics

• **Processing

[!IMPORTANT] This is a beta release. The beta and stable flavors use different application IDs and install side-by-side — they do not share data. When the first stable release ships, you will need to install the stable APK separately and re-download your models.

Features

Core

• Hardware-accelerated inference — runs on your phone's GPU or CPU via Google's LiteRT LM runtime
• Streaming — SSE and non-streaming response modes
• Configurable listen port — default 8000, binds to all interfaces
• Keep-alive idle timeout — (configurable) auto-unloads model after inactivity, reloads on next request

Multimodal and Reasoning

• Vision — send images as base64 data URIs in chat messages
• Audio — inline audio or Whisper-compatible /v1/audio/transcriptions endpoint with automatic stereo-to-mono downmix
• Thinking — chain-of-thought mode with <think> tag output (Gemma 4)
• Tool calling — (experimental) runtime tool schemas injected via system prompt

Models

• One-tap download — grab models from HuggingFace, import .litertlm from local storage, or add custom model sources via JSON URL with ~24h auto-refresh
• Search and sort — filter models by name, sort by size, alphabetical, or default order
• Per-model inference settings — temperature, top-K, top-P, max tokens, thinking mode, system prompt
• Accelerator selection — choose GPU or CPU per model
• Model updates — automatic detection with notifications and per-model update indicators
• Benchmarking — prefill speed, decode speed, TTFB, init time with shareable results

Always On

• Auto-start on boot and app launch — (configurable) server launches automatically with your configured default model
• Foreground notification — Stop Server and Copy URL quick actions
• Battery optimization exemption — prompt to prevent Android from killing the service

Server Configuration

• Authentication — optional bearer token auth and CORS support
• Custom system prompts — global and per-model
• Client sampler override — force server-side temperature/top-k/top-p
• Context management — (configurable) automatic overflow handling: truncate history, compact tool schemas, trim prompt
• Log persistence — local database with configurable retention and auto-pruning

Monitoring and Observability

• Status screen — model name, uptime, request count, decode speed, TTFB, context utilization
• Logs screen — per-request cards with timing, token counts, expandable bodies, JSON highlighting, real-time preview, search, filtering, and log export
• Prometheus metrics — /metrics endpoint with 29 metrics (memory, throughput, latency, errors)
• Health endpoint — extended metrics for Home Assistant REST sensors
• Remote server control — REST API endpoints for server control via Home Assistant

App

• Automatic update checks — via GitHub Releases
• HuggingFace OAuth — sign in to download gated models
• Report Issue — autofills GitHub issue template with relevant device info

Integrations