olla

thushanApache-2.0活跃维护

轻量级 LLM 基础设施代理和负载均衡器，能在本地与远程推理后端之间做智能路由、自动故障转移和统一模型发现。

High-performance lightweight proxy and load balancer for LLM infrastructure. Intelligent routing, automatic failover and unified model discovery across local and remote inference backends.

v0.1.0发布于 2026-11-01 00:19 UTC338star +2 / 7 天60fork同步于 2026-11-10 12:01 UTC

LLM 代理负载均衡智能路由故障转移推理后端

对比 GitHub 官网 ai amd golang intel llama-cpp

怎么上手

由 README 提炼，执行前请确认

原文

bash <(curl -s https://raw.githubusercontent.com/thushan/olla/main/install.sh)
docker run -t --name olla -p 40114:40114 ghcr.io/thushan/olla:latest

硬件要求按 README 说明

README 未说明具体硬件要求；Olla 本身是 Go 编写的代理程序，推理负载由后端承担，支持 x86 与 ARM64 平台。

支持 Apple Silicon有 Docker 镜像

下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼

没人工点评？写下你的经验

轻量高性能 LLM 推理代理与负载均衡器，统一路由本地和远程后端并自动故障转移

轻量级 LLM 基础设施代理和负载均衡器，能在本地与远程推理后端之间做智能路由、自动故障转移和统一模型发现。

核心能力

按优先级路由请求，连接失败时自动重试并无缝切换端点
KV-cache 感知的粘性会话，把多轮对话固定到同一后端
聚合各后端模型目录，提供 OpenAI 兼容的跨提供方路由
持续健康检查配合熔断器，端点恢复后自动刷新模型发现
内置 Sherpa 与 Olla 双代理引擎，可选熔断和连接池

容易踩的坑

默认监听 40114 端口，需与客户端配置一致；用 go install 安装不会附带前端管理面板；需自行准备并配置 Ollama、vLLM 等推理后端，Olla 本身不提供模型。

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

用户点评

登录用户的真实使用记录，可按版本和硬件筛选

共 0 条

还没有人点评。第一个说出真实体验的人最有价值。

README 原文

未翻译

在 GitHub 打开

Olla is a high-performance, low-overhead, low-latency proxy and load balancer for managing LLM infrastructure. It intelligently routes LLM requests across self-hosted inference nodes with a wide variety of natively supported endpoints and extensible enough to support others. Olla provides model discovery and unified model catalogues within each provider, enabling seamless routing to available models on compatible endpoints.

Olla works alongside API gateways like LiteLLM or orchestration platforms like GPUStack, focusing on making your existing LLM infrastructure reliable through intelligent routing and failover. You can choose between two proxy engines: Sherpa for simplicity and maintainability or Olla for maximum performance with advanced features like circuit breakers and connection pooling.

Olla Single OpenAI

Single CLI application and config file is all you need to go Olla!

For large GPU deployments, enterprise and data centre use, see TensorFoundry FoundryOS. For an inference control plane, consider Alloy.

Key Features

🔄 Smart Load Balancing: Priority-based routing with automatic failover and connection retry
📌 Sticky Sessions: KV-cache-aware affinity routing that pins multi-turn conversations to the same backend
🔍 Smart Model Unification: Per-provider unification + OpenAI-compatible cross-provider routing
⚡ Dual Proxy Engines: Sherpa (simple) and Olla (high-performance)
🎯 Advanced Filtering: Profile and model filtering with glob patterns for precise control
💊 Health Monitoring: Continuous endpoint health checks with circuit breakers and automatic recovery
🔁 Intelligent Retry: Automatic retry on connection failures with immediate transparent endpoint failover, tunable with proxy.retry
🔧 Self-Healing: Automatic model discovery refresh when endpoints recover
📊 Request Tracking: Detailed response headers and statistics
⚡🔄 Anthropic Messages API: Passthrough for backends with native support; automatic translation for others
🧬 Embeddings and Rerank: Text Embeddings Inference support, with model-less embedding and rerank requests routed to a host serving that kind
🔐 Backend Authentication: Outbound auth for local backends requiring API keys, bearer tokens, or basic auth
🛡️ Production Ready: Rate limiting, request size limits, optional CORS for browser clients, configurable proxy/server timeouts, graceful shutdown
⚡ High Performance: Sub-millisecond endpoint selection with lock-free atomic stats
🎯 LLM-Optimised: Streaming-first design with optimised timeouts for long inference
⚙️ High Performance: Designed to be very lightweight & efficient, runs on less than 50Mb RAM.
📈 Admin Dashboard: Read-only fleet overview embedded in the binary at /internal/ui/, on by default and loopback-only

Herd Dashboard

Olla ships with an embedded (v0.0.29+), read-only dashboard at /internal/ui/ - fleet health, a filterable herd table with selectable columns, sticky session, translator and process cards, and the discovered model inventory, served straight from the binary. No extra port, no separate web server, on by default and loopback-only.

Start Olla and open http://localhost:40114/internal/ui/ to see your herd.

Supported Backends

Olla proxies the following self-hosted inference backends. The Anthropic API column shows whether /olla/anthropic/ requests are forwarded as-is (native passthrough) or translated from Anthropic format to OpenAI format on the fly.

Backend	Docs	Anthropic API
Ollama	Integration	⚡ Passthrough (v0.14.0+)
LM Studio	Integration	⚡ Passthrough (v0.4.1+)
vLLM	Integration	⚡ Passthrough (v0.11.1+)
vLLM-MLX	Integration	⚡ Passthrough
llama.cpp	Integration	⚡ Passthrough (b4847+)
Docker Model Runner	Integration	⚡ Passthrough
oMLX	Integration	⚡ Passthrough (v0.4.2+)
SGLang	Integration	⚡ Passthrough (v0.5.9+)
LMDeploy	Integration	⚡ Passthrough (v0.13.0+)
Lemonade SDK	Integration	🔄 Translation
LiteLLM	Integration	🔄 Translation
OpenAI-compatible (generic)	Integration	🔄 Translation
Text Embeddings Inference	Integration	N/A (no chat or text generation)

⚡ Passthrough: Olla forwards Anthropic-format requests directly; the backend handles them natively, with no translation overhead.
🔄 Translation: Olla converts Anthropic ↔ OpenAI format automatically; any OpenAI-compatible backend works transparently.

LiteLLM is the recommended bridge when you need Olla to reach hosted cloud APIs (OpenAI, Anthropic, Bedrock, etc.). Learn more about how they compare Olla vs LiteLLM and where each fits, where each wins etc.

Platform Support

Olla runs on multiple platforms and architectures:

Platform	AMD64	ARM64	Notes
Linux	✅	✅	Full support including Raspberry Pi 4+
macOS	✅	✅	Intel and Apple Silicon (M1/M2/M3/M4)
Windows	✅	✅	Windows 10/11 and Windows on ARM
Docker	✅	✅	Multi-architecture images (amd64/arm64)

Quick Start

Installation

# Download latest release (auto-detects your platform)
bash <(curl -s https://raw.githubusercontent.com/thushan/olla/main/install.sh)

# Docker (automatically pulls correct architecture)
docker run -t \
  --name olla \
  -p 40114:40114 \
  ghcr.io/thushan/olla:latest

# Or explicitly specify platform (e.g., for ARM64)
docker run --platform linux/arm64 -t \
  --name olla \
  -p 40114:40114 \
  ghcr.io/thushan/olla:latest

You can still use the go install method, but you won't get the bundled front-end, but you can see Admin Dashboard: building the frontend.

# Install via Go
go install github.com/thushan/olla@latest

Or you can build and run it from source:

# Build from source
git clone https://github.com/thushan/olla.git && cd olla && make build-release
# Run Olla
./bin/olla

# Build Docker image locally (no goreleaser required)
git clone https://github.com/thushan/olla.git && cd olla && make docker-build-local
docker run -p 40114:40114 ghcr.io/thushan/olla:local-amd64

Verification

When you have everything running, you can check it's all working with:

# Check health of Olla
curl http://localhost:40114/internal/health

# Check endpoints
curl http://localhost:40114/internal/status/endpoints

# Check models available
curl http://localhost:40114/internal/status/models

For detailed installation and deployment options, see Getting Started Guide.

Querying Olla

Olla exposes multiple API paths depending on your use case:

Path	Format	Use Case
`/olla/proxy/`	OpenAI	Routes to any backend, universal endpoint
`/olla/anthropic/`	Anthropic	Claude-compatible clients (passthrough or translated)
`/olla/openai/`	OpenAI	OpenAI-compatible route (`type: "openai"` is an alias for `openai-compatible`)
`/olla/{provider}/`	OpenAI	Target a specific backend type (e.g. `/olla/vllm/`, `/olla/ollama/`)

OpenAI-Compatible (Universal Proxy)

Use /olla/proxy/ as the universal entry point that routes to any backend.

# Chat completion (routes to best available backend)
curl http://localhost:40114/olla/proxy/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama3.2", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 100}'

# Streaming
curl http://localhost:40114/olla/proxy/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama3.2", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 100, "stream": true}'

# List all models across backends
curl http://localhost:40114/olla/proxy/v1/models

Anthropic Messages API

You can use /olla/anthropic

# Chat completion (passthrough for supported backends, translated for others)
curl http://localhost:40114/olla/anthropic/v1/messages \
  -H "Content-Type: application/json" \
  -H "x-api-key: not-needed" \
  -H "anthropic-version: 2023-06-01" \
  -d '{"model": "llama3.2", "max_tokens": 100, "messages": [{"role": "user", "content": "Hello"}]}'

# Streaming
curl http://localhost:40114/olla/anthropic/v1/messages \
  -H "Content-Type: application/json" \
  -H "x-api-key: not-needed" \
  -H "anthropic-version: 2023-06-01" \
  -d '{"model": "llama3.2", "max_tokens": 100, "messages": [{"role": "user", "content": "Hello"}], "stream": true}'

Provider-Specific Endpoints

Use provider specific patterns (Eg. /olla/ollama)

# Target a specific backend type directly
curl http://localhost:40114/olla/ollama/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama3.2", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 100}'

# Other providers: /olla/vllm/, /olla/vllm-mlx/, /olla/lm-studio/, /olla/llamacpp/, etc.

Examples

We've also got ready-to-use Docker Compose setups for common scenarios:

Common Architectures

Home Lab: Olla → Multiple Ollama (or OpenAI Compatible - eg. vLLM) instances across your machines
Hybrid Cloud: Olla → Local endpoints + LiteLLM → Cloud APIs (OpenAI, Anthropic, Bedrock, etc.)
Enterprise: Olla → GPUStack cluster + vLLM servers + LiteLLM (cloud overflow)
Development: Olla → Local + Shared team endpoints + LiteLLM (API access)

See integration patterns for detailed architectures.

For a robust enterprise setup, consider TensorFoundry FoundryOS.

🌐 OpenWebUI Integration

Complete setup with OpenWebUI + Olla load balancing multiple Ollama instances or unify all OpenAI compatible models.

See: examples/ollama-openwebui/
Services: OpenWebUI (web UI) + Olla (proxy/load balancer)
Use Case: Web interface with intelligent load balancing across multiple Ollama servers with Olla

Quick Start:

cd examples/ollama-openwebui
# Edit olla.yaml to configure your Ollama endpoints
docker compose up -d
# Access OpenWebUI at http://localhost:3000

You can learn more about OpenWebUI Ollama with Olla or see OpenWebUI OpenAI with Olla.

🤖 Anthropic Message API / CLI Tools - Claude Code, OpenCode, Crush

Olla's Anthropic Messages API support (v0.0.20+) is enabled by default, allowing you to use CLI tools like Claude Code with local AI models on your machine via /olla/anthropic. It operates in two modes depending on your backend:

⚡ Passthrough: requests are forwarded as-is for backends with native Anthropic support (vLLM, vLLM-MLX, SGLang, LMDeploy, llama.cpp, Ollama, LM Studio, Docker Model Runner, oMLX)
🔄 Translation: Anthropic ↔ OpenAI format conversion for backends that don't natively support the Anthropic Messages API

Still actively being improved -- please report any issues or feedback.

We have examples for:

Learn more about Anthropic API Translation.

Documentation

Full documentation is available at https://thushan.github.io/olla/

Getting Started - Getting Started with Olla
Integrations - See which LLM backends are supported by Olla
Comparisons - Compare with LiteLLM, GPUStack, LocalAI
Olla Concepts - Understand Key Olla concepts
Configuration - Extensive configuration documentation
API Reference - Olla System API Reference
Development - Contributing and development guide

Guides

In-depth guides on the TensorFoundry blog:

What is an LLM Proxy? - Why you end up needing one
Olla vs LiteLLM - Where each fits, where each wins
Deploying LLMs on Your Own Infrastructure - A practical guide
LLM Inference Servers Compared - vLLM, SGLang, llama.cpp and Ollama

🤝 Contributing

We welcome contributions! Please open an issue first to discuss major changes.

🤖 AI Disclosure

This project has been built with the assistance of AI tools for documentation, test refinement, and code reviews.

We've utilised GitHub Copilot, Anthropic Claude, JetBrains Junie, Codex & TensorFoundry Kaizen for documentation, code reviews, test refinement and troubleshooting.

We also utilise CodeRabbit for AI-driven code reviews for PRs prior to human review.

🙏 Acknowledgements

@pterm/pterm - Terminal UI framework
@puzpuzpuz/xsync - High-performance concurrent maps
@golangci/golangci-lint - Go linting
@dkorunic/betteralign - Struct alignment optimisation

📄 License

Licensed under the Apache License 2.0. See LICENSE for details.

Built by TensorFoundry, Australian AI infrastructure for running LLMs on your own hardware.

🌐 TensorFoundry • 📖 Documentation • 🐛 Issues • 🚀 Releases

Made with ❤️ for the LLM community

怎么上手

项目说明

核心能力

容易踩的坑

最近发版

What's in this release

Quick Start

What's New in v0.1.0

Behaviour Changes

What's in this release

Quick Start

What's New in v0.0.29

Admin Dashboard

Observability

Reliability

Quick Start

What's New in v0.0.28

New Backends & Compatibility

Authentication & Security

CORS

Anthropic / Translation

Configuration Tunables

What's in this release

Quick Start

Changelog

Other

What's in this release

Quick Start

Changelog

Other

用户点评

README 原文

Key Features

Herd Dashboard

Supported Backends

Platform Support

Quick Start

Installation

Verification

Querying Olla

OpenAI-Compatible (Universal Proxy)

Anthropic Messages API

Provider-Specific Endpoints

Examples

Common Architectures

🌐 OpenWebUI Integration

🤖 Anthropic Message API / CLI Tools - Claude Code, OpenCode, Crush

Documentation

Guides

🤝 Contributing

🤖 AI Disclosure

🙏 Acknowledgements

📄 License

相关项目