ai-gateway

High-performance AI gateway in Go. Route LLM requests across 30 providers via a single OpenAI-compatible API.

📖 Documentation: docs.ferrolabs.ai

🔀 30 providers, 2,500+ models — one API ⚡ 13,925 RPS at 1,000 concurrent users (v1.0.0 benchmark) 📦 Single static binary, no external services required, 32 MB base memory

---

Quick Start

Under two minutes from install to first response.

Platform / tool	Install
macOS, Linux	curl -fsSL https://get.ferrolabs.ai | sh
Windows	irm https://get.ferrolabs.ai/install.ps1 | iex
Homebrew	brew install ferro-labs/tap/ferrogw
Scoop	scoop bucket add ferrolabs https://github.com/ferro-labs/homebrew-tap then scoop install ferrogw
npm	npm install -g ferrogw
Python	uv tool install ferrogw
Docker	docker run -p 8080:8080 ghcr.io/ferro-labs/ai-gateway:latest
Go	go install github.com/ferro-labs/ai-gateway/cmd/ferrogw@latest — builds from source, without the dashboard
Debian, RPM, Alpine	.deb, .rpm and .apk packages on the releases page

Then go from nothing to a served request:

export OPENAI_API_KEY=sk-your-key     # ferrogw init detects this and writes the matching target
ferrogw init                          # writes config.yaml, prints your master key
export GATEWAY_CONFIG=./config.yaml   # the server reads a config file only when this is set
export MASTER_KEY=fgw_your-master-key # the key ferrogw init printed
ferrogw serve                         # starts the server on :8080

ferrogw init prints the master key once and never writes it to disk — save it yourself, in your .env file or a secret manager.

Docker runs the server itself, so there is no ferrogw init to print a master key — choose your own. Pass both variables by name so their values stay off the command line, where ps would show them:

export OPENAI_API_KEY=sk-your-key
export MASTER_KEY=fgw-any-strong-secret-you-choose
docker run -p 8080:8080 -e OPENAI_API_KEY -e MASTER_KEY ghcr.io/ferro-labs/ai-gateway:latest

Passing them by name keeps the values out of process arguments only — they are still readable in the container's environment and through docker inspect, so use Docker secrets or your platform's secret storage in production.

First request

export MASTER_KEY=fgw_your-master-key   # the key ferrogw init printed, in whichever shell you curl from

curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "Hello from Ferro Labs AI Gateway"}]
  }' | jq

Every release is signed with keyless cosign and ships an SPDX SBOM — verification steps are in SECURITY.md.

---

Why Ferro Labs AI Gateway

AI gateways differ most under load, in throughput, latency and memory. Ferro Labs AI Gateway is written in Go from the ground up for real-world throughput — a single binary that routes LLM requests with predictable latency and minimal resource usage.

Feature	Ferro Labs	LiteLLM	Bifrost	Kong AI
Language	Go	Python	Go	Go/Lua
Single binary	✅	❌	✅	❌
Providers	30	100+	20+	10+
MCP support	✅	❌	✅	❌
Response cache	✅	✅	✅	❌ (paid)
Guardrails	✅	✅	❌	❌ (paid)
OSS license	Apache 2.0	MIT	Apache 2.0	Apache 2.0
Managed cloud	Coming Soon	✅	✅	✅

---

Features

Capability	What it does	Reference
🔀 Routing	8 strategies — single, fallback, load balance, least latency, cost-optimized, content-based, A/B test, conditional — with per-target retry, failover, and model aliases	Docs →
🔌 30 providers	Chat and streaming everywhere; embeddings, images, rerank, moderations, speech-to-text, text-to-speech, and batch where the vendor offers them	Docs →
🛡️ Guardrails & plugins	Eleven built in — word filter, regex guard, PII redaction, secret scanning, prompt-injection shield, response schema validation, token/message limits, response cache, rate limiting, per-key budgets, request logging — and the plugin framework is public for writing your own	Docs →
🎯 Capability matrix	One declarative record of which OpenAI parameters each provider forwards, translates, or cannot express, served by GET /v1/capabilities	Docs →
🤖 MCP	Connects to stdio and Streamable HTTP tool servers, injects their tools into chat completions, and drives the agentic tool_calls loop itself	Docs →
📊 Observability	OpenTelemetry tracing and Prometheus metrics, one trace ID across logs and spans — a zero-allocation no-op until enabled	Docs →
🖥️ Dashboard	Operations console compiled into the binary and served at / — traffic, spend, provider health, request logs, audit trail	Docs →

---

Dashboard

Every release binary serves a built-in operations console at / — same port as the API, compiled in with go:embed, no second image and no second origin. Sign in with your MASTER_KEY or any admin / read-only key and it reads the live gateway: traffic, spend, provider health, routing, plugins, request logs, and the audit trail.

The bundle is produced by the release pipeline, so a binary built with go install is the one exception: it answers / with a placeholder instead. Any other install method above ships the console.

Overview, Analytics (latency/TTFT/cost percentiles), Providers, Routing, Plugins, a Playground over the real routing path, Tracing, Request Logs, the Audit Trail, Configuration with history/rollback, and scoped API key management.

Run the gateway and open http://localhost:8080. To see it filled like the recording above, bring up the self-contained demo stack:

make up-fullstack   # gateway + Postgres + Jaeger + Prometheus + Grafana + mock upstream + load generator
# then open http://localhost:8080

Build and development details are in web/README.md.

---

Documentation

The root README is the overview; each subsystem keeps its own reference beside its code:

Reference	Covers
providers/README.md	The 30 providers, the per-provider endpoint matrix, and every /v1/* surface
config/README.md	Config loading, validation, ${VAR} secrets, declared models, trusted proxies
internal/strategies/README.md	All 8 routing strategies and their failure semantics
plugin/README.md	The plugin framework and the eleven built-in plugins
mcp/README.md	MCP tool servers, transports, the subprocess trust boundary, readiness
observability/README.md	Tracing setup, managed backends, emitted attributes, privacy levels, exporters
deploy/README.md	Dockerfiles, Compose files, the fullstack demo stack
web/README.md	Dashboard development and the embed contract
AGENTS.md	The complete operator/developer reference: architecture, every env var, request flow
SECURITY.md	Reporting, security posture, release verification
CONTRIBUTING.md	Branch strategy, commit conventions, provider/plugin checklists

---

Examples

Integration examples for common use cases are in ferro-labs/ai-gateway-examples:

Example	Description
basic	Single chat completion to the first configured provider
fallback	Fallback strategy — try providers in order with retries
loadbalance	Weighted load balancing across targets (70/30 split)
with-guardrails	Built-in word-filter and max-token guardrail plugins
with-mcp	Local MCP server with tool-calling integration
embedded	Embed the gateway as an HTTP handler inside an existing server

---

Configuration

One YAML/JSON file, named by GATEWAY_CONFIG, drives routing, guardrails, MCP tools, and observability:

strategy:
  mode: fallback              # 8 modes — see internal/strategies/README.md

targets:
  - virtual_key: openai       # an allowlist: only listed providers are routable
    retry: { attempts: 3 }    # per target, honoured under every routing mode
    concurrency: { max_concurrency: 32, queue_size: 1000 }
  - virtual_key: anthropic

aliases:
  fast: gpt-4o-mini

plugins:                      # guardrails first, then cache — see plugin/README.md
  - name: word-filter
    type: guardrail
    stage: before_request
    enabled: true
    config: { blocked_words: ["password", "secret"] }

mcp_servers:                  # tool servers — see mcp/README.md
  - name: search
    url: https://mcp.example.com/mcp
    headers: { Authorization: "Bearer ${SEARCH_TOKEN}" }

${VAR} references (braced form only) resolve when a component is constructed, never at file load — so secrets are never stored, served by GET /admin/config, or restored by a rollback. A bare $ is data; an undefined variable is an error.

The full annotated reference with every option is config.example.yaml / config.example.json, and the schema guide is config/README.md. ferrogw validate checks a file without starting the server.

Key environment variables

Variable	Purpose
MASTER_KEY	Bootstrap and break-glass admin credential (generated by ferrogw init); give each operator their own key from POST /admin/keys for day-to-day use
GATEWAY_CONFIG	Path to config YAML/JSON
GATEWAY_ENV	Set to production to enable production-mode safety guards: it refuses to start on ALLOW_UNAUTHENTICATED_PROXY=true or a * entry in CORS_ORIGINS, and warns when per-IP rate limiting is off, pprof is mounted, or the API key store is in-memory
PORT	Server port (default: 8080)
ALLOW_UNAUTHENTICATED_PROXY	Set to true to disable proxy-route auth (dev only; blocked when GATEWAY_ENV=production)
CORS_ORIGINS	Comma-separated allowed CORS origins; cross-origin is denied when unset. Matched literally — there is no wildcard, so list each origin explicitly
TRUSTED_PROXIES	CIDRs of trusted reverse proxies; forwarded headers are honored only from these (default: loopback). See config/README.md
<PROVIDER>_BASE_URL	Points a provider at a proxy, self-hosted server, or regional endpoint. It is the API root, used verbatim — write it exactly as the vendor documents it, version segment included (https://api.groq.com/openai/v1); a bare host resolves to the provider's own version segment

See AGENTS.md for the full environment variable reference including provider API keys, store backends, and OTel settings.

---

Observability

See everything your gateway does — every request, what it cost, how long it took, which provider served it, and which guardrails ran. Ferro Labs AI Gateway ships first-class OpenTelemetry tracing and Prometheus metrics out of the box, and stays at a zero-allocation no-op until you turn it on. Point it at Jaeger, Grafana, New Relic, LangSmith, Datadog, or Honeycomb — anything that speaks OTLP — and every request emits a gateway.request span carrying GenAI semantic conventions (gen_ai.*) plus ferro.* extensions for cost, routing, MCP tool calls, and stream timings. The same trace ID threads your logs, spans, and the X-Request-ID response header. Requests can name the end user and session they belong to (user, X-User-ID, X-Session-ID, or W3C baggage), and that identity rides the request span, every event, and request-log rows.

📈 Full observability guide → observability/README.md — managed-backend setup, endpoint & transport rules, every emitted attribute, privacy levels, and exporter plugins.

Bring up the gateway wired to a full monitoring stack — Prometheus, Grafana, and Jaeger — driven by generated traffic, in one command:

make up-fullstack   # then open Grafana at http://localhost:3000

Enable tracing with one variable (or the observability: config block — endpoint, protocol, sampling, privacy, headers are all documented in the guide):

export OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317
ferrogw serve

---

CLI

ferrogw is a single binary — no separate CLI tool required.

Command	Description
ferrogw	Start the gateway server (default)
ferrogw serve	Start the gateway server (explicit)
ferrogw init	First-run setup — generate master key and config
ferrogw validate	Validate a config file without starting
ferrogw doctor	Check environment (API keys, config, connectivity)
ferrogw status	Show gateway health and provider status
ferrogw version	Print version, commit, and build info
ferrogw admin keys list	List API keys
ferrogw admin keys create --name <name>	Create an API key (--scope, --expires-in)
ferrogw admin logs stats	Show request log statistics
ferrogw plugins	List registered plugins

Global flags available on all subcommands: --gateway-url, --api-key, --format (table/json/yaml).

---

Deployment

Local development

export OPENAI_API_KEY=sk-your-key
export MASTER_KEY=fgw_your-master-key
export GATEWAY_CONFIG=./config.yaml
make build && ./bin/ferrogw

Railway & Render

The deploy buttons at the top of this README provision either platform: Railway with SQLite on a volume (point API_KEY_STORE_DSN, CONFIG_STORE_DSN and REQUEST_LOG_STORE_DSN at paths under /data) or PostgreSQL, and Render from the repo's render.yaml Blueprint, which generates MASTER_KEY and wires the store DSNs to a managed Postgres automatically.

Docker Compose

Three Compose files in deploy/ follow the standard override pattern — a shared base, a dev override that builds from source, and a prod override with a pinned tag, health check, and resource limits. Run everything from the repository root:

make up             # dev: builds from source
IMAGE_TAG=v1.4.5 CORS_ORIGINS=https://your-domain.com make up-prod
make down           # tears down either

One container serves both the API and the dashboard — no second image, no second origin. Provider keys go in a repository-root .env or the environment. deploy/README.md has the full reference, including a self-contained PostgreSQL pairing and the fullstack observability stack.

Kubernetes via Helm

helm repo add ferro-labs https://ferro-labs.github.io/helm-charts
helm repo update
helm install ferro-gw ferro-labs/ai-gateway \
  --set env.OPENAI_API_KEY=sk-your-key

Helm charts: github.com/ferro-labs/helm-charts | ArtifactHub

---

Migrate to Ferro Labs AI Gateway

The gateway is OpenAI-compatible, so for a client already using an OpenAI-compatible SDK, migration — from another gateway or from calling a provider directly — is a base_url and api_key change: point the SDK at the gateway and present a gateway-issued credential in place of the provider's own. A client with its own API, such as LiteLLM's completion() below, also moves to the OpenAI SDK.

From LiteLLM

Before (LiteLLM):

from litellm import completion

response = completion(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}]
)

After (Ferro Labs AI Gateway):

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="your-ferro-api-key",
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
)

Provider API keys move to environment variables (OPENAI_API_KEY, ANTHROPIC_API_KEY, …); the model list becomes targets + aliases in config.yaml.

Why migrate from LiteLLM:

• 14x higher throughput at 150 concurrent users (2,447 vs 175 RPS)
• 23x less memory at peak load (47 MB vs 1,124 MB under streaming)
• Single binary — no Python environment, no pip, no virtualenv
• Predictable latency — p99 stays under 65 ms at 150 VU vs LiteLLM's timeouts at the same concurrency

From Portkey

The code change is the same one line — Ferro Labs uses the standard OpenAI SDK with no custom headers in self-hosted mode.

Why migrate from Portkey:

• Fully open source — no per-request pricing, no log limits
• Self-hosted — the gateway runs in your infrastructure, and prompts reach only the providers you configure
• No vendor lock-in — Apache 2.0 license
• MCP support — Portkey self-hosted lacks native MCP
• FerroCloud (coming soon) for teams that want a managed service

---

Performance

Every figure in this section comes from one run: Ferro Labs v1.0.0, measured 2026-03-23. Benchmarked against Kong OSS, Bifrost, LiteLLM, and Portkey on GCP n2-standard-8 (8 vCPU, 32 GB RAM) using a 60ms fixed-latency mock upstream — results reflect gateway overhead only. Later releases have not been re-measured; reproduce against the version you intend to run using the commands below.

VU	RPS	p50	p99	Memory
50	813	61.3ms	64.1ms	36 MB
150	2,447	61.2ms	63.4ms	47 MB
300	4,890	61.2ms	64.4ms	72 MB
500	8,014	61.5ms	72.9ms	89 MB
1,000	13,925	68.1ms	111.9ms	135 MB

At 1,000 VU: 13,925 RPS, p50 overhead 8.1ms, memory 135 MB. Against the live OpenAI API, the gateway itself adds 25 microseconds p50 in a typical plugin configuration, and 2 microseconds with no plugins enabled.

Full methodology, raw results, and flamegraph analysis: ferro-labs/ai-gateway-performance-benchmarks (make setup && make bench reproduces it).

---

FerroCloud

FerroCloud — the managed version of Ferro Labs AI Gateway with multi-tenancy, analytics, and cost governance — is coming soon.

👉 Join the waitlist at ferrolabs.ai

---

SDKs

Official client libraries for the Ferro Labs AI Gateway — and the standard OpenAI SDK works unchanged: point base_url at http://your-gateway:8080/v1.

SDK	Install	Repository
Python	pip install ferrolabs	ferro-labs/ferrolabs-python-sdk
TypeScript	npm install ferrolabs	ferro-labs/ferrolabs-typescript-sdk

from ferrolabs import FerroClient

client = FerroClient(
    base_url="http://localhost:8080/v1",
    api_key="your-ferro-api-key",
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
)

import { FerroClient } from "ferrolabs";

const client = new FerroClient({
  baseURL: "http://localhost:8080/v1",
  apiKey: "your-ferro-api-key",
});

const response = await client.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Hello" }],
});

---

Contributing

We welcome contributions. New providers go in this OSS repo only — never in FerroCloud. See CONTRIBUTING.md for branch strategy, commit conventions, and PR guidelines.

---

Community

• GitHub Discussions
• Discord
• Built with Ferro Labs AI Gateway? Open a PR to add to our showcase.

---

License

Apache 2.0 — see LICENSE.