llmgateway

LLM Gateway

LLM Gateway is an open-source API gateway for Large Language Models (LLMs). It acts as a middleware between your applications and various LLM providers, allowing you to:

• Route requests to multiple LLM providers (OpenAI, Anthropic, Google Vertex AI, and others)
• Manage API keys for different providers in one place
• Track token usage and costs across all your LLM interactions
• Analyze performance metrics to optimize your LLM usage

Features

• Unified API Interface: Compatible with the OpenAI API format for seamless migration
• Usage Analytics: Track requests, tokens used, response times, and costs
• Multi-provider Support: Connect to various LLM providers through a single gateway
• Performance Monitoring: Compare different models' performance and cost-effectiveness

Getting Started

You can use LLM Gateway in two ways:

• Hosted Version: For immediate use without setup, visit llmgateway.io to create an account and get an API key.
• Self-Hosted: Deploy LLM Gateway on your own infrastructure for complete control over your data and configuration.

Self-Hosted With Docker

Use Docker-managed volumes for the unified image. Do not bind-mount a host directory directly to /var/lib/postgresql/data, because PostgreSQL initialization inside the container needs to set permissions on that directory and that can fail depending on the host filesystem and ownership.

export LLM_GATEWAY_SECRET="$(openssl rand -base64 32 | tr -d '\n')"
export GATEWAY_API_KEY_HASH_SECRET="$(openssl rand -base64 32 | tr -d '\n')"
./scripts/run-unified-container.sh

If you prefer a one-off docker run, use named volumes instead of ~/some-host-dir:

docker volume create llmgateway_postgres
docker volume create llmgateway_redis

docker run -d \
  --name llmgateway \
  --restart unless-stopped \
  -p 3002:3002 \
  -p 3003:3003 \
  -p 3005:3005 \
  -p 3006:3006 \
  -p 3007:3007 \
  -p 4001:4001 \
  -p 4002:4002 \
  -v llmgateway_postgres:/var/lib/postgresql/data \
  -v llmgateway_redis:/var/lib/redis \
  -e AUTH_SECRET="$LLM_GATEWAY_SECRET" \
  -e GATEWAY_API_KEY_HASH_SECRET="$GATEWAY_API_KEY_HASH_SECRET" \
  ghcr.io/theopenco/llmgateway-unified:latest

Using LLM Gateway API

curl -X POST https://api.llmgateway.io/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -d '{
  "model": "gpt-4o",
  "messages": [
    {"role": "user", "content": "Hello, how are you?"}
  ]
}'

Development Setup

1. Install dependencies and set up the development environment:

   pnpm i && pnpm run setup
   

   This will install all dependencies, start Docker services, sync the database schema, and seed initial data.

   Note for WSL2 users: Ensure Docker Desktop is running with WSL integration enabled.

2. Start development servers:

   pnpm dev
   

3. Build for production:

   pnpm build
   

Folder Structure

• apps/ui: Next.js dashboard frontend
• apps/playground: Lounge — consumer AI chat app (Next.js)
• apps/code: Next.js Dev Plans + coding tools landing & dashboard
• apps/airside: Airside — self-serve provider portal (Next.js)
• apps/api: Hono backend
• apps/gateway: API gateway for routing LLM requests
• apps/docs: Documentation site
• ee/admin: Internal admin dashboard
• packages/db: Drizzle ORM schema and migrations
• packages/models: Model and provider definitions
• packages/shared: Shared types and utilities

Adding a provider

We do not accept pull requests that add new providers. Inference providers onboard through Airside, our self-serve provider portal, where you register and manage your model listings.

License

LLMGateway is available under a dual license:

• Open Source: Core functionality is licensed under AGPLv3 - see the LICENSE file for details.
• Enterprise: Commercial features in the ee/ directory require an Enterprise license; multi-organization administration requires a white-label license. See ee/LICENSE for details.

Enterprise features include:

• Advanced billing and subscription management
• Full request and response payload retention (Retain All Data); other plans are metadata only
• Custom provider key configurations
• Team and organization management
• Priority support
• And more to be defined

For enterprise licensing, please contact us at [email protected]