> ## Documentation Index
> Fetch the complete documentation index at: https://docs.runlayer.com/llms.txt
> Use this file to discover all available pages before exploring further.
# LLM Gateway
> Route any AI client's model traffic through Runlayer for cost optimization, budgets, and session visibility
The Runlayer LLM Gateway is a lightweight proxy that sits between your AI clients and model providers. Clients keep speaking the OpenAI or Anthropic API they already use; the gateway authenticates them with a Runlayer virtual key, applies model routing rules, meters cost per key, and streams every conversation into [Sessions](/platform-sessions).
The LLM Gateway is in early access as a self-hosted service. Talk to your Runlayer contact before rolling it out broadly.
Use the LLM Gateway to:
* **Optimize cost with model routing** — substitute models by rule (for example, serve `gpt-4.1` requests with `gpt-4o-mini`), including across providers
* **Degrade, don't deny** — when a virtual key's budget is spent, downgrade expensive models to cheaper ones instead of failing requests
* **Support any AI client** — Claude Code, Cursor, Codex, OpenAI/Anthropic SDKs, or any script that speaks either API
* **Keep provider keys out of client hands** — clients hold a Runlayer virtual key; provider API keys live only in the gateway
* **See everything in Sessions** — prompts, responses, per-call token usage, cost, and routing decisions, attributed to the real client and user
## Managing keys, rules, and provider credentials
Virtual keys and routing rules live in the Runlayer backend and are managed in the app under **Settings → LLM Gateway**: create keys (shown once, with optional USD budgets) and maintain the ordered rule list (glob pattern → target model, always or only over budget). Gateways poll the backend and hot-apply changes within their sync interval (30s by default); revoked keys stop authenticating on the same cadence. Keys are stored and synced as SHA-256 hashes — plaintext exists only in the one-time reveal at creation. Spend is durable: gateways report per-call cost to the backend, accumulated spend survives restarts and is shared across gateway replicas, and each key’s current spend shows next to its budget in the settings page. Provider API keys (Anthropic, OpenAI, OpenAI-compatible) come from **Settings → AI Providers** — add a provider there and every synced gateway can serve its models within the same interval, no gateway config change needed.
## How it works
The gateway exposes two API surfaces:
| Surface | Base URL | Used by |
| ---------------------- | ---------------------------------- | ------------------------------------------------------- |
| Anthropic Messages API | `https:///anthropic` | Claude Code, Anthropic SDKs |
| OpenAI API | `https:///openai/v1` | OpenAI SDKs, most tools with a "custom base URL" option |
Each request is authenticated with a **virtual key** (sent as `x-api-key` or `Authorization: Bearer`, whichever the client already uses). The gateway then:
1. Applies **routing rules** — first matching rule wins, and may substitute the provider and model
2. Calls the upstream provider, translating between API dialects when the rule routes cross-provider (an Anthropic client routed to an OpenAI model still receives an Anthropic-shaped response, including streaming)
3. Meters token usage and cost against the virtual key's budget
4. Emits the turn to Sessions
### Routing rules
A rule matches the requested model with a glob pattern and substitutes the target `provider/model`; it applies `always` (default) or only `over_budget`. Rules are ordered — first match wins. For example:
| Pattern | Routes to | When | Effect |
| --------------- | ---------------------------- | ----------- | ------------------------------------------------------------------------- |
| `gpt-4.1` | `openai/gpt-4o-mini` | Always | Every `gpt-4.1` request is served by `gpt-4o-mini` |
| `claude-opus-*` | `anthropic/claude-haiku-4-5` | Over budget | Opus traffic downgrades instead of failing once the key's budget is spent |
The same shape works as inline `rules` in the gateway config file for standalone/bootstrap use — `{ "model": "gpt-4.1", "to": "openai/gpt-4o-mini", "when": "always" }`.
Spend is computed from response usage against the `prices_per_mtok` table in the gateway config (or provider-reported cost when available).
### Latency
The gateway's processing overhead — auth, routing, dialect handling, metering — is roughly **0.1ms per request (p50) and under 0.3ms at p99** (measured on loopback against a mock upstream). Session streaming happens off the request path. In practice the added latency is dominated by the network hop to wherever the gateway is deployed, so place it close to your clients or providers; both are negligible next to model response times.
## Client support
Any client that can point at a custom API base URL works with the gateway. The matrix below is verified against each client's official docs. Use `/anthropic` for clients that speak the Anthropic Messages API and `/openai/v1` for OpenAI-compatible clients (both Chat Completions and the Responses API are served).
| Client | Support | Speaks | How to connect |
| ------------------------- | ------------- | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Claude Code | ✅ First-class | Anthropic | `ANTHROPIC_BASE_URL=/anthropic` + `ANTHROPIC_AUTH_TOKEN=` (env or `settings.json` `env` block — MDM-distributable). Sessions group automatically via its embedded session id |
| Claude Desktop | ✅ | Anthropic | `~/.claude/settings.json` `env.ANTHROPIC_BASE_URL`, or Developer mode → third-party inference gateway |
| Codex CLI | ✅ | OpenAI (Responses) | `~/.codex/config.toml` `[model_providers.runlayer]` with `base_url = "/openai/v1"`, `env_key = "OPENAI_API_KEY"` |
| GitHub Copilot | ✅ Chat only | OpenAI or Anthropic | VS Code "Manage Language Models" → Custom Endpoint provider; CLI: `COPILOT_PROVIDER_BASE_URL` + `COPILOT_PROVIDER_API_KEY`. Inline completions stay on GitHub models |
| Cline | ✅ | OpenAI or Anthropic | "OpenAI Compatible" provider Base URL, or Anthropic provider → "Use custom base URL" |
| Roo Code | ✅ | OpenAI or Anthropic | Same as Cline. Requires models with native tool calling |
| Continue.dev | ✅ | OpenAI or Anthropic | `config.yaml` per-model `apiBase`. o-series/gpt-5 model names default to the Responses API (served) |
| Aider | ✅ | OpenAI | `OPENAI_API_BASE=/openai/v1` + `openai/` model prefix |
| Zed | ✅ | OpenAI or Anthropic | `settings.json` `language_models.openai_compatible` / `anthropic_compatible` with `api_url` |
| Goose | ✅ | OpenAI or Anthropic | `ANTHROPIC_HOST` / `OPENAI_HOST`, or a custom provider JSON with `base_url` |
| OpenCode | ✅ | OpenAI or Anthropic | `opencode.json` provider with `options.baseURL` |
| LibreChat | ✅ | OpenAI or Anthropic | `librechat.yaml` custom endpoint `baseURL`, or `ANTHROPIC_REVERSE_PROXY` |
| Open WebUI | ✅ | OpenAI | `OPENAI_API_BASE_URL=/openai/v1` or Admin → Connections |
| OpenAI / Anthropic SDKs | ✅ | Native | `base_url` param or `OPENAI_BASE_URL` / `ANTHROPIC_BASE_URL` env — see examples below |
| LangChain / Vercel AI SDK | ✅ | OpenAI or Anthropic | `ChatOpenAI(base_url=…)` / `ChatAnthropic(anthropic_api_url=…)`; `@ai-sdk/openai-compatible` with `baseURL` |
| Cursor | ⚠️ Limited | OpenAI | "Override OpenAI Base URL" + custom API key — chat models only (not Tab or Agent mode), gateway must be publicly reachable, and requests still transit Cursor's backend |
| Gemini CLI | 🚧 Not yet | Gemini | Speaks only the Google GenAI dialect; the gateway doesn't expose a `/genai` surface today |
| Windsurf | ❌ | — | BYOK accepts an API key only; no custom base URL setting exists |
## Connecting clients manually
Point the client at the gateway and swap the provider API key for your Runlayer virtual key.
```bash Claude Code theme={null}
export ANTHROPIC_BASE_URL="https:///anthropic"
export ANTHROPIC_API_KEY=""
```
```python OpenAI SDK (Python) theme={null}
from openai import OpenAI
client = OpenAI(
base_url="https:///openai/v1",
api_key="",
)
```
```python Anthropic SDK (Python) theme={null}
from anthropic import Anthropic
client = Anthropic(
base_url="https:///anthropic",
api_key="",
)
```
```bash curl / custom apps theme={null}
curl https:///anthropic/v1/messages \
-H "x-api-key: " \
-H "x-runlayer-session-id: my-conversation-42" \
-H "content-type: application/json" \
-d '{"model": "claude-opus-4-8", "max_tokens": 1024,
"messages": [{"role": "user", "content": "Hello"}]}'
```
Custom applications should send an `x-runlayer-session-id` header with a stable per-conversation value so multi-turn conversations group into one session. Without it, the gateway groups turns by virtual key and first user message, which works for typical chat clients that resend full history.
## Connecting clients via MDM
For managed fleets, distribute the same two settings — base URL and virtual key — through your MDM instead of asking users to configure anything:
1. **Issue virtual keys** scoped the way you want attribution to work (per team or per user).
2. **Push the configuration** with the same MDM integrations used for [AI Watch deployment](/shadow-ai/deploy/index) and [auto-provisioning](/auto-provisioning/index) — a managed environment profile (macOS) or policy (Windows) that sets `ANTHROPIC_BASE_URL`, `OPENAI_BASE_URL`, and the corresponding API key variables for the clients you manage.
3. Traffic from every managed device now flows through the gateway with no per-user setup, and appears in Sessions attributed to the device's user.
This pairs with [Shadow AI](/shadow-ai/index) detection: AI Watch shows you which clients exist on the fleet; the gateway turns that discovered traffic into governed, metered, budget-controlled traffic.
## What you see in Sessions
Gateway conversations appear in [Sessions](/platform-sessions) like any other monitored client:
* The session card shows the **real client** (detected from the request's User-Agent — Claude Code, Cursor, an SDK, or the raw tool name for scripts) and the **requested model**, plus the user behind the virtual key
* The timeline shows each prompt and response, and a **Model Call** entry per request with token counts and cost
* When a routing rule substituted the model, the Model Call shows the routing explicitly: `claude-opus-4-8 → gpt-4o-mini`
## Configuration reference
The gateway is configured with a single JSON file. Values of the form `env.VAR` are resolved from the environment at startup.
| Field | Description |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `listen` | Address to serve on (default `127.0.0.1:8090`) |
| `providers` | Optional inline map of provider → `api_key` (and optional `base_url` override) — bootstrap/fallback; production credentials come from **Settings → AI Providers** via `backend` sync. Supported: `anthropic`, `openai`, `bedrock`, `vertex`, `azure`, `gemini`, and other major providers |
| `virtual_keys` | Inline client credentials (`key`, `name`, optional `budget_usd`) — bootstrap/fallback; production keys come from the backend via `backend` sync |
| `rules` | Inline routing rules (`model` glob, `to`, optional `when: over_budget`) — bootstrap/fallback; backend-managed rules take precedence once synced |
| `prices_per_mtok` | Per-model USD prices per million tokens (`input`, `output`) used for budget metering |
| `sessions` | Optional: `hooks_url` (your Runlayer API `/api/v1/hooks/events` endpoint) and `api_key` (a Runlayer API key) to stream conversations into Sessions |
| `backend` | Optional: `sync_url` (your Runlayer API `/api/v1/llm-gateway/sync-config` endpoint), `api_key`, and `sync_interval_seconds` (default 30). When set, virtual keys and routing rules from **Settings → LLM Gateway** and provider credentials from **Settings → AI Providers** are synced to the gateway automatically — the webapp becomes the source of truth, and file `providers`/`virtual_keys`/`rules` act as bootstrap/fallback |