LLM Gateway

The Runlayer LLM Gateway is a lightweight proxy that sits between your AI clients and model providers. Clients keep speaking the OpenAI or Anthropic API they already use; the gateway authenticates them with a Runlayer virtual key, applies model routing rules, meters cost per key, and streams every conversation into Sessions.

The LLM Gateway is in early access as a self-hosted service. Talk to your Runlayer contact before rolling it out broadly.

Use the LLM Gateway to:

Optimize cost with model routing — substitute models by rule (for example, serve gpt-4.1 requests with gpt-4o-mini), including across providers
Degrade, don’t deny — when a virtual key’s budget is spent, downgrade expensive models to cheaper ones instead of failing requests
Support any AI client — Claude Code, Cursor, Codex, OpenAI/Anthropic SDKs, or any script that speaks either API
Keep provider keys out of client hands — clients hold a Runlayer virtual key; provider API keys live only in the gateway
See everything in Sessions — prompts, responses, per-call token usage, cost, and routing decisions, attributed to the real client and user

Managing keys, rules, and provider credentials

Virtual keys and routing rules live in the Runlayer backend and are managed in the app under Settings → LLM Gateway: create keys (shown once, with optional USD budgets) and maintain the ordered rule list (glob pattern → target model, always or only over budget). Gateways poll the backend and hot-apply changes within their sync interval (30s by default); revoked keys stop authenticating on the same cadence. Keys are stored and synced as SHA-256 hashes — plaintext exists only in the one-time reveal at creation. Spend is durable: gateways report per-call cost to the backend, accumulated spend survives restarts and is shared across gateway replicas, and each key’s current spend shows next to its budget in the settings page. Provider API keys (Anthropic, OpenAI, OpenAI-compatible) come from Settings → AI Providers — add a provider there and every synced gateway can serve its models within the same interval, no gateway config change needed.

How it works

The gateway exposes two API surfaces:

Surface	Base URL	Used by
Anthropic Messages API	`https://<gateway-host>/anthropic`	Claude Code, Anthropic SDKs
OpenAI API	`https://<gateway-host>/openai/v1`	OpenAI SDKs, most tools with a “custom base URL” option

Each request is authenticated with a virtual key (sent as x-api-key or Authorization: Bearer, whichever the client already uses). The gateway then:

Applies routing rules — first matching rule wins, and may substitute the provider and model
Calls the upstream provider, translating between API dialects when the rule routes cross-provider (an Anthropic client routed to an OpenAI model still receives an Anthropic-shaped response, including streaming)
Meters token usage and cost against the virtual key’s budget
Emits the turn to Sessions

Routing rules

A rule matches the requested model with a glob pattern and substitutes the target provider/model; it applies always (default) or only over_budget. Rules are ordered — first match wins. For example:

Pattern	Routes to	When	Effect
`gpt-4.1`	`openai/gpt-4o-mini`	Always	Every `gpt-4.1` request is served by `gpt-4o-mini`
`claude-opus-*`	`anthropic/claude-haiku-4-5`	Over budget	Opus traffic downgrades instead of failing once the key’s budget is spent

The same shape works as inline rules in the gateway config file for standalone/bootstrap use — { "model": "gpt-4.1", "to": "openai/gpt-4o-mini", "when": "always" }. Spend is computed from response usage against the prices_per_mtok table in the gateway config (or provider-reported cost when available).

Latency

The gateway’s processing overhead — auth, routing, dialect handling, metering — is roughly 0.1ms per request (p50) and under 0.3ms at p99 (measured on loopback against a mock upstream). Session streaming happens off the request path. In practice the added latency is dominated by the network hop to wherever the gateway is deployed, so place it close to your clients or providers; both are negligible next to model response times.

Client support

Any client that can point at a custom API base URL works with the gateway. The matrix below is verified against each client’s official docs. Use /anthropic for clients that speak the Anthropic Messages API and /openai/v1 for OpenAI-compatible clients (both Chat Completions and the Responses API are served).

Client	Support	Speaks	How to connect
Claude Code	✅ First-class	Anthropic	`ANTHROPIC_BASE_URL=<gw>/anthropic` + `ANTHROPIC_AUTH_TOKEN=<virtual-key>` (env or `settings.json` `env` block — MDM-distributable). Sessions group automatically via its embedded session id
Claude Desktop	✅	Anthropic	`~/.claude/settings.json` `env.ANTHROPIC_BASE_URL`, or Developer mode → third-party inference gateway
Codex CLI	✅	OpenAI (Responses)	`~/.codex/config.toml` `[model_providers.runlayer]` with `base_url = "<gw>/openai/v1"`, `env_key = "OPENAI_API_KEY"`
GitHub Copilot	✅ Chat only	OpenAI or Anthropic	VS Code “Manage Language Models” → Custom Endpoint provider; CLI: `COPILOT_PROVIDER_BASE_URL` + `COPILOT_PROVIDER_API_KEY`. Inline completions stay on GitHub models
Cline	✅	OpenAI or Anthropic	”OpenAI Compatible” provider Base URL, or Anthropic provider → “Use custom base URL”
Roo Code	✅	OpenAI or Anthropic	Same as Cline. Requires models with native tool calling
Continue.dev	✅	OpenAI or Anthropic	`config.yaml` per-model `apiBase`. o-series/gpt-5 model names default to the Responses API (served)
Aider	✅	OpenAI	`OPENAI_API_BASE=<gw>/openai/v1` + `openai/<model>` model prefix
Zed	✅	OpenAI or Anthropic	`settings.json` `language_models.openai_compatible` / `anthropic_compatible` with `api_url`
Goose	✅	OpenAI or Anthropic	`ANTHROPIC_HOST` / `OPENAI_HOST`, or a custom provider JSON with `base_url`
OpenCode	✅	OpenAI or Anthropic	`opencode.json` provider with `options.baseURL`
LibreChat	✅	OpenAI or Anthropic	`librechat.yaml` custom endpoint `baseURL`, or `ANTHROPIC_REVERSE_PROXY`
Open WebUI	✅	OpenAI	`OPENAI_API_BASE_URL=<gw>/openai/v1` or Admin → Connections
OpenAI / Anthropic SDKs	✅	Native	`base_url` param or `OPENAI_BASE_URL` / `ANTHROPIC_BASE_URL` env — see examples below
LangChain / Vercel AI SDK	✅	OpenAI or Anthropic	`ChatOpenAI(base_url=…)` / `ChatAnthropic(anthropic_api_url=…)`; `@ai-sdk/openai-compatible` with `baseURL`
Cursor	⚠️ Limited	OpenAI	”Override OpenAI Base URL” + custom API key — chat models only (not Tab or Agent mode), gateway must be publicly reachable, and requests still transit Cursor’s backend
Gemini CLI	🚧 Not yet	Gemini	Speaks only the Google GenAI dialect; the gateway doesn’t expose a `/genai` surface today
Windsurf	❌	—	BYOK accepts an API key only; no custom base URL setting exists

Connecting clients manually

Point the client at the gateway and swap the provider API key for your Runlayer virtual key.

export ANTHROPIC_BASE_URL="https://<gateway-host>/anthropic"
export ANTHROPIC_API_KEY="<virtual-key>"

from openai import OpenAI

client = OpenAI(
    base_url="https://<gateway-host>/openai/v1",
    api_key="<virtual-key>",
)

from anthropic import Anthropic

client = Anthropic(
    base_url="https://<gateway-host>/anthropic",
    api_key="<virtual-key>",
)

curl https://<gateway-host>/anthropic/v1/messages \
  -H "x-api-key: <virtual-key>" \
  -H "x-runlayer-session-id: my-conversation-42" \
  -H "content-type: application/json" \
  -d '{"model": "claude-opus-4-8", "max_tokens": 1024,
       "messages": [{"role": "user", "content": "Hello"}]}'

Custom applications should send an x-runlayer-session-id header with a stable per-conversation value so multi-turn conversations group into one session. Without it, the gateway groups turns by virtual key and first user message, which works for typical chat clients that resend full history.

Connecting clients via MDM

For managed fleets, distribute the same two settings — base URL and virtual key — through your MDM instead of asking users to configure anything:

Issue virtual keys scoped the way you want attribution to work (per team or per user).
Push the configuration with the same MDM integrations used for AI Watch deployment and auto-provisioning — a managed environment profile (macOS) or policy (Windows) that sets ANTHROPIC_BASE_URL, OPENAI_BASE_URL, and the corresponding API key variables for the clients you manage.
Traffic from every managed device now flows through the gateway with no per-user setup, and appears in Sessions attributed to the device’s user.

This pairs with Shadow AI detection: AI Watch shows you which clients exist on the fleet; the gateway turns that discovered traffic into governed, metered, budget-controlled traffic.

What you see in Sessions

Gateway conversations appear in Sessions like any other monitored client:

The session card shows the real client (detected from the request’s User-Agent — Claude Code, Cursor, an SDK, or the raw tool name for scripts) and the requested model, plus the user behind the virtual key
The timeline shows each prompt and response, and a Model Call entry per request with token counts and cost
When a routing rule substituted the model, the Model Call shows the routing explicitly: claude-opus-4-8 → gpt-4o-mini

Configuration reference

The gateway is configured with a single JSON file. Values of the form env.VAR are resolved from the environment at startup.

Field	Description
`listen`	Address to serve on (default `127.0.0.1:8090`)
`providers`	Optional inline map of provider → `api_key` (and optional `base_url` override) — bootstrap/fallback; production credentials come from Settings → AI Providers via `backend` sync. Supported: `anthropic`, `openai`, `bedrock`, `vertex`, `azure`, `gemini`, and other major providers
`virtual_keys`	Inline client credentials (`key`, `name`, optional `budget_usd`) — bootstrap/fallback; production keys come from the backend via `backend` sync
`rules`	Inline routing rules (`model` glob, `to`, optional `when: over_budget`) — bootstrap/fallback; backend-managed rules take precedence once synced
`prices_per_mtok`	Per-model USD prices per million tokens (`input`, `output`) used for budget metering
`sessions`	Optional: `hooks_url` (your Runlayer API `/api/v1/hooks/events` endpoint) and `api_key` (a Runlayer API key) to stream conversations into Sessions
`backend`	Optional: `sync_url` (your Runlayer API `/api/v1/llm-gateway/sync-config` endpoint), `api_key`, and `sync_interval_seconds` (default 30). When set, virtual keys and routing rules from Settings → LLM Gateway and provider credentials from Settings → AI Providers are synced to the gateway automatically — the webapp becomes the source of truth, and file `providers`/`virtual_keys`/`rules` act as bootstrap/fallback

Enable

Control

Self-Hosted Deployment

Operations

Cookbook

Integrations

Connectors

Resources

Managing keys, rules, and provider credentials

How it works

Routing rules

Latency

Client support

Connecting clients manually

Connecting clients via MDM

What you see in Sessions

Configuration reference

​Managing keys, rules, and provider credentials

​How it works

​Routing rules

​Latency

​Client support

​Connecting clients manually

​Connecting clients via MDM

​What you see in Sessions

​Configuration reference

Managing keys, rules, and provider credentials

How it works

Routing rules

Latency

Client support

Connecting clients manually

Connecting clients via MDM

What you see in Sessions

Configuration reference