Use this path for one or many isolated tenants on a Kubernetes cluster (EKS or GKE).
End-to-end flow
- Provision cluster features and external per-tenant dependencies (Postgres, Redis, buckets, IAM / Workload Identity, streams).
- Install the
runlayer-operatorHelm chart once per cluster. - Create (or let the operator create) the platform namespace and seed required Secrets.
- Apply a
RunlayerInstance(raw YAML, optionalrunlayer-instanceHelm chart, or Terraform on AWS). - Wait for
status.phase: Ready, point DNS at the load balancer, then createMCPServerCRs if using Runlayer Deploy.
1. External dependencies
The operator does not provision cloud infrastructure. Each tenant needs the following out-of-band (Terraform, console, or your platform).Required for every tenant
Network: cluster nodes (or pods via CNI) must reach Postgres and Redis on private connectivity (VPC, Private Service Connect, peering). Tighten egress with
spec.security.networkPolicy.additionalEgressRules once CIDRs are known.
Configure time synchronization on cluster nodes; Runlayer pods inherit the node clock. Use the cloud provider’s local time service (Amazon Time Sync on AWS; Google’s internal NTP service on GCP), or your approved internal source. The operator does not configure host NTP.
Optional (feature-dependent)
Sample Terraform for Postgres, Redis, buckets, streams, and identity is embedded in External dependencies and Workload identity (AWS and GCP side by side). Adapt the samples to your IaC — Runlayer does not require a specific Terraform module.
Shared GCP project
When any component usesstreamBackend: pubsub, set one project for the instance:
projectId on per-component pubsub blocks (subscription / DLQ only).
Identity model (IRSA / Workload Identity)
The operator creates dedicated ServiceAccounts per workload and never embeds long-lived cloud keys. Annotate viaspec.serviceAccount:
AWS (IRSA):
AUDIT_LOG_SIEM_EXPORT_ASSUME_ROLE_ARN is not the K8s path).
Saved Bedrock providers for monitor consumers
The EKS application policy already grants scoped saved-Bedrock role assumption for separately provisioned monitor consumers. Additional ServiceAccounts need explicitapplication_irsa_additional_service_accounts trust and matching role annotations.
The saved provider role must trust that AWS identity with its configured external ID and grant model access; GKE consumers also need an AWS credential source because GCP Workload Identity does not authenticate to AWS.
2. Cluster requirements
Kubernetes version
Shared features (EKS and GKE)
EKS-specific
See Kubernetes prerequisites for the full cluster contract.
GKE-specific
Namespace layout
Override with
spec.deploy.platformNamespace / spec.deploy.mcpNamespace.
The operator only uses, labels, drains, or deletes a namespace it owns: one it created (labeled runlayer.com/instance-uid: <instance UID>) or a pre-created namespace annotated for that instance, including a platform namespace that also holds a co-located RunlayerInstance (see sequence A). Any other existing namespace, including one another instance owns, fails the instance with Available=False, reason NamespaceNotOwned, and the operator creates nothing there. This applies to the platform and MCP namespaces. An agents.k8s.namespace may be one you created or one shared with other instances; the operator refuses it only when it is another instance’s platform or MCP namespace or is annotated for another instance, and never drains or deletes it.
3. Install and upgrade the operator
Install (once per cluster)
Copy and customize values:runlayer-operator-promote-customer-distribution workflow.
main in account 412677576004.
Override image.repository to the staging operator image when using those charts.
crds/ directory and are applied on first helm install only:
platform.runlayer.com/v1alpha1—RunlayerInstancedeploy.runlayer.com/v1alpha1—MCPServer
Common operator values
Full values reference ships with the Helm chart you install from Customer Distribution ECR (
values.example.yaml in the chart package).
Upgrading the operator (manual CRD apply)
Helm does not upgrade CRDs on subsequent releases. When a release notes CRD schema changes, apply CRDs before upgrading the chart:kubectl apply -f ... is enough.
4. Prerequisites: namespace and Secrets
Install order for Secrets
The operator never creates, reads, or syncs Secret data (nosecrets write RBAC). You must create Secrets in the platform namespace.
Two workable sequences:
Many teams use sequence A with their own Terraform syncing Secrets Manager / Secret Manager into the platform namespace.
In sequence A, annotate the namespace with the
RunlayerInstance that may adopt it, or the operator refuses it (NamespaceNotOwned). Do the same for a pre-created MCP namespace. A pre-created agents namespace needs no annotation. The terraform-runlayer-instance module annotates the platform namespace it creates; it does not create the MCP namespace, so annotate that one yourself if you pre-create it. Namespaces already reconciled by the operator carry the instance UID label and need nothing.
Required Secrets (platform namespace)
Before creating
runlayer-app, obtain a Distribution reader key registered and active for this deployment from Runlayer. Set all three Distribution settings below so feature flags use Distribution.
Example (sequence A):
MCP Catalog (required)
Onboarding calls the MCP Catalog API. Without a key,GET /api/v1/catalog/ returns 503 and setup shows “Couldn’t load setup status”.
Contact Runlayer for the key — see Runlayer-provided inputs.
AI Watch binary distribution (when using AI Watch)
Installer resolution needs a Runlayer download token, a customer-owned object-storage cache, and workload permission to that cache. Without them the UI reports “No AI Watch installer is currently resolved”.
After setting these, restart backend/worker and run release discovery (Check now or the scheduled poll). Details: External dependencies and Workload identity.
Optional component Secrets
App secret
Platform Secret namedrunlayer-app by convention (spec.appSecretRef.name). Mounted with envFrom on backend and worker (and related backend-image jobs). Every key in the Secret becomes an environment variable.
Required keys
Common optional keys
AUTH_API_KEY may also appear here; it is also injected from spec.auth.apiKeyFrom (duplicate is harmless).
With terraform-runlayer-instance, the module copies RESEND_API_KEY from the tenant Secrets Manager secret into this Secret (empty when that key is absent); a key added to the Kubernetes Secret by hand is overwritten on the next apply, so put it in Secrets Manager: initial_tenant_secrets.resend_api_key on the dependencies module for a new tenant, or edit the existing tenant secret directly (the bootstrap resource ignores later changes).
Common non-secret configuration via the same Secret
BecauseenvFrom loads all keys, teams often put non-sensitive settings in runlayer-app as well (or use a separate ConfigMap pattern outside the operator). Typical production keys:
AI Watch signed policy snapshot
AI Watch devices verify the policy snapshot served atGET /api/v1/ai-watch/config/signed offline. The backend signs it with this deployment’s key, and a Runlayer root delegation per organization tells devices to trust that key. Until both are set, the endpoint returns 404; it also does when either is invalid, and the backend logs ai_watch_policy_snapshot_signing_misconfigured. To generate the key and obtain or renew delegations, see Runlayer-provided inputs.
AI_WATCH_SNAPSHOT_SIGNING_KEY to spec.extraEnv with valueFrom.secretKeyRef.
Kubernetes does not reload Secret-backed environment variables. After setting or rotating the seed, roll the pods by changing the runlayer.com/app-secret-hash annotation on the RunlayerInstance (the operator copies it onto the backend, worker and frontend pod templates):
spec.extraEnv rolls the pods by itself. A delegation is bound to one key, so a new seed needs new delegations for every organization; change both in the same rollout.
With terraform-runlayer-instance, put the seed in the tenant Secrets Manager secret as AI_WATCH_SNAPSHOT_SIGNING_KEY (initial_tenant_secrets.ai_watch_snapshot_signing_key on the dependencies module for a new tenant; edit an existing tenant secret directly) and pass the delegations JSON as ai_watch_snapshot_delegations. Each reaches runlayer-app only when non-empty, and the module sets the annotation from the Secret contents, so an apply rolls the pods. With the runlayer-instance chart, set spec.extraEnv and annotations in its values.
Runlayer support access
Runlayer support reaches your deployment’s/api/v1/debug/* endpoints through the Runlayer ops console, which signs every request. Your backend verifies the signature against your configured trusted keys and encrypts its response so only that request can read it.
Support access also requires DEBUG_WORKOS_CLIENT_ID in the runlayer-app Secret, configured for Runlayer’s ops WorkOS environment, not the tenant’s. The backend accepts only tokens from Runlayer’s ops WorkOS organization: DEBUG_ALLOWED_WORKOS_ORG_ID defaults to it, so leave it unset. For MCP access, coordinate DEBUG_MCP_AUTHKIT_DOMAIN with Runlayer support. WorkOS authentication runs before signature verification; signing trust alone does not enable access.
Obtain the reviewed production ops-console JWKS from Runlayer support and set its JSON contents as DEBUG_OPS_REQUEST_SIGNING_JWKS in the runlayer-app Secret (one line is fine). With terraform-runlayer-instance, pass the JSON string as debug_ops_request_signing_jwks. Do not trust staging console keys in production. Runlayer support coordinates key rotations separately from module releases; publish replacement keys before switching signing keys. Unset, blank, or empty trust disables support debug access once backend signature enforcement is deployed.
Replay protection uses the existing writer Postgres database; the deployment’s migration Job creates its table automatically. If replay storage is unavailable, debug requests return 503 without running the handler.
Runlayer Assistant inference. BEDROCK_ENABLED=true (typical AWS): new/unbound assistants use platform-billed Bedrock (RUNLAYER_ASSISTANT_BEDROCK_MODEL; IRSA/WI or BEDROCK_ACCESS_KEY_ID / BEDROCK_SECRET_ACCESS_KEY). BEDROCK_ENABLED=false (typical GCP/GKE): the assistant binds an org AI provider instead — recommended model if one is set, else the custom-agent default, else the oldest provider’s first model. Until an admin adds a provider in Settings → AI Providers, the assistant is still provisioned but chat is blocked. An admin can change the assistant’s model/provider in the agent panel (MANAGE_ORG_SETTINGS); a bound org provider is sticky and is not reset when Bedrock is later enabled. Prompt, name, connectors, and skills stay managed.
Any backend Settings field (see application config) can be supplied this way. Prefer CR fields for connectivity that the operator already knows how to wire (database, redis, auth, component toggles).
On AWS, when this deployment invokes Anthropic models on Bedrock (BEDROCK_ENABLED=true, or Bedrock-backed custom agent models), submit the one-time form in the account those calls come from:
Anthropic model access. Anthropic requires a use case details form before its models can be invoked in an AWS account, plus an accepted model agreement. You need this when the deployment invokes Anthropic models on Bedrock — typically
BEDROCK_ENABLED=true (Runlayer Assistant platform inference and optional Bedrock agent models). With BEDROCK_ENABLED=false, Runlayer Assistant binds an org AI provider instead; skip this step unless you also enable Bedrock inference elsewhere (for example BEDROCK_ANTHROPIC_MODELS_ENABLED for custom agents).The backend does both for you at startup and on login, in the account the Bedrock calls come from, using the model-access IAM permissions your deployment grants. Four settings describe the deployment on the form — BEDROCK_MODEL_ACCESS_COMPANY_NAME (defaults to Runlayer, the operator of the software making the calls), BEDROCK_MODEL_ACCESS_COMPANY_WEBSITE, BEDROCK_MODEL_ACCESS_INDUSTRY and BEDROCK_MODEL_ACCESS_INTENDED_USERS. Override them if you run this install yourself. They are never derived from your data: the form is submit-once, so a guessed answer would be permanent.Set BEDROCK_AUTO_MODEL_ACCESS=false to turn this off and do it by hand instead: Bedrock console → Model catalog → any Anthropic model → Submit use case details. Do the same if your deployment role predates the model-access permissions — the backend logs exactly which actions it is missing. AWS accepts the form once per account, or once at your AWS Organization’s management account, which member accounts inherit.Deploying outside the US? Also set RUNLAYER_ASSISTANT_BEDROCK_MODEL to your geography’s inference profile, for example au.anthropic.claude-opus-4-6-v1 in ap-southeast-2. The risk-categorization explainer follows BEDROCK_REGION on its own and uses Claude Haiku 4.5 outside the US, so the Anthropic form above covers it; only set RISK_CATEGORIZATION_MODEL if you want a different model, and pick one served in your Region. If Bedrock rejects the primary model it falls back to Amazon Nova Lite, which needs no access request.Access takes up to 15 minutes to propagate. Until it does — or if this is skipped entirely — Bedrock-backed agents fail with Model use case details have not been submitted for this account; see troubleshooting for the CLI checks and Region caveats.5. Deploy a RunlayerInstance
Required spec fields
Optional highlights:
database.port (5432), database.name (runlayer), redis.port / tls / credentials, components, deploy, serviceAccount, workloadScheduling, security, trustedCa, autoscaling, podDisruptionBudget, ingress, otelCollector, auditSpool, extraEnv (extra environment variables for every backend-image workload; unlike the ECS module’s additional_backend_env_vars, a name the operator sets itself is rejected rather than overridden). spec.deploymentTelemetry is accepted and ignored.
multiRegion (role: primary, region: <cloud region>) marks the instance as a member of a multi-region deployment: backend-image workloads get RUNLAYER_REGION_ROLE and a deployment.region resource attribute in front of any OTEL_RESOURCE_ATTRIBUTES you set in extraEnv. Leave it unset unless Runlayer asks you to set it; unset changes nothing. While it is set, declare OTEL_RESOURCE_ATTRIBUTES as a literal extraEnv value: a valueFrom entry is rejected, and a key of that name in the app secret is overridden.
Field-level detail is covered in the sections below and in the supporting guides linked at the top of this page.
Minimal example (EKS)
Minimal example (GKE)
Optional components (snippet)
When enabling an optional component, set itsimages.* field (and any required Secret refs). The operator rejects the CR if those are missing. status.phase: Ready / Available wait for every enabled optional Deployment (not only backend/frontend/worker).
Pull credentials for platform components: the operator sets no imagePullSecrets on platform component pods (spec.deploy.defaultImagePullSecrets applies to MCPServer pods only — the Helm chart’s toolguard.image.pullSecrets knob has no operator equivalent). Customer Distribution ECR (088332244652) supports cross-account pulls when Runlayer allowlists your AWS account or IAM pull principal during onboarding — use those image URIs directly on EKS once allowlisted. Prefer a GAR / private mirror for GKE, air-gapped networks, or orgs that require an internal registry copy.
extendDB needs no images.* entry — it defaults to a digest-pinned upstream ExtendDB image; set image only to override. It is the only optional component with a dedicated reconcile path: the init Job runs extenddb init (creates the catalog/data Postgres databases + a dedicated role + TLS cert from the bootstrap credentials, HTTPS port 18443) then mints a shared SigV4 access key and kubectl applys those Secrets/ConfigMaps into the platform and MCP namespaces (the reconciler never reads Secret payloads). Deployment/Service/PDB come up after that. RunlayerInstance.status.conditions reports ExtendDBReady: Deployment ready, credentials present, and the readiness probe’s cached Postgres connectivity check (extenddb catalog-check against both catalog and data DBs, refreshed at most every 5 minutes) — gated on the check’s connectivity banners only, so a catalog-integrity finding can’t take the whole Service NotReady. It is still not a live, between-ticks guarantee. ExtendDBReady gates MCPServer.spec.infrastructure.enable_db (rejected with “enable ExtendDB on the instance first” until then) — see infrastructure.enable_db. Enable ExtendDB on the instance before setting enable_db on MCPServers. MCPServers with enable_db still reconcile their Deployment while waiting; init failure does not wedge the rest of the instance. Init is create-once; a partial failure with no MCP keys synced yet gets one automatic reset (reconciler drops the half-created catalog/data DBs + role, then recreates the Job, capped by an instance annotation) before it needs a human — see Upgrading and recovering ExtendDB below. ExtendDB cannot be disabled while any MCPServer in the Deploy namespace still has enable_db: true (clear those first; undeploy deletes tables). Connection sizing for the shared Postgres host: External dependencies.
Upgrading and recovering ExtendDB
Bumpingcomponents.extendDB.image (or a helper image):
imageonly changes the Deployment —helm upgrade/ re-apply rolls the existing pods to the new tag. It does not rerunextenddb init: TLS cert/key, the catalog/data Postgres schema, and the shared SigV4 access key are untouched. There is no auto-upgrade Job in v1 — if the new version needs a schema migration, run it yourself against the catalog/data DBs before or after the roll, per the upstream ExtendDB release notes.kubectlImage/postgresClientImage/awsCliImageonly affect their respective one-shot Jobs (init, reset, and per-MCPServer create-table/delete-table) on their next run — they don’t restart anything already Running. Bump them ahead of a planned init/reset, or anytime for the next MCPServer table Job.- Keep Secrets/state (do nothing beyond the CR edit) for a routine image bump — this is the common case and preserves the live token store.
- Don’t delete a succeeded init Job expecting a fresh schema: the operator only recreates the Job, which reruns
extenddb initagainst the existing catalog/data DBs. Init isn’t idempotent, so it just failsInitFailed— the MCP keys Secret already exists at this point, so the reset Job (and its refuse-guards) never runs. - To actually get a clean Postgres schema with no live token store: clear
enable_dbon (or undeploy) every MCPServer on the instance first, then drop the catalog/data DBs (and the app role, if needed) yourself with the bootstrap credential, then delete the init Job so the operator recreates it. This destroys any tokens already stored in those tables — only do it once nothing depends on them. - Prefer disable → re-enable when you just want the adapter back without touching Postgres — it keeps the succeeded init Job + Secrets and skips re-init entirely.
- A second
InitFailedafter the one automatic reset, or any failure once MCP keys already exist: inspect the Job logs first (the condition message points there — “inspect the Job logs before doing anything manual” for a reset failure), then delete the Job and any partially-created catalog/data DBs before retrying. Once keys/tables exist, dropping the DBs destroys stored tokens.
extenddb-postgres image (and its helper images) are pulled directly from Docker Hub (docker.io/extenddb/...) today — not mirrored to Runlayer’s own registries. A Runlayer-run mirror is not yet available. Before relying on it for air-gapped or registry-restricted clusters:
- Confirm the digest you’re pinning (
image/kubectlImage/postgresClientImage/awsCliImage) is reachable from cluster nodes. - If not, mirror it into your own registry and set the CR override(s) above.
- Track re-pinning on every planned upgrade; nothing re-mirrors automatically.
auditConsumer and siemExport scale with replicas or spec.autoscaling.auditConsumer / spec.autoscaling.siemExport and deploy start-before-stop (new pods start before the old ones stop). The audit consumer’s insert is idempotent, so records read twice during the overlap are written once. On Kinesis, a readiness probe on both consumers holds the old pods until each new pod holds its share of the shards; a backend image without that signal passes the probe after 4 minutes, so each of its rollouts waits that long. On Kinesis each replica leases whole shards, so replicas past the stream’s open shard count sit idle; keep replicas / maxReplicas at or below it. On Kinesis, run Runlayer backend 1.28.775 or later (the first release with shard leasing) before going past one replica, and before upgrading to this operator if you run siemExport. An older image has no leases, so extra replicas, and the overlap of every deploy, read every shard: SIEM export exports duplicates (never loss; dedupe on audit_log_id), while the audit consumer writes each record once but multiplies Aurora writer load by the replica count. Pods on the same Kinesis enhanced fan-out consumer also fight over each shard’s subscription: a SubscribeToShard 5 s or more after another takes it over, so they keep cutting each other off. On Pub/Sub replicas share the subscription, with no shard cap. A rollout briefly runs twice the current pod count (maxSurge: 100%, the ECS 200/100), even above maxReplicas; on Kinesis the pods past the shard count idle but still count against cluster quota until the old pods stop. Resharding the audit stream (shard count or capacity mode, managed out of band) requires Runlayer backend 1.28.2313 or later (the first release with the ENG-6765 fix, #19256), or SIEM export skips the first records of new child shards. Session materializer always runs two Deployments: ingest (session-materializer, the stream consumer) and compactor (session-materializer-compactor, app.entrypoints.session_compactor), both on the {instance}-session-materializer ServiceAccount. On Kinesis both scale: set ingest replicas (up to the shard count; rolls start-before-stop) and compactor.replicas (default 1; compactor.resources default 1000m / 4096Mi, one fold per 4Gi). On Pub/Sub both stay at ≤ 1 (CEL). The operator rolls ingest only after the compactor has rolled out (SessionMaterializerReady reason WaitingForRollout meanwhile), so upgrade the operator before spec.version: an image without the compactor entrypoint leaves ingest on its previous image instead of running without a compactor. It also injects SESSION_* / HOOK_EVENTS_* into backend/worker (payload read + hook publish). For Pub/Sub, provision the hook-events subscription with message ordering enabled; publishers use ordering_key=session_id. Publish RPC timeout defaults to 5s (HOOK_EVENTS_GCP_PUBSUB_PUBLISH_TIMEOUT_SECONDS); override via that env or hookEvents.publishTimeoutSeconds on the CR. Infra (streams, checkpoints, buckets, ordered subscription) is out-of-band; see External dependencies. spec.agents env (RUNLAYER_AGENT_SANDBOX*) is injected onto backend and worker (and onto agentEventTrigger when that component is enabled) — see External dependencies. Procedures validation runs in the same sandbox runtime, so Procedures need spec.agents configured too; without it, creating or validating a Procedure returns 503.
agents.filesBucket is a bucket you own, so three things on it are yours to set:
- The backend role needs
s3:PutObjectTaggingalongsides3:PutObject. Files the backend hands a run (Slack attachments) are uploaded with arunlayer-kind=run-inputtag, and a taggedPutObjectis denied without that action — the upload fails and the attachment is silently dropped. Grants3:PutObjectTaggingon the agent-files bucket in the backend role (see Workload identity). - Those run inputs live under
sessions/{agent_id}/workspace/.runlayer-inputs/{run_scope}/inside the durable per-agent prefix, and nothing deletes them. They are only ever readable by the run they were delivered to, but to stop them accumulating add a lifecycle rule expiring objects taggedrunlayer-kind=run-inputafter ~7 days (plusnoncurrent_version_expirationif the bucket is versioned). Scope the rule to thesessions/prefix as well as the tag: published artifacts are copies that live underartifacts/, and a tag-only rule would reach them too. The ECS module ships this rule asexpire-run-inputs. - Artifact uploads from disk stage files under
artifact-uploads/until they are published, and publishing deletes them. The backend role needss3:GetObjectto read a staged file ands3:DeleteObjectto discard it on the agent-files bucket, alongsides3:PutObjectands3:PutObjectTagging; without Get, publishing fails as storage unavailable. To stop abandoned uploads (up to 25 MB each) accumulating, add a lifecycle rule expiring theartifact-uploads/prefix after 1 day (plusnoncurrent_version_expirationif the bucket is versioned). Nothing durable lives under that prefix. The ECS module ships this rule asexpire-artifact-uploads.
K8s Agent Sandbox (sandboxMode: k8s)
In-cluster runtime using kubernetes-sigs/agent-sandbox (SandboxClaim + SandboxWarmPool). The operator only wires env; it does not install the controller, Template, WarmPool, or isolation runtime.
1. Install the controller
2. Isolation on the nodes (gVisor or Kata — pick one)
Sandbox pods must not use the host
runc kernel. Nodes that schedule them need a RuntimeClass (kubectl get runtimeclass):
GKE add-on admission also requires
automountServiceAccountToken: false, runAsNonRoot, drop ALL capabilities, CPU+memory limits, and nodeSelector/toleration sandbox.gke.io/runtime=gvisor.
3. Template + WarmPool + RBAC, then set spec.agents.sandboxMode: k8s + spec.agents.k8s.warmPool. Ask Runlayer for Agent Sandbox Template / WarmPool examples if you choose sandboxMode: k8s.
Deploy (spec.deploy)
When enabled, backend/worker get
RUNLAYER_DEPLOY=K8S and related env; frontend gets PUBLIC_RUNLAYER_DEPLOY=K8S.
Wait for Ready
Pending → Installing → Migrating / Upgrading → Ready (or Degraded / Failed).
Available=True only when every enabled core and optional component Deployment is Ready (for example ToolguardReady, TopicCpuReady, TopicCpuFastReady, LLMGatewayReady). Disabled optionals are ignored. Ingress and Gateway are separate: when either is enabled, also watch IngressReady or GatewayReady for traffic (on GCE, IngressReady includes the llm-gateway GLB backend when the Ingress routes to it).
Then create MCP env Secrets in the MCP namespace and apply MCPServer CRs (create MCPServer CRs in the MCP namespace after Ready).
6. Optional: runlayer-instance Helm chart
The optional runlayer-instance Helm chart (same Customer Distribution OCI registry as the operator) renders one RunlayerInstance CR per Helm release. Prefer it for GitOps when you do not apply the CR with kubectl/Terraform.
v1 owns only the CR — Secrets and (when not using
spec.ingress) Ingress stay outside Helm.
name, namespace, labels, annotations, and spec (full RunlayerInstanceSpec passthrough). Do not put tenant CRs in the operator chart — operator upgrades must not re-render tenants.
7. Ingress, NetworkPolicy, scheduling, resources
Ingress
Opt-in (spec.ingress.enabled, default false). When enabled, the operator reconciles path routing:
On
gce / gce-internal, the operator creates BackendConfigs (backend, frontend, and llm-gateway when enabled) before annotating Services / applying Ingress so GKE GLBC does not stick on missing BackendConfig. Set controllerNamespace so NetworkPolicies allow the ingress controller → backend (AWS LBC, nginx). GKE also uses GCP health-check CIDR defaults.
On gce / gce-internal, and on Gateway classes gke-l7-*, the operator sets PRE_AUTH_TRUSTED_PROXY_HOPS=2 on backend-image workloads when every enabled edge is a Google load balancer. Google’s load balancer appends <client-ip>, <lb-ip> to X-Forwarded-For, and per-IP rate limits, the audit ip_address and the meta.request.ip policy field must all key on the client, not the load balancer. Other classes keep the backend default of 1 (ALB and nginx append only the client). A one-hop class beside a Google edge is rejected. With an external Ingress or Gateway, set the value yourself in runlayer-app (see App secret).
Do not enable certManager and gcp.managedCertificate on the same instance.
Gateway
Opt-in (spec.gateway.enabled, default false). Independent of Ingress: enabling one does not delete the other. The path table matches Ingress, including /anthropic and /openai when components.llmGateway.enabled is true. paths is a full replace. GKE does not allow one Service on both an Ingress and a Gateway, so gce / gce-internal and gke-l7-* together are rejected. A one-hop edge (ALB, nginx, or any other non-Google class, including attach mode with no gatewayClassName) beside a Google edge is also rejected: the backend would treat a caller-supplied X-Forwarded-For entry as the client IP.
gatewayClassName: alb reconciles LoadBalancerConfiguration (internet-facing, TLS 1.2 policy, optional aws.certificateArn) and a TargetGroupConfiguration per Service (IP targets, the same health checks as Ingress, frontend stickiness). The AWS controller ignores Kubernetes TLS Secrets. Gateway, HTTPRoutes, Certificates, and target-group configs stay in the platform namespace. In attach mode the load balancer config is the exception: it is created in each parent Gateway’s namespace, because infrastructure.parametersRef cannot name an object in another namespace. Set that Gateway’s parametersRef to status.gateway.loadBalancerConfiguration. The object name is {instance namespace}.{instance name}-lb-config, so two instances can share one Gateway namespace. A hyphen would collide (team-a/prod and team/a-prod), and a namespace cannot contain .. Hostname discovery uses spec.domain when certificateArn is empty. The operator deletes a Gateway object only when it carries this instance’s UID, so a customer Gateway that shares an inventory name is left in place.
gatewayClassName: gke-l7-* reconciles HealthCheckPolicy and GCPBackendPolicy per Service. TLS on a managed Gateway is one of a Kubernetes Secret (tls / certManager), gcp.certMap, or gcp.preSharedCerts. GKE Gateway does not support ManagedCertificate. gcp.staticIPName becomes a named address. The operator does not create Certificate Manager resources. Regional classes (gke-l7-rilb, gke-l7-regional-external-managed) need an active proxy-only subnet in the same region and VPC. Google rejects the forwarding rule without one, and the operator does not create the subnet. Global classes do not use it.
Any other class gets a generic Gateway and HTTPRoute. certManager reconciles a Certificate. The HTTPS listener is added only after the TLS Secret exists, and HTTP then redirects to HTTPS.
The cluster must already have the Gateway API CRDs. The operator does not install a GatewayClass. When Gateway is enabled, watch GatewayReady.
NetworkPolicies
Defaults isolate tenants and allow required DNS / platform / MCP paths. Extensions are append-only:deploy.enabled: true, platform policies come from RunlayerInstance; MCP policies come from each MCPServer.spec.security.networkPolicy.
Workload scheduling
spec.workloadScheduling— defaultnodeSelector/affinity/tolerationsfor platform pods + migration Jobspec.components.<name>.scheduling— per-component override (e.g. ToolGuard → GPU pool)
Default container resources
Whencomponents.<name>.resources is omitted, the operator applies these defaults:
¹ backend also carries a 2Gi ephemeral-storage request (no limit) for the audit-log spool; it is kept even when
components.backend.resources is overridden, unless the override sets an explicit ephemeral-storage request.
Override per component for smaller clusters. Topic CPU also enforces ECS-parity memory floors vs gunicornWorkers / fast.gunicornWorkers: the slow pool needs 4096 MiB per worker (one worker holds about 3 GiB of models resident and skill scans peak about 0.5 GiB above that), the fast pool 1024 MiB per worker and at least 2048 MiB. A RunlayerInstance below a floor fails validation, and a failed instance is not reconciled at all, not even a version bump, until the memory is raised or the worker count lowered. If you set deploy.resourceQuota.hard, ensure requests fit the quota.
Autoscaling and PDBs
-
spec.autoscaling.<component>— HPA whenenabled: trueand a CPU or memory target is set (backend,frontend,worker,sentryRelay,llmGateway,auditConsumer,siemExport) -
llmGatewaygets a default HPA (70% CPU, min 1, max 3) when enabled andautoscaling.llmGatewayis unset — setautoscaling.llmGateway.enabled: falseto disable -
auditConsumerandsiemExporthave no default HPA. The HPA averages CPU across pods, while ECS scales on the busiest task; shard leases spread load unevenly, so leave headroom in the target (ECS uses 75%). Keep their CPU request at 1 core or less, and set a request for every resource the HPA targets (the operator rejects a missing request, or a CPU request above 1 core; aresourcesoverride replaces the defaults, so it must carry those requests itself): the consumer is one process, so a pegged 2-core pod reads 50%. Before turning the HPA off, setcomponents.<component>.replicasto the current pod count and step it down, or every lease moves at once -
Default HPA
behavior(when unset): scale-up waits 60s and adds up to 50% or 2 pods per minute. Scale-down waits for a quiet fleet — 1800s forbackend(keeps replicas added by a spike until it is over), 300s for other components — and removes at most 10% per minute. Setbehaviorto replace the whole block. -
auditConsumerandsiemExportscale-down instead waits 900s and removes one pod per 900s: on Kinesis every scale-in moves shard leases. -
autoscaling.auditConsumer.kinesisLag(EKS,streamBackend: kinesis, needs a CPU target) adds pods when the audit consumer falls behind the stream while CPU stays under target, for example while it waits on the database writer: one more pod once the worst shard is more than 60s behind, two once it is more than 5 minutes behind, the same steps as ECS. CPU still removes pods. It readsAWS/KinesisSubscribeToShardEvent.MillisBehindLatest(Maximum) for theconsumerArnconsumer through KEDA 2.20 or later: the operator replacesaudit-consumer-hpawith a KEDAScaledObjectnamedaudit-consumer, and KEDA runs the HPAkeda-hpa-audit-consumer, which keeps the CPU target and bounds and, unless you setbehavior, waits 120s (not 60s) before scaling up. Install KEDA before the operator starts (restart the operator after installing it); without KEDA the operator keeps the CPU-only HPA and setsAutoscalingReadytoFalse(reasonKEDAUnavailable). KEDA needscloudwatch:GetMetricData: grant it to the KEDA operator’s IRSA or EKS Pod Identity role, or setkinesisLag.authenticationRefto a KEDATriggerAuthenticationin the platform namespace (or aClusterTriggerAuthentication,kind: ClusterTriggerAuthentication). If a ScaledObject you manage yourself (one without this instance’s owner labels or annotations) is already namedaudit-consumer, the operator never changes or deletes it. If it scales the audit consumer Deployment, the operator runs no HPA of its own (KEDA’s would fight it), andAutoscalingReadyisFalsesaying so; delete it to hand scaling back to the operator. If KEDA cannot read the metric for three polls in a row, the lag counts as zero, so CPU keeps scaling in, andAutoscalingReadyturnsFalse(reasonScaledObjectNotReady) naming the failing trigger until KEDA reads it again. The HPA ignores a change of 10% or less, so the one-pod step does nothing at 10 or more pods. The step counts running pods, so a rollout that takes longer than 120s while the lag stays above 60s can scale towardmaxReplicas; CPU then removes the extra pods one per 900s. If KEDA reports the ScaledObject not Ready (for example a missingTriggerAuthentication), or gives no verdict within 2 minutes (the KEDA CRDs are installed but its operator is not running), KEDA runs no HPA: the audit consumer keeps its current pod count without autoscaling, andAutoscalingReadyisFalsewith reasonScaledObjectNotReady. The operator does not bring back its own HPA then, because two HPAs would fight over the replica count once KEDA recovers. When you turnkinesisLagoff or delete the instance, the operator deletes the ScaledObject and waits for KEDA to remove it andkeda-hpa-audit-consumerbefore it recreatesaudit-consumer-hpa. If the KEDA CRDs are installed but no KEDA operator is running, it removes KEDA’sfinalizer.keda.shfrom the ScaledObject 2 minutes after deleting it, so neither step hangs. Pub/Sub has no lag scale-out. -
An HPA needs a request for every resource it targets, on every container in the pod, native sidecars included, and the pod’s requests must not sum to zero; otherwise that metric fails (
FailedGetResourceMetric). An HPA with no other working metric never scales; one with a working metric (say, memory beside a broken CPU target) still scales up on it but never scales down. Acomponents.<name>.resourcesoverride replaces the defaults, so a memory-only override drops the CPU request a CPU target needs (also checkotelCollector.resources, the backend’s sidecar). The operator still reconciles, and sets theAutoscalingReadycondition toFalse(reasonMissingResourceRequest) naming the component, resource and container:kubectl get runlayerinstance <name> -o jsonpath='{.status.conditions[?(@.type=="AutoscalingReady")]}'. A limit with no request counts. The check reads the operator’s pod spec, so a LimitRange default request or GKE Autopilot defaults can make it a false alarm (set the request explicitly to clear it), and it does not see sidecars injected by a mutating webhook. -
spec.podDisruptionBudget— optional PDBs for core Deployments
Trusted CA
spec.trustedCa mounts a customer CA ConfigMap and sets SSL_CERT_FILE / REQUESTS_CA_BUNDLE / NODE_EXTRA_CA_CERTS for enterprise TLS inspection.
OTEL collector sidecar
spec.otelCollector.enabled injects a native sidecar on the backend Deployment (Kubernetes ≥ 1.29). Requires reachable OTLP endpoints; hostname endpoints may need extra NetworkPolicy CIDRs. Configure reachable OTLP endpoints; see observability notes below. Name the Runlayer-operated endpoint runlayer-… and give it protocol: http: the sidecar exports its own health metrics (export queue depth, send failures, receiver refusals — no application data) to endpoints with that prefix and only those, which is how Runlayer monitors your deployment’s telemetry export.
Audit log spool
The backend’s local audit-log spool is enabled by default: audit records are appended to a spool on the pod’s local disk (a dedicatedemptyDir at /var/audit-spool) and delivered asynchronously by an in-process tailer, so MCP proxy requests do not wait on audit delivery. Records are typically delivered within seconds, and delivery is at-least-once with deduplication. If the spool cannot keep up or its disk budget fills, individual records fall back to inline delivery on the request path, so nothing is lost to backpressure.
- Node-disk encryption is required. The spool briefly holds pre-redaction audit payloads at rest on the node’s disk. On GKE, node disks are encrypted by default with Google-managed keys (CMEK optional). On EKS, encrypted EBS node volumes must be explicitly enabled. The operator cannot verify either.
- The backend pod’s shutdown drain window for the spool is 110 seconds (grace period 140s minus 30s headroom); records not delivered before the window expires are lost with the pod.
- After an outage of a delivery target (database or Kinesis), expect elevated write load while each backend pod drains its accumulated backlog. On small database instances,
AUDIT_LOG_SPOOL_DELIVERY_CONCURRENCY(default 8, per pod) can be lowered via the app secret to throttle the catch-up. - Each backend pod can hold up to ~512MB of spool data on node disk by default (
AUDIT_LOG_SPOOL_MAX_BYTES, settable viaspec.auditSpool.maxBytes; dead-letter bytes count against the same budget — a soft cap that worker processes re-check every 5 seconds, so bursts can briefly overshoot it). Budget node ephemeral storage accordingly when running many instances per cluster. The spool volume is bounded by an emptyDirsizeLimitderived as max(2 ×maxBytes, 1Gi) — raisingmaxBytesautomatically keeps the backstop above the budget — and backend pods carry an ephemeral-storage request (2Gi, no limit) so the scheduler accounts for the disk footprint. If spool usage ever exceeds thesizeLimit, the kubelet evicts the backend pod and its undelivered spool is discarded —spec.auditSpool.volumeSizeLimitoverrides the derivation; if you set it, keep it comfortably above themaxBytesbudget. - Alert on spool health. A spool that persistently rejects appends degrades silently to inline delivery (a latency regression, not data loss), and nothing alerts on it out of the box. See Monitoring the audit log spool below.
spec.auditSpool.enabled: false. New pods write audit logs inline again, and each old pod drains its residual spool within the shutdown window as it stops. No data migration is needed in either direction.
Disabling requires the operator 0.2.22 CRD: helm upgrade does not apply crds/, and against an older CRD the API server silently prunes spec.auditSpool, leaving the spool on. Apply the updated CRD first — see Upgrading the operator.
Monitoring the audit log spool
The spool degrades gracefully — records that it rejects are delivered inline instead of lost — so a persistently unhealthy spool is invisible unless you alert on it. The operator does not ship a spool alert; use the metrics and alert rules below to build one.Prerequisite: metrics export
The backend exports spool metrics over OTLP.spec.otelCollector.enabled (above) injects a collector sidecar and wires the backend’s OTLP/gRPC endpoint for you — but at least one configured endpoint must carry metrics: an endpoint restricted to signals: [traces] (or a configMapRef collector config without a metrics exporter) leaves the sidecar’s metrics pipeline on the nop exporter, and the spool metrics go nowhere.
Each backend worker process exports its own cumulative counters, tagged with a unique service.instance.id resource attribute. The operator’s generated sidecar config promotes that identity onto every datapoint, so the per-process series survive into your storage. If you replace the config via configMapRef, replicate that promotion (or enable resource_to_telemetry_conversion on a Prometheus/remote-write exporter): if the workers collapse into one series, the merged counters zigzag, every dip reads as a counter reset, and rate() / increase() overcount by orders of magnitude — the alerts below will page on fabricated spikes.
Metric names below are the Prometheus-translated forms (OTLP ingestion appends the _total / _seconds / _milliseconds / _bytes suffixes), as stored by Prometheus-compatible backends such as Mimir.
Fallback reasons
Every rejected append incrementsrunlayer_audit_spool_fallback_count_total with a reason label. Each reason has a different operational meaning:
Prometheus alert rules
A ready-to-adaptPrometheusRule (kube-prometheus-stack shape — adapt the labels/selectors to your stack, or lift the PromQL into whatever rule format your Prometheus uses). The operator does not ship these as manifests:
dead/ in the first place, the record is dropped outright. The matching counter (runlayer_audit_spool_dead_letter_write_failed_count_total) is disabled by default on self-hosted deployments, so a metric alert on it would silently never fire. If your stack supports log-based alerting, alert on the audit_spool_dead_letter_write_failed backend log line instead — it fires unconditionally.
Related docs
Kubernetes prerequisites
Cluster requirements for EKS and GKE
External dependencies
Postgres, Redis, streams, buckets + Terraform samples
Runlayer-provided inputs
Catalog key, WorkOS, download token, images
Workload identity
IRSA / GKE Workload Identity matrix
Deployment overview
Account strategy and isolation models
Network firewall
Client egress allowlists for the tenant hostname