Skip to main content
The unified Runlayer operator reconciles two custom resources in one controller: Use this path for one or many isolated tenants on a Kubernetes cluster (EKS or GKE).
Do not install the unified operator on a cluster that already runs runlayer-deploy-operator. Both reconcile MCPServer CRs.

End-to-end flow

  1. Provision cluster features and external per-tenant dependencies (Postgres, Redis, buckets, IAM / Workload Identity, streams).
  2. Install the runlayer-operator Helm chart once per cluster.
  3. Create (or let the operator create) the platform namespace and seed required Secrets.
  4. Apply a RunlayerInstance (raw YAML, optional runlayer-instance Helm chart, or Terraform on AWS).
  5. Wait for status.phase: Ready, point DNS at the load balancer, then create MCPServer CRs if using Runlayer Deploy.
Supporting guides (read before install):

1. External dependencies

The operator does not provision cloud infrastructure. Each tenant needs the following out-of-band (Terraform, console, or your platform).

Required for every tenant

Network: cluster nodes (or pods via CNI) must reach Postgres and Redis on private connectivity (VPC, Private Service Connect, peering). Tighten egress with spec.security.networkPolicy.additionalEgressRules once CIDRs are known. Configure time synchronization on cluster nodes; Runlayer pods inherit the node clock. Use the cloud provider’s local time service (Amazon Time Sync on AWS; Google’s internal NTP service on GCP), or your approved internal source. The operator does not configure host NTP.

Optional (feature-dependent)

Sample Terraform for Postgres, Redis, buckets, streams, and identity is embedded in External dependencies and Workload identity (AWS and GCP side by side). Adapt the samples to your IaC — Runlayer does not require a specific Terraform module.

Shared GCP project

When any component uses streamBackend: pubsub, set one project for the instance:
Do not put projectId on per-component pubsub blocks (subscription / DLQ only).

Identity model (IRSA / Workload Identity)

The operator creates dedicated ServiceAccounts per workload and never embeds long-lived cloud keys. Annotate via spec.serviceAccount: AWS (IRSA):
GCP (Workload Identity):
Prefer WI/IRSA over ECS-style assume-role env vars (AUDIT_LOG_SIEM_EXPORT_ASSUME_ROLE_ARN is not the K8s path).

Saved Bedrock providers for monitor consumers

The EKS application policy already grants scoped saved-Bedrock role assumption for separately provisioned monitor consumers. Additional ServiceAccounts need explicit application_irsa_additional_service_accounts trust and matching role annotations. The saved provider role must trust that AWS identity with its configured external ID and grant model access; GKE consumers also need an AWS credential source because GCP Workload Identity does not authenticate to AWS.

2. Cluster requirements

Kubernetes version

Shared features (EKS and GKE)

EKS-specific

See Kubernetes prerequisites for the full cluster contract.

GKE-specific

Namespace layout

Override with spec.deploy.platformNamespace / spec.deploy.mcpNamespace. The operator only uses, labels, drains, or deletes a namespace it owns: one it created (labeled runlayer.com/instance-uid: <instance UID>) or a pre-created namespace annotated for that instance, including a platform namespace that also holds a co-located RunlayerInstance (see sequence A). Any other existing namespace, including one another instance owns, fails the instance with Available=False, reason NamespaceNotOwned, and the operator creates nothing there. This applies to the platform and MCP namespaces. An agents.k8s.namespace may be one you created or one shared with other instances; the operator refuses it only when it is another instance’s platform or MCP namespace or is annotated for another instance, and never drains or deletes it.

3. Install and upgrade the operator

Install (once per cluster)

Copy and customize values:
OCI (Customer Distribution — production): versions declared production-ready via the runlayer-operator-promote-customer-distribution workflow.
OCI (staging / pre-promote): continuous push on main in account 412677576004. Override image.repository to the staging operator image when using those charts.
Use the Customer Distribution OCI chart for production installs. Runlayer can share a chart tarball if OCI pull is blocked in your environment. CRDs ship in the chart crds/ directory and are applied on first helm install only:
  • platform.runlayer.com/v1alpha1 — RunlayerInstance
  • deploy.runlayer.com/v1alpha1 — MCPServer

Common operator values

Full values reference ships with the Helm chart you install from Customer Distribution ECR (values.example.yaml in the chart package).

Upgrading the operator (manual CRD apply)

Helm does not upgrade CRDs on subsequent releases. When a release notes CRD schema changes, apply CRDs before upgrading the chart:
If CRDs were never Helm-managed, plain kubectl apply -f ... is enough.
Always apply CRDs from the same chart version you are upgrading to. Skipping CRD apply can leave the controller unable to reconcile new fields.
Action required for chart 0.2.75 when you pre-create tenant namespaces: the operator no longer adopts an existing namespace unless it owns it. Namespaces the operator already reconciled carry the instance UID label and keep working. For a new instance whose platform or MCP namespace you create yourself (sequence A below), annotate it with runlayer.com/owner-instance-namespace and runlayer.com/owner-instance-name naming the RunlayerInstance first, or the instance fails with NamespaceNotOwned. See Install order for Secrets. Deleting an instance now leaves any namespace it does not own in place, removing only that instance’s labeled objects there. After restoring a cluster from backup, recreated RunlayerInstance objects get new UIDs, so their namespaces still carry the old UID label; annotate those namespaces the same way to hand them back.
Action required for chart 0.2.73 before using autoscaling.auditConsumer.kinesisLag: apply the 0.2.73 CRDs before helm upgrade; the old CRD silently drops kinesisLag. Before rolling back below 0.2.73, remove kinesisLag and wait for the operator to delete the audit-consumer ScaledObject and recreate audit-consumer-hpa. An older operator neither manages nor deletes the ScaledObject, so KEDA’s HPA and the operator’s HPA fight over the replica count.
Action required for chart 0.2.67 before using audit consumer scaling: this version adds spec.autoscaling.auditConsumer to the RunlayerInstance CRD and allows Kinesis components.auditConsumer.replicas above 1.
  • Upgrade: apply the 0.2.67 CRDs before helm upgrade, as shown above. The old CRD silently drops autoscaling.auditConsumer (no HPA gets created) and still rejects replicas above 1.
  • Rollback below 0.2.67: first set components.auditConsumer.replicas to the current pod count, remove autoscaling.auditConsumer, and wait for the operator to delete audit-consumer-hpa. An older operator neither manages nor deletes that HPA, so the two fight over the replica count and every change moves shard leases.
Action required for chart 0.2.61 with components.siemExport on Kinesis: this version rolls SIEM export start-before-stop. On a Runlayer backend older than 1.28.775, keep one replica and expect each deploy to export its in-flight batch twice (never loss; dedupe on audit_log_id) until the backend is upgraded. See Optional components below.

4. Prerequisites: namespace and Secrets

Install order for Secrets

The operator never creates, reads, or syncs Secret data (no secrets write RBAC). You must create Secrets in the platform namespace. Two workable sequences: Many teams use sequence A with their own Terraform syncing Secrets Manager / Secret Manager into the platform namespace. In sequence A, annotate the namespace with the RunlayerInstance that may adopt it, or the operator refuses it (NamespaceNotOwned). Do the same for a pre-created MCP namespace. A pre-created agents namespace needs no annotation. The terraform-runlayer-instance module annotates the platform namespace it creates; it does not create the MCP namespace, so annotate that one yourself if you pre-create it. Namespaces already reconciled by the operator carry the instance UID label and need nothing.

Required Secrets (platform namespace)

Before creating runlayer-app, obtain a Distribution reader key registered and active for this deployment from Runlayer. Set all three Distribution settings below so feature flags use Distribution. Example (sequence A):

MCP Catalog (required)

Onboarding calls the MCP Catalog API. Without a key, GET /api/v1/catalog/ returns 503 and setup shows “Couldn’t load setup status”. Contact Runlayer for the key — see Runlayer-provided inputs.

AI Watch binary distribution (when using AI Watch)

Installer resolution needs a Runlayer download token, a customer-owned object-storage cache, and workload permission to that cache. Without them the UI reports “No AI Watch installer is currently resolved”. After setting these, restart backend/worker and run release discovery (Check now or the scheduled poll). Details: External dependencies and Workload identity.

Optional component Secrets

App secret

Platform Secret named runlayer-app by convention (spec.appSecretRef.name). Mounted with envFrom on backend and worker (and related backend-image jobs). Every key in the Secret becomes an environment variable.

Required keys

Common optional keys

AUTH_API_KEY may also appear here; it is also injected from spec.auth.apiKeyFrom (duplicate is harmless). With terraform-runlayer-instance, the module copies RESEND_API_KEY from the tenant Secrets Manager secret into this Secret (empty when that key is absent); a key added to the Kubernetes Secret by hand is overwritten on the next apply, so put it in Secrets Manager: initial_tenant_secrets.resend_api_key on the dependencies module for a new tenant, or edit the existing tenant secret directly (the bootstrap resource ignores later changes).

Common non-secret configuration via the same Secret

Because envFrom loads all keys, teams often put non-sensitive settings in runlayer-app as well (or use a separate ConfigMap pattern outside the operator). Typical production keys:

AI Watch signed policy snapshot

AI Watch devices verify the policy snapshot served at GET /api/v1/ai-watch/config/signed offline. The backend signs it with this deployment’s key, and a Runlayer root delegation per organization tells devices to trust that key. Until both are set, the endpoint returns 404; it also does when either is invalid, and the backend logs ai_watch_policy_snapshot_signing_misconfigured. To generate the key and obtain or renew delegations, see Runlayer-provided inputs.
To keep the seed in its own Secret instead, add AI_WATCH_SNAPSHOT_SIGNING_KEY to spec.extraEnv with valueFrom.secretKeyRef. Kubernetes does not reload Secret-backed environment variables. After setting or rotating the seed, roll the pods by changing the runlayer.com/app-secret-hash annotation on the RunlayerInstance (the operator copies it onto the backend, worker and frontend pod templates):
A change to spec.extraEnv rolls the pods by itself. A delegation is bound to one key, so a new seed needs new delegations for every organization; change both in the same rollout. With terraform-runlayer-instance, put the seed in the tenant Secrets Manager secret as AI_WATCH_SNAPSHOT_SIGNING_KEY (initial_tenant_secrets.ai_watch_snapshot_signing_key on the dependencies module for a new tenant; edit an existing tenant secret directly) and pass the delegations JSON as ai_watch_snapshot_delegations. Each reaches runlayer-app only when non-empty, and the module sets the annotation from the Secret contents, so an apply rolls the pods. With the runlayer-instance chart, set spec.extraEnv and annotations in its values.

Runlayer support access

Runlayer support reaches your deployment’s /api/v1/debug/* endpoints through the Runlayer ops console, which signs every request. Your backend verifies the signature against your configured trusted keys and encrypts its response so only that request can read it. Support access also requires DEBUG_WORKOS_CLIENT_ID in the runlayer-app Secret, configured for Runlayer’s ops WorkOS environment, not the tenant’s. The backend accepts only tokens from Runlayer’s ops WorkOS organization: DEBUG_ALLOWED_WORKOS_ORG_ID defaults to it, so leave it unset. For MCP access, coordinate DEBUG_MCP_AUTHKIT_DOMAIN with Runlayer support. WorkOS authentication runs before signature verification; signing trust alone does not enable access. Obtain the reviewed production ops-console JWKS from Runlayer support and set its JSON contents as DEBUG_OPS_REQUEST_SIGNING_JWKS in the runlayer-app Secret (one line is fine). With terraform-runlayer-instance, pass the JSON string as debug_ops_request_signing_jwks. Do not trust staging console keys in production. Runlayer support coordinates key rotations separately from module releases; publish replacement keys before switching signing keys. Unset, blank, or empty trust disables support debug access once backend signature enforcement is deployed. Replay protection uses the existing writer Postgres database; the deployment’s migration Job creates its table automatically. If replay storage is unavailable, debug requests return 503 without running the handler. Runlayer Assistant inference. BEDROCK_ENABLED=true (typical AWS): new/unbound assistants use platform-billed Bedrock (RUNLAYER_ASSISTANT_BEDROCK_MODEL; IRSA/WI or BEDROCK_ACCESS_KEY_ID / BEDROCK_SECRET_ACCESS_KEY). BEDROCK_ENABLED=false (typical GCP/GKE): the assistant binds an org AI provider instead — recommended model if one is set, else the custom-agent default, else the oldest provider’s first model. Until an admin adds a provider in Settings → AI Providers, the assistant is still provisioned but chat is blocked. An admin can change the assistant’s model/provider in the agent panel (MANAGE_ORG_SETTINGS); a bound org provider is sticky and is not reset when Bedrock is later enabled. Prompt, name, connectors, and skills stay managed. Any backend Settings field (see application config) can be supplied this way. Prefer CR fields for connectivity that the operator already knows how to wire (database, redis, auth, component toggles). On AWS, when this deployment invokes Anthropic models on Bedrock (BEDROCK_ENABLED=true, or Bedrock-backed custom agent models), submit the one-time form in the account those calls come from:
Anthropic model access. Anthropic requires a use case details form before its models can be invoked in an AWS account, plus an accepted model agreement. You need this when the deployment invokes Anthropic models on Bedrock — typically BEDROCK_ENABLED=true (Runlayer Assistant platform inference and optional Bedrock agent models). With BEDROCK_ENABLED=false, Runlayer Assistant binds an org AI provider instead; skip this step unless you also enable Bedrock inference elsewhere (for example BEDROCK_ANTHROPIC_MODELS_ENABLED for custom agents).The backend does both for you at startup and on login, in the account the Bedrock calls come from, using the model-access IAM permissions your deployment grants. Four settings describe the deployment on the form — BEDROCK_MODEL_ACCESS_COMPANY_NAME (defaults to Runlayer, the operator of the software making the calls), BEDROCK_MODEL_ACCESS_COMPANY_WEBSITE, BEDROCK_MODEL_ACCESS_INDUSTRY and BEDROCK_MODEL_ACCESS_INTENDED_USERS. Override them if you run this install yourself. They are never derived from your data: the form is submit-once, so a guessed answer would be permanent.Set BEDROCK_AUTO_MODEL_ACCESS=false to turn this off and do it by hand instead: Bedrock console → Model catalog → any Anthropic model → Submit use case details. Do the same if your deployment role predates the model-access permissions — the backend logs exactly which actions it is missing. AWS accepts the form once per account, or once at your AWS Organization’s management account, which member accounts inherit.Deploying outside the US? Also set RUNLAYER_ASSISTANT_BEDROCK_MODEL to your geography’s inference profile, for example au.anthropic.claude-opus-4-6-v1 in ap-southeast-2. The risk-categorization explainer follows BEDROCK_REGION on its own and uses Claude Haiku 4.5 outside the US, so the Anthropic form above covers it; only set RISK_CATEGORIZATION_MODEL if you want a different model, and pick one served in your Region. If Bedrock rejects the primary model it falls back to Amazon Nova Lite, which needs no access request.Access takes up to 15 minutes to propagate. Until it does — or if this is skipped entirely — Bedrock-backed agents fail with Model use case details have not been submitted for this account; see troubleshooting for the CLI checks and Region caveats.

5. Deploy a RunlayerInstance

Required spec fields

Optional highlights: database.port (5432), database.name (runlayer), redis.port / tls / credentials, components, deploy, serviceAccount, workloadScheduling, security, trustedCa, autoscaling, podDisruptionBudget, ingress, otelCollector, auditSpool, extraEnv (extra environment variables for every backend-image workload; unlike the ECS module’s additional_backend_env_vars, a name the operator sets itself is rejected rather than overridden). spec.deploymentTelemetry is accepted and ignored. multiRegion (role: primary, region: <cloud region>) marks the instance as a member of a multi-region deployment: backend-image workloads get RUNLAYER_REGION_ROLE and a deployment.region resource attribute in front of any OTEL_RESOURCE_ATTRIBUTES you set in extraEnv. Leave it unset unless Runlayer asks you to set it; unset changes nothing. While it is set, declare OTEL_RESOURCE_ATTRIBUTES as a literal extraEnv value: a valueFrom entry is rejected, and a key of that name in the app secret is overridden. Field-level detail is covered in the sections below and in the supporting guides linked at the top of this page.

Minimal example (EKS)

Minimal example (GKE)

Optional components (snippet)

When enabling an optional component, set its images.* field (and any required Secret refs). The operator rejects the CR if those are missing. status.phase: Ready / Available wait for every enabled optional Deployment (not only backend/frontend/worker). Pull credentials for platform components: the operator sets no imagePullSecrets on platform component pods (spec.deploy.defaultImagePullSecrets applies to MCPServer pods only — the Helm chart’s toolguard.image.pullSecrets knob has no operator equivalent). Customer Distribution ECR (088332244652) supports cross-account pulls when Runlayer allowlists your AWS account or IAM pull principal during onboarding — use those image URIs directly on EKS once allowlisted. Prefer a GAR / private mirror for GKE, air-gapped networks, or orgs that require an internal registry copy.
extendDB needs no images.* entry — it defaults to a digest-pinned upstream ExtendDB image; set image only to override. It is the only optional component with a dedicated reconcile path: the init Job runs extenddb init (creates the catalog/data Postgres databases + a dedicated role + TLS cert from the bootstrap credentials, HTTPS port 18443) then mints a shared SigV4 access key and kubectl applys those Secrets/ConfigMaps into the platform and MCP namespaces (the reconciler never reads Secret payloads). Deployment/Service/PDB come up after that. RunlayerInstance.status.conditions reports ExtendDBReady: Deployment ready, credentials present, and the readiness probe’s cached Postgres connectivity check (extenddb catalog-check against both catalog and data DBs, refreshed at most every 5 minutes) — gated on the check’s connectivity banners only, so a catalog-integrity finding can’t take the whole Service NotReady. It is still not a live, between-ticks guarantee. ExtendDBReady gates MCPServer.spec.infrastructure.enable_db (rejected with “enable ExtendDB on the instance first” until then) — see infrastructure.enable_db. Enable ExtendDB on the instance before setting enable_db on MCPServers. MCPServers with enable_db still reconcile their Deployment while waiting; init failure does not wedge the rest of the instance. Init is create-once; a partial failure with no MCP keys synced yet gets one automatic reset (reconciler drops the half-created catalog/data DBs + role, then recreates the Job, capped by an instance annotation) before it needs a human — see Upgrading and recovering ExtendDB below. ExtendDB cannot be disabled while any MCPServer in the Deploy namespace still has enable_db: true (clear those first; undeploy deletes tables). Connection sizing for the shared Postgres host: External dependencies.

Upgrading and recovering ExtendDB

Bumping components.extendDB.image (or a helper image):
  • image only changes the Deployment — helm upgrade / re-apply rolls the existing pods to the new tag. It does not rerun extenddb init: TLS cert/key, the catalog/data Postgres schema, and the shared SigV4 access key are untouched. There is no auto-upgrade Job in v1 — if the new version needs a schema migration, run it yourself against the catalog/data DBs before or after the roll, per the upstream ExtendDB release notes.
  • kubectlImage / postgresClientImage / awsCliImage only affect their respective one-shot Jobs (init, reset, and per-MCPServer create-table/delete-table) on their next run — they don’t restart anything already Running. Bump them ahead of a planned init/reset, or anytime for the next MCPServer table Job.
  • Keep Secrets/state (do nothing beyond the CR edit) for a routine image bump — this is the common case and preserves the live token store.
  • Don’t delete a succeeded init Job expecting a fresh schema: the operator only recreates the Job, which reruns extenddb init against the existing catalog/data DBs. Init isn’t idempotent, so it just fails InitFailed — the MCP keys Secret already exists at this point, so the reset Job (and its refuse-guards) never runs.
  • To actually get a clean Postgres schema with no live token store: clear enable_db on (or undeploy) every MCPServer on the instance first, then drop the catalog/data DBs (and the app role, if needed) yourself with the bootstrap credential, then delete the init Job so the operator recreates it. This destroys any tokens already stored in those tables — only do it once nothing depends on them.
  • Prefer disable → re-enable when you just want the adapter back without touching Postgres — it keeps the succeeded init Job + Secrets and skips re-init entirely.
  • A second InitFailed after the one automatic reset, or any failure once MCP keys already exist: inspect the Job logs first (the condition message points there — “inspect the Job logs before doing anything manual” for a reset failure), then delete the Job and any partially-created catalog/data DBs before retrying. Once keys/tables exist, dropping the DBs destroys stored tokens.
Mirror checklist: the upstream extenddb-postgres image (and its helper images) are pulled directly from Docker Hub (docker.io/extenddb/...) today — not mirrored to Runlayer’s own registries. A Runlayer-run mirror is not yet available. Before relying on it for air-gapped or registry-restricted clusters:
  • Confirm the digest you’re pinning (image / kubectlImage / postgresClientImage / awsCliImage) is reachable from cluster nodes.
  • If not, mirror it into your own registry and set the CR override(s) above.
  • Track re-pinning on every planned upgrade; nothing re-mirrors automatically.
auditConsumer and siemExport scale with replicas or spec.autoscaling.auditConsumer / spec.autoscaling.siemExport and deploy start-before-stop (new pods start before the old ones stop). The audit consumer’s insert is idempotent, so records read twice during the overlap are written once. On Kinesis, a readiness probe on both consumers holds the old pods until each new pod holds its share of the shards; a backend image without that signal passes the probe after 4 minutes, so each of its rollouts waits that long. On Kinesis each replica leases whole shards, so replicas past the stream’s open shard count sit idle; keep replicas / maxReplicas at or below it. On Kinesis, run Runlayer backend 1.28.775 or later (the first release with shard leasing) before going past one replica, and before upgrading to this operator if you run siemExport. An older image has no leases, so extra replicas, and the overlap of every deploy, read every shard: SIEM export exports duplicates (never loss; dedupe on audit_log_id), while the audit consumer writes each record once but multiplies Aurora writer load by the replica count. Pods on the same Kinesis enhanced fan-out consumer also fight over each shard’s subscription: a SubscribeToShard 5 s or more after another takes it over, so they keep cutting each other off. On Pub/Sub replicas share the subscription, with no shard cap. A rollout briefly runs twice the current pod count (maxSurge: 100%, the ECS 200/100), even above maxReplicas; on Kinesis the pods past the shard count idle but still count against cluster quota until the old pods stop. Resharding the audit stream (shard count or capacity mode, managed out of band) requires Runlayer backend 1.28.2313 or later (the first release with the ENG-6765 fix, #19256), or SIEM export skips the first records of new child shards. Session materializer always runs two Deployments: ingest (session-materializer, the stream consumer) and compactor (session-materializer-compactor, app.entrypoints.session_compactor), both on the {instance}-session-materializer ServiceAccount. On Kinesis both scale: set ingest replicas (up to the shard count; rolls start-before-stop) and compactor.replicas (default 1; compactor.resources default 1000m / 4096Mi, one fold per 4Gi). On Pub/Sub both stay at ≤ 1 (CEL). The operator rolls ingest only after the compactor has rolled out (SessionMaterializerReady reason WaitingForRollout meanwhile), so upgrade the operator before spec.version: an image without the compactor entrypoint leaves ingest on its previous image instead of running without a compactor. It also injects SESSION_* / HOOK_EVENTS_* into backend/worker (payload read + hook publish). For Pub/Sub, provision the hook-events subscription with message ordering enabled; publishers use ordering_key=session_id. Publish RPC timeout defaults to 5s (HOOK_EVENTS_GCP_PUBSUB_PUBLISH_TIMEOUT_SECONDS); override via that env or hookEvents.publishTimeoutSeconds on the CR. Infra (streams, checkpoints, buckets, ordered subscription) is out-of-band; see External dependencies. spec.agents env (RUNLAYER_AGENT_SANDBOX*) is injected onto backend and worker (and onto agentEventTrigger when that component is enabled) — see External dependencies. Procedures validation runs in the same sandbox runtime, so Procedures need spec.agents configured too; without it, creating or validating a Procedure returns 503. agents.filesBucket is a bucket you own, so three things on it are yours to set:
  • The backend role needs s3:PutObjectTagging alongside s3:PutObject. Files the backend hands a run (Slack attachments) are uploaded with a runlayer-kind=run-input tag, and a tagged PutObject is denied without that action — the upload fails and the attachment is silently dropped. Grant s3:PutObjectTagging on the agent-files bucket in the backend role (see Workload identity).
  • Those run inputs live under sessions/{agent_id}/workspace/.runlayer-inputs/{run_scope}/ inside the durable per-agent prefix, and nothing deletes them. They are only ever readable by the run they were delivered to, but to stop them accumulating add a lifecycle rule expiring objects tagged runlayer-kind=run-input after ~7 days (plus noncurrent_version_expiration if the bucket is versioned). Scope the rule to the sessions/ prefix as well as the tag: published artifacts are copies that live under artifacts/, and a tag-only rule would reach them too. The ECS module ships this rule as expire-run-inputs.
  • Artifact uploads from disk stage files under artifact-uploads/ until they are published, and publishing deletes them. The backend role needs s3:GetObject to read a staged file and s3:DeleteObject to discard it on the agent-files bucket, alongside s3:PutObject and s3:PutObjectTagging; without Get, publishing fails as storage unavailable. To stop abandoned uploads (up to 25 MB each) accumulating, add a lifecycle rule expiring the artifact-uploads/ prefix after 1 day (plus noncurrent_version_expiration if the bucket is versioned). Nothing durable lives under that prefix. The ECS module ships this rule as expire-artifact-uploads.

K8s Agent Sandbox (sandboxMode: k8s)

In-cluster runtime using kubernetes-sigs/agent-sandbox (SandboxClaim + SandboxWarmPool). The operator only wires env; it does not install the controller, Template, WarmPool, or isolation runtime. 1. Install the controller 2. Isolation on the nodes (gVisor or Kata — pick one) Sandbox pods must not use the host runc kernel. Nodes that schedule them need a RuntimeClass (kubectl get runtimeclass): GKE add-on admission also requires automountServiceAccountToken: false, runAsNonRoot, drop ALL capabilities, CPU+memory limits, and nodeSelector/toleration sandbox.gke.io/runtime=gvisor. 3. Template + WarmPool + RBAC, then set spec.agents.sandboxMode: k8s + spec.agents.k8s.warmPool. Ask Runlayer for Agent Sandbox Template / WarmPool examples if you choose sandboxMode: k8s.

Deploy (spec.deploy)

When enabled, backend/worker get RUNLAYER_DEPLOY=K8S and related env; frontend gets PUBLIC_RUNLAYER_DEPLOY=K8S.

Wait for Ready

Phases: Pending → Installing → Migrating / Upgrading → Ready (or Degraded / Failed). Available=True only when every enabled core and optional component Deployment is Ready (for example ToolguardReady, TopicCpuReady, TopicCpuFastReady, LLMGatewayReady). Disabled optionals are ignored. Ingress and Gateway are separate: when either is enabled, also watch IngressReady or GatewayReady for traffic (on GCE, IngressReady includes the llm-gateway GLB backend when the Ingress routes to it). Then create MCP env Secrets in the MCP namespace and apply MCPServer CRs (create MCPServer CRs in the MCP namespace after Ready).

6. Optional: runlayer-instance Helm chart

The optional runlayer-instance Helm chart (same Customer Distribution OCI registry as the operator) renders one RunlayerInstance CR per Helm release. Prefer it for GitOps when you do not apply the CR with kubectl/Terraform. v1 owns only the CR — Secrets and (when not using spec.ingress) Ingress stay outside Helm.
Values: name, namespace, labels, annotations, and spec (full RunlayerInstanceSpec passthrough). Do not put tenant CRs in the operator chart — operator upgrades must not re-render tenants.

7. Ingress, NetworkPolicy, scheduling, resources

Ingress

Opt-in (spec.ingress.enabled, default false). When enabled, the operator reconciles path routing: On gce / gce-internal, the operator creates BackendConfigs (backend, frontend, and llm-gateway when enabled) before annotating Services / applying Ingress so GKE GLBC does not stick on missing BackendConfig. Set controllerNamespace so NetworkPolicies allow the ingress controller → backend (AWS LBC, nginx). GKE also uses GCP health-check CIDR defaults. On gce / gce-internal, and on Gateway classes gke-l7-*, the operator sets PRE_AUTH_TRUSTED_PROXY_HOPS=2 on backend-image workloads when every enabled edge is a Google load balancer. Google’s load balancer appends <client-ip>, <lb-ip> to X-Forwarded-For, and per-IP rate limits, the audit ip_address and the meta.request.ip policy field must all key on the client, not the load balancer. Other classes keep the backend default of 1 (ALB and nginx append only the client). A one-hop class beside a Google edge is rejected. With an external Ingress or Gateway, set the value yourself in runlayer-app (see App secret). Do not enable certManager and gcp.managedCertificate on the same instance.

Gateway

Opt-in (spec.gateway.enabled, default false). Independent of Ingress: enabling one does not delete the other. The path table matches Ingress, including /anthropic and /openai when components.llmGateway.enabled is true. paths is a full replace. GKE does not allow one Service on both an Ingress and a Gateway, so gce / gce-internal and gke-l7-* together are rejected. A one-hop edge (ALB, nginx, or any other non-Google class, including attach mode with no gatewayClassName) beside a Google edge is also rejected: the backend would treat a caller-supplied X-Forwarded-For entry as the client IP. gatewayClassName: alb reconciles LoadBalancerConfiguration (internet-facing, TLS 1.2 policy, optional aws.certificateArn) and a TargetGroupConfiguration per Service (IP targets, the same health checks as Ingress, frontend stickiness). The AWS controller ignores Kubernetes TLS Secrets. Gateway, HTTPRoutes, Certificates, and target-group configs stay in the platform namespace. In attach mode the load balancer config is the exception: it is created in each parent Gateway’s namespace, because infrastructure.parametersRef cannot name an object in another namespace. Set that Gateway’s parametersRef to status.gateway.loadBalancerConfiguration. The object name is {instance namespace}.{instance name}-lb-config, so two instances can share one Gateway namespace. A hyphen would collide (team-a/prod and team/a-prod), and a namespace cannot contain .. Hostname discovery uses spec.domain when certificateArn is empty. The operator deletes a Gateway object only when it carries this instance’s UID, so a customer Gateway that shares an inventory name is left in place. gatewayClassName: gke-l7-* reconciles HealthCheckPolicy and GCPBackendPolicy per Service. TLS on a managed Gateway is one of a Kubernetes Secret (tls / certManager), gcp.certMap, or gcp.preSharedCerts. GKE Gateway does not support ManagedCertificate. gcp.staticIPName becomes a named address. The operator does not create Certificate Manager resources. Regional classes (gke-l7-rilb, gke-l7-regional-external-managed) need an active proxy-only subnet in the same region and VPC. Google rejects the forwarding rule without one, and the operator does not create the subnet. Global classes do not use it. Any other class gets a generic Gateway and HTTPRoute. certManager reconciles a Certificate. The HTTPS listener is added only after the TLS Secret exists, and HTTP then redirects to HTTPS. The cluster must already have the Gateway API CRDs. The operator does not install a GatewayClass. When Gateway is enabled, watch GatewayReady.

NetworkPolicies

Defaults isolate tenants and allow required DNS / platform / MCP paths. Extensions are append-only:
When deploy.enabled: true, platform policies come from RunlayerInstance; MCP policies come from each MCPServer.spec.security.networkPolicy.

Workload scheduling

  • spec.workloadScheduling — default nodeSelector / affinity / tolerations for platform pods + migration Job
  • spec.components.<name>.scheduling — per-component override (e.g. ToolGuard → GPU pool)

Default container resources

When components.<name>.resources is omitted, the operator applies these defaults: ¹ backend also carries a 2Gi ephemeral-storage request (no limit) for the audit-log spool; it is kept even when components.backend.resources is overridden, unless the override sets an explicit ephemeral-storage request. Override per component for smaller clusters. Topic CPU also enforces ECS-parity memory floors vs gunicornWorkers / fast.gunicornWorkers: the slow pool needs 4096 MiB per worker (one worker holds about 3 GiB of models resident and skill scans peak about 0.5 GiB above that), the fast pool 1024 MiB per worker and at least 2048 MiB. A RunlayerInstance below a floor fails validation, and a failed instance is not reconciled at all, not even a version bump, until the memory is raised or the worker count lowered. If you set deploy.resourceQuota.hard, ensure requests fit the quota.

Autoscaling and PDBs

  • spec.autoscaling.<component> — HPA when enabled: true and a CPU or memory target is set (backend, frontend, worker, sentryRelay, llmGateway, auditConsumer, siemExport)
  • llmGateway gets a default HPA (70% CPU, min 1, max 3) when enabled and autoscaling.llmGateway is unset — set autoscaling.llmGateway.enabled: false to disable
  • auditConsumer and siemExport have no default HPA. The HPA averages CPU across pods, while ECS scales on the busiest task; shard leases spread load unevenly, so leave headroom in the target (ECS uses 75%). Keep their CPU request at 1 core or less, and set a request for every resource the HPA targets (the operator rejects a missing request, or a CPU request above 1 core; a resources override replaces the defaults, so it must carry those requests itself): the consumer is one process, so a pegged 2-core pod reads 50%. Before turning the HPA off, set components.<component>.replicas to the current pod count and step it down, or every lease moves at once
  • Default HPA behavior (when unset): scale-up waits 60s and adds up to 50% or 2 pods per minute. Scale-down waits for a quiet fleet — 1800s for backend (keeps replicas added by a spike until it is over), 300s for other components — and removes at most 10% per minute. Set behavior to replace the whole block.
  • auditConsumer and siemExport scale-down instead waits 900s and removes one pod per 900s: on Kinesis every scale-in moves shard leases.
  • autoscaling.auditConsumer.kinesisLag (EKS, streamBackend: kinesis, needs a CPU target) adds pods when the audit consumer falls behind the stream while CPU stays under target, for example while it waits on the database writer: one more pod once the worst shard is more than 60s behind, two once it is more than 5 minutes behind, the same steps as ECS. CPU still removes pods. It reads AWS/Kinesis SubscribeToShardEvent.MillisBehindLatest (Maximum) for the consumerArn consumer through KEDA 2.20 or later: the operator replaces audit-consumer-hpa with a KEDA ScaledObject named audit-consumer, and KEDA runs the HPA keda-hpa-audit-consumer, which keeps the CPU target and bounds and, unless you set behavior, waits 120s (not 60s) before scaling up. Install KEDA before the operator starts (restart the operator after installing it); without KEDA the operator keeps the CPU-only HPA and sets AutoscalingReady to False (reason KEDAUnavailable). KEDA needs cloudwatch:GetMetricData: grant it to the KEDA operator’s IRSA or EKS Pod Identity role, or set kinesisLag.authenticationRef to a KEDA TriggerAuthentication in the platform namespace (or a ClusterTriggerAuthentication, kind: ClusterTriggerAuthentication). If a ScaledObject you manage yourself (one without this instance’s owner labels or annotations) is already named audit-consumer, the operator never changes or deletes it. If it scales the audit consumer Deployment, the operator runs no HPA of its own (KEDA’s would fight it), and AutoscalingReady is False saying so; delete it to hand scaling back to the operator. If KEDA cannot read the metric for three polls in a row, the lag counts as zero, so CPU keeps scaling in, and AutoscalingReady turns False (reason ScaledObjectNotReady) naming the failing trigger until KEDA reads it again. The HPA ignores a change of 10% or less, so the one-pod step does nothing at 10 or more pods. The step counts running pods, so a rollout that takes longer than 120s while the lag stays above 60s can scale toward maxReplicas; CPU then removes the extra pods one per 900s. If KEDA reports the ScaledObject not Ready (for example a missing TriggerAuthentication), or gives no verdict within 2 minutes (the KEDA CRDs are installed but its operator is not running), KEDA runs no HPA: the audit consumer keeps its current pod count without autoscaling, and AutoscalingReady is False with reason ScaledObjectNotReady. The operator does not bring back its own HPA then, because two HPAs would fight over the replica count once KEDA recovers. When you turn kinesisLag off or delete the instance, the operator deletes the ScaledObject and waits for KEDA to remove it and keda-hpa-audit-consumer before it recreates audit-consumer-hpa. If the KEDA CRDs are installed but no KEDA operator is running, it removes KEDA’s finalizer.keda.sh from the ScaledObject 2 minutes after deleting it, so neither step hangs. Pub/Sub has no lag scale-out.
  • An HPA needs a request for every resource it targets, on every container in the pod, native sidecars included, and the pod’s requests must not sum to zero; otherwise that metric fails (FailedGetResourceMetric). An HPA with no other working metric never scales; one with a working metric (say, memory beside a broken CPU target) still scales up on it but never scales down. A components.<name>.resources override replaces the defaults, so a memory-only override drops the CPU request a CPU target needs (also check otelCollector.resources, the backend’s sidecar). The operator still reconciles, and sets the AutoscalingReady condition to False (reason MissingResourceRequest) naming the component, resource and container: kubectl get runlayerinstance <name> -o jsonpath='{.status.conditions[?(@.type=="AutoscalingReady")]}'. A limit with no request counts. The check reads the operator’s pod spec, so a LimitRange default request or GKE Autopilot defaults can make it a false alarm (set the request explicitly to clear it), and it does not see sidecars injected by a mutating webhook.
  • spec.podDisruptionBudget — optional PDBs for core Deployments

Trusted CA

spec.trustedCa mounts a customer CA ConfigMap and sets SSL_CERT_FILE / REQUESTS_CA_BUNDLE / NODE_EXTRA_CA_CERTS for enterprise TLS inspection.

OTEL collector sidecar

spec.otelCollector.enabled injects a native sidecar on the backend Deployment (Kubernetes ≥ 1.29). Requires reachable OTLP endpoints; hostname endpoints may need extra NetworkPolicy CIDRs. Configure reachable OTLP endpoints; see observability notes below. Name the Runlayer-operated endpoint runlayer-… and give it protocol: http: the sidecar exports its own health metrics (export queue depth, send failures, receiver refusals — no application data) to endpoints with that prefix and only those, which is how Runlayer monitors your deployment’s telemetry export.

Audit log spool

The backend’s local audit-log spool is enabled by default: audit records are appended to a spool on the pod’s local disk (a dedicated emptyDir at /var/audit-spool) and delivered asynchronously by an in-process tailer, so MCP proxy requests do not wait on audit delivery. Records are typically delivered within seconds, and delivery is at-least-once with deduplication. If the spool cannot keep up or its disk budget fills, individual records fall back to inline delivery on the request path, so nothing is lost to backpressure.
  • Node-disk encryption is required. The spool briefly holds pre-redaction audit payloads at rest on the node’s disk. On GKE, node disks are encrypted by default with Google-managed keys (CMEK optional). On EKS, encrypted EBS node volumes must be explicitly enabled. The operator cannot verify either.
  • The backend pod’s shutdown drain window for the spool is 110 seconds (grace period 140s minus 30s headroom); records not delivered before the window expires are lost with the pod.
  • After an outage of a delivery target (database or Kinesis), expect elevated write load while each backend pod drains its accumulated backlog. On small database instances, AUDIT_LOG_SPOOL_DELIVERY_CONCURRENCY (default 8, per pod) can be lowered via the app secret to throttle the catch-up.
  • Each backend pod can hold up to ~512MB of spool data on node disk by default (AUDIT_LOG_SPOOL_MAX_BYTES, settable via spec.auditSpool.maxBytes; dead-letter bytes count against the same budget — a soft cap that worker processes re-check every 5 seconds, so bursts can briefly overshoot it). Budget node ephemeral storage accordingly when running many instances per cluster. The spool volume is bounded by an emptyDir sizeLimit derived as max(2 × maxBytes, 1Gi) — raising maxBytes automatically keeps the backstop above the budget — and backend pods carry an ephemeral-storage request (2Gi, no limit) so the scheduler accounts for the disk footprint. If spool usage ever exceeds the sizeLimit, the kubelet evicts the backend pod and its undelivered spool is discarded — spec.auditSpool.volumeSizeLimit overrides the derivation; if you set it, keep it comfortably above the maxBytes budget.
  • Alert on spool health. A spool that persistently rejects appends degrades silently to inline delivery (a latency regression, not data loss), and nothing alerts on it out of the box. See Monitoring the audit log spool below.
To revert to inline audit delivery, set spec.auditSpool.enabled: false. New pods write audit logs inline again, and each old pod drains its residual spool within the shutdown window as it stops. No data migration is needed in either direction. Disabling requires the operator 0.2.22 CRD: helm upgrade does not apply crds/, and against an older CRD the API server silently prunes spec.auditSpool, leaving the spool on. Apply the updated CRD first — see Upgrading the operator.

Monitoring the audit log spool

The spool degrades gracefully — records that it rejects are delivered inline instead of lost — so a persistently unhealthy spool is invisible unless you alert on it. The operator does not ship a spool alert; use the metrics and alert rules below to build one.

Prerequisite: metrics export

The backend exports spool metrics over OTLP. spec.otelCollector.enabled (above) injects a collector sidecar and wires the backend’s OTLP/gRPC endpoint for you — but at least one configured endpoint must carry metrics: an endpoint restricted to signals: [traces] (or a configMapRef collector config without a metrics exporter) leaves the sidecar’s metrics pipeline on the nop exporter, and the spool metrics go nowhere. Each backend worker process exports its own cumulative counters, tagged with a unique service.instance.id resource attribute. The operator’s generated sidecar config promotes that identity onto every datapoint, so the per-process series survive into your storage. If you replace the config via configMapRef, replicate that promotion (or enable resource_to_telemetry_conversion on a Prometheus/remote-write exporter): if the workers collapse into one series, the merged counters zigzag, every dip reads as a counter reset, and rate() / increase() overcount by orders of magnitude — the alerts below will page on fabricated spikes. Metric names below are the Prometheus-translated forms (OTLP ingestion appends the _total / _seconds / _milliseconds / _bytes suffixes), as stored by Prometheus-compatible backends such as Mimir.

Fallback reasons

Every rejected append increments runlayer_audit_spool_fallback_count_total with a reason label. Each reason has a different operational meaning:

Prometheus alert rules

A ready-to-adapt PrometheusRule (kube-prometheus-stack shape — adapt the labels/selectors to your stack, or lift the PromQL into whatever rule format your Prometheus uses). The operator does not ship these as manifests:
One residual false-fire mode: after a metrics-ingestion gap of 15 minutes or more (collector or scrape-pipeline outage), the dead-letters rule’s birth branches can re-fire on pre-existing series when ingestion resumes — silence the alert on pipeline recovery rather than treating it as new loss. One loss path is not covered by these rules: when the tailer fails to write a poison record into dead/ in the first place, the record is dropped outright. The matching counter (runlayer_audit_spool_dead_letter_write_failed_count_total) is disabled by default on self-hosted deployments, so a metric alert on it would silently never fire. If your stack supports log-based alerting, alert on the audit_spool_dead_letter_write_failed backend log line instead — it fires unconditionally.

Kubernetes prerequisites

Cluster requirements for EKS and GKE

External dependencies

Postgres, Redis, streams, buckets + Terraform samples

Runlayer-provided inputs

Catalog key, WorkOS, download token, images

Workload identity

IRSA / GKE Workload Identity matrix

Deployment overview

Account strategy and isolation models

Network firewall

Client egress allowlists for the tenant hostname