Runlayer Operator (Kubernetes)

The unified Runlayer operator reconciles two custom resources in one controller:

CRD	Purpose
`RunlayerInstance`	Tenant platform stack (backend, frontend, worker, migrations, optional components)
`MCPServer`	Hosted MCP server Deployments (Runlayer Deploy)

Use this path for one or many isolated tenants on a shared Kubernetes cluster (EKS or GKE). For legacy single-tenant Helm installs, see Helm + Kubernetes (anysource-chart) until you cut over.

Do not install the unified operator on a cluster that already runs runlayer-deploy-operator or anysource-chart. Both reconcile MCPServer CRs. Cut over one cluster at a time — see Migration.

End-to-end flow

Provision cluster features and external per-tenant dependencies (Postgres, Redis, buckets, IAM / Workload Identity, streams).
Install the runlayer-operator Helm chart once per cluster.
Create (or let the operator create) the platform namespace and seed required Secrets.
Apply a RunlayerInstance (raw YAML, optional runlayer-instance Helm chart, or Terraform on AWS).
Wait for status.phase: Ready, point DNS at the load balancer, then create MCPServer CRs if using Runlayer Deploy.

1. External dependencies

The operator does not provision cloud infrastructure. Each tenant needs the following out-of-band (Terraform, console, or your platform).

Required for every tenant

Dependency	Purpose	AWS	GCP
PostgreSQL	Primary datastore	Aurora / RDS PostgreSQL (recommended)	Cloud SQL for PostgreSQL
Redis	Cache / queues	ElastiCache (TLS recommended)	Memorystore for Redis
WorkOS	Auth (`AUTH_CLIENT_ID`, `AUTH_API_KEY`)	Same (SaaS)	Same (SaaS)
App crypto secrets	`SECRET_KEY`, `MASTER_SALT` (≥32 chars in production)	Secrets Manager → K8s Secret	Secret Manager → K8s Secret
Container images	Backend, frontend, worker (+ optionals)	ECR (or Runlayer registry)	Artifact Registry / mirrored images
DNS + TLS	Public hostname = `spec.domain`	Route 53 + ACM (typical)	Cloud DNS + Google-managed / cert-manager cert

Network: cluster nodes (or pods via CNI) must reach Postgres and Redis on private connectivity (VPC, Private Service Connect, peering). Tighten egress with spec.security.networkPolicy.additionalEgressRules once CIDRs are known.

Optional (feature-dependent)

Dependency	When needed	AWS	GCP
Object storage	Audit payload offload, tool-output offload, profiler, binary packages, SIEM export	S3 buckets	GCS buckets
Audit stream + consumer	`components.auditConsumer.enabled`	Kinesis Data Streams + EFO consumer, DynamoDB checkpoint table, DLQ S3 bucket	Pub/Sub subscription, DLQ GCS bucket
SIEM export	`components.siemExport.enabled`	Same stream pattern + destination S3 bucket	Pub/Sub + destination GCS bucket
Session materializer	`components.sessionMaterializer.enabled`	Hook-events Kinesis + DynamoDB checkpoints + DLQ S3 + session-payloads S3	Hook-events Pub/Sub topic+subscription + DLQ GCS + session-payloads GCS
Agent sandbox runtime	`spec.agents.sandboxMode`	Lambda function, or Bedrock AgentCore runtime (+ execution role, optional VPC/PrivateLink)	In-cluster kubernetes-sigs Agent Sandbox (`sandboxMode: k8s`) — see K8s Agent Sandbox
IAM for pods	Bedrock, S3, Kinesis, Secrets Manager, etc.	IRSA roles annotated on ServiceAccounts	Workload Identity GSA bound to KSA
GPU nodes	ToolGuard with `gpu.enabled: true` (default)	GPU node group + NVIDIA device plugin	GPU node pool / Autopilot accelerator allowlist + NVIDIA drivers
Topic CPU / ToolGuard model images	`components.topicCpu` / `toolguard`	Customer Distribution `088332244652.dkr.ecr.us-east-1.amazonaws.com/runlayer/toolguard` pins	Same (or GAR mirror)
LLM gateway image + user API key Secret	`components.llmGateway`	Customer Distribution `runlayer/llm-gateway` + K8s Secret with personal API key	Same
MCP telemetry	Deploy telemetry sidecar	OTLP collector endpoint reachable from MCP pods	Same
OTEL collector sidecar	`spec.otelCollector.enabled`	Reachable OTLP backends (Mimir/Tempo/etc.)	Same
Ingress controller	`spec.ingress.enabled` or external Ingress	AWS Load Balancer Controller (`alb`)	GKE Ingress (`gce` / `gce-internal`) or nginx
Encrypted worker-node volumes	Audit log spool (on by default) — holds pre-redaction payloads on node disk	Encrypted EBS node volumes	CMEK / Google-managed node-disk encryption
cert-manager	Optional TLS via Certificate CR	ClusterIssuer / Issuer you own	Same (or use ManagedCertificate)

AWS helper modules (optional):

Module	Role
`terraform-eks`	EKS cluster, node groups, ALB controller, IRSA scaffolding
`terraform-runlayer-dependencies`	Per-tenant Aurora + Redis + Secrets Manager
`terraform-runlayer-operator`	Operator Helm install on existing EKS
`terraform-runlayer-instance`	SM → K8s Secrets sync + `RunlayerInstance` + ALB Ingress

GCP: use equivalent Cloud SQL / Memorystore / IAM / GKE Ingress resources; wire the same Secret names and CR fields. There is no first-party GCP Terraform module yet — follow the CR examples below.

Shared GCP project

When any component uses streamBackend: pubsub, set one project for the instance:

spec:
  gcp:
    projectId: acme-prod   # sets GCP_PROJECT_ID once on platform workloads

Do not put projectId on per-component pubsub blocks (subscription / DLQ only).

Identity model (IRSA / Workload Identity)

The operator creates dedicated ServiceAccounts per workload and never embeds long-lived cloud keys. Annotate via spec.serviceAccount:

SA (name pattern)	Typical permissions
`{instance}-backend`	S3 (audit/tool-output/profiler/packages), Bedrock, optional SM; with session materializer enabled: hook-events publish (Kinesis PutRecord / Pub/Sub publish) + session-payload bucket read
`{instance}-worker`	Same class of AWS/GCP APIs the worker needs; with session materializer enabled: same hook-events publish + session-payload read as backend
`{instance}-audit-consumer`	Kinesis/DynamoDB/S3 or Pub/Sub + GCS DLQ
`{instance}-siem-export`	Stream read + destination bucket write
`{instance}-session-materializer`	Hook-events stream read + payload bucket R/W + DLQ write (Kinesis/DynamoDB/S3 or Pub/Sub/GCS)

AWS (IRSA):

spec:
  serviceAccount:
    backend:
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/runlayer-acme-backend
    worker:
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/runlayer-acme-worker
    auditConsumer:
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/runlayer-acme-audit-consumer
    siemExport:
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/runlayer-acme-siem-export
    sessionMaterializer:
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/runlayer-acme-session-materializer

GCP (Workload Identity):

spec:
  serviceAccount:
    backend:
      annotations:
        iam.gke.io/gcp-service-account: runlayer-acme-backend@PROJECT.iam.gserviceaccount.com
    worker:
      annotations:
        iam.gke.io/gcp-service-account: runlayer-acme-worker@PROJECT.iam.gserviceaccount.com
    auditConsumer:
      annotations:
        iam.gke.io/gcp-service-account: runlayer-acme-audit@PROJECT.iam.gserviceaccount.com
    siemExport:
      annotations:
        iam.gke.io/gcp-service-account: runlayer-acme-siem@PROJECT.iam.gserviceaccount.com
    sessionMaterializer:
      annotations:
        iam.gke.io/gcp-service-account: runlayer-acme-session-materializer@PROJECT.iam.gserviceaccount.com

Prefer WI/IRSA over ECS-style assume-role env vars (AUDIT_LOG_SIEM_EXPORT_ASSUME_ROLE_ARN is not the K8s path).

2. Cluster requirements

Kubernetes version

Requirement	Value
Minimum	1.29+ (Helm `kubeVersion`; native sidecars / KEP-753 for optional OTEL collector)
Recommended	Latest EKS / GKE stable (Runlayer EKS module defaults to 1.33)

Shared features (EKS and GKE)

Feature	Why
Helm 3.8+, `kubectl` with cluster-admin (or equivalent)	Install CRDs + ClusterRole
CNI with NetworkPolicy support	Default isolation; Calico / Cilium / VPC CNI + policy engine as applicable
Ability to pull app images	ECR/GAR auth, imagePullSecrets, or public mirrors
DNS resolution from pods	CoreDNS; private hosted zones / Cloud DNS for RDS/Redis hostnames
Persistent outbound HTTPS	WorkOS, Sentry (optional), catalog API, upstream MCP SaaS
Metrics server	Required if you enable HPA (`spec.autoscaling`)

EKS-specific

Item	Notes
AWS Load Balancer Controller	For `ingressClassName: alb` or Terraform ALB Ingress
EBS CSI (if needed)	Not required by core platform pods (stateless); may be needed by add-ons
OIDC provider + IRSA	Annotate workload SAs as above
Managed node groups	Size for backend defaults (~4 CPU / 8 Gi) or override `components.*.resources`
GPU node group	Only if ToolGuard GPU enabled
Security groups	Allow nodes → RDS/Redis; ALB → NodePort/target groups for backend:8000 / frontend:80

GKE-specific

Item	Notes
GKE Ingress / Gateway	Operator first-class path is Ingress class `gce` or `gce-internal` (GLB)
Workload Identity	Enable on cluster; bind KSAs to GSAs
BackendConfig CRD	Used when `spec.ingress.gcp.backendConfig` is enabled (default for `gce*`)
ManagedCertificate	Optional Google-managed TLS via `spec.ingress.gcp.managedCertificate`
Private clusters	Ensure Cloud SQL Auth Proxy / PSC / authorized networks so pods reach SQL/Redis
GPU pools	Same ToolGuard considerations

Namespace layout

Namespace	Contents
`runlayer-system`	Operator Deployment; typically all `RunlayerInstance` CRs
`runlayer-<customerId>`	Platform workloads + platform Secrets (default)
`runlayer-<customerId>-mcps`	`MCPServer` CRs + MCP Deployments (when `deploy.enabled`)

Override with spec.deploy.platformNamespace / spec.deploy.mcpNamespace.

3. Install and upgrade the operator

Install (once per cluster)

Copy and customize values:

cp infra/aws-helm/runlayer-operator/values.example.yaml values-runlayer-operator.yaml

OCI (Customer Distribution — production): versions declared production-ready via the runlayer-operator-promote-customer-distribution workflow.

helm upgrade --install runlayer-operator \
  oci://088332244652.dkr.ecr.us-east-1.amazonaws.com/runlayer-helm-charts/runlayer-operator \
  --version <chart-version> \
  --namespace runlayer-system --create-namespace \
  -f values-runlayer-operator.yaml

OCI (staging / pre-promote): continuous push on main in account 412677576004. Override image.repository to the staging operator image when using those charts.

helm upgrade --install runlayer-operator \
  oci://412677576004.dkr.ecr.us-east-1.amazonaws.com/runlayer-helm-charts/runlayer-operator \
  --version <chart-version> \
  --namespace runlayer-system --create-namespace \
  -f values-runlayer-operator.yaml \
  --set image.repository=412677576004.dkr.ecr.us-east-1.amazonaws.com/runlayer-operator

From the monorepo chart:

helm upgrade --install runlayer-operator infra/aws-helm/runlayer-operator \
  --namespace runlayer-system --create-namespace \
  -f values-runlayer-operator.yaml

CRDs ship in the chart crds/ directory and are applied on first helm install only:

platform.runlayer.com/v1alpha1 — RunlayerInstance
deploy.runlayer.com/v1alpha1 — MCPServer

Common operator values

Values key	Purpose
`image.repository` / `image.tag`	Operator image
`leaderElection.enabled`	Default `true`; keep on when `replicaCount > 1`
`telemetryProxy.*`	Cluster-wide MCP telemetry sidecar defaults (overridable per tenant)
`mcpWorkloads.dedicatedNodes.*`	Fallback MCP node scheduling
`otel.*`	Operator traces
`metrics.*`	Controller metrics + optional Prometheus `ServiceMonitor`

Full reference: chart README and CHART-DOCS.md.

Upgrading the operator (manual CRD apply)

Helm does not upgrade CRDs on subsequent releases. When a release notes CRD schema changes, apply CRDs before upgrading the chart:

# Helm-owned CRDs use server-side apply — force conflicts if Helm installed them first
kubectl apply --server-side --force-conflicts \
  -f infra/aws-helm/runlayer-operator/crds/platform.runlayer.com_runlayerinstances.yaml
kubectl apply --server-side --force-conflicts \
  -f infra/aws-helm/runlayer-operator/crds/deploy.runlayer.com_mcpservers.yaml

helm upgrade runlayer-operator \
  oci://088332244652.dkr.ecr.us-east-1.amazonaws.com/runlayer-helm-charts/runlayer-operator \
  --version <new-chart-version> \
  --namespace runlayer-system \
  -f values-runlayer-operator.yaml

If CRDs were never Helm-managed, plain kubectl apply -f ... is enough.

Always apply CRDs from the same chart version you are upgrading to. Skipping CRD apply can leave the controller unable to reconcile new fields.

4. Prerequisites: namespace and Secrets

Install order for Secrets

The operator never creates, reads, or syncs Secret data (no secrets write RBAC). You must create Secrets in the platform namespace. Two workable sequences:

Sequence	Steps
A — Namespace first (recommended for GitOps/TF)	Create `runlayer-<customerId>` → create Secrets → apply `RunlayerInstance`
B — CR first	Apply `RunlayerInstance` (operator creates namespaces) → create Secrets → operator retries until Ready

AWS Terraform path (terraform-runlayer-instance) uses A (TF owns the namespace + Secret sync from Secrets Manager).

Required Secrets (platform namespace)

Secret	Keys	CR reference
`runlayer-db`	`username`, `password`	`database.credentials.*From`
`runlayer-redis`	`password` or `REDIS_URL`	`redis.credentials.passwordFrom` / `urlFrom` (omit credentials if Redis has no auth)
`runlayer-auth`	`AUTH_CLIENT_ID`, `AUTH_API_KEY`	`auth.clientIdFrom`, `auth.apiKeyFrom`
`runlayer-app`	`SECRET_KEY`, `MASTER_SALT` (+ optional keys)	`appSecretRef.name`

Example (sequence A):

kubectl create namespace runlayer-acme

kubectl create secret generic runlayer-db -n runlayer-acme \
  --from-literal=username='runlayer' \
  --from-literal=password='…'

kubectl create secret generic runlayer-redis -n runlayer-acme \
  --from-literal=password='…'

kubectl create secret generic runlayer-auth -n runlayer-acme \
  --from-literal=AUTH_CLIENT_ID='…' \
  --from-literal=AUTH_API_KEY='…'

kubectl create secret generic runlayer-app -n runlayer-acme \
  --from-literal=SECRET_KEY='…' \
  --from-literal=MASTER_SALT='…'

Optional component Secrets

Secret	When	Keys
`runlayer-catalog`	`components.mcpCatalogApi.enabled`	`API_KEY`; optional `SENTRY_DSN`
`runlayer-sentry-relay`	`components.sentryRelay.enabled`	`public_key`, `secret_key`, `id` (key names configurable)
`runlayer-llm-gateway`	`components.llmGateway.enabled`	User/personal API key (conventional key `apiKey`) — not an org key; holder needs `MANAGE_ORG_SETTINGS`
TLS Secret	Manual / cert-manager ingress	Referenced by `spec.ingress.tls.secretName`

App secret

Platform Secret named runlayer-app by convention (spec.appSecretRef.name). Mounted with envFrom on backend and worker (and related backend-image jobs). Every key in the Secret becomes an environment variable.

Required keys

Key	Purpose
`SECRET_KEY`	JWT / session signing (min 32 characters in production)
`MASTER_SALT`	Encryption salt (min 32 characters in production)

Common optional keys

Key	Purpose
`SENTRY_DSN`	Error reporting (if not using relay-only)
`MCP_CATALOG_API_KEY`	Catalog API auth (prefer `components.mcpCatalogApi.apiKeyFrom` when in-cluster catalog is enabled)
`MCP_CATALOG_API_URL`	External catalog base URL when not using in-cluster catalog
`PROFILER_API_KEY`	On-demand memory profiling endpoint
`RUNLAYER_DOWNLOAD_TOKEN`	Binary release discovery token
`ANTHROPIC_API_KEY` / `OPENAI_API_KEY`	LLM providers when not using Bedrock-only
`OAUTH_AS_SIGNING_KEY_PEM`	OAuth authorization-server signing key (PEM)
`BEDROCK_ACCESS_KEY_ID` / `BEDROCK_SECRET_ACCESS_KEY`	Only if not using IRSA/WI for Bedrock

AUTH_API_KEY may also appear here; it is also injected from spec.auth.apiKeyFrom (duplicate is harmless).

Common non-secret configuration via the same Secret

Because envFrom loads all keys, teams often put non-sensitive settings in runlayer-app as well (or use a separate ConfigMap pattern outside the operator). Typical production keys:

Key	Example / notes
`AWS_REGION` / `AWS_DEFAULT_REGION`	e.g. `us-east-1`
`BEDROCK_ENABLED`	`true` / `false`
`BEDROCK_REGION`	e.g. `us-east-1`
`BEDROCK_ASSUME_ROLE_TRUSTED_AWS_ACCOUNT_ID`	Account ID for customer Bedrock roles
`AUDIT_PAYLOAD_S3_BUCKET`	Large audit payload offload. The same bucket also receives audit-spool dead letters exported (redacted) by the backend under `audit-spool-dead-letters/`; that evidence expires after 30 days (S3 lifecycle rule) — replay dead letters, or copy the objects out, before then
`TOOL_OUTPUT_S3_BUCKET`	Tool output offload
`TOOLGUARD_FEEDBACK_S3_BUCKET`	ToolGuard feedback
`PROFILER_S3_BUCKET`	Memray profiles
`BINARY_PACKAGES_S3_BUCKET`	Cached binary packages
`RUNLAYER_DOWNLOADS_BASE_URL`	Default `https://downloads.runlayer.com`
`DB_POOL_SIZE` / `DB_MAX_OVERFLOW`	Connection pool tuning
`SENTRY_AUDIT_EVENTS_ENABLED`	Default `false`
`OAUTH_BROKER_URL`	If using oauth-broker
`DIRECTORY_SYNC_EVENT_CRON`	Worker directory sync schedule
`DIRECTORY_SYNC_EVENT_SYNC_ENABLED`	`true` / `false`

Any backend Settings field (see application config) can be supplied this way. Prefer CR fields for connectivity that the operator already knows how to wire (database, redis, auth, component toggles). On AWS, invoking Anthropic models on Bedrock needs a one-time form submission in the account those calls are made from, even when BEDROCK_ENABLED is false:

One-time AWS step. Anthropic requires a use case details form before its models can be invoked in an AWS account, and Runlayer Assistant always routes through Bedrock — so this applies even when BEDROCK_ENABLED is false.Submit it in the account the Bedrock calls come from (your deployment’s account, or the account behind a Bedrock AI provider role you configure): Bedrock console → Model catalog → any Anthropic model → Submit use case details. AWS accepts it once per account, or once at your AWS Organization’s management account — member accounts inherit that.Deploying outside the US? Also set RUNLAYER_ASSISTANT_BEDROCK_MODEL to your geography’s inference profile, for example au.anthropic.claude-opus-4-6-v1 in ap-southeast-2.If skipped, this leaves Bedrock-backed agents failing with Model use case details have not been submitted for this account — see troubleshooting for the CLI checks and Region caveats.

5. Deploy a RunlayerInstance

Required `spec` fields

Field	Description
`customerId`	Tenant id; immutable; default ns prefix `runlayer-<customerId>`
`domain`	Public hostname (no scheme); immutable; drives `APP_URL` / CORS
`environment`	Label → `ENVIRONMENT` (`production`, `staging`, …)
`version`	App version tag; mirrored to `status.observedVersion` when Ready
`images.backend` / `frontend` / `worker`	`repository:tag`
`database.host` + `credentials.*From`	Postgres
`redis.host`	Redis (credentials optional)
`auth.clientIdFrom` / `apiKeyFrom`	WorkOS
`appSecretRef.name`	Platform Secret for `envFrom`

Optional highlights: database.port (5432), database.name (runlayer), redis.port / tls / credentials, components, deploy, serviceAccount, workloadScheduling, security, trustedCa, autoscaling, podDisruptionBudget, ingress, otelCollector, auditSpool, deploymentTelemetry. Authoritative field-level reference: RunlayerInstance CRD design.

Minimal example (EKS)

apiVersion: platform.runlayer.com/v1alpha1
kind: RunlayerInstance
metadata:
  name: acme-prod
  namespace: runlayer-system
  labels:
    runlayer.com/customer-id: acme
spec:
  customerId: acme
  domain: acme.example.com
  environment: production
  version: "1.0.0"
  images:
    backend: 123456789012.dkr.ecr.us-east-1.amazonaws.com/runlayer/backend:1.0.0
    frontend: 123456789012.dkr.ecr.us-east-1.amazonaws.com/runlayer/frontend:1.0.0
    worker: 123456789012.dkr.ecr.us-east-1.amazonaws.com/runlayer/worker:1.0.0
  database:
    host: acme-db.cluster-xxx.us-east-1.rds.amazonaws.com
    port: 5432
    name: runlayer
    credentials:
      usernameFrom: { name: runlayer-db, key: username }
      passwordFrom: { name: runlayer-db, key: password }
  redis:
    host: acme-redis.xxx.cache.amazonaws.com
    port: 6379
    tls: true
    credentials:
      passwordFrom: { name: runlayer-redis, key: password }
  auth:
    clientIdFrom: { name: runlayer-auth, key: AUTH_CLIENT_ID }
    apiKeyFrom: { name: runlayer-auth, key: AUTH_API_KEY }
  appSecretRef:
    name: runlayer-app
  components:
    backend: { enabled: true, replicas: 2 }
    frontend: { enabled: true, replicas: 2 }
    worker: { enabled: true, replicas: 1 }
  deploy:
    enabled: true
  serviceAccount:
    backend:
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/runlayer-acme-backend
    worker:
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/runlayer-acme-worker
  ingress:
    enabled: true
    ingressClassName: alb
    controllerNamespace: aws-load-balancer-controller
    annotations:
      alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:123456789012:certificate/…
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/target-type: ip

Minimal example (GKE)

apiVersion: platform.runlayer.com/v1alpha1
kind: RunlayerInstance
metadata:
  name: acme-prod
  namespace: runlayer-system
spec:
  customerId: acme
  domain: acme.example.com
  environment: production
  version: "1.0.0"
  images:
    backend: us-east1-docker.pkg.dev/PROJECT/runlayer/backend:1.0.0
    frontend: us-east1-docker.pkg.dev/PROJECT/runlayer/frontend:1.0.0
    worker: us-east1-docker.pkg.dev/PROJECT/runlayer/worker:1.0.0
  database:
    host: 10.x.x.x   # Cloud SQL private IP or PSC endpoint
    credentials:
      usernameFrom: { name: runlayer-db, key: username }
      passwordFrom: { name: runlayer-db, key: password }
  redis:
    host: 10.y.y.y   # Memorystore
    port: 6379
    tls: true
  auth:
    clientIdFrom: { name: runlayer-auth, key: AUTH_CLIENT_ID }
    apiKeyFrom: { name: runlayer-auth, key: AUTH_API_KEY }
  appSecretRef:
    name: runlayer-app
  components:
    backend: { enabled: true, replicas: 2 }
    frontend: { enabled: true, replicas: 2 }
    worker: { enabled: true, replicas: 1 }
  deploy:
    enabled: true
  serviceAccount:
    backend:
      annotations:
        iam.gke.io/gcp-service-account: runlayer-acme-backend@PROJECT.iam.gserviceaccount.com
    worker:
      annotations:
        iam.gke.io/gcp-service-account: runlayer-acme-worker@PROJECT.iam.gserviceaccount.com
  ingress:
    enabled: true
    ingressClassName: gce
    gcp:
      managedCertificate:
        enabled: true
      staticIPName: acme-prod-ip

Optional components (snippet)

When enabling an optional component, set its images.* field (and any required Secret refs). The operator rejects the CR if those are missing. status.phase: Ready / Available wait for every enabled optional Deployment (not only backend/frontend/worker). Pull credentials for platform components: the operator sets no imagePullSecrets on platform component pods (spec.deploy.defaultImagePullSecrets applies to MCPServer pods only — the Helm chart’s toolguard.image.pullSecrets knob has no operator equivalent). The Customer Distribution ECR references above therefore only work when the cluster’s node role can pull cross-account (EKS in an AWS account belonging to the Runlayer org). Clusters outside the org — including all GKE — must point images.* at a registry they can reach: the GAR mirror or a mirror in their own registry.

spec:
  images:
    mcpCatalogApi: REGISTRY/mcp-catalog-api:TAG
    sentryRelay: REGISTRY/sentry-relay:TAG
    toolguard: REGISTRY/toolguard:TAG   # or 088332244652.dkr.ecr.us-east-1.amazonaws.com/runlayer/toolguard:…
    topicCpu: REGISTRY/toolguard:runlayer-multimodel-guard-TAG   # or 088332244652.dkr.ecr.us-east-1.amazonaws.com/runlayer/toolguard:…
    llmGateway: REGISTRY/runlayer/llm-gateway:TAG
  components:
    mcpCatalogApi:
      enabled: true
      apiKeyFrom: { name: runlayer-catalog, key: API_KEY }
    toolguard:
      enabled: true
      gpu:
        enabled: false   # CPU-only / Autopilot without GPU capacity
    topicCpu:
      enabled: true
      # gunicornWorkers: 1          # slow pool (memory >= workers * 2048 MiB)
      # fast:
      #   gunicornWorkers: 3        # memory >= max(workers * 1024, 2048) MiB
    llmGateway:
      enabled: true
      apiKeyFrom: { name: runlayer-llm-gateway, key: apiKey }
    auditConsumer:
      enabled: true
      streamBackend: kinesis   # or pubsub
      kinesis:
        streamArn: arn:aws:kinesis:…
        consumerArn: arn:aws:kinesis:…/consumer/…
        checkpointTableName: runlayer-audit-checkpoints
        deadLetterBucket: runlayer-audit-dlq
      # pubsub:
      #   subscriptionId: audit-log-sub
      #   deadLetterBucket: acme-audit-dlq
    siemExport:
      enabled: true
      streamBackend: kinesis
      kinesis:
        streamArn: arn:aws:kinesis:…
        consumerArn: arn:aws:kinesis:…/consumer/…
        checkpointTableName: runlayer-siem-checkpoints
        deadLetterBucket: runlayer-siem-dlq
      export:
        s3Bucket: customer-siem-bucket
        # gcsBucket: customer-siem-gcs   # when streamBackend: pubsub
    sessionMaterializer:
      enabled: true
      streamBackend: kinesis   # or pubsub
      sessionsReadFromPostgres: true
      kinesis:
        streamArn: arn:aws:kinesis:…:stream/hook-events
        checkpointTableName: runlayer-session-materializer-checkpoints
        deadLetterBucket: runlayer-session-materializer-dlq
      payload:
        s3Bucket: runlayer-session-payloads
      hookEvents:
        streamName: hook-events
        region: us-east-1
        s3Bucket: runlayer-hook-events-parquet
      # pubsub:
      #   subscriptionId: hook-events-sub
      #   deadLetterBucket: acme-session-materializer-dlq
      # payload:
      #   gcsBucket: acme-session-payloads
      # hookEvents:
      #   topicId: hook-events
      #   publishTimeoutSeconds: 5  # optional; backend default is 5s
  agents:
    sandboxMode: agentcore   # or k8s
    filesBucket: acme-agent-sandbox-files
    agentcore:
      runtimeArn: arn:aws:bedrock-agentcore:us-east-1:123456789012:runtime/acme_prod_agent_sandbox
    # k8s:
    #   warmPool: runlayer-agent-pool
    #   namespace: runlayer-acme-agents   # optional; defaults to platform NS
  serviceAccount:
    sessionMaterializer:
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::…:role/runlayer-session-materializer
        # GKE: iam.gke.io/gcp-service-account: …@PROJECT.iam.gserviceaccount.com

For streamBackend: kinesis on auditConsumer / siemExport / sessionMaterializer, replicas must be ≤ 1 (CEL). Session materializer is always singleton (CEL) for both backends — compaction safety. It also injects SESSION_* / HOOK_EVENTS_* into backend/worker (payload read + hook publish). For Pub/Sub, provision the hook-events subscription with message ordering enabled; publishers use ordering_key=session_id. Publish RPC timeout defaults to 5s (HOOK_EVENTS_GCP_PUBSUB_PUBLISH_TIMEOUT_SECONDS); override via that env or hookEvents.publishTimeoutSeconds on the CR. Infra (streams, checkpoints, buckets, ordered subscription) is out-of-band; see the CRD design — Session materializer. spec.agents env (RUNLAYER_AGENT_SANDBOX*) is injected onto backend and worker (and onto agentEventTrigger when that component is enabled) — see the CRD design — Agents.

K8s Agent Sandbox (`sandboxMode: k8s`)

In-cluster runtime using kubernetes-sigs/agent-sandbox (SandboxClaim + SandboxWarmPool). The operator only wires env; it does not install the controller, Template, WarmPool, or isolation runtime. 1. Install the controller

Cluster	Install
GKE Autopilot or Standard ≥ `1.35.2-gke.1269000`	Enable Agent Sandbox on GKE: `gcloud beta container clusters update CLUSTER --location=REGION --enable-agent-sandbox`
EKS / GKE without the add-on	OSS CRDs + controller from the project site / GitHub

2. Isolation on the nodes (gVisor or Kata — pick one) Sandbox pods must not use the host runc kernel. Nodes that schedule them need a RuntimeClass (kubectl get runtimeclass):

	gVisor	Kata Containers
Isolation	Userspace kernel (no nested virt)	MicroVM, dedicated kernel
`runtimeClassName`	`gvisor`	`kata-qemu`
GKE Autopilot	Yes (add-on)	No
GKE Standard	Pool with `--sandbox=type=gvisor` (COS)	Ubuntu `UBUNTU_CONTAINERD` + Intel N2 + nested virt. Not E2 / N2D / T2A / COS. Kata on GKE (not a Google product — no GKE SLA)
EKS	Install runsc on the sandbox node group	Nested virt + Kata install on those nodes

GKE add-on admission also requires automountServiceAccountToken: false, runAsNonRoot, drop ALL capabilities, CPU+memory limits, and nodeSelector/toleration sandbox.gke.io/runtime=gvisor. 3. Template + WarmPool + RBAC, then set spec.agents.sandboxMode: k8s + spec.agents.k8s.warmPool. Spike manifests and both RuntimeClass snippets: docs-internal/agent-sandbox-k8s/ (README).

Deploy (`spec.deploy`)

Field	Default	Notes
`enabled`	`false`	Creates MCP namespace; enables MCPServer reconcile
`platformNamespace`	`runlayer-<customerId>`
`mcpNamespace`	`runlayer-<customerId>-mcps`
`deleteMcpServersOnDeprovision`	`true`
`resourceQuota.hard`	—	Applied to both platform and MCP namespaces
`telemetryProxy.*`	—	Per-tenant override of operator Helm defaults
`kubernetes.supportedPlatforms`	`["x86"]`	`x86` and/or `arm`

When enabled, backend/worker get RUNLAYER_DEPLOY=K8S and related env; frontend gets PUBLIC_RUNLAYER_DEPLOY=K8S.

Wait for Ready

kubectl wait runlayerinstance/acme-prod -n runlayer-system \
  --for=jsonpath='{.status.phase}'=Ready --timeout=20m

kubectl get runlayerinstance acme-prod -n runlayer-system -o yaml

Phases: Pending → Installing → Migrating / Upgrading → Ready (or Degraded / Failed). Available=True only when every enabled core and optional component Deployment is Ready (for example ToolguardReady, TopicCpuReady, TopicCpuFastReady, LLMGatewayReady). Disabled optionals are ignored. Ingress is separate: when Ingress is enabled, also watch IngressReady for traffic (on GCE, that includes the llm-gateway GLB backend when the Ingress routes to it). Then create MCP env Secrets in the MCP namespace and apply MCPServer CRs (see MCPServer design).

6. Optional: `runlayer-instance` Helm chart

The runlayer-instance chart renders one RunlayerInstance CR per Helm release. Prefer it for GitOps when you do not use Terraform.

Chart	Owns
`runlayer-operator`	CRDs + controller (once per cluster)
`runlayer-instance`	One tenant CR (once per tenant)

v1 owns only the CR — Secrets and (when not using spec.ingress) Ingress stay outside Helm.

cp infra/aws-helm/runlayer-instance/values.example.yaml values-acme.yaml
# edit REPLACE / REGISTRY placeholders; ensure Secrets exist in platform ns

helm upgrade --install acme-prod infra/aws-helm/runlayer-instance \
  --namespace runlayer-system \
  -f values-acme.yaml

OCI example:

helm upgrade --install acme-prod \
  oci://412677576004.dkr.ecr.us-east-1.amazonaws.com/runlayer-helm-charts/runlayer-instance \
  --version 0.1.0 \
  --namespace runlayer-system \
  -f values-acme.yaml

Values: name, namespace, labels, annotations, and spec (full RunlayerInstanceSpec passthrough). Do not put tenant CRs in the operator chart — operator upgrades must not re-render tenants.

7. Ingress, NetworkPolicy, scheduling, resources

Ingress

Opt-in (spec.ingress.enabled, default false). When enabled, the operator reconciles path routing:

Path	Service	Port
`/api/`, `/.well-known`, `/mcp`, `/docs`, `/redoc`	`{name}-backend`	8000
`/anthropic`, `/openai`	`llm-gateway`	8090 (only when `components.llmGateway.enabled`)
`/`	`{name}-frontend`	80

Mode	Config
AWS ALB + ACM	`ingressClassName: alb` + `alb.ingress.kubernetes.io/certificate-arn`
GCP GLB + managed cert	`ingressClassName: gce` + `gcp.managedCertificate.enabled: true`
cert-manager	`certManager.enabled` + `issuerRef` + `tls.secretName`
External	Leave `ingress.enabled: false`; manage Ingress in Terraform / GitOps

On gce / gce-internal, the operator creates BackendConfigs (backend, frontend, and llm-gateway when enabled) before annotating Services / applying Ingress so GKE GLBC does not stick on missing BackendConfig. Set controllerNamespace so NetworkPolicies allow the ingress controller → backend (AWS LBC, nginx). GKE also uses GCP health-check CIDR defaults. Do not enable certManager and gcp.managedCertificate on the same instance.

NetworkPolicies

Defaults isolate tenants and allow required DNS / platform / MCP paths. Extensions are append-only:

spec:
  security:
    networkPolicy:
      enabled: true
      additionalEgressRules:
        - to:
            - ipBlock: { cidr: 10.0.0.0/8 }
          ports:
            - protocol: TCP
              port: 5432
            - protocol: TCP
              port: 6379

When deploy.enabled: true, platform policies come from RunlayerInstance; MCP policies come from each MCPServer.spec.security.networkPolicy.

Workload scheduling

spec.workloadScheduling — default nodeSelector / affinity / tolerations for platform pods + migration Job
spec.components.<name>.scheduling — per-component override (e.g. ToolGuard → GPU pool)

Default container resources

When components.<name>.resources is omitted, operator defaults match anysource-chart:

Component	CPU request	Memory request/limit
backend	4000m	8192Mi
frontend	250m	512Mi
worker	1000m	4096Mi
mcpCatalogApi	100m	256Mi / 512Mi limit
sentryRelay	1000m	2048Mi
toolguard	1000m	6144Mi
llmGateway	1000m	2048Mi
topicCpu (slow)	2000m	8192Mi
topicCpu.fast	1000m	6656Mi
auditConsumer / siemExport / agentEventTrigger	500m	1024Mi
sessionMaterializer	512m	3072Mi

Override per component for smaller clusters. Topic CPU also enforces ECS-parity memory floors vs gunicornWorkers / fast.gunicornWorkers. If you set deploy.resourceQuota.hard, ensure requests fit the quota.

Autoscaling and PDBs

spec.autoscaling.<component> — HPA when enabled: true and a CPU or memory target is set (backend, frontend, worker, sentryRelay, llmGateway)
llmGateway gets a default HPA (70% CPU, min 1, max 3) when enabled and autoscaling.llmGateway is unset — set autoscaling.llmGateway.enabled: false to disable
spec.podDisruptionBudget — optional PDBs for core Deployments

Trusted CA

spec.trustedCa mounts a customer CA ConfigMap and sets SSL_CERT_FILE / REQUESTS_CA_BUNDLE / NODE_EXTRA_CA_CERTS for enterprise TLS inspection.

OTEL collector sidecar

spec.otelCollector.enabled injects a native sidecar on the backend Deployment (Kubernetes ≥ 1.29). Requires reachable OTLP endpoints; hostname endpoints may need extra NetworkPolicy CIDRs. See the CRD design — OTEL.

Audit log spool

The backend’s local audit-log spool is enabled by default: audit records are appended to a spool on the pod’s local disk (a dedicated emptyDir at /var/audit-spool) and delivered asynchronously by an in-process tailer, so MCP proxy requests do not wait on audit delivery. Records are typically delivered within seconds, and delivery is at-least-once with deduplication. If the spool cannot keep up or its disk budget fills, individual records fall back to inline delivery on the request path, so nothing is lost to backpressure.

Node-disk encryption is required. The spool briefly holds pre-redaction audit payloads at rest on the node’s disk. On GKE, node disks are encrypted by default with Google-managed keys (CMEK optional). On EKS, encrypted EBS node volumes must be explicitly enabled. The operator cannot verify either.
The backend pod’s shutdown drain window for the spool is 110 seconds (grace period 140s minus 30s headroom); records not delivered before the window expires are lost with the pod.
After an outage of a delivery target (database or Kinesis), expect elevated write load while each backend pod drains its accumulated backlog. On small database instances, AUDIT_LOG_SPOOL_DELIVERY_CONCURRENCY (default 8, per pod) can be lowered via the app secret to throttle the catch-up.
Each backend pod can hold up to ~512MB of spool data on node disk (AUDIT_LOG_SPOOL_MAX_BYTES; dead-letter bytes count against the same budget). Budget node ephemeral storage accordingly when running many instances per cluster.

To revert to inline audit delivery, set spec.auditSpool.enabled: false. New pods write audit logs inline again, and each old pod drains its residual spool within the shutdown window as it stops. No data migration is needed in either direction. Disabling requires the operator 0.2.22 CRD: helm upgrade does not apply crds/, and against an older CRD the API server silently prunes spec.auditSpool, leaving the spool on. Apply the updated CRD first — see Upgrading the operator.

Migration from legacy charts

Prefer a greenfield cluster or a maintenance window.
Install runlayer-operator.
Create Secrets + RunlayerInstance; verify Ready.
Migrate MCP workloads to MCPServer CRs in the tenant MCP namespace.
Switch DNS to the new Ingress / ALB / GLB.
Uninstall legacy runlayer-deploy-operator and anysource-chart on that cluster only.

Detailed plan: gradual migration plan.

Legacy Helm (single tenant)

anysource-chart for traditional single-tenant installs

EKS + Terraform

Provision EKS, RDS, and Redis before the operator

Deployment overview

Account strategy and isolation models

Network firewall

Client egress allowlists for the tenant hostname

Design references (GitHub):

​End-to-end flow

​1. External dependencies

​Required for every tenant

​Optional (feature-dependent)

​Shared GCP project

​Identity model (IRSA / Workload Identity)

​2. Cluster requirements

​Kubernetes version

​Shared features (EKS and GKE)

​EKS-specific

​GKE-specific

​Namespace layout

​3. Install and upgrade the operator

​Install (once per cluster)

​Common operator values

​Upgrading the operator (manual CRD apply)

​4. Prerequisites: namespace and Secrets

​Install order for Secrets

​Required Secrets (platform namespace)

​Optional component Secrets

​App secret

​Required keys

​Common optional keys

​Common non-secret configuration via the same Secret

​5. Deploy a RunlayerInstance

​Required spec fields

​Minimal example (EKS)

​Minimal example (GKE)

​Optional components (snippet)

​K8s Agent Sandbox (sandboxMode: k8s)

​Deploy (spec.deploy)

​Wait for Ready

​6. Optional: runlayer-instance Helm chart

​7. Ingress, NetworkPolicy, scheduling, resources

​Ingress

​NetworkPolicies

​Workload scheduling

​Default container resources

​Autoscaling and PDBs

​Trusted CA

​OTEL collector sidecar

​Audit log spool

​Migration from legacy charts

​Related docs

Legacy Helm (single tenant)

EKS + Terraform

Deployment overview

Network firewall

End-to-end flow

1. External dependencies

Required for every tenant

Optional (feature-dependent)

Shared GCP project

Identity model (IRSA / Workload Identity)

2. Cluster requirements

Kubernetes version

Shared features (EKS and GKE)

EKS-specific

GKE-specific

Namespace layout

3. Install and upgrade the operator

Install (once per cluster)

Common operator values

Upgrading the operator (manual CRD apply)

4. Prerequisites: namespace and Secrets

Install order for Secrets

Required Secrets (platform namespace)

Optional component Secrets

App secret

Required keys

Common optional keys

Common non-secret configuration via the same Secret

5. Deploy a RunlayerInstance

Required `spec` fields

Minimal example (EKS)

Minimal example (GKE)

Optional components (snippet)

K8s Agent Sandbox (`sandboxMode: k8s`)

Deploy (`spec.deploy`)

Wait for Ready

6. Optional: `runlayer-instance` Helm chart

7. Ingress, NetworkPolicy, scheduling, resources

Ingress

NetworkPolicies

Workload scheduling

Default container resources

Autoscaling and PDBs

Trusted CA

OTEL collector sidecar

Audit log spool

Migration from legacy charts

Related docs