> ## Documentation Index
> Fetch the complete documentation index at: https://docs.runlayer.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Helm + Kubernetes

> Deploy Runlayer on Kubernetes using Helm charts

<Note>
  **Multi-tenant EKS?** For shared clusters reconciling `RunlayerInstance` + `MCPServer` CRs, use the [Runlayer Operator](/deployment/runlayer-operator) chart instead of `anysource-chart`.
</Note>

## AWS EKS with Terraform (Recommended for Production)

For AWS production deployments, use the Terraform EKS module to provision infrastructure, then deploy the application with Helm. This provides better separation of concerns and security.

<Note>
  **New to EKS deployment?** See the comprehensive [EKS + Terraform deployment guide](/deployment/eks-terraform) for detailed instructions on provisioning EKS infrastructure.
</Note>

### Quick Start: EKS + Helm

**Step 1: Provision EKS Infrastructure**

```bash theme={null}
cd infra/aws-helm/terraform-eks

# Copy and edit configuration
cp terraform.tfvars.example terraform.tfvars
# Edit with your AWS account, VPC, and subnet details

# Deploy infrastructure
terraform init
terraform plan
terraform apply

# Configure kubectl
aws eks update-kubeconfig --region <your-region> --name <your-cluster-name>
```

**Step 2: Deploy Application with Helm**

```bash theme={null}
cd ../anysource-chart

# Add required repositories
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo add jetstack https://charts.jetstack.io
helm repo update

# Create secrets for external services
kubectl create secret generic anysource-db-secret \
  --from-literal=password="your-rds-password" \
  --namespace anysource

kubectl create secret generic anysource-redis-secret \
  --from-literal=password="your-redis-password" \
  --namespace anysource

# Deploy with AWS production values
# Option 1: Using chart directory
helm upgrade --install anysource . \
  --namespace anysource --create-namespace \
  -f values.example.yaml \
  --wait --timeout=10m

# Option 2: Using packaged chart (recommended for production)
helm upgrade --install anysource ./anysource-chart-1.0.0.tgz \
  --namespace anysource --create-namespace \
  -f values.example.yaml \
  --wait --timeout=10m
```

This approach provides:

* **Managed Infrastructure**: RDS, ElastiCache, VPC, IAM roles
* **Security**: IRSA, ALB with ACM certificates, private subnets
* **Scalability**: Auto-scaling groups, load balancing
* **Monitoring**: CloudWatch integration

For detailed EKS provisioning, see the [EKS + Terraform deployment guide](/deployment/eks-terraform) or the [Terraform EKS module README](https://github.com/runlayer/Runlayer/tree/main/infra/aws-helm/terraform-eks/README.md).

## AWS IAM Role Configuration

For AWS production deployments, you'll need to create an IAM role with the following configuration:

### Required Permissions

The role must have the following Bedrock permissions:

```json theme={null}
{
  "Effect": "Allow",
  "Action": [
    "bedrock:CreateGuardrail",
    "bedrock:GetGuardrail",
    "bedrock:ListGuardrails",
    "bedrock:UpdateGuardrail",
    "bedrock:DeleteGuardrail",
    "bedrock:ApplyGuardrail",
    "bedrock:InvokeModel",
    "bedrock:InvokeModelWithResponseStream"
  ],
  "Resource": "*"
}
```

It also needs AWS Marketplace permissions so the first invocation of a Bedrock model can auto-subscribe the account to the model's Marketplace offer (without these, model calls fail with a 403 access-denied error):

```json theme={null}
{
  "Effect": "Allow",
  "Action": [
    "aws-marketplace:Subscribe",
    "aws-marketplace:ViewSubscriptions"
  ],
  "Resource": "*"
}
```

### Trust Policy

The role must have this trust policy to allow EKS service account access:

```json theme={null}
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::<aws_account_id>:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/oidc_id"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-east-1.amazonaws.com/id/oidc_id:sub": "system:serviceaccount:<anysource_namespace>:anysource",
                    "oidc.eks.us-east-1.amazonaws.com/id/oidc_id:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}
```

Replace:

* `<aws_account_id>` with your AWS account ID
* `<oidc_id>` with your EKS cluster's OIDC provider ID
* `<anysource_namespace>` with your deployment namespace

### Configure in Values

Set the role ARN in your values file:

```yaml theme={null}
serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::<aws_account_id>:role/your-anysource-role"
```

## Overview

Deploy Runlayer on Kubernetes clusters using Helm charts. This guide covers multiple deployment patterns:

1. **AWS EKS + Terraform (Recommended)**: Provision infrastructure with Terraform, deploy application with Helm
2. **Development**: Use embedded databases for local testing
3. **Production**: External databases with high availability

**Helm Chart Location:** [`infra/aws-helm/anysource-chart/`](https://github.com/runlayer/Runlayer/tree/main/infra/aws-helm/anysource-chart/)

### AWS Production Architecture

The recommended AWS production deployment separates infrastructure provisioning from application deployment. The Terraform EKS module supports three deployment modes:

1. **Full Stack**: Create new VPC + new EKS cluster + application infrastructure
2. **Existing VPC**: Use existing VPC + create new EKS cluster + application infrastructure
3. **Existing EKS**: Use existing VPC + existing EKS cluster + application infrastructure only

```mermaid theme={null}
graph TB
    subgraph "1. Infrastructure Layer (Terraform)"
        EKS[EKS Cluster]
        RDS[(RDS PostgreSQL)]
        ElastiCache[(ElastiCache Redis)]
        ALB[ALB Load Balancer]
        VPC[VPC + Subnets]
        IAM[IAM Roles + IRSA]
    end

    subgraph "2. Application Layer (Helm)"
        Backend[Backend Pods]
        Frontend[Frontend Pods]
        Ingress[Ingress Controller]
        Secrets[Kubernetes Secrets]
        ConfigMaps[ConfigMaps]
    end

    Terraform --> EKS
    Terraform --> RDS
    Terraform --> ElastiCache
    Terraform --> VPC
    Terraform --> IAM

    Helm --> Backend
    Helm --> Frontend
    Helm --> Ingress
    Helm --> Secrets

    Backend --> RDS
    Backend --> ElastiCache
    Ingress --> ALB
```

<Note>
  For AWS production, first provision infrastructure with [EKS Terraform
  module](/deployment/eks-terraform), then deploy the application with this Helm
  chart. This separation provides better security, scalability, and operational
  control.
</Note>

## Quick Start

**Step 1: Navigate to the Helm chart**

```bash theme={null}
cd infra/aws-helm/anysource-chart
```

<Tabs>
  <Tab title="Minimal (Recommended)">
    **Get production-ready deployment with just 5 parameters:**

    ```bash theme={null}

    # Deploy with development values
    helm upgrade --install anysource . \
      --namespace anysource-dev --create-namespace \
      -f values-local.yaml \
      --atomic --wait

    # OR deploy with AWS production values (requires external RDS/ElastiCache)
    helm upgrade --install anysource . \
      --namespace anysource-prod --create-namespace \
      -f values.example.yaml \
      --atomic --wait
    ```

    **That's it!** You get production-ready defaults for everything else.

    <Note>
      **Deployment Telemetry**: The chart automatically sends deployment completion events to Sentry via Helm hooks for observability. Use `--atomic --wait` flags to ensure both success and failure events are captured. Telemetry gracefully degrades if SENTRY\_DSN is not configured.
    </Note>

    <Warning>
      Ensure your Kubernetes cluster has an ingress controller and certificate management for automatic SSL certificates.
      See the [SSL Certificate Management guide](/ssl-certificate-management) for detailed cert-manager vs ACM configuration.
    </Warning>
  </Tab>

  <Tab title="Custom Configuration">
    **Full control over all deployment settings:**

    ```bash theme={null}
    # Get full configuration template
    helm show values . > values-custom.yaml

    # Customize all settings as needed
    nano values-custom.yaml

    # Deploy with custom configuration
    helm upgrade --install anysource . \
      --namespace anysource --create-namespace \
      --values values-custom.yaml \
      --atomic --wait
    ```

    **100+ configuration options** for custom deployment.
  </Tab>
</Tabs>

## Configuration Options

## Prerequisites

<Steps>
  <Step title="Kubernetes Cluster">
    * Kubernetes 1.19+
    * Sufficient resources (4+ CPU cores, 8+ GB RAM)
    * Storage class for persistent volumes
    * Load balancer support (for ingress)
  </Step>

  <Step title="Required Tools">
    ```bash theme={null}
    # Helm 3.8+
    curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
    helm version

    # kubectl
    curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
    chmod +x kubectl && sudo mv kubectl /usr/local/bin/
    ```
  </Step>

  <Step title="AWS Production Requirements">
    * EKS cluster with AWS Load Balancer Controller
    * External RDS PostgreSQL and ElastiCache Redis
    * AWS ACM certificate
    * IAM role with Bedrock permissions and EKS OIDC trust policy
  </Step>

  <Step title="Ingress Controller (Required)">
    ```bash theme={null}
    # Install AWS Load Balancer Controller for EKS
    helm repo add eks https://aws.github.io/eks-charts
    helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
      --namespace kube-system \
      --set clusterName=your-cluster-name

    # OR use NGINX Ingress Controller
    helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
    helm install ingress-nginx ingress-nginx/ingress-nginx \
      --namespace ingress-nginx --create-namespace
    ```
  </Step>

  <Step title="Cert-Manager (Optional)">
    ```bash theme={null}
    # Install cert-manager for automatic SSL certificates (if not using AWS ACM)
    helm repo add jetstack https://charts.jetstack.io
    helm install cert-manager jetstack/cert-manager \
      --namespace cert-manager --create-namespace \
      --set installCRDs=true
    ```
  </Step>
</Steps>

## Deployment

### 1. Add Required Repositories

```bash theme={null}
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo add jetstack https://charts.jetstack.io
helm repo update
```

### 2. Configure Values

Copy and customize the example values file:

```bash theme={null}
cp values.example.yaml values-<environment>.yaml
# Edit values-<environment>.yaml with your specific configuration
```

### 3. Deploy Runlayer

<Tabs>
  <Tab title="Using Chart Directory (Iteration)">
    **Deploy directly from the chart source:**

    ```bash theme={null}
    cd infra/aws-helm/anysource-chart

    # With embedded databases (non-production)
    helm upgrade --install anysource . \
      --namespace anysource-dev --create-namespace \
      -f values-local.yaml \
      --atomic --wait

    # With external AWS services (production)
    helm upgrade --install anysource . \
      --namespace anysource --create-namespace \
      -f values.example.yaml \
      --atomic --wait
    ```

    Best for testing chart changes in non-production environments.
  </Tab>

  <Tab title="Using Packaged Chart (Production)">
    **Deploy from a packaged .tgz file:**

    ```bash theme={null}
    # Download the chart package (e.g., from your team or release artifacts)
    # Example: anysource-chart-1.0.0.tgz

    # With embedded databases (non-production)
    helm upgrade --install anysource ./anysource-chart-1.0.0.tgz \
      --namespace anysource-dev --create-namespace \
      -f values-local.yaml \
      --atomic --wait

    # With external AWS services (production)
    helm upgrade --install anysource ./anysource-chart-1.0.0.tgz \
      --namespace anysource --create-namespace \
      -f values.example.yaml \
      --atomic --wait
    ```

    **From a remote URL:**

    ```bash theme={null}
    helm upgrade --install anysource https://releases.example.com/anysource-chart-1.0.0.tgz \
      --namespace anysource --create-namespace \
      -f values.example.yaml \
      --atomic --wait
    ```

    Recommended for production deployments - ensures version-controlled, immutable releases.
  </Tab>
</Tabs>

<Note>
  Deployment takes 5-10 minutes. SSL certificate provisioning may add 2-5
  minutes.
</Note>

### 4. Verify

```bash theme={null}
# Check deployment status
kubectl get pods -n anysource
kubectl get ingress -n anysource
kubectl get hpa -n anysource

# View logs
kubectl logs -f deployment/anysource-backend -n anysource
kubectl logs -f deployment/anysource-frontend -n anysource

# Check events
kubectl get events -n anysource --sort-by='.lastTimestamp'
```

## Configuration

Copy `values.example.yaml` to create your environment-specific values file:

```bash theme={null}
cp values.example.yaml values-<environment>.yaml
```

### Required Configuration

Set these values in your values file:

```yaml theme={null}
global:
  domain: "your-domain.com"
  auth_client_id: "your-auth-client-id"

backend:
  secrets:
    SECRET_KEY: "your-jwt-secret-key-minimum-32-characters"
    MASTER_SALT: "your-master-salt-minimum-32-characters"
    AUTH_API_KEY: "your-auth-api-key"
```

### Database Configuration

**Embedded PostgreSQL (Development):**

```yaml theme={null}
postgresql:
  enabled: true
```

**External RDS (Production):**

```yaml theme={null}
postgresql:
  enabled: false
externalDatabase:
  enabled: true
  host: "your-rds-endpoint"
  readerHost: "your-rds-reader-endpoint"   # optional read-replica
  existingSecret: "anysource-db-secret"
```

Set `readerHost` to your RDS reader endpoint to offload read-heavy queries (audit logs, analytics) to a read replica. When omitted, all traffic goes to the primary host.

### Existing Backend Secret

By default the chart creates a Kubernetes Secret from the values in `backend.secrets`. To reference a pre-existing Secret instead (useful when secrets are managed by an external operator or Vault):

```yaml theme={null}
backend:
  existingSecret: "my-backend-secret"
```

The Secret must contain all keys the backend expects (`SECRET_KEY`, `MASTER_SALT`, `AUTH_API_KEY`, and optionally `SENTRY_DSN`). When `existingSecret` is set, `backend.secrets` values are ignored.

### Custom Trusted CA Bundle

If outbound HTTPS traffic passes through an enterprise proxy that terminates TLS with a private CA, mount the CA certificate so the backend trusts the proxy:

```yaml theme={null}
trustedCa:
  enabled: true
  pem: |-
    -----BEGIN CERTIFICATE-----
    ...your CA certificate...
    -----END CERTIFICATE-----
```

Alternatively, reference an existing ConfigMap:

```yaml theme={null}
trustedCa:
  enabled: true
  existingConfigMap: "enterprise-proxy-ca"
  key: "ca.crt"
```

The chart appends the PEM to the container's system CA bundle and configures Python/curl clients to use the merged file. The CA is injected into backend, worker, and cron job pods.

### Certificate Management

**AWS ACM (Recommended for AWS):**

```yaml theme={null}
awsCertificate:
  enabled: true
  arn: "arn:aws:acm:region:account:certificate/cert-id"
certManager:
  enabled: false
```

**Let's Encrypt (Non-AWS Deployment):**

```yaml theme={null}
certManager:
  enabled: true
  issuer:
    email: "your-email@example.com"
awsCertificate:
  enabled: false
```

## Architecture

```mermaid theme={null}
graph TB
    Internet[Internet] --> ALB[Application Load Balancer]
    ALB --> Frontend[Frontend Pods<br/>2 replicas]
    ALB --> Backend[Backend Pods<br/>2 replicas]

    Backend --> PostgreSQL[(PostgreSQL<br/>Primary + Replica)]
    Backend --> Redis[(Redis<br/>Master + Replica)]

    subgraph "Kubernetes Cluster"
        subgraph "anysource namespace"
            Frontend
            Backend
            PostgreSQL
            Redis
        end
    end
```

## Commands

### Install/Upgrade

<Tabs>
  <Tab title="Using Chart Directory">
    **Deploy directly from the chart directory:**

    ```bash theme={null}
    cd infra/aws-helm/anysource-chart

    helm upgrade --install anysource . \
      --namespace anysource --create-namespace \
      -f values-<environment>.yaml \
      --atomic --wait
    ```

    Use this when developing or testing chart changes locally.
  </Tab>

  <Tab title="Using Packaged Chart (.tgz)">
    **Deploy using a packaged Helm chart:**

    ```bash theme={null}
    # Download or obtain the packaged chart (e.g., anysource-chart-1.0.0.tgz)

    # Deploy from the packaged chart
    helm upgrade --install anysource ./anysource-chart-1.0.0.tgz \
      --namespace anysource --create-namespace \
      -f values-<environment>.yaml \
      --atomic --wait
    ```

    **Or use a remote URL:**

    ```bash theme={null}
    helm upgrade --install anysource https://your-repo/charts/anysource-chart-1.0.0.tgz \
      --namespace anysource --create-namespace \
      -f values-<environment>.yaml \
      --atomic --wait
    ```

    **Benefits of packaged charts:**

    * Version controlled and immutable deployments
    * Easy to distribute and archive
    * Can be stored in Helm repositories or object storage
    * Faster deployment (no need to clone repository)

    <Note>
      Always verify the chart package before deployment:

      ```bash theme={null}
      helm template anysource ./anysource-chart-1.0.0.tgz \
        -f values-<environment>.yaml --validate
      ```
    </Note>
  </Tab>
</Tabs>

### Uninstall

```bash theme={null}
helm uninstall anysource -n anysource
kubectl delete namespace anysource
```

### Debug

```bash theme={null}
# Debug with chart directory
helm template anysource . -f values-<environment>.yaml --debug --validate=false

# Debug with packaged chart
helm template anysource ./anysource-chart-1.0.0.tgz \
  -f values-<environment>.yaml --debug --validate=false
```

## Deployment Telemetry

The Helm chart automatically reports deployment status to Sentry via Helm hooks for observability and debugging.

### How It Works

The chart includes Helm hook Jobs that run automatically after each deployment:

* **post-upgrade hook**: Sends success event after successful deployment
* **post-rollback hook**: Sends failure event if deployment fails (triggered by `--atomic` flag)

Telemetry is enabled by default and gracefully degrades if `SENTRY_DSN` is not configured.

### Key Features

* **Automatic**: No wrapper scripts needed - hooks run on every `helm upgrade`
* **Secure**: SENTRY\_DSN only exposed to short-lived hook Jobs, not long-running pods
* **Reliable**: Exactly one event per deployment outcome (no duplicates on pod restarts)
* **Non-blocking**: Telemetry failures don't impact deployment success

### Configuration

Telemetry settings are read from your values file:

```yaml theme={null}
global:
  domain: "acme-corp.runlayer.com"  # Used as customer_id
  environment: "production"          # Environment tag
  deployment:
    infraVersion: "2.1.0"            # Infrastructure version for tracking

backend:
  secrets:
    SENTRY_DSN: "https://key@sentry.io/project"  # Required for telemetry

telemetry:
  enabled: true  # Enable/disable telemetry hooks (default: true)
```

### What Gets Reported

Each deployment event includes:

* Deployment status (success/failure)
* Customer ID (from `global.domain`)
* Infrastructure version (from `global.deployment.infraVersion`)
* Environment (from `global.environment`)
* Deployment duration and timestamp
* Operator and hostname metadata

### Disabling Telemetry

To disable telemetry hooks, set in your values file:

```yaml theme={null}
telemetry:
  enabled: false
```

### Advanced: Pre-Deployment Diff Telemetry

For pre-deployment diff capture (optional), use the wrapper script:

```bash theme={null}
cd infra/aws-helm/anysource-chart

# Enable pre-deployment diff telemetry
DEPLOY_TELEMETRY_DIFF=true \
  ./files/deploy-helm-with-telemetry.sh auto auto -f values-production.yaml
```

This wrapper sends an additional event with Helm diff output before deployment.

<Note>
  **Recommended**: Always use `--atomic --wait` flags with `helm upgrade` to ensure:

  * Deployments roll back automatically on failure
  * The post-rollback hook captures failure telemetry
  * Kubernetes waits for pods to be ready before marking success
</Note>

## Troubleshooting

### Check Status

```bash theme={null}
kubectl get pods -n anysource
kubectl get ingress -n anysource
kubectl get hpa -n anysource
```

### View Logs

```bash theme={null}
kubectl logs -f deployment/anysource-backend -n anysource
kubectl logs -f deployment/anysource-frontend -n anysource
```

### Check Events

```bash theme={null}
kubectl get events -n anysource --sort-by='.lastTimestamp'
```

## Support

For issues and questions, contact the Runlayer team at [team@runlayer.com](mailto:team@runlayer.com)

## Next Steps

<CardGroup cols={2}>
  <Card title="AWS EKS Provisioning" icon="aws" href="/deployment/eks-terraform">
    Use our Terraform EKS module to provision production-grade infrastructure:
    EKS clusters, RDS, ElastiCache, VPC, IAM roles, and monitoring. Then deploy
    applications with this Helm chart.
  </Card>
</CardGroup>