> ## Documentation Index
> Fetch the complete documentation index at: https://docs.runlayer.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Sentry Relay Vault Integration

> Automatically fetch Sentry Relay credentials from WorkOS Vault for EKS and ECS deployments

## Overview

The Runlayer platform supports automatic fetching of Sentry Relay credentials from WorkOS Vault for both EKS (Helm) and ECS (Terraform) deployments. This eliminates the need to manually manage relay credentials and ensures they are securely retrieved at deployment time.

**Deployment Methods:**

* **EKS/Helm**: Uses an init container to fetch credentials on pod startup
* **ECS/Terraform**: Uses Terraform's external data source to fetch credentials during deployment

## How It Works

### EKS/Helm Deployments

When `relay.vault.enabled` is set to `true`, an init container runs before the Sentry Relay starts:

1. The init container executes the fetch script from a ConfigMap
2. The script fetches relay credentials from WorkOS Vault using the `AUTH_API_KEY` from backend secrets
3. Credentials are written as `credentials.json` and `config.yml` to a shared in-memory volume (`emptyDir` at `/.relay`)
4. The main relay container starts with `--config /.relay` flag to read the fetched credentials
5. Relay processes telemetry events with the fetched credentials

If credentials are not available in the vault, the init container gracefully exits after writing a minimal config file, allowing the relay to start successfully but remain non-functional until credentials are provisioned.

### ECS/Terraform Deployments

Terraform uses an external data source to fetch credentials during the deployment:

1. Terraform calls the `vault-fetch-relay.sh` script as an external data source
2. The script fetches credentials from WorkOS Vault and returns them as JSON
3. Terraform creates ECS task definition environment variables with the credentials
4. ECS tasks start with the credentials already configured

If credentials are not available, Terraform receives empty values and deploys without Sentry Relay functionality.

## Prerequisites

### 1. WorkOS Vault Setup (Both EKS and ECS)

Ensure your Sentry Relay credentials are stored in WorkOS Vault with the following structure:

**Secret Name:** `runlayer-sentry-credentials`

**Secret Value (JSON):**

```json theme={null}
{
  "public_key": "your-relay-public-key",
  "secret_key": "your-relay-secret-key",
  "id": "your-relay-id",
  "sentry_dsn": "https://your-key@your-org.ingest.sentry.io/project-id"
}
```

<Note>
  The `sentry_dsn` field is optional and can be overridden per environment via the `backend.secrets.SENTRY_DSN` value (EKS) or Terraform variables (ECS).
</Note>

### 2. Provide WorkOS API Key

<Tabs>
  <Tab title="EKS/Helm">
    Ensure your `backend.secrets.AUTH_API_KEY` is set in your Helm values:

    ```yaml theme={null}
    backend:
      secrets:
        AUTH_API_KEY: "sk_live_your_workos_api_key"
        # ... other secrets ...
    ```

    The relay init container will automatically use this API key to fetch credentials from WorkOS Vault.
  </Tab>

  <Tab title="ECS/Terraform">
    Provide the WorkOS API key as a Terraform variable:

    ```hcl theme={null}
    # In your terraform.tfvars or via environment variable
    workos_api_key = "sk_live_your_workos_api_key"

    # Or via environment variable
    export TF_VAR_workos_api_key="sk_live_your_workos_api_key"
    ```
  </Tab>
</Tabs>

<Warning>
  Keep your WorkOS API key secure. This key has access to your vault secrets.
</Warning>

## Configuration

### EKS/Helm Configuration

In your Helm values file, configure the vault integration:

```yaml theme={null}
relay:
  enabled: true
  
  # Vault integration for automatic credential fetching via init container
  # When enabled, an init container will fetch credentials from WorkOS Vault
  # before the relay starts, writing them to a shared memory volume
  # Uses AUTH_API_KEY from backend.secrets to authenticate with WorkOS Vault
  vault:
    # Enable automatic credential fetching via init container
    enabled: true
    
    # Name of the secret in WorkOS Vault (optional, defaults shown)
    vaultSecretName: "runlayer-sentry-credentials"
    
    # WorkOS API base URL (optional, defaults shown)
    apiBase: "https://api.workos.com"
    
    # Init container configuration (optional)
    initContainer:
      image: "alpine:3.20"
      resources:
        requests:
          cpu: 100m
          memory: 128Mi
        limits:
          cpu: 200m
          memory: 256Mi

# Ensure AUTH_API_KEY is set in backend.secrets (used by relay init container)
backend:
  secrets:
    AUTH_API_KEY: "sk_live_your_workos_api_key"
    # ... other secrets ...
```

### Deploy with Vault Integration

```bash theme={null}
helm install anysource ./infra/aws-helm/anysource-chart \
  -f production-values.yaml \
  -n your-namespace
```

The init container will automatically run before each relay pod starts and fetch credentials.

### How the Init Container Works

When a relay pod starts with `relay.vault.enabled: true`:

1. **ConfigMap Creation**: Helm creates a ConfigMap named `<release-name>-relay-fetch-script` containing the fetch script
2. **Init Container Starts**: The init container mounts:
   * The ConfigMap at `/scripts/fetch-relay-credentials.sh` (read-only)
   * An empty memory volume at `/.relay` (read-write)
   * The `backend-secret` to read `AUTH_API_KEY`
3. **Credential Fetch**: The init container executes the script which:
   * Installs `curl`, `jq`, and `ca-certificates` in the Alpine container
   * Fetches credentials from WorkOS Vault using the API key
   * On success: Writes `credentials.json` and `config.yml` to `/.relay`
   * On failure: Writes minimal `config.yml` only (prevents relay crashes)
4. **Main Container Starts**: After the init container completes:
   * The relay container mounts `/.relay` as read-only
   * Relay starts with `--config /.relay` flag
   * Relay reads credentials and begins processing events

### ECS/Terraform Configuration

In your Terraform configuration, the vault integration is automatic when you provide the WorkOS API key:

```hcl theme={null}
# main.tf or relay.tf
data "external" "relay_credentials" {
  program = ["${path.module}/scripts/vault-fetch-relay.sh"]
  
  query = {
    api_key = var.workos_api_key
  }
}

# Use the fetched credentials in your ECS task definition
resource "aws_ecs_task_definition" "relay" {
  # ... other configuration ...
  
  container_definitions = jsonencode([{
    name  = "sentry-relay"
    image = "ghcr.io/getsentry/relay:25.10.0"
    
    environment = [
      {
        name  = "RELAY_PUBLIC_KEY"
        value = data.external.relay_credentials.result.public_key
      },
      {
        name  = "RELAY_SECRET_KEY"
        value = data.external.relay_credentials.result.secret_key
      },
      {
        name  = "RELAY_ID"
        value = data.external.relay_credentials.result.relay_id
      },
      # ... other environment variables ...
    ]
  }])
}
```

**Customization via Environment Variables:**

You can customize the vault fetch behavior by setting environment variables before running Terraform:

```bash theme={null}
export WORKOS_API_BASE="https://api.workos.com"  # Default
export WORKOS_VAULT_SECRET_NAME="runlayer-sentry-credentials"  # Default

terraform apply
```

## Graceful Degradation

### EKS/Helm

If credentials are not available in WorkOS Vault:

* The init container exits successfully (exit code 0)
* A minimal `config.yml` is written to the shared volume (prevents relay crashes)
* No credentials are written (relay starts but remains non-functional)
* Relay pods start successfully but cannot process telemetry
* The backend continues to work without Sentry telemetry

### ECS/Terraform

If credentials are not available in WorkOS Vault:

* The external data source returns empty values
* Terraform creates the task definition with empty credential environment variables
* ECS tasks start but relay remains non-functional
* The backend continues to work without Sentry telemetry

This allows deployments to proceed even when Sentry credentials are not yet provisioned.

## Security Considerations

### Credential Storage

<Tabs>
  <Tab title="EKS/Helm">
    When `relay.vault.enabled` is `true`:

    * The fetch script is stored in a ConfigMap and mounted into the init container
    * Credentials are fetched by an init container on each pod start
    * Credentials are written as `credentials.json` and `config.yml` to `/.relay` directory
    * The `/.relay` directory is an in-memory `emptyDir` volume (never written to disk, 1Mi size limit)
    * The volume is shared only between the init container and relay container (read-only for relay)
    * Credentials are automatically cleared when the pod terminates
    * **No RBAC permissions required** - init container only writes to local volume
  </Tab>

  <Tab title="ECS/Terraform">
    * Credentials are fetched during Terraform deployment
    * Credentials are stored in Terraform state (ensure state is encrypted)
    * Credentials are passed to ECS as environment variables in the task definition
    * ECS task definition is stored in AWS (encrypted at rest)
    * Consider using AWS Secrets Manager for additional security layer
  </Tab>
</Tabs>

### Network Access

<Tabs>
  <Tab title="EKS/Helm">
    The init container needs network access to reach the WorkOS Vault API at `https://api.workos.com`. Ensure your cluster's network policies allow outbound HTTPS traffic.
  </Tab>

  <Tab title="ECS/Terraform">
    The machine running Terraform (CI/CD runner, local workstation) needs network access to reach the WorkOS Vault API at `https://api.workos.com`. ECS tasks do not need direct access to WorkOS Vault.
  </Tab>
</Tabs>

## Troubleshooting

<Tabs>
  <Tab title="EKS/Helm">
    ### Check Init Container Status

    ```bash theme={null}
    # Check pod status
    kubectl get pods -n your-namespace -l app.kubernetes.io/component=sentry-relay

    # Check init container logs
    kubectl logs -n your-namespace <relay-pod-name> -c fetch-credentials
    ```

    ### Common Issues

    **Init Container Fails with "WORKOS\_API\_KEY not found"**

    Ensure `AUTH_API_KEY` is set in your `backend.secrets`:

    ```bash theme={null}
    # Check if the backend-secret exists and contains AUTH_API_KEY
    kubectl get secret backend-secret -n your-namespace -o jsonpath='{.data.AUTH_API_KEY}' | base64 -d
    ```

    **Relay Pod Starts but Relay is Non-Functional**

    Check the init container logs to see if credentials were fetched:

    ```bash theme={null}
    kubectl logs -n your-namespace <relay-pod-name> -c fetch-credentials
    ```

    If you see warnings about missing credentials, they may not be available in WorkOS Vault.
  </Tab>

  <Tab title="ECS/Terraform">
    ### Check Terraform External Data Source

    ```bash theme={null}
    # Run Terraform with debug logging
    TF_LOG=DEBUG terraform plan

    # Check if the script is executable
    ls -la infra/aws-terraform-ecs/scripts/vault-fetch-relay.sh

    # Test the script manually
    echo '{"api_key":"sk_live_your_key"}' | \
      ./infra/aws-terraform-ecs/scripts/vault-fetch-relay.sh
    ```

    ### Common Issues

    **Terraform Fails with "external program error"**

    Ensure the script is executable:

    ```bash theme={null}
    chmod +x infra/aws-terraform-ecs/scripts/vault-fetch-relay.sh
    ```

    **Empty Credentials in Task Definition**

    Check the Terraform output to see what the external data source returned:

    ```bash theme={null}
    terraform console
    > data.external.relay_credentials.result
    ```

    If all values are empty strings, credentials may not be available in WorkOS Vault.

    **Script Fails with "jq: command not found"**

    Ensure `jq` is installed on the machine running Terraform:

    ```bash theme={null}
    # macOS
    brew install jq

    # Ubuntu/Debian
    sudo apt-get install jq

    # Amazon Linux
    sudo yum install jq
    ```
  </Tab>
</Tabs>

**Credentials Not Found in Vault (Both)**

Contact Runlayer support to provision relay credentials for your deployment.

## Configuration Reference

### EKS/Helm Vault Settings

| Parameter                             | Description                                             | Default                       |
| ------------------------------------- | ------------------------------------------------------- | ----------------------------- |
| `relay.vault.enabled`                 | Enable automatic credential fetching via init container | `true`                        |
| `relay.vault.vaultSecretName`         | Name in WorkOS Vault                                    | `runlayer-sentry-credentials` |
| `relay.vault.apiBase`                 | WorkOS API base URL                                     | `https://api.workos.com`      |
| `relay.vault.initContainer.image`     | Container image for the init container                  | `alpine:3.20`                 |
| `relay.vault.initContainer.resources` | Resource limits for the init container                  | See values.yaml               |
| `backend.secrets.AUTH_API_KEY`        | WorkOS API key (used by relay init container)           | *(required)*                  |

**Note:** When `relay.vault.enabled` is `true`:

* The init container mounts a ConfigMap containing the fetch script at `/scripts/fetch-relay-credentials.sh`
* Credentials are written to `/.relay` directory (in-memory `emptyDir` volume)
* The relay container uses `--config /.relay` flag to load credentials on startup
* The `sentry-relay-secret` Kubernetes secret is not created (vault takes precedence)

### ECS/Terraform Environment Variables

The `vault-fetch-relay.sh` script supports the following environment variables:

| Variable                   | Description                 | Default                       |
| -------------------------- | --------------------------- | ----------------------------- |
| `WORKOS_API_BASE`          | WorkOS API base URL         | `https://api.workos.com`      |
| `WORKOS_VAULT_SECRET_NAME` | Secret name in WorkOS Vault | `runlayer-sentry-credentials` |

**Usage:**

```bash theme={null}
export WORKOS_API_BASE="https://api.workos.com"
export WORKOS_VAULT_SECRET_NAME="runlayer-sentry-credentials"
terraform apply
```

### Script Locations

| Deployment Type           | Script Location                                                   | Purpose                        | Input/Output                                                                                    |
| ------------------------- | ----------------------------------------------------------------- | ------------------------------ | ----------------------------------------------------------------------------------------------- |
| ECS/Terraform             | `infra/aws-terraform-ecs/scripts/vault-fetch-relay.sh`            | Terraform external data source | stdin: JSON with api\_key<br />stdout: JSON with credentials                                    |
| EKS/Helm (init container) | `infra/aws-helm/anysource-chart/files/fetch-relay-credentials.sh` | Init container script          | env vars: WORKOS\_API\_KEY, etc.<br />output: writes credentials.json and config.yml to /.relay |
| EKS/Helm (external)       | `scripts/helm-fetch-vault-credentials.sh`                         | CLI wrapper for testing        | env: WORKOS\_API\_KEY<br />output: temporary values file                                        |

**Script Differences:**

* **ECS script**: Reads API key from stdin (JSON), outputs credentials to stdout (JSON), uses curl with retry logic
* **EKS init container script**: Reads API key from environment variables, writes credentials to files (`credentials.json` and `config.yml`), uses curl with retry logic
* **EKS external script**: Calls the ECS script internally, converts JSON output to Helm values file format

<Note>
  The external Helm script (`scripts/helm-fetch-vault-credentials.sh`) uses the ECS script internally to avoid code duplication, then transforms the JSON output into a Helm values YAML file.
</Note>