Sentry Relay Vault Integration

Overview

The Runlayer platform supports automatic fetching of Sentry Relay credentials from WorkOS Vault for both EKS (Helm) and ECS (Terraform) deployments. This eliminates the need to manually manage relay credentials and ensures they are securely retrieved at deployment time. Deployment Methods:

EKS/Helm: Uses an init container to fetch credentials on pod startup
ECS/Terraform: Uses Terraform’s external data source to fetch credentials during deployment

How It Works

EKS/Helm Deployments

When relay.vault.enabled is set to true, an init container runs before the Sentry Relay starts:

The init container executes the fetch script from a ConfigMap
The script fetches relay credentials from WorkOS Vault using the AUTH_API_KEY from backend secrets
Credentials are written as credentials.json and config.yml to a shared in-memory volume (emptyDir at /.relay)
The main relay container starts with --config /.relay flag to read the fetched credentials
Relay processes telemetry events with the fetched credentials

If credentials are not available in the vault, the init container gracefully exits after writing a minimal config file, allowing the relay to start successfully but remain non-functional until credentials are provisioned.

ECS/Terraform Deployments

Terraform uses an external data source to fetch credentials during the deployment:

Terraform calls the vault-fetch-relay.sh script as an external data source
The script fetches credentials from WorkOS Vault and returns them as JSON
Terraform creates ECS task definition environment variables with the credentials
ECS tasks start with the credentials already configured

If credentials are not available, Terraform receives empty values and deploys without Sentry Relay functionality.

Prerequisites

1. WorkOS Vault Setup (Both EKS and ECS)

Ensure your Sentry Relay credentials are stored in WorkOS Vault with the following structure: Secret Name: runlayer-sentry-credentials Secret Value (JSON):

{
  "public_key": "your-relay-public-key",
  "secret_key": "your-relay-secret-key",
  "id": "your-relay-id",
  "sentry_dsn": "https://your-key@your-org.ingest.sentry.io/project-id"
}

The sentry_dsn field is optional and can be overridden per environment via the backend.secrets.SENTRY_DSN value (EKS) or Terraform variables (ECS).

2. Provide WorkOS API Key

EKS/Helm
ECS/Terraform

Ensure your backend.secrets.AUTH_API_KEY is set in your Helm values:

backend:
  secrets:
    AUTH_API_KEY: "sk_live_your_workos_api_key"
    # ... other secrets ...

The relay init container will automatically use this API key to fetch credentials from WorkOS Vault.

Provide the WorkOS API key as a Terraform variable:

# In your terraform.tfvars or via environment variable
workos_api_key = "sk_live_your_workos_api_key"

# Or via environment variable
export TF_VAR_workos_api_key="sk_live_your_workos_api_key"

Keep your WorkOS API key secure. This key has access to your vault secrets.

Configuration

EKS/Helm Configuration

In your Helm values file, configure the vault integration:

relay:
  enabled: true
  
  # Vault integration for automatic credential fetching via init container
  # When enabled, an init container will fetch credentials from WorkOS Vault
  # before the relay starts, writing them to a shared memory volume
  # Uses AUTH_API_KEY from backend.secrets to authenticate with WorkOS Vault
  vault:
    # Enable automatic credential fetching via init container
    enabled: true
    
    # Name of the secret in WorkOS Vault (optional, defaults shown)
    vaultSecretName: "runlayer-sentry-credentials"
    
    # WorkOS API base URL (optional, defaults shown)
    apiBase: "https://api.workos.com"
    
    # Init container configuration (optional)
    initContainer:
      image: "alpine:3.20"
      resources:
        requests:
          cpu: 100m
          memory: 128Mi
        limits:
          cpu: 200m
          memory: 256Mi

# Ensure AUTH_API_KEY is set in backend.secrets (used by relay init container)
backend:
  secrets:
    AUTH_API_KEY: "sk_live_your_workos_api_key"
    # ... other secrets ...

Deploy with Vault Integration

helm install anysource ./infra/aws-helm/anysource-chart \
  -f production-values.yaml \
  -n your-namespace

The init container will automatically run before each relay pod starts and fetch credentials.

How the Init Container Works

When a relay pod starts with relay.vault.enabled: true:

ConfigMap Creation: Helm creates a ConfigMap named <release-name>-relay-fetch-script containing the fetch script
Init Container Starts: The init container mounts:
- The ConfigMap at /scripts/fetch-relay-credentials.sh (read-only)
- An empty memory volume at /.relay (read-write)
- The backend-secret to read AUTH_API_KEY
Credential Fetch: The init container executes the script which:
- Installs curl, jq, and ca-certificates in the Alpine container
- Fetches credentials from WorkOS Vault using the API key
- On success: Writes credentials.json and config.yml to /.relay
- On failure: Writes minimal config.yml only (prevents relay crashes)
Main Container Starts: After the init container completes:
- The relay container mounts /.relay as read-only
- Relay starts with --config /.relay flag
- Relay reads credentials and begins processing events

ECS/Terraform Configuration

In your Terraform configuration, the vault integration is automatic when you provide the WorkOS API key:

# main.tf or relay.tf
data "external" "relay_credentials" {
  program = ["${path.module}/scripts/vault-fetch-relay.sh"]
  
  query = {
    api_key = var.workos_api_key
  }
}

# Use the fetched credentials in your ECS task definition
resource "aws_ecs_task_definition" "relay" {
  # ... other configuration ...
  
  container_definitions = jsonencode([{
    name  = "sentry-relay"
    image = "ghcr.io/getsentry/relay:25.10.0"
    
    environment = [
      {
        name  = "RELAY_PUBLIC_KEY"
        value = data.external.relay_credentials.result.public_key
      },
      {
        name  = "RELAY_SECRET_KEY"
        value = data.external.relay_credentials.result.secret_key
      },
      {
        name  = "RELAY_ID"
        value = data.external.relay_credentials.result.relay_id
      },
      # ... other environment variables ...
    ]
  }])
}

Customization via Environment Variables: You can customize the vault fetch behavior by setting environment variables before running Terraform:

export WORKOS_API_BASE="https://api.workos.com"  # Default
export WORKOS_VAULT_SECRET_NAME="runlayer-sentry-credentials"  # Default

terraform apply

Graceful Degradation

EKS/Helm

If credentials are not available in WorkOS Vault:

The init container exits successfully (exit code 0)
A minimal config.yml is written to the shared volume (prevents relay crashes)
No credentials are written (relay starts but remains non-functional)
Relay pods start successfully but cannot process telemetry
The backend continues to work without Sentry telemetry

ECS/Terraform

If credentials are not available in WorkOS Vault:

The external data source returns empty values
Terraform creates the task definition with empty credential environment variables
ECS tasks start but relay remains non-functional
The backend continues to work without Sentry telemetry

This allows deployments to proceed even when Sentry credentials are not yet provisioned.

Security Considerations

Credential Storage

EKS/Helm
ECS/Terraform

When relay.vault.enabled is true:

The fetch script is stored in a ConfigMap and mounted into the init container
Credentials are fetched by an init container on each pod start
Credentials are written as credentials.json and config.yml to /.relay directory
The /.relay directory is an in-memory emptyDir volume (never written to disk, 1Mi size limit)
The volume is shared only between the init container and relay container (read-only for relay)
Credentials are automatically cleared when the pod terminates
No RBAC permissions required - init container only writes to local volume

Network Access

EKS/Helm
ECS/Terraform

The init container needs network access to reach the WorkOS Vault API at https://api.workos.com. Ensure your cluster’s network policies allow outbound HTTPS traffic.

The machine running Terraform (CI/CD runner, local workstation) needs network access to reach the WorkOS Vault API at https://api.workos.com. ECS tasks do not need direct access to WorkOS Vault.

Troubleshooting

EKS/Helm
ECS/Terraform

Check Init Container Status

# Check pod status
kubectl get pods -n your-namespace -l app.kubernetes.io/component=sentry-relay

# Check init container logs
kubectl logs -n your-namespace <relay-pod-name> -c fetch-credentials

Common Issues

Init Container Fails with “WORKOS_API_KEY not found”Ensure AUTH_API_KEY is set in your backend.secrets:

# Check if the backend-secret exists and contains AUTH_API_KEY
kubectl get secret backend-secret -n your-namespace -o jsonpath='{.data.AUTH_API_KEY}' | base64 -d

Relay Pod Starts but Relay is Non-FunctionalCheck the init container logs to see if credentials were fetched:

kubectl logs -n your-namespace <relay-pod-name> -c fetch-credentials

If you see warnings about missing credentials, they may not be available in WorkOS Vault.

Check Terraform External Data Source

# Run Terraform with debug logging
TF_LOG=DEBUG terraform plan

# Check if the script is executable
ls -la infra/aws-terraform-ecs/scripts/vault-fetch-relay.sh

# Test the script manually
echo '{"api_key":"sk_live_your_key"}' | \
  ./infra/aws-terraform-ecs/scripts/vault-fetch-relay.sh

Common Issues

Terraform Fails with “external program error”Ensure the script is executable:

chmod +x infra/aws-terraform-ecs/scripts/vault-fetch-relay.sh

Empty Credentials in Task DefinitionCheck the Terraform output to see what the external data source returned:

terraform console
> data.external.relay_credentials.result

If all values are empty strings, credentials may not be available in WorkOS Vault.Script Fails with “jq: command not found”Ensure jq is installed on the machine running Terraform:

# macOS
brew install jq

# Ubuntu/Debian
sudo apt-get install jq

# Amazon Linux
sudo yum install jq

Credentials Not Found in Vault (Both) Contact Runlayer support to provision relay credentials for your deployment.

Configuration Reference

EKS/Helm Vault Settings

Parameter	Description	Default
`relay.vault.enabled`	Enable automatic credential fetching via init container	`true`
`relay.vault.vaultSecretName`	Name in WorkOS Vault	`runlayer-sentry-credentials`
`relay.vault.apiBase`	WorkOS API base URL	`https://api.workos.com`
`relay.vault.initContainer.image`	Container image for the init container	`alpine:3.20`
`relay.vault.initContainer.resources`	Resource limits for the init container	See values.yaml
`backend.secrets.AUTH_API_KEY`	WorkOS API key (used by relay init container)	(required)

Note: When relay.vault.enabled is true:

The init container mounts a ConfigMap containing the fetch script at /scripts/fetch-relay-credentials.sh
Credentials are written to /.relay directory (in-memory emptyDir volume)
The relay container uses --config /.relay flag to load credentials on startup
The sentry-relay-secret Kubernetes secret is not created (vault takes precedence)

ECS/Terraform Environment Variables

The vault-fetch-relay.sh script supports the following environment variables:

Variable	Description	Default
`WORKOS_API_BASE`	WorkOS API base URL	`https://api.workos.com`
`WORKOS_VAULT_SECRET_NAME`	Secret name in WorkOS Vault	`runlayer-sentry-credentials`

Usage:

export WORKOS_API_BASE="https://api.workos.com"
export WORKOS_VAULT_SECRET_NAME="runlayer-sentry-credentials"
terraform apply

Script Locations

Deployment Type	Script Location	Purpose	Input/Output
ECS/Terraform	`infra/aws-terraform-ecs/scripts/vault-fetch-relay.sh`	Terraform external data source	stdin: JSON with api_key stdout: JSON with credentials
EKS/Helm (init container)	`infra/aws-helm/anysource-chart/files/fetch-relay-credentials.sh`	Init container script	env vars: WORKOS_API_KEY, etc. output: writes credentials.json and config.yml to /.relay
EKS/Helm (external)	`scripts/helm-fetch-vault-credentials.sh`	CLI wrapper for testing	env: WORKOS_API_KEY output: temporary values file

Script Differences:

ECS script: Reads API key from stdin (JSON), outputs credentials to stdout (JSON), uses curl with retry logic
EKS init container script: Reads API key from environment variables, writes credentials to files (credentials.json and config.yml), uses curl with retry logic
EKS external script: Calls the ECS script internally, converts JSON output to Helm values file format

The external Helm script (scripts/helm-fetch-vault-credentials.sh) uses the ECS script internally to avoid code duplication, then transforms the JSON output into a Helm values YAML file.

​Overview

​How It Works

​EKS/Helm Deployments

​ECS/Terraform Deployments

​Prerequisites

​1. WorkOS Vault Setup (Both EKS and ECS)

​2. Provide WorkOS API Key

​Configuration

​EKS/Helm Configuration

​Deploy with Vault Integration

​How the Init Container Works

​ECS/Terraform Configuration

​Graceful Degradation

​EKS/Helm

​ECS/Terraform

​Security Considerations

​Credential Storage

​Network Access

​Troubleshooting

​Check Init Container Status

​Common Issues

​Check Terraform External Data Source

​Common Issues

​Configuration Reference

​EKS/Helm Vault Settings

​ECS/Terraform Environment Variables

​Script Locations

Overview

How It Works

EKS/Helm Deployments

ECS/Terraform Deployments

Prerequisites

1. WorkOS Vault Setup (Both EKS and ECS)

2. Provide WorkOS API Key

Configuration

EKS/Helm Configuration

Deploy with Vault Integration

How the Init Container Works

ECS/Terraform Configuration

Graceful Degradation

EKS/Helm

ECS/Terraform

Security Considerations

Credential Storage

Network Access

Troubleshooting

Check Init Container Status

Common Issues

Check Terraform External Data Source

Common Issues

Configuration Reference

EKS/Helm Vault Settings

ECS/Terraform Environment Variables

Script Locations