​

Troubleshooting

​

Quick Diagnostics

​

System Health Check

kubectl get pods -n anysource
kubectl get svc -n anysource
kubectl get events -n anysource --sort-by=.metadata.creationTimestamp

aws ecs list-services --cluster <cluster>
aws ecs list-tasks --cluster <cluster> --service-name <service>
aws ecs describe-services --cluster <cluster> --services <service>

​

Database Connectivity

# Test PostgreSQL connection
psql "postgresql://<user>:<password>@<rds-endpoint>:5432/<db>"

# Test Redis connection
redis-cli -h <redis-endpoint> -p 6379 ping

​

Common Issues

​

Service Won’t Start

1. Kubernetes: Inspect pod status and events
   • kubectl describe pod <pod> -n anysource
   • kubectl logs <pod> -n anysource
2. ECS: Inspect stopped tasks and CloudWatch logs
   • aws ecs describe-tasks --cluster <cluster> --tasks <task-id>
   • Review the task’s CloudWatch log group
3. Verify configuration values and secrets (Kubernetes Secrets or AWS SSM/Secrets Manager)
4. Check security groups and network connectivity between services

​

Database Connection Issues

1. Verify the database endpoint and credentials
2. Check security groups / network policies for database access
3. Review database logs in AWS (RDS logs / CloudWatch)
4. Validate environment variables or secret values used by the backend

​

Performance Issues

1. Check resource utilization (CloudWatch metrics or Kubernetes metrics)
2. Review database performance and slow query logs
3. Verify Redis cache connectivity and hit rate
4. Inspect application logs for errors and timeouts

​

ACM Certificate Issues

1. Verify a wildcard certificate exists: aws acm list-certificates --query "CertificateSummaryList[?contains(DomainName, '*')]"
2. If no wildcard certificate exists, either create one manually or set enable_acm_dns_validation = true to have Terraform create it.

1. Confirm the Route53 hosted zone exists in the AWS account running Terraform.
2. Ensure hosted_zone_name matches the zone name (e.g., staging.runlayer.com).
3. Re-run terraform apply so the _acme-challenge CNAME records are created automatically.

​

Authentication Problems

1. Verify authentication configuration
2. Check external identity provider connectivity
3. Review user permissions and roles
4. Check JWT token configuration

​

Runlayer Assistant: Anthropic use case details not submitted

Not Found: {"detail":"Model use case details have not been submitted for this
account. Fill out the Anthropic use case details form before using the model.
If you have already filled out the form, try again in 15 minutes."}

1. Check whether the form was already submitted, and whether the model is available:
   aws bedrock get-use-case-for-model-access
   aws bedrock get-foundation-model-availability --model-id anthropic.claude-opus-4-6-v1
   
   Requires AWS CLI 2.27.42+ and AmazonBedrockFullAccess on the calling principal.
2. If it was not submitted, submit it — Bedrock console → Model catalog → any Anthropic model → Submit use case details, or aws bedrock put-use-case-for-model-access --form-data <base64-encoded JSON>. Access is granted once the form is accepted, but allow up to 15 minutes for it to propagate before resubmitting or escalating.
3. AWS accepts the submission once per account, or once at your AWS Organization’s management account, with member accounts inheriting it. Check the management account before submitting again. Opt-in Regions require their own submission.
4. Confirm the model matches your Region’s geography. The default us. inference profile cannot be invoked from a non-US BEDROCK_REGION — set RUNLAYER_ASSISTANT_BEDROCK_MODEL to the matching profile (for example au.anthropic.claude-opus-4-6-v1 in ap-southeast-2).
5. If your organization restricts Regions with SCPs, allow bedrock:InvokeModel* in every destination Region of the inference profile.

​

Log Analysis

​

Application Logs

kubectl logs -n anysource deploy/backend -f
kubectl logs -n anysource deploy/webapp -f

aws logs tail /aws/ecs/<service> --follow

​

Database Logs

​

Enterprise Support

​

Support Information

• System Information: OS, deployment method, AWS region
• Error Messages: Complete error messages and stack traces
• Log Files: Relevant application and system logs
• Configuration: Sanitized configuration files (remove secrets)
• Steps to Reproduce: Detailed steps that led to the issue

​

Escalation Process

1. Level 1: Basic troubleshooting (this guide)
2. Level 2: Advanced diagnostics (contact support)
3. Level 3: Engineering escalation (critical issues)

​

Preventive Measures

• Regular Monitoring: Set up health checks and alerting
• Log Rotation: Configure proper log management
• Resource Monitoring: Monitor CPU, memory, and disk usage
• Backup Verification: Regularly test backup and restore procedures