module "runlayer_infrastructure". Examples enable features and do not represent the defaults.
Bedrock and Anthropic
On this ECS deployment path, Runlayer Assistant uses Bedrock in the deployment account by default (modelglobal.anthropic.claude-opus-4-8). Having “some Anthropic model” enabled in the console is not enough — that
specific model (or its base) must be available. To run it on one of your own providers instead (for example an
OpenAI-compatible gateway), an admin picks it under Runlayer Assistant model in Settings → AI Providers.
Module v28.1.x+ attaches the model-access IAM actions to the ECS task role
(GetFoundationModelAvailability, PutUseCaseForModelAccess, Marketplace
subscribe, etc.). You do not add those by hand unless an SCP / boundary strips
them.
does not have access to the Anthropic model yet,
see troubleshooting.
Agent runtime
Runlayer Assistant, Security Investigations, and Procedures run on the agent runtime (Bedrock AgentCore). Without it, Assistant chat fails withNo agent runtime is provisioned for this deployment.
enable_runlayer_agents only adds infrastructure
custom agents use (the agent event-trigger consumer) and requires the runtime.
AgentCore calls your application URL. The default PUBLIC network mode
reaches it over the internet; with WAF IP allowlisting, use
VPC mode and PrivateLink.
Before applying, check the runtime’s
egress and
AWS permissions; the
apply host also needs the AgentCore prerequisites in the module README.
Sandbox options are under Agent Sandbox.
Distribution configuration
The deployment guide configures Distribution with all three inputs below. Obtain a registered reader key from Runlayer before the first apply:flagd without both reader inputs. The
application default remains WorkOS when the provider input is omitted; setting
the key and URL alone does not switch providers. Existing WorkOS deployments
can stage the reader credentials before explicitly switching to flagd.
If Runlayer has registered your account for self-managed Distribution
registration, also pass the IDs Runlayer support provides and the module manages
your registry manifest as a Terraform resource:
distribution_api_key_previous to the
outgoing key, applying, waiting for ECS services to reach steady state, then
clearing it and applying again. Set distribution_status = "disabled" to revoke
access without destroying the stack; move openfeature_provider off flagd first.
Optional feature flags
Enable only the features you intend to operate. ToolGuard needs GPU quota and bootstrap egress. With WAF IP allowlisting, configure AgentCore VPC mode and PrivateLink before enabling the agent runtime.Which optional services need EC2 or a GPU
The application services run on ECS Fargate. Three optional ML services run on EC2 instead, and all three are off by default:
Rev always runs on
g6f.large. ToolGuard defaults to g6f.large, and runlayer_tool_guard_instance_type selects another single-GPU g6f, g6 or g6e size. Check GPU quota and availability of the chosen types in your private subnets’ AZs before you turn either on.
Runlayer Deploy is always enabled by the ECS module.
sts:AssumeRole only on arn:aws:iam::*:role/runlayer-bedrock-* and
arn:aws:iam::*:role/RunlayerBedrock*. The saved role must trust the caller’s AWS
account/identity with the saved external ID and grant the required model access.
When both BEDROCK_ENABLED and BEDROCK_ANTHROPIC_MODELS_ENABLED are enabled,
the consumer also receives nonstreaming bedrock:InvokeModel access to Claude
foundation models and the deployment account’s Claude inference profiles for
the legacy platform fallback.
Upgrade and apply the Terraform module alongside the backend/consumer image;
updating the image alone does not install this permission. After IAM propagation,
verify an authorized monitor canary and its dead-letter alarm.
ToolGuard capacity and scaling
ToolGuard scale-out speed. A new ToolGuard GPU host installs NVIDIA GRID drivers at boot and takes 10–15 minutes before it serves traffic, so a scale-out decision is only as fast as the spare hosts already running. That boot path also needs outbound HTTPS to Ubuntu, NVIDIA GRID S3, Docker, and related hosts — see Self-Hosted Egress Requirements → ToolGuard.runlayer_tool_guard_scaling_profile trades idle GPU cost for
time-to-capacity:
CPU targets are a percent of a
g6f.large task’s CPU reservation; on a larger
runlayer_tool_guard_instance_type the same CPU is a smaller percent
(balanced 35% becomes 15% on g6.xlarge).
balanced and aggressive keep one to two extra g6f.large instances
running at all times (billed even when idle) so the next task starts
immediately. Scale-in waits 15 minutes on every profile. The dashboard’s
Tasks: Desired vs Running panel shows how long scale-outs wait on new
hosts. The runlayer-tool-guard-scale-out-lag alarm fires only when desired
tasks exceed running tasks for 20 consecutive minutes — longer than a cold
GPU host boot — so a routine scale-out does not trip it; when it does fire,
new hosts are not arriving (check the Auto Scaling group and capacity
provider).
GPU signal on aggressive. CPU is a weak load signal on a GPU inference
host, so aggressive adds a second target-tracking policy on fleet-average
GPU utilization (target 60%). The policy tracks the existing
RunlayerToolGuard/GPUUtilization metric (Environment dimension,
Average statistic): every host whose ToolGuard answers /health 200
publishes one sample per minute, so the per-minute average is the mean across
serving GPUs. Hosts that are not serving (capacity-provider spares, hosts
still installing drivers or loading the model, hosts in the recycle drain)
publish nothing and are left out, so the spare pool cannot hold the average
under the target while serving GPUs are busy. No per-host series is added, so
CloudWatch metric cardinality is unchanged.
Scaling takes the higher of the CPU and GPU decisions, so the GPU policy
never forces the fleet below what the CPU or predictive policies request.
Hosts publish GPU metrics from their boot script every 60 seconds, so changes to the ToolGuard launch
template trigger an ASG rolling instance refresh that replaces the
hosts: tasks restart on new hosts as old hosts drain (each restart reloads the
model, so expect a capacity dip per batch). Until the refresh replaces them,
hosts from the previous launch template still publish every 5 minutes and so
land in only one of every five 1-minute averages.
GPU alarms. On every profile, with enable_runlayer_tool_guard_alarms
(default on), two alarms watch the same GPU utilization series, which CPU-based
scaling and the high-cpu alarm do not see. With today’s models a ToolGuard
host runs out of CPU well before its GPU, so GPU is rarely the limit.
runlayer-tool-guard-high-gpufires when fleet-average GPU utilization stays above 70% for 15 minutes (three 5-minute averages): the whole fleet is GPU-bound, usually after a model or batch change. It does not see one busy host among idle ones (one of two hosts at 100% averages about 50%).runlayer-tool-guard-gpu-peggedfires when the busiest host reads above 90% in 10 of 15 minutes: a GPU held by a stuck kernel or process, or a model or batch change that made serving GPU-bound. The metric has no host dimension, so runnvidia-smion each host. Replace a single pegged host; on a fleet at its minimum, raise the minimum by one first and let the new host serve, since a host takes up to 15 minutes to boot. If several hosts are high, roll back a recent model or batch change, or treat it likehigh-gpu.
runlayer-tool-guard-predict-p95 fires when the p95
of every successful predict request, fleet-wide and server-side, is above a
threshold in 12 of the last 15 minutes (minutes with fewer than 10 requests
don’t count): the fleet is past its latency knee. Short bursts of concurrent
requests slow only the minute they land in, so they don’t fire it. The threshold
defaults by runlayer_tool_guard_instance_type: 400 ms on g6f.large (and on
types not yet measured), 150 ms on g6.xlarge; during an instance type cutover,
the higher of the two. Set runlayer_tool_guard_predict_p95_alarm_threshold_ms
if your busy-hour p95 sits near the default. When it fires, check whether the
service is at its task ceiling (raise runlayer_tool_guard_max_count) or a
scale-out is stuck (scale-out-lag); after a model or batch change at normal
load, roll that change back.
high-gpu, gpu-pegged and predict-p95 are non-paging alerts. On ALARM they
notify the ToolGuard alerts topic and, with enable_monitoring, the topics in
critical_alerts_extra_sns_topic_arns. On OK they notify only those extra
topics, like the ToolGuard alarms that page.
GPU query timeout alarm. A host whose GPU or driver is wedged stops
publishing GPU utilization, so high-gpu, gpu-pegged and GPU scaling cannot
see it. Its nvidia-smi read gives up after 10 seconds and publishes
RunlayerToolGuard/GPUQueryTimeouts instead, and
runlayer-tool-guard-gpu-query-timeout fires when that happens in 3 of 5
minutes, counted across the fleet. It pages like gpu-errors. The same metric
with an InstanceId dimension names the host; terminate it. The alarm
deliberately pages instead of replacing the host automatically: a driver fault
on every host would otherwise replace the whole fleet into the same fault.
Predictive scaling. Because a reactive scale-out cannot beat the 10–15
minute host boot, the module also creates an ECS predictive scaling policy
on the ToolGuard service. It forecasts the next 48 hours of demand hourly
from up to 14 days of ToolGuard predict request history and raises the
task count runlayer_tool_guard_predictive_scheduling_buffer_seconds
(default 1200) before each forecast hour, so GPU hosts are booting before
the load arrives. conservative runs it in forecast_only mode (forecasts
are recorded but never acted on); balanced and aggressive run
forecast_and_scale. Override the profile default with
runlayer_tool_guard_predictive_scaling_mode (disabled, forecast_only,
forecast_and_scale). The first forecast appears after 24 hours of metrics;
review it in the ECS console (service → Service auto scaling) or with
aws application-autoscaling get-predictive-scaling-forecast, then flip
forecast_only → forecast_and_scale once the curve matches your traffic.
Predictive scaling only scales out; the CPU policy still handles scale-in and
unforecast bursts. Every policy stops below runlayer_tool_guard_max_count,
which reserves ceil(25%) of it, at least one host (8 allows 6 tasks, 4 allows
3, 2 allows 1), because each task needs a whole GPU host and a
rolling deploy starts each replacement task before it stops the old one.
runlayer_tool_guard_max_count stays the GPU host limit. In regions where ECS predictive scaling is unavailable
(for example il-central-1) the policy is skipped automatically; the
runlayer_tool_guard_predictive_scaling_mode_effective output reports the
mode actually applied.
Why the load is floored. The forecast learns daily and weekly patterns
from the load metric, so a few days with scanning switched off would teach
it that those weekdays are idle and under-provision them for the next two
weeks. Before forecasting, weekday hours (Mon–Fri UTC) below
runlayer_tool_guard_predictive_load_floor_fraction (default 0.5) of the
window’s mean hourly request count are raised to that floor, so off days
look like quiet business days instead of zero; weekends forecast from actual
traffic (runlayer_tool_guard_predictive_load_floor_weekdays_only = false
floors every day, 0 disables the floor). Capacity is sized so each task
receives runlayer_tool_guard_predictive_target_requests_per_task_per_minute.
Left null (default), it is the per-task rate measured for
runlayer_tool_guard_instance_type: 50 per minute on g6f.large (3000 per
hour), 250 on g6.xlarge, and 50 on other, unmeasured types (which
over-provisions a faster host). On g6f.large, 50 keeps ToolGuard predict
p95 near its floor; latency rises steeply once a task serves more than roughly
twice that rate. 250 on g6.xlarge keeps the same headroom: a benchmark
showed a g6.xlarge task serving at least five times what a g6f.large task
does. While runlayer_tool_guard_cutover is set, the lower of the two types’
rates applies. Raise the target to trade latency for fewer hosts, lower it for
more headroom.
Running tasks are read from Container Insights (RunningTaskCount) when it
is enabled, otherwise from the GPU Auto Scaling group’s in-service instance
count (one task per host). The Auto Scaling group count also includes the
spare hosts balanced and aggressive keep warm (roughly 20% and 30% of
the fleet), so without Container Insights requests per task read low by that
share and the forecast under-provisions. Pair forecast_and_scale with
container_insights = "enabled"; terraform plan warns when it is not.
Agent Sandbox (Optional)
Procedures also need the agent sandbox: their code is validated and test-run inside it. Without the agent runtime (enable_runlayer_agentcore_runtime = true),
creating or validating a Procedure returns 503. Select a network configuration that allows AgentCore to reach the application before applying.
sandboxMode: k8s, gVisor or Kata on the nodes) is operator/GKE — see Runlayer Operator — K8s Agent Sandbox.
Runlayer Assistant runs on Bedrock in this account by default, so it works without configuring a provider. Admins can move it to one of the workspace’s providers under Runlayer Assistant model in Settings → AI Providers. Model access is handled as described above; this module exposes it as bedrock_auto_model_access, bedrock_model_access_company_name, bedrock_model_access_company_website, bedrock_model_access_industry and bedrock_model_access_intended_users.
Backend egress for WorkOS AuthKit, Bedrock, and AgentCore is listed in Self-Hosted Egress Requirements.
LLM gateway
Microsoft Teams
Optional. Off unless all five values are set — the module variable is the client secret, the rest rideadditional_backend_env_vars.
Provisioning creates a per-agent Entra app registration and Azure Bot Service
resource in your Entra tenant and Azure subscription, so a self-managed
install needs its own multi-tenant app registration (Graph application
permissions Application.ReadWrite.All, AppCatalog.ReadWrite.All,
TeamsAppInstallation.ReadWriteForUser.All, admin-consented), an Azure
resource group whose IAM grants that app Azure Bot Service Contributor, and
the Microsoft.BotService provider registered on the subscription.
MS_TEAMS_APP_URL defaults to APP_URL (https://<domain_name>). Register
https://<domain_name>/api/v1/integrations/ms-teams/admin-consent-callback as a
Redirect URI (Authentication → Web) on that app registration, or admin consent
fails with AADSTS500113.
Each tenant that installs an agent’s bot must also enable Upload custom apps
in its Teams Admin Center setup policy; without it, publishing to the tenant’s
app catalog returns 403 no matter how many times it is retried.
domain_name is effectively immutable once agents are provisioned: each agent’s
Azure Bot resource stores the messaging endpoint captured at creation time and
is never rewritten, so changing the domain orphans existing bots until each is
deleted and recreated.
AWS Cloud Discovery
On by default. Admins connect AWS accounts from AI Inventory → Sources; each account creates a read-onlyRunlayerDiscovery-… role from a
CloudFormation stack that trusts only this deployment’s ECS task role, with an
external ID per connection. The module adds a private template bucket in your
deployment’s Region, a grant on the task role to assume those roles,
and the backend settings that point at both. Customers create the stack in the
same Region, so service control policies there must allow CloudFormation.
AWS_CLOUD_DISCOVERY_* keys in the operator guide;
GKE has no AWS identity, so AWS Cloud Discovery isn’t available there.
Customer stacks bake in the task role’s ARN, so keep project and
environment unchanged once accounts are connected. With restricted egress,
also allow sts.us-east-1.amazonaws.com, account.us-east-1.amazonaws.com,
and bedrock.<region>.amazonaws.com plus
bedrock-agentcore.<region>.amazonaws.com for every Region your customers’
accounts enable.
iMessage channel
Optional, off by default. Lets people text an agent from Messages; the user side is in iMessage. The stack doesn’t run any Apple hardware: Sendblue rents a phone number (“line”) that sends and receives iMessage (with SMS fallback) through its API. The stack has one line, Runlayer’s, configured in Terraform. One agent answers it at a time, so one workspace on the stack uses iMessage at a time: the first workspace whose admin turns it on holds the number until it turns it off. On a stack several workspaces share, agree on which one before you enable the channel.enable_imessage_channel sets IMESSAGE_CHANNEL_ENABLED, which shows the
Text message card on agent profiles in Go. The apply stores the key
pair and webhook secret in one Secrets Manager secret and hands them, with
SENDBLUE_FROM_NUMBER, to the backend service and the shared worker only. The
plan fails unless all four sendblue_* values are set with the flag.
Before you apply
- A Sendblue account with a dedicated line. Sign up at
sendblue.com (or
npx @sendblue/cli setup). The free plan shares one number across customers and only reaches contacts you verify first; for an agent anyone in the organization can text, use a dedicated line (Sendblue’s AI Agent plan is inbound-first: people text the line, the agent replies). Copy the line’s number and the API key pair (dashboard, orsendblue show-keys). - Who can reach it. Anyone can text the line. Runlayer ignores texts from people who haven’t linked their number to a Runlayer account, apart from one link reply every 10 minutes.
- Data path. Texts and the agent’s replies pass through Sendblue (SOC 2 Type II). Review that with your security team the same way as any other messaging integration.
Point Sendblue at the stack
In the line’s Sendblue account (Developer → Webhooks), add a receive webhook with theimessage_channel_sendblue_webhook_url output,
https://<domain_name>/api/v1/integrations/imessage/sendblue/receive, and
sendblue_webhook_secret as its secret. Sendblue sends the secret in the
sb-signing-secret header; the backend rejects any call without the right one.
Sendblue webhooks are per account. If the account has other lines, texts to
them reach the webhook too and are ignored: only texts to
sendblue_from_number start runs.
With WAF IP allowlisting on, the webhook must still reach
https://<domain_name>: allow Sendblue’s webhook egress IPs (ask Sendblue
support) in waf_allowlist_ipv4_cidrs, or texts get 403s and nothing happens.
The backend also needs outbound HTTPS to api.sendblue.co.
Give the line a name and photo
Optional. Without a contact profile, Messages shows the line as a bare number. Set one once with Sendblue’s contact sharing API and the line’s key pair. The photo must be a public JPEG or PNG URL:imessage_sendblue_name_share_failed; texts and replies are
unaffected.
Operating it
- Status. The agent page shows the number and Runlayer’s line, or No line. It does not call Sendblue, so a wrong key shows up as failed replies, not on that badge.
- Replies. Each reply is queued in the database, then sent right away.
One person’s replies go out one at a time and in order. If Sendblue is
down, rate-limits, or rejects the keys, the worker’s
imessage_outbox_sweep(every minute) retries it, up to 5 attempts over about 9 minutes, and the replies after it wait. A message Sendblue declines (for example, the person opted out) is not retried, and the rest of that answer is dropped. A reply still unsent after a day (the line’s config was missing) is dropped too. Logs:imessage_sendblue_send_failed. - Read receipts and typing. A text the agent takes is marked read, and
the agent shows typing while its run works. Both are best effort: failures
are logged (
imessage_sendblue_read_receipt_failed,imessage_sendblue_typing_failed) and change nothing else. - What “sent” means. A reply counts as sent once Sendblue accepts it. If Sendblue’s response is lost, the reply still counts as sent (Sendblue may have queued it) and is never resent, because Sendblue takes no idempotency key. Delivery to the phone is Sendblue’s from there. Failures after that (for example, an unreachable phone) show in Sendblue’s dashboard, matched by the message handle stored on the reply when one came back. If a run’s answer never reached the queue (the server failed right after the run finished), the sweep queues it within a couple of minutes.
- Rotating keys. Rotate in Sendblue, update the variables and apply; a new secret version rolls the backend and the worker onto it. Texts Sendblue sends with an old webhook secret get 401s, and Sendblue retries each one up to 3 times, so update both sides together.
- Offboarding an organization: turn iMessage off on the agent that
answers, with the API call in
Turn it on for your workspace. To retire
the channel, set
enable_imessage_channel = false, apply, and delete the webhook in Sendblue.
IMESSAGE_CHANNEL_ENABLED, SENDBLUE_FROM_NUMBER and the secrets
SENDBLUE_API_KEY_ID, SENDBLUE_API_SECRET_KEY and SENDBLUE_WEBHOOK_SECRET
on both the backend and the worker.
Audit SIEM, sessions, and hook events
Audit logs and session events use separate S3 exports. Enable either or both for ingestion into your SIEM:- Audit logs include tool activity, policy decisions, security events, and
administrative changes. Set
enable_audit_log_siem_export = true. Withaudit_log_write_target = "aurora", the worker exports through an outbox; audit-log Kinesis is optional. S3 audit archiving is a separate retention feature and does not enable SIEM export. See Audit-log export. - Session events include captured prompts, reasoning, tool calls, and
scanner verdicts. The hook-events pipeline is on by default (leave
enable_hook_events_pipelineunset); uncommentsession_siem_export_kinesis_consumer_configbelow to enable the dedicated export consumer. Enabling the hook-events pipeline alone does not enable session export. See Session export.
*_external_s3_bucket
settings and replace the example names. Both exports can use the same bucket
with separate *_s3_prefix values. Without an external destination, Terraform
creates a bucket for each enabled export in your deployment’s AWS account.
Direct writes use your deployment’s ECS task role. To write through another
IAM role, set *_assume_role_arn together with the required *_external_id
and *_external_s3_bucket for that export.
Configure destination bucket and KMS policies, plus your SIEM’s S3 reader
permissions, notifications, and schemas, separately.
Optionally set hook_events_alert_email to receive pipeline-health emails
through Amazon SNS. Alerts fire when the hook-events consumer Lambda’s error
rate exceeds 20% for two consecutive five-minute periods, or its processing lag
exceeds five minutes in three consecutive five-minute periods. Recovery notifications are also sent. Confirm the SNS
subscription email to receive notifications; leaving the setting unset disables
these email alerts. These alarms monitor the hook-events consumer Lambda, which
is provisioned only when ML data export (data_export_bucket_arn), the Sparkle
worker, or enable_hook_events_parquet_archive is set — with only session SIEM
export on, no Lambda and none of these email alarms are created. The session
SIEM exporter is a separate ECS consumer.
Session export starts with new events and does not backfill earlier sessions.
It contains unredacted content; session privacy settings do not apply to the
export, so restrict access to the destination bucket.
audit_log_write_target = "kinesis", keep its
existing audit-ingestion configuration. To export directly from that stream,
also enable the independent SIEM export consumer below, alongside
enable_audit_log_siem_export = true. This exports normal streamed events
without routing them through the database outbox. Set both audit prefix
variables to the same value so stream exports and outbox fallback exports use
the same destination prefix:
audit_log_id downstream.
Binary packages and downloads
AI Watch snapshot signing
Only for AI Watch Local Proxy. Generate the deployment key and get a delegation from Runlayer as described in AI Watch snapshot delegation, then pass both:backend-service, so no manual redeploy is needed.
- Renew before the delegation’s
not_after(at most 90 days): replace the file with the one Runlayer sends and apply. - Rotate the key by changing both inputs in the same apply: a delegation names one key, so a new key needs new delegations.