Default Runlayer app version:
6.4.0. See the Runlayer changelog.Improvements- Tenant key vault with one switch: a self-hosted deployment now turns
the tenant key vault on with
tenant_key_vault_enabled = true, once Runlayer confirms the deployment is onboarded. The module finds the vault secrets by the name of its task execution role, so no ARN is exchanged with Runlayer. The task execution role keeps its grants on the provisioned secrets, so ECS can still roll a failed deployment back to the task definition from before the vault. If you passdistribution_api_keytoday, leave it (andmcp_catalog_api_key) set on the apply that turns the vault on, for the same rollback, and remove them in a later apply. tenant_key_vault_kms_key_arnis deprecated: it is ignored and will be removed in a later release. Remove it from your module call.tenant_key_vault_secret_arnis deprecated: the tenant key vault now keeps each key in a secret of its own, named after your task execution role, so there is no single secret left to name. The input is ignored and will be removed in a later release. Remove it from your module call.
Default Runlayer app version:
6.3.0. See the Runlayer changelog.Improvements- Session search follows the indexer: activity and security-finding
filters on the Sessions page, and the agent session search tools, are now
available only when
enable_session_search_indexer = trueandsession_search_index_mode = "write". Other deployments keep text search and session id lookup. Nothing to change; to force a value, setSESSIONS_SEARCH_INDEX_READS_ENABLEDinadditional_backend_env_vars. - Tunnel diagnostics now report how long authenticated CONNECT streams stay open and why they close. The Runlayer backend also reports connection timings and failures by phase, from trust lookup through CONNECT admission.
- Tunnel image security updates include fixes in Go and OpenTelemetry.
- Aurora replica lag alarm: with
enable_monitoring = true, each Aurora instance gets an alarm on the critical-alerts topic when its replica lag stays aboverds_replica_lag_threshold_ms(default1000) for 5 minutes. Replica reads are stale while it fires. Nothing to change.
- Tunnel connector readiness keeps healthy targets available while slow, successful TCP readiness checks wait for their scheduled turn after startup.
Default Runlayer app version:
6.2.0. See the Runlayer changelog.No upgrade notes for this release.Default Runlayer app version:
6.1.0. See the Runlayer changelog.Breaking changes- Who is affected: only roots that never set
enable_hook_events_pipeline. In each case below, the simplest fix is to setenable_hook_events_pipeline = falsebefore you bump the module. - Inline Runlayer Deploy policy: with
runlayer_deploy_use_managed_policy = false,terraform planfails. Either turn the pipeline off, or remove that setting. - Digest-pinned backend image with the Parquet Lambda: if
ecr_repositories.backenduses a digest (@sha256:...) and you setdata_export_bucket_arn,enable_sparkle_worker = trueorenable_hook_events_parquet_archive = true,terraform planfails until you also sethook_events_consumer_image_uri. Either set it, or turn the pipeline off.
- Sessions are also stored in Postgres and S3 by default: if you never
set
enable_hook_events_pipeline, the next apply creates the hook-events pipeline (Kinesis stream, S3 buckets, session materializer and compactor ECS services, and their alarms), which stores new sessions in Postgres and S3 as well as Redis. Sessions are still read from Redis. A later release reads them from Postgres and S3, so keep the pipeline on; setenable_hook_events_pipeline = falseonly if you must skip it. - Session retention: Redis keeps sessions for 7 days, but the Postgres
rows and the session-payloads S3 bucket have no expiry, so captured prompts,
reasoning and tool calls are kept indefinitely. Raw hook events in the
Parquet bucket expire after a year. Records the session materializer or the
cold-analyzers consumer fail to process are kept for 3 years in their
dead-letter buckets. Check this with your compliance team before you
apply; setting
enable_hook_events_pipeline = falsebefore the bump keeps the 7-day behavior. - Other services follow the pipeline: the cold-analyzers consumer (an
always-on ECS service) also starts; set
enable_cold_analyzers_consumer = falseto skip it. If you setdata_export_bucket_arn, data-export sync and the hook-events Parquet consumer Lambda also start; the Lambda also starts withenable_sparkle_worker = trueorenable_hook_events_parquet_archive = true. - Cost: at default sizes the pipeline and the cold-analyzers consumer add
about $90 a month in
us-east-1, plus usage that grows with session traffic.
- AWS Cloud Discovery on by default: the apply adds a private template
bucket in your deployment’s Region, a bucket policy, and an
sts:AssumeRolegrant onRunlayerDiscovery-*roles to the ECS task role’s managed policy, so admins can connect AWS accounts from AI Inventory. Setenable_aws_cloud_discovery = falseto opt out. Run an app version from this release or later outsideus-east-1. If your egress is restricted, allowsts.us-east-1.amazonaws.com,account.us-east-1.amazonaws.com, andbedrockandbedrock-agentcorein each Region your customers’ accounts use. - AI Watch policy snapshot signing (opt-in): set
ai_watch_snapshot_signing_keyandai_watch_snapshot_delegationswith the values Runlayer provides. Off while both are empty. - Valkey 9.1 for the managed cache: new deployments should set
redis_engine = "valkey"(added in v49.0.0). On an existing deployment, setting it upgrades the cache in place and keeps endpoints; AWS keeps the data only on a best-effort basis. AWS warns that a single-node group, the default, may be unavailable for the whole upgrade. Settingredisagain makes Terraform recreate the groups empty. - Per-key tenant secrets no longer need the JSON secret: with
provisioned_tenant_secret_layout = "per_key"the module stops looking uprestricted/<project>-<env>-provisioned, so the plan no longer needs that secret to exist. The task execution role keeps a grant on it by name, so a rollback to an earlier task definition still starts while the secret exists. Nothing changes on the default"json"layout or withtenant_key_vault_secret_arn. - Alarm for a load balancer target group with no healthy targets: with
enable_monitoringon, the apply adds a<project>-<environment>-alb-<service>-no-healthy-hostsalarm for each public ALB target group. It fires when a service has had no healthy target for 10 minutes and alerts through the critical-alerts topic whencritical_alerts_enabledis on. New alarms wait up to an hour before they first evaluate.
Default Runlayer app version:
6.0.0. See the Runlayer changelog.Breaking changes- The WorkOS, Sentry and Resend keys come only from the provisioned tenant
secret: before bumping, make sure the Secrets Manager secret
restricted/<project>-<env_short>-provisionedexists in the deployment account (env_shortisprodforenvironment = "production", otherwise the environment name) with a JSON version carrying all six keys, an empty string for any you do not use:AUTH_API_KEY(your currentauth_api_keyvalue),RESEND_API_KEY,SENTRY_DSN,SENTRY_RELAY_PUBLIC_KEY,SENTRY_RELAY_SECRET_KEYandSENTRY_RELAY_ID(the Sentry values Runlayer gave you). Runlayer creates and fills it for managed deployments; a plan fails until it exists. auth_api_keyis removed: delete it from the module call and from the CI secret store that fed it, then rotate the WorkOS key with Runlayer, since copies remain in earlier state versions.tenant_secret_sourceaccepts only"provisioned":"input"fails the plan. The variable is now a no-op default; remove it from the module call before the next major, which drops it.- Relay without credentials: Terraform no longer checks whether Relay
credentials exist. Set
sentry_relay_enabled = falseon a deployment whose provisioned secret has noSENTRY_RELAY_*values, or the Relay task fails to start.sentry_dsnonly gates frontend Sentry; set it to the DSN the secret carries when you want frontend Sentry.
- Stateless roots apply after their primary: a stateless root
(
region_role = "stateless") now reads the keys from the primary’s secret throughcentral_state.provisioned_tenant_secret_arnand fails the plan while that is null. Apply the primary on this version first, then refreshcentral_stateand apply each stateless root. - Upgrade from input mode in a quiet window: the first apply rewrites the
app secret without the WorkOS and Sentry keys while earlier task revisions
still reference them, so an old task that restarts during the rollout fails
to start. Apply when you can wait for the new deployment to become healthy.
To back out, return to module version 48 with
tenant_secret_source = "provisioned"set and apply again: it reads the same provisioned secret and needs noauth_api_key. Earlier versions have no provisioned mode and needauth_api_keyagain. - Browser Sentry DSN is no longer read from WorkOS Vault: if browser
Sentry was turned on through the optional
frontend_sentry_dsnfield of the Vault item, set the module’sfrontend_sentry_dsninput to that public DSN before bumping, together withsentry_dsn. Without it, browser error and trace reporting stops. - Hook-events lag email removed: with
hook_events_alert_emailset,<project>-<environment>-hook-events-consumer-iterator-age(over 5 minutes behind) is deleted. Lag now alerts only through<project>-<environment>-hook-events-parquet-lambda-iterator-age, which needsenable_monitoringandcritical_alerts_enabledand fires over 15 minutes behind. To keep a lag alert, set both totrue.
- Tunnel collector autoscaling: collectors now scale on native ECS average CPU,
targeting 60% by default, even with central observability off. The default ceiling
is the larger of four and
tunnel_otel_collector_desired_count, which now sets the runtime floor; settunnel_otel_collector_max_capacityequal to that floor for fixed capacity, or tune the target withtunnel_otel_collector_cpu_target. - Model shed-rate alarms stop flipping on bursty load: every ToolGuard,
topic-cpu and guard-cpu
<project>-<service>-<model>-shed-rate-<env>alarm now pages when 503 load sheds passrunlayer_model_error_rate_threshold_percentin 3 of the last 4 ten-minute windows, and holds its state while the rate hovers near that line. Windows need at least twicerunlayer_model_error_rate_min_requestsrequests to count. - Legacy topic-cpu alarms removed: while
keep_legacy_runlayer_topic_cpuistrue(the default) orenable_runlayer_guard_cpu_splitisfalse, the apply deletes therunlayer-topic-cpu-no-tasks,-high-memory,-unhealthyand-fast-unhealthyalarms. With the split on, the guard-cpu stack alarms cover the same failures. With the split off, the legacy stack keeps only its per-model alarms (error rate, shed rate, model load failures).
- ToolGuard alarms no longer fire for other deployments: with several
deployments in one AWS account and region, each deployment’s ToolGuard
rotation-aborted,cuda-oom,gpu-errorsandworker-recyclesalarms fired on any of them. Each deployment now reads its own counts.
Default Runlayer app version:
5.0.0. See the Runlayer changelog.Improvements- Missing Runlayer Plugin alarm: with
enable_monitoringon, the apply adds<project>-<env>-runlayer-plugin-row-missing, which fires when the Runlayer Plugin is missing from the database. It notifiescritical_alerts_extra_sns_topic_arnsonly and never pages; its description says how to restore the plugin. No new variables. - ToolGuard predict-p95 alarm ignores short bursts: it now fires when predict
p95 is above the threshold in 12 of the last 15 minutes, instead of in three
5-minute windows in a row, so brief bursts of concurrent scans no longer fire
it. A value you set in
runlayer_tool_guard_predict_p95_alarm_threshold_msstill applies; you don’t need to change it. - ToolGuard replaces a stuck worker: with ToolGuard enabled and more
than one task running, a task that logs
ROTATION_ABORTEDand has been running for at least 30 minutes is stopped so ECS starts a fresh one. A replacement counts only once it is healthy in the load balancer. An abort that has to wait is remembered and retried, one task at a time. A younger task, or the last healthy task, is left alone. The rotation-aborted alarm still pages. Kubernetes is not covered. - Quieter token metering alarms: with
token_metering_task_config.enabled,token-metering-lagfires only when every run in an hour is over 2 hours and now goes tocritical_alerts_extra_sns_topic_arnsinstead of the critical alerts topic;run-durationfires on 3 slow runs in 8 ticks,tail-read-mismatchon mismatches in 2 of 6 hours, andquarantined-sessionson more than 2 in 6 hours. - ToolGuard replaces a task that stopped answering: with ToolGuard enabled, a task at least 30 minutes old whose container check fails and that has logged nothing for 10 minutes has its host drained and marked Unhealthy in Auto Scaling, then is stopped, so the replacement starts on a new machine. At most one host per 30 minutes, none while an earlier one is unfinished, and none while every task is quiet. Hosts in Standby or Detaching are left alone. Kubernetes is not covered.
- Two new ToolGuard alarms: with
enable_runlayer_tool_guard_alarmson, the apply adds<project>-runlayer-tool-guard-silent-task-replaced-<env>(fires once per host replaced this way) and<project>-runlayer-tool-guard-restart-errors-<env>(the replacement function failed five minutes in a row, or a host replacement has been stuck for an hour). Both follow the existing actionable ToolGuard alarm routing. No new variables.
- Worker autoscaling no longer stops tasks mid-job: the worker now scales
out on CPU/memory but never scales in, so it holds its peak size up to
worker_config.max_capacity. To pin or shrink it, setworker_config.desired_count,min_capacityandmax_capacityto the same value. - Removing an EC2-backed guard or ToolGuard stack no longer hangs: turning
off a stack that runs on its own EC2 hosts (topic CPU, guard CPU, ToolGuard,
rev) could fail the apply after 20 minutes with its ECS service stuck in
DRAINING. The hosts now keep their IAM permissions until they are gone, starting with the first apply after this upgrade; a stack turned off in that same apply is removed the old way. - Dropping the legacy topic CPU stack no longer hangs: with
keep_legacy_runlayer_topic_cpu = false, the apply no longer times out waiting for the legacy ECS service to drain. If you already dropped that stack, your next plan creates its unused instance role, instance profile and two policy attachments again. That’s expected and needs no action.
Default Runlayer app version:
4.2.0. See the Runlayer changelog.Improvements- Portable tunnel metrics: Tunnel applications publish OpenTelemetry utilization. The shared collector formats the existing CloudWatch metrics independently of optional central forwarding, using its existing image and HTTPS access. Collector defaults remain one 0.25-vCPU / 512-MiB task.
- Tunnel endpoint scaling: Compatible endpoints default to 256 concurrent streams. Ranged bounds enable automatic scaling at 80% average stream usage with 300-second cooldowns. Busy endpoints retain task protection until their streams finish naturally; idle endpoints can scale down. Upgrades keep their 90-second drain after healthy replacements arrive. Equal bounds retain fixed capacity. See the rollout and canary guidance in ECS configuration.
- Autoscaled audit-log consumers also scale on CPU: when
audit_log_kinesis_consumer_config.max_capacityis abovemin_capacityandcpuis1024or below, the consumer now tracks its busiest task’s CPU at 75%, which also owns scale-in; lag still adds tasks, and<project>-<environment>-audit-log-kinesis-consumer-lag-scale-inis removed. Keepmax_capacityat or below the stream’s open shard count. A fixed fleet (the default), or one withcpuabove1024, is unchanged. - Audit-log consumer CPU alarm follows autoscaling: when
audit_log_kinesis_consumer_config.max_capacityis abovemin_capacityandcpuis1024or below, the consumer’s paging 80% CPU alarm is replaced by<project>-<environment>-audit-log-kinesis-consumer-cpu-saturated: the busiest task above 90% for 15 minutes, sent to the non-paging Slack relay only (<project>-<environment>-audit-log-kinesis-iterator-age-highstill pages). Otherwise the consumer keeps the 80% alarm. - Two new token metering alarms: with
token_metering_task_config.enabled,enable_monitoringandcritical_alerts_enabledon, the apply adds<project>-<env>-token-metering-tail-read-mismatchand<project>-<env>-token-metering-tail-fallback-ratio. Both notifycritical_alerts_extra_sns_topic_arnsonly and never page. No new variables. - GPU driver compiler: ECS ToolGuard and Rev hosts now explicitly build the NVIDIA GRID driver with GCC 12. Applying this update rolls enabled ToolGuard and Rev GPU hosts through their existing instance refresh; benchmark hosts use the new bootstrap on their next launch.
- Token metering no longer runs out of memory on large sessions: with
token_metering_task_config.enabled = true, the meter and nightly coverage tasks now default to 5 GiB (token_metering_task_config.memory, was 4 GiB), which the default 1 GiBmax_archive_read_bytesneeds. If you setmemoryyourself, keep it, minusotel_collector_config.memory(default 512 MiB) when the OTel collector sidecar is enabled, at least 2.6xmax_archive_read_bytes+ 1.9 GiB. - GPU hosts join ECS on newer Ubuntu kernels: ToolGuard and Rev hosts now install NVIDIA GRID driver 580.178.04, which builds on the latest Ubuntu 22.04 AWS kernel. Applying rolls enabled ToolGuard and Rev GPU hosts through their existing instance refresh.
Default Runlayer app version:
4.0.0. See the Runlayer changelog.Action required- ToolGuard and Rev host refresh: with ToolGuard enabled, the apply changes the
ToolGuard launch template, so the ASG replaces every ToolGuard GPU host in a
rolling refresh, and each batch reloads the model. With Rev enabled
(
enable_runlayer_rev), Rev hosts boot from the same user-data script, so the apply also replaces every Rev GPU host the same way. Apply in a quiet window.
- Token metering coverage check runs as its own scheduled task: with
token_metering_task_config.enabled, the apply adds a nightly Fargate task (01:45 UTC, same size as the metering task) and two non-paging alarms:token-metering-coverage-failedandtoken-metering-coverage-heartbeat(no run reported in 26 hours). The heartbeat shows ALARM from about an hour after the upgrade until the first nightly run reports; that is expected. The check no longer runs on the worker service. - Audit SIEM export pauses for seconds, not minutes, during upgrades: with
audit_log_siem_export_kinesis_consumer_config.enabled, the consumer starts its replacement task before stopping the old one, and the new task claims the shards within one 30-second discovery interval. Delivery stays at-least-once: a deploy can export a few records twice. - Pinned old backend images: if you pin
app_versionto a backend older than1.28.775(before shard leasing), every deploy exports the records written during its overlap twice for as long as the pin stays. Nothing is lost; unpin, or move to a newer backend, to stop the duplicates. - Audit-log stream shard count can be pinned: set
audit_log_kinesis_shard_count = Nto make the audit-log stream PROVISIONED with N shards, giving lease-sharing consumers (audit-log, audit SIEM export) more shards than on-demand mode creates at your load; a plan then fails if either consumer’smax_capacityexceeds N. Unset (null), the stream stays ON_DEMAND. Requires Runlayer backend1.28.2313or later (the first release with the ENG-6765 fix, #19256), or SIEM export skips the first records of new child shards. Size N for peak writes too, and read the module README (“Audit-log stream shard count”) before changing it. - ToolGuard GPU alarms send recovery to the relay topics only:
<project>-runlayer-tool-guard-high-gpu-<environment>and<project>-runlayer-tool-guard-gpu-pegged-<environment>still notify the ToolGuard alerts topic on ALARM, but on OK they now notify only the topics incritical_alerts_extra_sns_topic_arns(withenable_monitoring), like the ToolGuard alarms that page. Subscribers of the ToolGuard alerts topic no longer get their OK messages. - Paging alarms for the ToolGuard prompt guard model: with
enable_runlayer_tool_guard = true,runlayer_tool_guard_modelsnow includesrunlayer-user-prompt-guardby default, so the apply adds its<project>-tool-guard-runlayer-user-prompt-guard-{error-rate,shed-rate,error-burst,load-failure}-<environment>alarms and their log metric filters. Like the other ToolGuard model alarms, they notify the ToolGuard alerts topic and the critical alerts topic.enable_runlayer_tool_guard_alarms = falseturns them off. - Predict p95 dashboard panel follows
runlayer_tool_guard_models: the ToolGuard “Predict p95, request-weighted” panel now charts only the models in that list. If you set it yourself, addrunlayer-user-prompt-guardto keep that model on the panel and to get its alarms. - ToolGuard predict-latency alarm: with ToolGuard on, the apply adds a log
metric filter on the ToolGuard log group and its metric
tool-guard.PredictDurationUs, even withenable_runlayer_tool_guard_alarmsoff. With alarms on it also adds<project>-runlayer-tool-guard-predict-p95-<environment>, which fires when the p95 of successful predict requests stays above a threshold for 15 minutes: 400 ms on g6f.large, 150 ms on g6.xlarge, and 400 ms (the slowest measured type’s) on other instance types. It does not page: on ALARM it notifies the ToolGuard alerts topic and, withenable_monitoring, the topics incritical_alerts_extra_sns_topic_arns; on OK, only those extra topics. If your normal busy-hour p95 sits near the default, setrunlayer_tool_guard_predict_p95_alarm_threshold_ms. - New paging alarm for a wedged ToolGuard GPU: with ToolGuard alarms on
(
enable_runlayer_tool_guard_alarms, defaulttrue), the apply adds<project>-runlayer-tool-guard-gpu-query-timeout-<environment>. It fires when a host’snvidia-smigets no answer within 10 seconds in 3 of 5 minutes, which means its GPU or driver is wedged and its GPU metrics have stopped. It notifies the same topics asrunlayer-tool-guard-gpu-errors. Rev hosts get no such alarm; they publish their GPU metrics to theRunlayerRevnamespace. - WorkOS, Sentry and Resend keys can live only in your Secrets Manager:
tenant_secret_source = "provisioned"makes tasks readAUTH_API_KEY,SENTRY_DSN,SENTRY_RELAY_*and the newRESEND_API_KEYfrom a secretrestricted/<project>-<env>-provisionedthat Runlayer creates and fills in your account, instead of from theauth_api_keyinput and the WorkOS Vault fetch;auth_api_keyis then left unset. Terraform only reads the secret. Contact Runlayer to schedule the switch; until then nothing changes, and"input"stays the default. New outputsprovisioned_tenant_secret_nameandprovisioned_tenant_secret_arnname the secret (null in input mode).
Default Runlayer app version:
3.2.0. See the Runlayer changelog.Improvements- ToolGuard predictive scaling sizes
g6.xlargefor its speed: ong6.xlargewithrunlayer_tool_guard_predictive_target_requests_per_task_per_minuteunset, the apply retunes the predictive policy in place from50to250requests per task per minute (duringrunlayer_tool_guard_cutover, the lower of the two types’). Withbalancedoraggressive, predictive scaling then launches about a fifth as many tasks; withconservative(default), only the recorded forecast changes. A value you set still wins: set it to50to keep today’s sizing. - Hook-events Parquet Lambda runs only when something reads its output:
with
enable_hook_events_pipeline = true, the apply now deletes the<project>-<environment>-hook-events-consumerLambda, its event source mapping, IAM role and log group, unlessdata_export_bucket_arnis set orenable_sparkle_worker = true. If you sethook_events_alert_email, it also deletes the<project>-<environment>-hook-events-alertsSNS topic, its email subscription, and the Lambda’s<project>-<environment>-hook-events-consumer-failure-rateand<project>-<environment>-hook-events-consumer-iterator-ageemail alarms. This frees each hook-events shard’s read budget for the session consumers. The stream and the hook-events bucket stay, with their objects. To keep the Lambda and its email alarms, setenable_hook_events_parquet_archive = true. - New paging alarm on Parquet Lambda lag: where the Lambda runs,
<project>-<environment>-hook-events-parquet-lambda-iterator-agefires when the Lambda is more than 15 minutes behind the stream for 15 minutes. - Audit SIEM export can autoscale on CPU: set
audit_log_siem_export_kinesis_consumer_config.max_capacityabovemin_capacity; see the module README “Audit SIEM export consumer scaling”. Unset, the fleet stays fixed atdesired_count. - Scale events can duplicate a few exports: each one can export a few
audit records twice; dedupe on
audit_log_id. - Audit SIEM export CPU alarm follows autoscaling: when
audit_log_siem_export_kinesis_consumer_config.max_capacityis abovemin_capacity, the consumer’s 80% CPU alarm is replaced by<project>-<environment>-audit-siem-export-kinesis-consumer-cpu-saturated: the busiest task above 90% for 15 minutes, sent to the non-paging Slack relay only (the SIEM export lag alarm still pages). A fixed fleet keeps the 80% alarm.
Default Runlayer app version:
3.1.0. See the Runlayer changelog.Action required- ToolGuard host refresh: Before upgrading with ToolGuard enabled, allow for a rolling GPU-host refresh and model reloads. Ubuntu hosts now use local Amazon Time Sync instead of public NTP pools.
- New ToolGuard GPU alarms: with ToolGuard alarms on
(
enable_runlayer_tool_guard_alarms, defaulttrue), the apply adds two alarms.<project>-runlayer-tool-guard-high-gpu-<environment>fires when fleet-average GPU utilization stays above 70% for 15 minutes (the whole fleet is GPU-bound).<project>-runlayer-tool-guard-gpu-pegged-<environment>fires when the busiest host reads above 90% in 10 of 15 minutes (one stuck host, or several held near full). Both notify the ToolGuard alerts topic and, withenable_monitoring, the topics incritical_alerts_extra_sns_topic_arns, on ALARM and OK. The other ToolGuard alarms send only ALARM to the ToolGuard alerts topic, so its subscribers will now also get OK messages from these two. Neither pages the critical-alerts topic.
Default Runlayer app version:
3.0.0. See the Runlayer changelog.Breaking changes- ALB 5XX alarm threshold is now a percentage:
alb_5xx_alarm_thresholdis removed. If you set it, remove it before upgrading. To tune the alarm, setalb_5xx_alarm_threshold_percent(default2).
- Lag alarms for the other audit and session stream consumers: with
enable_monitoringandcritical_alerts_enabled, the agent event trigger and audit SIEM export each get a critical alarm when they fall behind the audit stream, and the session materializer gets a non-paging one whensession_materializer_enhanced_fan_outis on. The agent event trigger and session materializer alarms are removed while that consumer is parked at zero tasks. - ALB 5XX alarm fires on error rate, not single errors: the
<project>-alb-5xx-errors-<environment>alarm now fires when ALB-generated 5XX exceedalb_5xx_alarm_threshold_percentof requests in 2 of 3 periods. Periods with fewer than 50 requests never breach. - New non-paging ALB 5XX drip alarm:
<project>-alb-5xx-drip-<environment>fires on more than 20 ALB-generated 5XX in 6 hours. It routes only tocritical_alerts_extra_sns_topic_arns, never the paging topic, so a slow error drip under the rate alarm stays visible. - Parked deployments send no alarm notifications: with
infra_parked = true, alarms stay in place but notify nobody. Delivery resumes when you unpark.
- Audit log archive heartbeat waits for the first run: a new deployment no
longer pages
<project>-<environment>-audit-log-archive-heartbeatbefore its first scheduled archive run. It still pages when runs stop or never complete. - Fewer false ToolGuard GPU alarms:
runlayer-tool-guard-cuda-oomandrunlayer-tool-guard-gpu-errorsno longer fire on the brief GPU memory pressure of a ToolGuard worker restart, which clears on its own. - Idle-transaction alarms no longer fire on brief backend pauses: the
default
db_idle_in_txn_max_duration_thresholdis now30seconds (was3). Both idle-transaction alarms use it. A few seconds of idle time from a busy backend or a deploy no longer pages. If you set this input, keep your value or remove it to use the new default. - Fewer false model load-failure alarms:
<service>-<model>-load-failureno longer fires when a ToolGuard worker restart briefly runs out of GPU memory. That case clears on its own while the old worker keeps serving.
Default Runlayer app version:
2.4.0. See the Runlayer changelog.Improvements- ToolGuard instance type changes keep the old fleet’s capacity: in the
apply that changes
runlayer_tool_guard_instance_type, setrunlayer_tool_guard_cutover = { from_instance_type = "<old type>", from_max_count = <old runlayer_tool_guard_max_count> }. Running tasks are not scaled in before their replacements serve. Remove it whenterraform planwarns that no old-type hosts remain. While set, plan needsautoscaling:DescribeAutoScalingGroups,ecs:DescribeServicesandec2:DescribeInstances. On a faster type, also setrunlayer_tool_guard_predictive_target_requests_per_task_per_minuteto its measured per-task rate; the default50is measured ong6f.large. - ToolGuard GPU Auto Scaling group gets a
RunlayerToolGuardTaskCeilingtag: withenable_runlayer_tool_guard, the first apply adds it in place (no host replacement). A cutover plan reads it so it never lowers the task ceiling in force.
- ToolGuard CPU target and high-CPU alarm scale with the host’s CPU:
with a
runlayer_tool_guard_instance_typeother thang6f.large, the scaling profile’s CPU target and the high-CPU alarm threshold are now a share of that host’s CPU, not of ag6f.large’s (ong6.xlarge, balanced goes from 35% to 15% and the alarm from 55% to 23.6%). The old values sat above what a ToolGuard task uses on those hosts. The new values aren’t yet benchmarked there.g6f.largeis unchanged. - Worker autoscaling now scales: with
worker_config.max_capacityabovemin_capacity, the worker now scales on CPU and memory between those bounds. Before this fix it stayed atmin_capacity. The apply replaces the worker’s autoscaling target and its two policies. - No backend alarm on every deploy: the backend and frontend unhealthy-targets alarms now fire only when targets are unhealthy in 5 of the last 10 minutes. They no longer fire for the minute a new task warms up during a deploy or scale-out.
Default Runlayer app version:
2.3.0. See the Runlayer changelog.Action required- ToolGuard keeps 25% of its GPU hosts free for rollouts: with
enable_runlayer_tool_guard, autoscaling now stops atrunlayer_tool_guard_max_countminus ceil(25%), at least one host, so a rolling deploy no longer stalls at the ceiling (6 allows 4 tasks, 8 allows 6). A service above the new ceiling scales in to it on apply. To keep your current task ceiling, raiserunlayer_tool_guard_max_countbefore upgrading (for 6 tasks, set 8). Ifrunlayer_tool_guard_desired_countequalsrunlayer_tool_guard_max_count, the warm floor also drops to the new ceiling unless you raise the max.
Default Runlayer app version:
2.2.0. See the Runlayer changelog.No upgrade notes for this release.Default Runlayer app version:
2.1.0. See the Runlayer changelog.Improvements- WAF no longer blocks attack text in prompt scans: the
bad_inputs_count_uri_regexdefault adds/api/v1/hooks/prompt/pre, so AWS KnownBadInputs body matches there count instead of block and the prompt reaches Prompt Guard. If you set the variable, compare it with the new default. - Audit consumer lag alarm: With
enable_monitoringandcritical_alerts_enabled, the existing audit iterator-age alarm now monitors the audit consumer’s enhanced fan-out lag. No usable telemetry becomesINSUFFICIENT_DATAwithout notifications; the threshold and notification routes stay the same. - Disabled audit consumers: The alarm is removed when
audit_log_kinesis_consumer_config.enabled = falseoraudit_log_kinesis_consumer_config.desired_count = 0, even if the audit stream remains enabled.
Default Runlayer app version:
1.42.0. See the Runlayer changelog.No upgrade notes for this release.Default Runlayer app version:
1.41.0. See the Runlayer changelog.Breaking changes- Terraform or OpenTofu 1.12.0 or newer: before bumping, upgrade the CLI
on the host that runs
terraform apply;terraform initnow rejects older versions. Terraform 1.11 already failedplanon recent releases.
- VPC peering routes stay in place across applies: with
vpc_peering_connectionsset, applies no longer delete and re-add the peer routes. The first plan after upgrading reportsaws_route.peering_privateas no longer managed by Terraform; nothing is deleted.
Default Runlayer app version:
1.40.0. See the Runlayer changelog.No upgrade notes for this release.Default Runlayer app version:
1.39.0. See the Runlayer changelog.Breaking changes- Slow-pool memory floors rise for the session-topic model: the next ToolGuard CPU image adds
runlayer-session-topic, and every slow-pool worker loads every model, soterraform plannow requiresrunlayer_topic_cpu_memory >= runlayer_topic_cpu_gunicorn_workers x 3584 MiB(x 4096 MiB withrunlayer_skill_guard_enabled, the default) while topic-cpu is enabled; the previous floors were 2048 / 2560. The defaultrunlayer_topic_cpu_memorymoves from 16384 to 20480. A pin at the old 16384 default still plans with the default 4 workers (4 x 4096 = 16384, sitting exactly on the floor). Configurations that fail after the bump: a pinned memory near the old floor, for example 4 workers x 8192, or 5 or 6 workers x 16384 with SkillGuard on (5 x 4096 = 20480, 6 x 4096 = 24576); 6 workers no longer fit the 20480 default either and need an explicit pin, and with the legacyrunlayer-topic-cpustack kept on the m7i.2xlarge that pin (24576) only fits beside arunlayer_topic_cpu_fast_memoryof about 6656 or less (24576 + 6656 = 31232, under the ~31.4k MiB an m7i.2xlarge registers to ECS; the plan-time check only enforces 32768 - 1024 = 31744, a budget measured on an xlarge, so sizing up to it can plan and then fail placement). Fix by raisingrunlayer_topic_cpu_memory, loweringrunlayer_topic_cpu_gunicorn_workers, or settingrunlayer_skill_guard_enabled = false. On hosts smaller than the m7i.2xlarge default you already pinrunlayer_topic_cpu_memory(the default never fit a 16 GiB host); that pin must now also clear the new per-worker floor, and an m7i.xlarge with the legacy stack kept (keep_legacy_runlayer_topic_cpu, the default) holds at most three slow workers (3 x 4096 = 12288 plus a 3072 MiB fast pool = 15360 = 16384 - 1024); four slow workers fit an xlarge only with SkillGuard off and the legacy stack dropped (4 x 3584 = 14336). A stack withenable_runlayer_topic_cpu = falsekeeps the old floor and is unaffected. The floors come from a measurement: one slow-pool worker withrunlayer-session-topicloaded measures 3,011 MiB resident on the candidate image (2026-10-22, full roster, +375 MiB over the same roster without the model), so 3584 MiB is that figure plus 573 MiB of margin, and 4096 MiB adds the ~0.5 GiB SkillGuard scan peak on top with the same margin. Follow-up: Kubernetes deployments are not covered by this module; the operator still enforces 2048 MiB per worker and moves to the same floors before the session-topic image ships there. enable_resolver_endpointremoved: delete it from your configuration;terraform planfails while it is set. If you set it totrue, the next apply deletes the Route53 Resolver inbound endpoint and its security group. If you still need private-zone resolution from outside the VPC (VPN or on-prem forwarders), create your own INBOUNDaws_route53_resolver_endpointin the VPC outside the module and re-point your forwarders to it before applying. Theresolver_endpoint_ipsoutput is also removed.
Default Runlayer app version:
1.38.0. See the Runlayer changelog.No upgrade notes for this release.Default Runlayer app version:
1.37.1. See the Runlayer changelog.Bug fixes- Upgrading with the CPU guard stacks no longer fails
terraform plan: withenable_runlayer_topic_cpu, bumping from 40.x to 41.0.0 failed withError: Cyclenaming the guard task definitions. Bump straight to this release instead; no other change is needed.
Default Runlayer app version:
1.37.0. See the Runlayer changelog.Breaking changes- The identity the CPU guard cutover wait runs as also needs
ecs:DescribeServices: withenable_runlayer_topic_cpuand the default waits,scripts/guard_cpu_wait.shnow also polls the stack’s ECS service, asbackend_migration_aws_rolewhen that is set, otherwise as the apply host’s own credentials. Addecs:DescribeServicesnext toelasticloadbalancing:DescribeTargetHealthon that identity before bumping; without it the apply fails before the backend has moved, and the error names the missing action. Setting bothrunlayer_guard_cpu_wait_for_steady_stateandwait_for_ecs_steady_statetofalseskips the wait instead.
- Schedule a tunnel interruption: with tunnels and
runlayer_obs_otel_enabled = true, confirm existing-VPC private subnets can reachotlp.obs.runlayer.comon HTTPS before upgrading. The apply restarts tunnel routers and drops live tunnel sessions while clients reconnect. - Schedule later observability-secret seeding too: on a tunnel deployment,
seeding
runlayer/otlp-ingestenables collector wiring on the next apply, which restarts tunnel routers and drops live sessions while clients reconnect. - Keep the old telemetry proxy through the collector soak: apply, wait for the collector and tunnel connections to stabilize, then verify metrics and traces before removing the old proxy.
- Disable export before rollback: set
runlayer_obs_otel_enabled = false, apply, and wait for tunnel connections to reconcile before restoring the old module and application. This temporarily stops tunnel metrics and traces; application logs continue whenrunlayer_obs_otel_logs_enabled = true.
- Tunnel telemetry uses a private local relay: routers and endpoints send
metrics and traces to a deployment-local collector. Set
tunnel_otel_collector_desired_count = 2for two replicas; logs remain in CloudWatch. - A failed first create of a CPU guard service recovers on the next
apply: a
guard-cpu-slow-path,guard-cpu-hot-pathorrunlayer-topic-cpuservice left tainted by a failed steady-state wait is now replaced destroy-then-create, so a rerun completes; previously every rerun failed withCreation of service was not idempotentuntil the state was untainted by hand.
Default Runlayer app version:
1.35.0. See the Runlayer changelog.No upgrade notes for this release.Default Runlayer app version:
1.34.0. See the Runlayer changelog.Breaking changes- The identity the cutover wait runs as needs
elasticloadbalancing:DescribeTargetHealth: withenable_runlayer_topic_cpuand the default waits, every apply that points the backend at a CPU guard stack (the two new stacks on the cutover, the old one with the split off) runsscripts/guard_cpu_wait.shon the apply host, which pollsdescribe-target-healthwith the AWS CLI asbackend_migration_aws_rolewhen that is set, otherwise as the apply host’s own credentials; the aws provider’sassume_roledoes not reach it. The module README’s role list forbackend_migration_aws_rolepreviously named onlyecs:RunTask,ecs:DescribeTasks,ecs:StopTaskandiam:PassRole; a role scoped to that getsAccessDeniedand the apply fails right after the new stacks exist, with the backend still on the old stack (nothing goes down, the script exits at once and names the missing action and the opt-out). Addelasticloadbalancing:DescribeTargetHealthto whichever identity runs the wait (the role whenbackend_migration_aws_roleis set, otherwise the apply host’s own credentials) and make sure the apply host has the AWS CLI (the migration runner already needs it) before bumping. To defer the whole change instead, setenable_runlayer_guard_cpu_split = falseandkeep_runlayer_guard_cpu_split_stacks = false(the first alone still creates both new stacks and runs the wait on the old one), and bothrunlayer_guard_cpu_wait_for_steady_stateandwait_for_ecs_steady_statetofalseuntil the IAM change lands. - AgentCore prune no longer uses apply-host
uv/local-exec: with AgentCore enabled, version prune runs in a module Lambda (daily schedule plus invoke before each runtime update). Removerunlayer_agentcore_control_aws_role_arnif you set it for v38 — that input is gone. Apply hosts no longer needuvor AgentCore list/delete IAM; the apply role only needslambda:InvokeFunctionon the prune function (the same credentials that create it).
- Two new CPU guard hosts are created beside the existing topic CPU
stack: with
enable_runlayer_topic_cpu, the apply addsguard-cpu-hot-path(runlayer_guard_cpu_hot_path_instance_type, defaultc7i.xlarge; serves the session tool scanner) andguard-cpu-slow-path(runlayer_topic_cpu_instance_type; topic_cascaded, session titling, tool removal, SkillGuard), each with its own cluster, ASG, NLB, log group and alarms, and points the backend at them. Nothing in the existingrunlayer-topic-cpustack is changed, replaced or removed, so there is no cutover window: the apply waits until both new services report steady state and their target groups have healthy targets (about 6 minutes for the hosts to boot;runlayer_guard_cpu_wait_for_steady_state, defaulttrue, or the module-widewait_for_ecs_steady_state), and only then rolls the backend as on any deploy, while the old stack keeps running. Bounds: the steady-state wait byecs_service_wait_timeout, the target health wait 15 minutes; its credentials are under Breaking changes above. Cost: a default deployment runs three CPU guard hosts instead of one until you drop the old stack (+1 m7i.2xlarge and +1 c7i.xlarge, about +12 vCPUs of On-Demand Standard quota), and one more than today after it; withrunlayer_topic_cpu_desired_count = Nthe duplicate is N slow hosts, and the hot path follows the same N unlessrunlayer_guard_cpu_hot_path_desired_countis set. Budget the new hosts in cost and EC2 quotas, and the extra apply time in CI timeouts. - Drop the old stack once the new ones are verified: after the apply,
confirm on the Grafana
topic-cpu-healthdashboard that scans and topic requests arrive on the new services (the ECS panels show theguard-cpu-*series next to the legacyrunlayer-topic-cpuone), thatSessionToolScannerConnectErrorCountstays at zero and that therunlayer-topic-cpuservice is idle: its CPU series flat, and the<model>.RequestCountmetrics in theRunlayerModels/<project>-<env>namespace (legacy pool only; the new pools publish underRunlayerModels/<project>-<env>/guard-cpu-slow-pathand.../guard-cpu-hot-path) at zero. Then, in a later apply, never the apply that bumps the module, setkeep_legacy_runlayer_topic_cpu = false: that plan destroys only the old cluster, ASG, NLB, log group (with its retained logs; export first if you need them), alerts SNS topic and alarms. In the same apply as the bump it would destroy the old NLB and service before the backend rolls, and the backend’s old tasks would keep calling them while the new hosts boot: SkillGuard imports fail closed and/prescans fail open for that window. Fresh installs are the exception and can set itfalsefrom day one instead of getting an idle old host. The old stack andkeep_legacy_runlayer_topic_cpuare removed in a future major release. - Subscribe to the new alerts topic before dropping the old stack: both
new stacks alarm to
<project>-guard-cpu-alerts-<env>, and, for as long as the old stack is kept, to the old<project>-runlayer-topic-cpu-alerts-<env>topic as well, so the subscriptions you have today page for the new hosts from the first apply on with nothing to do. The old topic and its subscriptions are untouched. Settingkeep_legacy_runlayer_topic_cpu = falseremoves the old topic, after which the new stacks’ alarms fire to the new topic alone: add your email / PagerDuty subscriptions to it before that apply, or the new hosts alarm without paging from then on. - Dashboards and alarm hooks keyed on
runlayer-topic-cpu: ECS metrics for live traffic now report underServiceNameguard-cpu-slow-pathandguard-cpu-hot-path; the log groups are<project>-guard-cpu-slow-path-logs-<env>and<project>-guard-cpu-hot-path-logs-<env>. The shipped Grafana dashboard is updated; private dashboards, SIEM filters or alarm subscriptions matching the old names need the new ones. Model error-rate alarms are added asguard-cpu-slow-path/<model>andguard-cpu-hot-path/<model>, reading per-model metrics from their own namespaces (RunlayerModels/<project>-<env>/guard-cpu-slow-pathand.../guard-cpu-hot-path) so they never sum with the legacy pool’s; thetopic-cpu/<model>alarms and theirRunlayerModels/<project>-<env>metrics stay until the old stack is dropped. Duration metric names and namespaces are unchanged. - Thread pin: the new slow path pins
OMP_NUM_THREADSper worker from the samerunlayer_topic_cpu_omp_threadsinput as the old slow pool (defaultnull= the slow host’s vCPUs /runlayer_topic_cpu_gunicorn_workers, 2 on the m7i.2xlarge default, 1 on an m7i.xlarge with 4 workers), derived from the new stack’s own instance type. An explicit value whosethreads x workersexceeds the host’s vCPUs fails the plan on both stacks. The old stack’s sizing inputs (runlayer_topic_cpu_fast_*, includingrunlayer_topic_cpu_fast_omp_threads) keep their meaning for the old stack until it is dropped; the hot path has its ownrunlayer_guard_cpu_hot_path_omp_threads. - Rollback is two applies, like the cutover:
enable_runlayer_guard_cpu_split = falsepoints the backend back at the oldrunlayer-topic-cpustack, which is always kept while the split is off, and leaves both new stacks running, idle (keep_runlayer_guard_cpu_split_stacks, new, defaulttrue). Terraform destroys a removed stack before it rolls the backend off it, so an apply that did both would leave the backend’s tasks pointed at stacks being torn down for the length of the roll; keeping them is what makes the rollback safe. While the old stack is still running the move is immediate. After you have dropped it (keep_legacy_runlayer_topic_cpu = false), the same apply re-creates it first and, with the wait on, only moves the backend once its tasks are running and its targets are healthy (about 6 minutes for the host to boot; the old service’s steady-state wait runs under the provider’s default 20-minute timeout). Once the backend has rolled, setkeep_runlayer_guard_cpu_split_stacks = falsein a second apply to remove the new stacks; or set the split back totrueto try again.
- The session tool scanner gets its own host:
/prescans no longer share a host with topic_cascaded, session titling, tool removal and SkillGuard. On a dedicated c7i.xlarge the scanner answers 64 % of a 70-concurrent burst inside its 0.8 s budget (27 % on the shared host) and 460/460 of a 20-burst at p95 423 ms; the slow path alone on its host brings topic_cascaded p50 from 1.5 s to 0.12 s with SkillGuard scanning concurrently. - The backend only moves onto a stack that answers: the backend task
definition is only rolled to the new NLBs after both
guard-cpu-*services exist and, withrunlayer_guard_cpu_wait_for_steady_state(new, defaulttrue), report steady state and have healthy targets; on a rollback the same holds for the old stack. A/prescan or SkillGuard import can never be sent to a target group whose host is still booting. Both new containers run the same plain/healthcheck the topic CPU pools run today; as today, that check is 200 whenever the server is up, even if a model failed to load, so it does not prove the scanner or SkillGuard model is loaded. That is a pre-existing property of the serving image, unchanged by this release and tracked as a follow-up on the image. Moving the backend and removing the stack it left are always separate applies (keep_legacy_runlayer_topic_cpu,keep_runlayer_guard_cpu_split_stacks). - New hot-path inputs:
runlayer_guard_cpu_hot_path_instance_type(c7i.xlarge),_desired_count(null: followsrunlayer_topic_cpu_desired_count, so a deployment on two topic CPU hosts gets two scanner hosts, not one),_gunicorn_workers(4),_memory(6144),_omp_threads(1),_queue_max(22, per worker,DYNAMIC_BATCH_QUEUE_MAX),_expected_concurrent_callers(16),_models(["runlayer-dynamic-policy"], becomes the container’sMODEL_FILTER; must stay a subset ofrunlayer_topic_cpu_modelsand must includerunlayer-dynamic-policy, the model the backend’s scanner requests, so a roster that would 404 every/preis refused at plan).workers x threadsmust fit the instance’s vCPUs and memory must fitworkers x 1024and the instance; validation refuses otherwise. - Hot-path alarms: no-running-tasks, memory > 85 %, CPU > 70 % for
15 minutes, and unhealthy-targets alarms on the hot-path service, on the
<project>-guard-cpu-alerts-<env>topic with the slow-path alarms (and on the old topic while the old stack is kept, see Action required). - Scanner connection errors are counted: a new
SessionToolScannerConnectErrorCountmetric (namespaceRunlayerTopicCpu/<project>-<env>, from backend logs) sits next to the existing timeout count on thetopic-cpu-healthdashboard, so a scanner endpoint that is unreachable shows separately from one that is slow. - Aurora lock-wait logging: Set
database_config.log_lock_waits = trueto log waits exceeding the existingdeadlock_timeout. The default leaves the parameter unspecified; setfalseto disable logging explicitly. - AgentCore prune uses a least-privilege Lambda role: the function can
list/delete AgentCore runtime versions (plus companion
DeleteAgentRuntimeEndpoint/DeleteWorkloadIdentitythat versioned delete authorizes) and write its own logs. It also runs once a day on EventBridge. The function zip ships a botocore data overlay so runtime boto3 can passagentRuntimeVersion. - Least-outstanding-requests routing for the backend: set
backend_load_balancing_algorithm = "least_outstanding_requests"to send each backend request to the task with the fewest in flight, which lowers median latency under load. It turns off the backend’s 60s slow start whenbackend_health_scaling_enabledis on (AWS can’t combine the two), and makes the backend hold each 5xx briefly so a task that fails fast can’t draw most of the traffic. If you pinapp_version, use a backend release that includes that hold. The default,round_robin, changes nothing. - Backend CPU target override:
backend_cpu_target_valuesets the backend’s CPU target-tracking target without replacingservices_configurations. A lower target runs more tasks for lower latency; recheckrds_database_connections_thresholdwhen you lower it. - Steadier backend scale-in: with
backend_health_scaling_enabled = true, the quiet-fleet scale-in now waits until CPU is below0.9 x 0.75of the target (47.25% for the default 70%), not 90% of it, so removing a quarter of the tasks lands under the target instead of triggering an immediate scale-out. Busy fleets may keep slightly more tasks.
Default Runlayer app version:
1.33.0. See the Runlayer changelog.Improvements- Customer-managed Aurora encryption: optionally set
rds_kms_key_arnbefore creating a cluster to use your own KMS key for database storage, backups, and snapshots. The default preserves existing key selection; changing an existing cluster’s key requires migration.
Default Runlayer app version:
1.32.0. See the Runlayer changelog.Action required- New session compactor service, ingest rolls after it: with
enable_hook_events_pipeline, the apply createssession-materializer-compactor-serviceand waits for it to be healthy before it rolls the existing materializer service. The compactor runs tasks only whilesession_parquet_store_enabledis on. Bump the module together with or beforeapp_version. If the store is already on when you bump andapp_versionis older than this module release, the apply fails on the compactor and leaves the materializer unchanged; raiseapp_versionand apply again. If the store is off, the apply succeeds either way: before you turn the store on, raiseapp_versionto this release or later. A mistake there is not caught by the apply (the compactor rolls at zero tasks); with monitoring on it shows up as the compactor’s running-tasks alarm (needs Container Insights) or busy dead-man alarm (needs critical alerts and autoscaling). Roll the module back only together withapp_version. - Fixed-size materializer configs must pin capacity: the
session_materializer_config.max_capacitydefault rises from 1 to 4, which turns on autoscaling (1–4 tasks) and makesdesired_countseed-only. If you setdesired_count(to 0 to stop ingest, or above 1 for a larger fixed fleet), also setmin_capacityandmax_capacityto the same value (for example both 1) before upgrading. Otherwise ingest restarts or resizes to 1–4 tasks on the upgrade.
- PrivateLink deployment access: Adds IAM read permissions for checking security-group rule capacity and waiting for ECS rollouts before removing obsolete access rules. Existing module inputs remain compatible.
- Materializer sizing split:
session_materializer_confignow sizes ingest only (new defaults 1 vCPU / 2 GB, autoscaling 1–4 tasks on ingest lag). Compaction is sized bysession_materializer_compactor_config(default 2 vCPU / 4 GB, 1–4 tasks on fold load; zero tasks whilesession_parquet_store_enabledis off). If you raisedsession_materializer_configfor compaction, move those values tosession_materializer_compactor_config. - Materializer deploys start new tasks first: ingest no longer stops before the new task starts. Upgrading from a module older than shard leasing, the first deploy may deliver some session events twice; nothing is lost.
Default Runlayer app version:
1.30.0. See the Runlayer changelog.Breaking changes- AgentCore prune needs deploy-account credentials on apply: with AgentCore
enabled, prune fails if the apply-host credentials are not the deploy
account (the aws provider account). Before bumping, set
runlayer_agentcore_control_aws_role_arnwhen terraform starts in a build/parent account, or ensure ambient credentials are already the deploy account.
- AgentCore prune no longer needs a preinstalled
uv: with AgentCore enabled, apply uses hostuvwhen present, otherwise bootstraps a pinneduvinto a temp dir (curl+ Astral installer, which then fetches the binary from GitHub releases). Air-gapped hosts should putuvonPATHor set the installer’s download-base env (UV_INSTALLER_GITHUB_BASE_URL/UV_DOWNLOAD_URL); mirroring onlyinstall.shis not enough. AgentCore control IAM from v34.0.0 is unchanged.
Default Runlayer app version:
1.29.0. See the Runlayer changelog.Improvements- Agent iMessage channel (opt-in):
enable_imessage_channel = truelets people text an agent from Messages through Sendblue, with no Apple hardware in the stack. Off by default. Set it with the line’ssendblue_from_number,sendblue_api_key_id,sendblue_api_secret_keyandsendblue_webhook_secret(the plan fails without all four), and add theimessage_channel_sendblue_webhook_urloutput as the line’sreceivewebhook in Sendblue. See ECS optional features → iMessage channel.
Default Runlayer app version:
1.28.1575. See the Runlayer changelog.Breaking changes- The agent runtime no longer needs
enable_runlayer_agents: withenable_runlayer_agentcore_runtime = trueandenable_runlayer_agents = false, the next apply creates the AgentCore runtime. Before bumping, give the apply hostuvand credentials for the AgentCore control APIs and, with WAF IP allowlisting, setrunlayer_agentcore_network_mode = "VPC"with PrivateLink; or setenable_runlayer_agentcore_runtime = falseto keep it off.
- Runlayer Assistant without custom Agents: set
enable_runlayer_agentcore_runtime = trueto run Runlayer Assistant, Security Investigations and Procedures.enable_runlayer_agentsis only needed for custom agents. - Two session-compaction alarms: deployments with
enable_monitoring,critical_alerts_enabledand a running session materializer get a non-paging warning when a session sits at the fold quarantine breaker (CompactionQuarantinedSessions >= 1for 15 min — it never self-heals), and a paging CRITICAL when, for 30 min straight, every sweep saw the oldest dirty session waiting more than 2h for compactor work (CompactionBacklogAgeSeconds, Minimum > 7200 — Minimum rides out the short ~24h blips that late-arriving events cause on a healthy queue). Both need the backend image this release ships: on older images the backlog-age gauge idles at ~86,400 s and the CRITICAL alarm stands in permanent breach, and the quarantine gauge is not emitted at all. If you pin an older app version, hold this module bump until you move the app forward.
Default Runlayer app version:
1.28.1480. See the Runlayer changelog.Action required- Check the topic-cpu slow pool’s memory per worker and image tag before
bumping, or pin SkillGuard off:
runlayer_skill_guard_enablednow defaults totrue, so every deployment withenable_runlayer_topic_cpu = truescores skills with SkillGuard after the bump, and two validations failplanfor shapes that cannot run it. Memory:runlayer_topic_cpu_memorymust be at least `runlayer_topic_cpu_gunicorn_workers- 2560
MiB while the switch is on (SkillGuard scans peak at ~2.2 GiB per worker, measured 2026-10-16 across the fleet); the module default (4 workers, 16384 MiB) passes, but a slow pool left on the xlarge-era 8192 MiB with 4 workers does not. Image:runlayer_topic_cpu_image_tagmust berunlayer-multimodel-guard-v202607291645or newer (the first bundle with the SkillGuard model); the module default passes. For each failing input, either raise it to the default or setrunlayer_skill_guard_enabled = falseto keep the previous skill scorer and the 2048 MiB/worker floor. Once SkillGuard is on it becomes the source of each skill's risk level: skills it rates higher may be newly blocked under your skill policies, and file preview follows that level. The switch is carried by this module only; the backend default stays off, so Kubernetes (operator) and docker-compose deployments are unchanged until they setRUNLAYER_SKILL_GUARD_ENABLED` themselves.
- 2560
- Stable public IPv4 block for the ALB: set
enable_alb_public_ipv4_pool = trueto give the public ALB its IPv4 addresses from one Amazon-provided block, output asalb_public_ipv4_cidr, that client networks can route or allowlist. It creates the account’s IPAM inregion, so an account that already has one there can’t turn it on. Turning it on changes the ALB’s public IPv4 addresses once; turning it off releases the block, and a later re-enable gets a different CIDR.
Default Runlayer app version:
1.28.1446. See the Runlayer changelog.Improvementsrunlayer_topic_cpu_fast_queue_max: new input (default22, per worker) settingDYNAMIC_BATCH_QUEUE_MAXon the topic CPU fast-pool container. Both topic CPU containers run one image and therefore one serving profile, which carries the slow pool’s shallow cap; the apply now sets the fast pool’s cap explicitly so it stops inheriting that value. 22 is one CPU thread’s in-budget capacity for a/prescan, so the container admits22 x runlayer_topic_cpu_fast_gunicorn_workersconcurrent scans (132 at the default 6) before shedding.0disables admission (rollback). No action required — the only visible change is a task definition revision.runlayer_topic_cpu_expected_concurrent_callers(default4),runlayer_topic_cpu_fast_expected_concurrent_callers(default16),runlayer_tool_guard_expected_concurrent_callers(default16): new inputs passed to each guard container asEXPECTED_CONCURRENT_CALLERS, the concurrency the pool is sized for. The serving image reports an admission cap below it at boot and in/health. Validation refuses a fast-pool cap below the declared fast-pool concurrency. No action required.
Default Runlayer app version:
1.28.1407. See the Runlayer changelog.No upgrade notes for this release.Default Runlayer app version:
1.28.1396. See the Runlayer changelog.No upgrade notes for this release.Default Runlayer app version:
1.28.1381. See the Runlayer changelog.No upgrade notes for this release.Default Runlayer app version:
1.28.1271. See the Runlayer changelog.No upgrade notes for this release.Default Runlayer app version:
1.28.1233. See the Runlayer changelog.No upgrade notes for this release.Default Runlayer app version:
1.28.1158. See the Runlayer changelog.Improvements- WAF no longer blocks code in hook and upload bodies: on the routes in
bad_inputs_count_uri_regex, AWS KnownBadInputs body matches now count instead of block, and header, path, query and method matches block. - More routes, any client: the
bad_inputs_count_uri_regexdefault adds AI Watch hooks and scan inventory, local MCP pre/post and tool sync, skill/plugin batch uploads, and skill file sync and scoring, andbad_inputs_count_user_agentnow defaults to""(was"Runlayer CLI"). If you set either, compare it with the new default. - Renamed WAF rule: the count-mode rule is now
AWS-AWSManagedRulesKnownBadInputsRuleSet-BodyCountRoutes(was...-CountRunlayerCli); its metric name is unchanged and now also records blocked requests. Update WAF log queries that match the old rule name.
Default Runlayer app version:
1.28.1115. See the Runlayer changelog.Breaking changes- AgentCore needs more IAM on apply: with AgentCore enabled, the AWS
credentials on the host that runs
terraform apply(not a providerassume_role) needbedrock-agentcore:ListAgentRuntimes,bedrock-agentcore:ListAgentRuntimeVersions,bedrock-agentcore:ListAgentRuntimeEndpointsandbedrock-agentcore:DeleteAgentRuntime. uvon the apply host: with AgentCore enabled, installuvwhereterraform applyruns.
- Redis connection alarms scale with your backend: they page on connections
per backend task above
redis_connections_per_backend_task_threshold(default750), so scale-outs and deploys no longer page. They need Container Insights; withcontainer_insights = "disabled"only the backstop below remains. redis_curr_connections_thresholdis now a raw backstop: default rises from2000to20000. If you set it, keep it above your max backend tasks × ~500. The critical Redis cluster’s alarm still uses your value, or2000if you never set one; setcritical_redis_curr_connections_thresholdto split them.- Deploys no longer hit AgentCore’s version cap: with AgentCore enabled,
each apply that updates the runtime deletes its oldest versions, keeping
runlayer_agentcore_runtime_versions_to_keep(default50).
Default Runlayer app version:
1.28.1078. See the Runlayer changelog.Improvements- Backend worker-death alarm: a new
…-backend-worker-deathsalarm pages when more than two backend uvicorn workers die within 15 minutes (theWorkerDiedmetric in theRunlayer/Backend/<project>-<environment>namespace, read from the backend task log group). Repeated deaths usually mean task memory is at the container limit and the kernel is killing workers. Created withenable_monitoringandcritical_alerts_enabled; nothing to configure.
Default Runlayer app version:
1.28.1073. See the Runlayer changelog.Breaking changes- Rev CPU variables removed: delete
runlayer_rev_gunicorn_workers,runlayer_rev_memoryandrunlayer_rev_cpu_sharefrom your configuration. Rev no longer runs in the topic CPU task.
- Rev runs as its own GPU service: with
enable_runlayer_rev = true, the apply removes the Rev container from the topic CPU task and creates a dedicated GPU ECS service (g6f.large, its own cluster, internal NLB on:8080, security group and alert topic). Rev is unavailable until the first GPU host boots, typically 10 to 15 minutes. Confirm g6f.large is available and within quota in your region before upgrading. - No-healthy-hosts alarm: the new alarm pages if no Rev host registers
within 20 minutes of the first apply. Set
enable_runlayer_rev_alarmsto control it.
runlayer_rev_desired_count: new input (default1) for the Rev GPU service task count.