Skip to main content
Offline evals let you answer “would this agent do better with a cheaper model?” or “does my new prompt regress anything?” using runs the agent has already done. Runlayer replays selected past runs under your changed configuration and scores the outcome against a matched control replay. Run an eval without any override and it becomes a pure regression test: each replay restores the original run’s captured setup and is scored against the historical run. Evals tab on the agent page, listing completed offline evals

​
How offline evals work

With at least one override, Runlayer spawns two fresh replays of the same task for every past run you select:
  • a control replay using the original run’s captured model and prompt
  • a candidate replay using the same setup plus your overrides
Scorers then compare the candidate against the control. Comparing two live replays (rather than the historical run) keeps the comparison fair — both sides run under the original run’s captured conditions, and only your override differs. The historical run’s output is still shown for reference. With no overrides, each selected run replays once restoring the original run’s captured setup, and scorers compare it against the historical run itself. Use this to catch regressions from drift outside the snapshot — for example, the underlying model’s behavior or the runtime that runs the replay changed between the original run and now. By default, replays never call your real tools. MCP tool calls are not sent to your real services — Runlayer simulates their results using the original run as reference, so the replay stays representative without side effects. An agent that normally sends Slack messages or writes to your database does neither during an eval. Sandbox-internal commands (shell, filesystem inside the sandbox) still execute. Memory writes are also disabled during replays. You can opt out of this per eval with the Tool mocking option in the eval dialog — non-default policies let some or all tool calls run against your real services and can perform real side effects. Simulated results are generated fresh for every replay, so the control and candidate can see slightly different data. Pick Replay recorded results to remove that noise: when a replay makes a call the original run made, with the same arguments, it gets the original run’s recorded result, including any error. Calls the original run never made are still simulated. This option is side-effect free too.

​
Run an eval

Evals live on the agent page under the Evals tab (visible to anyone who can edit the agent).
1

Open the eval dialog

On the Evals tab click New eval. Alternatively, on the Monitor tab click Select, pick up to 20 finished runs (shift-click selects a range), and click Run offline eval to start from those runs. The dialog then shows a summary of them; click Edit selection to change it.
2

Pick runs to replay

Select up to 20 finished runs from the last 30 days. To find them:
  • Search run text, or filter by status, start date, and topic. Shift-click a row to select a range, or use the header checkbox to select every run shown.
  • Quick picks replace the selection in one click: latest runs, failures from the last 7 days, a spread across topics, unretried write failures, or the most expensive runs.
  • Ask AI: describe the runs you want, for example “failed Slack runs this week where it never answered”, and press Enter. Status and date parts become normal filters; the rest becomes an AI match filter, and each matched row says why it matched. Matches are added to the selection, and you can uncheck any of them.
Picks stay selected when you change filters. Recurring agents work best — replaying several instances of the same scheduled task gives you a distribution instead of a single sample.
3

Choose what to change (optional)

Overrides are optional:
  • Model override — try a different model available to your workspace
  • Override system prompt — edit the agent’s prompt for the replay
  • Harness override — where bring-your-own-harness is enabled for your workspace
  • No override — replay under the original run’s captured setup and compare against the historical runs (regression test)
Then pick one or more scorers (see below) and click Run eval.
Run offline eval dialog with run selection, scorers, and overrides Cases replay in parallel in the background. The eval page updates live while the eval is running.

​
Scorers

Scorers judge each case and return a score from 0 to 1 (1 is best). Three built-in LLM scorers are available: You can also create custom scorers per agent. Click Author scorer with AI, describe what to measure in plain language (e.g. “Score 1 when the reply contains no apology boilerplate and is under 200 words”), and Runlayer generates a TypeScript scorer you can review, edit, and save. Custom scorers run with a 30-second timeout and no filesystem writes or child processes. The scorers you select are frozen into the eval when it starts — deleting or editing a scorer later doesn’t change past results.

​
Read the results

Each eval has a detail page with one row per replayed case: status, one column per scorer, and cost and runtime with deltas versus control. Aggregates for cost, runtime, and every scorer appear in the sidebar. Eval detail page with per-case scores, verdicts, and cost/runtime vs control Click a case to open it side by side: the historical output, the control output, and the candidate output, plus every scorer’s score and reasoning. From there you can jump to the full Historical, Control, or Candidate run traces. Eval case view comparing historical, control, and candidate outputs LLM judges are a signal, not ground truth — you can record your own verdict (Improved / Similar / Regressed) and notes on any case. Human verdicts are tallied in the sidebar alongside the scorer averages. Comparative scorers such as Baseline battle also show Agreement with you: how many of your labelled cases the judge gave the same verdict, plus a confusion matrix of your verdict against the judge’s. Until you’ve labelled at least 10 cases, the sidebar warns that the number is too noisy to trust.

​
Costs and limits

  • Each case runs two real agent replays (one for a no-override eval) plus scorer and tool-simulation LLM calls, billed to your configured AI provider like any other run. The Cost column is often the headline result when evaluating a cheaper model.
  • Up to 20 runs per eval; only finished runs can be replayed.
  • Replays appear in the agent’s run history like normal runs.
  • LLM scoring is non-deterministic — treat small score differences as noise and look at verdict counts and cost/runtime deltas across many cases.
  • Creating evals, recording verdicts, and managing scorers require edit access to the agent. If workspace redaction is enabled, a redacted run must be unredacted before it can be used as an eval source.