Skip to main content
Offline evals let you answer “would this agent do better with a cheaper model?” or “does my new prompt regress anything?” using runs the agent has already done. Runlayer replays selected past runs under your changed configuration and scores the outcome against a matched control replay. Evals tab on the agent page, listing completed offline evals

​
How offline evals work

For every past run you select, Runlayer spawns two fresh replays of the same task:
  • a control replay using the agent’s current model and prompt
  • a candidate replay using the same setup plus your overrides
Scorers then compare the candidate against the control. Comparing two live replays (rather than the historical run) keeps the comparison fair — both sides run under today’s conditions, and only your override differs. The historical run’s output is still shown for reference. Replays never call your real tools. MCP tool calls are intercepted and answered by an LLM that fabricates plausible results, primed with the tool calls recorded in the original run so the simulation stays realistic. An agent that normally sends Slack messages or writes to your database does neither during an eval. Sandbox-internal commands (shell, filesystem inside the sandbox) still execute. Memory writes are also disabled during replays.

​
Run an eval

Evals live on the agent page under the Evals tab (visible to anyone who can edit the agent).
1

Open the eval dialog

On the Evals tab click New eval. Alternatively, on the Monitor tab click Select, pick up to 20 finished runs, and click Run offline eval to start from those runs.
2

Pick runs to replay

Select up to 20 finished runs. Recurring agents work best — replaying several instances of the same scheduled task gives you a distribution instead of a single sample.
3

Choose what to change

An eval needs at least one override:
  • Model override — try a different model available to your workspace
  • Override system prompt — edit the agent’s prompt for the replay
  • Harness override — where bring-your-own-harness is enabled for your workspace
Then pick one or more scorers (see below) and click Run eval.
Run offline eval dialog with run selection, scorers, and overrides Cases replay in parallel in the background. The eval page updates live while the eval is running.

​
Scorers

Scorers judge each case and return a score from 0 to 1 (1 is best). Three built-in LLM scorers are available: You can also create custom scorers per agent. Click Author scorer with AI, describe what to measure in plain language (e.g. “Score 1 when the reply contains no apology boilerplate and is under 200 words”), and Runlayer generates a TypeScript scorer you can review, edit, and save. Custom scorers run with a 30-second timeout and no filesystem writes or child processes. The scorers you select are frozen into the eval when it starts — deleting or editing a scorer later doesn’t change past results.

​
Read the results

Each eval has a detail page with one row per replayed case: status, one column per scorer, and cost and runtime with deltas versus control. Aggregates for cost, runtime, and every scorer appear in the sidebar. Eval detail page with per-case scores, verdicts, and cost/runtime vs control Click a case to open it side by side: the historical output, the control output, and the candidate output, plus every scorer’s score and reasoning. From there you can jump to the full Historical, Control, or Candidate run traces. Eval case view comparing historical, control, and candidate outputs LLM judges are a signal, not ground truth — you can record your own verdict (Improved / Similar / Regressed) and notes on any case. Human verdicts are tallied in the sidebar alongside the scorer averages.

​
Costs and limits

  • Each case runs two real agent replays plus scorer and tool-simulation LLM calls, billed to your configured AI provider like any other run. The Cost column is often the headline result when evaluating a cheaper model.
  • Up to 20 runs per eval; only finished runs can be replayed.
  • Replays appear in the agent’s run history like normal runs.
  • LLM scoring is non-deterministic — treat small score differences as noise and look at verdict counts and cost/runtime deltas across many cases.
  • Creating evals, recording verdicts, and managing scorers require edit access to the agent. If workspace redaction is enabled, a redacted run must be unredacted before it can be used as an eval source.