> ## Documentation Index
> Fetch the complete documentation index at: https://docs.runlayer.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Agent Evals
> Replay past agent runs with a different model or prompt, score the results, and compare against a control — without touching your real tools.
Offline evals let you answer "would this agent do better with a cheaper model?"
or "does my new prompt regress anything?" using runs the agent has already
done. Runlayer replays selected past runs under your changed configuration and
scores the outcome against a matched control replay.
## How offline evals work
For every past run you select, Runlayer spawns **two fresh replays** of the
same task:
* a **control** replay using the agent's current model and prompt
* a **candidate** replay using the same setup plus your overrides
Scorers then compare the candidate against the control. Comparing two live
replays (rather than the historical run) keeps the comparison fair — both
sides run under today's conditions, and only your override differs. The
historical run's output is still shown for reference.
**Replays never call your real tools.** MCP tool calls are intercepted and
answered by an LLM that fabricates plausible results, primed with the tool
calls recorded in the original run so the simulation stays realistic. An agent
that normally sends Slack messages or writes to your database does neither
during an eval. Sandbox-internal commands (shell, filesystem inside the
sandbox) still execute. Memory writes are also disabled during replays.
## Run an eval
Evals live on the agent page under the **Evals** tab (visible to anyone who
can edit the agent).
On the **Evals** tab click **New eval**. Alternatively, on the **Monitor**
tab click **Select**, pick up to 20 finished runs, and click
**Run offline eval** to start from those runs.
Select up to **20 finished runs**. Recurring agents work best — replaying
several instances of the same scheduled task gives you a distribution
instead of a single sample.
An eval needs at least one override:
* **Model override** — try a different model available to your workspace
* **Override system prompt** — edit the agent's prompt for the replay
* **Harness override** — where bring-your-own-harness is enabled for your
workspace
Then pick one or more **scorers** (see below) and click **Run eval**.
Cases replay in parallel in the background. The eval page updates live while
the eval is running.
## Scorers
Scorers judge each case and return a score from **0 to 1** (1 is best). Three
built-in LLM scorers are available:
| Scorer | What it does |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| **Baseline battle** | Compares the candidate replay against the control and picks a verdict: **improved**, **similar**, or **regressed** |
| **Goal completion** | Rates how completely the replay accomplished the task |
| **Instruction adherence** | Checks whether the replay followed the agent's system instructions |
You can also create **custom scorers** per agent. Click **Author scorer with
AI**, describe what to measure in plain language (e.g. "Score 1 when the reply
contains no apology boilerplate and is under 200 words"), and Runlayer
generates a TypeScript scorer you can review, edit, and save. Custom scorers
run with a 30-second timeout and no filesystem writes or child processes.
The scorers you select are frozen into the eval when it starts — deleting or
editing a scorer later doesn't change past results.
## Read the results
Each eval has a detail page with one row per replayed case: status, one
column per scorer, and cost and runtime with deltas versus control.
Aggregates for cost, runtime, and every scorer appear in the sidebar.
Click a case to open it side by side: the historical output, the control
output, and the candidate output, plus every scorer's score and reasoning.
From there you can jump to the full **Historical**, **Control**, or
**Candidate** run traces.
LLM judges are a signal, not ground truth — you can record your own verdict
(**Improved** / **Similar** / **Regressed**) and notes on any case. Human
verdicts are tallied in the sidebar alongside the scorer averages.
## Costs and limits
* Each case runs **two real agent replays** plus scorer and tool-simulation
LLM calls, billed to your configured AI provider like any other run. The
**Cost** column is often the headline result when evaluating a cheaper
model.
* Up to **20 runs per eval**; only finished runs can be replayed.
* Replays appear in the agent's run history like normal runs.
* LLM scoring is non-deterministic — treat small score differences as noise
and look at verdict counts and cost/runtime deltas across many cases.
* Creating evals, recording verdicts, and managing scorers require edit
access to the agent. If workspace redaction is enabled, a redacted run must
be unredacted before it can be used as an eval source.