Golden run
A recorded run of a workflow or agent that you have checked and keep as the reference: its input, every step's result and every outside call, for replaying changes against.
Agents are hard to regression test because they call the outside world. Rerun last week's test and the search results, the ticket queue and the clock have all moved, and a write tool would really write. A golden run freezes one good run so a later version can be compared with it on the same footing. The outside calls are answered from the recording (a cassetteCassette: A recording of the outside calls a run made, with their answers, so a later test run can be answered from it instead of calling the real services.), so the test has no side effects and does not depend on other services being up.
What it holds
- the input that started the run;
- each step's output, in order;
- each tool call with its arguments and its result;
- the final output, with facts to compare such as status, cost and tokens.
Example
A support workflow answered "Where is my order 1182?" well: it called lookup_order, then get_shipment, and replied with a tracking link. You save that run as golden. Next week you switch to a cheaper model and replayReplay: Running a changed workflow or agent on the input of a recorded run, with its calls to the outside world answered from the recording, and comparing the result with the original. the golden run against the new version. The new model skips get_shipment and invents a date. The replay fails before any customer sees it.
Common confusion
A golden run is not a snapshot test of the final text. Model wording changes from run to run, so useful comparisons look at the path (which tools, with what arguments), the structure of the output, and limits on cost, rather than demanding identical sentences. Several golden runs together make a golden setGolden set: A curated collection of reference cases, each an input with a known-good result, that a new version of an AI workflow or agent must pass before it ships..
In L2M
On the Runs page, Save as golden keeps a finished run with its node results and, when the run was recorded with Record MCP calls, its MCP cassette and model tape. Replay answers MCP, code and outside steps from the recording and gives pass, fail or inconclusive. See Golden runs and replay.
Further reading
- Demystifying evals for AI agents (anthropic.com)