Skip to content

Golden set ​

A curated collection of reference cases, each an input with a known-good result, that a new version of an AI workflow or agent must pass before it ships.

It is also called a golden dataset. In eval terms it is the task list of a regression evalRegression eval: An eval that checks a system still does what it did before a change. It should pass nearly every time; a failure means something broke.: the cases that define "still works". Each case may be a recorded golden runGolden run: A recorded run of a workflow or agent that you have checked and keep as the reference: its input, every step's result and every outside call, for replaying changes against. or a hand-written input with an expected answer. The set as a whole is a contract, so changing it changes what "passed" means, and it deserves the same review as code.

What makes a good set:

  • Real cases. Anthropic's evals guide says "20-50 simple tasks drawn from real failures is a great start", and recommends turning user-reported failures into test cases.
  • Unambiguous cases. Its test for a good task: "two domain experts would independently reach the same pass/fail verdict". A case that people would argue about makes a noisy test.
  • A known-good result for each case, so a failure means the candidate changed, not that the case was never solvable.
  • Small enough to run on every change, which keeps it in the release path instead of in a nightly job nobody reads.
  • An owner. The same guide calls an eval suite "a living artifact that needs ongoing attention and clear ownership to remain useful". New failures get added, and cases that test retired behaviour get removed on purpose.

Example ​

A refunds workflow has a set of 15 cases: six common requests, five edge cases (partial refund, wrong currency, order older than 30 days) and four bugs from last quarter, each named for what it checks, such as "refund after 30 days is declined".

Common confusion ​

A golden set is not training data and not a public benchmark. It is private to your system and describes your behaviour, not general skill. Nor is it a pile of golden runs: the point is the selection.

In L2M ​

A golden set belongs to one workflow and holds its chosen golden runs. To test a change, a replayReplay: Running a changed workflow or agent on the input of a recorded run, with its calls to the outside world answered from the recording, and comparing the result with the original. job plays every golden run in the set against the version you pick (a draft, a saved version or a workflow file) and scores it. Hand-written cases can sit in a set, but replay jobs skip them. An enforced release gateRelease gate: A check in the release path that lets a new version of a workflow or agent go live only after its tests passed on that exact version. counts only cases an admin admitted, so the set cannot be padded with easy cases.

Further reading ​

This website (not the L2M product) uses Google Analytics, only after you accept cookies. Cookie settings