A golden dataset for LLM evaluation is a fixed, versioned set of test inputs, each stored with the behaviour you expect and a grader that returns pass or fail. Build it from failures your users have already hit. Mine production traces and support tickets, sort the failures into a taxonomy, write each one as a test case with a code grader, freeze the cases as a regression set, and add cases as new failures appear. Evals built from your own failures are the first gate in our guide to production-ready AI agents.

Where generic benchmarks and synthetic data fall short

Hamel Husain argues that product evals built around your users’ tasks “measure something different from general model benchmarks” (Your AI Product Needs Evals, 29 March 2024). His evals FAQ with Shreya Shankar (18 September 2026) says generic evaluations “create false confidence when you use them as quality measures”. OpenAI’s evaluation best practices list “biased design” as an anti-pattern and describe it as “creating eval datasets that don’t faithfully reproduce production traffic patterns”.

Synthetic inputs help before launch, but “synthetic data cannot tell you how common a failure is in production” (FAQ). Anthropic’s guide to agent evals (9 January 2026) names the better source when it says, “If you’re already in production, look at your bug tracker and support queue.”

How to build a golden dataset, step 1: collect failures from traces and tickets

The FAQ defines a trace as “a complete record of one user session with your application”. From each trace, export the user’s messages, each tool call with its arguments and results, the final reply, and the model and prompt versions. Pull candidates from three places:

  1. Support tickets and bug reports about the agent. Anthropic says “converting user-reported failures into test cases ensures your suite reflects actual usage”.
  2. Traces with a negative signal, such as a thumbs-down, a hand-off to a human, a tool error or a repeated request.
  3. A random sample of all traces, to catch failures nobody reported. The FAQ calls this “the simplest approach”.

Keep the tool results with each trace, so a case can replay them without calling live systems (the account object in the example below). Langfuse’s dataset documentation (vendor source) describes this workflow for “production traces where the application did not perform as expected”, with a source_trace_id on each item.

Redact personal data before it leaves production

Redact names, email addresses and account details inside production, before anything is exported or committed:

  • Run a PII detector over free text, plus rules for structured fields such as account IDs. Presidio, an MIT-licensed open-source library, detects common PII entities, and its replace operator defaults to typed placeholders such as <PHONE_NUMBER>.
  • Number placeholders within a case (<EMAIL_1>, <EMAIL_2>) so the conversation still reads correctly. If a grader depends on a value’s shape, substitute a synthetic value such as customer1@example.com.
  • Have a person read every case before it is committed. Presidio’s documentation warns that “there is no guarantee that Presidio will find all sensitive information”.
  • Keep the mapping from placeholders to real values inside production, or delete it. The UK ICO’s pseudonymisation guidance (10 October 2025, now under review) says pseudonymised data “is personal data in the hands of someone who holds the additional information” and may be anonymous for a recipient without it. Treat the committed set as personal data until your data protection officer agrees otherwise.

Step 2: read the failures and build an AI agent failure taxonomy

The core of Husain and Shankar’s error analysis is two passes borrowed from qualitative research:

  1. Open coding. A domain expert reads each trace and writes a free-text note on anything wrong, starting with the first failure, since “upstream errors can cause downstream issues”.
  2. Axial coding. Group the notes into named failure classes and count each class.

The FAQ recommends “reviewing at least 100 traces” and continuing until new traces stop revealing failure modes, which qualitative researchers call theoretical saturation. Then rank the classes by impact and count, and mark the ones that would stop a launch as critical.

A failure-taxonomy template

Keep the taxonomy in the repository next to the cases. The example rows describe the hypothetical billing-support agent from the code below. Replace “Seen in” with your counts (failing traces out of traces read) and two or three redacted IDs.

Failure class Definition (one sentence a second reviewer can apply) Seen in Impact Critical Grader Threshold
refund_policy Refunds a charge older than 30 days, or refuses a refund inside the window n of N traces, with IDs Money Yes Code: tool_not_called, tool_called, reply_matches 100%
cancel_timing Cancels at once when the user asked to cancel at the end of the billing period n of N, with IDs Revenue, trust Yes Code: field_equals, tool_not_called, reply_matches 100%
complaint_tone Argues with, or blames, a customer who reports a billing error n of N, with IDs Churn No LLM judge, calibrated on human labels Set with the product owner
unsorted Notes that fit no class yet n of N, with IDs Unknown No None yet None

Set each threshold with the product owner. In Husain’s words, “Your pass rate is a product decision, depending on the failures you are willing to tolerate.”

Step 3: write each failure as a test case with a grader

Each test case holds one input, the tool results the agent will see, the expected behaviour in one sentence, and the checks that decide pass or fail. Three tests for a case, from Anthropic’s guide:

  1. Two domain experts “would independently reach the same pass/fail verdict”.
  2. “Test both the cases where a behavior should occur and where it shouldn’t.” Below, refund-001 must not refund and refund-002 must.
  3. A reference solution, “a known working output that passes all graders”, proves every check can pass.

Code checks first

Use a deterministic code check whenever a rule can decide the case, such as an exact field value, a required or forbidden tool call, a regex over the reply or a schema check. Anthropic’s evaluation docs call code-based grading “fastest and most reliable, extremely scalable”, and the FAQ’s advice for CI is to “favor assertions or other deterministic checks over LLM-as-judge evaluators”.

  • Grade outcomes before paths. Anthropic’s guide recommends grading “what the agent produced, not the path it took”, so require a tool call only when the call is itself the rule, such as looking up the account before discussing a charge.
  • Eval tools ship the same checks. Promptfoo has regex, trajectory:tool-used and a not- prefix that negates any assertion, and Ragas has ToolCallAccuracy, which compares calls with reference calls, in order by default (vendor sources).

LLM-as-a-judge only where code can’t decide

Tone, or whether an explanation answers the customer’s question, needs judgement. For those, use an LLM-as-a-judge that returns pass or fail for one failure class, calibrated against human labels before it gates anything:

  1. The domain expert labels 100 to 200 examples for the class (FAQ).
  2. Split them as Husain’s LLM-as-a-judge guide suggests, with 10–20% as examples the judge prompt may include, 40–45% as a dev set for refining it, and 40–45% for one final test.
  3. Report the true positive rate (failures caught) and true negative rate (good outputs passed) on the test set, since “raw agreement can be misleading when classes are imbalanced” (Husain’s LLM-as-a-judge guide).

Our guide to calibrating an LLM as a judge covers the prompt, the bar and a script that corrects the judge’s pass rate for its errors.

A runnable example with a case schema and runner

The example uses only the Python standard library, and we ran it on Python 3.11.17, 3.12.3 and 3.13.16 with identical output. It loads JSONL cases, calls a stub agent, applies four code graders (exact field match, required tool call, forbidden action, regex) and prints pass rates per failure class. The stub has a planted bug. It refunds without checking the 30-day window.

The case schema

Each line of golden_cases.jsonl is one case with seven fields:

  • id: unique and never reused.
  • failure_class: a class name from the taxonomy.
  • source: the redacted trace or ticket the case came from.
  • added: the date the case joined the set.
  • expected: the expected behaviour in one sentence.
  • input: the user’s message plus the frozen tool results.
  • checks: the graders, each with a type and its parameters.
{"id": "refund-001", "failure_class": "refund_policy", "source": "ticket <TICKET_1>", "added": "2026-10-14", "expected": "No refund outside the 30-day window; explain the policy.", "input": {"message": "My annual plan renewed 45 days ago by mistake. Can I get a refund?", "account": {"days_since_charge": 45, "period_end": "2027-08-30"}}, "checks": [{"type": "tool_called", "tool": "lookup_account"}, {"type": "tool_not_called", "tool": "issue_refund"}, {"type": "reply_matches", "pattern": "(?i)\\b30[- ]day"}]}
{"id": "refund-002", "failure_class": "refund_policy", "source": "trace <TRACE_2>", "added": "2026-10-14", "expected": "Refund a duplicate charge inside the window.", "input": {"message": "I was charged twice this month. Please refund the duplicate.", "account": {"days_since_charge": 3, "period_end": "2026-11-12"}}, "checks": [{"type": "field_equals", "field": "intent", "value": "refund_request"}, {"type": "tool_called", "tool": "issue_refund"}]}
{"id": "cancel-001", "failure_class": "cancel_timing", "source": "trace <TRACE_3>", "added": "2026-10-14", "expected": "Schedule cancellation for the period end and state the date.", "input": {"message": "Please cancel at the end of my billing period, not today.", "account": {"days_since_charge": 12, "period_end": "2026-11-30"}}, "checks": [{"type": "field_equals", "field": "intent", "value": "cancel_at_period_end"}, {"type": "tool_not_called", "tool": "cancel_now"}, {"type": "reply_matches", "pattern": "until 2026-11-30\\b"}]}

The runner

"""Run a golden dataset against an agent. Python 3.11+, standard library only."""
import hashlib
import json
import re
import sys
from collections import Counter
from pathlib import Path


def stub_agent(case_input: dict) -> dict:
    """Stand-in for your agent. Replace with a call to the real one.

    It has a planted bug: it refunds without checking the 30-day window.
    """
    message = case_input["message"].lower()
    account = case_input["account"]  # tool results, frozen with the case
    calls = [{"name": "lookup_account"}]
    if "refund" in message:
        calls.append({"name": "issue_refund"})
        return {"fields": {"intent": "refund_request"}, "tool_calls": calls,
                "reply": "Done! Your refund is on its way."}
    if "cancel" in message and "end of" in message:
        calls.append({"name": "schedule_cancellation"})
        return {"fields": {"intent": "cancel_at_period_end"}, "tool_calls": calls,
                "reply": f"Your plan stays active until {account['period_end']}."}
    return {"fields": {"intent": "other"}, "tool_calls": calls,
            "reply": "I'll pass you to a colleague."}


# Each grader returns None on a pass, or the reason for a fail.
def field_equals(out: dict, check: dict) -> str | None:
    got = out["fields"].get(check["field"])
    if got != check["value"]:
        return f"{check['field']} is {got!r}, expected {check['value']!r}"
    return None


def tool_names(out: dict) -> set[str]:
    return {call["name"] for call in out["tool_calls"]}


def tool_called(out: dict, check: dict) -> str | None:
    return None if check["tool"] in tool_names(out) else f"{check['tool']} not called"


def tool_not_called(out: dict, check: dict) -> str | None:
    return f"{check['tool']} was called" if check["tool"] in tool_names(out) else None


def reply_matches(out: dict, check: dict) -> str | None:
    if re.search(check["pattern"], out["reply"]) is None:
        return f"no match for /{check['pattern']}/"
    return None


GRADERS = {"field_equals": field_equals, "tool_called": tool_called,
           "tool_not_called": tool_not_called, "reply_matches": reply_matches}
REQUIRED = {"id", "failure_class", "source", "added", "expected", "input", "checks"}


def load_cases(path: Path) -> list[dict]:
    cases = []
    lines = path.read_text(encoding="utf-8").splitlines()
    for number, line in enumerate(lines, start=1):
        if not line.strip():
            continue
        try:
            case = json.loads(line)
        except json.JSONDecodeError as err:
            sys.exit(f"{path}:{number}: {err}")
        missing = REQUIRED - case.keys()
        unknown = {c.get("type") for c in case.get("checks", [])} - GRADERS.keys()
        if missing or unknown:
            sys.exit(f"{path}:{number}: missing {sorted(missing)}, "
                     f"unknown check types {sorted(map(str, unknown))}")
        cases.append(case)
    return cases


def main(path: Path) -> int:
    cases = load_cases(path)
    digest = hashlib.sha256(path.read_bytes()).hexdigest()[:12]
    print(f"{path.name}  sha256 {digest}  {len(cases)} cases\n")
    passed, total = Counter(), Counter()
    for case in cases:
        out = stub_agent(case["input"])
        reasons = []
        for check in case["checks"]:
            reason = GRADERS[check["type"]](out, check)
            if reason:
                reasons.append(f"{check['type']}: {reason}")
        total[case["failure_class"]] += 1
        if reasons:
            print(f"FAIL {case['id']}  expected: {case['expected']}")
            for reason in reasons:
                print(f"     {reason}")
        else:
            passed[case["failure_class"]] += 1
            print(f"pass {case['id']}")
    print(f"\n{'failure class':<16}{'passed':>8}{'rate':>7}")
    for name, count in total.items():
        fraction = f"{passed[name]}/{count}"
        print(f"{name:<16}{fraction:>8}{passed[name] / count:>7.0%}")
    return 0 if sum(passed.values()) == len(cases) else 1


if __name__ == "__main__":
    sys.exit(main(Path(sys.argv[1] if len(sys.argv) > 1 else "golden_cases.jsonl")))

Output from a real run

Unedited output from 14 October 2026, with the exit status:

$ python3 run_golden.py golden_cases.jsonl; echo "exit status $?"
golden_cases.jsonl  sha256 2b6da0c2d9e9  3 cases

FAIL refund-001  expected: No refund outside the 30-day window; explain the policy.
     tool_not_called: issue_refund was called
     reply_matches: no match for /(?i)\b30[- ]day/
pass refund-002
pass cancel-001

failure class     passed   rate
refund_policy        1/2    50%
cancel_timing        1/1   100%
exit status 1

Exit status 1 fails the CI job, so a failing case can block the merge. To use the runner, replace stub_agent with a call to your agent that returns fields, tool_calls and reply. A calibrated judge becomes one more entry in GRADERS.

Step 4: freeze and version the regression set

A frozen set keeps cases and expected behaviours fixed between the runs you compare:

  • Keep golden_cases.jsonl in the agent’s repository and change it only through reviewed pull requests. One case per line gives one-line diffs.
  • Retire a case instead of editing its expected behaviour. Move it to retired_cases.jsonl with a date and a reason, and add a new case with a new id.
  • Tag each release of the set (golden-v1.4.0) and store the sha256 the runner prints with every result. Compare pass rates only between runs with the same digest.
  • Bump the major version when a definition changes (see criteria drift below).

Hosted tools version datasets too. Langfuse creates a version on “every add, update, delete, or archive of dataset items”, and LangSmith’s dataset docs describe a new version on every add, update or delete, plus tags such as “prod” (vendor sources). Mirror the set into a tool if you use one, and keep the canonical copy in git. OpenAI’s evaluation best practices page, for example, said its Evals platform would become read-only for existing users on 31 October 2026, and it is scheduled to shut down on 30 November 2026.

Re-run the frozen version whenever a provider ships a new model. Our guide to regression testing after model updates covers that run.

Step 5: add a case for every new failure

The set grows one confirmed failure at a time:

  1. When a ticket, an incident or a trace review shows a new failure, write its case before the fix. It should fail on the current build and pass with the fix, which Anthropic’s guide calls a fail-to-pass test.
  2. If the failure fits no class, add a class to the taxonomy with a definition and a threshold.
  3. Read a fresh random sample of traces each month. As Husain wrote in 2024, “You can never stop looking at data.”

The FAQ says such sets often grow “to 100 or more examples”, with coverage deciding the final size.

How many cases to start with

For a starting range, Anthropic’s guide says “20-50 simple tasks drawn from real failures is a great start”, because early changes to an agent have large effects that small samples detect, while mature agents “may need larger, more difficult evals to detect smaller effects”.

Arithmetic shows what each size can resolve. With n cases in a class, one case moves its pass rate by 100/n points. By 95% Wilson score intervals (our calculation), 9 passes out of 10 fits a true rate from 60% to 98%, 45 of 50 a rate from 79% to 96%, and 90 of 100 a rate from 83% to 94%. A 10-case class detects broken behaviour, and telling 92% from 96% needs an interval of about ±2 points, roughly 550 cases by the same method.

Start with every distinct failure your trace reading found, cover each critical class’s variants and should-not counterparts, then grow the classes whose thresholds need finer resolution.

Criteria drift and how to handle it

Shankar and colleagues named the effect in Who Validates the Validators? (arXiv, 18 April 2024), writing that “users need criteria to grade outputs, but grading outputs helps users define criteria.” Some criteria depend on the outputs a grader has seen, so a rubric written before grading changes during grading. In the FAQ’s example, a support agent escalates refunds to a human as policy requires, and only reading the interactions shows that the process frustrates users.

Plan for it:

  1. Write the detailed rubric only after reading traces yourself (the FAQ asks for at least 30 before you review an agent’s suggestions). It is “often better to wait until you’ve reviewed some examples”, the FAQ says.
  2. Keep the rubric in the repository, so a new definition of “good” lands in the same pull request as the relabelled cases.
  3. When a definition changes, relabel the affected cases, bump the major version and re-run the baseline.
  4. Re-test any LLM judge on the new labels, because its true positive and true negative rates were measured on the old ones.
  5. Record who decided and why. For lasting disagreements, the FAQ has a domain expert “make the final decision and record the reason”.

Our view

Start smaller than feels respectable. A first suite of 30 cases, all rebuilt from failures someone actually saw, beats 300 synthetic cases, because every one of the 30 is a failure a customer met and would meet again. Spend the saved time on the graders. A case with a vague grader passes things it shouldn’t, and nobody notices until a customer does.

Keep the cases that must succeed (a valid refund that should go through) next to the cases that must fail. A suite of only “should refuse” cases rewards an agent that refuses everything.

What a golden dataset can’t catch

  • Passing cases bound the failure rate, but they don’t rule failures out. Twenty passing cases with no failures still fit a true failure rate of up to 13.9% (exact binomial, one-sided 95%, our calculation). The bound falls to 5.8% with 50 cases and 3.0% with 100.
  • A hand-picked set’s pass rate says nothing about how often a failure happens in production. The FAQ assigns that to online monitoring of sampled traffic, and gives CI evals the job of protecting “against known regressions”.
  • It catches only failure classes someone has already seen. New ones surface when someone reads new traces.
  • One run per case can pass by chance. In Anthropic’s worked example, an agent with a 75% per-trial success rate passes three trials in a row about 42% of the time, so run critical cases several times and require every run to pass.
  • Frozen tool results test the agent’s decisions and leave the live tools untested. Credentials, scopes and prompt-injection paths need a separate security review, which our MCP server security guide covers.

If you’d rather we did this

Our production readiness review builds this suite from your traces and tickets, with the failure taxonomy on days 1–2, golden cases and graders on days 4–5, and handover in your repository and CI on day 10. It takes ten working days at a fixed price of USD 10,000.

Sources