When a provider ships a new model or sets a retirement date for yours, keep production on a pinned model ID, re-run the full regression suite on both models in the same window with several samples per case, and compare the runs per failure class before you switch. LLM regression testing is re-running a fixed suite of test cases after a change to the model, prompt or configuration, and checking each failure class against a baseline run. This runbook covers the model-update step of the guide to production-ready AI agents. It has a checklist, a Python script that compares two runs with an exact test, and a GitHub Actions workflow that fails the job on a regression.

The runbook checklist

Items 36 to 38 of our AI agent deployment checklist are the short form of these steps, for a launch sign-off.

  1. Record the new model ID, your current ID’s retirement date and the recommended replacement.
  2. Confirm that production sends an exact model ID from one configuration value.
  3. Read the migration guide and fix every request the new model rejects.
  4. Agree a threshold for every failure class before you see results.
  5. Run the full suite on both models in the same window, five samples per case.
  6. Run the current model against itself once to see your noise floor.
  7. Compare lost and gained cases per class, with an exact McNemar test and Wilson intervals.
  8. Read the trace of every lost case.
  9. Check cost per case and p95 latency against written budgets.
  10. Decide on go, go with conditions or no-go.
  11. Roll out in stages, reading production traces at each stage.
  12. Keep the old ID as the rollback until its retirement date.
  13. Record the run, the decision and who made it.

Pinned model versions, aliases and retirement dates

Pin because the same model name can behave differently over time. Chen, Zaharia and Zou compared the March 2023 and June 2023 versions of GPT-4 and GPT-3.5. GPT-4’s accuracy at telling prime from composite numbers fell from 84% to 51%, and both models made more formatting mistakes in generated code in June.

Anthropic

From the Claude 4.6 generation onwards, Anthropic’s model IDs carry no date (claude-sonnet-4-6, claude-opus-5), and each “maps to a single, fixed model snapshot”. Older models have dated IDs such as claude-sonnet-4-5-20250929 and aliases such as claude-sonnet-4-5 that resolve “to the most recent dated snapshot for that minor version”, so send the dated ID. Fixed weights don’t freeze the service. The same page says changes to the request router, safety classifiers or sampling logic occasionally cause minor differences in behaviour on an unchanged ID, so run the suite on a schedule as well as on releases.

Anthropic’s model deprecations page lists four states (Active, Legacy, Deprecated, Retired), promises at least 60 days’ notice before retiring a publicly released model and says “Requests to retired models fail.” On 30 September 2026 it deprecated claude-sonnet-4-5-20250929, retiring it on 30 November 2026 in favour of claude-sonnet-5-5, whose own tentative retirement date is “not sooner than” 28 September 2027. Anthropic announced Claude Opus 5.5 on 22 September 2026, Claude Sonnet 5.5 on 28 September 2026 and Claude Haiku 5.5 on 7 October 2026. That’s three candidates within 15 days.

OpenAI

OpenAI’s GPT-5 page lists the alias gpt-5 beside the dated snapshot gpt-5-2025-08-07 and says “Snapshots let you lock in a specific version of the model so that performance and behavior remain consistent.” The GPT-6.1 Sol page lists one undated ID, gpt-6.1-sol, and doesn’t say whether it can move, so check a model’s snapshot list before treating its ID as pinned.

OpenAI’s deprecations page gives at least 6 months’ notice for generally available models, at least 3 months for specialised variants and much shorter notice for previews (its example is 2 weeks), unless safety or compliance concerns require a faster timeline. Notices dated 11 June 2026 and 14 October 2026 shut down gpt-5-2025-08-07 and o3-2025-04-16 on 11 December 2026, and five GPT-4-series snapshots, gpt-4o-2024-08-06 among them, on 14 January 2027. OpenAI’s hosted Evals platform has been read-only since 31 October 2026 and is scheduled to shut down on 30 November 2026, so keep the suite and its results in your own repository.

Fix request changes before you measure quality

A new model can reject requests the old one accepted. For teams moving from Sonnet 4.5, Anthropic’s Sonnet 5.5 migration guide lists these changes:

  • sampling parameters (temperature, top_p and top_k) return a 400 error, so remove them
  • assistant-message prefill is not supported
  • forced tool use, a tool_choice of type any or tool, is not supported
  • a request with no thinking field runs with adaptive thinking, and thinking tokens are billed as output
  • the same text produces about 30% more tokens
  • the model declines “in more categories than Claude Sonnet 5”, including a general-harms category that benign work can trigger.

Run the suite through your production request code (client, system prompt, tool definitions, parameters), fix the errors first and record each fix as part of the candidate. The guide adds, “There’s no fixed mapping from a budget to an effort level, so run your evaluations at two or three levels.” Each effort level is a separate candidate.

What to re-run

When the model changes, re-run the whole suite, every class, built from real failures as in how to build a golden dataset from production failures. The same comparison covers prompt regression testing, with the changed prompt as the candidate.

Sample each case several times, because one run can pass by luck. In τ-bench, gpt-4o’s pass^1 in the retail domain was about 61%, while its pass^8 (“the chance that all k i.i.d. task trials are successful”, here with k = 8) was below 25%. This runbook uses five samples and counts a case as passed only if all five pass.

Run both models in the same window, from the same test harness commit, with the same graders. A baseline stored months ago mixes the model change with any infrastructure change since. Run the current model against itself once, and the cases that flip between two identical runs are your noise floor. If an LLM judge grades part of the suite, pin its model ID too, and re-check it against human labels when that model is deprecated, using the agreement test in our LLM-as-a-judge calibration guide.

Compare the runs per failure class

Compare per class, because an average hides the class that broke. In the example below, the overall pass rate moves from 90% to 87% while tool_arguments falls from 93% to 75%.

Both runs contain the same cases, so the comparison is paired. Per class, the script counts lost cases (pass on the baseline, fail on the candidate) and gained cases (the reverse) and runs the exact McNemar test, a binomial test of lost against gained with p = 0.5. Wikipedia’s McNemar’s test article notes, “The traditional advice has been to use the exact binomial test when b + c < 25”, b + c being the changed cases. The script prints a 95% Wilson score interval for each pass rate, an interval that “can be safely employed with small samples and skewed observations”, and splits the 5% error rate across classes (a Bonferroni correction), so four classes are each tested at 0.0125.

The script

Save it as evals/compare_runs.py. It needs Python 3.11 or later and only the standard library.

"""Compare two regression runs per failure class. Python 3.11+, standard library only.

Usage: python compare_runs.py BASELINE.jsonl CANDIDATE.jsonl   (or --demo)
Each line: {"id": "refund-001", "failure_class": "refund_policy", "samples": [true, true]}
A case passes only if every sample passed. Exit code 1 means: don't switch.
"""
import json
import math
import sys

THRESHOLDS = {"cancel_timing": 0.85, "output_format": 0.95,  # agreed before the run
              "refund_policy": 1.00, "tool_arguments": 0.90}
ALPHA, Z = 0.05, 1.96  # family-wise alpha, split across classes; 95% Wilson interval


def wilson(k, n):
    centre = (k + Z * Z / 2) / (n + Z * Z)
    half = Z / (n + Z * Z) * math.sqrt(k * (n - k) / n + Z * Z / 4)
    return max(0.0, centre - half), min(1.0, centre + half)


def mcnemar_exact(lost, gained):  # two-sided binomial test on the cases that changed
    n = lost + gained
    return min(1.0, 2 * sum(math.comb(n, i) for i in range(min(lost, gained) + 1)) / 2**n)


def load(path):
    run = {}
    with open(path, encoding="utf-8") as f:
        for row in map(json.loads, filter(str.strip, f)):
            if row["id"] in run:
                sys.exit(f"duplicate case {row['id']} in {path}")
            samples = row["samples"]  # an empty list counts as a fail
            run[row["id"]] = (row["failure_class"], len(samples) > 0 and all(samples))
    return run


def rate(k, n):
    low, high = wilson(k, n)
    return f"{k / n:.0%} ({100 * low:.0f}-{100 * high:.0f})"


def compare(base, cand):
    if base.keys() != cand.keys() or any(base[i][0] != cand[i][0] for i in base):
        sys.exit("both runs must contain the same cases in the same classes")
    classes = sorted({cls for cls, _ in base.values()})
    if missing := set(classes) - THRESHOLDS.keys():
        sys.exit(f"no threshold agreed for {sorted(missing)}")
    alpha, blocked = ALPHA / len(classes), False  # Bonferroni
    print(f"{'class':<20}{'cases':>5}  {'baseline':<14}{'candidate':<14}"
          f"{'lost':>4}{'gained':>7}{'p':>8}  result")
    for cls in classes:
        ids = [i for i in base if base[i][0] == cls]
        n, b, c = len(ids), sum(base[i][1] for i in ids), sum(cand[i][1] for i in ids)
        lost = sum(base[i][1] and not cand[i][1] for i in ids)
        gained = sum(cand[i][1] and not base[i][1] for i in ids)
        p, below = mcnemar_exact(lost, gained), c / n < THRESHOLDS[cls]
        result = ["regression" if lost > gained else "improvement"] if p < alpha else []
        result += [f"below {THRESHOLDS[cls]:.0%}"] if below else []
        blocked |= below or "regression" in result
        print(f"{cls:<20}{n:>5}  {rate(b, n):<14}{rate(c, n):<14}{lost:>4}{gained:>7}"
              f"{p:>8.4f}  {', '.join(result) or 'no change detected'}")
    print(f"alpha per class: {alpha:.4f}; decision: {'block' if blocked else 'pass'}")
    return blocked


def demo():  # made-up counts per class: (both pass, lost, gained, both fail)
    counts = {"cancel_timing": (33, 3, 2, 2), "output_format": (40, 0, 8, 2),
              "refund_policy": (24, 1, 0, 0), "tool_arguments": (44, 12, 1, 3)}
    base, cand = {}, {}
    for cls, groups in counts.items():
        pairs = [(True, True), (True, False), (False, True), (False, False)]
        cases = [pair for pair, count in zip(pairs, groups) for _ in range(count)]
        for i, (b, c) in enumerate(cases):
            base[f"{cls}-{i}"], cand[f"{cls}-{i}"] = (cls, b), (cls, c)
    return base, cand


if __name__ == "__main__":
    if sys.argv[1:] == ["--demo"]:
        runs = demo()
    elif len(sys.argv) == 3:
        runs = load(sys.argv[1]), load(sys.argv[2])
    else:
        sys.exit(__doc__)
    sys.exit(1 if compare(*runs) else 0)

Example output (made-up numbers)

With --demo, the script builds two runs from the made-up counts in demo(), so the numbers come from no agent or model. This is the real output, identical on Python 3.11.17, 3.12.3 and 3.13.16:

$ python3 compare_runs.py --demo; echo "exit status $?"
class               cases  baseline      candidate     lost gained       p  result
cancel_timing          40  90% (77-96)   88% (74-95)      3      2  1.0000  no change detected
output_format          50  80% (67-89)   96% (87-99)      0      8  0.0078  improvement
refund_policy          25  100% (87-100) 96% (80-99)      1      0  1.0000  below 100%
tool_arguments         60  93% (84-97)   75% (63-84)     12      1  0.0034  regression, below 90%
alpha per class: 0.0125; decision: block
exit status 1

Reading it class by class:

  • tool_arguments: 12 lost, 1 gained, p = 0.0034. A regression, and below its 90% threshold.
  • output_format: 8 gained, none lost, p = 0.0078. An improvement.
  • refund_policy: one lost case can’t reach significance (p = 1.0), but the class needs 100%, so the threshold blocks the switch.
  • cancel_timing: 3 lost, 2 gained. “No change detected” means the suite can’t see a change at this size, but a smaller one can still exist.

Exit status 1 fails a CI job. Input errors (a missing threshold, mismatched cases, a duplicate ID) also exit with 1, so a broken run can’t pass.

Thresholds and a decision rule

Agree a threshold for each class before the run, write it into THRESHOLDS in the repository and record who agreed it. A class where one failure moves money or changes customer data gets 100%, as refund_policy does here. Then decide with the verdict levels the pillar guide defines:

  • No-go: any class shows a significant regression or sits below its threshold. Production stays on the pinned ID.
  • Go with conditions: no blockers, but a lost case whose trace nobody can explain yet, or cost or latency over budget. Each condition gets an owner and a date, for example a pilot group only until a named review.
  • Go: no blockers, every lost case read and explained, and cost and latency within budget.

A baseline below threshold is a finding in its own right. In the example, the current model misses the 95% bar for output_format.

Cost and latency checks

Record input tokens, output tokens (thinking included) and latency for every sample, and compare cost per case and p50 and p95 latency per class. List prices alone mislead. Moving from Sonnet 4.5 to Sonnet 5.5 cuts Anthropic’s prices from $3 to $2 per million input tokens and from $15 to $10 per million output tokens, but the migration guide says the same text produces about 30% more tokens, and Anthropic’s Sonnet 5.5 announcement says that, compared with Sonnet 5, it “typically needs far fewer tokens to do the same work”. Only a run on your suite shows the net effect.

Write budgets next to the thresholds, for example cost per case at most 10% and p95 latency at most 20% above the baseline (illustrative numbers, so set your own), and measure latency on the API path, region and concurrency you use in production. Scheduled quality runs can use batch pricing, which tells you nothing about latency. Anthropic’s Batch API gives “a 50% discount on both input and output tokens”, and OpenAI lists Batch at 50% below Standard for gpt-6.1-sol.

Staged rollout, and rollback by pinning

Keep the model ID in one configuration value you can change without a deploy, and perform one rollback in staging before the rollout. Move traffic in stages, for example internal users, then 5%, 25% and 100%, and at each stage read production traces for the classes that lost cases.

Rollback means setting that value back to the old pinned ID, and it expires. On the Claude API, requests to claude-sonnet-4-5-20250929 fail from 30 November 2026, and Google Cloud shuts it down the same day (only Amazon Bedrock runs it later, to 8 April 2027), and on the OpenAI API requests to gpt-5-2025-08-07 fail from 11 December 2026. Plan the stages so the last one ends before the old ID’s retirement date, with time left to roll back.

Run the suite in GitHub Actions

The workflow runs on demand with a candidate ID typed into the run form, and every Monday with the candidate in the repository variable CANDIDATE_MODEL, while BASELINE_MODEL holds the pinned production ID. evals/run_suite.py and evals/requirements.txt stand in for your test harness, which must write one JSON line per case in the format the script reads. actions/checkout v7.0.1 and actions/setup-python v7.0.0 were the latest releases on 14 October 2026.

name: model-regression

on:
  workflow_dispatch:
    inputs:
      candidate_model:
        description: "Model ID to test against the pinned baseline"
        required: true
        type: string
  schedule:
    - cron: "17 2 * * 1" # 02:17 UTC every Monday

permissions:
  contents: read

jobs:
  compare:
    runs-on: ubuntu-latest
    timeout-minutes: 90
    env:
      BASELINE_MODEL: ${{ vars.BASELINE_MODEL }}
      CANDIDATE_MODEL: ${{ inputs.candidate_model || vars.CANDIDATE_MODEL }}
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.13"
      - run: test -n "$BASELINE_MODEL" && test -n "$CANDIDATE_MODEL"
      - run: pip install -r evals/requirements.txt
      - name: Run the suite on both models
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} # or your provider's API key
        run: |
          mkdir -p results
          python evals/run_suite.py --model "$BASELINE_MODEL" --samples 5 --out results/baseline.jsonl
          python evals/run_suite.py --model "$CANDIDATE_MODEL" --samples 5 --out results/candidate.jsonl
      - name: Compare per failure class
        run: python evals/compare_runs.py results/baseline.jsonl results/candidate.jsonl
      - name: Keep the raw results
        if: ${{ always() }}
        uses: actions/upload-artifact@v7
        with:
          name: regression-${{ github.run_id }}
          path: results/

A scheduled run has no inputs (the inputs context is filled only for workflow_dispatch and reusable workflows), so the expression falls back to the variable. Per GitHub’s events reference, scheduled runs use UTC unless you set a time zone, run on the default branch’s latest commit, “can be delayed during periods of high loads” and are disabled in a public repository after 60 days without activity. Artifacts are kept for 90 days by default (upload-artifact README). Between releases, set CANDIDATE_MODEL to the production ID, and the weekly run checks that the pinned model still meets every threshold. To see drift against the last accepted run, pass that run’s results file to compare_runs.py as the baseline.

What to record

Store these with every run:

  • the model IDs exactly as sent, and the platform, because IDs differ (Sonnet 4.5 is anthropic.claude-sonnet-4-5-20250929-v1:0 on Amazon Bedrock)
  • the date, the time and the commit SHA of prompts, tool definitions, test harness, graders and thresholds
  • request parameters such as effort, thinking settings and max_tokens
  • samples per case, raw outputs and every grader verdict
  • tokens, cost and latency per sample
  • the comparison report, the decision, who made it, and any conditions with owners and dates
  • the rollout stages with dates, and the rollback ID with its retirement date.

Our view

Treat a retirement date as a deadline for the regression run, not for the switch. Run the suite on the replacement as soon as the provider names it, because the run is cheap and the fixes it finds aren’t. The model deprecation dates page keeps the dates for Claude, OpenAI and Gemini in one place.

And never let an alias decide your model for you in production. Pin the dated ID, so the day the model changes is a day you chose.

What a regression run can’t prove

  • The suite tests only the failure classes you have already seen, so a new kind of failure passes it until someone finds one in a trace.
  • Small classes hide real changes. With four classes, a class needs at least 8 lost cases and none gained before the test flags it, and a 40-case class at 90% has a 95% interval of 77% to 96%. Thresholds and trace reading cover what the test can’t see.
  • The exact test is conservative. The McNemar’s test article reports simulations that found it “overly conservative” and describes a mid-P version with higher statistical power. This runbook accepts missed small drops in exchange for few false alarms.
  • Infrastructure changes can alter a pinned model’s behaviour, so last month’s pass doesn’t carry over.
  • The suite’s mix of classes differs from your traffic, so its cost per case is not your cost per conversation.

If you’d rather we did this

If you don’t have a suite yet, our production readiness review builds one from your traces in ten working days. Once you have one, our maintenance retainer re-runs it after every model release, reviews production traces monthly and fixes drift, for USD 3,000 a month for one agent or USD 6,000 for up to three agents with weekly runs.

Sources