AI agent maintenance with regression tests for every model change
When your provider ships a model or sets a deprecation date for yours, we run your evals on both models before you switch, then read your production traces every month and fix the drift. From USD 3,000 a month.
Before a model switch
| Failure class | Pinned model | Replacement | Result |
|---|---|---|---|
| Refund policy | 40 of 40 | 40 of 40 | Same |
| Wrong account | 36 of 36 | 33 of 36 | Regression |
| Tool timeout | 19 of 20 | 20 of 20 | Better |
| Injected instructions | 20 of 20 | 20 of 20 | Same |
Hold the switch. Wrong account fell to 33 of 36 on the replacement. Fix it, re-run, then switch before the retirement date.
Why every model change needs a test run
Anthropic announced Claude Opus 5.5 on 22 September 2026, Claude Sonnet 5.5 on 28 September and Claude Haiku 5.5 on 7 October. That’s three candidate models in 15 days. On 30 September its deprecations page set 30 November 2026 as the retirement date for claude-sonnet-4-5-20250929, and says “Requests to retired models fail.” OpenAI’s deprecations page shuts down gpt-5-2025-08-07 on 11 December 2026.
A new model can also reject requests the old one accepted. Anthropic’s Sonnet 5.5 migration guide says sampling parameters (temperature, top_p and top_k) return a 400 error and must be removed. And an unchanged model ID isn’t a frozen service. Anthropic’s model ID page says changes to request routing, safety classifiers or sampling can cause minor differences in behaviour. So the suite runs on a schedule as well as on every release.
The method we follow is written up in LLM regression testing after model updates: a runbook, and the upcoming retirement dates for Claude, OpenAI and Gemini models are on our model deprecation dates page.
What starts a run
- Your provider releases a model you might move to.
- Your provider sets a retirement date for the model ID you use.
- You change a prompt, a tool or a model setting. The suite runs from your CI on the pull request.
- A scheduled run falls due, monthly for one agent and weekly on the three-agent plan.
What a run includes
- The full suite on your current model and the candidate, in the same window, five samples per case.
- Results per failure class against thresholds you agreed before the run.
- A read of the trace behind every case that got worse.
- Cost per case and p95 latency against your budgets.
- A written recommendation for the switch (go, go with conditions or no-go), with a rollout plan and the rollback model ID’s retirement date.
Trace review, drift fixes and a report every month
- Trace review
We read up to 100 production traces per agent, starting with the ones that carry a negative signal (a thumbs-down, a hand-off to a human, a tool error) and topping up with a random sample. New failures get a name and a count, and become new cases.
- Drift fixes
Up to 16 hours a month on the one-agent plan and 32 on the three-agent plan, spent on new cases for new failures, graders brought back in line with human labels, and prompt or tool fixes. Every change arrives as a pull request you review.
- A short report
It covers the runs done and their results per failure class, the failures found in the traces and the cases added, and changes in cost per conversation and p95 latency. It also lists the retirement dates coming up for the model IDs you use, and the hours used.
AI agent maintenance price and terms
One agent
USD 3,000 per month
- Runs after every relevant release and retirement notice
- A scheduled run every month
- Up to 100 traces read every month
- Up to 16 hours of drift fixes
Up to three agents
USD 6,000 per month
- Runs after every relevant release and retirement notice
- A scheduled run every week, per agent
- Up to 100 traces read per agent every month
- Up to 32 hours of drift fixes
- Maintenance needs a regression suite. If we built yours in a production readiness review, it starts the month after. If you built your own, the first month includes a review of it.
- Invoiced monthly in advance. Cancel with 30 days’ notice.
- Model API costs for the runs are billed to you by your provider, because the suite runs in your CI with your keys.
What maintenance doesn’t cover
It isn’t an incident-response contract. Maintenance carries no response-time guarantee for outages in your systems. New features and new tools are quoted separately, or done at the forward deployed engineer rate. A suite that passes can’t prove that production traffic will. It only shows that the failures it knows about haven’t come back.
Send us the model ID you run and its retirement date
We’ll reply within one working day with what we’d run before you switch.