llm-handover is a free, open-source command-line tool for the weeks before a model you use is retired. It finds every OpenAI, Claude and Gemini model ID in your code, checks each one against the retirement dates on the providers’ own APIs, Amazon Bedrock, Google Cloud and Microsoft Foundry, and replays your own prompts on the old model and its replacement, so you see what changes before you switch. It needs Python 3.10 or newer and nothing else, and it’s licensed under Apache 2.0.
pip install llm-handover
llm-handover scan # find models, check dates
llm-handover run --tests cases.jsonl # test the replacements
What it does
- It finds every model ID in your code. That’s every ID in our dataset of retirement dates, plus anything shaped like an OpenAI, Anthropic or Google model ID, so models with no announced date are listed too.
- It checks each one against retirement dates on the providers’ own APIs, Amazon Bedrock, Google Cloud and Microsoft Foundry. This needs no API keys and no network.
- It replays your prompts on the old model and the new one and shows what changed: answers that are no longer valid JSON, missing keys, dropped tool calls, new refusals, rules your answers used to meet, and answers cut off at the token limit.
It doesn’t guess the replacement. It tests the one the provider names, and when the provider names more than one, or none, it tests nothing for that model until you choose one.
Install
pipx install llm-handover
Or use pip install llm-handover, or run it without installing it with uvx llm-handover. It uses only Python’s standard library, so it installs nothing else. It’s on PyPI as llm-handover, llm-handover --version shows the version, and every command has --help.
Find every LLM model ID in your code
Run llm-handover scan in your project, or give it a folder. This is the scan from the demo:
$ llm-handover scan agent
llm-handover 0.1.0 | dataset 2026-11-18 (bundled) | as of 2026-11-20
Scanned 4 files in agent (listed with git).
Found 3 model ID(s) in 4 place(s): 3 with dataset records, 0 without.
WITHIN 30 DAYS claude-sonnet-4-5-20250929 [fails]
Claude API: retires on 2026-11-30 (10 days left) -> claude-sonnet-5-5
Used in: triage.py:5
WITHIN 30 DAYS gpt-5 [fails]
OpenAI API: retires on 2026-12-11 (21 days left) -> gpt-6.1-sol
Azure OpenAI (Microsoft Foundry) [gpt-5 (version 2025-08-07)]: retires on 2027-02-09 (81 days left)
Used in: answer.py:5, extract.py:8
SCHEDULED gpt-5.1
OpenAI API: retires on 2027-04-01 (132 days left) -> gpt-6-sol
Used in: drafts.py:5
Result: FAILED. 2 model ID(s) at or above --fail-on 30d: claude-sonnet-4-5-20250929, gpt-5
Dates: Volans Aquilae, https://www.volansaquilae.com/guides/llm-model-deprecation-dates/ (CC BY 4.0).
In a git repository it reads the files git lists, so .gitignore is respected. Anywhere else it walks the folder. It skips dependency folders, lock files, minified files, binary files, files over 2 MB and, unless you pass --include-docs, documentation files.
A model ID only matches as a whole token, so gpt-5 doesn’t match inside gpt-5.1 or gpt-5-mini, and Bedrock’s region prefixes, as in us.anthropic.claude-..., are understood. Each place gets a confidence. It’s high when the ID is quoted or assigned to something named like a model setting (model: gpt-5, MODEL=gpt-5, --model gpt-5), and low otherwise, as in a comment. A model mentioned only in comments never fails a scan.
--fail-on decides when the scan exits with code 1: retired, 30d (the default), 90d, deprecated, any or never. --format json, markdown and github give output for scripts, pull request comments and GitHub Actions annotations.
Check deprecation and retirement dates
Every record gets exactly one label, as of today or the date you give with --as-of:
| Label | What it means |
|---|---|
| RETIRED | Retired, or the retirement date is today or earlier |
| WITHIN 30 DAYS | Retires within 30 days |
| WITHIN 90 DAYS | Retires in 31 to 90 days |
| MAY RETIRE ANY TIME | The provider gave only an earliest date, and it has passed |
| DEPRECATED, NO DATE | Deprecated or legacy, with no date yet |
| SCHEDULED | Retires in more than 90 days |
| NO DATE | No retirement date announced |
The dates are the same records as in our guide to LLM model deprecation dates and our model retirement checker: one record per model ID and platform, each with the provider page it was checked against. A copy ships inside llm-handover, so it works offline, and llm-handover update downloads the latest one. To look a model up:
$ llm-handover models gpt-5-mini
2 record(s), dataset 2026-11-18, as of 2026-11-20:
gpt-5-mini-2025-08-07 [openai-api] WITHIN 30 DAYS: retires on 2026-12-11 (21 days left) -> gpt-6.1-terra
gpt-5-mini [openai-api] WITHIN 30 DAYS: retires on 2026-12-11 (21 days left) -> gpt-6.1-terra
Dates: Volans Aquilae, https://www.volansaquilae.com/guides/llm-model-deprecation-dates/ (CC BY 4.0).
--platform limits the check to the platforms you use, with the names llm-handover models shows, such as openai-api, claude-api, gemini-api, bedrock or vertex-ai. Separate them with commas.
Test the replacement on your own prompts
A retirement date tells you when requests will start to fail. It doesn’t tell you whether the replacement behaves the same for your product. OpenAI’s deprecations page says to “run representative requests against both models”, and Anthropic’s says to test your applications with newer models “well before the retirement date”. llm-handover does that with your own prompts.
Write the prompts that matter as a test set: a text file with one JSON object per line. This is the first case from the demo, spread over several lines to read:
{
"id": "extract-001",
"target": "openai:gpt-5",
"messages": [
{
"role": "system",
"content": "Extract the invoice. Reply with JSON: vendor, total, due date."
},
{
"role": "user",
"content": "Northwind Traders, INV-2207, $1,240.50, due 15 January 2027."
}
],
"expect": {
"json": true
}
}
| Field | What it holds |
|---|---|
id |
A name for the case, unique in the file |
messages |
An optional system message, then user and assistant messages taking turns, ending with the user |
target |
The model the case was written for, as provider:model (needed by run) |
tools, tool_choice |
Tools the model may call, each with a JSON Schema |
expect |
Rules every answer must meet: json, contains, not_contains, regex or tool |
baseline |
The old model’s answer, if you saved it, so the old model isn’t called again |
max_tokens, temperature |
Optional request settings |
notes |
Anything you want to remember about the case |
Unknown fields are errors, so a typo can’t quietly switch a rule off, and every problem in the file is reported at once, with its line number, before any call is made. Then compare a new model with an old one:
llm-handover replay cases.jsonl --baseline openai:gpt-5 --candidate openai:gpt-6.1-sol
For each case, llm-handover takes the old model’s answer from the recorded baseline, or calls --baseline for it, asks the new model the same thing, and compares the two. Models are written as provider:model, with openai, anthropic, gemini or openai-compatible as the provider.
What the checks catch
| Check | The case regressed when | Also reported |
|---|---|---|
| Expectations | The old answer met a rule and the new one doesn’t | Rules both answers miss |
| JSON | The old answer was JSON and the new one isn’t, it changed between object and list, or top-level keys disappeared | New top-level keys |
| Tool calls | The new model calls no tool or different tools, drops argument names, or sends arguments that aren’t valid JSON | A tool call where the old model made none, and new argument names |
| Refusals | The new model refuses where the old one answered | |
| Empty answers | The new answer is empty and the old one wasn’t | |
| Cut-off answers | The new answer stopped at max_tokens |
|
| Length | The answer got more than 50% longer or shorter (a rough measure) | |
| Wording | Only noted: a big change in wording (a rough measure) |
A regressed case fails the run. Anything in the right-hand column, apart from wording, marks the case as changed, which fails the run only with --fail-on warning. If a call returns an error after its retries, for example because the new model rejects a setting your request uses, the case is marked failed with the provider’s message, and that fails the run too.
Run all three steps with one command
llm-handover run scans your code, checks the dates, and tests every model that’s at risk, using the replacement its provider names. Each case needs a target, and cases are grouped by it. This is the first run from the demo:
$ llm-handover run --tests cases.jsonl
[1/11] openai:gpt-5 extract-002: regressed
[2/11] openai:gpt-5 extract-001: regressed
[3/11] openai:gpt-5 answer-001: ok
[4/11] openai:gpt-5 answer-002: changed
[5/11] openai:gpt-5 answer-003: ok
[6/11] openai:gpt-5 refund-001: ok
[7/11] anthropic:claude-sonnet-4-5-20250929 triage-003: ok
[8/11] anthropic:claude-sonnet-4-5-20250929 triage-001: ok
[9/11] anthropic:claude-sonnet-4-5-20250929 triage-004: ok
[10/11] anthropic:claude-sonnet-4-5-20250929 triage-002: ok
[11/11] anthropic:claude-sonnet-4-5-20250929 triage-005: ok
openai:gpt-5 -> openai:gpt-6.1-sol: DID NOT PASS
3 ok, 1 changed, 2 regressed, 0 failed, 0 skipped
anthropic:claude-sonnet-4-5-20250929 -> anthropic:claude-sonnet-5-5: passed
5 ok, 0 changed, 0 regressed, 0 failed, 0 skipped
DID NOT PASS. Details: handover-runs/20261120-101502/report.md
Each group gets exactly one outcome:
| Outcome | When |
|---|---|
| Tested | The model is retired or retires within 90 days (change this with --risk), and its provider names one replacement, or you chose one |
| Not at risk | Its retirement is further away than --risk, or it has no date |
| Not in the dataset | We have no record for it on its provider’s own API, or on the platforms you chose for an OpenAI-compatible endpoint, and you didn’t choose a replacement |
| Needs a choice | It’s at risk, but the provider names no single replacement, or it’s called through an OpenAI-compatible endpoint |
A group that needs a choice fails the run until you name the replacement, for example with --candidate anthropic:claude-opus-4-5-20251101=anthropic:claude-opus-5-5. When the replacement is itself at risk, the output says so, so you don’t migrate twice. When the old model is already retired, only the answers recorded in baseline are compared. Models in your code that are at risk but have no test cases are listed, and --fail-on-untested makes them fail the run.
Read the report
Every run writes a folder under handover-runs/ with report.md, summary.json and scan.json, plus one folder per tested model with its own report.md, summary.json and results.jsonl, which holds both answers and every check for each case. The results contain the models’ answers, so treat them like your prompts. This is the top of the report for gpt-5 from the demo’s first run:
# llm-handover replay
- Old model: `openai:gpt-5` (0 recorded, 6 live)
- New model: `openai:gpt-6.1-sol`
- Run: completed, did not pass
- Cases: 6 total, 3 ok, 1 changed, 2 regressed, 0 failed, 0 skipped
## Regressed
| Case | What changed |
| --- | --- |
| `extract-001` | keys missing in the new answer: due_date |
| `extract-002` | keys missing in the new answer: due_date |
## Changed
| Case | What changed |
| --- | --- |
| `answer-002` | the new answer is 216% longer (heuristic) |
In the demo, the fix was to name the keys in the prompt, and in the test cases that copy it. The second run passed, with 5 ok and 1 changed for gpt-5, and the team could switch before the retirement date.
Run it in CI
The scan needs no keys, so it can run on every pull request and once a week, because dates change even when your code doesn’t. In GitHub Actions:
name: Model retirements
on:
pull_request:
schedule:
- cron: "0 7 * * 1"
jobs:
scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
with:
python-version: "3.12"
- run: pip install llm-handover
- run: llm-handover update || echo "Using the bundled dates"
- run: llm-handover scan --format github --fail-on 30d
With --format github, models at or above --fail-on become error annotations, and other models that are retired, retiring within 90 days or deprecated become warnings, each with its file and line. To test replacements in CI as well, run llm-handover run --tests cases.jsonl with your API keys as secrets, and keep the results folder as a build artefact.
Providers and API keys
| Provider | Key | Base address, if you need another |
|---|---|---|
openai |
OPENAI_API_KEY |
OPENAI_BASE_URL |
anthropic |
ANTHROPIC_API_KEY |
ANTHROPIC_BASE_URL |
gemini |
GEMINI_API_KEY or GOOGLE_API_KEY |
GEMINI_BASE_URL |
openai-compatible |
LLM_HANDOVER_COMPAT_API_KEY, if the endpoint needs one |
LLM_HANDOVER_COMPAT_BASE_URL (required) |
The openai-compatible provider covers anything that speaks OpenAI’s Chat Completions format, such as a LiteLLM proxy, OpenRouter or Azure OpenAI’s v1 API. Its key goes in an Authorization header, or in the header you name in LLM_HANDOVER_COMPAT_AUTH_HEADER, such as api-key for Azure. Amazon Bedrock and Google Vertex AI aren’t called directly in this version, so reach them through such a gateway.
Keys are read from environment variables only. They’re never written to any file, and they’re removed from any error message before it’s shown or saved. Addresses must use https, except localhost, and redirects are never followed, so a key can’t be sent on to another address.
Exit codes
| Code | Meaning |
|---|---|
| 0 | Passed |
| 1 | Didn’t pass: a model at or above --fail-on, a regression, a failed case, or a model that needs a choice |
| 2 | Couldn’t run as asked: a bad option, an invalid file, a missing key, or a file that can’t be read or written |
| 3 | An internal error in llm-handover |
| 130 | Stopped with Ctrl-C. The first Ctrl-C lets the cases already running finish and writes the report |
Limits
- The dates cover language models from OpenAI, Anthropic and Google, not their embedding, image or audio models. Other model IDs are found by their shape and listed without a date.
- A model ID built while your program runs, for example from two strings joined together, can’t be found by reading the code.
- Test cases are text conversations with optional tool definitions. Tool results, images, audio and streaming aren’t supported yet.
- The length, wording and refusal-wording checks are rough measures, and the report says so. Read the regressed cases before you decide.
- Calls cost what they always cost. A run makes one call to the new model per case, plus one to the old model for each case without a recorded answer, and repeats a call only after a rate limit, an overload, a timeout or a network error.
Questions and answers
How do I find which deprecated OpenAI, Claude or Gemini models my code uses?
Run llm-handover scan in your repository. It lists every model ID it finds, with the file and line, the retirement date on each platform and the replacement the provider names. It needs no API keys and doesn’t send your code anywhere.
How do I test a new model before the old one is retired?
Write the prompts that matter as a test set, each with the rules a good answer meets, and run llm-handover run --tests cases.jsonl. It sends the same prompts to the old model and the replacement and reports every case that got worse. If the old model has already been retired, it compares against the answers you recorded.
Does llm-handover send my code anywhere?
No. The scan reads your files on your own machine. The only network calls are the model calls that replay and run make with your keys, which send your test cases to your providers or to the addresses you set, and llm-handover update, which downloads the dates file from our site.
Does it work with Azure OpenAI, Amazon Bedrock or Google Vertex AI?
The dates cover all three, and --platform picks the ones you use. For replays, Azure OpenAI works through the openai-compatible provider, and Bedrock and Vertex AI work through a gateway such as a LiteLLM proxy.
How is this different from searching the code for model names?
A search finds strings. llm-handover also knows which IDs are real models, matches them as whole tokens, tells quoted settings from comments, gives each one its dates on every platform, and then tests the replacement on your prompts.
What does it cost?
Nothing. llm-handover is free and open source under the Apache 2.0 licence, and the dates are published under CC BY 4.0. You pay your providers only for the model calls that replays make.
If you’d rather we ran the migration
If you don’t have a test set yet, our production readiness review builds one from your traces in ten working days. Once you have one, our maintenance retainer re-runs it after every model release, reviews production traces monthly and fixes drift, for USD 3,000 a month for one agent or USD 6,000 for up to three agents with weekly runs. Our runbook for LLM regression testing after model updates shows how we decide whether a new model is safe to ship.
Sources
- OpenAI, Deprecations, the migration guidance, accessed 20 November 2026.
- Anthropic, Model deprecations, accessed 20 November 2026.
- Volans Aquilae, LLM model deprecation dates for Claude, OpenAI and Gemini, the dataset llm-handover 0.1.0 ships with, as of 18 November 2026.