llm-handover is a free, open-source command-line tool for the weeks before a model you use is retired. It finds every OpenAI, Claude and Gemini model ID in your code, checks each one against the retirement dates on the providers’ own APIs, Amazon Bedrock, Google Cloud and Microsoft Foundry, and replays your own prompts on the old model and its replacement, so you see what changes before you switch. It needs Python 3.10 or newer and nothing else, and it’s licensed under Apache 2.0.

pip install llm-handover
llm-handover scan                     # find models, check dates
llm-handover run --tests cases.jsonl  # test the replacements

What it does

  1. It finds every model ID in your code. That’s every ID in our dataset of retirement dates, plus anything shaped like an OpenAI, Anthropic or Google model ID, so models with no announced date are listed too.
  2. It checks each one against retirement dates on the providers’ own APIs, Amazon Bedrock, Google Cloud and Microsoft Foundry. This needs no API keys and no network.
  3. It replays your prompts on the old model and the new one and shows what changed: answers that are no longer valid JSON, missing keys, dropped tool calls, new refusals, rules your answers used to meet, and answers cut off at the token limit.

It doesn’t guess the replacement. It tests the one the provider names, and when the provider names more than one, or none, it tests nothing for that model until you choose one.

Install

pipx install llm-handover

Or use pip install llm-handover, or run it without installing it with uvx llm-handover. It uses only Python’s standard library, so it installs nothing else. It’s on PyPI as llm-handover, llm-handover --version shows the version, and every command has --help.

Find every LLM model ID in your code

Run llm-handover scan in your project, or give it a folder. This is the scan from the demo:

$ llm-handover scan agent
llm-handover 0.1.0 | dataset 2026-11-18 (bundled) | as of 2026-11-20
Scanned 4 files in agent (listed with git).
Found 3 model ID(s) in 4 place(s): 3 with dataset records, 0 without.

WITHIN 30 DAYS  claude-sonnet-4-5-20250929  [fails]
  Claude API: retires on 2026-11-30 (10 days left) -> claude-sonnet-5-5
  Used in: triage.py:5

WITHIN 30 DAYS  gpt-5  [fails]
  OpenAI API: retires on 2026-12-11 (21 days left) -> gpt-6.1-sol
  Azure OpenAI (Microsoft Foundry) [gpt-5 (version 2025-08-07)]: retires on 2027-02-09 (81 days left)
  Used in: answer.py:5, extract.py:8

SCHEDULED  gpt-5.1
  OpenAI API: retires on 2027-04-01 (132 days left) -> gpt-6-sol
  Used in: drafts.py:5

Result: FAILED. 2 model ID(s) at or above --fail-on 30d: claude-sonnet-4-5-20250929, gpt-5
Dates: Volans Aquilae, https://www.volansaquilae.com/guides/llm-model-deprecation-dates/ (CC BY 4.0).

In a git repository it reads the files git lists, so .gitignore is respected. Anywhere else it walks the folder. It skips dependency folders, lock files, minified files, binary files, files over 2 MB and, unless you pass --include-docs, documentation files.

A model ID only matches as a whole token, so gpt-5 doesn’t match inside gpt-5.1 or gpt-5-mini, and Bedrock’s region prefixes, as in us.anthropic.claude-..., are understood. Each place gets a confidence. It’s high when the ID is quoted or assigned to something named like a model setting (model: gpt-5, MODEL=gpt-5, --model gpt-5), and low otherwise, as in a comment. A model mentioned only in comments never fails a scan.

--fail-on decides when the scan exits with code 1: retired, 30d (the default), 90d, deprecated, any or never. --format json, markdown and github give output for scripts, pull request comments and GitHub Actions annotations.

Check deprecation and retirement dates

Every record gets exactly one label, as of today or the date you give with --as-of:

Label What it means
RETIRED Retired, or the retirement date is today or earlier
WITHIN 30 DAYS Retires within 30 days
WITHIN 90 DAYS Retires in 31 to 90 days
MAY RETIRE ANY TIME The provider gave only an earliest date, and it has passed
DEPRECATED, NO DATE Deprecated or legacy, with no date yet
SCHEDULED Retires in more than 90 days
NO DATE No retirement date announced

The dates are the same records as in our guide to LLM model deprecation dates and our model retirement checker: one record per model ID and platform, each with the provider page it was checked against. A copy ships inside llm-handover, so it works offline, and llm-handover update downloads the latest one. To look a model up:

$ llm-handover models gpt-5-mini
2 record(s), dataset 2026-11-18, as of 2026-11-20:
  gpt-5-mini-2025-08-07  [openai-api]  WITHIN 30 DAYS: retires on 2026-12-11 (21 days left) -> gpt-6.1-terra
  gpt-5-mini  [openai-api]  WITHIN 30 DAYS: retires on 2026-12-11 (21 days left) -> gpt-6.1-terra
Dates: Volans Aquilae, https://www.volansaquilae.com/guides/llm-model-deprecation-dates/ (CC BY 4.0).

--platform limits the check to the platforms you use, with the names llm-handover models shows, such as openai-api, claude-api, gemini-api, bedrock or vertex-ai. Separate them with commas.

Test the replacement on your own prompts

A retirement date tells you when requests will start to fail. It doesn’t tell you whether the replacement behaves the same for your product. OpenAI’s deprecations page says to “run representative requests against both models”, and Anthropic’s says to test your applications with newer models “well before the retirement date”. llm-handover does that with your own prompts.

Write the prompts that matter as a test set: a text file with one JSON object per line. This is the first case from the demo, spread over several lines to read:

{
    "id": "extract-001",
    "target": "openai:gpt-5",
    "messages": [
        {
            "role": "system",
            "content": "Extract the invoice. Reply with JSON: vendor, total, due date."
        },
        {
            "role": "user",
            "content": "Northwind Traders, INV-2207, $1,240.50, due 15 January 2027."
        }
    ],
    "expect": {
        "json": true
    }
}
Field What it holds
id A name for the case, unique in the file
messages An optional system message, then user and assistant messages taking turns, ending with the user
target The model the case was written for, as provider:model (needed by run)
tools, tool_choice Tools the model may call, each with a JSON Schema
expect Rules every answer must meet: json, contains, not_contains, regex or tool
baseline The old model’s answer, if you saved it, so the old model isn’t called again
max_tokens, temperature Optional request settings
notes Anything you want to remember about the case

Unknown fields are errors, so a typo can’t quietly switch a rule off, and every problem in the file is reported at once, with its line number, before any call is made. Then compare a new model with an old one:

llm-handover replay cases.jsonl --baseline openai:gpt-5 --candidate openai:gpt-6.1-sol

For each case, llm-handover takes the old model’s answer from the recorded baseline, or calls --baseline for it, asks the new model the same thing, and compares the two. Models are written as provider:model, with openai, anthropic, gemini or openai-compatible as the provider.

What the checks catch

Check The case regressed when Also reported
Expectations The old answer met a rule and the new one doesn’t Rules both answers miss
JSON The old answer was JSON and the new one isn’t, it changed between object and list, or top-level keys disappeared New top-level keys
Tool calls The new model calls no tool or different tools, drops argument names, or sends arguments that aren’t valid JSON A tool call where the old model made none, and new argument names
Refusals The new model refuses where the old one answered
Empty answers The new answer is empty and the old one wasn’t
Cut-off answers The new answer stopped at max_tokens
Length The answer got more than 50% longer or shorter (a rough measure)
Wording Only noted: a big change in wording (a rough measure)

A regressed case fails the run. Anything in the right-hand column, apart from wording, marks the case as changed, which fails the run only with --fail-on warning. If a call returns an error after its retries, for example because the new model rejects a setting your request uses, the case is marked failed with the provider’s message, and that fails the run too.

Run all three steps with one command

llm-handover run scans your code, checks the dates, and tests every model that’s at risk, using the replacement its provider names. Each case needs a target, and cases are grouped by it. This is the first run from the demo:

$ llm-handover run --tests cases.jsonl
[1/11] openai:gpt-5 extract-002: regressed
[2/11] openai:gpt-5 extract-001: regressed
[3/11] openai:gpt-5 answer-001: ok
[4/11] openai:gpt-5 answer-002: changed
[5/11] openai:gpt-5 answer-003: ok
[6/11] openai:gpt-5 refund-001: ok
[7/11] anthropic:claude-sonnet-4-5-20250929 triage-003: ok
[8/11] anthropic:claude-sonnet-4-5-20250929 triage-001: ok
[9/11] anthropic:claude-sonnet-4-5-20250929 triage-004: ok
[10/11] anthropic:claude-sonnet-4-5-20250929 triage-002: ok
[11/11] anthropic:claude-sonnet-4-5-20250929 triage-005: ok
openai:gpt-5 -> openai:gpt-6.1-sol: DID NOT PASS
  3 ok, 1 changed, 2 regressed, 0 failed, 0 skipped
anthropic:claude-sonnet-4-5-20250929 -> anthropic:claude-sonnet-5-5: passed
  5 ok, 0 changed, 0 regressed, 0 failed, 0 skipped
DID NOT PASS. Details: handover-runs/20261120-101502/report.md

Each group gets exactly one outcome:

Outcome When
Tested The model is retired or retires within 90 days (change this with --risk), and its provider names one replacement, or you chose one
Not at risk Its retirement is further away than --risk, or it has no date
Not in the dataset We have no record for it on its provider’s own API, or on the platforms you chose for an OpenAI-compatible endpoint, and you didn’t choose a replacement
Needs a choice It’s at risk, but the provider names no single replacement, or it’s called through an OpenAI-compatible endpoint

A group that needs a choice fails the run until you name the replacement, for example with --candidate anthropic:claude-opus-4-5-20251101=anthropic:claude-opus-5-5. When the replacement is itself at risk, the output says so, so you don’t migrate twice. When the old model is already retired, only the answers recorded in baseline are compared. Models in your code that are at risk but have no test cases are listed, and --fail-on-untested makes them fail the run.

Read the report

Every run writes a folder under handover-runs/ with report.md, summary.json and scan.json, plus one folder per tested model with its own report.md, summary.json and results.jsonl, which holds both answers and every check for each case. The results contain the models’ answers, so treat them like your prompts. This is the top of the report for gpt-5 from the demo’s first run:

# llm-handover replay

- Old model: `openai:gpt-5` (0 recorded, 6 live)
- New model: `openai:gpt-6.1-sol`
- Run: completed, did not pass
- Cases: 6 total, 3 ok, 1 changed, 2 regressed, 0 failed, 0 skipped

## Regressed

| Case | What changed |
| --- | --- |
| `extract-001` | keys missing in the new answer: due_date |
| `extract-002` | keys missing in the new answer: due_date |

## Changed

| Case | What changed |
| --- | --- |
| `answer-002` | the new answer is 216% longer (heuristic) |

In the demo, the fix was to name the keys in the prompt, and in the test cases that copy it. The second run passed, with 5 ok and 1 changed for gpt-5, and the team could switch before the retirement date.

Run it in CI

The scan needs no keys, so it can run on every pull request and once a week, because dates change even when your code doesn’t. In GitHub Actions:

name: Model retirements
on:
  pull_request:
  schedule:
    - cron: "0 7 * * 1"
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
      - run: pip install llm-handover
      - run: llm-handover update || echo "Using the bundled dates"
      - run: llm-handover scan --format github --fail-on 30d

With --format github, models at or above --fail-on become error annotations, and other models that are retired, retiring within 90 days or deprecated become warnings, each with its file and line. To test replacements in CI as well, run llm-handover run --tests cases.jsonl with your API keys as secrets, and keep the results folder as a build artefact.

Providers and API keys

Provider Key Base address, if you need another
openai OPENAI_API_KEY OPENAI_BASE_URL
anthropic ANTHROPIC_API_KEY ANTHROPIC_BASE_URL
gemini GEMINI_API_KEY or GOOGLE_API_KEY GEMINI_BASE_URL
openai-compatible LLM_HANDOVER_COMPAT_API_KEY, if the endpoint needs one LLM_HANDOVER_COMPAT_BASE_URL (required)

The openai-compatible provider covers anything that speaks OpenAI’s Chat Completions format, such as a LiteLLM proxy, OpenRouter or Azure OpenAI’s v1 API. Its key goes in an Authorization header, or in the header you name in LLM_HANDOVER_COMPAT_AUTH_HEADER, such as api-key for Azure. Amazon Bedrock and Google Vertex AI aren’t called directly in this version, so reach them through such a gateway.

Keys are read from environment variables only. They’re never written to any file, and they’re removed from any error message before it’s shown or saved. Addresses must use https, except localhost, and redirects are never followed, so a key can’t be sent on to another address.

Exit codes

Code Meaning
0 Passed
1 Didn’t pass: a model at or above --fail-on, a regression, a failed case, or a model that needs a choice
2 Couldn’t run as asked: a bad option, an invalid file, a missing key, or a file that can’t be read or written
3 An internal error in llm-handover
130 Stopped with Ctrl-C. The first Ctrl-C lets the cases already running finish and writes the report

Limits

  • The dates cover language models from OpenAI, Anthropic and Google, not their embedding, image or audio models. Other model IDs are found by their shape and listed without a date.
  • A model ID built while your program runs, for example from two strings joined together, can’t be found by reading the code.
  • Test cases are text conversations with optional tool definitions. Tool results, images, audio and streaming aren’t supported yet.
  • The length, wording and refusal-wording checks are rough measures, and the report says so. Read the regressed cases before you decide.
  • Calls cost what they always cost. A run makes one call to the new model per case, plus one to the old model for each case without a recorded answer, and repeats a call only after a rate limit, an overload, a timeout or a network error.

Questions and answers

How do I find which deprecated OpenAI, Claude or Gemini models my code uses?

Run llm-handover scan in your repository. It lists every model ID it finds, with the file and line, the retirement date on each platform and the replacement the provider names. It needs no API keys and doesn’t send your code anywhere.

How do I test a new model before the old one is retired?

Write the prompts that matter as a test set, each with the rules a good answer meets, and run llm-handover run --tests cases.jsonl. It sends the same prompts to the old model and the replacement and reports every case that got worse. If the old model has already been retired, it compares against the answers you recorded.

Does llm-handover send my code anywhere?

No. The scan reads your files on your own machine. The only network calls are the model calls that replay and run make with your keys, which send your test cases to your providers or to the addresses you set, and llm-handover update, which downloads the dates file from our site.

Does it work with Azure OpenAI, Amazon Bedrock or Google Vertex AI?

The dates cover all three, and --platform picks the ones you use. For replays, Azure OpenAI works through the openai-compatible provider, and Bedrock and Vertex AI work through a gateway such as a LiteLLM proxy.

How is this different from searching the code for model names?

A search finds strings. llm-handover also knows which IDs are real models, matches them as whole tokens, tells quoted settings from comments, gives each one its dates on every platform, and then tests the replacement on your prompts.

What does it cost?

Nothing. llm-handover is free and open source under the Apache 2.0 licence, and the dates are published under CC BY 4.0. You pay your providers only for the model calls that replays make.

If you’d rather we ran the migration

If you don’t have a test set yet, our production readiness review builds one from your traces in ten working days. Once you have one, our maintenance retainer re-runs it after every model release, reviews production traces monthly and fixes drift, for USD 3,000 a month for one agent or USD 6,000 for up to three agents with weekly runs. Our runbook for LLM regression testing after model updates shows how we decide whether a new model is safe to ship.

Sources