A penetration test is an authorised attempt to break into scoped systems with an attacker’s tools, to show which technical vulnerabilities can be exploited. AI red teaming is adversarial testing of an AI system’s behaviour, pushing a model or agent towards harmful, false or unintended outputs and actions. A production readiness review decides whether one agent, with its current model, prompts and tools, can take customer traffic, and ends in a written go/no-go verdict. In short, if the agent can change customer data or move money, we’d run the readiness review first and a targeted red team second. We’d also run the penetration test that enterprise security reviews ask for, and add a broad red team when the agent talks to the public about open-ended topics. This comparison extends our guide to production readiness for AI agents.

Definitions from NIST, OWASP, Microsoft and the NCSC

AI red teaming

NIST’s security glossary defines AI red-teaming, citing NIST SP 800-218A (July 2024), as “a structured testing effort to find flaws and vulnerabilities in an AI system”. NIST AI 600-1, the Generative AI Profile of the AI Risk Management Framework (July 2024), describes probing for inaccurate, harmful or discriminatory outputs and names four kinds of red team: the general public, domain experts, a mix of the two, and generative AI working alongside human teams.

The OWASP Gen AI Security Project’s GenAI Red Teaming Guide (version 1.0, 23 January 2025) combines human experts with automated tools to find safety, security, trust and performance gaps in the model and the application around it, with an appendix of 12 task areas for agents. Microsoft’s guide to planning LLM red teaming (updated 13 May 2026) says the term has grown from security attacks to many kinds of probing of AI systems, and tests at the base model, the application or both.

Penetration testing

The NCSC’s penetration testing guidance (published 8 August 2017, reviewed 10 January 2022) defines a penetration test as a way to gain assurance in an IT system’s security by trying to breach it with the tools and techniques an adversary would use, and warns against treating it as the main way to find vulnerabilities. In NIST SP 800-115 (September 2008), “assessors mimic real-world attacks” in four phases: planning, discovery, attack and reporting.

LLM penetration testing

An LLM penetration test applies that method to an application built on a large language model, treating prompts and the content the model reads as attack surfaces. The nearest public method is the OWASP AI Testing Guide (version 1, 26 November 2025), whose tests include indirect prompt injection (AITG-APP-02) and agentic behaviour limits (AITG-APP-06). The labels overlap. On the UK’s Digital Marketplace, Dionach’s AI Penetration Testing and Razorthorn’s AI red teaming service both describe simulated attacks on AI systems, so read the statement of work to learn the scope.

AI red teaming vs penetration testing, side by side

Aspect Penetration test AI red teaming Production readiness review (ours)
Goal Show which technical vulnerabilities an attacker can exploit Find ways to make the model or agent misbehave Decide whether one agent can take customer traffic
Scope Named hosts, applications, APIs and networks The base model, the application, or both One agent, with its traces, up to 10 tools and MCP (Model Context Protocol) servers, and its operations
Who does it Qualified testers, often a third party (for UK government, the NCSC’s CHECK scheme) Security testers, domain experts and, in NIST’s list, the general public Our engineers, and the founder signs the verdict
What it finds Exploitable flaws, and chains of them Jailbreaks, prompt injection, data leakage, harmful or false outputs, new kinds of harm Failure rates per failure class, over-broad credentials, token passthrough, injection paths to writes, missing kill switches
What it misses Agent behaviour, such as injected content, tool misuse and wrong answers Failure rates on ordinary traffic, regressions after model changes, the launch decision Systems the agent can’t reach, open-ended harms outside its tools, failures absent from the traces and scripted attacks
Method Planning, discovery, attack, reporting (SP 800-115) Threat modelling, adversarial scenarios, manual and automated attacks, reporting (OWASP) Failure taxonomy from traces, golden cases with code graders, baseline run, tool and MCP review, operational checks
Typical length Quoted in tester days, with no standard length One exercise or a continuing programme Ten working days
Deliverable A report of each issue, its risk and a fix (NCSC) Findings, with evidence of each successful attack An eval suite in your CI, an access page per tool, a written verdict
Standards used SP 800-115, OWASP’s Web Security Testing Guide 4.2 and, for LLM applications, the OWASP AI Testing Guide OWASP’s GenAI Red Teaming Guide, Top 10 for LLM Applications 2026 and Top 10 for Agentic Applications 2026, plus MITRE’s ATLAS and NIST AI 600-1 MCP Security Best Practices (2026-07-28) and OWASP’s LLM and agentic lists, with the edition next to each ID
How often Often yearly, and SP 800-115 says annual testing may be enough for networks Repeatedly (Microsoft’s AI Red Team calls it “a continuous process”) Each launch or new enterprise customer, and the suite then runs in CI on every prompt, tool or model change
Published price £900 to £1,150 a day for Razorthorn’s infrastructure penetration testing on G-Cloud 15 (pricing document) £1,000 to £1,350 a day for Razorthorn’s AI red teaming assessment on G-Cloud 15 (pricing document) USD 10,000 fixed

Both day rates come from one UK supplier’s G-Cloud 15 pricing document, so they compare like with like, but totals depend on how many days a scope needs. The LLM list IDs below come from the 2026 edition of the OWASP Top 10 for LLM Applications, posted on 3 August 2026, which moved Excessive Agency from LLM06 to LLM03. MITRE calls ATLAS “a public knowledge base of adversary TTPs targeting AI systems” (tactics, techniques and procedures).

What a penetration test misses in an AI agent

A penetration test scoped like a web application test sees the agent’s chat endpoint as one more input. Five kinds of failure sit outside that view.

Prompt injection through content

LLM01:2026 Prompt Injection (LLM01 in the 2025 edition too) counts retrieved content, tool output, images, audio and memory as routes in, so instructions can hide in an email, a database row or an MCP server’s output that the user never sees. The NCSC’s post Prompt injection is not SQL injection (8 December 2025) explains why the SQL injection fix doesn’t carry over. An LLM makes no distinction between data and instructions, so nothing plays the part of a parameterised query. Testing the chat window covers the direct route only, and a fixed list of known jailbreaks understates the risk. The 2026 entry cites a 2025 study in which adaptive attacks beat most of 12 recent defences more than 90% of the time, while static attacks almost never did.

Tool misuse inside granted permissions

In the OWASP Top 10 for Agentic Applications for 2026 (9 December 2025), ASI02 Tool Misuse and Exploitation covers an agent that stays within its permissions but “applies a legitimate tool in an unsafe or unintended way”, such as deleting valuable data. Every call passes the authorisation check, so an authorisation test passes too. Our guide to the OWASP agentic Top 10 gives a pre-launch test for ASI02 and the other nine entries.

Excessive agency

LLM03:2026 Excessive Agency (LLM06:2025) traces damaging actions to excessive functionality, permissions or autonomy, such as a document reader that can also delete documents. Finding it takes a review of each task against the agent’s tool list.

The confused deputy, in three forms

The MCP Security Best Practices (revision 2026-07-28) describe the OAuth form. It needs an MCP proxy server with one static client ID towards a third-party API, dynamic client registration, a third-party consent cookie and no per-client consent. A tester who knows the spec can find it, and our MCP server security guide has the test. The NCSC’s post describes the model form, an “inherently confusable deputy” that acts with its own access on instructions from whoever wrote the text it reads, and advises deterministic, non-LLM limits on what the system can do. ASI03 Identity and Privilege Abuse adds a multi-agent form, where a compromised low-privilege agent relays instructions to a high-privilege one. The model and multi-agent forms have no one-off fix, so the check is a design review that traces every route from untrusted text to a write.

Quality failures that aren’t security bugs

LLM03:2026 applies “regardless of what is causing the LLM to malfunction”, hallucination from benign prompts included. A refund for the wrong amount needs no attacker. Outputs are probabilistic, so OWASP’s red teaming guide calls for statistically rigorous testing. A failure in a small share of runs can pass a single attempt. And the NCSC’s guidance says a test shows only that systems are “not vulnerable to known issues” on the day it runs.

What AI red teaming misses

Three things a launch decision needs fall outside red teaming.

Systematic quality measurement

Microsoft’s planning guide says “red teaming is not a replacement for systematic measurement”. NIST AI 600-1 warns that jailbreaking or prompt-engineering tests “may not systematically assess validity or reliability risks”. The third lesson in Microsoft’s Lessons From Red Teaming 100 Generative AI Products (13 January 2025), drawn from more than 80 operations since 2021, is that AI red teaming is not safety benchmarking. A red team shows that an attack worked, but the agent’s failure rate on ordinary refund requests needs an eval suite.

Regression after model updates

A red-team report describes the model, prompts and tools it attacked, so after a model update nobody knows which findings still hold until someone tests again. OWASP’s guide puts drift outside the red team’s job, saying that detecting response degradation over time “belongs to ongoing monitoring”. A regression suite in CI answers for every failure class in one run, and our runbook for re-running a regression suite after a model update covers the comparison.

A go/no-go decision against thresholds

A red-team report lists what broke. Whether an injection that works in some runs should block a launch needs a threshold agreed before the test and a person who owns the decision. NIST AI 600-1 adds that red-team results need further analysis before they feed governance decisions.

Which test to run first

This order is our recommendation. No standard sets one. The rows run in priority order, so where several apply, start with the highest.

If Run first Then Why
Your agent can change customer data or move money A readiness review of tool access, routes from untrusted content to writes and thresholds per failure class A targeted red team of the write paths, after the fixes Typical blockers (a broad credential, token passthrough, a write with no confirmation step) are design flaws that exploit testing finds only on the routes it tries
An enterprise security review is due A third-party penetration test that includes the agent’s API, authentication and MCP servers The readiness verdict and access pages, for the questions about the agent CSA’s Cloud Controls Matrix asks for the “periodic performance of penetration testing by independent third parties” (control TVM-06)
A model update or retirement is coming The regression suite, on both models side by side Re-run the injection tests on write paths The model changes, but the infrastructure doesn’t
The agent talks to the public on open-ended topics and has no write tools A broad red team with domain experts and general-public testers Eval cases, with thresholds, for each class of finding The risk is in what it says, on topics nobody listed in advance
You ship an MCP server, ChatGPT plugin or Claude connector An MCP security audit against the 2026-07-28 specification Your usual penetration test of the hosting OAuth flaws such as token passthrough sit in the server, whichever client calls it

Where a production readiness review fits

Our production readiness review is an AI agent security assessment and an eval build in ten working days, fixed at USD 10,000. It covers:

  • an eval suite built from the agent’s real failures (before launch, from pilot conversations and scripted attacks), committed to your repository and CI
  • a security review of every tool and MCP server, covering credentials and scopes, token passthrough, the confused deputy, prompt-injection paths to write actions, SSRF in OAuth discovery, and logging
  • operational checks on monitoring, a kill switch and rollback, rate limits, escalation to a human and cost per conversation
  • a written verdict of go, go with conditions or no-go, with each finding’s severity, evidence and suggested fix.

The verdict lets you stop a launch on a named blocker, or launch on the signed date with conditions that have owners. The review covers up to 10 tools and MCP servers, with one re-run within 30 days of your fixes. It excludes penetration testing of systems the agent can’t reach, open-ended red teaming of harms unrelated to its tools, and compliance audits, so a customer who wants a third-party penetration test report still needs one.

Our view

For an agent that can write to customer data or move money, the cheapest risk reduction comes before any attack testing. Narrow each tool’s credential, confirm every write that untrusted content can reach, and measure the failure classes that move money. Design comes first because the NCSC warns that prompt injection may never be fully mitigated, so the controls that hold are limits on what the agent can do.

After that, a targeted red team of the write paths is worth paying for, and a broad one earns its cost when the audience and topics are open. When you compare AI red teaming services, ask each supplier which of your agent’s write paths its statement of work names. We’d wait to red team until an eval suite exists, so that each finding becomes a test case that survives the next model update. And list the agent’s endpoints and MCP servers in the penetration test’s written scope, because testers test what the scope names.

Where this comparison stops

  • Passing any of the three is evidence about the cases tried, not a guarantee about the ones nobody tried.
  • The prices are one supplier’s public day rates. Totals depend on scope, and other suppliers price differently.
  • The lists are dated. This comparison uses the LLM list’s 2026 edition, the agentic list of December 2025 and ATLAS content version 2026.09 of 14 September 2026, which its changelog shows updating monthly since May 2026. Attack classes found later aren’t on them.
  • It doesn’t cover the red teaming that model developers run on base models, or legal duties such as those under the EU AI Act.

If you’d rather we did this

Our production readiness review builds an eval suite from your agent’s real failures, reviews every tool and MCP server it can call and checks its operations, then gives you a written go/no-go verdict in ten working days for a fixed USD 10,000.

Sources

All pages opened on 19 October 2026.