Red Teaming Agentic AI: Why Testing the Model Is No Longer Enough

In January 2025, NIST's Center for AI Standards and Innovation published red team results that should have changed how enterprises test autonomous systems. Against an AI agent operating in simulated workspace, travel, Slack and banking environments, the strongest previously known hijacking attack succeeded 11% of the time. The strongest new attack developed by the red team succeeded 81% of the time. The model had not changed. The evaluation had.

That gap between what an organisation believes it has tested and what an adversary can actually achieve is the central problem in agentic security today. Most enterprises deploying AI agents are assessing them with methods designed for a different class of system, and the assurance those methods produce does not transfer.

What Agentic AI Red Teaming Tests That Model Testing Does Not

Agentic AI red teaming tests the actions a system can take, not merely the text it can be persuaded to produce. Conventional AI red teaming matured around generative models, where the worst outcome of a successful prompt injection was a harmful, biased or confidential string appearing in an output window. That discipline remains necessary, and its taxonomies — NIST AI 100-2, MITRE ATLAS, the OWASP LLM Top 10 — still describe real attack classes accurately.

What it does not describe is an attack surface composed of tool calls, persistent memory, delegated authority and inter-agent messaging. An agent does not simply answer. It queries a database, writes to a repository, sends an email, invokes another agent, and stores what it learned for the next session. Each of those capabilities converts a language vulnerability into an operational one. A jailbreak that produces objectionable text is a content failure; the same jailbreak in an agent with shell access is a remote code execution finding. Testing the model in isolation measures the first and says nothing about the second.

The Anatomy of an Agentic Attack Chain

Agentic compromise is rarely a single exploit. It is a chain in which each documented technique supplies a capability the next one requires, and no individual link would necessarily fail a model-level test.

Consider how three real disclosures compose. In February 2025, researcher Johann Rehberger demonstrated a technique he called delayed tool invocation against Google Gemini. Malicious instructions embedded in a document are not executed on contact, because the system correctly refuses to run tools on untrusted input. They sit dormant as a conditional — if the user later says X, do Y — and activate during a subsequent trusted interaction, when those protections no longer apply. The corrupted information then persists into long-term memory and shapes responses across future sessions. Google assessed the impact as low, noting that it requires user participation and that memory updates are surfaced to the user.

Persistence is only useful if it can reach something. That is what CVE-2025-32711, disclosed in 2025 and known as EchoLeak, supplied against Microsoft 365 Copilot. The attack required no user interaction at all, triggering the receipt of a crafted email. It evaded Microsoft's cross-prompt injection classifier, defeated link redaction by using reference-style Markdown, and exfiltrated data through a Teams proxy that content security policy already permitted.

The third link is authority. AWS security bulletin AWS-2025-019 documented prompt injection issues in Amazon Q Developer and Kiro in which malicious content could trigger code execution through find, grep and echo commands, and leak secrets over DNS using ping and dig, in some cases without human approval. Poisoned memory establishes persistence, a zero-click path establishes reach, and an over-permissioned tool turns both into consequence. An organisation that tested only for jailbreaks would have cleared all three.

Why the OWASP LLM Top 10 Is the Wrong Map for Agents

The OWASP Top 10 for Agentic Applications, published on 9 December 2025 under chair John Sotiropoulos with co-leads Keren Katz and Ron F. Del Rosario, exists because the LLM list does not cover this ground. Its ten entries are not a re-skin of the earlier taxonomy. They describe failures that only occur in systems that act.

The list runs from ASI01 Agent Goal Hijack and ASI02 Tool Misuse through ASI03 Identity and Privilege Abuse, ASI04 Agentic Supply Chain Vulnerabilities, ASI05 Unexpected Code Execution and ASI06 Memory and Context Poisoning, to ASI07 Insecure Inter-Agent Communication, ASI08 Cascading Failures, ASI09 Human-Agent Trust Exploitation and ASI10 Rogue Agents. The working group anchored several entries to the incidents above, citing EchoLeak under goal hijack, Amazon Q under tool misuse and the Gemini memory attack under context poisoning.

The practical consequence for a security leader is a scoping question with an uncomfortable answer. A supplier statement of work that promises coverage of the OWASP LLM Top 10 is promising a map of the wrong territory. Four of the ten agentic categories — identity abuse, inter-agent communication, cascading failures and rogue agents — have no meaningful equivalent in a test that never lets the system call a tool. Enterprises implementing security protocols for agentic autonomy need to confirm which taxonomy an assessment was actually built against.

Single-Shot Testing Is the Methodological Failure Underneath

The deeper problem is not which attacks are tested but how many times each one is attempted. The same NIST work found that when attacks were repeated rather than run once, the average success rate rose from 57% to 80% across 25 attempts per injection task. Nothing about the defence changed between the first attempt and the twenty-fifth.

This follows from the probabilistic nature of the systems. A model that refuses a malicious instruction 19 times out of 20 will comply on the twentieth, and an adversary is under no obligation to stop at one. A red team report that records a single blocked attempt per technique is not measuring resilience. It is sampling a distribution once and reporting the sample as the result.

In our assessments, the findings that persuade an executive audience are almost never the first-attempt results. They are the repeat-rate figures — the demonstration that a control which appeared to hold is actually a coin weighted slightly in the defender's favour, across an interface an attacker can query indefinitely.

What This Changes in Scoping and Procurement

Scoping an agentic assessment begins with an inventory that most organisations do not yet maintain: every agent in production, the tools each one can invoke, the data sources it retrieves from, the credentials it holds, and the other agents it can instruct. Attack surface here is defined by permissions, not parameters, and an inventory organised by model name will miss it entirely.

Three requirements follow. Testing must place adversarial instructions in the channels agents actually read — documents, email, ticket comments, retrieved web content — rather than only in the user prompt. It must run each technique repeatedly and report success as a rate rather than a binary. And it must be framed against an agentic taxonomy, with NIST's three-tier ARIA structure of model testing, adversarial red teaming and field testing offering a defensible way to show a regulator that all three layers were covered. For organisations subject to Article 55 of the EU AI Act, which requires adversarial testing for models presenting systemic risk, that documentation burden is already live, and it sits alongside the wider governance expectations now shaping cyber resilience and AI risk in critical infrastructure.

None of this makes model-level red teaming obsolete. It remains the foundation, and an agent built on an untested model inherits every weakness in it. But the assurance it produces stops at the boundary where the system begins to act, and for most enterprises that boundary was crossed some time ago. The question worth asking of any current assessment is a simple one: did the test let the system do anything, and did it try more than once?

Frequently Asked Questions

What is agentic AI red teaming? Agentic AI red teaming is adversarial testing of AI systems that take autonomous actions — invoking tools, writing to systems, retaining memory and delegating to other agents. It differs from model-level red teaming by evaluating the consequences of a successful manipulation rather than only whether the manipulation produces harmful text. The benchmark taxonomy is the OWASP Top 10 for Agentic Applications, published in December 2025.

How is it different from LLM penetration testing? LLM penetration testing generally targets the model and its prompt interface, assessing jailbreaks, data leakage and harmful output generation. Agentic testing additionally covers tool misuse, identity and privilege abuse, memory poisoning, inter-agent communication and cascading failure across multiple systems. A test that cannot invoke the agent's tools cannot assess four of the ten OWASP agentic categories.

Can automated tools red team AI agents? Automated tooling handles the scale requirement well, and scale matters here: NIST found average attack success rising from 57% to 80% when techniques were retried 25 times, which is impractical manually. Automation is weaker at discovering novel multi-step chains that combine memory persistence, a delivery path and an over-permissioned tool. Most mature programmes run automated testing continuously and commission expert-led assessments periodically.

How often should agentic systems be red teamed? Continuously in principle, because the tested configuration changes whenever a system prompt, tool permission, model version or connected data source changes — which in most deployments is weekly rather than annually. A point-in-time annual assessment describes a system that no longer exists. Practical programmes pair automated regression testing on every change with a deeper expert engagement each quarter or on major releases.

Does the EU AI Act require red teaming of AI agents? Article 55 of the EU AI Act requires adversarial testing for general-purpose AI models presenting systemic risk, and obligations attach primarily to model providers rather than every deployer. Organisations building agents on those models face related due-diligence and documentation expectations, and sector regulators have been more prescriptive in some cases. Legal obligations vary by role and jurisdiction, so this should be confirmed with counsel rather than inferred from the Act's headline requirements.