Testing AI agents for prompt injection

What is being tested?

Prompt-injection testing is frequently reported as a language exercise: an attacker writes a clever instruction, the model complies, and the transcript is the finding. That framing produces reports full of quotations and short of consequences.

The question worth answering is what the system can be made to do. An assistant that can be persuaded to say something impolite is a quality problem. An assistant that can be persuaded to call a tool, retrieve a record belonging to another customer, or send data to an address chosen by an attacker is a security problem, and the difference is the tools and permissions attached to the model rather than the model itself.

OWASP separates these in its 2025 list, with prompt injection at LLM01 and excessive agency at LLM06. The separation is useful in practice: injection is how an instruction arrives, and agency determines what the instruction can accomplish.

Why the problem is structural

A language model receives instructions and data through the same channel and has no reliable way to rank their authority. Text retrieved from a document, an email or a web page arrives in the same context window as the system prompt, and the model has been trained to follow instructions it finds in text.

NIST’s revised taxonomy, published in March 2025, covers indirect prompt injection explicitly, along with attacks on agent tool chains. It pairs each attack class with mitigations and states that those mitigations are not fool-proof. That sentence is the reason testing exists as a discipline. If a filter could be trusted, the work would be a procurement decision.

The most useful practitioner framing is Simon Willison’s. An agent is exposed when three things are true at once: it can reach private data, it processes untrusted content, and it has some way to send information outwards. Any two of the three are usually survivable. All three together give a hostile instruction a complete path, and no amount of detection tuning removes the path.

Where do the attacks actually come from?

Direct injection, where a user types an instruction into a chat box, is the easiest case to test and the least interesting. The payload passes through an interface someone is watching, and the damage is bounded by what that user could already do.

Indirect injection is the case most systems have never been tested against. The instruction is planted in content the system will read later: a PDF attached to a routine query, a support ticket, a calendar invitation, a page the agent browses, or the output of another tool. A legitimate user asks an ordinary question, retrieval pulls in the planted text, and the instruction executes with that user’s permissions. Nothing in the interaction looks unusual to the person who asked.

Johann Rehberger’s published research demonstrates these paths against shipped products rather than laboratory constructions, which is what makes it worth reading before designing a test plan. The attacks are not exotic. They rely on the ordinary behaviour of systems built to be helpful.

The attack path, and where the test assertion belongs

1. Untrusted contentA document, ticket, email or page carrying an instruction.

2. RetrievalThe system pulls it into context during an ordinary request.

3. The model reads itInstruction and data arrive through the same channel.

4. Tool callA function is invoked, with arguments the attacker influenced.

5. ConsequenceData leaves, a record changes, money moves. The test asserts here.

How the testing is done

Threat modelling comes before any attack. Draw what the system can read, what it can write, what it can spend, and which actions complete without a person. Most serious findings are visible on that diagram before a single payload is written, and the diagram determines which attacks are worth the time.

Baselining against a published taxonomy is next. Working from OWASP’s list for LLM applications and mapping to MITRE ATLAS means nothing is skipped because it seemed uninteresting, and it lets a client compare one firm’s coverage with another’s.

Automated probes establish the floor. Thousands of known attack strings across jailbreaks, leakage and encoding tricks are cheap to run and catch the obvious cases. They do not find the interesting ones, because the interesting ones depend on the specific tools and data this system holds.

Manual work follows the paths a scanner cannot reach: instructions planted in the retrieval corpus, tool calls with forged arguments, campaigns that fail in one message and succeed across fifteen, and the guardrail itself, which is a control with a false-negative rate that should be measured on the client’s own traffic rather than quoted from the vendor’s benchmark.

Reporting a probabilistic system

A model that refuses an attack once may accept it on the fourth attempt with different phrasing, or on the same phrasing at a different temperature. Testing once and recording the outcome produces false negatives by construction.

Every confirmed finding should therefore carry the full transcript, a severity, the fix, and how many attempts out of how many produced the result. An attack that works eighteen times in twenty is a different engineering problem from one that works six times in twenty, and both are real.

What should be left behind

Findings age quickly. Prompts get edited, models are replaced, retrieval corpora grow, and a report written in March describes a system that no longer exists by August. The tests do not age in the same way.

Every confirmed attack should become an assertion wired into the client’s own release pipeline, running on every deployment and on a schedule against the live configuration. This is the part clients rarely ask for and the part they keep. It converts a point-in-time assessment into a standing control, and it is the reason a retest after remediation is part of the engagement rather than an upsell.

The position we hold

Where a system combines private data, untrusted content and an outbound path, that combination is the finding. It can be reported before a single successful attack, and remediation means removing one of the three rather than tuning a classifier.

Detection has a place as a second layer. Sold as the primary defence, it substitutes a probability for an architecture, and the probability is being estimated against an adversary who gets to choose the wording.