Red Teaming LLM Agents: A Practical Framework for Production Teams
A five-phase red teaming framework for LLM agents: map trust boundaries, build attack corpora, test, triage findings, and gate releases.
The Agent Attack Surface Problem
LLM agents do not just generate text — they act. They call tools, write to memory, trigger workflows, and hand off tasks to other agents. That shift from answering to doing is what makes them valuable, and it is also what makes them dangerous. A chatbot that produces a bad answer is an embarrassment. An agent that produces a bad action — a refund issued, a record deleted, a credential leaked — is an incident.
Most organizations still test agents the way they tested chatbots: a handful of prompts, a manual review, a sign-off before launch. That approach does not survive contact with production. Agent behavior is nondeterministic, multi-step, and shaped by whatever data it retrieves at runtime. The most credible way to understand its failure modes is to attack it deliberately, repeatedly, and with the same rigor you apply to any other security-critical system — while recognizing that red teaming complements, and does not replace, static analysis of tool schemas, permission reviews, and output classifiers. (See: LLM agent security fundamentals)
This article lays out a five-phase red teaming framework built for production teams: map trust boundaries, build an adversarial corpus, execute structured testing, triage by exploitability and blast radius, and convert every finding into a regression test. The goal is not a one-time audit. It is a release gate that runs every time the model, the prompts, or the tools change.
Why Red Teaming LLM Agents Is Different From Chatbot Testing
Static QA assumes a fixed input space. You write test cases, you compare outputs, you pass or fail. Agents break that assumption at every layer. The same prompt can produce different tool calls on different runs — sometimes because of sampling temperature, more often because the agent re-plans after a retrieval result changes, a tool returns a different payload, or a retry loop takes a different path. A single task may span a dozen steps, and the failure may only appear at step nine. Emergent behaviors — loops, retries, unexpected tool chains — appear only under realistic load and realistic data.
The operational consequence: single-run pass/fail is meaningless for agents. Your test criteria have to be statistical — "this attack succeeds in fewer than k of n runs" — and your harness has to capture full execution traces, not just final outputs, because the failure you care about is usually mid-trajectory.
Tool use, memory, and multi-step execution expand the attack surface
Every tool an agent can call is a potential exploit primitive. A search tool becomes an exfiltration channel. A file-write tool becomes a persistence mechanism. An email tool becomes a phishing relay. Each permission you grant to make the agent useful is a permission an attacker can try to borrow.
Memory compounds the problem. When an agent persists state across sessions, a single poisoned input can survive long after the conversation that introduced it. An attacker who plants an instruction in a document the agent later retrieves may influence behavior days later, in a session they never touched — because the poisoned content gets re-surfaced by retrieval, folded into a session summary, or written to a long-term memory store that subsequent sessions read. The practical mitigation is provenance on every memory write: tag the source, the trust level, and the timestamp, and treat low-trust memory as untrusted input on every read, not just on the turn it was created. (See: indirect prompt injection and persistent memory poisoning)
Where AI safety and agent security diverge
These are related but distinct disciplines, and production teams need both.
- AI safety concerns harmful outputs: bias, toxicity, dangerous advice, policy violations.
- Agent security concerns unauthorized actions and data exposure: privilege escalation, exfiltration, tool misuse, cross-tenant leakage.
A safety-only review will miss an agent that politely and accurately exfiltrates a customer database. A security-only review will miss an agent that takes no unauthorized action but gives catastrophically bad advice. Red teaming for agents has to cover both, with separate test suites and separate success criteria.
A chatbot failure is a content problem. An agent failure is an incident with a timestamp, an actor, and often a compliance obligation attached.
A Five-Phase Red Teaming Framework for Production Agents
The framework below is designed to be repeatable and to fit inside an existing SDLC. Each phase produces an artifact the next phase consumes, and the whole loop re-runs on every material change.
Phase 1: Map the agent's trust boundaries and capabilities
You cannot attack what you have not inventoried. Start by documenting the agent's capability envelope: every tool it can call, every data source it can read, every system it can write to, and every point where a human is supposed to be in the loop.
For each capability, record:
- Inputs and outputs — what data crosses the boundary, and in which direction.
- Permissions — the actual scopes and credentials, not the intended ones. Pull these from the identity provider and the tool's own config; the design doc is usually wrong.
- Trust level of the source — is the data user-supplied, retrieved from the web, or pulled from an internal system?
- Human checkpoints — where approval is required, and whether it can be bypassed (e.g., by an agent that calls the tool directly instead of going through the approval flow).
The output is a trust-boundary diagram plus a table of capabilities ranked by sensitivity. This becomes your test plan's backbone: every high-sensitivity capability needs adversarial coverage in Phase 2, and the coverage percentage becomes a release-gate metric. (See: agent capability inventory template)
Phase 2: Build an adversarial test corpus
Treat your attack corpus like code. Version-control it, review it, and grow it with every incident. Seed it from four sources:
- Known prompt injection patterns — direct overrides, role-play escapes, delimiter confusion, encoding tricks.
- Tool-abuse payloads — malformed parameters, oversized inputs, chained calls designed to escalate.
- Domain-specific misuse cases — the ways your agent, in your business, could cause harm.
- Indirect injection samples — poisoned documents, web pages, and emails the agent might retrieve.
Tag each case with the threat category it targets and the capability it exercises. This makes coverage gaps visible and lets you run targeted subsets when a specific tool changes. (See: prompt injection payload library)
Phase 3: Execute structured adversarial testing
Run two tracks in parallel. Automated testing handles volume: fuzzing over inputs and parameters, and LLM-vs-LLM red teams where one model generates attacks and a second model judges outcomes against explicit success criteria (e.g., "did the target call the refund tool with a value not authorized by the user?"). Human-led testing handles the cases automation misses: multi-turn social engineering, business-logic abuse, and novel chains across tools.
Two non-negotiables for the execution environment:
- Sandbox everything. Run against mock or staging tools with fake credentials and rate limits. Red teaming an agent that can actually issue refunds or send email is how you cause the incident you were trying to prevent.
- Capture full traces. Log every model call, tool call, argument, and retrieval result per run. Without traces you cannot tell a real exploit from a flaky test, and you cannot build the regression test in Phase 5.
Report results as attack success rate (ASR) per capability and per threat category, plus a secondary metric — tool-call depth to failure — which tells you how far an attacker got before the agent was stopped. A low ASR with high depth-to-failure means your guardrails are firing late.
Phase 4: Triage by exploitability and blast radius
Not every finding is a release blocker. Score each one on two axes:
- Exploitability — can a realistic attacker reach it? (Low: requires unusual internal data. High: reachable from any user-supplied input or any retrievable document.)
- Blast radius — what is the worst outcome if it fires? (Low: cosmetic or reversible. High: data exfiltration, financial transaction, cross-tenant access, credential exposure.)
Sort findings into a 4×4 matrix and assign severity:
| Low blast radius | High blast radius | |
|---|---|---|
| High exploitability | Medium | Critical |
| Low exploitability | Low | High |
Because results are nondeterministic, define reproducibility explicitly: a finding is confirmed if it succeeds in at least k of n runs (k=3, n=10 is a reasonable starting point for high-blast-radius capabilities). Anything below the threshold goes to a "watch" list and gets re-run on the next cycle rather than being closed.
Phase 5: Remediate and convert findings into regression tests
Every confirmed finding produces two artifacts: a fix, and a regression test that fails until the fix holds. The regression test is the same attack case from Phase 2, promoted into the corpus with its success criteria and its k-of-n threshold. This is how the corpus grows and how the release gate stays honest.
Fixes usually fall into four buckets, in rough order of preference:
- Reduce the capability — remove the tool, narrow the scope, or split a high-privilege tool into a low-privilege one plus a human approval step.
- Add a deterministic guardrail — schema validation on tool arguments, allow-lists on destinations, output classifiers, and hard limits on transaction size or call count.
- Harden the prompt and the context — explicit trust labeling on retrieved content, instruction hierarchy, and provenance checks on memory reads.
- Add monitoring and kill switches — anomaly detection on tool-call patterns and the ability to disable a tool without redeploying the agent.
The release gate
The loop is only useful if it gates releases. Define pass criteria up front and enforce them in CI:
- Zero open Critical findings; High findings require a signed risk acceptance with an expiry date.
- ASR below threshold for every high-sensitivity capability (e.g., <5% across n=10 runs).
- Coverage of 100% of high-sensitivity capabilities from the Phase 1 inventory.
- All prior regression tests passing at their k-of-n thresholds.
Trigger the full loop on any material change: model version, system prompt, tool schema, permission scope, or retrieval corpus. Run the targeted subset on every deploy.
The point of this framework is not to prove the agent is safe. It is to make the agent's failure modes known, measured, and gated — so that the decision to ship is made with evidence rather than optimism.