Guardrailing Compound Agents: AI Safety Controls for Multi-Step Autonomous Workflows

Compound Agents 101: Anatomy of a Multi-Step Workflow

UC Berkeley researchers popularized the term "compound AI system" in 2024. It describes an AI system built from multiple interacting components — not a single model. Enterprises adopt compound agents because chaining steps turns chatbots into systems that complete real work. A single prompt cannot reconcile an ERP ledger; a planner with tools can.

Planner, tools, memory, sub-agents: the anatomy of a compound AI system

Four components appear in most production systems:

The planner decomposes a goal into steps. It is the LLM (large language model — a neural network trained to predict text) at the center of the loop.
Tools extend the agent's reach. They are external functions the agent can call: APIs, databases, code execution, browsers.
Memory persists context between steps or sessions. Vector databases and task-state stores are the common implementations.
Sub-agents specialize the work. The planner delegates to child agents, each with its own tools.

A finance example makes this concrete: the planner receives "reconcile Q3 invoices." It queries the ERP via a tool, delegates the math to a sub-agent, writes results to memory, and emails a summary. Four steps, four systems, four chances to fail.

The failure surface: composition, propagation, persistence

Single-model failures stay contained. Compound-agent failures compose. Three mechanisms matter:

Composition multiplies risk across steps. Five steps at 98% reliability yield roughly 90% end-to-end.
Propagation turns one step's bad output into the next step's trusted input.
Persistence lets memory store corrupted state, so one bad step poisons every future run.

Guardrails must therefore wrap the whole loop, not one call.

Threat Model for Autonomous Multi-Step Workflows

Security teams should treat every agent workflow as untrusted input meeting privileged execution. That framing drives everything below.

Indirect prompt injection via tools, MCP, and retrieved content

Prompt injection is attacker text that overrides an agent's instructions. Indirect prompt injection hides that text in content the agent reads — not in the user's prompt. Sources include emails, web pages, PDFs, support tickets, and retrieved documents in RAG (retrieval-augmented generation — fetching external knowledge at query time).

Two enterprise-relevant vectors deserve special attention:

MCP (Model Context Protocol — an open standard for connecting models to tools and data) can carry hidden instructions. Tool descriptions and metadata are attacker-writable in many setups, a technique called tool poisoning.
Retrieved content can smuggle instructions into context. In 2025, researchers disclosed "EchoLeak" (CVE-2025-32711), a zero-click exploit where a crafted email made Microsoft 365 Copilot exfiltrate data.

OWASP ranks prompt injection as the #1 risk in its 2025 Top 10 for LLM Applications.

Excessive agency and privilege creep

Excessive agency grants an agent more actions, permissions, or autonomy than the task requires. It is a named risk in the OWASP LLM Top 10. Privilege creep accumulates access rights beyond what a role needs. Agents accelerate both risks: teams grant broad tokens "temporarily," then ship. In incident reviews we have conducted, the never-revoked "temporary" token is the most common root cause.

Cascading failures: loops, context drift, error propagation

Not every threat involves an attacker. A cascading failure is one fault triggering others downstream. Watch for three patterns:

Loops burn compute budget. An agent retries a failing tool indefinitely.
Context drift pulls the working context away from the original goal, so agents act on stale assumptions.
Error propagation flows an early wrong number into downstream writes, reports, and payments.

Exfiltration paths: egress, side effects, data sinks

Ask three questions of every workflow: how does data leave, what does the agent change, where does output land?

Egress moves data through network calls to external domains. An outbound API request with encoded data is the classic exfiltration channel.
Side effects change the world outside the model: emails sent, records deleted, payments initiated.
Data sinks retain output — logs, data warehouses, third-party SaaS.

The average cost of a data breach reached $4.88M in 2024, per IBM — and agentic workflows multiply the paths that lead there.

Guardrail teardowns: our subscriber portal publishes a monthly breakdown of real agent incidents, with the control that would have stopped each one. Subscribe to build your threat model faster.

Reference Architecture: The Four-Layer Guardrail Stack

Effective guardrails are layered. Each layer assumes the others will sometimes fail. The stack wraps the agent loop from four sides: what comes in, what the agent does, what goes out, and how the whole process is governed.

Input layer: screening untrusted content

The input layer treats all non-user content as hostile. Core controls:

Injection classifiers score text for hidden instructions before it reaches the planner.
Provenance labeling marks whether content came from the user or from tools, so downstream steps know what to trust.
PII (personally identifiable information) masking removes sensitive fields before content enters context.
Canonicalization normalizes text to defeat obfuscation tricks like invisible Unicode characters.

Latency budget: small classifiers typically add 20–50 ms per screening call — negligible next to a tool call that takes seconds.

Tool/action layer: least privilege plus approval gates

This layer controls what the agent can do.

Least privilege limits every tool call to the minimum permissions needed. Prefer scoped credentials: short-lived tokens limited to one tool and one data scope.
Allowlists enumerate permitted domains, record types, and commands. Anything unlisted fails closed.
Parameter validation checks arguments against schemas before execution.
Approval gates require human confirmation above defined risk thresholds, such as payments over $500.

Output layer: filtering, DLP, schema validation

The output layer ensures nothing sensitive or malformed leaves the system.

DLP (data loss prevention) scans outbound content for secrets, credentials, and regulated data.
PII redaction strips or masks personal identifiers before output reaches users, logs, or downstream systems.
Schema validation rejects malformed or off-contract output, so a hallucinated field never reaches the tool that executes it.
Response bounding caps payload size and query breadth, so a single compromised step cannot drain a data warehouse in one call.

Lat