Inside an Agent Ops Team: What Breaks When You Run 40 AI Agents in Production
An agent ops lead shares what actually broke scaling to 40 AI agents in production—silent failures, cost creep, memory drift, and permission sprawl.
Editor's note: This is a composite case study. "Maya Okafor" and "Meridian Freight" are illustrative, and the figures below are representative of patterns we've seen across teams running agents at scale — not a single audited account. We've labeled them as such so you can judge the lessons on their merits.
The breaking point wasn't a model. It was a Tuesday.
At around 2 a.m., an agent responsible for reconciling vendor invoices began approving charges against a deprecated cost center. No errors fired. Latency stayed flat. Uptime dashboards glowed green. By the time a human noticed three days later, the company had miscoded roughly $180,000 in spend and triggered a downstream reporting scramble that ate two weeks of finance team time.
That incident is one of several we walked through with Maya Okafor, who leads agent operations at a mid-size logistics software company we're calling Meridian Freight. Over 14 months, her team scaled from a handful of experimental agents to 40 AI agents in production touching procurement, customer support triage, contract review, and internal IT. We asked her what actually broke, what they rebuilt, and what she'd tell any team about to hit the same wall.
Her thesis is blunt: the models held up fine. The operating model didn't.
The following is an edited conversation, organized around the failure modes and fixes that defined Meridian's year.
Meet the Team: How an Agent Ops Function Takes Shape
Walk me through the team. Who actually runs 40 agents?
When we started, "the AI team" was two ML engineers and a product manager. That worked at ten agents because everyone knew every agent. At forty, nobody could hold the system in their head. We now have a nine-person agent ops function sitting alongside — not inside — the core platform team.
The roles emerged rather than being designed. Nobody budgeted for them in year one.
The roles nobody budgeted for
- Agent reliability engineer. Owns the health of agent behavior in production: tracing, alerting, and the incident response loop. Think SRE, but for nondeterministic systems where "no error" is not the same as "correct."
- Eval and QA owner. Runs production evals, maintains golden datasets, and owns regression detection. This is a full-time job on its own.
- Prompt and context librarian. Manages shared prompts, retrieval corpora, tool schemas, and versioning. Sounds bureaucratic until you've had two agents silently diverging on the same policy document.
- Platform owner. Builds the shared runtime — orchestration, tool gateways, cost attribution, access control — so individual teams aren't rebuilding plumbing.
Where does agent ops sit in the org?
We debated this hard. The platform team wanted us embedded; leadership wanted a dedicated function. We landed on a hybrid: agent ops reports to the VP of Engineering but maintains a dotted line to the platform group. The argument that won was accountability. When an agent misbehaves at 2 a.m., someone needs to own the pager without negotiating with three other teams.
The dotted line matters more than it sounds. It means the platform owner can push shared runtime changes — a new tool gateway, a tracing schema update — without filing a ticket against a team that doesn't report to them. We formalized it as a weekly sync with a shared backlog, and the platform group treats our requests as first-class work rather than favors.
How did you justify the headcount?
We didn't pitch it as innovation. We pitched it as cost avoidance. We modeled the invoice incident, a support agent that had been over-escalating tickets for weeks, and the engineering hours burned on manual firefighting. The number that got attention: roughly 30% of senior engineering time was going to agent babysitting that should have been product work. (Illustrative figure.)
The way we framed the ask mattered. We didn't walk in with a headcount slide; we walked in with three incident timelines and the hours attached to each. When you can show a VP that a single silent failure cost two weeks of finance time, the conversation shifts from "can we afford this team" to "can we afford not to have it." I'd tell any team building this case to lead with the incidents, not the org chart.
What Broke First: The Failure Modes of Scaling AI Agents
Let's get specific. What broke, in order?
Silent failures first, then cost, then context, then permissions. Each one got worse as agent count grew, because agents interact.
Silent failures and alert fatigue
The core problem is that an agent can complete a task successfully and still be wrong. Traditional monitoring watches for errors and timeouts. Our agents rarely errored — they just did the wrong thing confidently.
"We had 99.9% uptime and a system that was quietly making bad decisions. Uptime monitoring told us nothing. That gap is where most of our early damage happened."
The fix was to stop treating availability as the reliability metric and start measuring behavior. We defined behavioral SLIs — task success rate against a labeled set, tool-call error rate, groundedness (did the answer trace to a retrieved source?), escalation rate, and outcome-level checks for anything touching money. Those became SLOs with error budgets, reviewed weekly. Uptime stayed on the dashboard; it just stopped being the thing we paged on.
Instrumentation came from OpenTelemetry traces emitted by our orchestration layer, shipped to Langfuse, with LangSmith used earlier in development for prompt iteration. Every agent run became a trace with spans for each reasoning step, tool call, and retrieval. That gave the reliability engineer something concrete to look at: a waterfall where a wrong decision is visible as a sequence of plausible-looking steps, not a stack trace. We also ran a nightly eval job against Arize Phoenix so drift showed up as a trend line rather than a surprise.
Once we started instrumenting behavior, the alert volume exploded. We went from a handful of infra alerts to hundreds of low-signal behavioral alerts a week. We tiered severity into three bands: P1 for anything touching money, customer-facing output, or permissions; P2 for degraded quality trends; P3 for drift signals that route to a weekly review instead of a pager. That cut page-worthy noise by roughly 80%. (Illustrative figure.) (Related: how to build an AI agent observability stack)
Cost creep and runaway loops
Token spend per agent was invisible at first because it rolled up into one bill. Per-agent attribution is genuinely hard: orchestration is shared, retries hide inside tool calls, retrieval re-embeds the same documents on every turn, and cached vs. uncached input tokens have wildly different costs. When we finally attributed spend per agent — and split input from output tokens — we found two agents accounting for nearly half of total spend. One was stuck in a retry loop on a flaky downstream API; the other re-retrieved the same large documents on every turn instead of caching them.
Per-agent cost attribution changed behavior immediately. Teams started treating token budget like any other resource. We now set hard ceilings per run and per step, not just per month, and alert on anomalous spend velocity — a single run that suddenly consumes 10x its normal tokens is a loop, and you want to catch it in minutes, not at the end of the month. (Related: controlling LLM token costs in production)
The retry loop is worth dwelling on because it's the classic case. The agent called a downstream pricing API, got a timeout, and retried — but the retry logic lived inside the tool wrapper, so the orchestration layer saw one tool call while the wrapper quietly made six. Multiply that across a busy afternoon and you have a line item nobody can explain. We fixed it by moving retries into the orchestration layer, where Temporal enforces a bounded retry policy with backoff and a hard attempt cap, and by tagging every span with a run ID so cost could be rolled up per agent and per run. The caching fix was simpler: a content-addressed cache in front of the retrieval layer so identical documents aren't re-embedded on every turn.
Eval Gates Before Deploy
One gap we didn't anticipate was that our deploy process had no opinion about agent quality. Code shipped through CI; prompts and tool schemas shipped through a wiki page and a Slack message. That's how two agents ended up on different versions of the same policy.
We now run eval gates in the deploy pipeline. Any change to a prompt, tool schema, or retrieval bundle triggers a regression suite against a golden dataset — a few hundred labeled cases per agent, weighted toward the failure modes we've actually seen. The gate blocks the deploy if task success drops below a threshold or if groundedness regresses. It runs in CI alongside the unit tests, so the eval owner isn't a human bottleneck; they own the dataset and the thresholds, not the button-pressing.
The cost of this was real: building the golden datasets took the eval owner and a domain expert several weeks per high-stakes agent. But it paid for itself the first time it caught a prompt change that improved the happy path and quietly broke escalation handling. Without the gate, that ships to production and we find out from customers.
Context and memory drift across agents
This one surprised us. Agents sharing a retrieval layer began diverging because the underlying corpus changed underneath them. One agent's "current" policy was another agent's stale snapshot. To be precise, this was retrieval-corpus version skew, not memory in the agent-state sense — but the two fail the same way, silently, and both change the agent's judgment rather than its settings.
"Context drift is the agent equivalent of configuration drift, except it's invisible and it changes the agent's judgment, not just its settings."
We fixed it with versioned context bundles: every agent pins to a specific corpus version and prompt version, and upgrades go through the eval owner and a cache-invalidation step. Boring, but it stopped the silent divergence. Long-term agent memory got the same treatment — versioned, scoped per agent, and never silently shared. (Related: managing context and memory for AI agents)
Permission sprawl and blast radius
Early agents got broad tool access because scoping was tedious. That's a known anti-pattern, and at forty agents it meant dozens of credentials with overlapping, over-broad permissions. The invoice incident was partly a permissions failure — the agent could write to a cost-center field it had no business touching, and nothing stopped the write.
The rebuild had four parts:
- Least-privilege tool scopes. Each agent gets the narrowest credential that lets it do its job — read-only where read-only will do.
- Per-action authorization, not per-agent. The check happens at the moment of a write, against the specific resource, not once at agent startup.
- Human-in-the-loop for irreversible actions. Any write above a threshold, or to a financial or customer-facing field, requires an approval step.
- Blast-radius containment. Dry-run mode, allow-lists, and rate limits on destructive tools so a misbehaving agent hits a wall before it hits production data.
The mechanics: credentials live in Vault, issued per agent with short TTLs, and authorization decisions run through OPA policies evaluated at call time. The policy for the invoice agent now reads, in effect, "may write to cost-center fields only for cost centers in the active list, only below a threshold, and only after an approval token is present." That single policy would have stopped the Tuesday incident cold. The human-in-the-loop checkpoint routes approvals to a queue in the finance team's existing tooling — we deliberately didn't build a new UI for it — and the runbook says an approver who doesn't recognize the vendor or amount rejects and escalates rather than investigating.
On-Call and Escalation for Agents
The piece nobody warns you about is the pager. We initially bolted agent alerts onto the existing platform on-call rotation, and it went badly — the platform engineers could see a failing trace but had no idea whether the agent's output was wrong, only that a span was slow or a tool call failed. We split the rotation: the agent reliability engineer owns P1 behavioral alerts, the platform team owns infrastructure, and there's a documented handoff when an alert turns out to be a runtime problem rather than a behavior problem.
The runbook is deliberately short and decision-oriented. For a P1, the first step is always pause the agent — flip it to dry-run mode, which we built as a one-line config change — before anyone debugs. Then triage: is the output wrong, or is the system wrong? If wrong output, check the last context bundle and prompt version change. If system, hand to platform. The runbook names the exact dashboards and the exact query to pull recent traces for the affected agent. We wrote it after the invoice incident, and the first time we used it in anger, the pause took under a minute.
What We'd Do Differently
If you started over tomorrow, what changes?
Three things. We'd define behavioral SLOs on day one instead of after the first incident. We'd build per-agent cost attribution and least-privilege tool scopes before scaling past ten agents, not after forty. And we'd treat the eval owner as a founding role, not a hire we made in month nine. The models were never the hard part. The operating model was — and it's cheaper to build it early than to rebuild it under a 2 a.m. pager.
If I had to compress it to one lesson for a team about to scale: the moment you can't hold every agent in your head, you've lost the ability to catch silent failures by intuition. That's the line where you need instrumentation, eval gates, cost attribution, and scoped permissions in place — not because the models are failing, but because the humans can no longer see what they're doing.