We Shipped 40 AI Agents to Production in 6 Months: A Lead Engineer's Retrospective
Meta Description: A lead engineer's honest retrospective on deploying 40 AI agents to production in 6 months — architecture lessons, failure modes, observability gaps, and practical advice for enterprise AI agent deployment.
Six months ago, I told my CTO we'd have 40 AI agents running in production by the end of the year. I was right — but not in the way I expected.
It started as a modest experiment. Three agents: one for customer support triage, one for invoice processing, one for inventory checks. By month three, we'd ballooned to 17. By month six, the number sat at 40 — and I'd accumulated enough scar tissue to write a proper retrospective.
This is that retrospective. Not the polished conference talk version. The real one.
From 3 agents to 40: the evolution of our production AI agent fleet over six months.
The Starting Point: Why 40 Agents?
When leadership asked me to lead our "AI automation initiative," I figured we'd automate 5–8 high-volume workflows and call it a year. Clean, manageable, defensible.
Then the business saw the first results and said: more.
The first agent — customer support triage — reduced our Tier 1 response time by 73%. The invoice processing agent eliminated a full-time-equivalent data entry role. Each win unlocked appetite for the next. By Week 8, we had a backlog of 60 potential agent use cases and a team of four engineers trying to build them all.
We stopped treating it like a project and started treating it like a platform.
Architecture Decisions That Saved Us (and One That Cost Us)
The Good: Hub-and-Spoke with a Shared Memory Layer
We settled on a hub-and-spoke model where a central orchestration layer manages agent registration, heartbeat, and task routing. Each agent owns its domain but draws on shared services: a vector database for institutional memory, a message queue for async task passing, and a unified logging sink.
This sounds obvious in hindsight. It wasn't obvious at Month 1.
Early agents were point-to-point. Agent A talked directly to Agent B. When we hit 12 agents, the graph became unreadable and debugging was a nightmare. The refactor to hub-and-spoke took three weeks and was the single highest-ROI engineering decision of the project.
The Good: Agent Templates from Day One
We created a base agent template with mandatory interfaces: health_check(), process(task), report_status(). Every new agent inherits from this template. It added about two days of upfront work. It saved us weeks of inconsistency later.
The Bad: We Underestimated the Observability Gap
Here's the mistake I regret most: we treated agents like microservices, but they don't behave like microservices.
A microservice, when it fails, usually fails fast and loudly. An AI agent can meander. It can produce plausible-but-wrong outputs at 2 AM and not trigger any alert because it didn't error — it just... drifted.
We spent Month 4 building what we should have built in Month 1: behavioral assertions. Not just "did the agent run?" but "did the agent produce an output within expected bounds?" We now run every agent output through a lightweight validator before it touches production systems.
The Hard Parts Nobody Warns You About
0. What About Agent Safety and Rogue Outputs?
Before diving into the specific technical challenges, here's one that doesn't get enough attention: what happens when an agent produces a harmful output?
We built a three-layer defense:
- Input validation at the orchestration layer — sanitize and constrain all inputs before they reach the agent.
- Output behavioral assertions — run every output through a lightweight validator that checks for expected format, value ranges, and known-bad patterns before the output reaches downstream systems.
- Human-in-the-loop gates — for high-stakes actions (financial transactions, external API calls, customer-facing responses), route through a human approval step or at minimum a shadow mode before full automation.
Most teams focus on the AI model. The operational risk actually lives in the edges — the inputs you didn't sanitize and the outputs you didn't validate.
1. Agent-to-Agent Authentication Was a Nightmare
With 40 agents, you have roughly 780 potential communication paths. We couldn't trust any agent to call any other agent without verification. We ended up building a lightweight mutual TLS setup with short-lived certs issued by our orchestration hub. It works. It's not elegant.
We also had to build a lightweight cert distribution system because off-the-shelf solutions didn't handle short-lived certs for ephemeral agent containers — that's a non-trivial operational problem worth anticipating. If anyone has a better pattern, I'm all ears.
2. Context Window Management Is a Real Problem
Several of our agents handle multi-step reasoning tasks. Early versions would context-stuff: dump as much history as possible into each prompt. Token costs spiked. Latency spiked. Outputs got incoherent.
The fix was a "context budgeting" system: each agent has a fixed context allocation, and long-running conversations get summarized and compressed every N turns. We built this in-house. LangChain has some solutions here but nothing out-of-the-box matched our needs. Worth noting: summarization has its own latency cost — it's a valid solution, not a free one.
3. Versioning Agents Without Versioning Conversations
When you update an agent, what happens to the 200 conversations currently in flight? We initially had no answer. Agents would silently start using new logic mid-conversation, producing jarring inconsistencies.
We now implement a "conversation affinity" model: each active conversation is pinned to the agent version that started it. New conversations use the latest version. It adds operational complexity but eliminates the silent behavior drift problem.
4. The "Shadow Agent" Problem
Because agents can call other agents, we had several cases where Agent A would call Agent B, which would call Agent C, which would call Agent A — creating loops that consumed resources without producing output.
We added a recursion guard with a depth limit and a task graph visualization tool so we could see the full call tree. This took a weekend to build and saved us from several production incidents.
What Actually Worked
Structured Logging with Agent ID Propagation
Every log entry includes agent_id, task_id, parent_task_id, and a deterministic trace_version. When something goes wrong, we can reconstruct the full call chain in under a minute. This sounds table stakes for software engineering — it is. We just had to build it ourselves because off-the-shelf APM tools didn't understand agent hierarchies.
The Agent Runbook Registry
We maintain a Markdown registry of every agent's purpose, expected inputs, known failure modes, and rollback procedure. It's owned by the engineer closest to that agent domain. Every production incident gets a runbook entry within 48 hours. This reduced our mean time to recovery from "who knows" to about 18 minutes.
Blue-Green Agent Deployments
We run two parallel environments: blue (current stable) and green (next release). New agent versions deploy to green first, receive 5% of traffic, and only graduate to blue after 24 hours of error-rate parity. This eliminated the "deploy and pray" pattern that plagued our early months.
Code Generation as an Agent Template
Perhaps surprisingly, the most valuable agent we built was one that generates boilerplate for new agents. Given a YAML specification of inputs, outputs, and business rules, it produces a skeleton agent that conforms to our template. It reduced new agent onboarding from two weeks to two days. The irony of using an AI agent to bootstrap AI agents is not lost on me.
Metrics and Business Impact
I was skeptical of vanity metrics going into this project. I'm still skeptical. But here are the numbers leadership cares about:
- 40 agents deployed across customer support, operations, finance, and engineering
- $1.2M in annualized labor cost reduction (roles eliminated or repurposed)
- 73% reduction in Tier 1 support response time (first agent)
- 94% reduction in invoice processing errors (second agent)
- 2.1x increase in workflow throughput for automated processes
- ~$180K in infrastructure costs annually — covering compute, orchestration layer, vector database, message queue, and observability stack
The number that surprised me most: agent-to-agent communication now accounts for 31% of all automated decisions — meaning most of the value comes not from individual agents but from how they collaborate.
Advice for Teams Deploying AI Agents to Production
If you're about to embark on something similar, here's what I'd tell you:
1. Start with observability, not agents. Build your logging, tracing, and validation framework before you write your first agent prompt. Retrofitting observability into 20 agents is painful. Retrofitting it into 3 is trivial.
2. Treat agents as products, not projects. Each agent has a lifecycle: it needs an owner, a roadmap, a deprecation strategy. Assign clear ownership upfront.
3. Invest in the template and the registry early. The two-day upfront cost of an agent template and runbook registry pays back within the first month.
4. Plan for failure modes specifically, not just errors. AI agents produce wrong outputs that look right. Build validators. Build canaries. Build alerts for drift, not just crashes.
5. Don't scale to 40 agents the way we did — scale intentionally. Our growth was driven by business demand, which is a good problem to have. But we would have been smarter to cap at 15–20 and solidify our platform before expanding.
What I'd Do Differently
Honestly? I'd slow down in Month 3.
We hit what I call the "sprawl crisis" at Month 3 — too many agents, too little shared infrastructure, too many firefights. We spent Month 4 in stabilization mode, which was necessary but humbling.
If I could re-run those six months, I'd do a formal platform hardening sprint at Month 3 before greenlighting additional agents. Maybe we'd have shipped 35 instead of 40 by Month 6. We'd have shipped 40 with more confidence and fewer late-night pages.
The other thing: I'd bring in a specialized MLOps engineer two months earlier. I have software engineering instincts. My colleague Sarah has ML production instincts. Those are different skill sets, and the project needed both from roughly Month 2 onward.
Q&A: Expert Answers to Common Practitioner Questions
Q: What's the minimum team size to run 40 agents in production?
At minimum: 1–2 engineers who own the orchestration platform, 1 engineer per 8–10 agents for domain-specific development, 0.5–1 MLOps engineer for prompt engineering and behavioral validation, and 1 on-call rotation. For 40 agents, that's roughly 6–8 engineers at steady state — plus platform investment up front. Running 40 agents with a team of 4 only works if most agents are stable after initial deployment.
Q: How do you handle an agent that goes rogue or produces harmful outputs?
Three-layer defense: input validation at the orchestration layer, output behavioral assertions before downstream systems receive anything, and human-in-the-loop gates for high-stakes actions (financial transactions, external API calls, customer-facing responses).
Q: What would you use instead of building in-house?
Depends on use case. For task automation: Microsoft AutoGen, LangGraph, or crew.ai provide solid orchestration frameworks. For conversational agents: Dialogflow, Voiceflow, or Rasa for dialogue management. For observability: LangSmith is purpose-built for LLM/agent tracing — we'd have used it if it existed when we started. The in-house route is only worth it if you have specific requirements (security compliance, unique agent hierarchies, legacy system integration) that off-the-shelf solutions can't handle.
Q: What's the biggest misconception teams have about AI agent production deployment?
That the hard problem is the AI model. It isn't. The hard problems are orchestration (coordinating multiple agents reliably), observability (knowing what your agent did and why), versioning (managing agent updates without disrupting in-flight work), and failure modes (agents fail silently and plausibly, not loudly and obviously). Most teams spend 80% of their time on observability and versioning and are surprised by it.
Closing
Forty AI agents in production over six months sounds like a flex. It was, partly. But the real story is the mess in the middle — the architectural pivots, the observability gaps, the silent drifts, the 2 AM pages that taught us more than any conference talk ever could.
If you're building AI agents in production, my one piece of unfiltered advice: the agents are the easy part. The platform, the observability, the operational rigor — that's the hard part. Budget accordingly.
The retrospective I wish I'd had at the start.
Ready to build your own AI agent platform? Start with the runbook, not the prompt.