From Pilot to Production: How a Fintech Scaled LLM Agents Across 40 Teams

The Pilot That Almost Died

Eighteen months ago, this fintech's LLM agent program was three teams deep and losing altitude. The pilot worked — sort of. A reconciliation agent shaved hours off back-office work, and a support triage assistant deflected routine tickets. But every new team that wanted in hit the same wall: no shared tooling, no evaluation standard, no compliance story. Each integration was a bespoke science project.

Then the CFO asked a question that changed everything: "If it works, why aren't forty teams using it?"

That question kicked off a rebuild that took the company from a stalled pilot to a production LLM agent platform serving 40 teams across engineering, operations, risk, and customer support — inside a regulated environment where real customer money and real auditors are always in the room.

We sat down with the VP of AI Platform Engineering (name withheld at the company's request) to unpack how they did it, what broke, and what enterprise teams should steal.

"We didn't scale an agent. We scaled a way of building agents — and that's a completely different problem."

Why This Fintech Went All-In on LLM Agents

The original pain was unglamorous. Back-office reconciliation, KYC document review, and support triage consumed thousands of human hours monthly, and error rates crept up as volume grew. Simple LLM features — summarization, classification — helped at the margins but couldn't own an end-to-end task.

Agents could. The work was inherently multi-step, tool-dependent, and stateful: pull records, cross-reference policy, flag exceptions, draft a response, route for approval. A chatbot can't do that. An agent with the right tools and boundaries can.

The Original Pilot: Scope, Team, and Success Metrics

The first pilot was deliberately small: six engineers, two product managers, one risk partner, and a 90-day clock. Scope was a single reconciliation workflow with a narrow tool surface — read from three internal systems, write to one queue.

Success criteria were concrete and measured against a defined baseline — the prior manual process, on the same case mix, over a 90-day evaluation window:

Task completion rate above 85% on a golden set of 500 historical cases, stratified across exception types (not a uniform random sample)
Human override rate below 20%, where an override is defined as any case where a human materially edited the agent's output or escalated it to the manual queue before completion
Median handling time cut by 40% versus the pre-pilot manual baseline
Zero material compliance findings, with all escalations triaged and resolved within the review cycle

They hit the numbers. They also discovered that "hitting the numbers in a lab" and "surviving contact with 40 teams" are different sports. A 500-case golden set is enough to validate a single workflow, but it is not enough to certify one for regulated production at scale — the team learned to widen and stratify the set before any workflow went live.

The Moment Leadership Asked for 40 Teams

The trigger wasn't a competitor or a board mandate. It was proven unit economics. The pilot's cost-per-task — inclusive of model tokens, gateway overhead, human review, and the eval harness itself — came in below the fully loaded human cost, with headroom. Once finance saw that curve, the ask inverted: instead of justifying expansion, the platform team had to justify not expanding.

The mandate that followed was blunt — make this available to 40 teams within two quarters. No new headcount for the platform team. That constraint forced every architectural and organizational decision that came next.

The Architecture That Survived Scale

The stack they landed on is boring on purpose. Boring survives audits.

Model gateway — a single choke point for all model calls, routing, rate limits, and provider failover. No team talks to a model provider directly.
Orchestration layer — the agent runtime: planning, tool invocation, memory, and retries.
Retrieval layer — permission-aware document and data retrieval, scoped per team.
Tool registry — every tool versioned, owned, and reviewed.
Observability — traces, token accounting, and eval hooks on every run.

The gateway was the single most important decision. It gave them one place to enforce policy, one place to swap models, and one place to measure cost — without touching 40 codebases. It also became the enforcement point for data residency: when a workflow was barred from external providers, the gateway rejected the route before a single token left the perimeter.

Agent Orchestration and Tool Boundaries

Early agents had broad tool access. That failed fast. A general-purpose agent with twenty tools hallucinated calls, picked the wrong API, and produced unexplainable actions.

The fix: narrow tool surfaces. Each agent gets the smallest set of tools that completes its task — often three or four. Tools are typed, validated, and reject malformed input at the boundary. If an agent needs a new capability, it's a reviewed pull request, not a prompt tweak.

"The most reliable agent we shipped had exactly three tools. The least reliable had twenty. That correlation was not a coincidence."

To be clear, that is one company's observation, not a universal law — but it aligns with published findings that tool-selection accuracy degrades as the number of available tools grows. Treat tool count as a design budget, not a convenience.

Evaluation Harness: Evals Before Features

This is the rule the VP calls non-negotiable: no agent ships without an eval suite.

Their harness has three layers:

Golden datasets — curated historical cases with known-good outcomes, versioned alongside code.
Regression suites — run on every prompt, model, or tool change; a drop blocks the merge.
Canary releases — new versions serve 5% of traffic behind a metric gate before full rollout.

The cultural shift was harder than the technical one. Teams used to ship prompts like config changes. Now a prompt change is a release artifact with a diff, a reviewer, and a rollback plan.

[ILLUSTRATION: Diagram of the agent platform stack — model gateway, orchestration, retrieval, tool registry, and observability layers feeding 40 downstream teams]

Governance, Risk, and Compliance in a Regulated Environment

Financial services governance isn't a checkbox — it's a design constraint. Data residency rules meant some agents could never call external models. Explainability requirements meant every decision needed a traceable rationale. Model risk management meant every model version was a governed artifact, not a floating endpoint.

The platform team's strategy was to make compliance a property of the platform, not a review gate bolted on at the end. Four mechanisms did the work:

Policy-as-code at the gateway. Residency, allowed-provider, and data-classification rules lived in versioned policy files, enforced on every call. A workflow that couldn't legally leave the perimeter simply had no route to an external model — enforced in code, not in a wiki.
A model registry with pinned versions. Every model and prompt version was registered, hashed, and tied to an approval record. Nothing reached production unpinned. When a provider deprecated a model, the registry surfaced every affected workflow before the cutoff, not after.
Immutable audit trails. Every agent run logged its inputs, retrieved sources, tool calls, model version, and output rationale to append-only storage. When an auditor asked "why did the agent flag this transaction," the answer was a replayable trace, not a reconstruction.
Human-in-the-loop gates by risk tier. Low-risk actions (drafting, routing) ran autonomously. Anything that moved money, changed a customer record, or triggered a regulatory report required a human approval step — with the agent's reasoning attached.

The result was not "zero escalations" — that isn't how regulated environments work. It was that escalations were expected, triaged, and traceable, and the program produced no material compliance findings across its production period.

The Organizational Model That Made 40 Teams Possible

No new headcount meant the platform team couldn't be a service desk. It became a product team with a paved road: a self-serve path where a new team could stand up a governed agent without platform engineers in the loop. Enablement ran through templates, reference agents, and a review board that approved tool surfaces and risk tiers — not through bespoke integrations.

The trade was explicit. Teams got speed and a compliance story for free. In return, they accepted the gateway, the registry, and the eval gate as non-negotiable. The constraint that looked like a limitation became the reason 40 teams could move at once without 40 separate risk reviews.

What Enterprise Teams Should Steal

Build the gateway first. One choke point for routing, policy, and cost is worth more than any single agent.
Make evals a merge gate, not a milestone. If a prompt change can't fail a build, it isn't governed.
Shrink tool surfaces until they hurt. Three well-typed tools beat twenty loosely scoped ones.
Treat compliance as platform behavior. Policy-as-code, a model registry, and immutable traces turn audit from a scramble into a query.
Turn constraints into the product. "No new headcount" is what forced a paved road — and the paved road is what scaled.

The pilot didn't scale. The platform did. That distinction is the whole story.