Inside a 40-Person Team's LLM Rollout: An Interview on AI Platform Ownership
How a 40-person team built an LLM platform: build vs. buy, AI platform ownership, shadow AI risk, and org design. A practical interview.
When a 40-person engineering organization decides to run its own LLM platform, the technical decisions get most of the attention. The organizational ones decide whether the rollout succeeds.
We spoke with Dana Whitfield, Director of Platform Engineering at a mid-market logistics software company, roughly six months after her team finished the third phase of an enterprise LLM adoption that began in early 2025. The company builds route-planning and warehouse-management software for regional carriers, and its engineering group sits at about 40 people — small enough that every headcount decision is visible, large enough that unmanaged AI usage had already become a governance problem.
This interview covers how they scoped the platform, who owns it, what broke, and what they'd change. The throughline: AI platform ownership is an org-design question as much as a technical one, and teams that treat it purely as infrastructure tend to rebuild the same governance gaps twice.
[ILLUSTRATION: Org chart showing the platform team's position relative to product engineering, data, and security, with the model gateway as a shared layer]
The Setup — Why a 40-Person Team Needed Its Own AI Platform
Walk me through the state of things before you started. What made this urgent?
We weren't starting from zero. By late 2024, maybe a third of the engineering team was already using LLM APIs for something — code review helpers, log summarization, a prototype that drafted customer-facing release notes. The problem was that none of it was visible. People had personal API keys on personal credit cards and were expensing them. One team had hardcoded a key into a CI job. Nobody knew the total spend, and nobody could answer the question "what customer data has gone through a third-party model this quarter?"
That last question is what escalated it. Our largest customer's security team asked it during a renewal review, and we couldn't answer it credibly.
The trigger: shadow AI usage and ad-hoc API keys
So the trigger was compliance, not capability.
Mostly, yes. We had three concrete risks: data leakage through unlogged API calls, runaway spend with no caps, and wildly inconsistent output quality because everyone was using different models with different prompts and no evaluation.
The spend one sounds minor until you see the numbers. We found an internal tool that was re-summarizing a document set on every dashboard refresh — no caching, and a retry loop that fired on transient errors without backing off. It burned low four figures in a month with zero users noticing, because the cost was invisible. That single finding did more to get the platform funded than any architecture deck.
Build vs. buy: the decision framework they used
How did you decide what to build?
We used a simple split: buy the commodity layer, build the differentiation. We bought model access directly from two providers and a managed observability tool for traces. We built the model gateway, the policy layer, and the internal self-serve portal, because those encode how our company handles data and approvals, and no vendor knows that.
The framework we actually applied was three questions:
- Does this touch our data residency or customer contracts? If yes, build or heavily configure.
- Is this a solved problem with a mature market? If yes, buy and don't look back.
- Will we need to change this every quarter? If yes, own it, because vendor roadmaps won't move at our pace.
The gateway passed all three. The observability tool passed only the second, and that was fine.
Who Owns the AI Platform? Org Design and Team Structure
This is the part most teams get wrong. How did you structure it?
We created a platform team of four people — about 10% of engineering, which was our judgment call for this scope, not a benchmark — with a named owner and a written charter. The charter was the single most useful artifact of the whole project, because it let us say no to things.
The platform team's charter and headcount allocation
The four roles:
- Platform lead (me, at 50% allocation): roadmap, vendor relationships, and the final call on policy.
- Two ML/platform engineers (full-time): the gateway, evals infrastructure, and the self-serve portal.
- A security liaison (25% allocation from the security team): PII handling, logging review, and incident response.
We also named a product owner for internal AI tooling — one of the platform engineers, part-time — because internal developers are users, and users need someone accountable for their experience.
The charter said three things explicitly: the platform team owns the gateway, the evaluation harness, and the shared prompt library. It does not own individual use cases. Product teams own their own use cases and are accountable for their own quality.
That boundary took two revisions to get right. The first version had the platform team reviewing every prompt, which made us a bottleneck within a month.
Where ownership sits: engineering, data, or a hybrid
Did you consider putting this under data?
We did, and we rejected it — not because data wasn't relevant, but because the platform's primary customers were software engineers, and we judged that sitting next to them was the fastest path to adoption. Our data team is three people and focused on the warehouse and BI. Making them own an LLM gateway would have meant a queue.
We landed on a hybrid: engineering owns the platform, data owns the evaluation datasets and quality review, security owns the policy. The tension point is real, though. Embedded use-case teams occasionally want to move faster than the gateway's release cadence, and we've had to be explicit that bypassing the gateway is a policy violation, not a shortcut.
We enforce that with controls, not documentation. Concretely: outbound traffic from build and runtime environments is allow-listed to the gateway only; direct provider API endpoints are blocked at the egress proxy; and developers do not get provider keys at all — the gateway issues scoped, budgeted credentials per team. If someone tries to call a provider directly, the request fails and we get an alert.
The mindset shift that made it work: treat the platform as a product with internal customers, not as a shared utility that people are lucky to have. That framing is well-worn in platform-engineering circles, but it's the one that changed how we prioritized.
Controlling spend and data flow
You named runaway spend as a risk. What does the control model look like now?
Three layers. First, attribution: every request carries a team and use-case tag, so spend rolls up per product team and per feature, not just per provider. Second, budgets: each team gets a monthly token budget on its scoped key, with alerts at 50%, 80%, and 100%, and hard cutoffs at the cap unless someone justifies an increase. Third, caching and dedup: the gateway caches identical prompts within a window, which killed the class of bug that started this whole thing.
On data flow, the gateway logs every request with a redaction pass for PII before anything leaves our boundary. That's what let us answer the customer security question — we can produce a per-quarter report of which models saw which data classes.
Evaluation and Quality: What the Harness Actually Does
You called the evaluation harness a platform-owned artifact. What does it evaluate?
The harness is the part we under-invested in first and regret most. It does two jobs: regression testing on prompt and model changes, and per-use-case quality tracking that product teams own.
For regression: each use case has a golden dataset — a few dozen to a few hundred inputs with expected or graded outputs — checked into the repo alongside the prompt. When a prompt changes or we swap a model version, the change runs through CI against that dataset, and a drop beyond a threshold blocks the merge. That gate is what stops silent quality regressions when a provider ships a model update.
For quality tracking: product teams own their own eval sets and their own thresholds; the platform team owns the harness, the grading tooling, and the reporting. We use a mix of exact-match and rubric-based grading depending on the task, and we keep a human review sample because automated grading has its own failure modes.
What broke?
Two things. Early on, teams treated the golden datasets as write-once and never refreshed them, so the evals drifted away from real traffic — we now require a quarterly refresh from sampled production inputs, redacted. And our first grading setup was too lenient; it passed outputs that a human reviewer would have rejected. Tightening the rubrics was unglamorous work that mattered more than any model upgrade.
What They'd Change
Six months in, what would you do differently?
Three things. Start the eval harness on day one, not phase two — we shipped faster early and paid for it in quality regressions. Write the charter before the architecture, because the boundary disputes cost more time than any technical decision. And instrument cost from the first request, because invisible spend is how you end up explaining a four-figure surprise to finance.
The thing I'd keep: the decision to make the platform a product with named internal customers. It's the reason adoption didn't require a mandate.
"The moment we started treating internal developers as customers with a right to a good experience — and not as people who should be grateful for a gateway — the adoption problem solved itself. The mandate was never the lever. The product was."
Dana Whitfield is Director of Platform Engineering at a mid-market logistics software company. Details have been reviewed for accuracy; the company name is withheld at its request.