Machine Learningfeature storeML infrastructureonline offline skewfeature engineering

Feature Stores After the Hype: When You Actually Need One (And When You Don't)

Feature stores aren't mandatory. Use these concrete signals to decide when a feature store earns its place in your ML stack—and when it doesn't.

The Feature Store Hype Cycle, in Retrospect

Feature stores were sold as mandatory infrastructure. For a stretch between 2020 and 2022, it was hard to read an ML platform architecture diagram that didn't have one wedged between the data warehouse and the model serving layer. Every serious company, the pitch went, needed a feature store — the same way it needed a data warehouse or a CI/CD pipeline.

Three years on, the picture is messier. Adoption is uneven. Some teams run feature stores at the center of their platform and would not give them up. Others bought one during the peak, wired up a handful of features, and now pay for seats and compute that mostly sit idle. A few ripped theirs out entirely and went back to warehouse tables and a thin serving layer.

The arc is familiar to anyone who has watched ML infrastructure mature. Enthusiasm peaked around 2021: a wave of vendors, eye-watering funding rounds, and conference talks that treated "we have a feature store" as a maturity milestone. Then came consolidation — acquisitions, quiet shutdowns, and open-source projects that stopped shipping releases. What's left is a quieter utility phase, where feature stores are one component among many rather than the centerpiece of a platform.

The pattern repeats across ML infrastructure. Tools get adopted for status and architecture-diagram appeal before teams have the pain they solve. Vector databases followed a similar curve. So did MLOps platforms, and so will whatever the next category is. This is less a failure of the tools than of the buying process: it's easier to evaluate a product demo than to diagnose your own bottlenecks.

This article isn't a victory lap for skeptics or a defense of the category. It's a calibration exercise. What follows is a concrete, signal-based test for whether a feature store earns its place in your stack, drawn from what the technology actually does rather than what vendors say it does. (Related: what an ML platform actually needs to include.)

The question isn't whether feature stores are good technology. It's whether you have the specific pain they were built to treat.

What a Feature Store Actually Does

Strip away the marketing and a feature store is a system with two halves and a consistency guarantee between them.

Core capabilities

  • Offline store: historical feature values for training, typically backed by a warehouse or data lake, with point-in-time correctness so training rows reflect only data that would have been available at prediction time.
  • Online store: low-latency key-value serving for real-time inference, usually a Redis-class store.
  • Feature definitions and pipelines: a single definition that materializes to both stores.
  • Feature discovery and reuse: a catalog so teams can find existing features instead of rebuilding them.
  • Lineage and monitoring: tracking where features come from and detecting drift or freshness problems.

What it is not

A feature store is not a model registry, not an orchestrator, not a replacement for your data platform or warehouse, and not a fix for bad data quality. It doesn't clean your data, and it won't make a leaky label pipeline correct. If your upstream tables are wrong, a feature store will serve wrong values faster and with better lineage. (See model registry vs. feature store for where the boundary sits.)

The online/offline consistency problem, stated precisely

This is the core technical justification, so it's worth stating plainly. In training, you compute features from historical data in batch — a SQL query over months of events. In serving, you compute the same features in real time from a stream or a key-value lookup. Unless both paths are built from the same definition and the same logic, they drift. A model trained on one distribution gets served another, and accuracy quietly degrades in production.

Point-in-time correctness is subtler than "only use data available at prediction time." The failure mode most teams miss is late-arriving data. Suppose you build a training set on Monday using a feature table as it looked Monday, then a backfill lands Wednesday that retroactively changes values for events from two weeks ago. Your labels were computed against the Monday snapshot; your next retrain uses the Wednesday snapshot. Same query, different answers, drifted model — with no code change to blame. A correct PIT join keys on the event timestamp of each row and reconstructs the state of the feature as of each label's timestamp, not the state of the table when the query ran. That distinction between event time and processing time is where most "we have PIT joins" implementations actually break.

A feature store's job is to make that drift structurally difficult rather than a thing you catch in a postmortem. Note the hedge: a single definition reduces skew, it does not eliminate it. Logic can still diverge inside a shared definition — different windowing in a streaming vs. batch path, different null handling, different timezone assumptions. What a feature store buys you is one place to get it right and one place to look when it's wrong.

Diagram showing a single feature definition materializing to both an offline store for training and
Diagram showing a single feature definition materializing to both an offline store for training and

When You Actually Need a Feature Store

Need scales with model count and team count, not company size. A 5,000-person company with one ML team and three models is a weaker candidate than a 200-person company with four teams shipping a dozen models. Here are the observable signals. Treat each as a heuristic, not a gate.

Signal 1: Online/offline skew is causing production incidents

You'll know it when a model performs well in offline evaluation and then underperforms in production, and the root cause turns out to be a feature computed differently in the two paths. The classic version: a batch job fills a feature with the day's final value, but real-time serving sees a partial-day value at prediction time.

This is the highest-value problem a feature store solves. If you've had repeated skew-related incidents — say, two or more in the past year — and your fix each time was "add a test and hope," you have the pain. (Background: detecting training-serving skew.)

Signal 2: Multiple models share features across teams

You'll know it when two teams independently build the same "customer 30-day order count" feature, with slightly different definitions, and nobody notices until a model comparison goes sideways. The tell is a meeting where someone asks "which order count?" and there's more than one answer.

Feature reuse is where the ROI math starts to work. If you have more than three teams building models on overlapping entities — users, accounts, transactions, products — you are almost certainly paying for duplicated feature engineering. The cost isn't just engineering hours; it's the reconciliation work when two teams' features disagree and someone has to decide which is right.

Signal 3: You need real-time features and have no serving layer

If your models are batch-scored nightly, you may not need a feature store at all — warehouse tables and a scheduled job will do. The moment you need sub-100ms feature lookups at request time, you need an online store, and the question becomes whether to build a thin one yourself or adopt a feature store that gives you one for free. Building a Redis-backed lookup service is a week of work; keeping it consistent with your training pipeline for two years is the actual cost.

Signal 4: Feature definitions are living in notebooks and tribal knowledge

If the canonical definition of a feature lives in one senior engineer's head, and every new model re-derives it from scratch, you have a discovery and governance problem. A catalog with lineage solves it. Note that this signal alone can often be addressed with a lightweight internal registry and documentation discipline — you don't always need the full system.

When You Don't Need One

The honest counter-case, which vendors won't make for you:

  • One team, a handful of models, batch scoring. Warehouse tables plus scheduled jobs are simpler, cheaper, and easier to debug. A feature store adds a dependency and an operational surface for no payoff.
  • Your skew problem is really a data quality problem. If features are wrong because upstream events are malformed or late, a feature store will faithfully serve the wrong values. Fix the pipeline first.
  • You only need an online store. A managed key-value store with a well-tested write path is a legitimate answer. You're buying consistency guarantees, not a database.
  • You're pre-product-market-fit. The feature store is a scaling tool. Adopting it before you have multiple teams and models is architecture-diagram appeal, which is exactly how the 2021 wave happened.

What to Actually Evaluate If You Buy

If the signals above point to "yes," the evaluation is concrete:

  • Online store latency and freshness. What's the p99 lookup latency at your expected QPS? Is freshness streaming, micro-batch, or batch? The difference matters more than the vendor's benchmark.
  • Point-in-time join correctness under late data. Ask specifically how the system handles backfills and event-time vs. processing-time. Most demos skip this.
  • Cost at your cardinality. Online store cost scales with entity count and update frequency. Model it against your actual entity counts, not a demo dataset.
  • Exit cost. Can you export feature definitions and historical values cleanly? A feature store you can't leave is a feature store that owns your roadmap.

The Decision

Ask one question: do I have a specific, recurring pain that a feature store was built to treat? If you have skew incidents, cross-team feature duplication, and a real-time serving requirement, the answer is yes, and the technology will pay for itself. If you have one of those, adopt the narrowest tool that solves it. If you have none, skip it — and revisit when the pain shows up. The category is fine. The buying process is what needs calibration.

ShareX / TwitterLinkedIn
← Back to Learn