Feature store: from notebook features to reliable production ML

A model works in the notebook and fails in production, quietly, returning worse predictions than the offline evaluation promised. The usual culprit is not the model but the features: the values it sees in production are computed differently from the ones it trained on. A feature store exists to close that gap. This article covers what it solves, how offline and online serving differ, and when a feature store is worth the operational cost.

What a feature store solves

A feature is an input to a machine learning model: a customer's average order value, the number of failed logins in the last hour, the age of an account. In a notebook, features are computed with a few lines of code against historical data. In production, the same features must be computed against live data, fast, for every prediction. When those two computations drift apart, the model degrades in ways nobody can see.

A feature store centralizes feature definitions so they are computed once and consumed everywhere. It stores the historical values for training and serves the current values for inference, from the same definition. That single source of truth is the whole point: it removes an entire class of silent production failures.

Offline and online: the two halves

A feature store has two stores with different jobs, and understanding the split is understanding the tool.

The offline store holds the full history of feature values, typically in a data lake or warehouse. It is optimized for throughput: building a training dataset means reading millions of historical feature values at once. Latency does not matter here; completeness and correctness do.

The online store holds only the latest value of each feature, in a low-latency key-value store (Redis, DynamoDB or similar). It is optimized for a single job: given an entity id, return its current features in a few milliseconds so a live prediction can be made in real time.

The feature store keeps these two in sync from one definition, so the feature a model trained on offline is exactly the feature it is served online. That synchronization is the mechanism that eliminates training-serving skew.

Point-in-time correctness and data leakage

The subtlest bug in machine learning is data leakage: training a model on information that would not have been available at prediction time, which inflates offline accuracy and collapses in production.

Feature stores address this with point-in-time correct joins. When building a training set for an event that happened on a given date, the store returns the feature values as they were on that date, not their current values. Without this, a model predicting churn might accidentally train on a feature updated after the customer already churned, learning from the future. Point-in-time correctness is a defining capability of a real feature store, and doing it by hand in SQL is error-prone enough that it is a strong reason to use one.

Reuse and governance

The second justification for a feature store is organizational, not technical. In a company with several models, the same features get reinvented by every team: one team's "active user" is subtly different from another's, computed in a different pipeline, tested differently.

A feature store makes features a shared, discoverable, versioned asset. A feature is defined once, documented, owned, and reused across models. That reduces duplicated pipelines, makes features auditable (which matters when a regulator asks how a decision was made), and lets a new model start from proven inputs rather than rebuilding them. This is where the feature store connects to broader data governance: features become governed data products, not scattered notebook code.

When you need one, and when you do not

A feature store is infrastructure, and infrastructure has a cost. It is not the right first move for every project.

You probably do not need one when there is a single model, features are few and computed in batch, and there is no real-time serving requirement. A well-managed table in the warehouse does the job, and adding a feature store is complexity without payoff.

You likely do need one when several models share features, when you serve predictions online with tight latency, when point-in-time correctness is hard to guarantee by hand, or when features must be governed and audited. The open-source Feast is a common starting point; managed options like Tecton or the feature stores built into Databricks and Vertex AI trade control for less operational burden.

The decision follows the same logic as the rest of the ML platform: adopt the capability when the problem demands it, not because it appears on an architecture diagram.

Failure patterns

From the field, the patterns that undermine feature infrastructure:

1. Training-serving skew ignored. Features computed one way offline and another online, so the model silently underperforms in production.

2. Leakage from naive joins. Training on current feature values instead of point-in-time values, inflating offline metrics.

3. A feature store for one model. Heavy infrastructure adopted before reuse or online serving justifies it.

4. No ownership. Features in the store with no accountable owner, drifting out of date until consumers quietly distrust them.

5. Online store as an afterthought. Building offline features first and discovering at deployment that serving them in milliseconds is a separate, unsolved problem.

Talk through your ML feature infrastructure

DNA Solutions helps European enterprises put machine learning into reliable production: feature stores where they are justified, point-in-time correct training data, synchronized online serving, and governed, reusable features instead of scattered notebook code. Whether you are fighting training-serving skew or deciding whether a feature store is worth it at all, we size the infrastructure to the models you actually run. Talk to us.