Feature Store Architecture for Production Machine Learning

A feature store is a system that publishes versioned feature definitions and serves the same values to training jobs and online inference, so models do not train on one history and predict on another. Production teams add one when feature logic starts to fork between notebooks and serving paths, not because a vendor checklist said every ML platform needs one.

The architecture is a data product with two read paths. The offline path produces point-in-time correct training sets. The online path returns low-latency values for a live request. If those paths do not share definitions, the store is a cache with a misleading name.

This article covers when you need a store, how to split offline and online, and which controls keep features auditable on private infrastructure.

When a Feature Store Earns Its Complexity

A warehouse view or a set of documented SQL jobs is enough when one team owns one model and the serving path can recompute features from the same source. A feature store starts to pay off when several models reuse the same entities, when serving cannot wait for a warehouse query, or when auditors ask which definition produced a score last Tuesday.

The failure mode that usually triggers the project is training-serving skew. A notebook engineers a rolling average with one window and one null policy. The service implements a faster approximation. Accuracy drops in production and nobody can replay the training join. The store exists to make that replay possible.

Skip the platform if you do not yet have shared entities, a second consumer, or a latency constraint. Buying software will not create those conditions.

Offline and Online Paths

Point-in-time correctness is the offline path's only non-negotiable. For each training label timestamp, the join may use only feature values that existed at or before that time. A naive join against current tables leaks the future and produces models that cannot be reproduced.

The online path trades historical completeness for latency. It stores the latest published value per entity key. That is sufficient for most scoring and insufficient for training. Do not train from the online store unless you are deliberately accepting that limitation and documenting it.

Storage, Compute, and Freshness

Feature stores fail operationally when people treat them as a single database. Offline materialization is a batch compute problem: scans, aggregations, and partitioned writes. Online serving is a lookup problem: hot keys, memory, and tail latency. Size them separately.

Freshness is a product decision. A fraud model may need event-updated balances within seconds. A weekly churn model may tolerate a nightly batch. Publish a freshness SLO per feature family and measure it. A store that cannot tell you how stale a key is will be distrusted within a quarter.

On dedicated infrastructure, keep feature compute next to the data boundary that already holds the sources. Moving raw events to a SaaS feature platform just to get a registry UI recreates the residency problem you solved for model training. Open-source options such as Feast, or a thin registry over warehouse tables, can run on private AI infrastructure beside the models that consume them.

Governance That Models Can Pin

A usable registry records owner, source query or transform, entity keys, value type, version, and deletion or retention rules. Models then pin a version set. "Always use production" is not a pin; it is a moving target.

Access control belongs on the entity, not only on the table. The same customer key can feed a marketing model and a credit model with different legal bases. Online stores are especially easy to over-expose because they look like generic key-value caches to application teams.

For regulated data, treat feature values as derived personal data. They inherit retention, access logging, and erasure obligations from the source. If a source record must be deleted, the offline partitions and the online key must be in the deletion map. That is storage work, which is why feature serving belongs in AI storage architecture conversations rather than only in ML platform conversations.

How the GPU Layer Connects

Feature stores do not replace GPU scheduling. They change GPU efficiency. Training jobs that wait on leaked or rebuilt features waste reserved accelerators. Serving jobs that compute heavy features on the request path steal GPU time from the model.

Precompute what you can, keep request-time compute small, and schedule training materialization so GPUs are not blocked on warehouse scans. The OnePlus Platform, OneSource Cloud's AI orchestration platform, is the layer that then assigns GPUs to the training jobs that consume those tables. Feature freshness and GPU utilization are one pipeline, even if two teams own the tickets.

FAQ

Do we need a feature store for machine learning?

You need one when multiple models share entities, when serving cannot recompute features in time, or when you must reproduce a historical training set. A single model with simple features can start with documented warehouse jobs and add a store later.

What is the difference between an offline and online feature store?

The offline path builds point-in-time training sets. The online path returns the latest published values for a live key. They must share definitions. They almost never share the same storage engine.

Can we use a data warehouse as a feature store?

Yes for offline features, if you enforce point-in-time joins and versioned definitions. Warehouses are usually the wrong online store because lookup latency and concurrency do not match scoring SLOs.

How does a feature store affect GPU cost?

Indirectly. Correct, precomputed features reduce wasted training reruns and keep request-time work off the GPU. A store that is chronically stale or skewed creates silent accuracy debt and extra training cycles.

Should a feature store run on private infrastructure?

If the source data cannot leave your boundary, the derived features cannot either. Run the registry and the online store in the same control domain as the models. Managed feature services are viable only after residency and deletion review.

Summary

A production feature store is a shared definition layer with an offline training path and an online serving path. Build it when skew, reuse, or latency make ad-hoc SQL unsafe. Enforce point-in-time joins, pin versions on models, and treat feature values as regulated data when the source is regulated. Keep GPU scheduling and feature freshness on the same operating calendar.

If features and models must stay in one U.S. boundary, OneSource Cloud can host the training and serving GPUs beside your store. Request an architecture review to map feature freshness SLOs onto reserved capacity.