Traditional ETL (Extract, Transform, Load) moves data for dashboards and reports. ML pipelines have different requirements: they need point-in-time correctness (features must reflect what was known at prediction time, not the future), reproducibility (same input → same features every time), dual serving (same features for training and inference), and freshness (real-time features for online predictions). Common orchestrators: Apache Airflow (the standard for batch pipelines), Prefect (modern Python-native alternative), Dagster (asset-based, great for data-aware orchestration). For streaming: Apache Kafka + Flink or Spark Structured Streaming.

Training-serving skew occurs when the features used during training differ from those used during inference. This happens when: training uses a Python script but serving uses a Java service (different implementations of the same logic), training uses historical data but serving uses real-time data (different data sources), or training computes features in batch but serving computes them on-the-fly (different computation paths). Even tiny differences — like a different rounding method or timezone handling — can cause significant prediction errors. Feature stores solve this by providing a single source of truth for features in both training and serving.

A feature store is a centralized system that manages the lifecycle of ML features: definition, computation, storage, and serving. It has three layers: (1) Transformation layer — defines how features are computed from raw data (SQL, Spark, or Python). (2) Storage layer — an offline store (data warehouse/lake for training) and an online store (low-latency key-value store for serving). (3) Serving layer — APIs to retrieve features for both training (batch) and inference (real-time). The key guarantee: the same feature definition produces the same value in both training and serving, eliminating training-serving skew.

Feast (Feature Store) is an open-source feature store originally developed at Gojek and now maintained by Tecton. It provides: feature definitions in Python (declarative), offline serving via point-in-time joins against your data warehouse, online serving via Redis/DynamoDB/SQLite for low-latency lookups, and materialization (syncing features from offline to online store). Feast is lightweight — it doesn’t run its own compute; it integrates with your existing data infrastructure. You define features, Feast handles the rest: storage, serving, and consistency.

Tecton is a managed feature platform (commercial, founded by the creators of Feast) that goes beyond Feast in several ways: built-in compute (Tecton runs the transformations; Feast relies on external compute), real-time features (stream processing from Kafka/Kinesis with sub-second latency), aggregation engine (optimized time-windowed aggregations like “average spend in last 30 days”), and feature monitoring (drift detection on feature values). Tecton is the right choice for teams that need real-time streaming features, enterprise SLAs, and don’t want to manage infrastructure. Feast is better for teams that want open-source flexibility and already have compute infrastructure.

Bad data is the #1 cause of ML failures in production. Data validation catches issues before they reach the model: Great Expectations (the most popular open-source data validation framework — define “expectations” like “column X should never be null” or “values should be between 0 and 1”), Pandera (lightweight schema validation for Pandas DataFrames), TensorFlow Data Validation (TFDV) (detects anomalies and schema drift), and dbt tests (SQL-based data quality checks in your transformation pipeline). Run these checks at every stage: ingestion, transformation, and before training.

Features fall into three categories by freshness: Batch features (computed daily/hourly from the data warehouse — e.g., “average spend in last 30 days”). Streaming features (computed in real-time from event streams — e.g., “number of transactions in last 5 minutes”). On-demand features (computed at request time from the input — e.g., “is this IP address in a known VPN list?”). Most ML systems use a mix: 80% batch features (cheap, easy) + 15% streaming features (for freshness-sensitive signals) + 5% on-demand features (for request-specific context). Start with batch; add streaming only when freshness is proven to improve model performance.

One of the biggest ROI drivers of a feature store is feature reuse. Without a feature store, every team independently computes “average spend in last 30 days” — with slightly different implementations. With a feature store, the feature is defined once, computed once, and shared across all models. This reduces: duplicated compute (one pipeline instead of five), inconsistency (one definition, one result), and onboarding time (new models can browse and reuse existing features). A feature catalog (like a data catalog but for features) lets data scientists search for existing features before building new ones.