Data quality monitoring for ML pipelines means validating every training batch and every serving window against a schema derived from training data: column presence and type, null rate, value domain, freshness, row volume and distribution. Make it a blocking gate before training and a sampled, alerting monitor at serving. Great Expectations, Deequ and TFDV cover the checks.
The failure this catches is not a crash. An upstream team renames a column or starts writing 0 where it used to write NULL, the DAG goes green, and the model scores every request with a value it never saw in training. Latency and error rate stay flat; the first signal is a business metric a week later. Google’s Data Validation for Machine Learning paper (Breck et al., MLSys 2019) opens on this gap: research attention goes to training and inference, far less to the quality of the data fed to them.
Data quality is a different question from drift. Drift asks whether the world changed; data quality asks whether the pipeline broke. Both move input statistics, which is why ETL bugs get routed to the model owner. Types of data drift covers the population shifts a statistical test is built for. The checks below target breakage that should never reach the model.
What should data quality monitoring for ML pipelines check?
Six checks, each compared against a reference: the training schema, the training statistics or the pipeline schedule. Schema, completeness, validity, freshness and volume are deterministic and cheap, so run them on every batch. Distribution checks are statistical and noisier, and belong one tier lower in alert severity. Training-serving skew is a seventh check with its own method.
| Check | Measures against the reference | Catches |
|---|---|---|
| Schema | Column presence, dtype, values per feature | Renamed column, int arriving as string |
| Completeness | Null ratio per feature | Join-key mismatch emptying a feature |
| Validity | Share of values inside the training domain | Unit change, unseen category, sentinel -1 |
| Freshness | Now minus newest event timestamp | Stalled upstream job, stale features |
| Volume | Row count vs the same slot last week | Missing partition, duplicated backfill |
| Distribution | Histogram distance (PSI, JS, L-infinity) | Re-bucketing, a default swallowing a category |
The table cannot see skew: the same entity’s features computed by the batch and online paths, diffed row by row. That method is in training-serving skew detection.
The metric that matters: out-of-domain rate
Put the per-feature out-of-domain rate on the dashboard next to the null ratio. Job success and row count are the obvious candidates, and both miss the common failure.
out_of_domain_rate(f, w) = |{ x in w : x_f is not null and x_f not in D_f }| / |w|
null_ratio(f, w) = |{ x in w : x_f is null }| / |w|
D_f is the domain recorded in the training schema: a [min, max] range for numerics, the observed category set for categoricals, a pattern for IDs. w is the batch or serving window. Alert on the null ratio’s change against the reference, not its absolute value, since some features are legitimately sparse.
Why it beats the alternatives:
- Job success measures code, not data. A renamed column that silently becomes all-null raises no exception.
- Row count catches a missing partition and misses corruption inside a full-size batch.
- PSI or Jensen-Shannon divergence detects the change but blends breakage with real population movement, and its magnitude depends on binning. PSI says something moved; out-of-domain rate says what fraction of rows carry values the model never saw. That maps straight to a runbook step: find the upstream change that introduced them.
The gap is sentinels. Great Expectations’ missing-data guide notes that a NULL account number and an account number of 0 mean different things. A pipeline that starts imputing 0 passes the null check and may sit inside the numeric domain. Only a validity rule that excludes the sentinel, or a distribution check, catches it.
Where do the checks run in the pipeline?
Checks run at three points: a blocking gate on training data before a fit starts, a blocking or quarantining gate on each batch of materialized features, and a non-blocking sampled monitor on serving traffic. Rejecting requests at serving usually costs more than scoring them with a degraded feature, so serving checks alert rather than block.
Pick the library by where the data lives:
- Spark tables: Deequ writes “unit tests for data” (
hasSize,isComplete,isUnique,isContainedIn), with PyDeequ as the Python wrapper. The VLDB 2018 paper behind it translates constraints into Spark aggregation queries, computes metrics incrementally as data grows, and runs anomaly detection on the quality-metric time series. - pandas and SQL warehouses: Great Expectations (GX Core 1.x), where each expectation takes a
mostlyfraction for tolerated failures. - TFX pipelines: TensorFlow Data Validation infers a schema from statistics, supports environments (a label present in training, absent at serving), and ships skew and drift comparators: L-infinity distance for categorical features, Jensen-Shannon divergence for numeric or categorical ones. Breck et al. describe the validation system inside TFX, which Google reports hundreds of product teams use to validate several petabytes of production data per day.
- Reports: Evidently’s DataSummaryPreset with
include_tests=Truederives test conditions such as min-max ranges from a reference dataset, and falls back to heuristics like “no missing values” without one.
Best drift detection libraries for Python compares the statistical layer; SentryML covers the observability-platform side.
Wiring it up
A daily feature batch, validated with GX Core, gating the training job and pushing results to a Prometheus Pushgateway. The thresholds are examples; derive yours from the training reference and review them like code.
import great_expectations as gx
import pandas as pd
from prometheus_client import CollectorRegistry, Gauge, push_to_gateway
PIPELINE = "fraud_features_daily"
# Domains and tolerances come from the training reference.
EXPECTATIONS = [
gx.expectations.ExpectColumnValuesToNotBeNull(column="txn_amount", mostly=0.999),
gx.expectations.ExpectColumnValuesToBeBetween(
column="txn_amount", min_value=0, max_value=50_000
),
gx.expectations.ExpectColumnValuesToBeInSet(
column="merchant_country", value_set=["US", "CA", "GB", "DE", "FR"], mostly=0.99
),
gx.expectations.ExpectColumnValuesToNotBeNull(column="account_age_days"),
]
def validate_batch(df: pd.DataFrame) -> bool:
context = gx.get_context()
batch = (
context.data_sources.add_pandas(name="features")
.add_dataframe_asset(name=PIPELINE)
.add_batch_definition_whole_dataframe("daily")
.get_batch(batch_parameters={"dataframe": df})
)
registry = CollectorRegistry()
passed = Gauge("ml_dq_expectation_passed", "1 if the expectation passed",
["pipeline", "expectation", "column"], registry=registry)
null_ratio = Gauge("ml_feature_null_ratio", "Null fraction per feature",
["pipeline", "feature"], registry=registry)
rows = Gauge("ml_batch_rows", "Rows in the batch", ["pipeline"], registry=registry)
newest = Gauge("ml_batch_max_event_unixtime", "Newest event timestamp",
["pipeline"], registry=registry)
ok = True
for exp in EXPECTATIONS:
result = batch.validate(exp)
passed.labels(PIPELINE, type(exp).__name__, exp.column).set(int(result.success))
ok = ok and result.success
for col in df.columns:
null_ratio.labels(PIPELINE, col).set(float(df[col].isna().mean()))
rows.labels(PIPELINE).set(len(df))
newest.labels(PIPELINE).set(df["event_ts"].max().timestamp())
push_to_gateway("pushgateway:9091", job=PIPELINE, registry=registry)
return ok
if __name__ == "__main__":
features = pd.read_parquet("s3://feature-store/fraud/daily/latest.parquet")
if not validate_batch(features):
raise SystemExit("data quality gate failed; training not started")
The Pushgateway exists for batch jobs Prometheus cannot scrape. Use a dedicated CollectorRegistry so process metrics from the default registry stay out of the push.
Alert rules, with deterministic failures paging and statistical moves opening a ticket:
groups:
- name: ml-data-quality
rules:
- alert: FeatureExpectationFailed
expr: ml_dq_expectation_passed == 0
labels:
severity: page
annotations:
summary: "{{ $labels.expectation }} failed on {{ $labels.column }}"
- alert: FeatureBatchStale
expr: time() - ml_batch_max_event_unixtime > 26 * 3600
labels:
severity: page
- alert: FeatureNullRatioStep
expr: (ml_feature_null_ratio - ml_feature_null_ratio offset 7d) > 0.02
for: 1h
labels:
severity: ticket
The offset 7d comparison keeps day-of-week seasonality from firing the null-ratio rule.
What you’ll see
Healthy charts are boring. Null ratios run as flat lines with small noise around the reference, out-of-domain rate sits near zero with rare blips, row count follows the weekly traffic shape, and freshness resets on every run.
The shapes that mean trouble:
- A step at one timestamp across several features from the same source table. Upstream deploy or schema change. Read the upstream commit log before touching the model.
- One feature’s null ratio steps up alone. A broken join, or a source that stopped populating one field.
- Out-of-domain rate creeping up over weeks on one categorical. New categories arriving from a product launch or a new market. That is drift, not breakage; route it to the model owner and widen the domain deliberately.
- Freshness that stops resetting. The upstream job stalled and features are stale while every task reports success.
- All green, model metric down. The checks pass on the wrong thing. Look at skew, then at monitoring without ground truth.
Caveats
- Inferred schemas overfit the snapshot. TFDV’s
infer_schemauses conservative heuristics, and the domain it learns is the training sample’s. Commit the schema file, review diffs, and change domains on purpose. - Tolerances drive false alarms. Without
mostly, a single null fails the expectation. Set tolerances from each metric’s historical variance and tier alerts as described in SLOs and alerting for ML systems. - Cardinality blowup. The Prometheus naming guide warns that every unique combination of label pairs is a new time series. Never label by entity ID or raw category value; keep per-value breakdowns in the batch report, not the TSDB.
- Sampling cost. Full scans on large tables are compute you pay every run. Deequ’s incremental metrics help on partitioned data; at serving, sample and store the sample size with each result.
- Dropping rows creates skew and leakage. If the training gate drops failing rows while serving scores them, the model never learned the rows it now sees. Derive domains and imputation values from the training split only; statistics computed over eval or later windows leak into the baseline and flatter offline metrics.
- Validation is not a security control. Checks catch accidents. Poisoned data crafted to stay inside ranges and null budgets passes them; AISec covers that threat model.
FAQ
what is the difference between data quality and data drift
Data quality failures are broken data; data drift is real data that changed. A renamed column, a null-filled join or a unit change is a quality failure and gets fixed upstream. A new segment shifting a feature’s distribution is drift and may require retraining. Both move input statistics, so route alerts by shape: step changes point to breakage.
which tool is best for data quality checks in ml pipelines
Pick by where your data lives: Deequ or PyDeequ for Spark tables, Great Expectations for pandas and SQL warehouses, TFDV for TFX pipelines with built-in schema inference and skew comparators. Evidently adds readable reports with reference-derived tests. All four are open source, so runtime integration decides, not license cost.
should data quality checks block model training
Yes, deterministic checks should block training. A failed schema, completeness or volume check means the fit would learn from broken data, and a model trained on a missing partition can pass offline evaluation on the same broken split. Distribution checks are noisier; let them open a ticket for review instead of hard-failing the run.
how often should you run data quality checks on ml features
Run them on every batch, at the cadence the batch is produced, since the checks are aggregation queries over data the pipeline already reads. For streaming or online features, evaluate a sampled window on a fixed interval matched to how fast a bad value would hurt, and record the sample size alongside every result.
Related across the network
- Model Monitoring in Production: Metrics and Triggers — sentryml.com
- Machine Learning Pipeline: Stages, Code, and Failure Gates — sentryml.com