ML Monitoring Report
Connected cyan cylindrical nodes float above a blue grid platform beside an upright interface panel with colored dots and lines.
drift-detection

Data Drift Detection in ML: Methods, Tests, and Practice

Statistical tests, reference windows, multivariate checks, threshold tuning, and when a data drift alert should actually trigger a model retrain.

By ML Monitoring Report Editorial · · Updated · 13 min read

Data drift detection is the practice of comparing the distribution of a model’s live inputs against a reference sample, usually the training data, and flagging features whose distributions have moved past a set threshold. Each feature gets a statistical test or distance (KS, chi-square, PSI, Wasserstein, Jensen-Shannon); results roll up into a dataset-level verdict that opens an investigation. No labels are needed, so it is usually the first production alarm a team builds. If you are choosing between KS, PSI, and Wasserstein for a given feature type, the Drift Test Selector recommends a test from your data shape and detection goal.

A pipeline can continue serving plausible predictions after its input mix changes. Input-distribution monitoring provides evidence for investigation while outcomes are still unavailable. Pair it with monitoring models without ground truth so a changed distribution is not mistaken for measured performance loss.

What Is Data Drift in Machine Learning

Data drift is a change in the distribution of a model’s input features between training time and inference time: the joint or marginal distribution P(X) has shifted. The term is used loosely, so precision helps. The model learned a mapping from some P_train(X) to an output space; if P_prod(X) diverges significantly, predictions land outside the model’s reliable operating range.

This is distinct from concept drift, where the relationship between inputs and the target changes, meaning P(Y|X) shifts rather than P(X). Both matter, but they require different monitoring strategies. Data drift can be detected without access to labels; concept drift detection usually requires ground truth feedback with some lag. The full comparison, including prediction drift, is in concept drift vs data drift explained.

A third category, label drift (shifts in P(Y)), is relevant in classification tasks but is often conflated with concept drift. In practice, most teams start with input feature monitoring and add label-based signals once they have a reliable feedback loop. For the full set, including the schema, unit, and missingness drifts that come out of your own pipeline rather than the world, see types of data drift.

Drift detection vs model drift detection

Drift detection is the umbrella term for any monitor that compares a current window of data, predictions, or errors against a reference and alerts on a significant change. What it watches decides whether you need labels:

Monitored signalWhat shiftsLabels neededTypical detector
Data driftInput distribution P(X)NoKS, chi-square, PSI, Wasserstein, Jensen-Shannon
Prediction driftModel output distributionNoThe same distances applied to scores
Concept driftInput-to-target relationshipYes, often delayedDDM, ADWIN, Page-Hinkley on the error stream
Model driftAccuracy, MAE, other metricsYes, or an estimateRealized metrics, or NannyML’s CBPE and DLE estimators

Without labels, model drift detection relies on estimation, and NannyML’s CBPE documentation says the estimator stays accurate under data drift but will not work under concept drift.

How to Detect Data Drift, Step by Step

Detecting data drift in production comes down to seven steps:

  1. Log what the model saw: post-transformation features with timestamp, model version, and transformation version.
  2. Version a fixed reference: training data, or a stable production window covering a full business cycle.
  3. Set the current window and a minimum row count; skip windows and segments below it rather than scoring them.
  4. Match the test to column type and sample size. P-value tests such as KS suit small samples and distances suit large ones; the methods and tools sections below give the defaults.
  5. Store the score, method, threshold, and reference version, not only a drifted flag.
  6. Aggregate: share of drifting columns, separate rules for critical features, and a multivariate check.
  7. Alert on persistence, then investigate. A single noisy window should not page anyone.

A minimal per-column check with SciPy’s ks_2samp:

import pandas as pd
from scipy.stats import ks_2samp

ref = pd.read_parquet("reference_v3.parquet")        # versioned reference
cur = pd.read_parquet("serving_2026-10-07.parquet")  # current window

res = ks_2samp(ref["txn_amount"].dropna(), cur["txn_amount"].dropna())
print(res.statistic, res.pvalue)

With the default method='auto', SciPy attempts an exact p-value only when both samples have fewer than 10,000 rows. The dropna() hides a rising null rate, and so does Evidently, which filters empty values before scoring drift. Track null share as its own metric.

The Three Decisions That Define a Drift Detector

Every drift detection setup, whether it is Evidently, NannyML, Arize, Fiddler, or a cron job you wrote yourself, reduces to three choices.

Reference window. What are you comparing against? The common options are a frozen snapshot of the training data or a trailing production window such as the last 30 days. Training-data references catch divergence from what the model actually learned; trailing references catch sudden change while tolerating slow seasonal movement. Many teams run both. Using the full training dataset is common but has a known weakness: training data is often preprocessed and filtered in ways that make it a poor proxy for live production traffic, while a rolling baseline can mask gradual drift by continuously shifting the reference point.

Comparison test. For each feature you need a way to score the gap between reference and current. Numeric features typically get a two-sample test or a distance metric; categorical features usually get a binned divergence such as PSI or Jensen-Shannon distance.

Aggregation. Per-feature results roll up into a dataset-level verdict, most commonly “share of features drifting,” plus an optional multivariate check. Univariate tests miss correlation changes. NannyML’s multivariate method fits PCA on the reference data (capturing about 65% of variance by default), reconstructs incoming batches through that compressed representation, and treats a rise in reconstruction error as drift. In their example, a three-feature “butterfly” dataset rotated 90 degrees about the z-axis leaves every univariate test green while reconstruction error jumps.

For high-dimensional inputs like images or embeddings, the empirical evidence says reduce first, then test. The NeurIPS 2019 study Failing Loudly compared shift-detection pipelines across perturbation types and reported that two-sample testing on top of a learned dimensionality reduction, notably the softmax outputs of the pretrained classifier itself, detected shift best. The paper’s framing is the whole discipline in one phrase: models fail silently on out-of-distribution data unless you build the alarm yourself.

Data Drift Detection Methods: Statistical Tests Compared

Choose a statistic based on the feature type and the question. The Evidently comparison illustrates that methods can react differently to the same shift; its experimental results are not universal alert thresholds. In that vendor benchmark (first published in 2022, updated July 2025), KS flagged a 0.5% shift once a dataset passed 100,000 objects, while PSI registered significant change at 100,000 observations only above a 10% shift.

MethodTypical inputWhat to inspect
KS two-sample testContinuous valuesBoth the empirical-CDF distance and the sample-sensitive p-value
PSIFixed bins or category proportionsBin definitions, smoothing, and the size of the distribution change
Wasserstein distanceNumeric valuesFeature units and any normalization applied
Jensen-Shannon distanceCategory or bin proportionsConsistent support, logarithm base, and implementation
Chi-square testCategorical countsExpected counts and treatment of rare categories

SciPy’s KS reference defines the statistic and its hypothesis test. Statistical significance alone is not a measurement of prediction harm. Require enough observations for the intended test and choose an operationally meaningful effect threshold.

The PSI formula compares reference and current bin shares. Fix bins from the reference and document empty-bin handling. Conventional cutoffs are starting points to evaluate, not universal retraining rules. PSI vs CSI distinguishes a score-distribution check from the same calculation on each characteristic.

Raw Wasserstein distance uses the input feature’s units; it is not simply the change in the median. Normalized implementations rescale it, so record that definition before comparing columns. Jensen-Shannon is symmetric and finite where raw KL divergence can be undefined, but its numeric range depends on the logarithm base.

Separate effect size from the alert policy

Use choosing drift metrics: PSI, KS and calibration to distinguish distribution checks from outcome-quality checks. Even distance estimates vary with the sample and binning. Calibrate thresholds on representative stable windows and replay known incidents if available.

A dataset-level drifting-column share can summarize movement. Retain separate checks for critical features: a single broken required input can matter even when the aggregate share is small. Assign ownership and response severity in the ML model monitoring framework.

Data Drift Detection Tools and Their Defaults

Every tool runs the same reference-versus-current comparison, but default statistics and thresholds differ, so a 0.1 in one tool is not a 0.1 in another. Defaults below were checked against vendor documentation in October 2026.

ToolTypeDrift statisticsDefault alert rule
EvidentlyPython libraryKS, chi-square, z-test up to 1,000 reference rows; Wasserstein, Jensen-Shannon abovep ≤ 0.05, or distance ≥ 0.1
NannyMLPython libraryKS, Jensen-Shannon, Wasserstein, Hellinger, chi-square, L-infinity; PCA reconstruction errorReconstruction error beyond 3 standard deviations of the reference mean
Alibi DetectPython libraryFeature-wise KSChosen p-value, Bonferroni or FDR corrected
Azure Machine LearningManaged serviceJensen-Shannon, PSI, normalized Wasserstein, KS, Pearson’s chi-squaredConfigurable per metric
Vertex AI Model Monitoring v1Managed serviceJensen-Shannon (numeric), L-infinity (categorical)0.3 for every feature
SageMaker Model MonitorManaged serviceMaximum CDF difference; L-infinity or chi-squared for categoricals0.1

Vertex AI’s v2 monitoring, tied to a model version, is in Preview and its overview lists no default threshold. SageMaker Model Monitor is closed to new customers; existing users keep it, but AWS plans no new features. Library trade-offs are compared in best drift detection libraries for Python.

How to Monitor Data Drift in Production

Define the current window, reference version, model version, and transformation version before scheduling the job. Batch systems can compare complete periods; streaming systems require explicit choices about window size and the order in which observations arrive. Longer windows can hide a brief change by mixing it with unaffected traffic.

Segment the data where behavior or consequences differ. A dataset-wide distance can conceal movement within a product route, device type, or customer cohort. Check sample sufficiency within each segment as well as across the full dataset.

Scope the monitor deliberately. With training data as the reference, Azure Machine Learning can limit drift monitoring to the top N features by importance, and it tracks null, data type error, and out-of-bounds rates as a separate data quality signal. Run data quality monitoring beside drift; per-feature thresholds and alert tiers are covered in feature drift monitoring best practices.

The current Evidently report example shows the data wrappers, report execution, and HTML/JSON exports. Use it as the implementation entry point; this page focuses on detector design. Evidently’s drift documentation explains its column-type and sample-size defaults. Persist the method and threshold with every result so reports remain comparable after a configuration change.

Compare the actual post-transformation features seen by the model. A production-versus-production baseline can miss a persistent mismatch with training. Add training-serving skew detection using matched inputs through both computation paths.

What to inspect after an alert

  • One feature moves. Inspect its upstream source, encoding, missingness, and affected segments.
  • Several related features move together. Check shared joins and transformations, then consider a changed population.
  • Movement accumulates over successive windows. Compare with the fixed reference as well as a rolling one, and evaluate whether labeled performance has changed.

None of these patterns uniquely identifies a root cause. Retain deployment records and feature summaries with the alert to make the diagnosis reproducible.

When to Act on a Drift Signal

A drift alert should trigger one of three responses, not automatically a full retraining cycle.

Investigate first. Confirm the drift is real and not an artifact of a data pipeline issue: a schema change upstream, a missing feature value getting imputed differently, a change in how the feature is computed. Silent pipeline bugs and genuine distributional shifts produce identical statistical signatures.

Assess impact before retraining. If the drifting feature is not among the top contributors to model predictions (check feature importance or SHAP values), the drift may be ignorable in the short term. Trigger retraining when drift in high-importance features exceeds thresholds, not when any feature moves.

Rule out the other mechanism. Input drift and a changed feature-to-label relationship look nothing alike in their fix but similar on a dashboard. If accuracy is falling while these input tests stay flat, the detectors described here are the wrong instrument entirely and you want the error-rate and performance-estimation methods in concept drift detection: DDM, ADWIN, and Page-Hinkley instead.

Choose the retraining strategy. Evaluate recent labeled data after confirming its quality. A drift threshold can open a retraining evaluation; it should not by itself approve replacing the serving model.

Embedding-store reliability: monitor retrieval separately

An embedding store introduces a second question: whether approximate search is still returning the neighbors intended by the retrieval design. Availability and latency alone cannot answer it. Weaviate’s vector-index documentation describes how search parameters trade recall against speed.

Use a representative query sample with exact nearest-neighbor results over a specified corpus snapshot. Define ANN recall@k as the overlap between the approximate and exact top-k sets, divided by k, and aggregate across queries. Match filters, distance metric, and corpus version. This measures agreement with exact vector search, not human relevance or the correctness of the generated answer.

A suggested monitoring panel separates four failure classes:

Failure classMeasurementDiagnostic use
ANN recall regressionRecall@k against exact searchCompare index settings, corpus changes, and search parameters
Stale or mixed embedding versionsEncoder-version coverage and indexing backlogIdentify documents encoded with an incompatible model or awaiting refresh
Dimension or preprocessing mismatchVector length, norm distribution, distance metricCompare ingestion and query encoder configurations
Corpus or query distribution changeFixed-model embedding summaries and query-to-result similarity distributionsLocate changing topics or segments for relevance review

Pinecone’s index setup documentation specifies the dimension and similarity metric when bringing vectors. Record the encoder and preprocessing version as additional application metadata. Equal dimensions alone do not establish compatibility. A changed norm distribution is a diagnostic clue; cosine similarity does not inherently require callers to supply unit-length vectors.

For an encoder migration, a conservative runbook is to prepare a separate index, backfill it, and check version coverage and retrieval quality before routing queries to it. Keep the prior encoder and index paired until cutover; retain them for rollback. Dual writes can keep both corpora current during the transition when the application supports that arrangement. These are proposed operating controls, not measured vendor guarantees.

Track a sampled query-to-result similarity histogram alongside the recall benchmark. A score-distribution shift can help locate a retrieval change but does not prove the results became less relevant. Recompute exact benchmarks when the corpus snapshot changes rather than comparing against obsolete neighbors.

Keep service telemetry too. Milvus documents monitoring with Prometheus and Grafana; those operational panels complement the retrieval-quality checks above. Carry encoder and index version IDs into outcome analysis so a failed answer can be traced to the retrieval version that supplied its context.

The distinction between retrieval degradation and generation quality is developed in monitoring tabular models vs LLM systems.

Caveats

Drift is not damage. Input movement can occur without lower accuracy, while quality can decline without detectable marginal input drift. Pair distribution checks with suitable outcome monitoring.

Binning changes the question. PSI needs consistent support and an explicit smoothing policy. High-cardinality identifiers may be better monitored through missingness, cardinality, and new-value rates than thousands of sparse bins.

Multiple comparisons need a policy. Testing many features repeatedly increases the chance of false alarms. Use statistical correction where appropriate, practical effect thresholds, and a selected set of actionable alerts. Multivariate checks complement marginal tests; neither replaces all the other checks.

Reference changes must be explicit. Keep a record of why the baseline was refreshed. A rolling window answers a different question from a fixed training reference and can conceal cumulative movement.

FAQ

which test is best for detecting data drift

No single test is best; it depends on sample size and the shift you care about. KS and chi-square suit small samples, roughly under 1,000 rows. On large windows, Wasserstein for numeric and Jensen-Shannon for categorical features avoid flagging trivial shifts. Keep PSI where risk teams already report it, with thresholds set from history.

what threshold indicates data drift

No universal threshold exists, and tools ship different defaults. Evidently flags Wasserstein or Jensen-Shannon scores of 0.1 or more on references over 1,000 rows, and p-values of 0.05 or less on smaller ones. Vertex AI Model Monitoring v1 uses 0.3, SageMaker Model Monitor 0.1. Calibrate per feature on known-stable windows.

how often should you run data drift checks

Run drift checks as often as enough new data accumulates for a stable comparison. Azure Machine Learning recommends daily monitoring when daily traffic is sufficient, otherwise weekly or monthly. Set a minimum row count per window and segment, keep reference and current windows from overlapping, and match the cadence to how fast someone can respond.

Sources

  1. Which test is the best? Comparing 5 drift detection methods on large datasets — Evidently AI
  2. Measuring Data Drift with the Population Stability Index — Fiddler AI
  3. Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift (NeurIPS 2019)
  4. Multivariate Drift Detection via PCA Reconstruction Error — NannyML docs
  5. scipy.stats.ks_2samp — SciPy reference
  6. Data drift metrics documentation — Evidently AI
  7. Vector indexing — Weaviate documentation
  8. Create an index — Pinecone documentation
  9. Deploy monitoring services — Milvus documentation
  10. Univariate drift detection — NannyML documentation
  11. Performance estimation (CBPE and DLE) — NannyML documentation
  12. Kolmogorov-Smirnov drift detector (KSDrift) — Alibi Detect documentation
  13. Model monitoring in production — Azure Machine Learning, Microsoft Learn
  14. Introduction to Vertex AI Model Monitoring — Google Cloud documentation
  15. Schema for constraints (constraints.json) — Amazon SageMaker Model Monitor
Subscribe

ML Monitoring Report — in your inbox

Production ML monitoring, drift, and reliability. Sent only when there is something worth sending.

No spam. Unsubscribe anytime.

Related