Data drift detection is the practice of comparing the distribution of a model’s live inputs against a reference sample, usually the training data, and flagging features whose distributions have moved past a set threshold. Each feature gets a statistical test or distance (KS, chi-square, PSI, Wasserstein, Jensen-Shannon); results roll up into a dataset-level verdict that opens an investigation. No labels are needed, so it is usually the first production alarm a team builds. If you are choosing between KS, PSI, and Wasserstein for a given feature type, the Drift Test Selector recommends a test from your data shape and detection goal.
A pipeline can continue serving plausible predictions after its input mix changes. Input-distribution monitoring provides evidence for investigation while outcomes are still unavailable. Pair it with monitoring models without ground truth so a changed distribution is not mistaken for measured performance loss.
What Is Data Drift in Machine Learning
Data drift is a change in the distribution of a model’s input features between training time and inference time: the joint or marginal distribution P(X) has shifted. The term is used loosely, so precision helps. The model learned a mapping from some P_train(X) to an output space; if P_prod(X) diverges significantly, predictions land outside the model’s reliable operating range.
This is distinct from concept drift, where the relationship between inputs and the target changes, meaning P(Y|X) shifts rather than P(X). Both matter, but they require different monitoring strategies. Data drift can be detected without access to labels; concept drift detection usually requires ground truth feedback with some lag. The full comparison, including prediction drift, is in concept drift vs data drift explained.
A third category, label drift (shifts in P(Y)), is relevant in classification tasks but is often conflated with concept drift. In practice, most teams start with input feature monitoring and add label-based signals once they have a reliable feedback loop. For the full set, including the schema, unit, and missingness drifts that come out of your own pipeline rather than the world, see types of data drift.
Drift detection vs model drift detection
Drift detection is the umbrella term for any monitor that compares a current window of data, predictions, or errors against a reference and alerts on a significant change. What it watches decides whether you need labels:
| Monitored signal | What shifts | Labels needed | Typical detector |
|---|---|---|---|
| Data drift | Input distribution P(X) | No | KS, chi-square, PSI, Wasserstein, Jensen-Shannon |
| Prediction drift | Model output distribution | No | The same distances applied to scores |
| Concept drift | Input-to-target relationship | Yes, often delayed | DDM, ADWIN, Page-Hinkley on the error stream |
| Model drift | Accuracy, MAE, other metrics | Yes, or an estimate | Realized metrics, or NannyML’s CBPE and DLE estimators |
Without labels, model drift detection relies on estimation, and NannyML’s CBPE documentation says the estimator stays accurate under data drift but will not work under concept drift.
How to Detect Data Drift, Step by Step
Detecting data drift in production comes down to seven steps:
- Log what the model saw: post-transformation features with timestamp, model version, and transformation version.
- Version a fixed reference: training data, or a stable production window covering a full business cycle.
- Set the current window and a minimum row count; skip windows and segments below it rather than scoring them.
- Match the test to column type and sample size. P-value tests such as KS suit small samples and distances suit large ones; the methods and tools sections below give the defaults.
- Store the score, method, threshold, and reference version, not only a drifted flag.
- Aggregate: share of drifting columns, separate rules for critical features, and a multivariate check.
- Alert on persistence, then investigate. A single noisy window should not page anyone.
A minimal per-column check with SciPy’s ks_2samp:
import pandas as pd
from scipy.stats import ks_2samp
ref = pd.read_parquet("reference_v3.parquet") # versioned reference
cur = pd.read_parquet("serving_2026-10-07.parquet") # current window
res = ks_2samp(ref["txn_amount"].dropna(), cur["txn_amount"].dropna())
print(res.statistic, res.pvalue)
With the default method='auto', SciPy attempts an exact p-value only when both samples have fewer than 10,000 rows. The dropna() hides a rising null rate, and so does Evidently, which filters empty values before scoring drift. Track null share as its own metric.
The Three Decisions That Define a Drift Detector
Every drift detection setup, whether it is Evidently, NannyML, Arize, Fiddler, or a cron job you wrote yourself, reduces to three choices.
Reference window. What are you comparing against? The common options are a frozen snapshot of the training data or a trailing production window such as the last 30 days. Training-data references catch divergence from what the model actually learned; trailing references catch sudden change while tolerating slow seasonal movement. Many teams run both. Using the full training dataset is common but has a known weakness: training data is often preprocessed and filtered in ways that make it a poor proxy for live production traffic, while a rolling baseline can mask gradual drift by continuously shifting the reference point.
Comparison test. For each feature you need a way to score the gap between reference and current. Numeric features typically get a two-sample test or a distance metric; categorical features usually get a binned divergence such as PSI or Jensen-Shannon distance.
Aggregation. Per-feature results roll up into a dataset-level verdict, most commonly “share of features drifting,” plus an optional multivariate check. Univariate tests miss correlation changes. NannyML’s multivariate method fits PCA on the reference data (capturing about 65% of variance by default), reconstructs incoming batches through that compressed representation, and treats a rise in reconstruction error as drift. In their example, a three-feature “butterfly” dataset rotated 90 degrees about the z-axis leaves every univariate test green while reconstruction error jumps.
For high-dimensional inputs like images or embeddings, the empirical evidence says reduce first, then test. The NeurIPS 2019 study Failing Loudly compared shift-detection pipelines across perturbation types and reported that two-sample testing on top of a learned dimensionality reduction, notably the softmax outputs of the pretrained classifier itself, detected shift best. The paper’s framing is the whole discipline in one phrase: models fail silently on out-of-distribution data unless you build the alarm yourself.
Data Drift Detection Methods: Statistical Tests Compared
Choose a statistic based on the feature type and the question. The Evidently comparison illustrates that methods can react differently to the same shift; its experimental results are not universal alert thresholds. In that vendor benchmark (first published in 2022, updated July 2025), KS flagged a 0.5% shift once a dataset passed 100,000 objects, while PSI registered significant change at 100,000 observations only above a 10% shift.
| Method | Typical input | What to inspect |
|---|---|---|
| KS two-sample test | Continuous values | Both the empirical-CDF distance and the sample-sensitive p-value |
| PSI | Fixed bins or category proportions | Bin definitions, smoothing, and the size of the distribution change |
| Wasserstein distance | Numeric values | Feature units and any normalization applied |
| Jensen-Shannon distance | Category or bin proportions | Consistent support, logarithm base, and implementation |
| Chi-square test | Categorical counts | Expected counts and treatment of rare categories |
SciPy’s KS reference defines the statistic and its hypothesis test. Statistical significance alone is not a measurement of prediction harm. Require enough observations for the intended test and choose an operationally meaningful effect threshold.
The PSI formula compares reference and current bin shares. Fix bins from the reference and document empty-bin handling. Conventional cutoffs are starting points to evaluate, not universal retraining rules. PSI vs CSI distinguishes a score-distribution check from the same calculation on each characteristic.
Raw Wasserstein distance uses the input feature’s units; it is not simply the change in the median. Normalized implementations rescale it, so record that definition before comparing columns. Jensen-Shannon is symmetric and finite where raw KL divergence can be undefined, but its numeric range depends on the logarithm base.
Separate effect size from the alert policy
Use choosing drift metrics: PSI, KS and calibration to distinguish distribution checks from outcome-quality checks. Even distance estimates vary with the sample and binning. Calibrate thresholds on representative stable windows and replay known incidents if available.
A dataset-level drifting-column share can summarize movement. Retain separate checks for critical features: a single broken required input can matter even when the aggregate share is small. Assign ownership and response severity in the ML model monitoring framework.
Data Drift Detection Tools and Their Defaults
Every tool runs the same reference-versus-current comparison, but default statistics and thresholds differ, so a 0.1 in one tool is not a 0.1 in another. Defaults below were checked against vendor documentation in October 2026.
| Tool | Type | Drift statistics | Default alert rule |
|---|---|---|---|
| Evidently | Python library | KS, chi-square, z-test up to 1,000 reference rows; Wasserstein, Jensen-Shannon above | p ≤ 0.05, or distance ≥ 0.1 |
| NannyML | Python library | KS, Jensen-Shannon, Wasserstein, Hellinger, chi-square, L-infinity; PCA reconstruction error | Reconstruction error beyond 3 standard deviations of the reference mean |
| Alibi Detect | Python library | Feature-wise KS | Chosen p-value, Bonferroni or FDR corrected |
| Azure Machine Learning | Managed service | Jensen-Shannon, PSI, normalized Wasserstein, KS, Pearson’s chi-squared | Configurable per metric |
| Vertex AI Model Monitoring v1 | Managed service | Jensen-Shannon (numeric), L-infinity (categorical) | 0.3 for every feature |
| SageMaker Model Monitor | Managed service | Maximum CDF difference; L-infinity or chi-squared for categoricals | 0.1 |
Vertex AI’s v2 monitoring, tied to a model version, is in Preview and its overview lists no default threshold. SageMaker Model Monitor is closed to new customers; existing users keep it, but AWS plans no new features. Library trade-offs are compared in best drift detection libraries for Python.
How to Monitor Data Drift in Production
Define the current window, reference version, model version, and transformation version before scheduling the job. Batch systems can compare complete periods; streaming systems require explicit choices about window size and the order in which observations arrive. Longer windows can hide a brief change by mixing it with unaffected traffic.
Segment the data where behavior or consequences differ. A dataset-wide distance can conceal movement within a product route, device type, or customer cohort. Check sample sufficiency within each segment as well as across the full dataset.
Scope the monitor deliberately. With training data as the reference, Azure Machine Learning can limit drift monitoring to the top N features by importance, and it tracks null, data type error, and out-of-bounds rates as a separate data quality signal. Run data quality monitoring beside drift; per-feature thresholds and alert tiers are covered in feature drift monitoring best practices.
The current Evidently report example shows the data wrappers, report execution, and HTML/JSON exports. Use it as the implementation entry point; this page focuses on detector design. Evidently’s drift documentation explains its column-type and sample-size defaults. Persist the method and threshold with every result so reports remain comparable after a configuration change.
Compare the actual post-transformation features seen by the model. A production-versus-production baseline can miss a persistent mismatch with training. Add training-serving skew detection using matched inputs through both computation paths.
What to inspect after an alert
- One feature moves. Inspect its upstream source, encoding, missingness, and affected segments.
- Several related features move together. Check shared joins and transformations, then consider a changed population.
- Movement accumulates over successive windows. Compare with the fixed reference as well as a rolling one, and evaluate whether labeled performance has changed.
None of these patterns uniquely identifies a root cause. Retain deployment records and feature summaries with the alert to make the diagnosis reproducible.
When to Act on a Drift Signal
A drift alert should trigger one of three responses, not automatically a full retraining cycle.
Investigate first. Confirm the drift is real and not an artifact of a data pipeline issue: a schema change upstream, a missing feature value getting imputed differently, a change in how the feature is computed. Silent pipeline bugs and genuine distributional shifts produce identical statistical signatures.
Assess impact before retraining. If the drifting feature is not among the top contributors to model predictions (check feature importance or SHAP values), the drift may be ignorable in the short term. Trigger retraining when drift in high-importance features exceeds thresholds, not when any feature moves.
Rule out the other mechanism. Input drift and a changed feature-to-label relationship look nothing alike in their fix but similar on a dashboard. If accuracy is falling while these input tests stay flat, the detectors described here are the wrong instrument entirely and you want the error-rate and performance-estimation methods in concept drift detection: DDM, ADWIN, and Page-Hinkley instead.
Choose the retraining strategy. Evaluate recent labeled data after confirming its quality. A drift threshold can open a retraining evaluation; it should not by itself approve replacing the serving model.
Embedding-store reliability: monitor retrieval separately
An embedding store introduces a second question: whether approximate search is still returning the neighbors intended by the retrieval design. Availability and latency alone cannot answer it. Weaviate’s vector-index documentation describes how search parameters trade recall against speed.
Use a representative query sample with exact nearest-neighbor results over a specified corpus snapshot. Define ANN recall@k as the overlap between the approximate and exact top-k sets, divided by k, and aggregate across queries. Match filters, distance metric, and corpus version. This measures agreement with exact vector search, not human relevance or the correctness of the generated answer.
A suggested monitoring panel separates four failure classes:
| Failure class | Measurement | Diagnostic use |
|---|---|---|
| ANN recall regression | Recall@k against exact search | Compare index settings, corpus changes, and search parameters |
| Stale or mixed embedding versions | Encoder-version coverage and indexing backlog | Identify documents encoded with an incompatible model or awaiting refresh |
| Dimension or preprocessing mismatch | Vector length, norm distribution, distance metric | Compare ingestion and query encoder configurations |
| Corpus or query distribution change | Fixed-model embedding summaries and query-to-result similarity distributions | Locate changing topics or segments for relevance review |
Pinecone’s index setup documentation specifies the dimension and similarity metric when bringing vectors. Record the encoder and preprocessing version as additional application metadata. Equal dimensions alone do not establish compatibility. A changed norm distribution is a diagnostic clue; cosine similarity does not inherently require callers to supply unit-length vectors.
For an encoder migration, a conservative runbook is to prepare a separate index, backfill it, and check version coverage and retrieval quality before routing queries to it. Keep the prior encoder and index paired until cutover; retain them for rollback. Dual writes can keep both corpora current during the transition when the application supports that arrangement. These are proposed operating controls, not measured vendor guarantees.
Track a sampled query-to-result similarity histogram alongside the recall benchmark. A score-distribution shift can help locate a retrieval change but does not prove the results became less relevant. Recompute exact benchmarks when the corpus snapshot changes rather than comparing against obsolete neighbors.
Keep service telemetry too. Milvus documents monitoring with Prometheus and Grafana; those operational panels complement the retrieval-quality checks above. Carry encoder and index version IDs into outcome analysis so a failed answer can be traced to the retrieval version that supplied its context.
The distinction between retrieval degradation and generation quality is developed in monitoring tabular models vs LLM systems.
Caveats
Drift is not damage. Input movement can occur without lower accuracy, while quality can decline without detectable marginal input drift. Pair distribution checks with suitable outcome monitoring.
Binning changes the question. PSI needs consistent support and an explicit smoothing policy. High-cardinality identifiers may be better monitored through missingness, cardinality, and new-value rates than thousands of sparse bins.
Multiple comparisons need a policy. Testing many features repeatedly increases the chance of false alarms. Use statistical correction where appropriate, practical effect thresholds, and a selected set of actionable alerts. Multivariate checks complement marginal tests; neither replaces all the other checks.
Reference changes must be explicit. Keep a record of why the baseline was refreshed. A rolling window answers a different question from a fixed training reference and can conceal cumulative movement.
FAQ
which test is best for detecting data drift
No single test is best; it depends on sample size and the shift you care about. KS and chi-square suit small samples, roughly under 1,000 rows. On large windows, Wasserstein for numeric and Jensen-Shannon for categorical features avoid flagging trivial shifts. Keep PSI where risk teams already report it, with thresholds set from history.
what threshold indicates data drift
No universal threshold exists, and tools ship different defaults. Evidently flags Wasserstein or Jensen-Shannon scores of 0.1 or more on references over 1,000 rows, and p-values of 0.05 or less on smaller ones. Vertex AI Model Monitoring v1 uses 0.3, SageMaker Model Monitor 0.1. Calibrate per feature on known-stable windows.
how often should you run data drift checks
Run drift checks as often as enough new data accumulates for a stable comparison. Azure Machine Learning recommends daily monitoring when daily traffic is sufficient, otherwise weekly or monthly. Set a minimum row count per window and segment, keep reference and current windows from overlapping, and match the cadence to how fast someone can respond.