“Data drift” is used as if it names one thing. It names at least three formally distinct failure modes, each with a different cause, a different detection signal, and a different correct response. Confusing them is why teams retrain a model that did not need retraining, and why they miss the one that did.
The taxonomy below separates the statistical types, which come from the research literature and are well defined, from the operational types, which come from your own pipeline and have no agreed names at all despite being the more common cause of a production alert.
For scorecards, PSI vs CSI explains how score-level stability can coexist with shifts in individual characteristics. That comparison helps interpret marginal drift signals without treating them as evidence of model accuracy.
The formal taxonomy: which part of the joint distribution moved
Every supervised model is a bet on a joint distribution P(X, Y). That joint can be factorised two ways, P(X)·P(Y|X) or P(Y)·P(X|Y), and the standard taxonomy from the unifying view on dataset shift is built from which factor moves.
Covariate shift. P(X) changes, P(Y|X) holds. The inputs look different; the rule mapping inputs to outcomes is still correct. A demand model sees a new city’s traffic, a fraud model sees a new payment method, a credit model sees applicants from a marketing campaign aimed at a younger segment. This is what most people mean by “data drift” and it is the only type a label-free monitor sees directly.
Prior probability shift, also called label shift. P(Y) changes, P(X|Y) holds. The base rate moved. Fraud attempts triple during a card-testing wave; churn doubles after a price rise. The features conditional on each class look exactly as they did in training, but the mix of classes underneath the traffic is different. A classifier tuned to a 2% positive rate is miscalibrated at 6% even though nothing about its learned boundary is wrong.
Concept drift. P(Y|X) changes. The relationship itself broke. The same input now implies a different outcome: a transaction pattern that was fraudulent last year is normal behaviour now that a legitimate app produces it. This is the expensive one, because input monitoring is structurally blind to it. The mechanics, and how it differs from the two above, are worked through in concept drift vs data drift explained.
Prediction drift is often listed as a fourth type. It is not a type; it is a symptom. P(Ŷ) moving tells you something upstream changed without telling you which of the three it was.
| Type | What moves | What holds | Needs labels to see | Typical cause |
|---|---|---|---|---|
| Covariate shift | P(X) | P(Y|X) | No | New segment, new channel, seasonality |
| Prior probability shift | P(Y) | P(X|Y) | Yes, or a base-rate proxy | Base-rate change, campaign, attack wave |
| Concept drift | P(Y|X) | Sometimes P(X) | Yes | Behaviour change, policy change, adversary |
| Prediction drift | P(Ŷ) | Nothing implied | No | Downstream of any of the above, or a bug |
The practical value of the table is the third column. Two of the four are visible without ground truth, which is why a monitoring stack that only watches inputs feels complete right up until the day it is not. When labels arrive weeks late, the workarounds are covered in monitoring models when ground truth is late.
The shapes drift takes over time
Type answers what moved. Shape answers how fast, and it decides which detector can see it at all. the concept drift survey sorts change into sudden, gradual, incremental, and recurring, and the same four shapes apply to covariate shift just as well as to concept drift.
Sudden change is easy: any windowed comparison catches a step function. Incremental change is the dangerous one, because a rolling reference window drifts along with the data and each comparison looks clean while the model slides a long way from where it started. Recurring change, mostly seasonality, is the biggest single source of false alarms, since a December window compared against a summer baseline “drifts” every year on schedule. Which detector suits which shape is the subject of concept drift detection: DDM, ADWIN, and Page-Hinkley.
The drift types that come from your pipeline, not the world
Most production drift alerts are not the world changing. They are your own data platform changing, and none of these have names in the research literature. the data validation paper documents these as the failures that a schema-based validator catches and a distribution-based drift test does not.
Schema drift. A column is added, removed, renamed, or retyped upstream. The model still scores, because the serving code fills a default, and the default is now a large fraction of your traffic.
Unit and encoding drift. Cents become dollars, seconds become milliseconds, a country field switches from ISO-2 to ISO-3, a boolean starts arriving as the strings “true” and “false”. The distribution shifts violently and the cause is a deployment, not a customer.
Missingness drift. The rate of nulls in a feature changes because an upstream service began timing out. If your imputation replaces nulls with the training median, the feature’s monitored distribution can look more stable while the model’s real information content collapses.
Cardinality drift. A categorical feature gains levels that did not exist at training time. Unseen categories create bins with zero reference mass, which is exactly the case that breaks a naive divergence calculation, as covered in population stability index explained.
Feature-computation drift. The offline transformation and the online transformation stop agreeing, usually after someone edits one of the two. Input distributions can look identical on both sides while the values fed to the model are wrong. This is a distinct failure with its own detection strategy in training-serving skew: the failure drift detection misses.
Feedback-loop drift. The model’s own outputs change the data it later sees. A recommender that promotes an item makes that item popular, so next month’s training data says it was always popular. Nothing is broken and every monitor is green, but the reference distribution is now partly authored by the model.
Which monitor catches which
| Signal you already collect | Catches | Misses |
|---|---|---|
| Per-feature distance (PSI, KS, Wasserstein) | Covariate shift, unit and cardinality drift | Concept drift, feature-computation drift |
| Schema and null-rate validation | Schema drift, missingness drift | Anything distribution-shaped |
| Prediction distribution P(Ŷ) | Prior probability shift, most upstream bugs | Which cause produced it |
| Realised performance on labels | Concept drift, prior probability shift | Nothing, but arrives too late to be an alert |
| Estimated performance without labels | Covariate-driven performance loss | Concept drift by construction |
| Offline versus online feature parity | Feature-computation drift | Real-world shift |
The row that surprises teams is the second-to-last. Label-free performance estimators assume P(Y|X) is stable and reweight for the input shift; that assumption is what makes them useful and it is also precisely why they cannot see concept drift. Reading an estimator’s flat line as evidence that the model is fine is a category error.
To pick a specific statistical test for a specific feature type, sample size, and label latency, the Drift Test Selector walks the decision directly rather than making you infer it from the table.
A triage order that saves time
When an alert fires, work from cheapest and most likely to most expensive:
- Check for a deployment. Correlate the alert timestamp with releases to the feature pipeline and its upstream services. A large fraction of “drift” is a change you shipped.
- Check whether every feature moved at once. Simultaneous drift across unrelated features is a pipeline event, not the world. Real covariate shift is usually localised to a few related columns.
- Check the calendar. Compare against the same period last year before against last month.
- Check nulls and cardinality separately from distributions. These have distinct fixes and are frequently the actual cause of a distance-metric spike.
- Only then look at P(Ŷ) and, when labels land, at realised performance. A stable input distribution with degrading performance is the concept drift signature, and it is the one case where retraining on fresh labels is the right answer.
Wiring those checks into a written spec with owners and thresholds, rather than into ad hoc dashboards, is the subject of the ML model monitoring framework.
Caveats
Virtual drift is real drift that does not matter. Input distributions can move without touching the decision boundary. Retraining on it burns capacity and can make the model worse by fitting a transient.
The types co-occur. A new market launch produces covariate shift, prior probability shift, and cardinality drift at the same time. The taxonomy is a diagnostic aid, not a claim that alerts arrive one at a time.
Univariate monitors miss joint shifts. Every marginal distribution can hold steady while the correlation structure between features changes completely. Multivariate checks, typically PCA reconstruction error or a domain classifier, exist for this and are covered in data drift detection in ML.
Thresholds are not transferable. Evidently’s drift documentation makes the point that the right test and cutoff depend on column type and sample size. Inherited numbers are a starting point to calibrate away from, not a standard.
Related on this site
- Concept Drift vs Data Drift Explained (and Prediction Drift)
- Data Drift Detection in ML: Methods, Tests, and Practice
- Concept Drift Detection: DDM, ADWIN, and Page-Hinkley
- Population Stability Index: How PSI Flags Feature Drift
- Training-Serving Skew: The Failure Drift Detection Misses
- ML Model Monitoring Framework: A Practical Blueprint
