Data drift is a change in the inputs a model receives, P(X). Concept drift is a change in the relationship between those inputs and the target, P(Y|X). You can detect data drift from inputs alone, without labels. Confirming concept drift usually takes ground-truth outcomes. Either one can cause model drift, which is the drop in predictive performance that follows.
The difference tells you what evidence to collect before you act. A distribution alert alone cannot tell you whether to repair a feature pipeline, collect labels, or evaluate a replacement model. IBM’s model drift overview and Evidently’s data drift guide both draw the same line.
Concept drift vs data drift: the formal distinction
A supervised model maps inputs X to a target Y.
Data drift is a change in the input distribution, P(X). In the pure covariate-shift case, P(Y|X) remains unchanged. For example, a hypothetical transaction model might receive a different mix of transaction amounts while the relationship between those amounts and the outcome stays the same. An input-drift test detects the changed mix; it does not establish that the model’s predictions are worse.
Concept drift is a change in P(Y|X). The same input can now have a different expected outcome. For example, the behavior associated with an account feature might change while that feature’s frequency stays stable. Measuring the input distribution alone cannot reveal this change.
Prediction drift is a change in the distribution of model outputs, P(Ŷ). A classifier predicting the positive class more frequently has prediction drift. That can reflect changed inputs, changed scoring code, or a different model version. It is a symptom to investigate.
Model drift is the umbrella term for a deployed model’s predictive performance degrading over time. IBM’s definition ties it to changes in the data or in the relationships between input and output variables. Of the three shifts above, data drift and concept drift can cause it; prediction drift can accompany it but does not establish it. One signal confirms model drift: performance measured on predictions whose outcome labels have matured.
The research literature uses different names for the same split. Gama et al.’s survey on concept drift adaptation (ACM Computing Surveys 46(4), 2014) frames concept drift as a change in the relation between input data and target. Later work that builds on it, such as this continual-learning drift analysis, calls a P(Y|X) change “real” concept drift. It calls a change in the data distribution that leaves the input-label relation intact “virtual” drift, also known as covariate shift or dataset shift. When a paper says “virtual drift”, read “data drift”.
The cases can coexist. Pure concept drift need not change the predictions of a fixed model: if its inputs and scoring function remain unchanged, its output distribution can stay stable even as its answers become less useful. Data drift also does not guarantee performance degradation; that depends on the model and the affected population.
Label-distribution changes and pipeline-origin shifts are covered in types of data drift.
What each shift requires to detect
| Shift | What changes | Needs outcome labels? | What the signal establishes | Typical first response |
|---|---|---|---|---|
| Data drift | Input distribution P(X) | No | The observed input samples differ | Check the pipeline, then the affected segments |
| Prediction drift | Output distribution P(Ŷ) | No | The model’s score or class mix differs | Trace back to inputs, code, or model version |
| Concept drift | Input-to-outcome relationship P(Y|X) | Usually needed for confirmation | The learned relationship may no longer hold | Retrain or redesign on recent, valid labels |
| Model drift | Realized performance (accuracy, AUC, error) | Yes | The model is measurably worse | Diagnose which shift caused it before acting |
Data and prediction drift can be measured as soon as a sufficiently large current window is available. Concept-drift confirmation depends on the outcome-label process. If labels take weeks to mature, a dashboard cannot turn input statistics into immediate ground truth.
This is why monitoring models without ground truth separates leading indicators, performance estimates, and retrospective outcome reporting.
Model drift vs data drift: cause and effect
Model drift is the outcome and data drift is one possible cause. Data drift describes the inputs, while model drift describes performance. A model can see heavy data drift and lose no accuracy if the shifted features carry little weight. It can also lose accuracy with no data drift at all when the concept changes underneath it.
Concept drift vs model drift is the same cause-and-effect pair. Concept drift is the other main cause of model drift, and the harder one to see. Treat the three terms as layers of one investigation:
- Data drift answers “did the inputs move?” You can measure it today.
- Concept drift answers “does the old input-to-outcome mapping still hold?” You can answer it once labels mature.
- Model drift answers “is the model worse?” Only labeled performance settles it.
Some vendors use “model drift” loosely for any of the three. When a dashboard labels an alert “model drift”, check whether it is computed from labels. If it is not, it is a data drift or prediction drift alert under another name.
Concept drift in forecasting
In time series forecasting, concept drift means the process generating the series has changed, so patterns learned from history stop predicting the future. Ground truth for a forecast only arrives once the horizon has passed. That makes labels late by design. Wikipedia’s concept drift article uses a weekly sales-prediction model degrading under seasonal shopping changes as its main example.
Two recent papers show how the problem is handled:
- Liu, Godahewa, Bandara and Bergmeir (arXiv:2304.01512, April 2023) address concept drift in global forecasting models. Their two methods, Error Contribution Weighting and Gradient Descent Weighting, blend a model trained on the most recent part of each series with one trained on the full history. They use LightGBM as the base learner.
- Zhao and Shen (arXiv:2412.08435, accepted to KDD 2025) argue that because true values appear only after the forecast horizon, the latest training samples lag the current test sample. That gap can itself introduce concept drift. Their Proceed framework estimates the drift and adjusts model parameters before predicting.
Three practical checks follow from this. Track forecast error per horizon on the forecasts that have matured, since a 28-day-ahead error tells you nothing until 28 days later. Compare against the same season last year before you call a change drift. And treat a forecast that misses consistently in one direction as a stronger concept-drift signal than a single noisy week.
Data drift in data pipelines: the StreamSets meaning
StreamSets uses “data drift” for a different problem than ML monitoring does: unannounced changes in the data a pipeline carries. The March 2016 release of StreamSets Data Collector v1.2 described data drift as unpredictable, unending mutations to data that start at the source and break downstream applications. The company reported that 96% of respondents to its survey had encountered it.
An Elastic post from November 2015 describes Data Collector adapting to changes in schema, semantics and infrastructure. Wikipedia uses the same split. Structural drift is a schema change. Semantic drift is a change in what the data means while its structure stays the same.
Both matter for models. A schema change upstream often surfaces as a sudden data drift alert. A semantic change, such as a field switching units, can look like data drift or, if the values stay in range, like concept drift. Rule these out with data quality monitoring for ML pipelines before you start a statistical drift investigation.
Detecting data drift without labels
Compare a current input window with a documented reference sample. That reference can be a training or validation snapshot, or a stable production period. Keep transformations and units consistent across both samples.
Hypothesis tests such as KS for continuous features and chi-square for categorical frequencies ask whether the observed differences are consistent with the null model. Sample size matters: a small p-value does not measure operational harm.
Distance metrics such as PSI and Wasserstein express distributional differences as a magnitude. The Population Stability Index uses fixed bins and needs an explicit policy for empty bins. Raw Wasserstein distance is expressed in feature units; normalized implementations use a scaling convention that must be recorded.
In scorecard terminology, score-level PSI and per-characteristic CSI apply the stability calculation to different variables. PSI vs CSI explains why a stable score histogram can hide offsetting input shifts.
The metric decision belongs in choosing drift metrics: PSI, KS and calibration. For window design, multivariate checks, and retrieval inputs, use data drift detection in ML. A metric threshold should be calibrated against the relevant baseline, sample size, and acceptable response burden.
A current Evidently drift report
The Evidently ML quickstart uses Dataset objects with a DataDefinition. The following reader-runnable example assumes two existing DataFrames with the named columns. It compares inputs; outcome monitoring is a separate step.
from evidently import DataDefinition, Dataset, Report
from evidently.presets import DataDriftPreset
definition = DataDefinition(
numerical_columns=["amount", "account_age_days"],
categorical_columns=["channel", "device_type"],
)
reference = Dataset.from_pandas(
reference_df, data_definition=definition
)
current = Dataset.from_pandas(
current_df, data_definition=definition
)
report = Report([DataDriftPreset()], include_tests=True)
result = report.run(current, reference)
result.save_html("drift_report.html")
metrics = result.dict()
payload = result.json()
The object returned by run() provides the exports. When migrating an older report, replace the legacy evidently.report and metric_preset imports and update the data wrappers. Calling export methods on the report builder instead of its result is an API mismatch. Record the installed library version alongside the report.
Evidently’s customization documentation describes how to select methods and thresholds per column. Use explicit choices when comparing successive reports; changed detector defaults can otherwise look like changed data.
Detecting concept drift with delayed outcomes
Monitor labeled cohorts. Track classification or regression performance on predictions whose outcomes have matured. Compare cohorts with similar label availability, class coverage, and sampling rules. An AUC drop measured on a small, selectively labeled subset is not equivalent to a drop across the complete serving population.
Track the label process itself. Missing labels, changed annotation criteria, and different maturation times can move apparent performance. Separate the time of prediction from the time its label arrived so a backfilled report can be reconstructed.
Use streaming detectors on an appropriate signal. ADWIN, DDM, and Page-Hinkley can be applied to monitored streams such as errors or losses, subject to each method’s assumptions. Feeding a detector raw inputs asks about input change; feeding it prediction errors asks about a different signal. The differences are covered in concept drift detection methods.
Keep proxies labeled as proxies. Complaints, conversion, or rejection rates can help prioritize investigation, but they also respond to product and population changes. They do not isolate concept drift.
NannyML’s CBPE documentation describes estimating classification performance from calibrated probabilities without current labels. Its assumptions do not make it a general concept-drift detector. Reconcile estimates against realized performance when labels arrive, and investigate discrepancies in calibration, population mix, or label quality.
Reading data and performance panels together
This diagnostic table preserves the dashboard distinction: input movement and outcome degradation are separate observations.
| Input distributions | Labeled performance | Interpretation to investigate | First checks |
|---|---|---|---|
| Stable | Stable | No detected regression in these monitored views | Check sample coverage and important segments |
| Shifted | Stable | Data drift without observed performance damage | Inspect pipelines, feature importance, and affected cohorts |
| Stable | Degraded | Possible concept drift, skew, or label-quality change | Compare feature paths, label definitions, and model versions |
| Shifted | Degraded | Changed population with a quality regression; several causes remain possible | Validate data and labels before evaluating retraining |
These rows are diagnostic prompts, not proofs of causality. A dataset-wide average can hide a failing subgroup. Likewise, stable individual feature distributions do not guarantee a stable joint distribution.
A high-cardinality input can appear to change simply because new identifiers arrive. Check new-category rates and group meaningful categories rather than interpreting every new ID as a model incident. Require a documented minimum sample and compare seasonal windows before escalating a short-lived spike.
A diagnostic order before retraining
- Validate the pipeline. Check schema, units, null handling, joins, and feature freshness. Compare the transformations actually used during training and serving; training-serving skew detection covers matched-record checks.
- Locate the movement. Inspect the features, score bands, and segments that changed. Use importance as a prioritization aid, not as proof that a feature cannot matter.
- Validate outcomes. Check maturity, missingness, and label definitions before interpreting a performance change.
- Evaluate a response. Repair a broken upstream feed when that is the cause. If the population or target relationship changed, evaluate a candidate trained on suitable recent examples and valid labels.
- Record the decision. Keep the reference version, evidence, owner, and action in the ML model monitoring framework specification.
Retraining on freshly collected inputs with stale or incorrectly joined labels can preserve the original problem. A drift alert should open an investigation or evaluation, not silently approve a new model.
Where the tabular distinction needs adaptation
For an LLM system, input movement can involve changes in topics, languages, document mix, or retrieval results. Quality often requires task-specific scoring and reviewed examples rather than a single classification label. The distinction between distribution movement and quality degradation still helps, but the instruments change.
Monitoring tabular models vs LLM systems maps those differences. The production-sampling and judge-review workflow is included in ML model monitoring best practices.
Common interpretation errors
Treating prediction drift as concept-drift confirmation. An output shift can have several causes, while a changed target relationship can leave a fixed model’s outputs unchanged.
Treating stable PSI as proof of model quality. PSI compares binned distributions. It does not evaluate correctness, interactions between features, or unmonitored segments.
Replacing every reference with the latest window. A rolling comparison answers a short-term question and can conceal cumulative movement. Keep a versioned fixed reference when long-term stability is the question.
Assuming drift gives a fixed warning period. There is no universal number of hours between an input alert and a quality failure. Detection latency depends on the change, the window, and the label process.
Related on this site
- Types of Data Drift: Covariate, Label, and Concept
- Best Drift Detection Libraries for Python
- SLOs and Alerting for ML Systems
FAQ
what is concept drift in machine learning
Concept drift is a change over time in the relationship between a model’s inputs and the outcome it predicts, written P(Y|X). The same input now implies a different result, so a model trained on the old relationship gives answers that look normal but are increasingly wrong. Confirming it normally requires comparing predictions against outcome labels.
is model drift the same as data drift
No. Data drift means the input distribution changed. Model drift means the model’s measured performance got worse. Data drift is one cause of model drift and concept drift is the other. Many data drift alerts never hurt accuracy, because the shifted features matter little to the model. Only labeled performance confirms model drift.
can concept drift happen without data drift
Yes. If the outcome an input implies changes while input frequencies stay the same, every input monitor stays green while accuracy falls. A fraud pattern that becomes legitimate behavior is a typical example. That is why input drift tests cannot rule out concept drift, and why labeled cohorts or error-stream detectors are still needed.
what are the types of data drift
The usual ML taxonomy has covariate shift (inputs change), prior probability or label shift (the class mix changes) and concept drift (the input-to-outcome relationship changes). Prediction drift is a symptom, not a separate type. Pipeline teams also use “data drift” for schema, semantic and infrastructure changes, which is how StreamSets uses the term.
how do you detect concept drift without labels
You cannot confirm it without labels, but you can narrow the search. Watch prediction distributions, estimate performance from calibrated probabilities where the method’s assumptions hold, and track business proxies. Treat all three as leading indicators. When outcomes mature, check realized performance on labeled cohorts or run a detector such as ADWIN on the error stream.