ML Monitoring Report
Isometric conveyor feeding cubes into a camera-topped sorting machine that routes cubes, spheres and pyramids into three bins, a model drift metaphor.
drift-detection

Concept Drift vs Data Drift vs Model Drift: Key Differences

Concept, data, prediction and model drift compared: what changes, whether labels are needed, detection latency, monitoring signals and the right response.

By ML Monitoring Report Editorial · · Updated · 12 min read

Data drift is a change in the inputs a model receives, P(X). Concept drift is a change in the relationship between those inputs and the target, P(Y|X). You can detect data drift from inputs alone, without labels. Confirming concept drift usually takes ground-truth outcomes. Either one can cause model drift, which is the drop in predictive performance that follows.

The difference tells you what evidence to collect before you act. A distribution alert alone cannot tell you whether to repair a feature pipeline, collect labels, or evaluate a replacement model. IBM’s model drift overview and Evidently’s data drift guide both draw the same line.

Concept drift vs data drift: the formal distinction

A supervised model maps inputs X to a target Y.

Data drift is a change in the input distribution, P(X). In the pure covariate-shift case, P(Y|X) remains unchanged. For example, a hypothetical transaction model might receive a different mix of transaction amounts while the relationship between those amounts and the outcome stays the same. An input-drift test detects the changed mix; it does not establish that the model’s predictions are worse.

Concept drift is a change in P(Y|X). The same input can now have a different expected outcome. For example, the behavior associated with an account feature might change while that feature’s frequency stays stable. Measuring the input distribution alone cannot reveal this change.

Prediction drift is a change in the distribution of model outputs, P(Ŷ). A classifier predicting the positive class more frequently has prediction drift. That can reflect changed inputs, changed scoring code, or a different model version. It is a symptom to investigate.

Model drift is the umbrella term for a deployed model’s predictive performance degrading over time. IBM’s definition ties it to changes in the data or in the relationships between input and output variables. Of the three shifts above, data drift and concept drift can cause it; prediction drift can accompany it but does not establish it. One signal confirms model drift: performance measured on predictions whose outcome labels have matured.

The research literature uses different names for the same split. Gama et al.’s survey on concept drift adaptation (ACM Computing Surveys 46(4), 2014) frames concept drift as a change in the relation between input data and target. Later work that builds on it, such as this continual-learning drift analysis, calls a P(Y|X) change “real” concept drift. It calls a change in the data distribution that leaves the input-label relation intact “virtual” drift, also known as covariate shift or dataset shift. When a paper says “virtual drift”, read “data drift”.

The cases can coexist. Pure concept drift need not change the predictions of a fixed model: if its inputs and scoring function remain unchanged, its output distribution can stay stable even as its answers become less useful. Data drift also does not guarantee performance degradation; that depends on the model and the affected population.

Label-distribution changes and pipeline-origin shifts are covered in types of data drift.

What each shift requires to detect

ShiftWhat changesNeeds outcome labels?What the signal establishesTypical first response
Data driftInput distribution P(X)NoThe observed input samples differCheck the pipeline, then the affected segments
Prediction driftOutput distribution P(Ŷ)NoThe model’s score or class mix differsTrace back to inputs, code, or model version
Concept driftInput-to-outcome relationship P(Y|X)Usually needed for confirmationThe learned relationship may no longer holdRetrain or redesign on recent, valid labels
Model driftRealized performance (accuracy, AUC, error)YesThe model is measurably worseDiagnose which shift caused it before acting

Data and prediction drift can be measured as soon as a sufficiently large current window is available. Concept-drift confirmation depends on the outcome-label process. If labels take weeks to mature, a dashboard cannot turn input statistics into immediate ground truth.

This is why monitoring models without ground truth separates leading indicators, performance estimates, and retrospective outcome reporting.

Model drift vs data drift: cause and effect

Model drift is the outcome and data drift is one possible cause. Data drift describes the inputs, while model drift describes performance. A model can see heavy data drift and lose no accuracy if the shifted features carry little weight. It can also lose accuracy with no data drift at all when the concept changes underneath it.

Concept drift vs model drift is the same cause-and-effect pair. Concept drift is the other main cause of model drift, and the harder one to see. Treat the three terms as layers of one investigation:

  1. Data drift answers “did the inputs move?” You can measure it today.
  2. Concept drift answers “does the old input-to-outcome mapping still hold?” You can answer it once labels mature.
  3. Model drift answers “is the model worse?” Only labeled performance settles it.

Some vendors use “model drift” loosely for any of the three. When a dashboard labels an alert “model drift”, check whether it is computed from labels. If it is not, it is a data drift or prediction drift alert under another name.

Concept drift in forecasting

In time series forecasting, concept drift means the process generating the series has changed, so patterns learned from history stop predicting the future. Ground truth for a forecast only arrives once the horizon has passed. That makes labels late by design. Wikipedia’s concept drift article uses a weekly sales-prediction model degrading under seasonal shopping changes as its main example.

Two recent papers show how the problem is handled:

  • Liu, Godahewa, Bandara and Bergmeir (arXiv:2304.01512, April 2023) address concept drift in global forecasting models. Their two methods, Error Contribution Weighting and Gradient Descent Weighting, blend a model trained on the most recent part of each series with one trained on the full history. They use LightGBM as the base learner.
  • Zhao and Shen (arXiv:2412.08435, accepted to KDD 2025) argue that because true values appear only after the forecast horizon, the latest training samples lag the current test sample. That gap can itself introduce concept drift. Their Proceed framework estimates the drift and adjusts model parameters before predicting.

Three practical checks follow from this. Track forecast error per horizon on the forecasts that have matured, since a 28-day-ahead error tells you nothing until 28 days later. Compare against the same season last year before you call a change drift. And treat a forecast that misses consistently in one direction as a stronger concept-drift signal than a single noisy week.

Data drift in data pipelines: the StreamSets meaning

StreamSets uses “data drift” for a different problem than ML monitoring does: unannounced changes in the data a pipeline carries. The March 2016 release of StreamSets Data Collector v1.2 described data drift as unpredictable, unending mutations to data that start at the source and break downstream applications. The company reported that 96% of respondents to its survey had encountered it.

An Elastic post from November 2015 describes Data Collector adapting to changes in schema, semantics and infrastructure. Wikipedia uses the same split. Structural drift is a schema change. Semantic drift is a change in what the data means while its structure stays the same.

Both matter for models. A schema change upstream often surfaces as a sudden data drift alert. A semantic change, such as a field switching units, can look like data drift or, if the values stay in range, like concept drift. Rule these out with data quality monitoring for ML pipelines before you start a statistical drift investigation.

Detecting data drift without labels

Compare a current input window with a documented reference sample. That reference can be a training or validation snapshot, or a stable production period. Keep transformations and units consistent across both samples.

Hypothesis tests such as KS for continuous features and chi-square for categorical frequencies ask whether the observed differences are consistent with the null model. Sample size matters: a small p-value does not measure operational harm.

Distance metrics such as PSI and Wasserstein express distributional differences as a magnitude. The Population Stability Index uses fixed bins and needs an explicit policy for empty bins. Raw Wasserstein distance is expressed in feature units; normalized implementations use a scaling convention that must be recorded.

In scorecard terminology, score-level PSI and per-characteristic CSI apply the stability calculation to different variables. PSI vs CSI explains why a stable score histogram can hide offsetting input shifts.

The metric decision belongs in choosing drift metrics: PSI, KS and calibration. For window design, multivariate checks, and retrieval inputs, use data drift detection in ML. A metric threshold should be calibrated against the relevant baseline, sample size, and acceptable response burden.

A current Evidently drift report

The Evidently ML quickstart uses Dataset objects with a DataDefinition. The following reader-runnable example assumes two existing DataFrames with the named columns. It compares inputs; outcome monitoring is a separate step.

from evidently import DataDefinition, Dataset, Report
from evidently.presets import DataDriftPreset

definition = DataDefinition(
    numerical_columns=["amount", "account_age_days"],
    categorical_columns=["channel", "device_type"],
)
reference = Dataset.from_pandas(
    reference_df, data_definition=definition
)
current = Dataset.from_pandas(
    current_df, data_definition=definition
)

report = Report([DataDriftPreset()], include_tests=True)
result = report.run(current, reference)
result.save_html("drift_report.html")
metrics = result.dict()
payload = result.json()

The object returned by run() provides the exports. When migrating an older report, replace the legacy evidently.report and metric_preset imports and update the data wrappers. Calling export methods on the report builder instead of its result is an API mismatch. Record the installed library version alongside the report.

Evidently’s customization documentation describes how to select methods and thresholds per column. Use explicit choices when comparing successive reports; changed detector defaults can otherwise look like changed data.

Detecting concept drift with delayed outcomes

Monitor labeled cohorts. Track classification or regression performance on predictions whose outcomes have matured. Compare cohorts with similar label availability, class coverage, and sampling rules. An AUC drop measured on a small, selectively labeled subset is not equivalent to a drop across the complete serving population.

Track the label process itself. Missing labels, changed annotation criteria, and different maturation times can move apparent performance. Separate the time of prediction from the time its label arrived so a backfilled report can be reconstructed.

Use streaming detectors on an appropriate signal. ADWIN, DDM, and Page-Hinkley can be applied to monitored streams such as errors or losses, subject to each method’s assumptions. Feeding a detector raw inputs asks about input change; feeding it prediction errors asks about a different signal. The differences are covered in concept drift detection methods.

Keep proxies labeled as proxies. Complaints, conversion, or rejection rates can help prioritize investigation, but they also respond to product and population changes. They do not isolate concept drift.

NannyML’s CBPE documentation describes estimating classification performance from calibrated probabilities without current labels. Its assumptions do not make it a general concept-drift detector. Reconcile estimates against realized performance when labels arrive, and investigate discrepancies in calibration, population mix, or label quality.

Reading data and performance panels together

This diagnostic table preserves the dashboard distinction: input movement and outcome degradation are separate observations.

Input distributionsLabeled performanceInterpretation to investigateFirst checks
StableStableNo detected regression in these monitored viewsCheck sample coverage and important segments
ShiftedStableData drift without observed performance damageInspect pipelines, feature importance, and affected cohorts
StableDegradedPossible concept drift, skew, or label-quality changeCompare feature paths, label definitions, and model versions
ShiftedDegradedChanged population with a quality regression; several causes remain possibleValidate data and labels before evaluating retraining

These rows are diagnostic prompts, not proofs of causality. A dataset-wide average can hide a failing subgroup. Likewise, stable individual feature distributions do not guarantee a stable joint distribution.

A high-cardinality input can appear to change simply because new identifiers arrive. Check new-category rates and group meaningful categories rather than interpreting every new ID as a model incident. Require a documented minimum sample and compare seasonal windows before escalating a short-lived spike.

A diagnostic order before retraining

  1. Validate the pipeline. Check schema, units, null handling, joins, and feature freshness. Compare the transformations actually used during training and serving; training-serving skew detection covers matched-record checks.
  2. Locate the movement. Inspect the features, score bands, and segments that changed. Use importance as a prioritization aid, not as proof that a feature cannot matter.
  3. Validate outcomes. Check maturity, missingness, and label definitions before interpreting a performance change.
  4. Evaluate a response. Repair a broken upstream feed when that is the cause. If the population or target relationship changed, evaluate a candidate trained on suitable recent examples and valid labels.
  5. Record the decision. Keep the reference version, evidence, owner, and action in the ML model monitoring framework specification.

Retraining on freshly collected inputs with stale or incorrectly joined labels can preserve the original problem. A drift alert should open an investigation or evaluation, not silently approve a new model.

Where the tabular distinction needs adaptation

For an LLM system, input movement can involve changes in topics, languages, document mix, or retrieval results. Quality often requires task-specific scoring and reviewed examples rather than a single classification label. The distinction between distribution movement and quality degradation still helps, but the instruments change.

Monitoring tabular models vs LLM systems maps those differences. The production-sampling and judge-review workflow is included in ML model monitoring best practices.

Common interpretation errors

Treating prediction drift as concept-drift confirmation. An output shift can have several causes, while a changed target relationship can leave a fixed model’s outputs unchanged.

Treating stable PSI as proof of model quality. PSI compares binned distributions. It does not evaluate correctness, interactions between features, or unmonitored segments.

Replacing every reference with the latest window. A rolling comparison answers a short-term question and can conceal cumulative movement. Keep a versioned fixed reference when long-term stability is the question.

Assuming drift gives a fixed warning period. There is no universal number of hours between an input alert and a quality failure. Detection latency depends on the change, the window, and the label process.

FAQ

what is concept drift in machine learning

Concept drift is a change over time in the relationship between a model’s inputs and the outcome it predicts, written P(Y|X). The same input now implies a different result, so a model trained on the old relationship gives answers that look normal but are increasingly wrong. Confirming it normally requires comparing predictions against outcome labels.

is model drift the same as data drift

No. Data drift means the input distribution changed. Model drift means the model’s measured performance got worse. Data drift is one cause of model drift and concept drift is the other. Many data drift alerts never hurt accuracy, because the shifted features matter little to the model. Only labeled performance confirms model drift.

can concept drift happen without data drift

Yes. If the outcome an input implies changes while input frequencies stay the same, every input monitor stays green while accuracy falls. A fraud pattern that becomes legitimate behavior is a typical example. That is why input drift tests cannot rule out concept drift, and why labeled cohorts or error-stream detectors are still needed.

what are the types of data drift

The usual ML taxonomy has covariate shift (inputs change), prior probability or label shift (the class mix changes) and concept drift (the input-to-outcome relationship changes). Prediction drift is a symptom, not a separate type. Pipeline teams also use “data drift” for schema, semantic and infrastructure changes, which is how StreamSets uses the term.

how do you detect concept drift without labels

You cannot confirm it without labels, but you can narrow the search. Watch prediction distributions, estimate performance from calibrated probabilities where the method’s assumptions hold, and track business proxies. Treat all three as leading indicators. When outcomes mature, check realized performance on labeled cohorts or run a detector such as ADWIN on the error stream.

Sources

  1. What is data drift in ML, and how to detect and handle it — Evidently AI
  2. Which test is the best? Comparing 5 drift detection methods — Evidently AI
  3. Data Drift vs. Concept Drift: What Are the Main Differences? — Deepchecks
  4. What Is Model Drift? — IBM
  5. Confidence-based Performance Estimation (CBPE) — NannyML
  6. Data and ML checks — Evidently documentation
  7. Customize data drift — Evidently documentation
  8. A survey on concept drift adaptation (Gama et al., ACM Computing Surveys 46(4), 2014) — TU Eindhoven research portal
  9. Understanding Continual Learning Settings with Data Distribution Drift Analysis — arXiv
  10. Concept drift — Wikipedia
  11. Handling Concept Drift in Global Time Series Forecasting — arXiv
  12. Proactive Model Adaptation Against Concept Drift for Online Time Series Forecasting — arXiv
  13. New Release of StreamSets Data Collector Takes on Data Drift — Yahoo Finance
  14. Elasticsearch plus StreamSets for reliable data ingestion — Elastic
Subscribe

ML Monitoring Report — in your inbox

Production ML monitoring, drift, and reliability. Sent only when there is something worth sending.

No spam. Unsubscribe anytime.

Related