ML Monitoring Report
Glowing isometric grid platform with cyan data nodes linked by wavy curves flowing toward an upright blue report panel, evoking feature drift monitoring.
drift-detection

Feature Drift Monitoring Best Practices for Production ML

Seven feature drift monitoring rules: fixed references, effect-size thresholds, importance weighting, segments, and alerts that require persistence.

By ML Monitoring Report Editorial · · 7 min read

The feature drift monitoring best practices that survive production come down to seven: version a fixed reference, pick the test by feature type and sample size, threshold on effect size rather than p-values, weight alerts by feature importance, monitor segments separately, require drift to persist across windows, and route every alert to a runbook that checks pipelines first.

Here is the failure those rules prevent. A churn model with 140 features runs a nightly KS test on every column. With a few million rows, nearly every p-value rounds to zero, dozens of features “drift” each day, and the team mutes the alert channel. Then an upstream duration field switches from seconds to milliseconds, and the one alert that mattered looks like all the others.

What is feature drift, and what should you measure?

Feature drift is a change in the distribution of a single model input between a reference window and a current window, measured per feature as an effect size. It is the marginal, label-free slice of data drift, so you can compute it as soon as traffic arrives, long before ground truth lands. It tells you the inputs moved. It does not tell you predictions got worse.

Feature drift is a shift in P(x). A change in P(y|x) is concept drift, and detecting it takes labels, as covered in concept drift detection methods. Types of data drift lays out the full taxonomy, including the unit and schema changes your own pipeline causes. Measure a per-feature distance, plus null share.

The metric that matters: normalized effect size, not a p-value

Use normalized Wasserstein distance for numeric features and Jensen-Shannon distance for categoricals:

d_num(x_i) = W1(P_ref(x_i), P_cur(x_i)) / std_ref(x_i)   # reference std units
d_cat(x_i) = JS(p_ref, q_cur), log base 2                 # bounded in [0, 1]

A KS p-value answers “how sure am I these differ?” That confidence grows with sample size whether or not the difference matters. In Evidently’s comparison of five drift tests, a vendor benchmark on its own synthetic and public datasets rather than an independent study, KS flagged a 0.5% shift at 100,000 observations. PSI ignored that shift at every sample size, and at 100,000 rows it reacted only above 10%. Normalized Wasserstein landed between the two and reads in standard-deviation units, which an on-call engineer can reason about.

Evidently’s default drift logic follows the same reasoning. With 1,000 or fewer reference rows, it runs KS on numeric columns and chi-squared on categoricals, both at p ≤ 0.05. Above 1,000 rows it switches to Wasserstein and Jensen-Shannon with a 0.1 threshold. Google’s Vertex AI Model Monitoring uses Jensen-Shannon divergence for numeric features, L-infinity distance for categoricals, and a default threshold of 0.3. Thresholds do not transfer between tools.

PSI is still a reasonable choice when a credit-risk team already reports it. See population stability index explained and PSI vs KS vs calibration.

Weight by importance before you alert

A drift score on a feature the model barely uses is trivia. Combine the per-feature scores into one model-level number, weighted by attribution:

D_model = sum_i w_i * d(x_i),   w_i = mean|SHAP_i| / sum_j mean|SHAP_j|

Then watch the attributions themselves. Amazon SageMaker Clarify ranks features by attribution in training and live data, compares the two rankings with NDCG, and alerts when NDCG falls below 0.90. The same page says Model Monitor is no longer open to new customers, but the method works in any pipeline that logs SHAP values.

Seven feature drift monitoring best practices

1. Version a fixed reference and run a rolling one beside it

Build the reference from the post-transform features the model trained on, or from a stable production window that covers a full business cycle. Store its version with every score. A rolling reference catches sudden steps. The fixed one catches the slow accumulation that a rolling baseline absorbs.

2. Match the statistic to type and volume

Use distances for large windows and hypothesis tests for small ones. If you run KS on high-volume traffic, sample to a fixed n so its sensitivity stays constant as traffic varies.

3. Calibrate thresholds on stable history

Start from the tool’s default, then replay weeks you know were healthy and any incidents you can reconstruct. Set each feature’s threshold above its own stable noise. A single global threshold fires constantly on noisy features and stays silent on tight ones.

4. Tier alerts by importance

Features near the top of the attribution ranking can page. The rest open tickets. Store the tier as a metric label so Alertmanager handles the routing.

5. Slice by segment

Drift confined to one slice disappears in the aggregate. In the same Evidently benchmark, Jensen-Shannon run on the full dataset barely registered large shifts confined to a 20% segment. Score each segment that has business meaning and enough rows.

6. Require persistence in two windows

The SRE Workbook’s chapter on alerting on SLOs grades alerts on precision, recall, detection time and reset time, and pairs each long window with a short one about 1/12 its length. Applied to drift, that means paging only when both the 24-hour and the 2-hour average exceed the threshold. The long window filters out blips. The short one clears the alert soon after a fix ships. More in SLOs and alerting for ML systems.

7. Check your pipeline before blaming the world

A unit change, a new default imputation or a swapped encoding looks the same on a drift chart as a real population shift. Evidently drops empty values before computing drift, and its docs say drift tests do not react to rising null counts. Before anyone retrains, run data quality monitoring and training-serving skew detection.

Wiring it up

A nightly batch job scores each feature per segment and pushes the results to a Prometheus Pushgateway. Prometheus describes service-level batch jobs as the usual valid use for the Pushgateway. The alert rules encode the tiers and the two-window persistence check.

import pandas as pd
from prometheus_client import CollectorRegistry, Gauge, push_to_gateway
from scipy.spatial.distance import jensenshannon
from scipy.stats import wasserstein_distance

MODEL = "churn-v14"
REF_VERSION = "train-2026-09-01"
CRITICAL = {"tenure_days", "plan_type", "sessions_7d"}  # top by mean |SHAP|


def num_drift(ref: pd.Series, cur: pd.Series) -> float:
    ref, cur = ref.dropna(), cur.dropna()
    std = ref.std()
    if pd.isna(std) or std == 0:
        return float(cur.nunique() > 1)
    return wasserstein_distance(ref, cur) / std


def cat_drift(ref: pd.Series, cur: pd.Series) -> float:
    cats = pd.Index(ref.dropna().unique()).union(cur.dropna().unique())
    p = ref.value_counts(normalize=True).reindex(cats, fill_value=0)
    q = cur.value_counts(normalize=True).reindex(cats, fill_value=0)
    return float(jensenshannon(p, q, base=2))


def publish(ref: pd.DataFrame, cur: pd.DataFrame, num_cols: list[str],
            cat_cols: list[str], segment: str) -> None:
    # ref and cur are already filtered to this segment
    reg = CollectorRegistry()
    drift = Gauge("feature_drift_score", "Effect size vs fixed reference",
                  ["model", "feature", "tier", "segment", "ref_version"], registry=reg)
    nulls = Gauge("feature_null_share", "Null share in current window",
                  ["model", "feature", "tier", "segment"], registry=reg)
    for col in num_cols + cat_cols:
        score = num_drift if col in num_cols else cat_drift
        tier = "critical" if col in CRITICAL else "standard"
        drift.labels(MODEL, col, tier, segment, REF_VERSION).set(score(ref[col], cur[col]))
        nulls.labels(MODEL, col, tier, segment).set(cur[col].isna().mean())
    push_to_gateway("pushgateway:9091", job=f"feature-drift-{MODEL}",
                    registry=reg, grouping_key={"segment": segment})
groups:
  - name: feature-drift
    rules:
      - alert: CriticalFeatureDrift
        expr: |
          avg_over_time(feature_drift_score{tier="critical"}[24h]) > 0.1
          and
          avg_over_time(feature_drift_score{tier="critical"}[2h]) > 0.1
        labels:
          severity: page
        annotations:
          runbook_url: "runbooks/feature-drift.md"
      - alert: FeatureDriftSustained
        expr: avg_over_time(feature_drift_score{tier="standard"}[3d]) > 0.1
        labels:
          severity: ticket
      - alert: FeatureNullShareJump
        expr: feature_null_share > 2 * avg_over_time(feature_null_share[7d] offset 1d) + 0.01
        for: 1h
        labels:
          severity: ticket

If you use Evidently, the equivalent is DataDriftPreset(num_method="wasserstein", cat_method="jensenshannon") with per_column_threshold overrides, as described in its customization docs. For other options, see best drift detection libraries for Python.

What you’ll see

On a per-feature panel, with the threshold drawn as a horizontal line:

  • Healthy: flat bands below the threshold, with repeating daily or weekly ripples.
  • One feature steps up at a deploy or upstream release: almost always a pipeline change. Check units, defaults and encodings.
  • Related features step together: a shared join, source table or backfill, occasionally a real population change such as a new market.
  • A slow ramp against the fixed reference, flat against the rolling one: real-world drift, and grounds for a retraining evaluation.
  • A sawtooth that climbs and resets: a scheduled reference refresh is hiding accumulation.
  • One segment red, aggregate green: the reason to slice by segment.

Caveats

Multiple comparisons. Scoring 140 features across six segments every day runs thousands of comparisons a week, and some will fire on noise. Tiering and persistence rules are the control.

Cardinality blowup. Prometheus warns that every unique label combination creates a new time series, and that labels should never hold unbounded values like user IDs. Keep segments to a short fixed list, and delete the series for retired reference versions.

Stale pushes. The Pushgateway never forgets a series unless you delete it through its API. If the drift job dies, its last score stays on the chart and looks exactly like stability. Alert on job freshness as well.

Point-in-time leakage. If training used backfilled aggregates that saw data from after each example’s timestamp, the online feature will never match the reference. That reads as permanent drift when the actual problem is skew. Build the reference with point-in-time correct joins.

High-dimensional inputs. Per-dimension drift on embeddings is noise. Failing Loudly, an independent NeurIPS 2019 study, found that across the shifts it examined, two-sample tests performed best after dimensionality reduction with pretrained classifiers.

Drift is not degradation. Inputs can move with no loss of accuracy, and accuracy can fall while the inputs stay flat. Feature drift gives you early warning while labels are pending. For what to pair it with, see monitoring without ground truth.

FAQ

what is a good threshold for feature drift

A good starting threshold is the default for your statistic, then calibrated per feature: 0.1 for normalized Wasserstein or Jensen-Shannon in Evidently, 0.3 for Vertex AI’s distances, or the 0.1 and 0.2 PSI bands. Replay stable weeks and known incidents, then set each feature’s threshold just above its observed stable noise.

how often should you check for feature drift

Check feature drift at a cadence your team can act on, usually daily for batch-scored models and hourly windows for high-traffic online models. Each window needs enough rows per segment to give a stable score. Checking more often than you can respond adds alert volume without shortening the time to a fix.

should feature drift trigger model retraining

No, feature drift should trigger an investigation, not an automatic retrain. First rule out pipeline bugs and skew, then check whether the drifting features carry meaningful attribution, then evaluate a candidate on recent labeled data. Retraining on a broken feature teaches the model the bug and hides it from the next drift check.

how do you monitor drift on high-cardinality categorical features

Monitor high-cardinality categoricals with coverage metrics instead of full distribution tests: the share of current values not seen in the reference, the share that falls into the top-k reference categories, and null share. Jensen-Shannon over thousands of sparse categories is noisy, and raw values in Prometheus labels create a cardinality problem.

Sources

  1. Data drift: how it works and default methods — Evidently AI documentation
  2. Customize data drift parameters — Evidently AI documentation
  3. Which test is the best? Comparing 5 drift detection methods on large datasets — Evidently AI (vendor benchmark)
  4. Introduction to Vertex AI Model Monitoring — Google Cloud documentation
  5. Feature attribution drift for models in production — Amazon SageMaker documentation
  6. Alerting on SLOs — The Site Reliability Workbook
  7. Metric and label naming — Prometheus documentation
  8. When to use the Pushgateway — Prometheus documentation
  9. Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift (NeurIPS 2019)
#feature-drift #drift-detection #model-monitoring #prometheus #evidently #alerting
Subscribe

ML Monitoring Report — in your inbox

Production ML monitoring, drift, and reliability. Sent only when there is something worth sending.

No spam. Unsubscribe anytime.

Related