ML Model Monitoring Framework: A Practical Blueprint

An ML model monitoring framework defines the models to watch, their signals and reference data, alert thresholds, responsible teams, and response actions. It is an operating specification that a monitoring tool implements.

This blueprint covers inventory, tiering, baseline policy, severity routing, and review. For software selection, use SentryML’s model monitoring tools comparison; for daily implementation choices, use ML model monitoring best practices.

Step 1: tier the models before instrumenting anything

Every downstream decision depends on two properties, and both are properties of the model’s situation, not its architecture: how much damage a wrong prediction does, and how long you wait for the label that would prove it wrong.

The following tiers are an illustrative policy, not a required standard. Record label latency independently: a critical model can still have outcomes that take months to mature.

Keep the model owner, baseline version, and baseline review date in the inventory. Reassess a tier when an experiment starts influencing a customer-facing decision.

Label latency is the constraint people underrate. If ground truth arrives in 30 days, no amount of tooling gives you an accuracy alert today, and a framework that assumes otherwise produces a monitoring plan that quietly does nothing. The strategies for that blind window are their own subject, covered in monitoring models when ground truth is late or never arrives.

Tiering is not only an engineering convenience. The revised US interagency model risk guidance issued on 17 April 2026 (SR 26-2, which supersedes SR 11-7) emphasises a risk-based approach tailored to a banking organisation’s model risk profile and the size and complexity of its operations, rather than one uniform standard for every model. It is guidance, not an enforceable rule, but it is the same idea as tiering under a different name; the regulated version of this table is discussed in PSI vs CSI.

Tiering is also the step that keeps cost sane. Tier 1 justifies per-request logging and a real-time metric store. Tier 3 justifies a nightly batch job over a sample. Skipping this step is why monitoring bills surprise people.

Step 2: pick signals per plane, not per feature

Select an actionable signal set across four planes. Azure ML’s monitoring guidance recommends combining signals and choosing a feature subset where appropriate.

Infrastructure. Latency percentiles, error rate, throughput, resource saturation. This plane is solved by ordinary SRE practice and should reuse the same Prometheus or Datadog stack the rest of the platform uses. Record service health alongside statistical quality.

Inputs. Schema conformance, null and cardinality rates, volume, and distribution drift on a selected feature set. Start with important features and required inputs from external or manually entered sources. Expand where consequences or observed failures justify it. Which statistical test to use per feature type is a separate decision, laid out in choosing monitoring metrics: PSI, KS, and calibration, and what each test is being asked to catch is enumerated in types of data drift.

Outputs. Prediction distribution, score calibration, confidence histogram, and segment-level rates. Prediction drift requires no labels, but it provides no guaranteed warning interval. It is also ambiguous by construction: a shifted score distribution is consistent with input drift, concept drift, or a correctly working model responding to a genuinely changed world.

Outcomes. Accuracy, AUC, precision at k, business KPI, whatever the model is actually judged on. Available only when labels are, which is why the tier table routes most models away from this plane as their primary alert.

A useful integrity check on the whole set: training-serving skew shows up on none of these planes if you only compare production against production. Catching it requires the reference distribution to come from the training pipeline itself.

Step 3: derive thresholds, do not inherit them

PSI above 0.25 is a threshold from credit-risk scorecards. KS at p < 0.05 is a threshold from a hypothesis test that was never meant to run daily on a million rows. Both are borrowed, and borrowed thresholds are the largest single source of alert fatigue in ML monitoring.

The framework’s rule should be a procedure, not a number:

1. Take twelve months of historical production data, or as much as exists.
2. Replay the detector over it in the same windowing you will use live.
3. Read off the distribution of the drift statistic during periods where the model was known to be healthy.
4. Set the warning threshold at roughly the 95th percentile of that distribution and the critical threshold where the statistic sat during known past incidents.
5. Record the resulting numbers, the date, and the window they were fitted on, in the spec.

This turns “is 0.18 bad?” into an answerable question. It also surfaces seasonality before it pages someone: a retail feature that legitimately shifts every November will show that shift in the backtest, and the threshold accommodates it instead of firing on it. For the mechanics of window sizing and test choice underneath this procedure, see data drift detection in ML: methods, tests, and practice.

Re-fit thresholds on a schedule. A threshold fitted eighteen months ago against a distribution that has legitimately moved is a threshold that has stopped meaning anything.

Step 4: write the spec down

Keep one versioned specification per model. The worked example below applies the combined-signal approach in the cited Azure monitoring guidance and the input-parity and prediction-quality checks in Google’s ML Test Score. Every sample, threshold, model identifier, and response policy below is hypothetical. These calculations illustrate a specification; they are not production measurements, vendor defaults, or recommended universal cutoffs.

Reference data, units, and normalization

Assume a toy binary classifier with two numeric inputs. Use four feature observations per window so every calculation is visible. Outcomes use a separate cohort of 20 predictions with fully matured labels. These samples are deliberately too small to establish a production alert policy.

For equal-sized, equally weighted numeric samples, calculate the one-dimensional Wasserstein distance as the mean absolute difference between their sorted values. Divide by the reference interquartile range (IQR) to express movement in fixed reference units. Use linearly interpolated quartiles, freeze the reference and scale, and do not normalize against the current window.

Reject empty, nonfinite, or mismatched feature samples before calculating drift. A zero IQR requires an explicit alternative scale or a value-change check; it must not be silently divided away. The small helper below only handles equal-sized windows. A production implementation must also specify missing-data handling and its distance calculation for unequal window sizes.

Calculate the warning and critical levels

Construct ten synthetic healthy replay windows by adding d * IQR to each reference value, for d in [0.02, 0.04, 0.06, 0.08, 0.10, 0.12, 0.14, 0.16, 0.18, 0.20]. Each window’s normalized distance is exactly d, for either feature. With linear interpolation, the 95th-percentile position is 0.95 * (10 - 1) = 8.55, giving a warning threshold of 0.18 + 0.55 * (0.20 - 0.18) = 0.191.

For a hypothetical critical rehearsal, shift payloads by 9 KB and account ages by 27 days. The normalized distances are 9 / 15 = 0.60 and 27 / 45 = 0.60. Select 0.60 as this toy policy’s critical threshold. This is a deliberately injected change, not a historical incident or a statistically estimated critical percentile. In native units the warning levels are 0.191 * 15 = 2.865 KB and 0.191 * 45 = 8.595 days; the critical levels are 9 KB and 27 days.

For outcomes, use the fixed prediction vector [0] * 10 + [1] * 10. Reference labels [1] + [0] * 9 + [0] + [1] * 9 give two errors: 2 / 20 = 0.10. The illustrative policy warns at one additional error, (2 + 1) / 20 = 0.15, and becomes critical at three additional errors, (2 + 3) / 20 = 0.25. These are explicit policy choices, not significance tests. Incomplete or immature labels produce an unavailable outcome metric, never an assumed zero error rate.

Work the current windows through the policy

The input-skew rehearsal deliberately doubles serving payload values while keeping the reference transformation fixed. In this constructed case, that is the known cause; in production a distance alert alone would not prove skew. The degraded cohort uses labels [1, 1, 1] + [0] * 7 + [0, 0, 0] + [1] * 7, so it has six mismatches against the same predictions despite healthy input distances.

The following calculation reproduces the distances, thresholds, and outcome rates:

import numpy as np

reference = {
    "payload_kb": np.array([10, 20, 30, 40], dtype=float),
    "account_age_days": np.array([30, 60, 90, 120], dtype=float),
}
scales = {
    name: float(np.diff(np.percentile(values, [25, 75]))[0])
    for name, values in reference.items()
}


def normalized_distance(name, current):
    ref = reference[name]
    cur = np.asarray(current, dtype=float)
    scale = scales[name]
    if (cur.shape != ref.shape or cur.size == 0
            or not np.isfinite(cur).all() or not np.isfinite(ref).all()
            or not np.isfinite(scale) or scale <= 0):
        raise ValueError("need equal finite samples and a positive reference IQR")
    return float(np.mean(np.abs(np.sort(cur) - np.sort(ref))) / scale)


shifts = np.arange(1, 11) * 0.02
healthy_replay = {
    name: [normalized_distance(name, values + d * scales[name]) for d in shifts]
    for name, values in reference.items()
}
warning = {name: float(np.percentile(values, 95))
           for name, values in healthy_replay.items()}
critical = {
    "payload_kb": normalized_distance("payload_kb", reference["payload_kb"] + 9),
    "account_age_days": normalized_distance(
        "account_age_days", reference["account_age_days"] + 27
    ),
}
windows = {
    "healthy": ([11.5, 21.5, 31.5, 41.5], [34.5, 64.5, 94.5, 124.5]),
    "warning": ([16, 26, 36, 46], [34.5, 64.5, 94.5, 124.5]),
    "input_skew": ([20, 40, 60, 80], [34.5, 64.5, 94.5, 124.5]),
    "degradation": ([11.5, 21.5, 31.5, 41.5], [34.5, 64.5, 94.5, 124.5]),
}
predictions = np.array([0] * 10 + [1] * 10)
baseline_labels = np.array([1] + [0] * 9 + [0] + [1] * 9)
degraded_labels = np.array([1, 1, 1] + [0] * 7 + [0, 0, 0] + [1] * 7)
baseline_errors = int(np.count_nonzero(predictions != baseline_labels))
outcome_warning = (baseline_errors + 1) / predictions.size
outcome_critical = (baseline_errors + 3) / predictions.size
print("input warning", warning, "input critical", critical)
print("outcome warning", outcome_warning, "outcome critical", outcome_critical)
for scenario, columns in windows.items():
    distances = {name: round(normalized_distance(name, values), 6)
                 for name, values in zip(reference, columns)}
    labels = degraded_labels if scenario == "degradation" else baseline_labels
    print(scenario, distances, float(np.mean(predictions != labels)))

Connect the calculations to the YAML

This is a proposed specification format, not executable configuration for a particular vendor. The values below come from the calculations above. Severity uses >=, with critical evaluated before warning. Evaluate each input separately and retain both input and outcome alert reasons.

example_kind: "hypothetical"
model: "example-scorer-v4"
tier: 1
owner: "model-operations"
reference:
  source: "synthetic-reference-v1"
  refresh: "version explicitly; recalculate scales and replay thresholds"
  payload_kb: [10, 20, 30, 40]
  account_age_days: [30, 60, 90, 120]
signals:
  inputs:
    test: "equal-weight 1-D Wasserstein divided by frozen reference IQR"
    window: "four synthetic observations versus four reference observations"
    percentile_method: "linear"
    invalid_sample: "data-quality alert; do not emit a drift score"
    features:
      payload_kb:
        unit: "KB"
        reference_iqr: 15
      account_age_days:
        unit: "days"
        reference_iqr: 45
  outcomes:
    metric: "classification_error_rate"
    cohort_size: 20
    baseline_errors: 2
    label_gate: "all 20 labels must be present and mature"
    incomplete_labels: "unavailable; retain input monitoring"
thresholds:
  fitted_on: "synthetic replay v1; ten healthy shifts and injected changes"
  normalized_distance_warn: 0.191
  normalized_distance_critical: 0.60
  error_rate_warn: 0.15
  error_rate_critical: 0.25
  comparison: ">="
  precedence: "critical before warning; preserve all alert reasons"
actions:
  healthy: "record results; continue monitoring"
  input_warn: "ticket model-operations to compare transformations and segments"
  input_critical: "page model-operations; inspect and correct upstream skew"
  outcome_warn: "ticket model-operations for mature-cohort quality review"
  outcome_critical: "page model-operations; evaluate a validated fallback"
  drift_confirmed: "investigate cause; do not automatically retrain"
review_cadence: "replay before replacing any reference or threshold"

owner routes the alert, fitted_on records the threshold evidence, and drift_confirmed keeps a distribution change from automatically triggering retraining. Healthy drift statistics do not override a confirmed outcome alert. Expand the synthetic replay into representative, seasonally matched historical windows and documented incident rehearsals before adopting this policy for a real service.

Step 5: define the action for every alert

An alert with no documented action is a notification, and notifications get muted. Each threshold in the spec maps to exactly one of four outcomes:

• Investigate. A human opens the drift report and decides. Correct for most tier 2 input-drift alerts.
• Roll back. The previous model version is restored. Correct when the alert fired within hours of a deployment.
• Fix upstream. The drift is a data engineering bug, not model degradation. Check this possibility before evaluating retraining.
• Retrain. Evaluate a retrain, on a labelled window, with the decision gated on offline metrics. Slowest and most expensive, and correct less often than teams assume.

Distinguishing the second and third from the fourth is the entire diagnostic value of the framework. The mechanism you are trying to name is whether the input distribution moved, the input-to-label relationship moved, or the pipeline broke, which is exactly the split covered in concept drift vs data drift explained.

Burn-rate style alerting keeps this from becoming a stream of one-off judgement calls; the adaptation of SRE error budgets to models with late labels is covered in SLOs and alerting for ML systems.

Severity routing and the performance floor

The following proposed routing table makes the response policy explicit. Set a performance floor before an incident, using the relevant quality metric, label availability, and consequences of failure.

Include the model and baseline versions, affected segments, sample size, and runbook link in each alert. Estimated performance needs its own uncertainty and escalation policy; it should not silently be treated as realized ground truth.

Step 6: set a review cadence

Monitoring rots. Features are deprecated, models are retrained, thresholds drift out of relevance, and dashboards accumulate panels that nobody has looked at in a year.

• Weekly. Tier 3 signal review, five minutes, in an existing team meeting.
• Monthly. Per-model spec review for tier 1 and 2. Did anything fire? Was the action taken the right one? Is fitted_on older than two quarters?
• Quarterly. Threshold re-fit and feature-set review. Delete signals that have never fired and never would have.
• On every retrain. Refresh the reference dataset and re-run the backtest. Keep the prior reference available for longitudinal comparison.

Where this sits against published frameworks

This blueprint is deliberately narrow. Two well-known references cover the surrounding ground and are worth reading against it.

Google’s ML Test Score is a 28-point production-readiness rubric, of which seven points are monitoring tests: notification on upstream dependency changes, data invariants in training and serving inputs, training-serving feature parity, model staleness, numerical stability, computational performance regression, and prediction quality regression on served data. Scoring an existing system against it is a fast way to find which plane above is missing entirely.

The NIST AI Risk Management Framework organises the same territory as governance rather than engineering. Its MEASURE function maps closely to the signal-selection step and its MANAGE function to the action step, which makes it the useful vocabulary when the monitoring spec has to satisfy an audit rather than an on-call rotation.

Google’s MLOps maturity levels are worth using honestly. Most teams asking for a monitoring framework are at level 0, with a manually deployed model and no automated retraining. The framework above works at level 0. It does not require a feature store, a model registry, or a pipeline orchestrator, and building those before the spec exists is a well-trodden way to spend a quarter and still not know when the model breaks.

A maturity ladder for the next improvement

This is a proposed implementation sequence, not Google’s MLOps maturity scale:

For LLM workloads, retain inventory, baseline, routing, and review while changing the signals. Version judge rubrics and retrieval inputs alongside the model. Monitoring tabular models vs LLM systems maps the signal changes, and the production quality sampling workflow covers the review loop.

What to build first

A practical implementation order is:

1. Pipeline liveness and schema conformance on every model. These checks can detect stale or malformed inputs without distribution tests.
2. Prediction distribution monitoring on every model. Label-free, one signal, moves early.
3. Input drift on the selected feature set, thresholds backtested, for tier 1 and 2 only.
4. Performance and estimated performance where labels or reliable proxies exist.
5. Segment-level breakdowns, which almost always reveal that an aggregate metric was hiding a subgroup failure.

Tooling comes last, once you know which of those five you need; the landscape and fit questions are covered in SentryML’s model monitoring tools comparison, and the open-source options in best drift detection libraries for Python. If the framework only tells you which statistical test belongs on a given feature, the Drift Test Selector answers that question directly.

Related on this site

• ML Model Monitoring Best Practices for Production Systems
• Concept Drift vs Data Drift Explained (and Prediction Drift)
• Types of Data Drift: Covariate, Label, and Concept
• PSI vs CSI: Stability Metrics for Credit Models
• Data Drift Detection in ML: Methods, Tests, and Practice
• SLOs and Alerting for ML Systems: Borrowing From SRE
• Monitoring Models When Ground Truth Is Late or Never Arrives