An ML model monitoring framework is one versioned specification per model: its tier, label regime, signals, backtested thresholds, alert routes, the action each alert triggers, and a review cadence. Build it in six steps: tier by consequence and label latency, choose signals, backtest thresholds, write the spec, map alerts to actions, and schedule reviews.
The monitoring tool implements the spec; it does not write it. Instrumentation detail lives in ML model monitoring best practices. Below are the decisions above that layer and the numbers behind them: a label sample-size table, a backtest you can run on public data, a spec template, a cost worksheet and a governance crosswalk.
Framework at a glance
| Step | Decision | Artifact |
|---|---|---|
| 1. Tier | How much damage a wrong prediction does, and how long its label takes to mature | Consequence tier table, tier x label-regime matrix, min_cohort from the sample-size formula |
| 2. Signals | Which checks run on the infrastructure, input, output and outcome planes | signals list with plane, statistic, window and segment keys |
| 3. Thresholds | Which warn and critical levels hold up on the model’s own history | Fitted thresholds with fitted_on and the backtest counts |
| 4. Spec | Where every decision above is recorded | One versioned spec file per model |
| 5. Actions | What each alert triggers and where it routes | Alert-to-action map and severity routing |
| 6. Review | How often the spec, thresholds and reference are revisited | Review cadence in the spec’s review field |
Step 1: tier by consequence, then by label regime
Every later decision depends on two properties of the model’s situation, not its architecture: how much damage a wrong prediction does, and how long you wait for the label that proves it wrong. Record them separately, because a critical model can still have outcomes that take months to mature.
| Tier | Consequence | Response preparation |
|---|---|---|
| 1 | Direct material impact per decision | On-call route, documented performance floor, validated fallback |
| 2 | Aggregate product or revenue impact | Owning team and response deadline |
| 3 | Internal or advisory output | Named team and review cadence |
| 4 | Experimental, shadow or offline | Release gate before wider use |
The tiers are illustrative; reassess one when an experiment starts influencing a customer-facing decision. Banks will recognise the idea: under the April 2026 interagency guidance (SR 26-2), “model purpose, together with model exposure, determines model materiality”, and oversight of immaterial models “may consist of identifying those models and monitoring model performance and conditions”, as Sullivan & Cromwell’s summary quotes it. The regulated version is in PSI vs CSI.
Label latency falls into one of three regimes in NannyML’s taxonomy: instant (its example is arrival-time estimation, scorable once the trip ends), delayed, or absent. Crossing regime with tier gives each model its primary alert plane and outcome-review cadence:
| Tier | Instant labels | Delayed labels | Absent labels |
|---|---|---|---|
| 1 | Outcomes page once the minimum cohort is reached | Predictions and inputs page; outcome review as each cohort matures | Predictions and inputs page; monthly human-label audit |
| 2 | Outcomes ticket; weekly review | Prediction drift tickets; monthly matured-cohort review | Drift tickets; quarterly audit sample |
| 3 | Outcome trend in the weekly review | Pipeline and volume checks; quarterly outcome review | Pipeline and volume checks only |
| 4 | Offline evaluation at the release gate | Same | Same |
The rule underneath the table: an outcome check cannot run more often than the label maturity lag plus the time needed to accumulate a minimum cohort of matured labels. Estimation and holdout strategies for the blind window are in model monitoring without ground truth.
How many matured labels an outcome alert needs
Small cohorts turn outcome alerts into coin tosses. At a 10% baseline error rate, a cohort of 20 predictions contains three or more errors about 32% of the time by chance (the binomial tail, binom.sf(2, 20, 0.10)), so a warn-at-three rule fires on roughly one healthy cohort in three.
Size the cohort instead. The one-sided formula for testing a proportion against a known baseline, from the NIST/SEMATECH e-Handbook, section 7.2.4.2, is N ≥ ((z₁₋α·√(p₀(1−p₀)) + z₁₋β·√(p₁(1−p₁))) / δ)². With α = 0.05 one-sided and 80% power:
| Baseline error | Detect +1 point | Detect +2 points | Detect +5 points |
|---|---|---|---|
| 2% | 1,398 (28 d / 2.8 d) | 391 (7.8 d / 0.8 d) | 80 (1.6 d / 0.2 d) |
| 5% | 3,118 (62 d / 6.2 d) | 822 (16 d / 1.6 d) | 150 (3.0 d / 0.3 d) |
| 10% | 5,728 (115 d / 11.5 d) | 1,471 (29 d / 2.9 d) | 253 (5.1 d / 0.5 d) |
Brackets show accumulation time at 50 and at 500 matured labels per day; add the maturity lag on top. Halving the detectable effect roughly quadruples the labels, so a low-volume Tier 1 model cannot promise to catch a one-point rise within a month. The formula treats the baseline as known; if it comes from a small validation set, a two-sample calculation needs more. A weekly check at α = 0.05 on a healthy model still fires about 2.6 times a year per metric, which belongs in the routing decision. Reproduce the table, and check cells with few expected errors against the exact binomial:
from math import ceil, sqrt
from scipy.stats import binom, norm
def min_cohort(p0, delta, alpha=0.05, power=0.80):
p1, za, zb = p0 + delta, norm.ppf(1 - alpha), norm.ppf(power)
return ceil(((za * sqrt(p0 * (1 - p0)) + zb * sqrt(p1 * (1 - p1))) / delta) ** 2)
def exact_check(n, p0, delta, alpha=0.05):
c = next(k for k in range(n + 1) if binom.sf(k - 1, n, p0) <= alpha)
return c, binom.sf(c - 1, n, p0), binom.sf(c - 1, n, p0 + delta) # count, alpha, power
for p0 in (0.02, 0.05, 0.10):
print(p0, [min_cohort(p0, d) for d in (0.01, 0.02, 0.05)])
print(exact_check(80, 0.02, 0.05))
Step 2: pick signals per plane, not per feature
Four planes: infrastructure (latency, errors, throughput, on the platform’s existing SRE stack), inputs (schema, nulls, volume, drift on a selected feature set), outputs (prediction distribution and calibration, label-free but ambiguous about cause) and outcomes (the metric the model is judged on, when labels exist). Azure ML’s guidance recommends combining signals and monitoring a feature subset. Per-feature test choice is in PSI vs KS vs calibration, and what each test can see is in types of data drift. One check belongs in every spec: training-serving skew stays invisible unless the reference comes from the training pipeline.
Step 3: backtest thresholds on real history
PSI bands of 0.1 and 0.25 are credit-scorecard convention (PSI explained), and a KS p-value on a million rows flags differences nobody cares about. The spec needs thresholds fitted to the model’s own history, plus evidence that the fitted threshold beats the convention. Rehearse the procedure on public data, then point it at your own logs.
Data. The NYC TLC trip records are monthly Parquet files of yellow-taxi trips with pickup and drop-off times, zones and distances, published with about a two-month delay. The task is trip-duration prediction, an instant-label problem. The period contains a dated change to check alerts against: the Congestion Relief Zone toll began on 5 January 2025 (MTA), taxi riders pay $0.75 per zone trip, and TLC added a cbd_congestion_fee column from 2025 data.
Design. Train and reference on January to March 2023. Fit thresholds on April 2023 to March 2024; evaluate on April 2024 to June 2025. Compute weekly PSI on trip_distance (an input) and on predicted minutes (the output), with 10 quantile bins frozen from the reference. The realized metric is weekly mean absolute error; a week counts as degraded when MAE exceeds the holdout MAE by more than 10%, a tolerance to replace with your own floor. Compare PSI ≥ 0.1, PSI ≥ 0.2 and the 95th percentile of PSI over healthy fit-period weeks.
pip install pandas pyarrow scikit-learn scipy matplotlib
mkdir -p tlc
for m in 2023-{01..12} 2024-{01..12} 2025-{01..06}; do
curl -sSf -o "tlc/yellow_tripdata_${m}.parquet" \
"https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_${m}.parquet"
done
import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingRegressor
COLS = ["tpep_pickup_datetime", "tpep_dropoff_datetime",
"trip_distance", "PULocationID", "DOLocationID"]
X = ["trip_distance", "PULocationID", "DOLocationID", "hour", "dow"]
def load(month, frac=0.1): # lower frac if memory is tight
df = pd.read_parquet(f"tlc/yellow_tripdata_{month}.parquet", columns=COLS)
start, t = pd.Timestamp(f"{month}-01"), df.tpep_pickup_datetime
df = df[(t >= start) & (t < start + pd.offsets.MonthBegin())]
df = df.sample(frac=frac, random_state=0)
df["minutes"] = (df.tpep_dropoff_datetime
- df.tpep_pickup_datetime).dt.total_seconds() / 60
df["hour"] = df.tpep_pickup_datetime.dt.hour
df["dow"] = df.tpep_pickup_datetime.dt.dayofweek
return df[df.minutes.between(1, 180) & df.trip_distance.between(0.1, 100)]
def edges(base, bins=10):
inner = np.quantile(base, np.linspace(0, 1, bins + 1)[1:-1])
return np.r_[-np.inf, np.unique(inner), np.inf]
def psi(base, cur, e, eps=1e-6):
p = np.clip(np.histogram(base, e)[0] / len(base), eps, None)
q = np.clip(np.histogram(cur, e)[0] / len(cur), eps, None)
return float(np.sum((q - p) * np.log(q / p)))
months = [str(p) for p in pd.period_range("2023-01", "2025-06", freq="M")]
data = pd.concat([load(m) for m in months], ignore_index=True)
ref = data[data.tpep_pickup_datetime < "2023-04-01"]
train = ref.sample(frac=0.8, random_state=0)
hold = ref.drop(train.index)
model = HistGradientBoostingRegressor(random_state=0).fit(train[X], train.minutes)
ref_pred = model.predict(hold[X])
base_mae = np.mean(np.abs(ref_pred - hold.minutes))
live = data[data.tpep_pickup_datetime >= "2023-04-01"].copy()
live["pred"] = model.predict(live[X])
live["err"] = (live.pred - live.minutes).abs()
live["week"] = live.tpep_pickup_datetime.dt.to_period("W").dt.start_time
e_dist, e_pred = edges(hold.trip_distance), edges(ref_pred)
w = pd.DataFrame([
{"week": wk, "n": len(g), "mae": g.err.mean(),
"psi_distance": psi(hold.trip_distance, g.trip_distance, e_dist),
"psi_pred": psi(ref_pred, g.pred, e_pred)}
for wk, g in live.groupby("week")])
w = w[w.n >= 0.5 * w.n.median()].copy() # drop partial edge weeks
w["degraded"] = w.mae > 1.10 * base_mae
fit, test = w[w.week < "2024-04-01"], w[w.week >= "2024-04-01"]
for stat in ("psi_distance", "psi_pred"):
p95 = fit.loc[~fit.degraded, stat].quantile(0.95)
for name, thr in (("0.1", 0.1), ("0.2", 0.2), (f"p95={p95:.3f}", p95)):
a = test[stat] >= thr
print(stat, name, "alerts", int(a.sum()),
"on degraded weeks", int((a & test.degraded).sum()),
"degraded weeks missed", int((~a & test.degraded).sum()))
w.to_csv("weekly_drift.csv", index=False)
Chart the weekly series against the three thresholds, with degraded weeks shaded and the toll start marked:
import matplotlib.pyplot as plt
p95 = fit.loc[~fit.degraded, "psi_pred"].quantile(0.95)
fig, ax = plt.subplots(figsize=(10, 4))
for i, wk in enumerate(w.week[w.degraded]):
ax.axvspan(wk, wk + pd.Timedelta(days=7), color="#e4e3de", lw=0,
label="MAE >10% above holdout" if i == 0 else None)
ax.plot(w.week, w.psi_distance, color="#2a78d6", lw=2, label="PSI, trip_distance")
ax.plot(w.week, w.psi_pred, color="#eb6834", lw=2, label="PSI, predicted minutes")
for y, ls, text in ((0.1, ":", "0.1"), (0.2, "--", "0.2"), (p95, "-.", "fitted p95")):
ax.axhline(y, color="#52514e", lw=1, ls=ls)
ax.text(w.week.iloc[-1], y, " " + text, va="center", color="#52514e", fontsize=8)
ax.axvline(pd.Timestamp("2025-01-05"), color="#52514e", lw=1, label="Toll starts")
ax.set_ylabel("weekly PSI vs Q1 2023 reference")
ax.spines[["top", "right"]].set_visible(False)
ax.legend(frameon=False, loc="upper left")
fig.savefig("weekly_psi.png", dpi=150, bbox_inches="tight")
What to record. For each statistic and threshold, store the backtest counts in the spec’s thresholds block next to fitted_on (fields in the Step 4 template): total alerts in the evaluation period, alerts on degraded weeks, degraded weeks missed, and whether an alert fired within two weeks of the 5 January 2025 toll start.
How to read it. Alerts on healthy weeks are the threshold’s pager cost; degraded weeks without an alert are its blind spots. If a fixed band fires on many healthy weeks, it is noise for this model. If no drift threshold catches the degraded weeks, input drift is not a leading indicator here: keep it as a ticket and let the outcome plane page, which an instant-label model can afford. With about 50 healthy fit weeks, the p95 is set by the top two or three, so treat it as a rough cut and re-fit as history grows. The fit year includes Thanksgiving and the December holidays; decide explicitly whether to exclude those weeks or compare against a same-season reference, and write the decision down. Window and test mechanics are in data drift detection in ML.
Step 4: write the spec, one versioned file per model
A blank template, each field defined in its comment:
model: "" # registry name and version under watch
owner: "" # team queue that receives alerts, never a person
tier: 0 # 1-4 from the consequence table
consequence: "" # what one wrong prediction costs, and to whom
labels:
regime: "" # instant | delayed | absent
maturity_days: 0 # days until ~95% of eventual labels exist, measured from history
matured_per_day: 0 # labels per day that reach maturity
primary_alert_plane: "" # from the tier x label-regime table
reference:
dataset: "" # versioned snapshot ID
window: "" # date range it covers
refresh: "" # what triggers replacement; the prior reference is kept
signals: [] # name, plane, statistic, window, segment keys
thresholds:
method: "" # backtest percentile | SLO floor | documented convention
fitted_on: "" # data range, fit date, commit of the backtest
warn: {} # per signal
critical: {} # per signal
backtest_alerts_total: {} # per signal and threshold: alerts in the evaluation period
backtest_alerts_on_degraded_weeks: {} # per signal and threshold: alerts that fell on degraded weeks
backtest_degraded_weeks_missed: {} # per signal and threshold: degraded weeks with no alert
alerted_within_two_weeks_of_known_change: {} # per signal and threshold: true | false for a dated change
outcome_check:
metric: "" # e.g. error rate on the matured cohort
baseline: 0.0 # value and its source
detect: 0.0 # smallest increase that must be caught
alpha: 0.05
power: 0.80
min_cohort: 0 # from the sample-size formula
cadence: "" # no shorter than maturity_days + min_cohort / matured_per_day
routing: {} # severity -> page | ticket | review queue
actions: {} # alert -> investigate | roll back | fix upstream | retrain evaluation
fallback: "" # validated fallback, or "none" with the reason
review: "" # Step 6 cadence
governance_refs: [] # clauses this spec evidences (see the crosswalk)
Three contrasting archetypes, filled in. The values are illustrative policy choices; the point is which fields move with tier and label latency.
| Field | Card fraud | Same-day ranker | Regulated credit scorecard |
|---|---|---|---|
| tier | 1 | 2 | 1 |
| labels.regime | Delayed | Instant | Delayed |
| labels.maturity_days | Measured from chargeback history; Visa card-absent fraud (10.4) can be disputed up to 120 calendar days after processing (Chargebacks911) | Same session | Months: defaults surface over the performance window |
| primary_alert_plane | Predictions and inputs, plus estimated performance | Outcomes | Score PSI and characteristic CSI |
| outcome_check.cadence | Each weekly cohort once mature | Daily rolling cohort | Monthly or quarterly vintages |
| reference | Frozen training snapshot plus same-season window | Rolling four weeks plus frozen launch baseline | Frozen, validated development sample |
| thresholds.method | Backtest percentile on prediction drift | SLO floor on the outcome metric | Documented PSI bands plus backtest |
| routing (critical) | Page fraud on-call | Page only for serving failure | Ticket model validation; page for pipeline failure |
| fallback | Rules engine | Previous model version | Prior scorecard version |
| label bias | Blocked transactions never produce chargebacks | Position bias in clicks | Declined applicants never show repayment |
| governance_refs | SR 26-2 if a bank model | ML Test Score Monitor 1–7 | SR 26-2; EU AI Act Annex III 5(b) |
The last row is where the archetypes diverge most. The EU AI Act’s Annex III point 5(b) lists systems that evaluate the creditworthiness of natural persons or establish their credit score as high-risk, “with the exception of AI systems used for the purpose of detecting financial fraud”. One lender can therefore owe an Article 72 monitoring plan for the scorecard while the fraud model falls outside point 5(b). Handling the label-bias row is covered in monitoring without ground truth.
Step 5: map every alert to one action
An alert with no documented action is a notification, and notifications get muted. Each threshold maps to one of four actions:
- Investigate. A human reads the drift report and decides. Right for most Tier 2 input-drift alerts.
- Roll back. Restore the previous version. Right when the alert fires within hours of a deployment.
- Fix upstream. The shift is a data engineering bug. Rule this out before any retraining.
- Evaluate a retrain. On a labelled window, gated on offline metrics, never triggered by a drift score alone.
Telling the middle two apart from the last is the framework’s diagnostic value: did the inputs move, did the input-to-label relationship move, or did the pipeline break (concept drift vs data drift)? Route by severity: serving failure or confirmed quality below the documented floor pages on-call; required-input failure or material drift tickets the owner with a deadline; low-impact trends wait for the scheduled review. Every alert carries the model and baseline versions, affected segments, sample size and runbook link. Burn-rate alerting for late-label models is in SLOs and alerting for ML systems.
Step 6: set the review cadence
- Weekly. Tier 3 signal review, five minutes in an existing meeting. Tier 1 matured-cohort review where the outcome cadence allows.
- Monthly. Spec review for Tiers 1 and 2: what fired, was the action right, is
fitted_onolder than two quarters? - Quarterly. Re-run the backtest, re-fit thresholds, delete signals that never fired and never would have.
- On every retrain. New reference and a fresh backtest; keep the prior reference for longitudinal comparison.
What tiering costs: a worksheet
Prices are AWS US East list prices as published in September 2026: S3 Standard at $0.023 per GB-month (first 50 TB) and $0.005 per 1,000 PUT requests; CloudWatch custom metrics at $0.30 per metric-month and alarms at $0.10 per alarm metric-month, with 10 of each free; Athena at $5 per TB scanned; Fargate at about $0.0405 per vCPU-hour and $0.00444 per GB-hour. Reviewer time uses the BLS median data-scientist wage of $57.80 an hour (May 2025), a wage rather than a loaded cost. Volumes and hours are example assumptions.
| Line item | Tier 1: log every request | Tier 3: nightly 1% sample |
|---|---|---|
| Volume | 50 rps × 2 KB = 8.64 GB/day, 259.2 GB/month | 2.59 GB/month |
| Storage, 12 months retained | 3,110 GB: $71.54/month | 31.1 GB: $0.72/month |
| Writes | One object per request, 129.6M PUTs: $648/month. One per minute: $0.22 | One object per night: under $0.01 |
| Drift job | Athena over the trailing 7 days, daily (60.5 GB × 30): $9.07/month, or Fargate 2 vCPU/8 GB for 15 min nightly: $0.87/month | Fargate 15 min weekly: $0.12/month |
| Metrics and alarms | 25 metrics + 25 alarms: $6.00/month (after the 10 free metrics and 10 free alarms) | 3 + 3, inside the free tier |
| Reviewer time | 8 h/month (weekly cohort review, monthly spec review, amortised quarterly re-fit): $462.40 | 20 min/month: $19.27 |
Two things stand out. Unbatched writes alone ($648) cost more than the rest of the Tier 1 column combined, reviewer time included, and people cost more than all the batched infrastructure. The other trap is metric cardinality: breaking 20 feature-drift metrics out by 10 segments makes 200 metrics, $60 a month before alarms. Tiering keeps the bill sane because Tier 3 models get no per-request logs, per-segment metrics or weekly human review.
Governance crosswalk
| Step | ML Test Score | NIST AI RMF 1.0 | SR 26-2 | EU AI Act Art. 72 | ISO/IEC 42001:2023 |
|---|---|---|---|---|---|
| Inventory, tiering | none | GOVERN 1.6 | Materiality from purpose and exposure | 72(1) proportionate to risks | 9.1 what to monitor |
| Signals | Monitor 1–7 | MEASURE 2.4 | Ongoing monitoring | 72(2) collect, document, analyse | 9.1 what to measure |
| Thresholds | Monitor 2, Monitor 7 | MEASURE 2.4 | none | 72(2) continuous compliance | 9.1 methods giving valid results |
| Routing | none | GOVERN 1.5 roles | none | 72(3) plan | none |
| Response | none | MANAGE 4.1 | none | none | none |
| Review | 0.5 manual, 1 automated | GOVERN 1.5 frequency | Risk-based, no fixed validation cadence | 72(3) plan | 9.1 when to monitor and evaluate |
“None” means no clause maps cleanly, not that the document is silent.
ML Test Score. The rubric has 28 tests; the seven monitoring tests cover dependency-change notification, data invariants, training-serving skew, staleness, numerical stability, computational performance and prediction quality. Per the paper’s scoring, a test earns 0.5 points when run manually with results documented and distributed, and 1 point when automated and repeated; the final score is the minimum across the four sections, and 0 reads as “more of a research project than a productionized system”. Score the monitoring section for each Tier 1 model first.
NIST AI RMF. GOVERN 1.6 asks for AI system inventories resourced by risk priority; GOVERN 1.5 wants roles defined and “the frequency of periodic review” determined. MEASURE 2.4 requires that functionality and behaviour “are monitored when in production”. MANAGE 4.1 requires implemented post-deployment monitoring plans covering user input, “appeal and override, decommissioning, incident response, recovery, and change management”, which is Step 5’s action table.
SR 26-2. Ongoing monitoring asks whether a model performs as expected as products, exposures, clients, data relevance or market conditions change; Sullivan & Cromwell notes there is no required validation cadence. Generative and agentic AI models “are not within the scope” (OCC Bulletin 2026-13), so LLM specs need other anchors.
EU AI Act and ISO. Article 72 makes the post-market monitoring plan part of the Annex IV technical documentation and set 2 February 2026 for the Commission’s template. ISO/IEC 42001 clause 9.1 requires deciding what to monitor, how, when, and how results are analysed, with documented evidence; check the licensed text before an audit.
LLM workloads and where to start
For LLM workloads, keep inventory, tiering, spec, routing and review, and change the signals; monitoring tabular models vs LLM systems maps what transfers. Most teams asking for a framework sit at level 0 of Google’s MLOps maturity levels, with manual deployment and no automated retraining, and nothing here needs a feature store or orchestrator. Start with liveness, schema and prediction-distribution checks on every model, then pick tools: Python drift libraries, or the Drift Test Selector for per-feature tests.
FAQ
what should an ml model monitoring framework include?
It should include a model inventory with an owner and tier for each model, the label regime and maturity lag, signals on each plane, thresholds with the data they were fitted on, a severity-to-route table, one documented action per alert, and a review schedule. Keep one versioned file per model so every change is auditable.
how many labels do i need to detect a drop in model accuracy?
It depends on the baseline error rate and the smallest rise you must catch. At a 5% baseline, detecting a two-point rise with one-sided alpha 0.05 and 80% power takes about 822 matured labels; a one-point rise takes about 3,118. Divide by matured labels per day, then add the label maturity lag.
is a psi threshold of 0.2 good for drift alerts?
Only as a starting guess. The 0.1, 0.2 and 0.25 cutoffs are inherited conventions, not values fitted to your model. Replay a year of weekly PSI, fit a percentile over weeks where the realized metric stayed healthy, and keep whichever threshold catches degraded weeks with the fewest alerts on healthy ones.
does the eu ai act require monitoring ml models after deployment?
For high-risk AI systems, yes. Article 72 requires providers to establish and document a post-market monitoring system proportionate to the system’s risks, based on a plan that forms part of the Annex IV technical documentation. Credit scoring of natural persons is listed as high-risk in Annex III; fraud detection is excluded from that entry.