ML Monitoring Report
Flat isometric illustration of a dotted blue grid plane with a rising cyan curve between two glowing endpoints and one bright white node.
monitoring-practice

ML Model Monitoring Framework: A Practical Blueprint

Six steps to an ML model monitoring framework: tiering, label cohort sizes, backtested PSI thresholds, a YAML spec, alert actions and governance mapping.

By ML Monitoring Report Editorial · · Updated · 14 min read

An ML model monitoring framework is one versioned specification per model: its tier, label regime, signals, backtested thresholds, alert routes, the action each alert triggers, and a review cadence. Build it in six steps: tier by consequence and label latency, choose signals, backtest thresholds, write the spec, map alerts to actions, and schedule reviews.

The monitoring tool implements the spec; it does not write it. Instrumentation detail lives in ML model monitoring best practices. Below are the decisions above that layer and the numbers behind them: a label sample-size table, a backtest you can run on public data, a spec template, a cost worksheet and a governance crosswalk.

Framework at a glance

StepDecisionArtifact
1. TierHow much damage a wrong prediction does, and how long its label takes to matureConsequence tier table, tier x label-regime matrix, min_cohort from the sample-size formula
2. SignalsWhich checks run on the infrastructure, input, output and outcome planessignals list with plane, statistic, window and segment keys
3. ThresholdsWhich warn and critical levels hold up on the model’s own historyFitted thresholds with fitted_on and the backtest counts
4. SpecWhere every decision above is recordedOne versioned spec file per model
5. ActionsWhat each alert triggers and where it routesAlert-to-action map and severity routing
6. ReviewHow often the spec, thresholds and reference are revisitedReview cadence in the spec’s review field

Step 1: tier by consequence, then by label regime

Every later decision depends on two properties of the model’s situation, not its architecture: how much damage a wrong prediction does, and how long you wait for the label that proves it wrong. Record them separately, because a critical model can still have outcomes that take months to mature.

TierConsequenceResponse preparation
1Direct material impact per decisionOn-call route, documented performance floor, validated fallback
2Aggregate product or revenue impactOwning team and response deadline
3Internal or advisory outputNamed team and review cadence
4Experimental, shadow or offlineRelease gate before wider use

The tiers are illustrative; reassess one when an experiment starts influencing a customer-facing decision. Banks will recognise the idea: under the April 2026 interagency guidance (SR 26-2), “model purpose, together with model exposure, determines model materiality”, and oversight of immaterial models “may consist of identifying those models and monitoring model performance and conditions”, as Sullivan & Cromwell’s summary quotes it. The regulated version is in PSI vs CSI.

Label latency falls into one of three regimes in NannyML’s taxonomy: instant (its example is arrival-time estimation, scorable once the trip ends), delayed, or absent. Crossing regime with tier gives each model its primary alert plane and outcome-review cadence:

TierInstant labelsDelayed labelsAbsent labels
1Outcomes page once the minimum cohort is reachedPredictions and inputs page; outcome review as each cohort maturesPredictions and inputs page; monthly human-label audit
2Outcomes ticket; weekly reviewPrediction drift tickets; monthly matured-cohort reviewDrift tickets; quarterly audit sample
3Outcome trend in the weekly reviewPipeline and volume checks; quarterly outcome reviewPipeline and volume checks only
4Offline evaluation at the release gateSameSame

The rule underneath the table: an outcome check cannot run more often than the label maturity lag plus the time needed to accumulate a minimum cohort of matured labels. Estimation and holdout strategies for the blind window are in model monitoring without ground truth.

How many matured labels an outcome alert needs

Small cohorts turn outcome alerts into coin tosses. At a 10% baseline error rate, a cohort of 20 predictions contains three or more errors about 32% of the time by chance (the binomial tail, binom.sf(2, 20, 0.10)), so a warn-at-three rule fires on roughly one healthy cohort in three.

Size the cohort instead. The one-sided formula for testing a proportion against a known baseline, from the NIST/SEMATECH e-Handbook, section 7.2.4.2, is N ≥ ((z₁₋α·√(p₀(1−p₀)) + z₁₋β·√(p₁(1−p₁))) / δ)². With α = 0.05 one-sided and 80% power:

Baseline errorDetect +1 pointDetect +2 pointsDetect +5 points
2%1,398 (28 d / 2.8 d)391 (7.8 d / 0.8 d)80 (1.6 d / 0.2 d)
5%3,118 (62 d / 6.2 d)822 (16 d / 1.6 d)150 (3.0 d / 0.3 d)
10%5,728 (115 d / 11.5 d)1,471 (29 d / 2.9 d)253 (5.1 d / 0.5 d)

Brackets show accumulation time at 50 and at 500 matured labels per day; add the maturity lag on top. Halving the detectable effect roughly quadruples the labels, so a low-volume Tier 1 model cannot promise to catch a one-point rise within a month. The formula treats the baseline as known; if it comes from a small validation set, a two-sample calculation needs more. A weekly check at α = 0.05 on a healthy model still fires about 2.6 times a year per metric, which belongs in the routing decision. Reproduce the table, and check cells with few expected errors against the exact binomial:

from math import ceil, sqrt
from scipy.stats import binom, norm

def min_cohort(p0, delta, alpha=0.05, power=0.80):
    p1, za, zb = p0 + delta, norm.ppf(1 - alpha), norm.ppf(power)
    return ceil(((za * sqrt(p0 * (1 - p0)) + zb * sqrt(p1 * (1 - p1))) / delta) ** 2)

def exact_check(n, p0, delta, alpha=0.05):
    c = next(k for k in range(n + 1) if binom.sf(k - 1, n, p0) <= alpha)
    return c, binom.sf(c - 1, n, p0), binom.sf(c - 1, n, p0 + delta)  # count, alpha, power

for p0 in (0.02, 0.05, 0.10):
    print(p0, [min_cohort(p0, d) for d in (0.01, 0.02, 0.05)])
print(exact_check(80, 0.02, 0.05))

Step 2: pick signals per plane, not per feature

Four planes: infrastructure (latency, errors, throughput, on the platform’s existing SRE stack), inputs (schema, nulls, volume, drift on a selected feature set), outputs (prediction distribution and calibration, label-free but ambiguous about cause) and outcomes (the metric the model is judged on, when labels exist). Azure ML’s guidance recommends combining signals and monitoring a feature subset. Per-feature test choice is in PSI vs KS vs calibration, and what each test can see is in types of data drift. One check belongs in every spec: training-serving skew stays invisible unless the reference comes from the training pipeline.

Step 3: backtest thresholds on real history

PSI bands of 0.1 and 0.25 are credit-scorecard convention (PSI explained), and a KS p-value on a million rows flags differences nobody cares about. The spec needs thresholds fitted to the model’s own history, plus evidence that the fitted threshold beats the convention. Rehearse the procedure on public data, then point it at your own logs.

Data. The NYC TLC trip records are monthly Parquet files of yellow-taxi trips with pickup and drop-off times, zones and distances, published with about a two-month delay. The task is trip-duration prediction, an instant-label problem. The period contains a dated change to check alerts against: the Congestion Relief Zone toll began on 5 January 2025 (MTA), taxi riders pay $0.75 per zone trip, and TLC added a cbd_congestion_fee column from 2025 data.

Design. Train and reference on January to March 2023. Fit thresholds on April 2023 to March 2024; evaluate on April 2024 to June 2025. Compute weekly PSI on trip_distance (an input) and on predicted minutes (the output), with 10 quantile bins frozen from the reference. The realized metric is weekly mean absolute error; a week counts as degraded when MAE exceeds the holdout MAE by more than 10%, a tolerance to replace with your own floor. Compare PSI ≥ 0.1, PSI ≥ 0.2 and the 95th percentile of PSI over healthy fit-period weeks.

pip install pandas pyarrow scikit-learn scipy matplotlib
mkdir -p tlc
for m in 2023-{01..12} 2024-{01..12} 2025-{01..06}; do
  curl -sSf -o "tlc/yellow_tripdata_${m}.parquet" \
    "https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_${m}.parquet"
done
import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingRegressor

COLS = ["tpep_pickup_datetime", "tpep_dropoff_datetime",
        "trip_distance", "PULocationID", "DOLocationID"]
X = ["trip_distance", "PULocationID", "DOLocationID", "hour", "dow"]

def load(month, frac=0.1):  # lower frac if memory is tight
    df = pd.read_parquet(f"tlc/yellow_tripdata_{month}.parquet", columns=COLS)
    start, t = pd.Timestamp(f"{month}-01"), df.tpep_pickup_datetime
    df = df[(t >= start) & (t < start + pd.offsets.MonthBegin())]
    df = df.sample(frac=frac, random_state=0)
    df["minutes"] = (df.tpep_dropoff_datetime
                     - df.tpep_pickup_datetime).dt.total_seconds() / 60
    df["hour"] = df.tpep_pickup_datetime.dt.hour
    df["dow"] = df.tpep_pickup_datetime.dt.dayofweek
    return df[df.minutes.between(1, 180) & df.trip_distance.between(0.1, 100)]

def edges(base, bins=10):
    inner = np.quantile(base, np.linspace(0, 1, bins + 1)[1:-1])
    return np.r_[-np.inf, np.unique(inner), np.inf]

def psi(base, cur, e, eps=1e-6):
    p = np.clip(np.histogram(base, e)[0] / len(base), eps, None)
    q = np.clip(np.histogram(cur, e)[0] / len(cur), eps, None)
    return float(np.sum((q - p) * np.log(q / p)))

months = [str(p) for p in pd.period_range("2023-01", "2025-06", freq="M")]
data = pd.concat([load(m) for m in months], ignore_index=True)
ref = data[data.tpep_pickup_datetime < "2023-04-01"]
train = ref.sample(frac=0.8, random_state=0)
hold = ref.drop(train.index)
model = HistGradientBoostingRegressor(random_state=0).fit(train[X], train.minutes)
ref_pred = model.predict(hold[X])
base_mae = np.mean(np.abs(ref_pred - hold.minutes))

live = data[data.tpep_pickup_datetime >= "2023-04-01"].copy()
live["pred"] = model.predict(live[X])
live["err"] = (live.pred - live.minutes).abs()
live["week"] = live.tpep_pickup_datetime.dt.to_period("W").dt.start_time
e_dist, e_pred = edges(hold.trip_distance), edges(ref_pred)
w = pd.DataFrame([
    {"week": wk, "n": len(g), "mae": g.err.mean(),
     "psi_distance": psi(hold.trip_distance, g.trip_distance, e_dist),
     "psi_pred": psi(ref_pred, g.pred, e_pred)}
    for wk, g in live.groupby("week")])
w = w[w.n >= 0.5 * w.n.median()].copy()  # drop partial edge weeks
w["degraded"] = w.mae > 1.10 * base_mae
fit, test = w[w.week < "2024-04-01"], w[w.week >= "2024-04-01"]

for stat in ("psi_distance", "psi_pred"):
    p95 = fit.loc[~fit.degraded, stat].quantile(0.95)
    for name, thr in (("0.1", 0.1), ("0.2", 0.2), (f"p95={p95:.3f}", p95)):
        a = test[stat] >= thr
        print(stat, name, "alerts", int(a.sum()),
              "on degraded weeks", int((a & test.degraded).sum()),
              "degraded weeks missed", int((~a & test.degraded).sum()))
w.to_csv("weekly_drift.csv", index=False)

Chart the weekly series against the three thresholds, with degraded weeks shaded and the toll start marked:

import matplotlib.pyplot as plt

p95 = fit.loc[~fit.degraded, "psi_pred"].quantile(0.95)
fig, ax = plt.subplots(figsize=(10, 4))
for i, wk in enumerate(w.week[w.degraded]):
    ax.axvspan(wk, wk + pd.Timedelta(days=7), color="#e4e3de", lw=0,
               label="MAE >10% above holdout" if i == 0 else None)
ax.plot(w.week, w.psi_distance, color="#2a78d6", lw=2, label="PSI, trip_distance")
ax.plot(w.week, w.psi_pred, color="#eb6834", lw=2, label="PSI, predicted minutes")
for y, ls, text in ((0.1, ":", "0.1"), (0.2, "--", "0.2"), (p95, "-.", "fitted p95")):
    ax.axhline(y, color="#52514e", lw=1, ls=ls)
    ax.text(w.week.iloc[-1], y, " " + text, va="center", color="#52514e", fontsize=8)
ax.axvline(pd.Timestamp("2025-01-05"), color="#52514e", lw=1, label="Toll starts")
ax.set_ylabel("weekly PSI vs Q1 2023 reference")
ax.spines[["top", "right"]].set_visible(False)
ax.legend(frameon=False, loc="upper left")
fig.savefig("weekly_psi.png", dpi=150, bbox_inches="tight")

What to record. For each statistic and threshold, store the backtest counts in the spec’s thresholds block next to fitted_on (fields in the Step 4 template): total alerts in the evaluation period, alerts on degraded weeks, degraded weeks missed, and whether an alert fired within two weeks of the 5 January 2025 toll start.

How to read it. Alerts on healthy weeks are the threshold’s pager cost; degraded weeks without an alert are its blind spots. If a fixed band fires on many healthy weeks, it is noise for this model. If no drift threshold catches the degraded weeks, input drift is not a leading indicator here: keep it as a ticket and let the outcome plane page, which an instant-label model can afford. With about 50 healthy fit weeks, the p95 is set by the top two or three, so treat it as a rough cut and re-fit as history grows. The fit year includes Thanksgiving and the December holidays; decide explicitly whether to exclude those weeks or compare against a same-season reference, and write the decision down. Window and test mechanics are in data drift detection in ML.

Step 4: write the spec, one versioned file per model

A blank template, each field defined in its comment:

model: ""               # registry name and version under watch
owner: ""               # team queue that receives alerts, never a person
tier: 0                 # 1-4 from the consequence table
consequence: ""         # what one wrong prediction costs, and to whom
labels:
  regime: ""            # instant | delayed | absent
  maturity_days: 0      # days until ~95% of eventual labels exist, measured from history
  matured_per_day: 0    # labels per day that reach maturity
primary_alert_plane: "" # from the tier x label-regime table
reference:
  dataset: ""           # versioned snapshot ID
  window: ""            # date range it covers
  refresh: ""           # what triggers replacement; the prior reference is kept
signals: []             # name, plane, statistic, window, segment keys
thresholds:
  method: ""            # backtest percentile | SLO floor | documented convention
  fitted_on: ""         # data range, fit date, commit of the backtest
  warn: {}              # per signal
  critical: {}          # per signal
  backtest_alerts_total: {}                     # per signal and threshold: alerts in the evaluation period
  backtest_alerts_on_degraded_weeks: {}         # per signal and threshold: alerts that fell on degraded weeks
  backtest_degraded_weeks_missed: {}            # per signal and threshold: degraded weeks with no alert
  alerted_within_two_weeks_of_known_change: {}  # per signal and threshold: true | false for a dated change
outcome_check:
  metric: ""            # e.g. error rate on the matured cohort
  baseline: 0.0         # value and its source
  detect: 0.0           # smallest increase that must be caught
  alpha: 0.05
  power: 0.80
  min_cohort: 0         # from the sample-size formula
  cadence: ""           # no shorter than maturity_days + min_cohort / matured_per_day
routing: {}             # severity -> page | ticket | review queue
actions: {}             # alert -> investigate | roll back | fix upstream | retrain evaluation
fallback: ""            # validated fallback, or "none" with the reason
review: ""              # Step 6 cadence
governance_refs: []     # clauses this spec evidences (see the crosswalk)

Three contrasting archetypes, filled in. The values are illustrative policy choices; the point is which fields move with tier and label latency.

FieldCard fraudSame-day rankerRegulated credit scorecard
tier121
labels.regimeDelayedInstantDelayed
labels.maturity_daysMeasured from chargeback history; Visa card-absent fraud (10.4) can be disputed up to 120 calendar days after processing (Chargebacks911)Same sessionMonths: defaults surface over the performance window
primary_alert_planePredictions and inputs, plus estimated performanceOutcomesScore PSI and characteristic CSI
outcome_check.cadenceEach weekly cohort once matureDaily rolling cohortMonthly or quarterly vintages
referenceFrozen training snapshot plus same-season windowRolling four weeks plus frozen launch baselineFrozen, validated development sample
thresholds.methodBacktest percentile on prediction driftSLO floor on the outcome metricDocumented PSI bands plus backtest
routing (critical)Page fraud on-callPage only for serving failureTicket model validation; page for pipeline failure
fallbackRules enginePrevious model versionPrior scorecard version
label biasBlocked transactions never produce chargebacksPosition bias in clicksDeclined applicants never show repayment
governance_refsSR 26-2 if a bank modelML Test Score Monitor 1–7SR 26-2; EU AI Act Annex III 5(b)

The last row is where the archetypes diverge most. The EU AI Act’s Annex III point 5(b) lists systems that evaluate the creditworthiness of natural persons or establish their credit score as high-risk, “with the exception of AI systems used for the purpose of detecting financial fraud”. One lender can therefore owe an Article 72 monitoring plan for the scorecard while the fraud model falls outside point 5(b). Handling the label-bias row is covered in monitoring without ground truth.

Step 5: map every alert to one action

An alert with no documented action is a notification, and notifications get muted. Each threshold maps to one of four actions:

  • Investigate. A human reads the drift report and decides. Right for most Tier 2 input-drift alerts.
  • Roll back. Restore the previous version. Right when the alert fires within hours of a deployment.
  • Fix upstream. The shift is a data engineering bug. Rule this out before any retraining.
  • Evaluate a retrain. On a labelled window, gated on offline metrics, never triggered by a drift score alone.

Telling the middle two apart from the last is the framework’s diagnostic value: did the inputs move, did the input-to-label relationship move, or did the pipeline break (concept drift vs data drift)? Route by severity: serving failure or confirmed quality below the documented floor pages on-call; required-input failure or material drift tickets the owner with a deadline; low-impact trends wait for the scheduled review. Every alert carries the model and baseline versions, affected segments, sample size and runbook link. Burn-rate alerting for late-label models is in SLOs and alerting for ML systems.

Step 6: set the review cadence

  • Weekly. Tier 3 signal review, five minutes in an existing meeting. Tier 1 matured-cohort review where the outcome cadence allows.
  • Monthly. Spec review for Tiers 1 and 2: what fired, was the action right, is fitted_on older than two quarters?
  • Quarterly. Re-run the backtest, re-fit thresholds, delete signals that never fired and never would have.
  • On every retrain. New reference and a fresh backtest; keep the prior reference for longitudinal comparison.

What tiering costs: a worksheet

Prices are AWS US East list prices as published in September 2026: S3 Standard at $0.023 per GB-month (first 50 TB) and $0.005 per 1,000 PUT requests; CloudWatch custom metrics at $0.30 per metric-month and alarms at $0.10 per alarm metric-month, with 10 of each free; Athena at $5 per TB scanned; Fargate at about $0.0405 per vCPU-hour and $0.00444 per GB-hour. Reviewer time uses the BLS median data-scientist wage of $57.80 an hour (May 2025), a wage rather than a loaded cost. Volumes and hours are example assumptions.

Line itemTier 1: log every requestTier 3: nightly 1% sample
Volume50 rps × 2 KB = 8.64 GB/day, 259.2 GB/month2.59 GB/month
Storage, 12 months retained3,110 GB: $71.54/month31.1 GB: $0.72/month
WritesOne object per request, 129.6M PUTs: $648/month. One per minute: $0.22One object per night: under $0.01
Drift jobAthena over the trailing 7 days, daily (60.5 GB × 30): $9.07/month, or Fargate 2 vCPU/8 GB for 15 min nightly: $0.87/monthFargate 15 min weekly: $0.12/month
Metrics and alarms25 metrics + 25 alarms: $6.00/month (after the 10 free metrics and 10 free alarms)3 + 3, inside the free tier
Reviewer time8 h/month (weekly cohort review, monthly spec review, amortised quarterly re-fit): $462.4020 min/month: $19.27

Two things stand out. Unbatched writes alone ($648) cost more than the rest of the Tier 1 column combined, reviewer time included, and people cost more than all the batched infrastructure. The other trap is metric cardinality: breaking 20 feature-drift metrics out by 10 segments makes 200 metrics, $60 a month before alarms. Tiering keeps the bill sane because Tier 3 models get no per-request logs, per-segment metrics or weekly human review.

Governance crosswalk

StepML Test ScoreNIST AI RMF 1.0SR 26-2EU AI Act Art. 72ISO/IEC 42001:2023
Inventory, tieringnoneGOVERN 1.6Materiality from purpose and exposure72(1) proportionate to risks9.1 what to monitor
SignalsMonitor 1–7MEASURE 2.4Ongoing monitoring72(2) collect, document, analyse9.1 what to measure
ThresholdsMonitor 2, Monitor 7MEASURE 2.4none72(2) continuous compliance9.1 methods giving valid results
RoutingnoneGOVERN 1.5 rolesnone72(3) plannone
ResponsenoneMANAGE 4.1nonenonenone
Review0.5 manual, 1 automatedGOVERN 1.5 frequencyRisk-based, no fixed validation cadence72(3) plan9.1 when to monitor and evaluate

“None” means no clause maps cleanly, not that the document is silent.

ML Test Score. The rubric has 28 tests; the seven monitoring tests cover dependency-change notification, data invariants, training-serving skew, staleness, numerical stability, computational performance and prediction quality. Per the paper’s scoring, a test earns 0.5 points when run manually with results documented and distributed, and 1 point when automated and repeated; the final score is the minimum across the four sections, and 0 reads as “more of a research project than a productionized system”. Score the monitoring section for each Tier 1 model first.

NIST AI RMF. GOVERN 1.6 asks for AI system inventories resourced by risk priority; GOVERN 1.5 wants roles defined and “the frequency of periodic review” determined. MEASURE 2.4 requires that functionality and behaviour “are monitored when in production”. MANAGE 4.1 requires implemented post-deployment monitoring plans covering user input, “appeal and override, decommissioning, incident response, recovery, and change management”, which is Step 5’s action table.

SR 26-2. Ongoing monitoring asks whether a model performs as expected as products, exposures, clients, data relevance or market conditions change; Sullivan & Cromwell notes there is no required validation cadence. Generative and agentic AI models “are not within the scope” (OCC Bulletin 2026-13), so LLM specs need other anchors.

EU AI Act and ISO. Article 72 makes the post-market monitoring plan part of the Annex IV technical documentation and set 2 February 2026 for the Commission’s template. ISO/IEC 42001 clause 9.1 requires deciding what to monitor, how, when, and how results are analysed, with documented evidence; check the licensed text before an audit.

LLM workloads and where to start

For LLM workloads, keep inventory, tiering, spec, routing and review, and change the signals; monitoring tabular models vs LLM systems maps what transfers. Most teams asking for a framework sit at level 0 of Google’s MLOps maturity levels, with manual deployment and no automated retraining, and nothing here needs a feature store or orchestrator. Start with liveness, schema and prediction-distribution checks on every model, then pick tools: Python drift libraries, or the Drift Test Selector for per-feature tests.

FAQ

what should an ml model monitoring framework include?

It should include a model inventory with an owner and tier for each model, the label regime and maturity lag, signals on each plane, thresholds with the data they were fitted on, a severity-to-route table, one documented action per alert, and a review schedule. Keep one versioned file per model so every change is auditable.

how many labels do i need to detect a drop in model accuracy?

It depends on the baseline error rate and the smallest rise you must catch. At a 5% baseline, detecting a two-point rise with one-sided alpha 0.05 and 80% power takes about 822 matured labels; a one-point rise takes about 3,118. Divide by matured labels per day, then add the label maturity lag.

is a psi threshold of 0.2 good for drift alerts?

Only as a starting guess. The 0.1, 0.2 and 0.25 cutoffs are inherited conventions, not values fitted to your model. Replay a year of weekly PSI, fit a percentile over weeks where the realized metric stayed healthy, and keep whichever threshold catches degraded weeks with the fewest alerts on healthy ones.

does the eu ai act require monitoring ml models after deployment?

For high-risk AI systems, yes. Article 72 requires providers to establish and document a post-market monitoring system proportionate to the system’s risks, based on a plan that forms part of the Annex IV technical documentation. Credit scoring of natural persons is listed as high-risk in Annex III; fraud detection is excluded from that entry.

Sources

  1. MLOps: Continuous delivery and automation pipelines in machine learning — Google Cloud Architecture Center
  2. The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction (IEEE Big Data 2017)
  3. NIST AI Risk Management Framework (AI RMF 1.0)
  4. Monitor data drift on models deployed to production — Azure Machine Learning documentation
  5. SR 26-2: Revised Interagency Guidance on Model Risk Management (Federal Reserve, 17 April 2026)
  6. Model monitoring in production — Azure Machine Learning documentation
  7. The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction — Google Research publication page
  8. The ML Test Score notes: monitoring tests, scoring and score bands (GitHub gist)
  9. NIST AI RMF Playbook: GOVERN
  10. NIST AI RMF Playbook: MEASURE
  11. NIST AI RMF Playbook: MANAGE
  12. EU AI Act, Article 72: Post-Market Monitoring by Providers and Post-Market Monitoring Plan for High-Risk AI Systems
  13. EU AI Act, Annex III: High-Risk AI Systems Referred to in Article 6(2)
  14. OCC Bulletin 2026-13: Model Risk Management: Revised Guidance (17 April 2026)
  15. Federal Banking Agencies Issue Revised Guidance on Model Risk Management — Sullivan & Cromwell LLP
  16. ISO/IEC 42001 Clause 9 Performance Evaluation: Guidance — Cyberzoni
  17. TLC Trip Record Data — NYC Taxi and Limousine Commission
  18. Congestion Relief Zone frequently asked questions — MTA
  19. Congestion Relief Zone Toll: Taxis and FHVs — MTA
  20. NIST/SEMATECH e-Handbook of Statistical Methods, 7.2.4.2 Sample sizes required
  21. Why is Machine Learning Monitoring in production hard? (availability of ground truth) — NannyML
  22. Visa Reason Code 10.4: Other Fraud, Card-Absent Environment — Chargebacks911
  23. Amazon S3 Pricing
  24. Amazon CloudWatch Pricing
  25. Amazon Athena Pricing
  26. AWS Fargate Pricing
  27. Data Scientists: Occupational Outlook Handbook — U.S. Bureau of Labor Statistics
Subscribe

ML Monitoring Report — in your inbox

Production ML monitoring, drift, and reliability. Sent only when there is something worth sending.

No spam. Unsubscribe anytime.

Related