A model passes every offline eval, ships, and runs fine for two quarters. Then approval rates drift, or click-through on a ranking model quietly sags, and nobody gets paged because accuracy-based alerts need ground truth that has not arrived yet. If you have searched for population stability index explained, you are already at the point every team hits eventually: you need a number that flags a shifted input distribution before the label ever shows up.
That number is PSI. It is older than most of the MLOps tooling built around it, coming out of credit-risk scorecard monitoring at banks where regulators expect a documented drift check. It migrated into general ML monitoring for the same reasons it worked there: it does not need labels, it collapses a whole feature distribution into one comparable scalar, and thirty years of credit-risk practice already agreed on what “bad” looks like. Several monitoring tools expose PSI, but available methods and defaults vary by implementation and feature type.
For the scorecard distinction between score distributions and individual inputs, read PSI vs CSI. Choose complementary checks with PSI vs KS vs calibration, and use monitoring models without ground truth to plan for outcomes that arrive later. The ML model monitoring framework assigns owners and actions to those signals.
The metric that matters
PSI compares a reference distribution (usually your training set, or last week’s traffic) against a current one, bucket by bucket:
PSI = Σ_b (Actual_b - Expected_b) * ln(Actual_b / Expected_b)For each bin b, Expected_b is the proportion of the reference sample that falls in that bin and Actual_b is the proportion of the current sample. Sum that term across every bin and you get a single non-negative scalar per feature: zero when the distributions match exactly, growing as they diverge.
Why this over a raw KS test or a mean/variance comparison? Three reasons. First, a KS test gives you a p-value tied to sample size; at production scale, with millions of rows, it rejects the null on noise-level shifts that do not matter, and it will not tell you how much the distribution moved. PSI produces a magnitude, which is what lets you set the same threshold across a feature measured in dollars and one measured in log-odds. Second, PSI is symmetric and bounded below by zero regardless of which side you call “reference,” so you do not get sign confusion across dashboards. Third, unlike a mean/variance check, PSI buckets the distribution first, so it catches shape changes such as a bimodal split or a fat tail appearing, which a first-moment comparison sails right past. PSI is not always the right pick, though, and the Drift Test Selector will tell you when a feature’s type, volume, or your detection goal points at Wasserstein or a KS test instead.
Fiddler’s writeup points out that PSI is mathematically a symmetrized form of KL divergence, so you inherit the information-theoretic interpretation without having to explain KL divergence to a risk committee. There is a firmer theoretical floor than the practitioner heuristic suggests: a 2025 arXiv paper on information-theoretic credit risk modeling proves that PSI, computed between two outcome groups, is exactly Jeffreys divergence, the symmetrized form of KL. That is why PSI behaves like a proper divergence (non-negative, zero only at equality) rather than an ad hoc statistic someone invented for a compliance memo.
The convention most teams use, per Arthur’s docs:
These are inherited numbers, not derived ones, and treating them as universal is the most common way a PSI monitor becomes noise. Backtesting your own thresholds against a healthy historical window is a step in the ML model monitoring framework; the conventional bands below are the starting point you calibrate away from.
- PSI < 0.1 — little to no shift, no action
- 0.1 ≤ PSI < 0.25 — moderate shift, worth watching
- PSI ≥ 0.25 — significant shift, treat as a retraining or investigation trigger
Fiddler’s own cutoff is tighter, 0.2 instead of 0.25, which is a reminder that these bands are industry convention, not a theorem. Calibrate them against your own feature’s historical volatility before wiring alerts to page anyone.
Bins are the whole game
PSI’s number depends entirely on how you bucket. Two conventions dominate:
- Equal-width bins, fixed-width ranges across the variable’s span. WhyLabs defaults to 30 equal-width bins for its PSI implementation, and Arthur recommends 10 to 20 equal-width bins between the reference min and max.
- Equal-frequency (quantile) bins, with edges chosen so the reference distribution has equal counts per bin, typically 10 bins in the credit-scoring convention.
Too few bins wash out real shifts; too many turn sampling noise into false alarms, especially on a small production batch.
Whichever you pick, compute the bin edges once from the reference distribution and freeze them. If you recompute edges from each new batch, you are comparing two distributions that were each binned to look uniform, which drives PSI toward zero regardless of actual drift. Evidently’s data drift documentation makes this explicit in its API: the reference dataset supplied at report creation time is what defines the bins used for every later comparison.
Wiring it up
PSI ships in every mainstream drift library, so the choice is rarely about the metric and usually about what else the library has to cover: batch tabular, streaming, or embeddings. Python drift detection libraries compared lays out that split.
Legacy API example: pin evidently==0.6.7 in an isolated environment. This example uses that version’s DataFrame API throughout; it is not a current-API quickstart. Supply numeric feature DataFrames as train_df and prod_window_df. The Evidently drift documentation covers method customization, but API generations must not be mixed.
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset
report = Report(metrics=[
DataDriftPreset(stattest="psi", stattest_threshold=0.2, drift_share=0.7)
])
report.run(reference_data=train_df, current_data=prod_window_df)
result = report.as_dict()
drift = next(
metric["result"] for metric in result["metrics"]
if metric["metric"] == "DatasetDriftMetric"
)
print(drift["number_of_drifted_columns"], drift["share_of_drifted_columns"])Here, 0.2 is the illustrative per-column PSI cutoff and drift_share=0.7 sets the dataset-level decision threshold on the share of drifted columns. Review individual columns too: an aggregate rule can hide a failure in one important feature. Calibrate both thresholds against representative history.
If you would rather not pull in a monitoring library, PSI is short enough to hand-roll, which is worth doing at least once so you understand where the failure modes below come from:
import numpy as np
def finite_sample(values, name):
values = np.asarray(values, dtype=float)
if values.ndim != 1 or values.size == 0 or not np.isfinite(values).all():
raise ValueError(f"{name} must be a nonempty, finite, 1-D sample")
return values
def reference_edges(reference, bins=10):
reference = finite_sample(reference, "reference")
if isinstance(bins, bool) or not isinstance(bins, (int, np.integer)) or bins < 2:
raise ValueError("bins must be an integer of at least 2")
if reference.min() == reference.max():
raise ValueError("constant reference: use an explicit value-change check")
quantiles = np.linspace(0, 100, bins + 1)[1:-1]
interior = np.unique(np.percentile(reference, quantiles))
return np.r_[-np.inf, interior, np.inf]
def psi(expected, actual, bins=10, eps=1e-6):
expected = finite_sample(expected, "expected")
actual = finite_sample(actual, "actual")
if not np.isfinite(eps) or not 0 < eps < 1:
raise ValueError("eps must be finite and between 0 and 1")
edges = reference_edges(expected, bins)
exp_counts = np.histogram(expected, edges)[0]
act_counts = np.histogram(actual, edges)[0]
exp_pct = exp_counts / exp_counts.sum()
act_pct = act_counts / act_counts.sum()
exp_pct = np.clip(exp_pct, eps, None)
act_pct = np.clip(act_pct, eps, None)
exp_pct /= exp_pct.sum()
act_pct /= act_pct.sum()
return float(np.sum((act_pct - exp_pct) * np.log(act_pct / exp_pct)))
# Hypothetical overflow example: all 12 current observations count.
reference = np.arange(10, dtype=float)
current = np.r_[reference, 100.0, 100.0]
edges = reference_edges(reference)
assert np.histogram(reference, edges)[0].sum() == reference.size == 10
assert np.histogram(current, edges)[0].sum() == current.size == 12
print(round(psi(reference, current), 6)) # 0.164792Only the interior boundaries come from reference quantiles; infinite outer edges retain values below or above the reference range. Repeated quantiles collapse to one boundary. Keep the same reference and boundaries across runs. This numeric example rejects empty, nonfinite, multidimensional, and constant-reference inputs; handle missing data separately and use a value-change check for a constant reference. The eps floor prevents zero probabilities, and each distribution is normalized again after smoothing. These choices are part of the metric definition, so record them with its thresholds.
Push the per-feature PSI value to whatever you already scrape metrics with. A prometheus_client gauge keyed on feature_name is enough to get it on a Grafana panel next to latency and throughput:
from prometheus_client import Gauge
psi_gauge = Gauge("model_feature_psi", "PSI vs training reference", ["feature_name"])
psi_gauge.labels(feature_name="transaction_amount").set(psi_value)If you are logging with MLflow, attach it as a metric on the scoring run (mlflow.log_metric("psi_transaction_amount", psi_value)) so PSI history lives next to the model version it was scored against, rather than floating in a separate dashboard with no link back to what shipped.
For a wider view of where PSI sits relative to KS tests, Jensen-Shannon distance, and Wasserstein distance, see our guide to data drift detection methods, and SentryML’s monitoring-metrics taxonomy maps the full set and when each one earns its keep.
What you will see
Good: a per-feature PSI line hovering under 0.1 on a rolling 7- or 30-day window, flat enough that you stop looking at it except in a weekly review. Bad: one feature, say transaction amount after a pricing change, or a categorical field after a new signup source gets turned on, climbing past 0.25 while everything else stays flat. That isolation is the useful signal: PSI computed per feature tells you which input changed, not just that something did, so triage starts at the feature instead of the whole pipeline. Credit-risk practice gives that per-feature version its own name, the Characteristic Stability Index, and reads it against the score-level number; PSI vs CSI covers why the pair catches offsetting shifts that either one alone reports as stable.
The other useful signal is persistence rather than peak. A one-hour spike that decays back down is usually a batch job hiccup or a partial data-pipeline delay, not drift worth acting on. Real covariate shift, from a new customer segment, a currency added to checkout, or an upstream schema change that started zero-filling a field, climbs and stays there.
If every feature’s PSI moves in lockstep, suspect the pipeline before the world: a schema change, a null-fill default that changed, or a reference window that rolled over a seasonal boundary. Holiday traffic compared against a summer baseline will “drift” every year, and that is a calendar problem, not a model problem. Those pipeline-side shifts sit alongside the textbook covariate, label and concept cases in types of data drift, which is the classification step to run before a PSI spike gets called drift at all.
Caveats
Empty bins break the math. If a bin has zero actual or expected observations, the log term goes to infinity. The eps smoothing above is the standard workaround, and Fiddler flags it as the most common source of a PSI job crashing silently or reporting inf. Depending on which side is zero, PSI either blows up or silently reports zero contribution from a bin that should have flagged loudly. Pick an epsilon and document it, because it changes the number.
Categorical cardinality blows up fast. A categorical feature with hundreds of levels effectively becomes hundreds of near-empty bins, and PSI on it is mostly noise. Cap or group rare categories before computing PSI, the same way you would handle high-cardinality features anywhere else in the pipeline.
Bin count and edges are not portable across teams. Two teams computing PSI on the same feature with different bin counts will get different numbers and possibly different alert outcomes. Arize’s PSI explainer notes this is why the metric works best as a within-team, within-pipeline standard rather than a cross-org benchmark.
PSI tells you that something moved, not whether it hurts. It is a marginal, univariate statistic. It checks each feature’s distribution independently, has no visibility into how features move together, and none at all into whether P(y|x) has changed. A model can pass every PSI check on inputs while its actual relationship to the label has quietly broken; that is concept drift, and PSI will not see it. A feature can also drift because of a benign product change, such as new UI copy shifting session length, that does not touch model performance at all. Pair PSI on inputs with an accuracy or calibration metric once labels arrive. The ML model monitoring framework keeps input stability separate from performance evidence and response policy.
Decide the reference type up front and say so on the dashboard. A fixed training snapshot catches slow drift away from training but goes stale as the world legitimately changes; a rolling baseline catches sudden shifts but can mask a slow multi-month drift because each window looks only slightly different from the last.
