Brier Score

The Brier score (BS) is the mean squared error of a probabilistic prediction. For a binary outcome y in {0,1} and a predicted probability p in [0,1] over N subjects, BS = (1/N) * sum_i (p_i - y_i)^2. It ranges from 0 (perfect: every prediction equals the realized outcome) to 1 (worst: confident and always wrong); a non-informative model that always predicts the marginal prevalence q achieves BS = q*(1-q). Originating in weather forecast verification (Brier, 1950), it is now the standard overall performance measure for clinical and RWE prediction models because, unlike discrimination metrics (AUC, the c-statistic) that depend only on the ranking of predictions, the Brier score penalizes predictions that are mis-scaled — a model can rank perfectly (AUC 1.0) yet be badly calibrated and earn a poor Brier score.

Core conceptual distinction — the decomposition

The Brier score's analytic value is that it splits into interpretable parts. Grouping predictions into K bins of (approximately) constant predicted probability, Murphy's (1973) decomposition is BS = reliability - resolution + uncertainty, where reliability = (1/N) * sum_k n_k * (p_bar_k - o_bar_k)^2 (the squared gap between mean predicted probability p_bar_k and observed outcome rate o_bar_k in each bin — lower is better, this is the calibration term), resolution = (1/N) * sum_k n_k * (o_bar_k - o_bar)^2 (how far each bin's outcome rate departs from the overall rate o_bar — higher is better, this is discrimination/sharpness), and uncertainty = o_bar*(1 - o_bar) (the irreducible variance of the outcome, fixed by the data).

A second, equivalent split is calibration + refinement, where refinement = uncertainty - resolution. The lesson: a low Brier score can be achieved either by good calibration or by high resolution, so the raw score conflates two distinct virtues — the decomposition is what lets you attribute a model's Brier performance to its calibration versus its sharpness.

Scaling

Because the achievable Brier score depends on outcome prevalence (a rare outcome makes a low absolute BS easy), the raw score is not comparable across datasets with different prevalence. The scaled Brier score (a.k.a. Brier skill score), BS_scaled = 1 - BS / BS_ref where BS_ref = q*(1-q) is the score of the prevalence-only model, rescales to a 0-to-1 "proportion of maximum achievable" metric (1 = perfect, 0 = no better than predicting the marginal rate, negative = worse than the null model). Report the scaled Brier score for cross-study comparison; report the raw score with its decomposition within a study.

Pros, cons, and trade-offs

• vs the c-statistic / AUC (`roc-auc-discrimination-rwe`): The Brier score is a proper scoring rule — it is minimized in expectation only by the true probabilities, so it cannot be gamed by mis-scaling and it rewards honest probability estimates. AUC measures only discrimination (ranking) and is invariant to any monotone re-scaling of predictions, so it is silent about calibration. Prefer the Brier score (with its decomposition) when the predicted probability itself will be used (for thresholds, expected cost, decision curves); prefer AUC when only the rank order matters and you want a prevalence-invariant discrimination summary. Report both — they answer different questions.
• vs a calibration plot / Hosmer-Lemeshow: A calibration plot (and the calibration slope/intercept) shows where and how miscalibration occurs across the probability range, information the single Brier number compresses away; the reliability term of the decomposition is the scalar summary of exactly that plot. Prefer the calibration plot to diagnose the shape of miscalibration; use the Brier score / reliability term as the one-number summary and for model selection. The Hosmer-Lemeshow test is a discredited goodness-of-fit hypothesis test (power depends on N, arbitrary binning); the reliability decomposition is the preferred quantitative alternative.
• vs the logarithmic score (log loss): Both are proper scoring rules. Log loss penalizes confident errors far more harshly (it is unbounded as p -> 0 for a positive case) and is the right loss when extreme over-confidence must be punished; the Brier score is bounded, more robust to a handful of catastrophic predictions, and decomposes cleanly into calibration and refinement. Prefer the Brier score for interpretable decomposition and robustness; prefer log loss when calibrated tail probabilities are critical.

When NOT to use — and when it is actively misleading or dangerous

• Comparing raw Brier scores across datasets with different prevalence. A rare outcome makes a low raw BS trivially attainable (the null model already scores q*(1-q) ~= q for small q); comparing raw scores across populations confounds model quality with base rate. Use the scaled Brier score and report prevalence.
• As a substitute for the calibration plot when the question is where the model fails. The single Brier number cannot tell you whether the model is over-confident at high risk and under-confident at low risk; treating a good Brier score as proof of calibration across the whole range is misleading. Inspect the plot or the bin-wise reliability.
• For a model that will only ever be used as a rank-ordering tool. If only the ordering of patients matters (e.g., waitlist prioritization) and no probability is acted on numerically, the calibration that the Brier score rewards is irrelevant; AUC or a rank-based metric is the appropriate target and optimizing Brier may add nothing.
• On predictions evaluated on the same data used to fit them, without correction. An apparent (resubstitution) Brier score is optimistically biased; report a Brier score from cross-validation, bootstrap optimism correction, or an external validation sample, exactly as for any other performance measure.

Data-source operational depth

• Claims (FFS): The Brier score requires a binary outcome label that is itself a computable phenotype, so its honesty inherits the phenotype's misclassification — a noisy outcome label inflates the irreducible-uncertainty term and caps the achievable score. Build predictions and outcomes on FFS-observable person-time only; Medicare Advantage enrollees generate no claims, so an MA-only span yields a fabricated "no event" label that biases both the outcome rate o_bar and the reliability term. For time-to-event outcomes use the time-dependent (inverse-probability-of-censoring-weighted) Brier score so administrative censoring does not masquerade as event-free survival.
• EHR: Richer features usually improve resolution (sharper, more separated predictions), but encounter-driven ascertainment makes an out-of-system event an unobserved "negative," biasing the outcome label and therefore the calibration term; require demonstrable in-system activity before labeling a subject event-free, and treat informative loss to follow-up with censoring-weighted Brier scores.
• Registry / linked: Adjudicated registry outcomes give the cleanest binary labels, tightening every term of the decomposition; linked claims-EHR-vital-records additionally supplies a death index so a competing terminal event is not miscoded as event-free. Linkage selects the linkable subset, so check that the validation sample's prevalence matches the deployment population before trusting a scaled Brier score.

Worked example

A logistic model predicts 1-year mortality in a linked claims-registry cohort of N = 5,000 with observed prevalence o_bar = 0.10 (so uncertainty = 0.10*0.90 = 0.090, and the prevalence-only null model scores BS_ref = 0.090). Suppose the fitted model achieves raw BS = 0.072. Then the scaled Brier score = 1 - 0.072/0.090 = 0.20 — the model explains 20% of the maximum achievable improvement over predicting the marginal rate, a modest but real gain.

Decomposing across deciles of predicted risk: if reliability = 0.004 (small squared calibration gaps) and resolution = 0.022, then BS = reliability - resolution + uncertainty = 0.004 - 0.022 + 0.090 = 0.072, confirming the arithmetic and attributing most of the improvement to resolution (the model separates risk groups) with good calibration (low reliability term).

The same model might post AUC = 0.74; reporting AUC alone would have hidden the calibration story, and reporting raw BS alone would have hidden that 0.072 is only modestly better than the 0.090 null. Report all three — scaled Brier 0.20, the reliability/resolution split, and a calibration plot — for a complete performance picture.

Interpreting the output

In the large-scale worked example, the model achieves Brier score = 0.072, scaled Brier = 0.20, reliability = 0.004, and resolution = 0.022, against a null Brier of 0.090.

(1) Formal interpretation. The Brier score is the mean squared error between predicted probabilities and binary outcomes. A score of 0.072 is lower (better) than the null score of 0.090, which a model predicting the marginal event rate for everyone would achieve. The scaled Brier of 0.20 expresses this improvement as a proportion of the maximum possible reduction, with 0 = no skill and 1 = perfect calibration and discrimination.

The Murphy decomposition separates BS into reliability (average squared calibration gap between predicted and observed — lower is better) and resolution (variance of predicted probabilities — higher means the model spreads risk groups apart). A reliability term of 0.004 indicates predictions are close to observed rates on average; a resolution of 0.022 indicates meaningful risk-group separation. When calibration slope < 1, predictions are too extreme and shrinkage or recalibration is needed; a slope > 1 indicates predictions are too conservative.

(2) Practical interpretation. A scaled Brier of 0.20 is a modest but real gain over the null — adequate for population-level risk stratification but not sufficient for individual-level clinical decisions that require tighter calibration. The low reliability (0.004) confirms predictions track observed rates well, which matters whenever predicted probabilities drive enrollment thresholds or resource-allocation rules. Always pair the Brier score with AUC to separate calibration from discrimination: the same model here posts AUC = 0.74, showing acceptable rank ordering; the Brier adds the calibration dimension that AUC cannot see.

Decision diagram

flowchart TD
  BS[Brier score<br/>mean&#40;p - y&#41;^2] --> D{Murphy decomposition}
  D --> Rel[Reliability<br/>calibration gap per bin<br/>LOWER is better]
  D --> Res[Resolution<br/>bin rates vs overall rate<br/>HIGHER is better]
  D --> Unc[Uncertainty<br/>o&#183;&#40;1-o&#41; irreducible<br/>fixed by the data]
  Rel --> Eq[BS = Reliability - Resolution + Uncertainty]
  Res --> Eq
  Unc --> Eq
  Eq --> Scaled[Scaled Brier = 1 - BS / &#40;o&#183;&#40;1-o&#41;&#41;<br/>comparable across prevalence]

flowchart LR
  P[Predicted probabilities + binary outcomes<br/>from a validation sample] --> Q{What is the question?}
  Q -->|Rank order only| AUC[Use AUC / c-statistic<br/>prevalence-invariant discrimination]
  Q -->|Probability is acted on| Brier[Compute Brier score]
  Brier --> Cal[Pair with calibration plot<br/>+ slope/intercept]
  Brier --> Cmp{Comparing across<br/>different prevalence?}
  Cmp -->|Yes| Sc[Report scaled Brier score]
  Cmp -->|No| Raw[Report raw Brier + decomposition]
  Cal --> Fix[If miscalibrated -> recalibrate<br/>intercept/slope or refit]

Worked example

Scenario

A research team builds a model that predicts whether a patient will be hospitalized within 90 days. They test it on five patients from a validation dataset. For each patient the model outputs a predicted probability of hospitalization, and at the end of 90 days the team records whether hospitalization actually occurred (1 = yes, 0 = no). The team wants to compute the Brier score to summarize how well the model's probabilities matched reality, and to discuss what the score tells them about calibration.

Dataset

Validation rows: one row per patient with the model's predicted probability and the observed 90-day hospitalization outcome.

Steps

Result

Brier score = (0.01 + 0.04 + 0.09 + 0.16 + 0.04) / 5 = 0.34 / 5 = 0.068. The null model (always predict 0.60) would score 0.240, so the model substantially outperforms a guess based on the overall event rate alone. The model is reasonably well-calibrated: its average predicted probability (0.56) is close to the observed event rate (0.60), and the largest single squared error (0.16, from P04 who was hospitalized but only scored 0.6) flags the one patient where the model was most under-confident.

Trade-offs

Runnable example

import numpy as np
from sklearn.metrics import brier_score_loss

def brier_decomposition(y_true, y_prob, n_bins=10):
    """Raw Brier score, scaled Brier (skill) score, and Murphy's decomposition."""
    y_true = np.asarray(y_true, float)
    y_prob = np.asarray(y_prob, float)
    N = y_true.size
    o_bar = y_true.mean()                       # overall outcome rate
    bs = brier_score_loss(y_true, y_prob)       # raw mean squared error
    bs_ref = o_bar * (1 - o_bar)                # prevalence-only null model
    bs_scaled = 1 - bs / bs_ref                 # Brier skill score

    # Bin by predicted probability for the reliability/resolution split.
    edges = np.linspace(0.0, 1.0, n_bins + 1)
    idx = np.clip(np.digitize(y_prob, edges[1:-1]), 0, n_bins - 1)
    reliability = resolution = 0.0
    for k in range(n_bins):
        m = idx == k
        n_k = m.sum()
        if n_k == 0:
            continue
        p_bar_k = y_prob[m].mean()              # mean prediction in bin
        o_bar_k = y_true[m].mean()              # observed rate in bin
        reliability += n_k * (p_bar_k - o_bar_k) ** 2
        resolution  += n_k * (o_bar_k - o_bar) ** 2
    reliability /= N
    resolution  /= N
    uncertainty = o_bar * (1 - o_bar)
    return {"brier": bs, "brier_scaled": bs_scaled,
            "reliability": reliability, "resolution": resolution,
            "uncertainty": uncertainty,
            "decomp_check": reliability - resolution + uncertainty}

# Worked example: 5000 subjects, prevalence 0.10, a moderately useful model.
rng = np.random.default_rng(1)
y = rng.binomial(1, 0.10, size=5000)
p = np.clip(0.10 + 0.25 * (y - 0.10) + rng.normal(0, 0.05, size=5000), 1e-4, 1 - 1e-4)
res = brier_decomposition(y, p)
print({k: round(v, 4) for k, v in res.items()})

library(Hmisc)

# Worked example: prevalence 0.10, a moderately useful model on 5000 subjects.
set.seed(1)
y <- rbinom(5000, 1, 0.10)
p <- pmin(pmax(0.10 + 0.25 * (y - 0.10) + rnorm(5000, 0, 0.05), 1e-4), 1 - 1e-4)

# val.prob returns Brier, calibration intercept/slope, c-statistic together.
vp <- val.prob(p = p, y = y, pl = FALSE)
brier   <- vp["Brier"]
bs_ref  <- mean(y) * (1 - mean(y))           # prevalence-only null model
cat("raw Brier       =", round(brier, 4), "\n")
cat("scaled Brier    =", round(1 - brier / bs_ref, 4), "\n")  # Brier skill score

# Time-dependent (IPCW) Brier score for a censored Cox model at a horizon.
# library(pec); library(survival)
# fit <- coxph(Surv(time, status) ~ x1 + x2, data = d, x = TRUE)
# pe  <- pec(list(Cox = fit), Surv(time, status) ~ 1, data = d,
#            times = 365, cens.model = "marginal")   # IPCW Brier at 1 year
# print(pe)

/* Per-subject squared error; phat from PROC LOGISTIC (... / outpred=scored). */
data se;
  set work.scored;
  sq_err = (phat - y)**2;
run;

/* Raw Brier score, overall outcome rate, and the prevalence-only reference. */
proc sql;
  create table brier as
  select mean(sq_err)            as brier,
         mean(y)                 as o_bar,
         calculated o_bar * (1 - calculated o_bar) as bs_ref,
         1 - calculated brier / calculated bs_ref as brier_scaled  /* skill score */
  from se;
quit;

/* Reliability / resolution via decile bins of the predicted probability. */
proc rank data=work.scored out=binned groups=10;
  var phat; ranks bin;
run;

proc sql;
  create table decomp as
  select sum(n_k * (p_bar_k - o_bar_k)**2) / sum(n_k) as reliability,
         sum(n_k * (o_bar_k - g_rate)**2)  / sum(n_k) as resolution,
         min(g_rate) * (1 - min(g_rate))             as uncertainty
  from (select bin,
               count(*)   as n_k,
               mean(phat) as p_bar_k,
               mean(y)    as o_bar_k,
               (select mean(y) from binned) as g_rate
        from binned group by bin);
quit;

proc print data=brier   noobs; run;
proc print data=decomp  noobs; run;   /* brier ~= reliability - resolution + uncertainty */

Citations

1. [1]Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, Pencina MJ, Kattan MW. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21(1):128-138.
2. [2]Brier GW. Verification of forecasts expressed in terms of probability. Monthly Weather Review. 1950;78(1):1-3.
3. [3]Murphy AH. A new vector partition of the probability score. Journal of Applied Meteorology. 1973;12(4):595-600.

1. [4]Rufibach K. Use of Brier score to assess binary predictions. Journal of Clinical Epidemiology. 2010;63(8):938-939.

1. [5]Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW. Calibration: the Achilles heel of predictive analytics. BMC Medicine. 2019;17(1):230.