Brier Score

In plain language

The Brier score measures how accurate a model's predicted probabilities are for a yes/no outcome — it is the average of the squared differences between each patient's predicted probability and their actual result (0 for no event, 1 for event), so a score of 0 means every prediction was exactly right. It captures two things at once: whether the model's probabilities are in the right ballpark for similar patients (that property is called calibration) and whether the model can actually separate high-risk patients from low-risk ones. A Brier score of 0 is perfect; a score equal to the outcome rate multiplied by one minus the outcome rate is what you would get from a model that ignores all patient information and just guesses the overall event rate for everyone.

The Brier score (BS) is the mean squared error of a probabilistic prediction. For a binary outcome y in {0,1} and a predicted probability p in [0,1] over N subjects, BS = (1/N) sum_i (p_i - y_i)^2. It ranges from 0 (perfect: every prediction equals the realized outcome) to 1 (worst: confident and always wrong); a non-informative model that always predicts the marginal prevalence q achieves BS = q(1-q). Originating in weather forecast verification (Brier, 1950), it is now the standard overall performance measure for clinical and RWE prediction models because, unlike discrimination metrics (AUC, the c-statistic) that depend only on the ranking of predictions, the Brier score penalizes predictions that are mis-scaled — a model can rank perfectly (AUC 1.0) yet be badly calibrated and earn a poor Brier score.

Core conceptual distinction — the decomposition

The Brier score's analytic value is that it splits into interpretable parts. Grouping predictions into K bins of (approximately) constant predicted probability, Murphy's (1973) decomposition is BS = reliability - resolution + uncertainty, where reliability = (1/N) sum_k n_k (p_bar_k - o_bar_k)^2 (the squared gap between mean predicted probability p_bar_k and observed outcome rate o_bar_k in each bin — lower is better, this is the calibration term), resolution = (1/N) sum_k n_k (o_bar_k - o_bar)^2 (how far each bin's outcome rate departs from the overall rate o_bar — higher is better, this is discrimination/sharpness), and uncertainty = o_bar(1 - o_bar) (the irreducible variance of the outcome, fixed by the data). A second, equivalent split is calibration + refinement, where refinement = uncertainty - resolution. The lesson: a low Brier score can be achieved either by good calibration or* by high resolution, so the raw score conflates two distinct virtues — the decomposition is what lets you attribute a model's Brier performance to its calibration versus its sharpness.

Scaling

Because the achievable Brier score depends on outcome prevalence (a rare outcome makes a low absolute BS easy), the raw score is not comparable across datasets with different prevalence. The scaled Brier score (a.k.a. Brier skill score), BS_scaled = 1 - BS / BS_ref where BS_ref = q*(1-q) is the score of the prevalence-only model, rescales to a 0-to-1 "proportion of maximum achievable" metric (1 = perfect, 0 = no better than predicting the marginal rate, negative = worse than the null model). Report the scaled Brier score for cross-study comparison; report the raw score with its decomposition within a study.

Pros, cons, and trade-offs

- vs the c-statistic / AUC (`roc-auc-discrimination-rwe`): The Brier score is a proper scoring rule — it is minimized in expectation only by the true probabilities, so it cannot be gamed by mis-scaling and it rewards honest probability estimates. AUC measures only discrimination (ranking) and is invariant to any monotone re-scaling of predictions, so it is silent about calibration. Prefer the Brier score (with its decomposition) when the predicted probability itself will be used (for thresholds, expected cost, decision curves); prefer AUC when only the rank order matters and you want a prevalence-invariant discrimination summary. Report both — they answer different questions. - vs a calibration plot / Hosmer-Lemeshow: A calibration plot (and the calibration slope/intercept) shows where and how miscalibration occurs across the probability range, information the single Brier number compresses away; the reliability term of the decomposition is the scalar summary of exactly that plot. Prefer the calibration plot to diagnose the shape of miscalibration; use the Brier score / reliability term as the one-number summary and for model selection. The Hosmer-Lemeshow test is a discredited goodness-of-fit hypothesis test (power depends on N, arbitrary binning); the reliability decomposition is the preferred quantitative alternative. - vs the logarithmic score (log loss): Both are proper scoring rules. Log loss penalizes confident errors far more harshly (it is unbounded as p -> 0 for a positive case) and is the right loss when extreme over-confidence must be punished; the Brier score is bounded, more robust to a handful of catastrophic predictions, and decomposes cleanly into calibration and refinement. Prefer the Brier score for interpretable decomposition and robustness; prefer log loss when calibrated tail probabilities are critical.

When to use

Reporting the overall performance of any RWE prediction or prognostic model that outputs probabilities (risk of hospitalization, mortality, treatment response) alongside discrimination (AUC) and a calibration plot, as the modern framework (Steyerberg et al., 2010) recommends; selecting among candidate models when the predicted probability will feed a downstream decision (risk-based eligibility, expected-cost calculations, decision-curve analysis); tracking performance over time or across sites with the scaled Brier score; quantifying calibration with the reliability term rather than the Hosmer-Lemeshow test.

When NOT to use — and when it is actively misleading or dangerous

- Comparing raw Brier scores across datasets with different prevalence. A rare outcome makes a low raw BS trivially attainable (the null model already scores q(1-q) ~= q for small q); comparing raw scores across populations confounds model quality with base rate. Use the scaled Brier score and report prevalence. - As a substitute for the calibration plot when the question is where* the model fails. The single Brier number cannot tell you whether the model is over-confident at high risk and under-confident at low risk; treating a good Brier score as proof of calibration across the whole range is misleading. Inspect the plot or the bin-wise reliability. - For a model that will only ever be used as a rank-ordering tool. If only the ordering of patients matters (e.g., waitlist prioritization) and no probability is acted on numerically, the calibration that the Brier score rewards is irrelevant; AUC or a rank-based metric is the appropriate target and optimizing Brier may add nothing. - On predictions evaluated on the same data used to fit them, without correction. An apparent (resubstitution) Brier score is optimistically biased; report a Brier score from cross-validation, bootstrap optimism correction, or an external validation sample, exactly as for any other performance measure.

Data-source operational depth

- Claims (FFS): The Brier score requires a binary outcome label that is itself a computable phenotype, so its honesty inherits the phenotype's misclassification — a noisy outcome label inflates the irreducible-uncertainty term and caps the achievable score. Build predictions and outcomes on FFS-observable person-time only; Medicare Advantage enrollees generate no claims, so an MA-only span yields a fabricated "no event" label that biases both the outcome rate o_bar and the reliability term. For time-to-event outcomes use the time-dependent (inverse-probability-of-censoring-weighted) Brier score so administrative censoring does not masquerade as event-free survival. - EHR: Richer features usually improve resolution (sharper, more separated predictions), but encounter-driven ascertainment makes an out-of-system event an unobserved "negative," biasing the outcome label and therefore the calibration term; require demonstrable in-system activity before labeling a subject event-free, and treat informative loss to follow-up with censoring-weighted Brier scores. - Registry / linked: Adjudicated registry outcomes give the cleanest binary labels, tightening every term of the decomposition; linked claims-EHR-vital-records additionally supplies a death index so a competing terminal event is not miscoded as event-free. Linkage selects the linkable subset, so check that the validation sample's prevalence matches the deployment population before trusting a scaled Brier score.

Worked example

A logistic model predicts 1-year mortality in a linked claims-registry cohort of N = 5,000 with observed prevalence o_bar = 0.10 (so uncertainty = 0.10*0.90 = 0.090, and the prevalence-only null model scores BS_ref = 0.090). Suppose the fitted model achieves raw BS = 0.072. Then the scaled Brier score = 1 - 0.072/0.090 = 0.20 — the model explains 20% of the maximum achievable improvement over predicting the marginal rate, a modest but real gain. Decomposing across deciles of predicted risk: if reliability = 0.004 (small squared calibration gaps) and resolution = 0.022, then BS = reliability - resolution + uncertainty = 0.004 - 0.022 + 0.090 = 0.072, confirming the arithmetic and attributing most of the improvement to resolution (the model separates risk groups) with good calibration (low reliability term). The same model might post AUC = 0.74; reporting AUC alone would have hidden the calibration story, and reporting raw BS alone would have hidden that 0.072 is only modestly better than the 0.090 null. Report all three — scaled Brier 0.20, the reliability/resolution split, and a calibration plot — for a complete performance picture.

Interpreting the output

In the large-scale worked example, the model achieves Brier score = 0.072, scaled Brier = 0.20, reliability = 0.004, and resolution = 0.022, against a null Brier of 0.090.

(1) Formal interpretation. The Brier score is the mean squared error between predicted probabilities and binary outcomes. A score of 0.072 is lower (better) than the null score of 0.090, which a model predicting the marginal event rate for everyone would achieve. The scaled Brier of 0.20 expresses this improvement as a proportion of the maximum possible reduction, with 0 = no skill and 1 = perfect calibration and discrimination. The Murphy decomposition separates BS into reliability (average squared calibration gap between predicted and observed — lower is better) and resolution (variance of predicted probabilities — higher means the model spreads risk groups apart). A reliability term of 0.004 indicates predictions are close to observed rates on average; a resolution of 0.022 indicates meaningful risk-group separation. When calibration slope < 1, predictions are too extreme and shrinkage or recalibration is needed; a slope > 1 indicates predictions are too conservative.

(2) Practical interpretation. A scaled Brier of 0.20 is a modest but real gain over the null — adequate for population-level risk stratification but not sufficient for individual-level clinical decisions that require tighter calibration. The low reliability (0.004) confirms predictions track observed rates well, which matters whenever predicted probabilities drive enrollment thresholds or resource-allocation rules. Always pair the Brier score with AUC to separate calibration from discrimination: the same model here posts AUC = 0.74, showing acceptable rank ordering; the Brier adds the calibration dimension that AUC cannot see.

Worked example

Scenario

A research team builds a model that predicts whether a patient will be hospitalized within 90 days. They test it on five patients from a validation dataset. For each patient the model outputs a predicted probability of hospitalization, and at the end of 90 days the team records whether hospitalization actually occurred (1 = yes, 0 = no). The team wants to compute the Brier score to summarize how well the model's probabilities matched reality, and to discuss what the score tells them about calibration.

Dataset

Validation rows: one row per patient with the model's predicted probability and the observed 90-day hospitalization outcome.

Steps

• For each patient, subtract the observed outcome from the predicted probability, then square the result. This gives the squared error for that patient.

• P01: predicted 0.9, outcome 1. Difference = 0.9 - 1 = -0.1. Squared error = (-0.1)^2 = 0.01.

• P02: predicted 0.8, outcome 1. Difference = 0.8 - 1 = -0.2. Squared error = (-0.2)^2 = 0.04.

• P03: predicted 0.3, outcome 0. Difference = 0.3 - 0 = 0.3. Squared error = (0.3)^2 = 0.09.

• P04: predicted 0.6, outcome 1. Difference = 0.6 - 1 = -0.4. Squared error = (-0.4)^2 = 0.16.

• P05: predicted 0.2, outcome 0. Difference = 0.2 - 0 = 0.2. Squared error = (0.2)^2 = 0.04.

• Add up all five squared errors: 0.01 + 0.04 + 0.09 + 0.16 + 0.04 = 0.34.

• Divide by the number of patients to get the mean: 0.34 / 5 = 0.068. That is the Brier score.

• Now check calibration in plain terms. Three of the five patients had the event (outcome rate = 3/5 = 0.60). The model assigned probabilities of 0.9, 0.8, 0.3, 0.6, and 0.2; the average predicted probability is (0.9+0.8+0.3+0.6+0.2)/5 = 0.56, close to the 0.60 observed rate — a sign of reasonable overall calibration. A null model that predicted 0.60 for everyone would score (0.60-1)^2 + (0.60-1)^2 + (0.60-0)^2 + (0.60-1)^2 + (0.60-0)^2 = 0.16+0.16+0.36+0.16+0.36 = 1.20; divided by 5 that is 0.240. The actual model (0.068) is well below that null benchmark, meaning it adds real information beyond just knowing the event rate.

Result

Brier score = (0.01 + 0.04 + 0.09 + 0.16 + 0.04) / 5 = 0.34 / 5 = 0.068. The null model (always predict 0.60) would score 0.240, so the model substantially outperforms a guess based on the overall event rate alone. The model is reasonably well-calibrated: its average predicted probability (0.56) is close to the observed event rate (0.60), and the largest single squared error (0.16, from P04 who was hospitalized but only scored 0.6) flags the one patient where the model was most under-confident.

Runnable example

import numpy as np
from sklearn.metrics import brier_score_loss

def brier_decomposition(y_true, y_prob, n_bins=10):
    """Raw Brier score, scaled Brier (skill) score, and Murphy's decomposition."""
    y_true = np.asarray(y_true, float)
    y_prob = np.asarray(y_prob, float)
    N = y_true.size
    o_bar = y_true.mean()                       # overall outcome rate
    bs = brier_score_loss(y_true, y_prob)       # raw mean squared error
    bs_ref = o_bar * (1 - o_bar)                # prevalence-only null model
    bs_scaled = 1 - bs / bs_ref                 # Brier skill score

    # Bin by predicted probability for the reliability/resolution split.
    edges = np.linspace(0.0, 1.0, n_bins + 1)
    idx = np.clip(np.digitize(y_prob, edges[1:-1]), 0, n_bins - 1)
    reliability = resolution = 0.0
    for k in range(n_bins):
        m = idx == k
        n_k = m.sum()
        if n_k == 0:
            continue
        p_bar_k = y_prob[m].mean()              # mean prediction in bin
        o_bar_k = y_true[m].mean()              # observed rate in bin
        reliability += n_k * (p_bar_k - o_bar_k) ** 2
        resolution  += n_k * (o_bar_k - o_bar) ** 2
    reliability /= N
    resolution  /= N
    uncertainty = o_bar * (1 - o_bar)
    return {"brier": bs, "brier_scaled": bs_scaled,
            "reliability": reliability, "resolution": resolution,
            "uncertainty": uncertainty,
            "decomp_check": reliability - resolution + uncertainty}

# Worked example: 5000 subjects, prevalence 0.10, a moderately useful model.
rng = np.random.default_rng(1)
y = rng.binomial(1, 0.10, size=5000)
p = np.clip(0.10 + 0.25 * (y - 0.10) + rng.normal(0, 0.05, size=5000), 1e-4, 1 - 1e-4)
res = brier_decomposition(y, p)
print({k: round(v, 4) for k, v in res.items()})

library(Hmisc)

# Worked example: prevalence 0.10, a moderately useful model on 5000 subjects.
set.seed(1)
y <- rbinom(5000, 1, 0.10)
p <- pmin(pmax(0.10 + 0.25 * (y - 0.10) + rnorm(5000, 0, 0.05), 1e-4), 1 - 1e-4)

# val.prob returns Brier, calibration intercept/slope, c-statistic together.
vp <- val.prob(p = p, y = y, pl = FALSE)
brier   <- vp["Brier"]
bs_ref  <- mean(y) * (1 - mean(y))           # prevalence-only null model
cat("raw Brier       =", round(brier, 4), "\n")
cat("scaled Brier    =", round(1 - brier / bs_ref, 4), "\n")  # Brier skill score

# Time-dependent (IPCW) Brier score for a censored Cox model at a horizon.
# library(pec); library(survival)
# fit <- coxph(Surv(time, status) ~ x1 + x2, data = d, x = TRUE)
# pe  <- pec(list(Cox = fit), Surv(time, status) ~ 1, data = d,
#            times = 365, cens.model = "marginal")   # IPCW Brier at 1 year
# print(pe)