PROBAST (Prediction model Risk Of Bias ASsessment Tool)

What it is

— PROBAST (Prediction model Risk Of Bias ASsessment Tool) is a domain-structured, signalling-question instrument for assessing the risk of bias and applicability of studies that develop, validate, or update a multivariable clinical prediction model (diagnostic or prognostic). It was published by Wolff et al. in 2019 in Annals of Internal Medicine alongside a companion Explanation & Elaboration paper (Moons et al. 2019, same issue). PROBAST is the appraisal companion to the TRIPOD reporting guideline: TRIPOD tells authors what to report; PROBAST tells reviewers how to judge whether a prediction model study is at low or high risk of bias. Both are maintained by the EQUATOR Network. PROBAST has four domains: D1 Participants (source data and case-mix adequate?), D2 Predictors (well-defined, not using post-index information?), D3 Outcome (valid definition, appropriately assessed, blinded to predictors?), and D4 Analysis (sample size, missing data handling, predictor selection, overfitting control, calibration reported?). Within each domain, signalling questions (answered low/high/unclear) drive an overall domain risk-of-bias rating, and the four domain ratings combine into an overall risk-of-bias judgment. A separate applicability assessment — across the same three phases as ROBINS-I — captures whether the study's population, predictors, or outcome definition are appropriate for the review question even if the study is otherwise at low risk of bias.

When to use

— Apply PROBAST whenever you are appraising a prediction model study rather than reporting one: in a systematic review or meta-analysis of prediction models (the primary design context for which PROBAST was developed); in a guideline panel or HTA subgroup weighting model- based evidence; in a regulatory or payer submission where a risk-stratification or enrichment model must be credibly evaluated; or in an internal evidence-quality gate before a model is cited or deployed. Decision rule: if the study's deliverable is a multivariable prediction model and the question is "is this model at risk of bias?", use PROBAST; if the question is "is this model reported completely?", use TRIPOD. If the study is an observational comparative-effectiveness study (not a prediction model), use ROBINS-I or GRACE instead. PROBAST is relevant whether the model was built on claims, EHR, registry, or prospectively collected data; it adapts because its signalling questions probe the methodological fundamentals rather than a specific data type.

What it requires (checklist domains)

— PROBAST enforces appraisal across four domains with ~20 signalling questions in total. Domain 1 — Participants: was the data source and sampling frame appropriate to the prediction task? Were inclusion/exclusion criteria pre-specified and applied consistently? Is there risk of selection bias (e.g., case-control designs applied to inherently cohort-structured questions)? Domain 2 — Predictors: are all predictors well-defined and measured at the correct time relative to the prediction horizon? Is there risk that predictor measurement was influenced by knowledge of outcome (look-ahead bias)? Were candidate predictors not selected post-hoc based on univariable screening in ways that inflate the final model? Domain 3 — Outcome: is the outcome defined clearly and measured validly? Was outcome assessment blinded to predictor values, or is there differential measurement? Was follow-up time adequate and consistent with the prediction horizon? Domain 4 — Analysis: was sample size adequate (events-per-variable or events-per-parameter)? Was missing data handled appropriately (not complete-case only)? Were statistical methods for model building (including overfitting control via regularisation, bootstrap, or cross-validation) appropriate? Was calibration assessed in addition to discrimination? Was model performance assessed in an independent validation sample rather than just in the training data? Each domain is rated low / high / unclear risk of bias; overall bias is rated high if any domain is high (a conservative rule that reflects how one flawed domain can invalidate a model's usefulness).

When NOT to use — limitations and common misapplications

— PROBAST is an appraisal tool, not a reporting checklist; do not hand it to authors as a writing guide (use TRIPOD for that). It is also not a numeric quality score: tallying signalling-question "lows" into a sum and using that as a meta-analytic weight is a methodological error — PROBAST supports structured domain-level judgments, not arithmetic. Common failure modes: (1) Wrong study type — applying PROBAST to a comparative- effectiveness observational study (use ROBINS-I or GRACE) or a single diagnostic test (use QUADAS-2) rather than a multivariable prediction model. (2) Confusing PROBAST and TRIPOD — a study can be fully TRIPOD-compliant (reported everything) and still be at high PROBAST risk of bias (e.g., if the predictor selection was data-driven and overfit). (3) Ignoring the calibration item — Domain 4 requires that calibration was assessed; reviewers often accept an AUC/C-statistic alone and rate D4 low-risk inappropriately. (4) Over-applying the "unclear = low" shortcut — unclear risk should not default to low risk when the information was not reported; absent calibration reporting or missing sample-size justification should drive uncertain or high ratings. (5) Applicability vs bias confusion — a model may be at low risk of bias but highly inapplicable to the target population (different healthcare system, different case-mix, older predictor definitions); PROBAST's applicability domain is meant to capture this and should not be collapsed into the bias assessment. (6) Checklist theater — completing PROBAST forms without engaging the underlying methodological questions produces misleading reassurance; a high Domain 4 rating because calibration was never assessed is a substantive finding, not a technicality.

How it maps to this catalog

— PROBAST's four domains map directly to the methodological concepts that implement what PROBAST only judges. Domain 1 (Participants / data adequacy) is implemented by fit-for-purpose-data-assessment-rwe and claims-analysis (data-source fitness and case-mix adequacy). Domain 2 (Predictors — no look-ahead, pre-specified) is implemented by time-zero-index-date-alignment-rwe (ensuring predictors are measured before the prediction horizon) and diagnosis-phenotype-algorithm-1ip-2op-time-window-rwe (how predictor variables are defined in administrative data). Domain 3 (Outcome validity) is implemented by claims-outcome-algorithm-ppv-sensitivity-rwe and ehr-phenotyping-algorithms-rwe (validated outcome algorithms with PPV/sensitivity). Domain 4 (Analysis — sample size, overfitting, calibration) is implemented by sample-size-power-precision-rwe (events-per-variable justification), multiple-imputation-longitudinal-rwe (appropriate missing-data handling), and prediction-model-validation-recalibration-rwe (internal/external validation, calibration, recalibration). The discrimination and calibration reporting that PROBAST requires is visualised through visualizations-pharmacoepidemiology-rwe (calibration plots, ROC/AUC, decision curves). The reporting obligation PROBAST assumes is fulfilled with TRIPOD (the companion guideline in this catalog). Whenever a systematic review of prediction models is appraised with PROBAST, the evidence synthesis itself should follow PRISMA-2020 or PRISMA-DTA and the overall certainty judgment may incorporate GRADE principles.