JBI Critical Appraisal Checklist for Systematic Reviews and Research Syntheses

What it is

— The JBI Critical Appraisal Checklist for Systematic Reviews and Research Syntheses is an 11-item critical-appraisal (risk-of-bias) tool maintained by JBI (formerly the Joanna Briggs Institute, Adelaide) as part of the JBI suite of design-specific appraisal instruments and the JBI Manual for Evidence Synthesis. It is the instrument a reviewer applies to an already-published systematic review to judge whether that review was conducted with enough methodological rigor that its conclusions can be believed and reused. It is the appraisal layer of JBI's umbrella-review / "review of reviews" methodology: when you synthesize the findings of multiple systematic reviews, this checklist is how you formally rate the trustworthiness of each included review. It sits alongside sibling JBI checklists for RCTs, cohort, case-control, prevalence, qualitative, and economic-evaluation studies — each tuned to the biases of its design.

When to use

— Use the JBI SR checklist when the unit of appraisal is a systematic review itself, not a primary study. The canonical decision contexts are: (1) conducting an umbrella review / overview of reviews, where each candidate review must be appraised before its effect estimates are pooled or narratively synthesized; (2) building the evidence-synthesis backbone of an HTA or payer dossier, where reviewers must defend why some published reviews were weighted and others discounted; (3) peer-reviewed evidence synthesis that cites prior reviews as inputs rather than re-extracting all primary data; and (4) any comparative-effectiveness landscape where multiple overlapping reviews exist and the analyst must rank their credibility. Decision rule for which tool: appraise a systematic review with JBI SR (or its sibling AMSTAR 2); appraise the primary studies inside a review with the design-matched JBI checklist (RCT, cohort, case-control) or ROBINS-I; report your own review with PRISMA 2020 and register the protocol with PRISMA-P — those are reporting guidelines, not appraisal tools, and are not interchangeable with JBI SR.

What it requires

— The checklist enforces 11 methodological domains that map to the credible-synthesis questions a regulator or HTA reviewer will ask: (1) an explicit and appropriate review question; (2) inclusion criteria appropriate to that question; (3) a search strategy adequate to find the relevant evidence; (4) adequate sources and resources searched (databases, grey literature, languages); (5) appropriate appraisal criteria applied to the included studies; (6) critical appraisal by two or more reviewers independently; (7) methods to minimize error in data extraction; (8) appropriate methods to combine studies (meta-analysis assumptions, heterogeneity, model choice); (9) assessment of publication bias; (10) policy/practice recommendations supported by the reported data; and (11) appropriate, specific directives for new research. For real-world-data evidence synthesis these domains acquire teeth: "appropriate appraisal criteria" must mean the included observational studies were judged on design transparency, data fitness-for-use, phenotype/algorithm validation, time-zero alignment, estimand specification with intercurrent-event handling, confounding control, attrition/missing-data accounting, and quantitative sensitivity/bias analysis — not a generic "low/high quality" stamp. "Appropriate methods to combine studies" must confront the fact that pooling effect estimates across non-randomized database studies with different time-zero, confounding, and exposure definitions can manufacture spurious precision.

When NOT to use — limitations and common misapplications

— (1) It is a risk-of-bias instrument, not a reporting checklist and not a numeric quality score. Tallying "yes" answers into a sum and thresholding it ("8/11 = high quality") is a documented misuse — JBI explicitly discourages converting the items into a cutoff score; each domain must be judged and reported individually. (2) It appraises the review, not the evidence base. A methodologically flawless systematic review of biased observational studies is still a rigorous synthesis of weak evidence; passing JBI SR does not upgrade the certainty of the underlying RWE, which is the job of GRADE and of design-level appraisal of the primary studies. (3) Wrong unit of appraisal: using JBI SR to appraise a single cohort or RCT (use the design-matched JBI checklist or ROBINS-I) or, conversely, using a primary-study tool to appraise a review. (4) Checklist-as-theater: completing the form to satisfy a journal or dossier template without the two-independent-reviewer process, documented disagreements, or an audit trail defeats the instrument. (5) Substituting it for reporting compliance: a review can score well on JBI SR yet still need PRISMA 2020 for transparent reporting — the two are complementary, not redundant. (6) Completing the checklist does not make an observational synthesis causal; estimand and confounding judgments still rest on the primary-study designs.

How it maps to this catalog

— When you apply the JBI SR appraisal criteria to a synthesis of real-world-data studies, the substantive judgments are implemented by concepts in this repo. Item 5 ("appropriate appraisal criteria") for included database studies is implemented by target-trial-emulation (the reference frame for judging whether each study aligned eligibility, treatment assignment, and time zero), active-comparator-new-user (the design that controls confounding by indication and immortal-time bias you should look for), high-dimensional-propensity-score-hdps-rwe (the confounding-control adequacy you should demand), and estimands-ate-att-intercurrent-events-rwe (whether each study even specified what it was estimating). Whether the included studies validated their outcomes and exposures — a precondition for trusting any pooled estimate — is implemented by diagnosis-phenotype-algorithm-1ip-2op-time-window-rwe. Item 7 (data-extraction error) and item 8 (combining studies) for claims/EHR evidence depend on attrition-and-loss-to-follow-up-rwe (whether cohorts were comparably retained) and claims-analysis (whether the data substrate could support the question at all).

Applied note (claims/EHR/registry RWE)

In an HTA umbrella review of, say, comparative cardiovascular safety drawn from several claims- and EHR-based systematic reviews, JBI SR is the gate at the review level, but its credibility hinges on the primary-study judgments above. A review that pooled studies with incompatible time-zero definitions, unvalidated outcome phenotypes, and unaddressed Medicare Advantage claims attrition should fail item 5 and item 8 even if its search and extraction were immaculate. Record each domain's judgment with the catalog concept that justified it, so the dossier shows why a given published review was up- or down-weighted — not merely that a checklist was filled in.