AMSTAR 2

What it is

— AMSTAR 2 ("A MeaSurement Tool to Assess systematic Reviews," version 2) is a critical-appraisal instrument that judges the methodological quality of a completed systematic review. It was published by Shea and colleagues in the BMJ in 2017 as a major revision of the original 2007 AMSTAR, broadening the tool so it can appraise reviews that include randomized trials, non-randomized studies of interventions (NRSI), or both. AMSTAR 2 has 16 items; seven are designated critical domains because flaws in them undermine the validity of a review's conclusions (Item 2: a pre-registered protocol; Item 4: a comprehensive, reproducible literature search; Item 7: a justified list of excluded studies; Item 9: an appropriate risk-of-bias assessment of the included studies; Item 11: appropriate meta-analytic methods; Item 13: accounting for risk of bias when interpreting results; Item 15: an adequate investigation of publication/small-study bias). The remaining nine items are non-critical. AMSTAR 2 is maintained by its developer group (Bruyère / Ottawa) at amstar.ca and is widely adopted by HTA agencies and methods journals. Critically, it does not produce a numeric summary score; the developers deliberately replaced the original AMSTAR sum-score with a structured rating of overall confidence in the results of the review.

When to use

— Apply AMSTAR 2 when you must appraise an existing systematic review (your own, or one you are citing/synthesizing) of healthcare interventions, and you need a defensible, transparent statement of how much confidence its findings warrant. This is the appraisal layer for evidence syntheses in HTA/payer dossiers (NICE, CADTH, and similar bodies routinely expect AMSTAR 2 for submitted SRs), in peer-reviewed overviews and umbrella reviews, and in regulatory evidence packages that lean on published syntheses. Decision rule for choosing AMSTAR 2 vs siblings: use AMSTAR 2 to appraise the quality of the review itself; use ROBIS if you specifically want a risk-of-bias (not quality) judgment of the review's process; use PRISMA 2020 / PRISMA-P to report or register a review (those are reporting checklists, not appraisal tools); use GRADE to grade the certainty of the body of evidence for each outcome (a different question from the review's methodological quality). AMSTAR 2 fits SRs of interventions — including reviews that synthesize real-world evidence from claims, EHR, or registry sources — but is not intended for reviews of diagnostic accuracy, prognosis, or purely qualitative/scoping reviews.

What it requires

— AMSTAR 2 enforces, item by item: an a priori protocol with pre-specified PICO and analysis plan (Item 2); explicit study-design eligibility justification (Item 3); a comprehensive search of at least two databases plus supplementary sources, with strategy and date reported (Item 4); duplicate study selection and data extraction (Items 5–6); a complete list of excluded full-text studies with reasons (Item 7); adequate description of included studies in PICO detail (Item 8); a satisfactory risk-of-bias assessment of the included primary studies using an appropriate tool (Item 9) and reporting of funding sources of those studies (Item 10); appropriate methods for statistical combination, including heterogeneity assessment (Item 11) and assessment of the impact of risk of bias on the pooled estimate (Item 12); accounting for individual-study risk of bias when discussing results (Item 13); explanation and investigation of heterogeneity (Item 14); an adequate investigation of publication/small-study bias when meta-analysis was performed (Item 15); and disclosure of the reviewers' conflicts of interest (Item 16). The overall rating follows an explicit rule based on weaknesses in the seven critical domains (Items 2, 4, 7, 9, 11, 13, 15): no critical weakness and no more than one non-critical weakness = high confidence; more than one non-critical weakness (but no critical flaw) = moderate; exactly one critical flaw (with or without non-critical weaknesses) = low; more than one critical flaw (with or without non-critical weaknesses) = critically low. The count of non-critical weaknesses never pushes a review below moderate — only critical-domain flaws drive a Low or Critically-low rating. For an SR that pools real-world data, Item 9 is the hinge — it requires that the included observational studies were appraised with a fit-for- purpose risk-of-bias tool (e.g., ROBINS-I), which is where design transparency, time-zero alignment, confounding control, and attrition would actually be scrutinized at the primary-study level.

When NOT to use — limitations and common misapplications

— AMSTAR 2 is an appraisal tool for systematic reviews, and most failures come from using it as something it is not. (1) It is not a reporting checklist: completing AMSTAR 2 does not satisfy PRISMA 2020, and a review can be well reported yet score critically low, or vice versa. (2) It is not a risk-of-bias instrument for primary studies — do not apply AMSTAR 2 items to an individual RCT or cohort study; that is the job of RoB 2 or ROBINS-I, and AMSTAR 2 invokes those tools at Item 9 rather than replacing them. (3) It is not GRADE: AMSTAR 2 judges how the review was conducted, not the certainty of the evidence for a given outcome; the two are complementary, not interchangeable. (4) Do not compute a numeric AMSTAR 2 score — summing "Yes" answers resurrects the discredited original-AMSTAR sum-score behavior and obscures that a single critical-domain failure should collapse confidence; the developers and subsequent commentary (e.g., Lorenz et al. 2020) emphasize the structured overall rating, not an arithmetic total. (5) It is the wrong tool for the design if the object of appraisal is a scoping review, narrative review, or a review of diagnostic-test accuracy. (6) Checklist-as-theater: marking items "Yes" without verifying the protocol, the excluded-studies list, or the risk-of-bias assessment of included RWE studies produces a hollow rating that senior reviewers will see through. (7) Appraising a review of observational evidence does not make that evidence causal — a high AMSTAR 2 rating certifies the review's methods, not the internal validity of the underlying real-world studies.

How it maps to this catalog

— AMSTAR 2 lives at the synthesis layer, one level above the primary-study methods this catalog documents, so the mapping is by delegation rather than item-for-item. Item 9 requires that the systematic review appraised its included studies with an appropriate risk-of-bias tool; when those included studies are real-world observational analyses, that tool (typically ROBINS-I) is what then interrogates exactly the methods catalogued here. An AMSTAR 2 appraiser of an RWE-inclusive review should therefore expect the underlying primary studies to have implemented: a defensible design and time-zero alignment via `active-comparator-new-user` and `target-trial-emulation`; confounding control via `high-dimensional-propensity-score-hdps-rwe`; a clearly specified target estimand and handling of intercurrent events via `estimands-ate-att-intercurrent-events-rwe`; validated outcome/exposure definitions via `diagnosis-phenotype-algorithm-1ip-2op-time-window-rwe`; and transparent handling of follow-up loss via `attrition-and-loss-to-follow-up-rwe`. Item 4's data-source and search transparency, and the data-fitness considerations that determine whether claims/EHR/registry sources can answer the review's question, connect to `claims-analysis`. Applied note for claims/EHR/registry RWE: when you appraise a meta-analysis that pools, say, claims-based cohort studies, a "Yes" on Item 9 is only credible if the review actually examined whether the included studies used incident-user, active- comparator designs with valid phenotype algorithms and pre-specified estimands — i.e., the review's risk-of-bias step reached down into the concepts above. If it did not, Item 9 is a critical weakness and overall confidence drops to low or critically low regardless of how polished the review's prose is.