Systematic Review

In plain language

A systematic review answers one focused question by finding every relevant study, not just the convenient few, and combining what they found. You write down the rules in advance (which studies count, how you will search, how you will judge their quality), register that plan publicly, then follow it so another team could repeat your work and get the same set of studies. Some reviews end by statistically pooling the numbers into one combined estimate, but pooling is optional; the review itself is the disciplined search-and-appraisal process. Its main weakness is honest: if the underlying studies are flawed, a tidy review cannot turn them into good evidence.

A systematic review (SR) is a secondary research design that treats the body of primary evidence on a focused question as its unit of analysis. Unlike a narrative review, an SR is governed by a pre-registered protocol (PROSPERO, or a fixed protocol for a regulatory PASS) that fixes the PICOTS eligibility frame, the search strategy, the screening and data-extraction rules, the risk-of-bias instrument, and the synthesis plan before results are seen. Quantitative pooling (meta-analysis) is an optional downstream step, not the SR itself: a methodologically complete SR may legitimately end in a structured narrative or tabular synthesis when pooling is inappropriate.

Core conceptual distinction

The SR is defined by process discipline, not by whether numbers are combined. Three separable design choices do the work. (1) Systematic vs narrative: an exhaustive, documented, reproducible search and dual independent screening replace the convenience sampling of literature that makes narrative reviews selection-biased. (2) Protocol-first vs data-driven: locking eligibility, outcomes, and analysis in a registered protocol prevents the outcome-switching and post-hoc subgroup mining that inflate false positives. (3) Synthesis vs pooling: synthesis is the integration step; meta-analysis is one synthesis tool, valid only when studies are clinically and methodologically similar enough that a common estimand exists. The SR's estimand is therefore a property of the evidence base (e.g., "the comparative effect of A vs B across all eligible RWE and trial studies"), and its central threat is not confounding within a study but selection and reporting bias across studies — missing trials, missing outcomes, and language/publication filtering.

Pros, cons, and trade-offs

(specific and comparative). - vs a narrative/expert review: the SR's reproducible search and dual screening remove cherry-picking and let another team regenerate the included set. Cost: months of effort, and the rigor is wasted if the underlying studies are themselves biased — an SR cannot manufacture evidence quality it does not have ("garbage in, garbage out"). Prefer the SR for any decision-grade or guideline question; reserve narrative reviews for scoping or framing. - vs a scoping review: an SR answers a narrow, pre-specified effect/association question and appraises risk of bias; a scoping review maps a literature's breadth and gaps without effect estimation or formal appraisal. Prefer the SR when the question is a focused PICOTS contrast; prefer scoping when the question is "what evidence exists?". - vs a single large pragmatic trial or a single well-powered RWE study: the SR aggregates power and improves external validity by spanning settings, but inherits every primary study's design flaws and adds across-study heterogeneity and ecological/aggregation bias. Prefer a single high-quality study when one exists that directly answers the question in the target population; prefer the SR when no single study is decisive or when consistency across settings is itself the question. - vs jumping straight to meta-analysis: pooling without the SR scaffolding (protocol, exhaustive search, RoB) is just a weighted average of a biased sample of studies. The SR is the prerequisite that makes a pooled number interpretable.

When to use

A focused, answerable PICOTS question; a decision that requires the totality of evidence (HTA submissions, clinical guidelines, payer dossiers, regulatory benefit-risk); a literature large or contested enough that a defensible, reproducible selection process matters; or as the evidence-generation front end before an indirect treatment comparison or network meta-analysis. Always register the protocol (PROSPERO) and follow PRISMA 2020 reporting.

When NOT to use — and when it is actively misleading or dangerous

- The question is exploratory or the field is immature. With three heterogeneous studies, a "systematic review with meta-analysis" lends spurious authority to a pooled estimate dominated by one study; a scoping review or a single well-conducted study is more honest. - The included evidence is uniformly high risk of bias. Synthesizing biased studies produces a precise but wrong answer — narrow confidence intervals around a biased pooled effect are more dangerous than no estimate, because precision is read as certainty. GRADE-rate down for risk of bias and say so plainly. - Clinical/methodological diversity is too large for a common estimand. Pooling across incomparable populations, comparators, or outcome definitions (the classic "mixing apples and oranges") yields an average that describes no real patient. Default to structured narrative synthesis; do not force a forest plot. - The decision needs patient-level effect modification. Aggregate-data SRs cannot recover within-study interactions; an individual-patient-data meta-analysis or a single richly covariate'd RWE study is required. - Time-sensitive signal detection. A 12-month SR is the wrong instrument for an emerging safety signal that needs a rapid analysis of a live data network.

Data-source operational depth

The SR's "data" are the included primary studies, but in RWE/HEOR those studies are themselves built on claims, EHR, registry, or linked data, and an SR is only as trustworthy as its handling of how those substrates bias the primary estimates. Treat data-source characteristics as extraction fields and appraisal criteria, not afterthoughts. - Claims-based primary studies: extract and appraise the washout/lookback length, the continuous-enrollment requirement, exposure operationalization (NDC + `fill_date` + `days_supply`), and outcome algorithm (e.g., 1-inpatient-or-2-outpatient). Failure mode: pooling studies that silently mixed Medicare Advantage and fee-for-service person-time — MA-only enrollees lack FFS claims, so their "no prior fill" washout is missingness, and studies that did not exclude MA-only time carry differential exposure/outcome ascertainment that surfaces as unexplained heterogeneity. Workaround: code an "FFS-only vs mixed" extraction flag and pre-specify it as a subgroup or meta-regression covariate. - EHR-based primary studies: appraise whether exposure is the order vs the dispensing, and whether loss to follow-up (patients leaving the system) was treated as informative. Failure mode: out-of-network care makes outcomes differentially under-captured; pooling EHR and claims studies without flagging capture completeness conflates true effect differences with ascertainment differences. - Registry-based primary studies: strong for adjudicated outcomes and severity but weak for complete exposure; appraise linkage to fills and to a death index. Failure mode: registries with voluntary enrollment carry selection that no SR-level adjustment can remove. - Linked claims–EHR–vital-records studies: the strongest substrate but the smallest, linkable subset; appraise linkage selection. Differential competing risks by exposure in elderly claims populations (e.g., one drug skewed to frailer patients who die before the non-fatal outcome) and immortal time in procedure studies (follow-up started at diagnosis rather than at the procedure) are the two within-study biases an RWE SR most often must catch in appraisal and, where present, exclude or down-weight rather than blindly pool.

Worked example (claims-style logic in the included studies)

Question (PICOTS): in adults with type 2 diabetes (P), does a second-generation sulfonylurea (I) vs a DPP-4 inhibitor (C) increase incident heart failure (O) over ≥6 months (T) in routine-care administrative data (S)? Protocol registered on PROSPERO; PRISMA 2020 followed. (1) Search MEDLINE/Embase/Web of Science plus the conference and regulatory grey literature; dedupe; dual independent title/abstract and full-text screening with a third-reviewer tiebreak. (2) Eligibility encodes the RWE design quality directly: include only active-comparator new-user studies that required ≥365-day continuous enrollment, a drug-free washout (no sulfonylurea or DPP-4 fill in the lookback), time-zero at the first qualifying fill, and an HF outcome algorithm with reported PPV. (3) Extraction (dual) captures: adjusted hazard ratio and 95% CI, the log-HR and its SE (SE = (ln(upper) − ln(lower)) / (2 × 1.96)), the data source, an `ffs_only` flag (1 if MA-only person-time was excluded, else 0), washout days, and `days_supply`-based grace period for the as-treated window. (4) Appraise each study with ROBIS/AMSTAR-2-aligned domains plus RWE-specific items (immortal time, competing risks, time-zero alignment). (5) Synthesis: if clinical/methodological diversity is acceptable, pool with a DerSimonian–Laird (or REML) random-effects inverse-variance model, report I² and the prediction interval, and run the pre-specified `ffs_only` subgroup; if diversity is too large, present a structured narrative with a harvest plot and do not pool. (6) GRADE the certainty of the body of evidence (start "low" for observational, move up/down for magnitude, dose-response, consistency, and residual confounding) and report a transparent PRISMA flow accounting for every excluded record.

Worked example

Scenario

You want to answer one focused question: in adults with type 2 diabetes, does a particular older diabetes drug raise the risk of heart failure compared with a newer alternative? Before searching, you register a protocol that states the question, which studies qualify, and how you will appraise them. You then run the search and walk every record down the funnel, throwing studies out only for reasons you wrote down in advance. This table is that funnel: each row is a stage and the count of records still in play, and the drops between stages must add up exactly.

Dataset

The PRISMA-style record count an analyst tracks from the first database search to the studies actually synthesized. Each stage subtracts a documented number of records.

Steps

• Start with everything the searches returned: 1,240 records pulled from multiple databases plus conference and regulatory grey literature.

• The same study often appears in more than one database, so remove the 240 duplicate records first: 1,240 minus 240 leaves 1,000 distinct records to screen.

• Two reviewers independently read each title and abstract against the pre-registered eligibility rules; clearly irrelevant ones are dropped. Here 870 are excluded, leaving 130 worth reading in full.

• Both reviewers then read the full text of those 130 and exclude 121 more, each for a documented reason (wrong population, wrong comparator, no usable effect estimate), which leaves 9 studies.

• Each of the 9 surviving studies gets a risk-of-bias appraisal, so a study that, for example, started follow-up at the wrong moment is flagged rather than trusted blindly.

• Because the 9 studies ask the same focused question in similar ways, the review can pool them in a meta-analysis; a scoping review would instead stop here, having mapped what exists without judging quality or combining results.

Result

1,240 identified minus 240 duplicates = 1,000 screened; 1,000 minus the 870 + 121 = 991 excluded with documented reasons = 9 studies included in the synthesis. Every record is accounted for, so the funnel reconciles.

Runnable example

import numpy as np
import pandas as pd

def prisma_balance(flow: dict) -> dict:
    # Reconciles the PRISMA 2020 flow: every identified record is screened, excluded, or included.
    screened = flow["identified"] - flow["duplicates"]
    excluded = flow["excluded_screening"] + flow["excluded_fulltext"]
    derived_included = screened - excluded
    assert derived_included == flow["included"], (
        f"PRISMA flow does not reconcile: {derived_included} vs reported {flow['included']}"
    )
    return {"records_screened": screened, "records_excluded": excluded, "studies_included": flow["included"]}

def rob_summary(studies: pd.DataFrame) -> pd.DataFrame:
    # Cross-tab of risk-of-bias by data source -- the table a GRADE panel needs to rate certainty.
    return (studies.pivot_table(index="data_source", columns="rob_overall",
                                values="study_id", aggfunc="count", fill_value=0))

def log_effect_se(ci_low: float, ci_high: float) -> float:
    # SE of the log-effect reconstructed from a reported 95% CI (HR/OR/RR are log-normal).
    return (np.log(ci_high) - np.log(ci_low)) / (2 * 1.959964)

def random_effects_pool(studies: pd.DataFrame) -> dict:
    yi = np.log(studies["effect"].to_numpy())                       # log effect per study
    sei = np.array([log_effect_se(lo, hi)
                    for lo, hi in zip(studies["ci_low"], studies["ci_high"])])
    wi = 1.0 / sei**2                                               # fixed-effect (inverse-variance) weights
    ybar_fe = np.sum(wi * yi) / np.sum(wi)
    Q = np.sum(wi * (yi - ybar_fe) ** 2)                           # Cochran's Q
    k = len(yi)
    C = np.sum(wi) - np.sum(wi**2) / np.sum(wi)
    tau2 = max(0.0, (Q - (k - 1)) / C)                            # DerSimonian-Laird between-study variance
    wi_re = 1.0 / (sei**2 + tau2)
    ybar = np.sum(wi_re * yi) / np.sum(wi_re)
    se = np.sqrt(1.0 / np.sum(wi_re))
    i2 = max(0.0, (Q - (k - 1)) / Q) * 100 if Q > 0 else 0.0      # I-squared (% variation from heterogeneity)
    # 95% prediction interval (Higgins-Thompson): where a NEW study's true effect is expected to fall.
    from scipy import stats  # optional; comment out and use 1.96 if scipy is unavailable
    t = stats.t.ppf(0.975, df=k - 2) if k > 2 else 1.959964
    pi = (ybar - t * np.sqrt(tau2 + se**2), ybar + t * np.sqrt(tau2 + se**2))
    return {
        "pooled_effect": float(np.exp(ybar)),
        "ci_95": (float(np.exp(ybar - 1.959964 * se)), float(np.exp(ybar + 1.959964 * se))),
        "tau2": float(tau2), "I2_pct": float(i2), "Q": float(Q), "k": int(k),
        "prediction_interval": (float(np.exp(pi[0])), float(np.exp(pi[1]))),
    }

def ffs_subgroup(studies: pd.DataFrame) -> pd.DataFrame:
    # Pre-specified subgroup: claims studies that excluded MA-only person-time vs those that did not.
    return (studies.assign(grp=np.where(studies["ffs_only"] == 1, "FFS-only", "mixed/other"))
                   .groupby("grp", group_keys=False)
                   .apply(lambda g: pd.Series(random_effects_pool(g))))

library(metafor)
library(dplyr)

# Reconstruct the log effect (yi) and its sampling variance (vi) from each study's reported 95% CI.
prep_effects <- function(studies) {
  studies %>%
    mutate(yi = log(effect),
           sei = (log(ci_high) - log(ci_low)) / (2 * qnorm(0.975)),
           vi = sei^2)
}

rob_summary <- function(studies) {
  # Risk-of-bias by data source -- the evidence-profile input for GRADE.
  with(studies, table(data_source, rob_overall))
}

pool_random_effects <- function(studies) {
  d <- prep_effects(studies)
  # DerSimonian-Laird (method = "DL"); switch to "REML" for the modern default.
  fit <- rma(yi = yi, vi = vi, data = d, method = "DL")
  pi <- predict(fit, transf = exp)          # 95% prediction interval, back-transformed
  list(
    pooled_effect = as.numeric(exp(coef(fit))),
    ci_95 = exp(c(fit$ci.lb, fit$ci.ub)),
    tau2 = fit$tau2, I2_pct = fit$I2, Q = fit$QE, k = fit$k,
    prediction_interval = c(pi$pi.lb, pi$pi.ub)
  )
}

# Pre-specified subgroup / meta-regression: did the claims study exclude MA-only person-time?
ffs_metareg <- function(studies) {
  d <- prep_effects(studies)
  rma(yi = yi, vi = vi, mods = ~ factor(ffs_only), data = d, method = "DL")
}