ISPOR Questionnaire for Relevance and Credibility of Observational/RWD Studies

What it is

The "ISPOR RWD Questionnaire" in this catalog refers to the ISPOR Good Practice Task Force questionnaire for assessing the relevance and credibility of observational (real-world data) studies for health-care decision making (Berger et al., 2014, an ISPOR-AMCP-NPC Good Practice Task Force Report). It is a structured appraisal instrument — a set of yes/no/can't-answer questions grouped into a relevance domain (does the study answer the decision-maker's actual question: population, intervention, comparators, outcomes, setting, generalizability?) and a credibility domain (design, data, analysis, reporting, interpretation, conflicts of interest). It was developed under the auspices of ISPOR (the professional society for health economics and outcomes research) and is maintained as part of ISPOR's Good Practices for Outcomes Research reports. Two later ISPOR-ISPE joint task-force reports extend the same appraisal logic specifically to routinely collected RWD: the good-practices recommendations for RWD studies of treatment/comparative effectiveness (Berger et al., 2017) and the reporting framework for healthcare-database studies to improve reproducibility and validity assessment (Wang et al., 2017). Treat the 2014 questionnaire as the appraisal instrument and the 2017 pair as its RWD-specific operationalization — they are complementary, not competing, documents.

When to use

Reach for this questionnaire when you must judge whether a completed (or proposed) observational/RWD study should inform a decision — a payer or HTA coverage/formulary review weighing a manufacturer's RWE dossier; an evidence team triaging published database studies for a comparative-effectiveness or safety question; a journal or regulatory reviewer screening a non-interventional submission; or an internal "go/no-go" credibility check before a study is cited in a value story. Use the relevance arm first: a methodologically flawless study that answers the wrong PICOTS question is useless to the decision at hand, and the questionnaire forces that judgment before any credibility scoring. Use it alongside, not instead of, design-stage and reporting-stage tools: pre-specify with HARPER or STaRT-RWE, report with STROBE/RECORD(-PE), and appraise relevance-plus-credibility with this ISPOR instrument. When the object of appraisal is specifically a healthcare-database study (claims, EHR, registry, linked), pull in the Wang 2017 reporting framework so that the credibility questions are answerable from the documentation the study should have provided.

What it requires

The instrument's domains map directly onto the operational decisions that make or break an RWD study, and a credible answer to each requires the artifacts a transparent study should already have produced: - Relevance / PICOTS alignment — the study population, exposure/intervention, comparator(s), outcomes, timing, and setting must match the decision question; generalizability/transportability to the target population is assessed explicitly, not assumed. - Data fitness-for-use — the data source(s) must be characterized (provenance, capture mechanism, linkage, lag/completeness, payer mix) and shown adequate for the exposures, outcomes, and covariates needed. - Phenotype / algorithm validity — exposure, outcome, and covariate definitions must be operationalized with code lists and, where claimed, validation metrics (PPV, sensitivity). - Time-zero alignment — index-date definition must avoid immortal time and align eligibility, treatment assignment, and start of follow-up. - Estimand and intercurrent events — the target estimand (population, treatment strategy, handling of switching, discontinuation, death) must be stated, not left implicit. - Confounding control — design (active comparator, new-user) and analysis (PS/hdPS, g-methods) must be adequate to the confounding structure, with balance diagnostics shown. - Attrition and missing data — loss to follow-up, censoring, and missingness must be reported and addressed. - Sensitivity / quantitative bias analysis — robustness to key assumptions (unmeasured confounding via E-value, negative controls, alternative specifications) must be demonstrated. Crucially, the questionnaire also requires transparency about pre-specification and conflicts of interest — whether the protocol and analysis plan predated the analysis, and who funded and conducted the work.

When NOT to use — limitations and common misapplications

This is a relevance-and-credibility appraisal instrument, not a risk-of-bias tool and not a numeric quality score. Do not convert its yes/no answers into a summed "quality score" and rank studies by it — the developers explicitly designed it for structured judgment, not scoring, and summary scores obscure which specific threat invalidates a study. For formal, signalling-question risk-of-bias assessment of a non-randomized study (e.g., within a systematic review), use ROBINS-I, which is the fit-for-purpose instrument; this questionnaire complements but does not replace it. Completing the questionnaire does not make an observational study causal — answering "yes" to the design questions documents that confounding was addressed, not that it was eliminated; residual and unmeasured confounding remain, which is why the sensitivity- analysis domain matters. Do not use it as a protocol template (that is HARPER / StaRT-RWE) or as a reporting checklist for the final manuscript (that is STROBE / RECORD-PE) — substituting one for another leaves real gaps that reviewers will find. Beware appraisal-as-theater: ticking boxes without inspecting the underlying code lists, validation studies, balance tables, and attrition diagrams produces a clean-looking checklist over an uncredible study. Finally, applying the generic 2014 questionnaire to a complex healthcare-database study without the Wang 2017 reporting expectations will leave the data-fitness and reproducibility questions effectively unanswerable.

How it maps to this catalog

Each appraisal domain has an implementing concept that supplies the operational detail a credible answer requires: - Relevance / PICOTS → `picots-framework-rwe` (and `generalizability-transportability-external-validity-rwe` for the target-population judgment). - Data fitness-for-use → `fit-for-purpose-data-assessment-rwe`, with `claims-analysis` and `medicare-ffs-ma-commercial-claims-differences-rwe` for source-specific capture and payer-mix nuances. - Phenotype / algorithm validity → `diagnosis-phenotype-algorithm-1ip-2op-time-window-rwe`. - Design / confounding control → `active-comparator-new-user` and `high-dimensional-propensity-score-hdps-rwe`, with `target-trial-emulation` as the pre-specification scaffold that makes the relevance and credibility questions answerable by construction. - Estimand / intercurrent events → `estimands-ate-att-intercurrent-events-rwe`. - Attrition / missing data → `attrition-and-loss-to-follow-up-rwe`. - Sensitivity / quantitative bias analysis → `e-value-sensitivity-analysis`. - Reporting artifacts (attrition flow, balance diagnostics) → `visualizations-pharmacoepidemiology-rwe`. Applied note (claims/EHR/registry RWE). A payer assessing a manufacturer's claims-based comparative-effectiveness study should run the relevance arm against `picots-framework-rwe`, then demand the credibility evidence concretely: the data-fitness narrative (`fit-for-purpose-data-assessment-rwe`), the outcome code list and PPV (`diagnosis-phenotype-algorithm-1ip-2op-time-window-rwe`), confirmation that the design is active-comparator/new-user with a balance table (`active-comparator-new-user`, `high-dimensional-propensity-score-hdps-rwe`), the attrition funnel (`attrition-and-loss-to-follow-up-rwe`), and an E-value or negative-control result (`e-value-sensitivity-analysis`). A "yes" with no supporting artifact is a "no."