Newcastle-Ottawa Scale (NOS)

What it is

The Newcastle-Ottawa Scale (NOS) is a critical-appraisal / risk-of-bias instrument for non-randomized observational studies, developed by Wells, Shea, O'Connell and colleagues at the Ottawa Hospital Research Institute (OHRI) in collaboration with the University of Newcastle (Australia) and maintained by OHRI rather than by EQUATOR or Cochrane. It exists as two separate forms — one for cohort studies and one for case-control studies — that share a common architecture but use different items. Appraisal is done by awarding stars across three domains: Selection (up to 4 stars — representativeness/definition of the exposed and non-exposed or case and control groups, and ascertainment of exposure or outcome at baseline), Comparability (up to 2 stars — whether the study controlled for the most important confounder and for additional confounders, with the reviewer naming the factors), and Outcome (cohort) or Exposure (case-control) ascertainment (up to 3 stars — blinding/record linkage, length and adequacy of follow-up, or non-response/ascertainment method). The maximum score is nine stars. NOS is deliberately lightweight: a trained reviewer can complete a form in roughly 20-30 minutes, which is why it became the default appraisal tool for observational evidence in thousands of meta-analyses.

When to use

Use NOS when you must appraise the methodological quality / risk of bias of individual cohort or case-control studies as part of a systematic review or meta-analysis of observational evidence — the standard context in a peer-reviewed evidence-synthesis manuscript, an HTA/payer evidence dossier that summarizes the observational literature, or a regulatory background section that grades existing non-randomized studies. Decision rules for choosing NOS vs siblings: (1) pick the cohort form for prospective/retrospective cohorts and the case-control form for case-control designs — do not mix them. (2) For appraising non-randomized studies of interventions (comparative drug or procedure effects), the Cochrane Handbook now recommends ROBINS-I (Sterne 2016) over NOS; reserve NOS for etiologic/prognostic exposure-outcome questions or where a fast, transparent appraisal across many studies is needed. (3) NOS appraises study-level conduct, not reporting completeness — if the task is to check whether a routinely-collected-data study is fully reported, use RECORD/RECORD-PE (a STROBE extension), not NOS. (4) NOS is not an evidence-certainty grading system; certainty of the pooled body of evidence is graded with GRADE, which sits downstream of per-study appraisal.

What it requires

Completing a NOS form requires the appraiser to make and document explicit judgments about: (a) selection — whether the exposed/case group is representative and the comparison group is drawn from the same source population, and whether exposure (case-control) or outcome (cohort) was secure and absent at baseline; (b) comparability — which confounders the study controlled for, with the reviewer pre-specifying the most important confounder (and a second factor) that must be adjusted to earn the comparability stars; and (c) outcome/exposure ascertainment — objective/record-linked vs self-reported ascertainment, blinding, and adequacy and length of follow-up with accounting for those lost. For real-world-data studies this maps onto exposure and outcome phenotype/algorithm validity (was the claims/EHR case definition validated?), data fitness-for-use (is the source population representative and the comparison group from the same data?), and attrition / loss to follow-up (is follow-up long enough and is dropout accounted for?).

When NOT to use — limitations and common misapplications

NOS is widely used and widely criticized, and a defensible appraisal must acknowledge this. (1) It is an appraisal tool, not a validated quality score. Summing stars into a single number and applying a cut-off (the ubiquitous "≥7 stars = high quality") is not validated; the developers never established score thresholds, and meta-regression or subgroup analysis on NOS totals can mislead. (2) Poor inter-rater reliability and ambiguous wording. Stang (2010) showed the "comparability" star hinges entirely on which confounder the reviewer chooses to require, "adequacy of follow-up" is arbitrary, and several items are open to divergent interpretation; Hartling (2013) empirically found low reliability between independent reviewers. Pre-specify, in your protocol, the required confounders and the follow-up threshold before appraisal. (3) It does not capture the dominant biases in pharmacoepidemiology. NOS has no item for immortal time bias, time-zero / index-date misalignment, prevalent-user (depletion-of-susceptibles) bias, or confounding by indication — a claims or EHR study can earn the full nine stars and still be biased to the point of uselessness for a regulatory or HTA decision. (4) Wrong tool for the question: using NOS to appraise non-randomized intervention effects where ROBINS-I is expected, using it as a reporting checklist (STROBE/RECORD-PE), or using it to grade body-of-evidence certainty (GRADE). (5) Appraisal-as-theater — filling in stars to satisfy a journal without letting the result change the synthesis or the interpretation.

How it maps to this catalog

NOS asks whether a study controlled bias; the catalog concepts specify how. The comparability domain (control of confounding) is operationalized by `active-comparator-new-user` (design-stage confounding-by-indication control) and `high-dimensional-propensity-score-hdps-rwe` (analytic confounding control) — both address exactly the star NOS reserves for "controlled for the most important factor." The selection domain (secure, comparable groups defined from the same source) is implemented through `diagnosis-phenotype-algorithm-1ip-2op-time-window-rwe` (validated exposure/outcome definitions) and `claims-analysis` (source-population and data-fitness considerations). The outcome/follow-up domain maps to `attrition-and-loss-to-follow-up-rwe`. The biases NOS structurally misses are precisely where the catalog adds value: `target-trial-emulation` and the time-zero alignment it enforces close the immortal-time and prevalent-user gaps, and `estimands-ate-att-intercurrent-events-rwe` makes explicit the causal contrast and intercurrent-event handling that a star count cannot represent. Applied note for claims/EHR/registry RWE: treat a high NOS score as necessary-but-insufficient. Before trusting a highly-starred observational study in an evidence dossier, separately confirm that exposure/outcome phenotypes were validated, that time zero was aligned for both arms (no immortal time), that the design used an active comparator and new-user restriction where confounding by indication is plausible, and that loss to follow-up was assessed as potentially informative — none of which NOS interrogates.