Qualitative Evidence Synthesis

In plain language

Qualitative evidence synthesis is a way to gather findings from many studies that used interviews or focus groups to learn about patients' experiences, then combine those findings into a single, richer picture. Instead of averaging numbers, reviewers read across all the studies, notice shared ideas, and build a higher-level conclusion — for example, 'patients stop a new injectable therapy mainly because they feel embarrassed injecting in public, not because of medical side effects.' Each conclusion comes with an honest rating of how confident we are, based on the quality and number of studies behind it. It cannot compare how well two drugs work — that is a different kind of question — but it is exactly what health agencies need when they ask 'what do patients actually experience?'

Qualitative evidence synthesis (QES)

is the systematic identification, critical appraisal, and integration of findings from primary qualitative studies (interviews, focus groups, ethnography, open-ended survey text) to answer questions about experiences, perceptions, acceptability, feasibility, and barriers/facilitators that quantitative effect estimates cannot address. It is a full systematic-review method — protocol-driven, with an explicit search, transparent study selection, structured appraisal, a named synthesis method, and a graded statement of confidence in each finding — not an informal narrative summary. In HEOR and RWE it most often supplies the patient-experience and acceptability evidence that quantitative comparative-effectiveness and economic analyses leave unanswered.

Core conceptual distinction

QES is defined by what it synthesizes (qualitative findings, i.e., themes and author interpretations) and how it integrates them (interpretive or aggregative methods), and these are separable choices. (1) Aggregative vs interpretive synthesis: aggregative methods (meta-aggregation, framework synthesis, textual narrative synthesis) tally and pool findings against a pre-specified structure and stay close to participants' voices — well suited to HTA decision questions with a defined scope. Interpretive methods (meta-ethnography, thematic synthesis, grounded-theory synthesis) generate new second- and third-order constructs that go beyond any single study — well suited to theory-building. (2) Unit of analysis: the data are the published findings of primary studies (first-order = participant quotes, second-order = original authors' interpretations), and the synthesis produces third-order analytical themes. The output is not an effect estimate, a pooled prevalence, or a measure of "how many patients" — quantifying themes ("60% of studies mentioned cost") is vote-counting and is considered a misuse. The estimand-analogue is a set of review findings, each carrying a CERQual confidence rating (high / moderate / low / very low) built from four components: methodological limitations, coherence, adequacy of data, and relevance.

Pros, cons, and trade-offs

- vs quantitative meta-analysis / network meta-analysis: QES answers why and how (mechanisms, acceptability, lived burden) where meta-analysis answers how much (pooled effect). Cost: no effect size, no transitivity logic, no I-squared; findings are interpretive and not "pooled" in a statistical sense. Prefer QES for the patient-perspective, treatment-burden, and implementation questions that increasingly drive HTA patient submissions; prefer meta-analysis for comparative efficacy/safety. - vs an unsystematic narrative review of qualitative studies: QES adds a protocol, reproducible search, transparent selection, formal appraisal (CASP/Cochrane), a named synthesis method, and CERQual grading — making the conclusions auditable. Cost: substantially more labor and methodological expertise. Prefer QES whenever the synthesis informs a decision (HTA, guideline, label, payer dossier); a narrative overview is acceptable only for scoping or background. - vs mixed-methods / segregated convergent reviews: A standalone QES isolates and rigorously synthesizes the qualitative strand; a convergent or segregated mixed-methods review then juxtaposes it with a quantitative strand to explain heterogeneity or contextualize effects. Prefer standalone QES when the question is purely about experience; embed it in a mixed-methods review when the goal is to interpret why a quantitative effect varies.

When to use

Decision questions about patient/clinician experience, treatment burden, acceptability, adherence drivers, equity, and implementation barriers — especially the patient-perspective and context sections of HTA submissions (NICE, CADTH), guideline development (WHO, Cochrane QIMG), and payer dossiers where the lived experience of a therapy matters. Use it when a body of primary qualitative research exists and the goal is a transparent, gradeable synthesis of findings rather than a count of effects.

When NOT to use — and when it is actively misleading or dangerous

- The question is comparative effectiveness, safety, prevalence, or cost. QES cannot estimate effects; presenting synthesized themes as evidence of how well a drug works, or quantifying themes into pseudo-prevalences, is a category error that decision-makers can mistake for effect evidence. - The evidence base is thin or one or two large studies dominate. With few studies, "adequacy of data" is low, third-order interpretation overreaches, and CERQual confidence collapses to low/very low — a synthesis that looks authoritative but rests on almost no data is dangerous in a decision context. - Findings are pooled as if commensurable counts (vote-counting). Treating "number of studies reporting a theme" as a measure of importance conflates research attention with patient salience and is explicitly cautioned against. - The included studies are methodologically weak in the same direction. If all primary studies recruit articulate, high-engagement volunteers, the synthesis amplifies a shared selection bias; methodological-limitations and relevance components of CERQual must flag this rather than averaging it away. - Reviewers lack qualitative methods expertise. Mechanically extracting quotes without engaging the interpretive layer produces a quote-collage, not a synthesis, and misrepresents the original authors' meaning.

Data-source operational depth

The "data sources" of a QES are types of source studies and documents, each with distinct failure modes and workarounds. - Primary peer-reviewed qualitative studies (interviews/focus groups/ethnography): the core substrate. Failure modes: thin "findings" sections that report quotes without the authors' interpretation (limits second-order data); poorly reported sampling so saturation/transferability cannot be judged; over-representation of articulate, engaged participants. Workaround: extract first- and second-order data verbatim into an audit-able table, appraise with CASP, and down-rate adequacy/relevance in CERQual when reporting is thin. - Mixed-methods studies: the qualitative strand is often buried as a sub-study with truncated reporting. Failure mode: the qualitative findings are subordinated to the trial and under-described. Workaround: extract only the qualitative component, contact authors for fuller findings, and record that the strand was secondary. - Grey literature and conference abstracts: improve coverage and reduce publication/dissemination bias but carry thin description and no peer review. Failure mode: an abstract states a theme with no supporting data to appraise. Workaround: include for comprehensiveness but mark low methodological confidence and never let an un-appraisable abstract anchor a high-confidence finding. - Patient-organization and regulatory patient-experience reports (e.g., HTA patient submissions): highly relevant to decision questions but not peer-reviewed and potentially advocacy-shaped. Failure mode: selection toward motivated, organized patients. Workaround: treat as a distinct source type, triangulate against peer-reviewed studies, and document potential conflicts in the relevance component. - Cross-language / cross-setting evidence: restricting to English or to one health system narrows transferability. Failure mode: a finding "confident" only because the evidence base is homogeneous in a way that hides setting effects. Workaround: justify language limits explicitly and use the relevance component to flag setting mismatch between the evidence and the decision context.

Worked example (HTA patient-perspective synthesis)

Question: what shapes initiation and persistence with GLP-1 receptor agonists from the perspective of adults with type 2 diabetes, to inform the patient-experience section of an HTA submission. (1) Protocol & question framing: use SPIDER (Sample = adults with T2D; Phenomenon of Interest = experience of GLP-1 RA therapy; Design = interviews/focus groups; Evaluation = acceptability, burden, persistence; Research type = qualitative) rather than PICO, and register the protocol. (2) Search: MEDLINE + CINAHL + PsycINFO with qualitative search filters, supplemented by reference-chaining and grey literature; English-language restriction justified and recorded as a relevance caveat. (3) Selection: dual independent screening, PRISMA flow, disagreements resolved by a third reviewer. (4) Appraisal: CASP qualitative checklist per study, feeding the methodological- limitations component of CERQual. (5) Extraction: capture first-order (participant quotes) and second-order (author interpretations) data into a structured table. (6) Synthesis: Thomas & Harden three-stage thematic synthesis — free line-by-line coding of findings, organization into descriptive themes, then generation of analytical (third-order) themes such as "injection identity and stigma," "weight-loss as motivator vs side-effect-driven discontinuation," and "cost and access friction." (7) Confidence: rate each review finding with GRADE-CERQual across methodological limitations, coherence, adequacy, and relevance, producing an Evidence Profile and a Summary of Qualitative Findings table. (8) Reporting: report against the ENTREQ statement so the HTA reviewer can audit every step from search to graded finding. The deliverable is a small set of analytical themes each carrying an explicit confidence rating — not a count of studies and not an effect estimate.

Worked example

Scenario

A health technology assessment body needs to understand what shapes whether adults with type 2 diabetes start and stay on a once-weekly injectable diabetes therapy. Four qualitative studies are identified — each interviewed or conducted focus groups with patients and reported its own themes. The synthesis reviewers read across all four and build a single higher-order conclusion.

Dataset

Findings table: one row per study, showing the key theme that study reported — the raw material for synthesis.

Steps

• Read each study's findings section and attach a short label to every idea mentioned — for example: 'needle fear,' 'routine disruption,' 'public embarrassment,' 'device feel.'

• Group the labels that share the same underlying idea across studies: 'needle fear' (Alvarez) and 'injection anxiety' (Chen) belong together under a descriptive theme called 'Anxiety about the injection act itself'; 'public embarrassment' (Okonkwo) belongs under 'Social visibility and stigma of injecting.'

• Step back and ask what these descriptive themes are saying together: three of four studies describe some form of social or psychological discomfort around the injection — not the medicine's effects on the body.

• Write the analytical theme: 'The primary barriers to initiating and continuing this therapy are social and psychological — stigma, embarrassment, and disruption of public routine — rather than medical side effects.'

• Rate confidence: four studies from four countries all point in the same direction, methods are sound, and the patient populations match the decision context. Confidence is rated moderate (not high, because the number of studies is still relatively small).

Result

Synthesized analytical theme: 'Barriers to starting and continuing weekly injectable therapy center on social stigma and how injecting fits into public daily life, not primarily on side effects' — rated moderate CERQual confidence, supported consistently across all four included studies from four different countries. No numbers are pooled; the output is a graded interpretive statement, not an effect size or percentage.

Runnable example

import pandas as pd

# CERQual: start at "high" confidence, then down-grade for concerns in each of the four components.
_DOWNGRADE = {"none": 0, "minor": 0, "moderate": 1, "serious": 2}
_LEVELS = ["very low", "low", "moderate", "high"]  # index 3 == high

def cerqual_rating(findings: pd.DataFrame,
                   appraisal: pd.DataFrame,
                   relevance: pd.DataFrame,
                   coherence: dict[str, str],          # finding_id -> 'none'/'minor'/'moderate'/'serious'
                   min_studies_adequate: int = 4) -> pd.DataFrame:
    rows = []
    for fid, grp in findings[findings["supports"]].groupby("finding_id"):
        studies = grp["study_id"].unique()
        n = len(studies)

        # 1) Methodological limitations: worst CASP concern among contributing studies.
        casp = appraisal.loc[appraisal["study_id"].isin(studies), "casp_concerns"]
        meth = max((_DOWNGRADE[c] for c in casp), default=2)

        # 2) Coherence: reviewer-assigned (fit between data and the finding).
        coh = _DOWNGRADE[coherence.get(fid, "moderate")]

        # 3) Adequacy: thin data when few studies / quotes support the finding.
        adeq = 0 if n >= min_studies_adequate else (1 if n >= 2 else 2)

        # 4) Relevance: indirect setting match to the decision context down-grades.
        rel_map = relevance[relevance["finding_id"] == fid]["setting_match"]
        rel = {"direct": 0, "partial": 1, "indirect": 2}.get(
            rel_map.mode().iat[0] if len(rel_map) else "indirect", 2)

        score = 3 - (meth + coh + adeq + rel)        # subtract total concern from "high"
        rating = _LEVELS[max(0, min(3, score))]
        rows.append({"finding_id": fid, "n_studies": n, "meth_limits": meth,
                     "coherence": coh, "adequacy": adeq, "relevance": rel,
                     "cerqual_confidence": rating})
    return pd.DataFrame(rows).sort_values("finding_id").reset_index(drop=True)

library(dplyr)
library(tidyr)

thematic_synthesis <- function(codes, descriptive, analytical) {
  # Stage 1->2: attach descriptive themes to coded findings.
  coded <- codes |>
    left_join(descriptive, by = "code")

  # Stage 2->3: roll descriptive themes up to analytical (third-order) themes.
  mapped <- coded |>
    left_join(analytical, by = "descriptive_theme")

  # Summary of Qualitative Findings skeleton: contributing studies per analytical theme,
  # with how much of the support is second-order (author interpretation) vs first-order (quote).
  mapped |>
    group_by(analytical_theme) |>
    summarise(
      n_studies        = n_distinct(study_id),
      n_descriptive    = n_distinct(descriptive_theme),
      prop_second_order = mean(order == 2L),   # higher = richer interpretive support
      .groups = "drop"
    ) |>
    arrange(desc(n_studies))
    # NB: n_studies informs CERQual 'adequacy' - it is NOT a vote-count of importance.
}