Markov Transition Probabilities from Real-World Data

A Markov (state-transition) cost-effectiveness model partitions a disease into a finite set of mutually exclusive, collectively exhaustive health states and moves a hypothetical cohort through them in fixed-length cycles. The numbers that drive everything the model produces — life-years, QALYs, and costs accrued in each state — are the transition probabilities: the per-cycle probability of moving from state i to state j. This concept is about estimating those probabilities from real-world longitudinal data (claims, EHR, registry, or linked sources), as opposed to lifting them from published trials or expert opinion. It is the data-engineering and statistical step that sits upstream of the decision model, not the economic evaluation itself.

Core conceptual distinction

Three things must be separated and pre-specified.

1. Probability vs rate. A transition rate is an instantaneous hazard (events per person-time, continuous); a transition probability is the chance of being in state j one cycle later (bounded 0–1, cycle-length-dependent). Cohort Markov models consume probabilities, but the honest way to get them from time-to-event data is to estimate rates and convert with the matrix exponential p = exp(Q·Δt), where Q is the transition-intensity matrix — not the naive 1 − exp(−rate·Δt), which is only correct for a single competing transition and silently mis-allocates probability when a state has more than one exit (Welton & Ades 2005).
2. Fully vs partially observed transitions. If you see every transition the moment it happens (e.g., a death index), counts give consistent estimates. If you observe state only at intermittent, irregular visits (the usual claims/EHR reality), the data are panel/interval-censored: an intervening state could have been missed, and count-ratio estimators are biased. Continuous-time multistate models (the `msm` framework, Jackson 2011) recover the intensity matrix from snapshots.
3. Time-homogeneity. The simplest model assumes a single Q for all cycles; real progression usually depends on time-in-state (semi-Markov / tunnel states) and calendar/age, so a homogeneous matrix can be badly wrong over a lifetime horizon. The estimand is therefore the cycle-specific transition-probability matrix for the target decision population, with uncertainty propagated (Dirichlet/multinomial draws for PSA per Briggs 2012), not a single point matrix.

Pros, cons, and trade-offs

• vs trial-derived or literature transition probabilities: RWD-derived transitions reflect the real treated population, settings, and adherence, and can be estimated for states and subgroups no trial reports. Cost: they inherit all of claims/EHR's measurement error (state misclassification, informative visit timing, left truncation), and naive count ratios from irregular data are biased. Prefer RWD when the decision population diverges from trial populations or when long-horizon/real-adherence transitions are needed — but validate against external benchmarks.
• vs a partitioned-survival model (PSM): PSM reads area-under-the-curve directly from extrapolated PFS/OS curves and needs no transition structure, which makes it simple but structurally unconstrained — post-progression survival is implied, not modeled, and the three curves can imply impossible state occupancy. A Markov/multistate model enforces a coherent state structure and lets you model post-progression mortality explicitly. Prefer Markov when the disease has clinically meaningful intermediate states and back-transitions; prefer PSM for two-state oncology problems with mature survival curves where transition data are thin.
• vs naive count-ratio (cohort-count) estimation: counting i→j moves and dividing by time-at-risk-in-i is trivial and transparent, but ignores interval censoring and competing risks, and cannot borrow strength across cells. A continuous-time multistate model (msm / Aalen-Johansen for the empirical matrix) is the methodologically correct upgrade at the cost of convergence fragility and stronger Markov assumptions. Prefer multistate estimation whenever states are observed only at irregular visits or death competes with progression.

When NOT to use — and when it is actively misleading or dangerous

• Irregular observation treated as exact-time data. Plugging interval-censored claims snapshots into count-ratio formulas, or applying 1 − exp(−rate) per transition when a state has competing exits, produces matrices whose rows look fine but whose long-run state occupancy is wrong by the time it compounds over 40+ cycles. This is the most common and most dangerous error: it is invisible in one cycle and catastrophic in a lifetime model.
• Disenrollment misread as a clinical transition. In claims, a patient who leaves the plan (especially to Medicare Advantage, where fee-for-service claims vanish) simply stops being observed. If censoring is not modeled, the cohort appears to "transition out," biasing every downstream probability. Never code end-of-data as a health-state move.
• A two-state, no-back-transition problem. If the disease is really just alive→dead with one progression event and mature survival curves exist, a Markov model adds structure (and assumptions) you do not need — a partitioned-survival or simple survival extrapolation is more honest.
• Sparse cells. When some i→j transitions have a handful of events, the matrix is unstable and PSA will swing wildly; Markov chaining propagates that instability. Collapse states or borrow strength (hierarchical/evidence synthesis, Welton & Ades 2005) rather than shipping a point matrix.
• Strong time-in-state dependence forced into a homogeneous matrix. Diseases with accelerating hazards (e.g., late CKD) violate the memoryless assumption; a single homogeneous Q understates progression and overstates survival.

Data-source operational depth

• Claims (FFS vs MA): State must be derived per cycle from diagnosis/procedure/drug codes and any linked labs, because claims have no native "state" field. Pin a fixed cycle length, assign each person's state at each cycle boundary, and decide a missing-data rule (last-observation-carried-forward within a max gap vs treat as censored). Failure modes: Medicare Advantage enrollees lack FFS person-time, so apparent transitions are observation gaps — restrict to continuously FFS-enrolled spans (Parts A/B/D) and censor at MA switch. Differential disenrollment by health state (sicker patients change plans or die) makes "lost to follow-up" informative. Left truncation: prevalent patients enter mid-trajectory, so the apparent starting-state mix is not the inception mix. Death is poorly captured in claims alone — without a death index, the absorbing state leaks into "censored," which inflates survival; link to vital records or use a hospice/inpatient-discharge-disposition proxy and acknowledge its incompleteness.
• EHR: States from labs/vitals/problem lists are richer (true eGFR for CKD staging, EF for HF) but visit-driven: a patient seen more often has more chances to be reclassified, so sicker patients accrue spurious transitions (ascertainment by visit frequency). External-care leakage means transitions happening outside the system are missed. Use the irregular visit times explicitly in a continuous-time model rather than forcing a fixed grid.
• Registry: Disease-specific registries often record state directly and adjudicate it (cancer stage, dialysis start), which is ideal for the transition structure, but capture of competing mortality and of inter-state utilization cost may be weak — link to claims for costs and to a death index for the absorbing transition.
• Linked claims–EHR–vital records: The ideal substrate (EHR-derived states + claims completeness + reliable death), but linkage selects the linkable subset (transportability) and introduces date discrepancies between lab, claim, and death dates that must be reconciled before assigning a state to a cycle boundary.

Worked claims example

Goal: a 3-month-cycle transition matrix for CKD progression (G3a → G3b → G4 → ESRD → Death, with possible back-transitions among the eGFR stages and Death/ESRD absorbing or near-absorbing) to feed a lifetime cost-utility model, using a linked claims+lab dataset.

1. Cohort/inception: adults with ≥2 eGFR values defining G3a/G3b and ≥365 days continuous FFS enrollment (Parts A/B) before the first qualifying eGFR (`index_date`), excluding Medicare Advantage person-time.
2. State assignment per cycle: lay a 91-day cycle grid from `index_date`; at each boundary assign the CKD stage from the most recent eGFR within the prior 180 days (carry forward); if no eGFR within the window and the person is still enrolled, treat the cycle as missing (interval-censored) rather than carrying an old value indefinitely. ESRD is set from the first dialysis/transplant procedure or revenue code (`dx`/`px` on the medical claim); Death from the linked vital-records `death_date`.
3. Censoring: stop person-time at the earliest of disenrollment, switch to MA, or end of data — these are censoring, never a transition to Death.
4. Estimation: because eGFR is observed at irregular lab dates, fit a continuous-time multistate (msm) model on the (person, eGFR-date, state) panel with Death as an exactly-timed absorbing state, recover the intensity matrix Q, and convert to a 91-day probability matrix via the matrix exponential; report the empirical Aalen-Johansen state-occupancy as a check.
5. Uncertainty: draw the transition matrix from the multivariate distribution of the fitted intensities (or Dirichlet draws of each row for a count-based sensitivity version) for probabilistic sensitivity analysis.
6. Diagnostics: compare modeled vs observed stage occupancy over follow-up, test time-homogeneity (split early vs late follow-up), and run a disenrollment-as-informative-censoring sensitivity analysis.

Interpreting the output

The worked example shows a 3-state Markov model (Stable, Progressed, Dead) with a 3-month cycle. After one cycle, a starting cohort of 1,000 in the Stable state redistributes to 585 Stable, 280 Progressed, and 135 Dead.

(1) Formal interpretation. Each row of the transition probability matrix must sum to 1.0, confirming that every patient in a state either stays or moves to exactly one other state — the matrix is stochastic. The result 585 + 280 + 135 = 1,000 confirms cohort conservation for this cycle. The 3-month cycle length means each probability represents the likelihood of a state change within a quarter; the same patient-level hazard translated to a 1-month cycle would yield different (generally smaller) per-cycle probabilities. A critical limitation is the Markov memorylessness assumption: the transition probability from Stable to Progressed is the same regardless of how many prior cycles the patient has already spent in Stable. When time in state is clinically meaningful (e.g., time since treatment start predicts progression risk), a tunnel-state extension, a multistate model with time-in-state covariates, or a DES approach is required.

(2) Practical interpretation. The 13.5% Dead probability from Stable over one 3-month cycle is the dominant driver of life-years in the model. Analysts should compare modeled state occupancy at each cycle against observed Kaplan-Meier curves or registry prevalence as a validation check — if the modeled progression curve diverges from the observed data by cycle 4, the transition probabilities need reestimation or the time-homogeneity assumption needs relaxing. Any PSA draws over the transition matrix must respect row-sum = 1 (use Dirichlet distributions for each row, not independent betas).

Decision diagram

stateDiagram-v2
  direction LR
  G3a: CKD G3a
  G3b: CKD G3b
  G4: CKD G4
  ESRD: ESRD (dialysis/transplant)
  Dead: Death (absorbing)
  G3a --> G3b
  G3b --> G3a
  G3b --> G4
  G4 --> G3b
  G4 --> ESRD
  G3a --> Dead
  G3b --> Dead
  G4 --> Dead
  ESRD --> Dead

flowchart TD
  Raw[Irregular claims/EHR observations<br/>eGFR labs, dx/px, linked death date] --> Grid[Lay fixed cycle grid<br/>e.g. 91-day cycles from index]
  Grid --> Assign[Assign state at each cycle boundary<br/>LOCF within max gap, else interval-censored]
  Assign --> Censor[Mark censoring: disenroll / MA switch / end of data<br/>NOT a transition]
  Censor --> Est{Observation pattern?}
  Est -->|Dense grid + death index| Count[Count-ratio matrix<br/>i to j moves / at-risk in i]
  Est -->|Irregular snapshots| MS[Continuous-time multistate<br/>fit Q, P = exp of Q times cycle]
  Count --> Valid[Validate vs Aalen-Johansen occupancy<br/>test time-homogeneity]
  MS --> Valid
  Valid --> PSA[Propagate uncertainty<br/>Dirichlet rows / fitted-intensity draws for PSA]

Worked example

Scenario

A health economist is building a simple cost-effectiveness model for a new cancer drug. She defines three health states: Stable (disease has not progressed), Progressed (disease has worsened), and Dead. Using one year of claims data she estimates the per-cycle (3-month) probabilities of moving between these states. She starts with a cohort of 1,000 patients and wants to see how they redistribute after one cycle.

Dataset

Transition probability matrix estimated from claims data. Each row is the origin state; each column is the destination state after one 3-month cycle. Every row sums to 1.0.

Steps

Result

After one 3-month cycle the cohort of 1,000 redistributes to 585 Stable, 280 Progressed, and 135 Dead. Total = 585 + 280 + 135 = 1,000, confirming the cohort is conserved. The model repeats this multiplication every cycle, chaining the matrix forward over the full time horizon to accumulate life-years and costs in each state.

Trade-offs

Runnable example

import pandas as pd
import numpy as np

def empirical_transition_matrix(states: pd.DataFrame) -> pd.DataFrame:
    s = states.sort_values(["person_id", "cycle"]).copy()

    # Pair each observed cycle with the SAME person's next observed cycle.
    s["next_cycle"] = s.groupby("person_id")["cycle"].shift(-1)
    s["next_state"] = s.groupby("person_id")["state"].shift(-1)

    # Keep only adjacent cycles (no gap) so an unobserved intervening state
    # is not silently collapsed into a single transition.
    moves = s[s["next_cycle"] == s["cycle"] + 1]

    order = sorted(set(s["state"]))
    counts = (moves.groupby(["state", "next_state"])
                   .size()
                   .unstack(fill_value=0)
                   .reindex(index=order, columns=order, fill_value=0))

    row_tot = counts.sum(axis=1).replace(0, np.nan)
    P = counts.div(row_tot, axis=0)

    # Absorbing/unseen-exit rows: keep the cohort in-state (self-loop = 1).
    P = P.fillna(0.0)
    empty_rows = P.sum(axis=1) == 0
    for st in P.index[empty_rows]:
        P.loc[st, st] = 1.0
    return P  # rows sum to 1.0; pass to PSA via row-wise Dirichlet draws of `counts`

library(msm)
library(expm)

CYCLE_DAYS <- 91  # 3-month model cycle

# qmat: allowed transitions get a small positive initial intensity; 0 = structurally impossible.
# Example 5-state CKD: 1=G3a 2=G3b 3=G4 4=ESRD 5=Death (5 absorbing).
qmat <- rbind(
  c(0, 0.1, 0,   0,   0.02),
  c(0.05, 0, 0.1, 0,   0.03),
  c(0, 0.05, 0,  0.1,  0.05),
  c(0, 0,   0,   0,    0.10),
  c(0, 0,   0,   0,    0))

fit <- msm(state ~ obs_time, subject = person_id, data = panel,
           qmatrix = qmat, deathexact = 5, gen.inits = TRUE)

# Recover the estimated intensity matrix Q and convert to a CYCLE_DAYS probability matrix.
Q <- qmatrix.msm(fit, ci = "none")
P_cycle <- expm(Q * CYCLE_DAYS)          # rows sum to 1; this is the Markov-model input
# pmatrix.msm(fit, t = CYCLE_DAYS) gives the same matrix directly, with CIs for PSA.

%let cycle_gap = 1;  /* require strictly adjacent observed cycles */

/* Pair each observed cycle with the same person's next observed cycle. */
proc sql;
  create table moves as
  select a.person_id,
         a.state as from_state length=12,
         b.state as to_state   length=12
  from work.states a
  inner join work.states b
    on a.person_id = b.person_id
   and b.cycle = a.cycle + &cycle_gap;
quit;

/* Cell counts and row totals -> transition probabilities. */
proc freq data=moves noprint;
  tables from_state*to_state / out=cellcounts(keep=from_state to_state count);
run;

proc sql;
  create table tprob as
  select c.from_state, c.to_state, c.count,
         c.count / t.row_total as p_transition format=8.5
  from cellcounts c
  inner join (select from_state, sum(count) as row_total
              from cellcounts group by from_state) t
    on c.from_state = t.from_state
  order by from_state, to_state;
quit;

/* cellcounts feeds row-wise Dirichlet(count + prior) draws for probabilistic sensitivity analysis. */

Citations

1. [1]Sonnenberg FA, Beck JR. Markov models in medical decision making: a practical guide. Medical Decision Making. 1993;13(4):322-338.
2. [2]Welton NJ, Ades AE. Estimation of Markov chain transition probabilities and rates from fully and partially observed data: uncertainty propagation, evidence synthesis, and model calibration. Medical Decision Making. 2005;25(6):633-645.
3. [3]Caro JJ, Briggs AH, Siebert U, Kuntz KM. Modeling good research practices - overview: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-1. Value in Health. 2012;15(6):796-803.
4. [4]Briggs AH, Weinstein MC, Fenwick EAL, Karnon J, Sculpher MJ, Paltiel AD. Model parameter estimation and uncertainty analysis: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-6. Medical Decision Making. 2012;32(5):722-732.

1. [5]Jackson CH. Multi-state models for panel data: the msm package for R. Journal of Statistical Software. 2011;38(8):1-28.