Health-Economic Model Verification, Calibration, and Validation

Definition and scope

Separate activities for checking correct implementation, fitting uncertain model inputs to calibration targets, and assessing whether a decision model adequately represents its intended real-world use. The target population, use, and analysis remain protocol decisions; this card explains the choices and their consequences rather than approving a study.

Operational specification

Write an intended-use statement, conceptual model, input provenance, verification cases, calibration targets and likelihood/loss, and independent validation targets. Distinguish face validity, internal verification, cross-model comparison, external validation, and prospective predictive validation. Prespecify clinically meaningful tolerances rather than declaring any small numerical discrepancy acceptable.

Analysis and diagnostics

Test boundary cases, population accounting, timing and units; assess parameter identifiability and alternative calibrations; then compare prespecified predictions with evidence not used to calibrate them. Report uncertainty in predictions and observations and any systematic residual patterns. Maintain a traceable correction log and validation status for each output relevant to the decision.

When NOT to use and common errors

A model can be implemented correctly and still be clinically wrong. A calibration fit is not independent validation. Agreement between two models using the same wrong inputs is not external validation, and a single aggregate mean can conceal misfit across time or subgroups.

Worked example and implementation boundary

A synthetic external validation set has observed one-year mortality risk 0.12 and model prediction 0.15. Calculate signed prediction error, defined as predicted minus observed; acceptability requires a separately specified clinical tolerance and uncertainty assessment. The model overpredicts by 0.03, or 3 percentage points. This number alone does not establish whether the model is fit for purpose. The code performs that explicitly stated calculation with synthetic inputs. It is not a complete fitted analysis and does not demonstrate the identification assumptions, estimate sampling uncertainty, or certify application in another population.

Data-source and reporting requirements

Preserve the provenance, dates, units, and observation rule for each input. Claims capture services and transactions under observable benefits; absence of a record is not proof that an event or resource use did not occur. EHR and registry measurements depend on the collection process and may miss care outside the system. Linked sources add reconciliation and selection questions. Primary collection can improve a missing domain but still requires valid measurement and follow-up. Report which of these limitations affects the stated use, which evidence supports the chosen inputs, and what remains uncertain.

Worked example

Scenario

A synthetic external validation set has observed one-year mortality risk 0.12 and model prediction 0.15. Calculate signed prediction error, defined as predicted minus observed; acceptability requires a separately specified clinical tolerance and uncertainty assessment.

Dataset

Synthetic inputs with units defined in the scenario; no patient data.

Steps

Result

The model overpredicts by 0.03, or 3 percentage points. This number alone does not establish whether the model is fit for purpose.

Trade-offs

Code example

predicted = 0.15
observed = 0.12
result = predicted-observed
assert abs(result - 0.03) < 1e-9
print(result)

predicted <- 0.15
observed <- 0.12
result <- predicted-observed
stopifnot(abs(result - 0.03) < 1e-9)
print(result)

Citations

1. [1]Eddy DM, et al. Model Transparency and Validation: A Report of the ISPOR-SMDM Modeling Good Research Practices Task Force-7. Value in Health. 2012.
2. [2]Vemer P, et al. AdViSHE: A Validation-Assessment Tool of Health-Economic Models for Decision Makers and Model Users. PharmacoEconomics. 2016.