Background. Increasing interest in using disease simulation models to evaluate public health programs has led to closer scrutiny of model methods and rising demand for evidence that models accomplish their intended goals.

Purpose. The purpose of this study was to identify obstructive lung disease simulation models from the recent literature and evaluate validation of those models.

Methods. A PubMed search was used to identify obstructive lung disease simulation models published from 2000 to 2005. Publications describing the models were reviewed to assess four types of model validation: first-order internal validity (verification/debugging), second-order internal validity (comparison of model estimates against data from studies used to construct the model), third-order internal validity (comparison of model estimates against published data not used to construct the model), and external (predictive) validity.

Results. Eight disease simulation models were identified, four for asthma and four for chronic obstructive pulmonary disease (COPD). Most were from Europe (75%) and funded by pharmaceutical companies (88%). Seven of eight were Markov models; one was a dynamic cohort model. Four models were developed for specific cost-utility analyses (CUA), three for use in future CUA, and one for exploratory analysis of future COPD burden. Seven of eight models included no mention of first-order internal validation; one did so implicitly. Five of eight models included tests of second-order internal validity. All were simulations of source study cohorts comparing modeled with observed outcomes. Disease exacerbation was the most common endpoint. Only one model included a statistical test of modeled versus observed outcomes. Two of eight models included tests of third-order internal validity. In both, modeled point estimates of COPD prevalence or exacerbation were compared to point estimates from published studies. No models included tests of predictive validity or plans for predictive validation. Validation was typically described as acceptable in qualitative terms, despite near-universal absence of criteria for judging adequacy of validation.

Conclusion. Most recent obstructive lung disease simulation models lack appropriate validation. For simulation modeling to be accepted as a tool for evaluating the impact of public health programs, models must be validated to the extent necessary to credibly simulate health outcomes of interest. Defining the requisite level of validation in diverse decision contexts is an important next step in promoting simulation models as practical decision tools.