Purpose: Model uncertainty has received relatively limited attention, likely partly due to lack of clarity on the exact definition and scope of model uncertainty, and partly due to lack of guidance on systematic assessment of this kind of uncertainty. Our objectives were to develop and apply a systematic approach to assess model (structure and process) uncertainty, using an example of rotavirus vaccination in
Method: Based on a series of working definitions of key model elements, we developed five alternative Markov models of rotavirus disease. We examined how the epidemiological outcomes and cost-effectiveness estimates (2004 US dollars per disability-adjusted life year [DALY] averted) for rotavirus vaccination compared to no vaccination would change as model elements were systematically varied. Specifically, 1) to assess model structure uncertainty, we compared model outcomes while varying model substructure, holding methodological choices, parameter values, and model process constant; 2) to assess modeling process uncertainty, we varied modeling process within each given model, standardizing on methodological choices and parameter values; and 3) to illustrate how to examine model uncertainty in the context of a probabilistic sensitivity analysis, for selected sets of model structure and process, we performed second-order Monte Carlo simulations and presented the results as cost-effectiveness acceptability curves.
Result: The estimated numbers of severe rotavirus disease cases (i.e., death, hospitalization, or outpatient visit) were very close across different combinations of model structure examined, but the numbers of non-severe outcomes (i.e., mild diarrhea or asymptomatic infection) varied widely across the different models. As more complexities were added to model structure, the point estimates of incremental cost-effectiveness ratios ranged from $510 to $620 per DALY averted. The results indicated moderate sensitivity to the process choices examined, showing about 13% differences, at maximum, in incremental cost-effectiveness ratios among different sets of processes within each model.
Conclusion: Our findings suggest that different model structure and process lead to relatively modest differences in estimated cost-effectiveness of rotavirus vaccination, but that intermediate epidemiologic outcomes varied more substantially depending on choices of model structure. We caution against generalizing the quantitative results in this example, but suggest that the approach taken in this study may serve as a template for other examinations of model uncertainty.
Candidate for the Lee B. Lusted Student Prize Competition