A SYSTEMATIC APPROACH TO EXPLORING THE IMPACT OF CALIBRATION UNCERTAINTY IN DISEASE MODELS

Purpose:

Economic and quantitative analyses of disease interventions typically assess costs and benefits of interventions using a model-based representation of the disease.  This requires the specification of numerous model parameters that represent transitions among health states.  Data associated with these transitions are generally limited and imperfect, and some transitions are not directly observable.  As a result, calibration techniques are employed to identify model parameters that achieve consistency between model outputs and available data.   The existence of multiple parameter selections that achieve similarly high levels of consistency lead to a phenomenon commonly known as "calibration uncertainty."  Systematic assessment of the impact of calibration uncertainty is an open area of investigation.

Methods:

We created a model of the calibration process that integrates mathematical representations of disease dynamics, goodness-of-fit measures involving common calibration targets, and some forms of internal and face validity.  Working with a Markovian representation of a disease, we examined cost-effectiveness using a net monetary benefit (NMB) criterion.  In this setting, we examined the impact of calibration uncertainty using both "perfect" and "sampled" data.  We performed a robust investigation of the impact of calibration uncertainty by using a modification of our model of the calibration process. 

Results:

We illustrate the effect of calibration uncertainty, and show that probabilistic sensitivity analysis is not sufficient to mitigate its impact.  Our methods permit an examination of the range of NMB observable over the parameter selections that fall within the calibration uncertainty set, as illustrated in Figure 1.  Our methods also provide a mechanism for incorporating expert judgement within the integrated model, which can substantially reduce the impact of calibration uncertainty.

Conclusion:

A wholistic and systematic approach to model calibration can lead to an enhanced understanding of calibration uncertainty and a quantification of its impact on analyses of disease interventions.  Incorporating the judgement of the disease expert within this approach can yield significant reduction of uncertainty.