CALIBRATION AND PREDICTIVE ACCURACY OF MIRCOSIMULATION MODELS

Purpose:

   The purpose of this study is to discuss statistical methods for calibrating and assessing the predictive accuracy of continuous time, dynamic Microsimulation Models used in Medical Decision Making.

Method:

   We apply two fundamental approaches, a Bayesian and an Empirical one, to calibrate the Microsimulation Lung Cancer (MILC) model, a streamlined MSM that describes the natural history of lung cancer and predicts important outcomes, such as lung cancer incidence and mortality. We compare these two methods in terms of theoretical aspects, the potential overlap in the resulting values, as well as the validity of the predictions of the final calibrated model each one produces.

   Furthermore we discuss statistical methods for an important yet rather overlooked aspect of MSMs, namely the assessment of the predictive accuracy of this type of models. In particular, we run a simulation study to compare the individual predictions from the calibrated MILC model with simulated outcomes, using C-statistics, a group of methods that has been widely used for assessing the predictive accuracy of survival models. We also compare the performance of C-statistics with other methods aimed at testing the deviations of survival distributions predicted by the calibrated MSM from the simulated truth.

Result:

   While Empirical Calibration methods prove more efficient, Bayesian methods seem to perform better especially when calibration targets involve rare outcomes. C-statistics are not very sensitive in capturing deviations of the individual predictions from the simulated truth.  Methods based on the comparison of predicted with observed survival distributions prove more effective for assessing the predictive accuracy of continuous time MSMs.

Conclusion:

   An effective calibration procedure of an MSM should combine an Empirical approach, for a more efficientl specification of plausible values for the model parameters, with a Bayesian method that will provide more accurate results choosing appropriate starting values from the previously defined ranges. In addition, techniques based on the comparison of predicted with observed survival curves seem to outperform C-statistics with regards to the assessment of the accuracy of the individual predictions received from a continuous time MSM.