SETTINGS OF AN OPTIMIZATION-ALGORITHM-BASED CALIBRATION APPROACH CAN GREATLY IMPACT CALIBRATION PRECISION: A NUMERICAL STUDY TO SHOW THAT THE DEVIL IS IN THE DETAILS

Purpose:

Modelers can improve unsatisfactory model calibration results by changing the requirements for model parameters or outcomes to be calibrated, which can increase model uncertainty. In this study, we show that the implementation of the Optimization algorithm-based Calibration Approach (OCA) can significantly improve calibration results without increasing model uncertainty.

Methods:

Using OCA, modelers first transform a calibration task into an optimization problem. In the problem, the objective function quantifies the calibration gap (distance between model outcomes and calibration targets), and constraint functions enforce calibration requirements, such as maintaining feasible bounds of calibration parameters. Modelers then choose an optimization algorithm to minimize the calibration gap, keeping all feasible calibration sets for uncertainty analysis, and report the optimal calibration set for future base-case analysis.

We followed this procedure to calibrate the HIV Optimization and Prevention Economics (HOPE) model, a compartmental model of HIV disease progression and transmission in the United States. We calibrated 123 out of 870 parameters to fit three model outcomes, prevalence, incidence, and deaths, to their targets. We compared the difference in calibration precision when the standard OCA procedure was altered by the choice of (1) the optimization algorithm—pattern search (PS) algorithm versus simulated annealing (SA) algorithm; (2) the search starting point—random point versus good point (low objective function value); and (3) the search strategy—general search over all calibrated parameters versus concentrated search over prioritized parameters. We used the best calibration solution from the Latin Hypercube algorithm and one-way sensitivity analysis results around this solution to inform (2) and (3), respectively.

Results:

Table 1 summarizes the results collected from running each of the 6 settings of OCA for no more than 48 hours to solve the calibration problem. Although the choice of optimization algorithm and starting point did not seem to significantly impact the calibration precision, the concentrated search strategy based on insights from the one-way sensitivity analysis of calibrated parameters significantly improved the calibration performance, decreasing the gaps between model outcomes and their corresponding targets shown in the standard OCA by 61% ((4.75-1.85)/4.75) and 51% ((5.68-2.81)/5.68) for PS and SA, respectively.

Conclusion:

Modelers should explore how different OCA options can close the calibration gap before resorting to relaxing calibration requirements, such as altering the model outcome target values.