OPTIMIZING PARAMETERS FOR SCREENING MODELS: A COMPARISON OF DIFFERENT PARAMETER SEARCH ALGORITHMS
Candidate for the Lee B. Lusted Student Prize Competition
Purpose:
We evaluated the performance of six parameter search algorithms, an important aspect of calibration (parameter estimation) of microsimulation models.
Methods:
We have currently implemented four parameter search algorithms (Nelder-Mead, Genetic Algorithm, Response Surface, and Simulated Annealing; SCORE function and “Bayesian Optimization” are work in progress) in the MISCAN-Colon microsimulation model for colorectal cancer. We first used the MISCAN-Colon model with fixed, known parameters to generate a hypothetical dataset of observations for calibration. Next, we cut loose two parameters and used the four different search algorithms to estimate these parameters based on the hypothetical dataset. These parameters are either two weakly or two highly correlated parameters. Each calibration was performed with 100 unique sets of random starting values for the parameters that were calibrated to address parameter and stochastic uncertainty. The performance of each algorithm was assessed by comparing the estimated parameter with the underlying parameter used to generate the hypothetical dataset in terms of bias and Root Mean Squared Prediction Error (RMSPE). We performed an additional analysis on a more narrow parameter search area (50 unique sets of random starting values). For this the RMSPE is expected to be lower compared to the wide parameter search area because the parameter space is limited to an area more closely to the underlying parameter values. Furthermore, we compared the computation time of the calibration procedures.
Results:
In the wide parameter search area for both 2-parameter cases the Nelder-Mead algorithm performed best in terms of RMSPE (0.017 and 0.177, Table). In addition, Nelder-Mead needed on average significantly less time to reach that result. In the narrow parameter search area Nelder-Mead is again the best performing algorithm in the two weakly correlated parameter case. However, in the highly correlated parameter case the Genetic Algorithm performs best (RMSPE of 0.083). Comparing a wide with a narrow parameter search area, the Genetic Algorithm, Simulated Annealing and Response Surface perform significantly better when the search area is more narrow.
Conclusion:
In case of calibration of two parameters, Nelder-Mead is best used as a parameter search algorithm in calibration. The algorithm leads to accurate and rapid estimation of parameters in various settings. Restricting the parameter search area significantly improved performance of calibration using the Genetic Algorithm, Simulated Annealing and Response Surface.
