METHODS OF DISEASE MODEL CALIBRATION: THEORY AND PRACTICE

Overview: This intermediate-level course is intended to introduce the science and art behind disease model calibration methods. We encourage participants to discuss how calibration could be used to improve simulation models from their own research.

Background: In the context of disease modeling, “calibration” refers to the systematic adjustment of model inputs such that the resulting model outputs better reflect setting-specific observed disease outcomes. This course encompasses theoretical and practical aspects of model calibration methodology, including: 1) Identifying calibration targets; 2) Determining model inputs to be calibrated; 3) Selecting goodness of fit criteria; and 4) Implementing an appropriate parameter search approach. The International Society for Pharmacoeconomics and Operations Research (ISPOR)-SMDM Joint Modeling Good Research Practices Task Force acknowledges model calibration as a potentially useful component of input parameter estimation, model validation, and uncertainty analysis. Model calibration is particularly useful when there are limited or nonexistent data on transition probabilities, but disease endpoint data are available. For example, setting-specific statistics on cervical cancer disease outcomes are often available, whereas there are virtually no data sources that can directly inform specific cervical disease progression inputs.

Format Requirements: The format of this course combines lecture-style presentations with hands-on spreadsheet exercises to introduce and build on theoretical and practical aspects of model calibration. The initial third of the course reviews the conceptual foundations of disease modeling (briefly), situations where model calibration is useful, and the main phases of carrying out model calibration. The second third of the course utilizes hands-on spreadsheet exercises to walk participants through various calibration parameter search and evaluation approaches. The final third of the course introduces advanced topics and related spreadsheet demonstrations. Proficiency with Microsoft Excel is a required, and familiarity with Excel macros/Visual Basic for Applications (VBA) programming and/or Solver Add-in is preferred but not required. This course is taught at an intermediate level and is intended for those with experience or interest in programming or evaluating disease models. Participants should bring laptop computers for the interactive exercises.

Description and Objectives: In the context of disease modeling, “calibration” refers to the systematic adjustment of model inputs such that the resulting model outputs better reflect setting-specific, observed disease outcomes. This course encompasses the main theoretical and practical aspects of model calibration methodology. Stylized practice models and realistic oncology disease models programmed in Microsoft Excel are utilized to give participants hands-on experience employing commonly-used and cutting edge model calibration techniques. The main objectives of the course are: 

• To review modeling circumstances that particularly benefit from calibration;
• To introduce the main methodological phases of model calibration, specifically: 1) Identifying calibration inputs/outputs;  2) Determining goodness of fit criteria, such as windows-based targets, minimizing deviation/least-squares approaches, and likelihood-based functions;  3) Selecting and implementing parameter search algorithms, including manual adjustment of model parameters, random searches, and optimization techniques (linear programming and directed-search algorithms);
• To demonstrate how to implement these methods using interactive spreadsheet exercises;
• To highlight several advanced topics.

Specific advanced topics include:

• Comparisons of manually-intensive versus computationally-intensive parameter search strategies (such as simulated annealing, Latin hypercube, and Nelder-Mead search algorithms);
• Identification of and correction for bias introduced from calibrating longitudinal models to cross-sectional data;
• Probabilistic and deterministic uncertainty analysis for calibrated disease models;
• Bayesian approaches to model calibration.