Course Level: Intermediate
Format Requirements: The format of this course combines lecture-style presentations with hands-on spreadsheet exercises to introduce and build on theoretical and practical aspects of model calibration. The initial third of the course reviews the conceptual foundations of disease modeling (briefly), situations where model calibration is useful, and the main phases of carrying out model calibration. The second third of the course utilizes hands-on spreadsheet exercises to walk participants through various calibration parameter search and evaluation approaches. The final third of the course introduces advanced topics and related spreadsheet demonstrations. Proficiency with Microsoft Excel is a required, and familiarity with Excel macros/Visual Basic for Applications (VBA) programming and/or Solver Add-in is preferred but not required. This course is taught at an intermediate level and is intended for those with experience or interest in programming or evaluating disease models.
Background: In the context of disease modeling, “calibration” refers to the systematic adjustment of model inputs such that the resulting model outputs better reflect setting-specific observed disease outcomes. This course encompasses theoretical and practical aspects of model calibration methodology, including: 1) Identifying calibration targets 2) Determining model inputs to be calibrated; 3) Selecting goodness of fit criteria; and 4) Implementing an appropriate parameter search approach. The International Society for Pharmacoeconomics and Operations Research (ISPOR) Task Force on Good Research Practices for Modeling Studies recommends model calibration whenever endpoint data are available and independent of the data used to initially inform input parameters. Model calibration is particularly useful when there are limited or nonexistent data on transition probabilities, but disease endpoint data are available. For example, setting-specific statistics on cervical cancer disease outcomes are often available, whereas there are virtually no data sources that can directly inform specific cervical disease progression inputs.
Description and Objectives: In the context of disease modeling, “calibration” refers to the systematic adjustment of model inputs such that the resulting model outputs better reflect setting-specific, observed disease outcomes. This course encompasses the main theoretical and practical aspects of model calibration methodology. Stylized practice models and realistic oncology disease models programmed in Microsoft Excel are utilized to give participants hands-on experience employing commonly-used and cutting edge model calibration techniques. The main objectives of the course are:
- To review modeling circumstances that particularly benefit from calibration;
- To introduce the main methodological phases of model calibration, specifically: 1) Identifying calibration inputs/outputs; 2) Determining goodness of fit criteria, such as windows-based targets, minimizing deviation/least-squares approaches, and likelihood-based functions; 3) Selecting and implementing parameter search algorithms, including manual adjustment of model parameters, random searches, and optimization techniques (linear programming and directed-search algorithms);
- To demonstrate how to implement these methods using interactive spreadsheet exercises;
- To highlight several advanced topics.
Specific advanced topics include:
- Comparisons of manually-intensive versus computationally-intensive parameter search strategies;
- Identification of and correction for bias introduced from calibrating longitudinal models to cross-sectional data;
- Probabilistic and deterministic uncertainty analysis for calibrated disease models.