METHODS FOR DEVELOPING AND EVALUATING PREDICTION MODELS FOR DYNAMIC DECISION-MAKING

Overview: Effective decision-making in settings of long-term follow-up relies on dynamic prediction rules that can accurately capture updated risks over time and guide medical decisions regarding monitoring and treatment. This course will cover methods for the development and evaluation of dynamic prediction rules that may be used for decision-making over time.

Background: Many medical decisions involve using accumulated information on patients under surveillance to predict transitions in future health status, such as progression of disease or advancement to death. At any given point in time, an individual’s longitudinal measures up to that time-point may be used to update the predicted risk of future adverse outcomes and guide medical decisions regarding monitoring and treatment. For example, high-risk individuals may be targeted for preventative strategies or aggressive treatments, whereas less frequent follow-up may be recommended for low-risk individuals. In this course, participants will learn flexible approaches for modeling dynamic prediction rules for risk of an adverse outcome at a future time using longitudinal trajectories up to the time of prediction, as well as summary measures for evaluating predictive performance that are appropriate for censored survival outcomes.

Format Requirements: The course will include a combination of didactic lectures and hands-on exercises that participants will work through using their own computers. Participants will be introduced to R packages that are currently available for evaluating the predictive accuracy of survival models. This segment will include hands-on training and demonstration of how to use these R packages for answering research questions. Real-data examples for analysis will be provided and the instructors will discuss implementation and interpretation. Data sets and files needed for the course will be distributed during the course session. This course is designed for those with basic understanding of statistics (e.g. methods for correlated data and survival analysis). Some experience with programming in R is preferred, but not required.

Description and Objectives:

Many medical decisions involve using accumulated information on patients under surveillance to predict transitions in future health status, such as progression of disease or advancement to death. At any given point in time, an individual’s longitudinal measures up to that time-point may be used to update the predicted risk of future adverse outcomes and guide medical decisions regarding monitoring and treatment. For example, high-risk individuals may be targeted for preventative strategies or aggressive treatments, whereas less frequent follow-up may be recommended for low-risk individuals. In this course, participants will learn: (1) Approaches for modeling dynamic prediction rules for risk of an adverse outcome at a future time using longitudinal trajectories up to the time of prediction, (2) Methods for evaluating predictive performance using summary measures that are appropriate for censored survival outcomes. While performance measures for evaluating prediction models for binary outcomes are well established, methods appropriate for survival outcomes are less well known. The methods we discuss are particularly important in that they allow for appropriate handling of censored outcomes commonly encountered with clinical event time data.

Methods will be illustrated using examples from HIV, end stage renal disease, and organ transplantation settings. The main objectives of this course are to overview modern statistical methods for the following:

• Model development:
  • Appropriately modeling both the longitudinal trajectory of predictors and their relationship with the survival outcome that can change over time
  • Comparison of recently proposed approaches 
• Model evaluation:
  • Measures of calibration and discrimination accuracy that are appropriate for survival outcomes
  • Methods for assessing the time-varying performance of a prediction model using predictive accuracy concepts of time-dependent sensitivity and specificity
  • Comparison of ROC curve based approaches to other approaches such as decision curve analysis and value of information analysis