A NOVEL TOOL TO EVALUATE THE ACCURACY OF PREDICTING SURVIVAL IN CYSTIC FIBROSIS

Purpose: To present a novel tool for evaluating the time-dependent accuracy of risk prediction models in cystic fibrosis (CF) that are to be used to guide decisions and prioritize limited donor lungs to patients who are at the greatest risk of death without transplantation.

Methods: We extend standard concepts of sensitivity and specificity to survival outcomes and demonstrate an approach to answer the following questions: If a risk prediction model were used to guide clinical decisions, then what is the time-dependent prognostic accuracy of such a model, and how frequently should risk predictions be updated through the use of time-varying risk scores?

   For any time t, among the patients who are still alive, cases are defined as those who die at time t and controls as those who survive through time t. The sensitivity and specificity at t are the error rates in classifying subjects at that time, summarized using the area under the time-dependent Receiver Operating Characteristic (ROC) curve, or AUC(t), interpreted as the probability that given a random case who dies at time t and a random control who survives beyond t, the case has a higher marker value than the control. We estimate AUC(t) using a nonparametric rank-based approach.

   Using this tool, we compare the performance of forced expiratory volume in one second (FEV₁), a standard measure of pulmonary function used in practice for recommending lung transplantation, to risk scores derived from previously proposed multivariate Cox regression models combining FEV₁ with other clinical factors. 

Results: Applying the proposed method to CF registry data, we show that several multivariate models proposed in the literature perform no better than FEV₁ alone. Not surprisingly, the performance of a baseline FEV₁ measurement declines over time, from AUC=0.9 at baseline to AUC=0.6 20 years later. In contrast, an annually updated FEV₁ measurement consistently maintains an AUC of 0.9 over time. Furthermore, FEV₁ updated every 2 years has similar performance to annually updated FEV₁.

Conclusions: It is imperative to continue to develop models that more accurately predict survival for individuals with CF. Our proposed evaluation tool that can serve as the basis for developing better clinical predictors and may also help guide clinical practice with regards to the optimal timing and frequency of updating patient information.