| To Register SMDM Homepage | |
|
|
|
||||
Purpose. Use Statistical Process Control Methods to identify when a clinical model needs to adapt to environmental variations in order to maintain its value. Methods. SPC was used to scrutinize the performance of the Score of Neonatal Physiology II model (SNAPII), a validated logistic regression model that predicts in-hospital mortality for neonates in the intensive care unit. The SNAPII model was applied to a database of 3437 newborns admitted to 7 neonatal units in New England from 1994 to 1996. To accommodate temporal analysis, cases were chronologically arranged and grouped into 14 sequential periods containing equal number of cases. The c-index (equivalent to area under the ROC curve) was used to quantify model performance. To identify periods in which model performance fell below SPC-determined limits, we graphed the exponentially weighted moving average (EWMA) for each period. The EWMA approach can manage non-normally distributed data such as the c-index and detect small sustained deviations in its trend. The same analysis (experiment #2) was repeated on the SNAPII model for different clinical scenarios. These scenarios were artificially constructed by varying the patient data. The analysis of experiment #3 applied a risk-adjusted SNAPII model that accounted for the variations made in experiment #2. Results. The C-indices resulting from applying the SNAPII model to the 14 periods were within the SPC limits of their overall mean, 0.88 (remained in statistical control); thereby reaffirming the model's validity. The model's performance shifted out of statistical control when applied to data containing unanticipated predictor-outcome relationships. The performance loss was partially recovered by risk-adjusting the model for the new relations. Conclusion. SPC framework can detect changes that could remain unnoticed using the current quality control measures, thereby helping to determine when and how a model may need to adapt to a new environment in order to remain useful. Such identification may improve understanding of how patients, diseases, and the environment interact in a dynamic clinical setting.
See more of Poster Session - Public Health; Methodological Advances
See more of The 26th Annual Meeting of the Society for Medical Decision Making (October 17-20, 2004)