Purpose: Prolonged QT interval is a relatively rare (prevalence about 1 in 6,000) heart condition that can be associated with long QT syndrome and that can have a fatal interaction with some QT-interval elongating medications. A study was undertaken to verify that an expert system under development was correctly implemented and that the system was effective in changing outcomes. This report focuses on the verification portion of the analysis.
Method: The operational rule was developed using experts within the Mayo Health Care system and implemented using the GE Blaze rule-based expert system (RBES). The algorithm was ported to R (a simpler programming environment) to permit testing against historical data. Data from cases run through the Blaze system while in "silent mode" (behind the scenes without notifications to physicians) were used to verify that the R algorithm and the Blaze algorithm agreed. Data from 22 months of ECG measurements (n=126,283, 97.923 unique patients) were collected and the ECG lab calls for "Semi-urgent findings" related to prolonged QT were compared with the predicted RBES calls using the R based version of the algorithm. Sensitivity, specificity, positive predictive and negative predictive values were computed.
Result: In addition to verifying that the algorithm was implemented correctly, we discovered that the expert system rule was not identical to the rule being used in the ECG lab. We found 421 patients who should have identified as having "prolonged QT interval" were not so identified. The difference was identified as being primarily due to the lab not reporting events in patients with pacemakers. How to handle the presence of a pacemaker had not been documented in the instructions to the lab. When the analysis revealed this discrepancy the experts were able to change the lab's rules for dealing with patients with pacemakers. Adjusting for this, the sensitivity was 0.825, specificity was 0.998, PPV was .455 and NPV was essentially 1 (as expected for conditions with a low prevalence rate).
Conclusion: The process of creating a rule-based expert system with sound verification and validation analysis can improve processes by providing rigorous and documented reviews of those processes. A thorough evaluation of clinical decision support systems may enable a more seamless implementation.
Candidate for the Lee B. Lusted Student Prize Competition