EVALUATING THE POTENTIAL PERFORMANCE GAIN END-USERS COULD ACHIEVE IN DIFFERENTIATING TRANSIENT ISCHEMIC ATTACK (TIA) OR MILD STROKE FROM NON-ISCHEMIC MIMIC CONDITIONS THROUGH USE OF A CLINICAL DECISION SUPPORT RULE

Purpose: To evaluate the increase in predictive performance over the unassisted baseline performance.of end-users who use a clinical decision rule (CDR) to differentiate acute cerebrovascular syndrome (ACVS; i.e., TIA/mild stroke) from non-ischemic mimic conditions (e.g., migraine, seizure) achieved on historical patient cases.

Method: Emergency department referrals from 2014 to an outpatient TIA clinic were randomly sampled to construct a real world test data set containing 100 ACVS and 100 mimic cases.  Cases consisted of patient age, sex, blood pressure, and history of presenting illness—a free text description of the patient’s self-reported event.

A pre-test/post-test design was used to assess increases in predictive performance resulting from use of the CDR. Test cases were presented to three, first-year medical students.  Students were instructed to review the cases and decide on the basis of their unassisted clinical judgement the probability (0–100%) of each case being ACVS (≥ 50%).  Concurrently, students coded each case for the presence of clinical features relevant to the CDR.  After all the cases were completed, students’ coding of the clinical features was analyzed off-line with the CDR.  Students’ pre-test predictions were then compared to the off-line results they would have obtained if they had actually used the CDR during case evaluation (hypothetical post-test).  To facilitate comparisons, ROC measures were averaged across all students within the pre- and post-test assessments.

Result: Students’ mean unassisted performance measures were:  AUC 0.72, sensitivity 0.65, specificity 0.7, and accuracy 0.67.  Mean CDR performance measures were:  AUC 0.80, sensitivity 0.8, specificity 0.65, and accuracy 0.73.  Individually, each student would have experienced an increase in his or her predictive accuracy if the CDR had been used during case evaluation:  pre-test (0.69, 0.67, 0.67) vs post-test (0.74, 0.73, 0.71), respectively.  Moreover, CDR sensitivity measures were more consistent across students than during pre-test:  pre-test (0.64, 0.49, 0.81) vs post-test (0.84, 0.77, 0.79), respectively.

Conclusion: The results suggest that our CDR has the potential to improve end-users’ ability to differentiate ACVS from mimic conditions.  The ACVS CDR would standardize clinical decision making by reducing individual differences in end-users’ unassisted clinical assessments.  As such, our CDR could assist medical staff without advanced stroke training in the areas of patient triage and medical imaging requisition.