COMPARING ANNOUNCED WITH UNANNOUNCED STANDARDIZED PATIENTS FOR ASSESSING DIAGNOSTIC ERROR

Purpose: To estimate the increase in difficulty when assessments of clinicians’ propensity to make contextual errors in diagnosis are performed by unannounced vs. announced standardized patients (SPs).

Method: We conducted a combined re-analysis of data from two previously published studies using the same actors, cases, and outcomes but varying in whether clinicians knew they were seeing SPs and in the level of expertise of the clinician. Participants in the original studies included 111 board-certified internists visited by unannounced SPs in their practices in the Midwest USA, and 59 4^th year medical students examining announced SPs in a clinical performance center at a Midwestern medical school. We compared likelihood of probing a biomedical red flag mentioned by the standardized patient, likelihood of probing a contextual red flag mentioned by the standardized patient, likelihood of writing an appropriate treatment plan.

Result: Attending physicians measured with USPs significantly underperformed 4^th year medical students measured with SPs in the probing of the biomedical red flag (adjusted OR=0.45 (0.30 to 0.67)),  in the probing of the contextual red flag (adjusted OR=0.66 (0.45 to 0.99) and in planning appropriate care (adjusted OR=0.43 (0.27 to 0.67)).

Conclusion: Assuming that attending physicians are in fact more competent than medical students, and at least as motivated to provide correct care for patients as medical students are to perform well in a no-stakes assessment, the outstanding difference between performance measures was the SP procedure. This reanalysis provides a lower bound estimate of the difference in difficulty: USPs are at least 1.5 to 2.3 times more difficult than equivalent SP cases. This study has the usual limitations of a secondary data analysis. Nevertheless, our findings suggest that traditional SP assessments, such as those used for licensure, may not be sufficiently sensitive to predict important lapses in how physicians make diagnostic decisions in actual practice. As other assessment methods, such as chart review or patient surveys, are also insensitive to overlooking information in the clinical encounter, further study and targeted use of unannounced standardized patients in practice may be warranted.