HOW WELL DO CLINICIANS INTERPRET CLINICAL EVIDENCE?

Purpose: The success of evidence-based medicine depends, in part, on how well clinicians are able to interpret scientific data and integrate it into their clinical practice. We investigated 41 primary care physicians' abilities to interpret scientific data by comparing their subjective comparisons of estimated outcomes of colorectal cancer screening strategies with those based on the actual data.
Methods: After a brief review of current guidelines, the physicians completed a multi-criteria analysis of 10 colorectal cancer screening options using the Analytic Hierarchy Process (AHP).  The analysis included blinded comparisons between quantitative outcome estimates for 3 criteria: effectiveness in preventing cancer, risk of serious screening-related side effects (perforation and major bleeding), and false positive rates. The results include normalized, ratio scales indicating how well the options were judged to meet each criterion. We assessed how well the physicians’ judgments agreed with the data presented by calculating difference scores using the formula O – E, where O represents the observed score derived from the physicians’ subjective data comparisons and E represents the expected score based on the data presented.
Results: The mean age of the study physicians was 44.7 years; 63% were male.  For 2 criteria, preventing cancer and risk of serious side effects, the physicians overvalued the best strategies; the mean difference scores were 0.26 and 0.04 respectively. In contrast, in terms of avoiding false positive screening tests, the best strategies were undervalued with a mean difference score of -0.15. For all 3 criteria, the subjective scores assigned to the best option(s) were the most variable with standard deviations equal to 0.15, 0.15, and 0.06 for prevent cancer, avoid side effects, and avoid false positives respectively.  Within each domain, there were statistically significant differences among the options in the mean difference scores, F = 119.2, 5.94, and 261.6, all p values < 0.001.
Conclusion: Primary care physicians’ subjective data interpretations of colorectal cancer screening outcome data varied and frequently did not accurately reflect the actual data upon which they were based. If confirmed, these findings suggest that the benefits of evidence-based medicine may be negated by variation in clinicians’ assessments of the evidence. If so, increased attention to training clinicians in data interpretation and synthesis would be indicated.