Background: The everyday work of physicians includes the accurate diagnosis of illnesses; if these judgments are inaccurate they have real consequences for patients’ health. To evaluate physicians’ judgment accuracy, a so-called bare-bones meta-analysis is frequently applied. However, with a bare-bones meta-analysis there is a danger of 75% that the true judgment accuracy will be underestimated and the data heterogeneity overestimated, leading to false identification of moderator variables (see Wittmann, 1988, 2009). To show that data analysis without a psychometric correction erroneously leads to the conclusion that medical decision making is inaccurate, we took medical lens-model studies as an example. Within the lens-model framework, it is unique to evaluate judgments against a criterion (e.g. actual illness) and to break them down by an equation to reveal the underlying accuracy factors (Tucker, 1964). Are environmental (e.g. diagnostic instrument) or personal (e.g. physicians’ judgment ability) factors responsible for accurate medical judgments? It is important to know these factors and to properly inform and support physicians in improving their judgments. To do so, it is possible to construct expert models that build on these factors to improve and support physicians’ judgments (Goldberg, 1970).
Methods: To evaluate a) the overall judgment accuracy in medical science in the lens-model framework and b) to reveal the underlying accuracy factors and c) the success of existing expert models we applied a psychometric meta-analysis approach (Hunter & Schmidt, 2004) to correct the data base for possible artefacts.
Results: Compared to a bare-bones meta-analysis our psychometric meta-analysis clearly reduced data heterogeneity. Judgment achievement across tasks reached a high level (r = .53; varcorr = .00; N = 258; k = 10). The underlying environmental (r = .67; varcorr = .00) and personal (r = .96; varcorr = .00) factors were both important, revealing that medical tasks can be judged well and physicians actually judge very consistently. The bare-bones meta-analysis revealed only a moderate judgment accuracy (r = .40), falsely implying that physicians’ accuracy is possibly biased. The psychometric meta-analysis, as a better measurement, reveals physicians’ true judgment accuracy and increases the success of expert models due to artefact correction.
Conclusion: Our example analysis of lens-model studies highlights the need for psychometric analyses to correct the data base for artefacts before the accuracy or bias of physicians’ judgment and decision-making can be evaluated properly.
See more of: The 32nd Annual Meeting of the Society for Medical Decision Making