Xin He, BS, Jonathan M. Links, PhD, Benjamin M. W. Tsui, PhD, and Eric C. Frey, PhD. Johns Hopkins Medical Institutions, Baltimore, MD
Purpose: ROC analysis is well established in the evaluation of binary task performances. However, the evaluation and optimization of some medical systems involve distinguishing among three diagnostic alternatives. We have developed a practical, though not fully general, three-class ROC analysis method based on decision theory. Methods: Four decision-theoretic optimization strategies were investigated, including the maximum utility criterion, the maximum-correctness criterion, the maximum likelihood criterion, and the Neyman-Pearson criterion. Results: All optimization strategies lead to the use of the ideal observer as the optimal observer. Ideal observer uses the likelihood ratios as the optimal decision variables. Both Neyman-Pearson and maximum utility criterion lead to a decision model that has a 2D decision plane and a decision structure with 5 degrees-of-freedom. However, this model does not provide a tractable way of determining the decision structure on the decision plane. We made the assumption that wrong decisions have equal utilities under the same hypothesis for the maximum utility criterion, and the assumption that wrong decisions have equal Lagrange multipliers under the same hypothesis for the Neyman-Pearson criterion. Then, using a log transformation, both criteria lead to a decision model that has a 2D decision plane and a decision structure with 2 degrees-of-freedom. Moving the decision structure over the 2D decision plane and computing the true class 1 fraction (T1F), true class 2 fraction (T2F), and true class 3 fraction (T3F) defines a 3-class ROC surface. The volume under the surface (VUS) serves as a figure-of-merit for the three-class task performance. The maximum-correctness criterion also leads to the same 2D decision plane and the same ROC surface without any assumptions while the maximum likelihood criterion results in an operating point on this ROC surface. Conclusions: All four optimization strategies, when combined with a single assumption, lead to the same optimal observer and the same three-class ROC surface. Thus the proposed 3-class ROC analysis method is meaningful according to all four decision strategies.
See more of Poster Session II
See more of The 27th Annual Meeting of the Society for Medical Decision Making (October 21-24, 2005)