MISSES VS. FALSE ALARMS IN HIGH-STAKES DECISIONS: QUANTIFYING BENEFITS–HARMS RATIOS FOR POLICY ANALYSIS IN MEDICAL DECISION MAKING

Purpose: Individual and institutional decision makers in the health care sector are—knowingly or unknowingly—faced with the inevitable tradeoff between setting a lax diagnostic threshold to avoid misses and setting a strict one to avoid false alarms. Often it is unclear whether the adopted tradeoff is aligned with the values of affected stakeholders. Unfortunately, decision makers are often unable or reluctant to reveal—or simply unaware of—the "benefits–harms ratio" (BHR) their decisions imply. Here I show how to use signal detection theory (SDT) to quantify a BHR and illustrate its application as policy analysis tool using several case studies.

Method(s): The BHR is calculated as the ratio of the decision maker's threshold (SDT's beta) and the cost-neutral threshold (i.e., simply minimizing the errors irrespective of whether they are misses or false alarms). A BHR represents the relative importance between the following two utility differences: (1) The harms of an incorrect "positive" decision, that is, the decrease in utility of falsely claiming the event (false alarm) instead of correctly rejecting the event (correct rejection) and (2) the benefits of a correct "positive" decision, that is, the increase in utility when correctly detecting the event (hit) instead of missing it (miss).

Result(s): Applying this approach to data from emergency medicine (ED) and HIV diagnostics reveals that BHRs vary substantially in unexpected or arguably undesirable ways. For example, based on the reported variation in decision thresholds of 28 ED physicians in a study of trauma triaging (Mohan et al., 2014 MDM), I can show that patients are subject to substantial variation in BHRs depending on which physician they happen to visit (ranging from 16 to 45 for the middle 90% of the distribution).

Conclusion(s): The BHR approach is a broadly applicable tool for policy analyses in medical decision making that allows quantifying decision makers' implied value judgments. It could be used to train decision makers and institutions by giving them feedback about their BHRs; alternatively, BHRs could be used to select among decision makers or methods. In this way, the BHR could support evaluating and possibly revising current health care practices (but also current practices in other domains, such as public safety, justice, business, environment, education, meteorology, military, and government).