WHEN, WHY, AND FOR WHOM ARE NATURAL FREQUENCY FORMATS MOST EFFECTIVE? A META-ANALYSIS

Purpose: Medical professionals and the public are poor at solving Bayesian inference problems when presented in the form of conditional probabilities (e.g., determining the probability that an individual has a disease given that they tested positive).  When the same information is presented as natural frequencies, performance is improved.  The current meta-analysis sought to reconcile twenty years of research on natural frequencies, to clarify what is a natural frequency and to identify when, why, and for whom the format is most effective. 

Method(s): To identify papers that compared conditional probability and natural frequency Bayesian reasoning tasks, we conducted a systematic review across three major scientific and medical databases: Ovid(psych), Web of Science, and Pubmed.  Cited reference searches were conducted on key papers and we requested relevant or unpublished papers from the JDM mailing list that we may have otherwise missed in our systematic search.  Thirty relevant papers were identified from which 90 effects were subsequently analysed.

Result(s): A broad range of potential moderators were coded.  These included moderators related to individual characteristics (e.g., numeracy, education), problem representation (e.g., congruence between answer and problem format, use of visual aids, menu), and methodology (e.g., use of incentives, scoring protocol).  Results revealed the expected natural frequency facilitation effect: on average, performance was enhanced for natural frequency formats when compared to conditional probabilities.  A number of variables moderated the effect: e.g., the size of the effect was reduced when both formats used visualisations, short menu versions, and when the problem format matched the answer format for frequency versions.   

Conclusion(s): The meta-analysis supports the consensus that performance on Bayesian inference tasks is facilitated by natural frequency formats when compared to conditional probability formats.  Despite this result, a non-trivial amount of people continue to have difficulty with Bayesian inference problems even when presented as natural frequencies.  We discuss gaps in the literature, suggest future research directions, and suggest methodological approaches to capture further information about how people acquire and update information.  We advise how to improve the communication of medical test statistics to professionals and the public.