Purpose: Subjective probability distributions for an uncertain quantity are often essential inputs in decision analyses and cost effectiveness analyses. However, eliciting such distributions from experts is often laborious and can be fraught with difficulties. We explored the practicality of the “trial roulette” method, which is a simple procedure for eliciting subjective probability distributions and has occasionally been used in eliciting clinical opinion. Rather than following the standard procedure of answering a series of questions about their subjective distributions, experts simply construct a distribution “on-line” by placing poker chips representing equal densities of the distribution into the “bins” of a pre-printed grid.

Methods: We report a series of comparisons based on three studies in which 121 participants provided over 700 subjective probability distributions. Three factors were varied in a series of experiments: (1) the number of chips (20 versus 33), (2) the number of bins used to represent a given range of values, and (3) the physical appearance of the bins (physical width was proportional versus not proportional to numerical width).

Results: The trial roulette method proved to be a highly reliable method of eliciting subjective distributions. For 141 test-retest comparisons of the same quantity assessed on two occasions, the median test-retest reliability for the densities assigned across 10 bins was r = .94. Varying the number of chips and the physical appearance of the bins had no systematic effect upon responses. However, increasing the number of bins covering a given range of values for the uncertain quantity has a strong tendency to increase the density assigned to that region. For 77% of 52 separate group-wise comparisons, a higher density was assigned to a given region when it was represented by 2 or 3 bins rather than a single bin. A lower density was assigned on only 4% of occasions.

Conclusions: This procedure represents a promising method of eliciting subjective probability distributions, which even participants with modest levels of statistical sophistication found easy to use. However, it is susceptible to the kinds of scale effects that beset other methods, and in the light of this we discuss recommendations for the best way to structure practical elicitation exercises.