Purpose: With most medical treatments, there is uncertainty regarding the outcomes for individual patients. In order to make informed decisions, patients need to know the likelihood of the potential benefits and harms. A purpose of this study was to evaluate two different question formats for measuring quantitative knowledge of risks and benefits.
Methods: People aged 35 and older self-identified through an online advertisement as never having colon cancer. Respondents received a prepaid incentive. The survey contained items covering knowledge of colon cancer and screening tests, preferences for different testing methods, screening history, and demographics We used a split ballot design for the 4 quantitative knowledge questions. Half of the respondents were asked the questions in an open-ended format (for example: Out of every 100 people, about how many will get colon cancer?). The other half were asked the questions in a closed-ended format (Out of every 100 people, about how many will get colon cancer? Please choose the number that you think is closest to the correct answer: 2, 6, 14, 24, or 43). Respondents were also randomly assigned to complete the survey on the web or on paper. In order to compare the responses, a range was created around each closed-ended response, and those who answered the open-ended question were assigned to the corresponding range. We compared response rates and item difficulty of the two formats.
Results: We received 338 responses (overall response rate 89%), 167 open-ended and 171 closed-ended instruments. 32% of the respondents were male, 39% were age 50 or older (mean age = 49), 57% were college graduates. Respondents were willing and able to answer both question formats. For the open-ended questions, the number of missing responses per question ranged from 2-4. There were no missing data for the closed-ended questions. The number of missing responses did not vary by the mode of administration. There were no significant differences in the number of correct responses in the open and closed-ended formats. (chi square = .063 - .477)
Conclusions: The open-ended and closed-ended questions performed similarly in this sample. Given the increased precision afforded by open-ended questions, researchers developing knowledge tests should consider using them to assess knowledge of the likelihood of benefits and harms.
Candidate for the Lee B. Lusted Student Prize Competition