Purpose: There is ample evidence supporting early and consistent use of disease-modifying drugs, such as methotrexate, for managing rheumatoid arthritis (RA). Yet, patients are reluctant to start treatment due to potential side effects. For those who are considering methotrexate, we have developed an interactive web-based decision aid called the ANSWER. The current study aimed to: 1) assess the usability of the ANSWER prototype; and 2) identify important components of usability testing from the patient’s perspective.
Method: ANSWER consists of: 1) six animated patient stories and narrated information on evidence surrounding methotrexate; and 2) interactive questionnaires to clarify patients’ treatment preferences. Eligible participants for the usability test were patients with RA. We used the concurrent think-aloud method, whereby participants were asked to verbalise their thoughts while using the ANSWER. Sessions were audiotaped and field notes taken. Participants completed the System Usability Scale (SUS) to assess the overall usability of the decision aid (range=0-100; higher=more user friendly). We conducted content analysis to identify major themes to understand the user experience.
Result: 15 patients participated in the usability testing; the majority were age 50 or over and university/college graduates (n=8, 53.4%). The median disease duration was 5 years (IRQ=0.83-10). Participants took an average of 56.1 minutes (SD=34.8) to complete the program. The mean SUS score was 81.2 (SD=13.5), indicating high usability. Content analysis of audiotapes and field notes revealed three themes that participants focused on while testing the ANSWER: 1) user engagement (i.e., the relevance of the decision aid design and content to the user); 2) information quality (i.e., clarity and credibility of information); and 3) user-tool interaction (i.e., consistency of the design; ease of use). Across these themes, participants commented extensively on the overall integration of the content and navigation (e.g., challenges of moving from one task to the next). We made revisions to the prototype based on the findings.
Conclusion: Although the SUS score indicated high usability, findings from the think-aloud sessions highlighted additional areas where further modifications were needed for the online ANSWER decision aid. Our results highlight the importance of direct observation methods in usability testing. With an increasing number of online and mobile decision aids being developed, further research to advance the methodology of usability testing is warranted.
See more of: The 34th Annual Meeting of the Society for Medical Decision Making