PS2-18
SIX TESTS FOR ASSESING THE VALIDITY AND RELIABILITY OF A DISCRETE-CHOICE EXPERIMENT
Methods: Following rigorous development and best-practice guidance, a paired-comparison DCE with 6 attributes was implemented to assess treatment preference of patients with type 2 diabetes in a nationally representative sample in the US. A blocked D-efficient Bayesian experimental design using priors from a pilot survey was developed. Results were analyzed using mixed-logit and latent class models. We assessed lexicographic preferences by examining frequency of participants that traded based on one attribute only and non-attendance by examining preference heterogeneity in latent class models. We assessed parallel-forms reliability through choices across three survey versions and test-retest reliability through choices for a repeat choice task. We assessed predictive validity by predicting choices for a holdout task and external validity by comparing results to published preference studies.
Results: 552 participants (response rate 66%) completed the DCE. The D-efficient Bayesian design lowered but did not eliminate lexicographic preferences. In this study, 15% of participants displayed lexicographic preferences vs. 41% during the pilot test which used an orthogonal design (p=0.002). Two latent preference classes were identified. Participant in class 2 (68.2% of participants) assigned almost no importance to cost (p=0.72), indicating the presence of attribute non-attendance in this group. The three survey versions displayed parallel-forms reliability; observed choices for the holdout task (p=0.15) and estimated choice models (p>0.1 for each attribute comparison) did not differ across versions. Test-retest reliability was higher than expected (p<0.001) with 76% of participants answering the repeat task the same twice (54% expected). The estimated choice model accurately predicted choices for the holdout task; no differences were found between predicted and observed choices (p=0.28). The standardized attribute importance scores of this study fell within the range of results published in DCEs, suggesting that these study results have external validity.
Conclusion: This study met five out of six tests to assess validity and reliability of a DCE. Due to the complexity of stated-preference methods, studies will not always meet all six test but they might help researchers gain greater insights into the results of their preference studies.