OVERLAP AND THE EFFICIENCY OF DISCRETE-CHOICE EXPERIMENTS: EVIDENCE FROM A RANDOMIZED EXPERIMENT

Purpose: Experimental design (ED) for discrete-choice experiments (DCE) are often driven by statistical efficiency and ignore the impact on respondent efficiency. We tested for differences between two EDs, with and without overlap, as part of a large national-representative survey of adults’ preferences for participation in genetic research.

Method: We constructed two EDs for a DCE with six attributes of three levels each, one allowing for overlap by including all attribute-level pairs (i.e. AA, AB, AC, BA, BB, BC, CA, CB, CC) consisting of 81 choice tasks and a second that omitted overlapped pairs (e.g. AA, BB, CC) with 72 choice tasks. The D-optimality was lower for the overlap ED (66.6%), compared to the one without overlap (100%). Implemented in a nationally-representative sample of adults from the Knowledge Networks panel, respondents were randomized to receive a block of 9 tasks from either ED. A choice model was estimated via a conditional logistic regression, with differences between the results of the two EDs assessed via the likelihood ratio test, comparison rates of opt-out, and the Swait and Louviere test for scale.

Result: 1546 respondents participated, of which 22 not completing all tasks and 12 always choosing the same profile were excluded. Quotas were used to ensure balance across blocks and EDs, leading to 504 respondents for each ED. Respondents always choosing the opt-out was marginally lower in the overlap ED (p=0.40), which was more significant when account for any type of opt-out (p=0.11). Statistically, there was no difference between the two models in terms of parameter estimates (p=0.06) and no difference in scale was observed across the two designs (p=0.23). This said, the estimates from the design with overlap ED were 1.06 times higher (95% CI: 0.95, 1.18). Post-hoc D-efficiency was higher in the non-overlap ED (p<0.01), and was 1.21 times higher than the overlap model (95% CI: 1.15, 1.26), which was reflected in statistically lower standard errors (p=0.01).

Conclusion: The ED with no overlap was more statistically efficient, but more respondents choose the opt-out and estimated parameters tended to be lower – both indicative poorer respondent efficiency. Assuming within group homogeneity is a limitation in our analysis, and subsequent analysis using mixed-logit and hierarchical-Bayes estimation revealed a statistically-significant difference between the two EDs.