ARE MECHANICAL TURK WORKER SAMPLES REPRESENTATIVE OF HEALTH STATUS AND HEALTH BEHAVIORS IN THE U.S.?
Methods: Two identical surveys, one amongst regular MTurk workers (“workers”) and one amongst “masters”, a higher paid subset of workers, were launched. The survey asked questions about demographics, health behavior, access, and physical and mental health status. Chi-squared tests were used to compare workers’ versus masters’ responses for categorical variables. Continuous variables were examined using Welch’s t-test and the Kolmogorov-Smirnov test for distributions. Workers’ responses were compared to national samples from the 2014 Behavioral Risk Factor Surveillance System and the 2012 National Health Interview Survey using Poisson regression, controlling for age.
Results: No statistically significant differences were detected between workers (n = 592) and masters (n = 393). In general, Mturk users were younger (35.6 years, SD = 10.8) and more highly educated than the national population. After controlling for age, Mturk users were significantly less likely to be vaccinated for flu (RR = 0.608, 95% CI: 0.601 – 0.616), less likely to smoke (0.729, 95% CI: 0.725 – 0.734), less likely to have asthma (0.667, 95% CI: 0.654 – 0.680), less likely to self-report being in excellent health (0.868, 95% CI: 0.868 – 0.870), and over twice as likely to screen positive for depression (2.28, 95% CI: 1.75 – 2.97). Mturk users were also slightly less likely to have participated in physical activity (0.983, 95% CI: 0.979 – 0.987) or have health insurance (0.950, 95% CI: 0.946 – 0.954) relative to national samples.
Conclusion: While Mturk may be an expedient means to recruit survey respondents, its workers are not a generalizable population with regards to health status and health behaviors. Sample weights may need to be employed in data analyses of MTurk surveys to ensure representativeness. In particular, our findings raise questions about the validity of MTurk surveys that relate to older adult populations, which are not prevalent among Mturk workers.