PS 3-17 ARE MECHANICAL TURK WORKER SAMPLES REPRESENTATIVE OF HEALTH STATUS AND HEALTH BEHAVIORS IN THE U.S.?

Purpose: Amazon’s Mechanical Turk (MTurk) is a crowdsourcing platform that shows promise as an effective participant pool for surveys with quick turnaround time and low cost. MTurk is already frequently used to pilot health surveys and experiments in the field of decision psychology. The purpose of this study was to determine representativeness of the MTurk population with regards to general health status, health behaviors, health care access, and mental health in the U.S.

Methods: Two identical surveys, one amongst regular MTurk workers (“workers”) and one amongst “masters”, a higher paid subset of workers, were launched. The survey asked questions about demographics, health behavior, access, and physical and mental health status. Chi-squared tests were used to compare workers’ versus masters’ responses for categorical variables. Continuous variables were examined using Welch’s t-test and the Kolmogorov-Smirnov test for distributions. Workers’ responses were compared to national samples from the 2014 Behavioral Risk Factor Surveillance System and the 2012 National Health Interview Survey using Poisson regression, controlling for age.

Results: No statistically significant differences were detected between workers (n = 592) and masters (n = 393). In general, Mturk users were younger (35.6 years, SD = 10.8) and more highly educated than the national population. After controlling for age, Mturk users were significantly less likely to be vaccinated for flu (RR = 0.608, 95% CI: 0.601 – 0.616), less likely to smoke (0.729, 95% CI: 0.725 – 0.734), less likely to have asthma (0.667, 95% CI: 0.654 – 0.680), less likely to self-report being in excellent health (0.868, 95% CI: 0.868 – 0.870), and over twice as likely to screen positive for depression (2.28, 95% CI:  1.75 – 2.97). Mturk users were also slightly less likely to have participated in physical activity (0.983, 95% CI: 0.979 – 0.987) or have health insurance (0.950, 95% CI: 0.946 – 0.954) relative to national samples.

Conclusion: While Mturk may be an expedient means to recruit survey respondents, its workers are not a generalizable population with regards to health status and health behaviors. Sample weights may need to be employed in data analyses of MTurk surveys to ensure representativeness. In particular, our findings raise questions about the validity of MTurk surveys that relate to older adult populations, which are not prevalent among Mturk workers.