Thursday, January 27, 2011

online labor markets

If you do experiments in the social sciences, please, for the love of god, read this paper.

It doesn't make any sense to pay $15 per datapoint in a lab if it costs 15 cents on an online labor market. Plus, you don't have to worry about recruitment, scheduling, payment, actually going to the lab, etc. Plus, external validity is a heck of a lot better when anyone in the world can do your experiment, not just some very WEIRD Cal students, and when they are viewing it as a real job rather than a silly game some researchers are manipulating you with.

Since this isn't yet the standard, any experiment that uses Amazon's Mechanial Turk (MTurk) labor market, or anything similar, will be met with plenty of skepticism. Since I plan on doing my experiments on MTurk, I would like to avoid having to have this discussion a hundred times, which I can do by getting everyone else to do theirs on MTurk first. So do so, please =)

Keep in mind, however, that design considerations are different online. The overview paper above doesn't go into that too much (and the experimental economics field ignores these details, like how do you present information to the subjects and how do you write the instructions, etc, way too much in general).

No comments: