The CrowdFlower Blog

One of the most important things we do at Dolores Labs is track the reputation of the workers on Amazon Mechanical Turk, so we know who is trustworthy and who is giving us questionable data.

But how do the Turkers (the people doing the work on AMT) know which requesters are trustworthy? In some ways it’s even more important for Turkers to know who is trustworthy, because requesters are allowed to refuse payment for a job with no recourse.

At Dolores Labs, we aspire to be as fair as possible, and only refuse payment when a Turker is giving us completely worthless results. But since our system is automated and does a large volume of tasks, we’ve surely made some mistakes.

Turkers can complain on message boards about bad requesters, but we’ve always felt it would be good for Turkers and the AMT marketplace to be have better information about requester’s reputation. So it was a pleasure to help my college friend Lilly Irani and her colleague Six Silberman to build Turkopticon, a Firefox plugin that lets Turkers report bad requesters.

Anyone can download the Turkopticon plugin, and they will be able to see all complaints that have been filed against a requester embedded inside the AMT interface. I would encourage all Turkers to download the plugin and help make Mechanical Turk a more transparent marketplace!

We helped Lilly and Six collect the seed data on which requesters were good and bad by creating a Turk task for them. We asked three questions, based on the complaints that we most often see: “How fair has this requester been in approving or rejecting your work?”, “How promptly has this requester approved your work and paid you?”, and “How well has this requester paid for the amount of time their HITs take?”.

It’s nice to see that the majority of requesters were reviewed positively:

We didn’t solicit data on ourselves in the initial set for fear of a conflict of interest. Happily, since Turkopticon was released Dolores Labs has only gotten good reviews. We would love to hear comments from any Turkers who read this blog for other ways that we could improve the marketplace and have a more open dialogue.

AMT is a labor market, and markets can’t function without trust between buyer and seller. Another online marketplace, eBay, has a similar trust problem — how do you know whether a seller will reliably send you what you just purchased? EBay has developed an important reputation and feedback system for sellers that greatly improves transparency. One of the amazing things about AMT is that it’s worked at all without such a system. We’re happy to help improve this situation.

By the way, Lilly has a very different perspective on Mechanical Turk than we do. Her blog is worthwhile reading.