How do photos uploaded to social networking sites reflect back on you? We scraped the profiles of about one hundred people on a social network and had Turkers guess those people’s traits – including age, ethnicity, intelligence, political affiliation, and intoxication. To be fair, we included pictures of ourselves in the batch.

You can see the photos ordered on three axes here:

More details after the jump:

(more…)

As anyone who follows political races knows, different sources can report the same event in very different ways. We took nearly six thousand recent articles over the past month about Clinton and Obama and sent them to Mechanical Turk to be classified as favorable or unfavorable for the respective candidates. We scraped the articles from Google News restricted to several sources, and threw in front page headlines from Digg.

Here is the graph for favorability scores, aggregated by source. We found that Digg was far and away the most favorable for Obama.

The next graph tracks overall news favorability by date. To provide some context, we compared it with the change in Obama stock on the Intrade prediction market.

More details after the jump:

(more…)

Martin Wattenberg at IBM Research took our color names data and made a cool new cloud view:

Instead of plotting each individual color name like in the original, he grouped together identical names, took an average position, and sized the word by frequency. That’s why the more common names like “red” and “green” are large. This really helps readability (and, I’ll admit, the black background works a bit better :))

Thanks to Martin for sending this on!

I just packaged and released the data set for our color names experiment. It has 10,000 color/label pairs.

This is the download link. Read on for more details:

(more…)

What would you call these colors?

We showed thousands of random colors like this to people on Mechanical Turk and asked what they would call them. Here’s what they said:

The above picture contains about 1,300 colors and the names for them that Turkers gave.  Each is printed in its color and positioned on a color wheel.  Just looking around, there sure seem to be different regions for different names.  But there are also rich sets of modifiers (”light”, “dark”, “sea”), multiword names (”army green”), and fun obscure ones (”cerulean”). To help look at all this, we also made a color label explorer, so you can search for different terms and see different parts of the space. If the link doesn’t work for you, here are a few examples:

This study is basically the same design as the famous World Color Survey, where anthropologists showed color patches to speakers of many different languages and asked for names, to test the universality of language.  Of course, we have mostly native English speakers. However, we can get much more data.  (The above picture and links use only a small percentage of all the colors and names we collected.)  There’s tons more that can be done. Want to make a better visualizer?  Statistical analysis of colors to name terms?  Let us know and we should be able to get this data set online.

UPDATE 3/18: I posted the data set.

-Brendan

Human annotators are great at providing basic information about images. We were wondering if we could find something interesting about magazine covers. Stumbling upon 2800 Sports Illustrated cover images going back to 1954, we sent them to Mechanical Turk, asking people to identify the race and gender of the person featured (if any), and what sport was depicted. There are lots of interesting things in this data; this post will touch on just a few we’ve had time to whip together some graphs for.

Here is a historical graph of the frequency of how often people of different races appear on the cover of Sports Illustrated. The story is simple and striking:

Next: which sports get featured on the cover? Here’s a chart for several sports over that same time.

It might be possible to find links between the careers of famous athletes and rises and falls their sports’ popularity; for example, boxing peaks in the 70’s (Muhammad Ali?), basketball peaks in the 90’s (Michael Jordan?) and golf bounces back in the 90’s after a long decline (Tiger Woods?).

Many other sports appear in the data, too; for this chart, we made sure to pick the three most common, and a few other particularly interesting ones. Percentages don’t add up to 100% because we didn’t plot all the other sports, including things like horse racing which used to be much more popular. If you’re really curious, here’s the full chart of all sports we asked about, including many of the smaller ones.

-Brendan

The first time I used Amazon’s Mechanical Turk it was at a search engine startup, Powerset, and I used it to compare the quality of a few versions of our early internal algorithm with Yahoo and Google. We were thinking we would have to hire a team of people that would spend their entire day comparing the quality of results.

As an experiment, I set up a task with no quality control, put in about fifty bucks and let it run overnight. The data that came back was noisy but I was able to find meaningful differences between the search engines. Completely on my own. I didn’t have to get approval to hire people, put my experimental design through a committee and wait a month for the results to come back. I could design the experiment empirically, doing meta experiments on the data collection process itself.

Back when I was thinking about what machine learning papers to write at Stanford, the conversation always hinged on what kind of data sets were available. We’d go research what data was out there and then figure out what we wanted to do. We’d spend a ton of time wrangling data designed for one purpose into another. I think it’s the same in lots of disciplines that use data.

Here at Dolores Labs, we’ve built tools and processes to quickly and efficiently collect lots of data on Mechanical Turk and other places. I hope that this blog gives us a chance to play with our technology. Back when I made my first AMT jobs, I thought about all the crazy experiments I wanted to run. Overnight, could you figure out what airline carrier was the cheapest? Could you find the exact threshold where what most people call “red” becomes what most people call “orange”? Could you quantify the difference in sentiment between FOX news and NPR?

When I was in college, I had an art teacher who made everyone draw twenty pictures a day. I hope these experiments are like those pictures. Sloppy and fun and occasionally brilliant.

We’ve been brainstorming experiments that we’d like to run, but if there’s any data set that you’d like to suggest send us an email. Maybe we can make this deal: if you have a cool idea, we’ll collect the data for you, and you guest post a short analysis.

Our first experiments will be posted shortly, and many more to come. I hope you enjoy em!

-Lukas