The CrowdFlower Blog

I never really understood the need for photo moderation until Chris and I built our first webapp together. This webapp was called FaceStat, a website where you could upload photos of yourself and get “feedback” from other people on things like whether or not you looked friendly and if you could win a fight with a medium-sized dog. It was a surprisingly popular, extremely viral website with horrible retention and no prayer of a viable business model.

At first we just let people upload whatever photos they wanted and comment however they wanted. Not surprisingly, things quickly got out of hand. Hours after we released the website we started seeing extremely disturbing uploads. We tried to use a user-moderation system but there were lots of false positives and it pained me to subject unsuspecting users to some of the things people uploaded. We tried automated systems but they were extremely unreliable.

This experience caused us to build CrowdSifter, a content moderation crowdsourcing tool, where our FaceStat app was the first customer. It worked really well and we got a few early customers. But our customers seemed interested in lots of tasks besides content moderation, so we decided to build a general purpose crowdsourcing system that we called CrowdFlower. Eventually CrowdSifter fell into disrepair and we shut it down and moved the existing customers over to our general crowdsourcing platform. CrowdFlower became so popular that we renamed our company CrowdFlower and we got in the business of helping companies get very high quality work done through crowdsourcing. We signed up over 50 channels, over 3 million contributors and hundreds of name-brand customers. We built out an enterprise sales team and learned how to deal with procurement departments. We even did some work for the US Government.

But as our engagement sizes rose, our sales costs rose too, and we stopped being able to service anyone except customers with the largest crowdsourcing jobs. Which in some ways completely contradicts the promise of crowdsourcing – a super-flexible workforce. We have a self-service site with some happy customers, but the learning curve is steep and while we’re working on improvements, I’m not sure if it’s possible to make a general-purpose crowdsourcing tool that’s also easy for everyone to use.

So we decided to build specific self-service products that would be easy to use, reliably fast and high-quality. I wanted to start with an application that we had lots of experience with and one where we felt like we really understood our customer’s needs. So I chose content moderation. The customers have expanded in the past few years with iPhone developers getting lots and lots of user generated images and living in constant fear of apple shutting them down for objectionable content. I’ve seen several indie app developers sign up for our beta and in particular I hope that this is a great product for them.

We plan on releasing several of these self-service applications over the next few months. If there are any crowdsourcing applications you would especially like to see, please let me know.

TL;DR We’ve offered crowdsourced photo moderation as a service for years, but now you can buy it with clear pricing, no sign-up fee and without ever talking to a sales person. And you should! Go to crowdflower.com/rtfm.

On Monday we conducted a sentiment analysis of futurist Ray Kurzweil’s SXSW 2012 keynote address, based on Tweets about his talk sent out during and immediately after his talk. In roughly two-three hours, our crowdsource-powered workforce analyzed 1,188 such Tweets, determining the sub-topic of each Tweet and its general sentiment (negative, positive, or neutral). While Kurzweil spoke to a capacity crowd of thousands, the reaction (according to our sentiment analysis) was decidedly mixed.

Here’s the results:

These answers were selected by our contributors from a number of options we put to them, including:

It’s about expanding our intelligence or IQ
It’s about Ray Kurzweil’s current projects
It’s about the singularity
It’s about artificial intelligence
It’s about life extension
It’s about Ray Kurzweil’s thoughts and predictions on the future
It’s about virtual reality
It’s about Transhumanism
It’s about nanotechnology
It’s about leaving his SXSW talk
It’s about SOMETHING ELSE (Other)

34% of the Tweets fell into the Other category, 34% were about AI, 10% were related to Kurzweil’s predictions of the future and current projects, and 2% about virtual reality. Interestingly, though Kurzweil’s talk was entitled “Expanding Our Intelligence Without Limit”, only a fraction of Tweets were related to that theme. We saw a similar pattern with Rainn Wilson’s SXSW speech, which was meant to promote his new site SoulPancake, but which also garnered minimal Tweets.

Having selected the sub-topic of each Tweet, our contributors then judged the sentiment — in other words, the author’s emotional tone regarding these Kurzweil topics. Notably, 38% were neutral, while 32% were positive, and 30% negative:

It’s interesting that both Kurzweil’s keynote and Rainn Wilson’s keynote drew such a significant negative reaction, despite the popularity of both celebrities in their respective fields. Perhaps if both speakers directly responded to the Twitter back channel throughout their talk (which typically happens in many SXSW panels and presentations), the overall response might have been more positive. It’s certainly worth considering for future SXSW keynotes.

How did we get this data? As we explained in a post last year, CrowdFlower’s crowdsource-driven sentiment analyses are fairly unique, in that they aggregate human judgements of Tweets, as opposed to an automated, computerized sampling, which is more likely to miss nuances of expression.

Again, check out our sentiment analysis of Rainn Wilson’s SXSW 2012 talk using the same methodology. Speaking of Tweets, don’t miss our top twelve Tweets from the SXSW talk of CrowdFlower founder Lukas Biewald.

Thanks to CrowdFlower’s Josh Eveleth for all the help putting this project together!