Comments on: Wisdom of small crowds, part 1: how to aggregate Turker judgments for classification (the threshold calibration trick)

By: brendano

brendano — Fri, 15 Aug 2008 18:28:18 +0000

Bob - yes, indeed some of the high-disagreement cases are genuinely hard. Not all of them though. I’ve only looked at this anecdotally.

By: Bob Carpenter

Bob Carpenter — Thu, 14 Aug 2008 22:27:02 +0000

I’d expect that blending in an estimate of the accuracy of the users would help separate out the confident assignments from others. This might not work in this setting, though, where annotators only have a go at very few data points. Also, estimating the prevalence of a given category can also provide helpful information.

Have you looked at uncertain or borderline cases for which there is a lot of disagreement to see if they are in fact hard cases?

By: brendano

brendano — Tue, 17 Jun 2008 21:01:22 +0000

Logan: It’s all about the R! I wrote the routines myself for the separation dotplot and the confusion barplot. I’ve never seen anything quite like the confusion barplot before; I’d be curious if anyone else has used it. (It would need modifications for problems where the class distribution isn’t as balanced; for example, where true negatives dominate, just ditch the TN bars. I have a bunch of plots here where I tried all possible stacking orders; it’s fun, they emphasize different ratios among the confusion categories.) The code is currently at http://github.com/brendano/dlanalysis but it’s rather tangled up with other things.

I really recommend the ROCR package (in R) as a well-done, reusable package for all the standard confusion plots, e.g. P/R and ROC, expected costs, etc etc. I was thinking of porting my plotting routines into their system, eventually…

By: Logan

Logan — Tue, 17 Jun 2008 18:24:49 +0000

Great explanation. What did you use to make the red and blue confusion matrix plot? Is that in an R package? Nice way to visualize it.

By: Paul (from Belgium)

Paul (from Belgium) — Tue, 17 Jun 2008 09:50:28 +0000

Hey Luke and the others,
Now I get the service that your company is providing on top of Amazon Turk, and it’s quite cool I must say!