Comments on: Crowdsourcing an Ethical Dilemma

By: David

David — Sat, 18 Jul 2009 00:00:16 +0000

These are all horrible situations, but I would find it permissable to save the 1 person only in the switch dilemma, not the other two. Even if it is to save more lives, they would involve the active murder of an innocent to do so. If I were 1 of the patients in the surgeon scenario, I wouldn’t even want him to save me by killing the traveller. I would rather die than live with part of a murdered person’s body in me. On the other hand, with the footbridge dilemma, while I could not push the fat man off, throwing ones self off the bridge onto the track to stop the trolley from running over the 5 would be a TRULY brave thing to do.

By: Chris MacDonald

Chris MacDonald — Tue, 16 Jun 2009 21:26:39 +0000

Not to be picky, but is this really “crowdsourcing?” Is a survey — even a sophisticated one like this, really an instance of crowdsourcing? Doesn’t crowdsourcing involve getting some job done? The only way a survey could be taken as getting something done would be if the results were taken to be normative, or as implying the “right” answer to a question. But then Comment #8 above needs a serious answer.

By: johnfromukiah

johnfromukiah — Wed, 20 May 2009 01:56:13 +0000

I heard this same experiment on NPR’s Radio Lab and the same correlations were found. If I remember correctly, one conclusion made by the hosts was that it was perhaps the technology, the remote control if you will, of the lever in the first scenario that made it easier to say “yes” (implications for using drones over Pakistan?). However, after taking a bit of time to think about it, I think there is another subtle difference between the scenarios that can help explain the different choices made. In scenario A, there is a significant delay between the pulling the switch and the train reaching the man. Scenario B is less time, but still time between pushing the man and the death of the man. Scenario C is the least amount of time between your physical action and the death of the man. I would venture to say that if you shortened the distance in scenario A such that the man would instantly die when you pulled the lever, less people would choose “yes”. Moreover, I think that the reason why these time intervals matter is that despite the foregone conclusions of each scenario (that the man will die) logic states that the longer the interval between your choice and the death, the more likely something else may happen that will prevent the death (e.g. the man will untie himself from the tracks, the man’s friend will rescue him, the train will be able to stop).

By: Norman Creaney

Norman Creaney — Wed, 13 May 2009 13:40:36 +0000

@Lukas wrt monotonicity – the irrationality of individuals is a given – its the aggregate that matters.
Norman

By: Carey

Carey — Tue, 12 May 2009 19:49:34 +0000

Idiocy of the Crowds (#8), I think you’ve read far too much into this. I don’t see anywhere that states what should be done, or what’s true or false. You can’t have a fallacious argument if there is no argument at all. I see this as simply presenting some data, which may or may not be interesting to the reader.

By: Cezary

Cezary — Tue, 05 May 2009 23:50:36 +0000

You assume what is a rational decision too quickly. I imagine that for tested people it might be a problem of what other people would think of him taking any decision. Have you tested your turks for religious beliefs? Why would 30 people be better for me than 10? $30 is better than $10, but people? They behave differently, they might all destroy your house as very stressed survivors. Wouldn’t they be more likely to attribute their survival to some supernatural force and not the decisionmaker not being thankful at all? Wouldn’t they want after the survival help from my taxes from the government? There are many issues to analyze here.

By: Andy Eggers

Andy Eggers — Wed, 21 Jan 2009 22:41:58 +0000

Yeah, Lukas, I think on reflection I would seriously moderate what I said, not so much because I think the data is fine but because I think that even if some turkers were not taking the questions seriously you can still get some good estimates out of the data. For example, even if you don’t know exactly what the “bad guys” were doing (and thus your estimate of the frequency of yes’s in a particular scenario was biased), as long as you think they were doing the same thing in answering the different scenarios you could still get an unbiased estimate of the difference in the mean proportions between scenarios. (Although you would need to at least estimate how much of the data was bad.)

Anyway, I concede that good experiments could be done on turkers, but as with all research it’s hard to come up with good questions. Personally I’m not so interested in this one but I will keep an eye out for some.

By: Lukas

Lukas — Wed, 14 Jan 2009 17:41:58 +0000

Andy – I agree the lack of monotonicity is troubling but I disagree with your conclusion that “mturk can really only be used seriously for tasks that can be validated via ESP game-like checks.”

Each individual seems illogical, but the aggregate is a mononically increasing function in the number of people sacrificed, and there is a clear distinction between the three scenarios. So the aggregate looks surprisingly logical. This indicates that most workers weren’t clicking randomly.

Thinking about the problem for myself. If you asked me if I wanted to switch the trolley track to kill 5 people I’m not sure I would give the same answer each time. Do you think that you could remain completely internally consistent if presented with this question over and over with different values of N?

-Lukas

By: Andy Eggers

Andy Eggers — Wed, 14 Jan 2009 14:11:30 +0000

The lack of monotonicity is the most interesting and troubling thing here: in many cases it seems that the turkers were responding purely randomly, which makes sense since this was the fastest way to get paid. This makes me think that mturk can really only be used seriously for tasks that can be validated via ESP game-like checks.

By: Idiocy of the Crowds

Idiocy of the Crowds — Wed, 14 Jan 2009 02:31:19 +0000

Crowdsourcing ethical dilemmas is an incredibly dumb idea. The “wisdom of the crowds” has jumped the shark. Firstly, yes, you might get data on what people will do, but that still doesn’t answer what you should do, which is the whole point of ethics. Secondly, the whole experiment is logically flawed. It’s an appeal to popular opinion, which is a fallacy. Just because a crowd believes in something, doesn’t make whatever position they hold true.

By: Travis Kalanick

Travis Kalanick — Sun, 11 Jan 2009 03:56:57 +0000

Couple things:
1) For each hypothetical number of lives saved, can you provide the number of turker samples you got? For example, can you provide the number of turkers answered Scenario A for 10 lives saved?

2) This is a really cool post, but would be nice if either at the beginning or the end, you provided 3 or 4 bullets for key takeaways. . . would be easier for me to synthesize for discussions at cocktail parties :)

By: Mosh

Mosh — Thu, 08 Jan 2009 17:02:08 +0000

Luke, your last point about people having to answer all three questions at once makes me think there might be an Innocent Bystanders Threshold that’s been exceeded by the time people get to the last question. “Ooh, I already killed some people. I think that’s enough for today.” Did you guys ask the questions in the same order every time?

By: brendano

brendano — Thu, 08 Jan 2009 00:09:10 +0000

Bill: yeah that was a typo for this post.

By: lukas

lukas — Tue, 06 Jan 2009 18:30:59 +0000

Paul – there is definitely a very strong correlation. Our second chart tries to show this (I realize it’s a little confusing, but it actually lays out all the raw data in a graphical form). One worry I have is that the correlation may be artificially inflated by the fact that people had to answer all three questions at once.

By: Paul

Paul — Tue, 06 Jan 2009 17:52:34 +0000

Is there a correlation between a specific Turker’s answer on all three questions? Can you say that someone responding yes on A for a low number (i.e. more directly comparing the number of lives saved in each of the two outcomes) is more likely to answer yes on B? One expects a correlation…