Sorites Paradox on Mechanical Turk

Sorites Paradox is something like this: Is this tile red? Sure. What about this tile ? No, it looks orange. Would you say that two sufficiently similar tiles are the same color? I suppose so, if they were so similar that I couldn’t tell them apart (if you can tell these particular tiles apart, kudos, but image two even more similar tiles). So, if we had a long line of tiles that slowly progressed from red to orange, and each pair of adjacent tiles was so similar that you couldn’t tell them apart, where would the red tiles stop and the orange tiles begin?

Some philosophers puzzle over this even today. The problem is that logic appears to contradict intuition. Classical logic concludes that there must be a red tile next to a non-red tile. Intuition concludes that this is pretty silly when we can’t tell any two adjacent tiles apart.

Now that you’re in a philosophical mood, you might ask, what does it mean for a tile to be red? Good question. Some philosophers say that the meaning of a word is defined by how people use it. So let’s ask Mechanical Turk.

We took 8 tiles on the gradient from red to orange. We showed each tile to 10 different turkers, and asked them whether the tile was red or not. This plot shows how many people said that each tile was red:

This chart suggests that there is no clear boundary between red and orange. It also suggests that it doesn’t really make sense to say that all tiles are either red or not-red, since people disagree about tiles in the middle. It might make more sense to say that a tile is 70% red, or 20% red. This view is called “Group consensus” on the Wikipedia page for Sorites Paradox.

We could probably run this experiment with more people over more tiles to get an even smoother curve from red to orange. If we did this, then we could develop a better intuition for what happens with adjacent tiles. No individual can distinguish them, but can the crowd distinguish them?

It might also be fun to capture the gradient of other terms, like “tall” or “rich”.

TurKit Online

We have been busy this past month or so developing an online version of TurKit, which was used to run this experiment. This version allows people to execute long running experiments “in the cloud”, without leaving their personal computer on all night. TurKit online runs on Google App Engine, and uses your Google id. The entire web-app is open source.

Where’s the code and data for this experiment?

The online version of TurKit doesn’t export projects yet, but it will soon. We’ll post the code and data soon.