Response Time Over 24 Hours
- Total Cost: $3.60
- Running Time: 24 hours (1 new HIT with 5 assignments each 20 minutes)
- 72 HITs with 5 assignments, each paying $0.01
- TurKit code for this experiment: response-time.zip response-time.js
- TurKit Version: 0.1.29
This experiment revisits the question of response time initially explored as part of our small where’s the duct tape example. Every 20 minutes over 24 hours we posted a HIT with 5 assignments. The assignments were drawn from 12 tasks resembling the duct tape example – turkers were shown a picture containing an object and asked to click on the object. You can try for yourself in our object recognition test page.
Turkers successfully identified the object in almost all cases. The main exception being the following picture in which we asked turkers to find sunglasses:
Only about half of turkers shown this example correctly identified the sunglassses. If you’re having trouble, they’re on top of the file cabinet.
You can see the results visually here. These graphs were created with Protovis, which unfortunately only works with Firefox at the moment, and so are repeated below. The blue color represents the first turker to submit, accept, or complete the HIT…the aqua, green, yellow, and pink represent later turkers.
What’s great about this is that without doing much of anything to encourage quick responses, we were still able to get an answer within 100 seconds most of the time.
In this experiment, we also looked to see how many assignments you would need to put out to get the fastest submissions. This is not straightforward because turkers can accept a HIT quickly and then wait a while before submitting it – it’s not clear why they would want to do this but they do. In the worst case, all 5 assignments are accepted quickly but none are quickly submitted. We could avoid this situation by setting a shorter maximum time, but this might have adverse effects on the pickup time of the HIT and is something to consider in future iterations of the experiment. In this study we used the TurKit default maximum of 60 minutes.
In 40 cases (56%), the turker who was first to accept the HIT was also the first to submit it. 17 2nd-acceptors submitted first, 10 3rd-acceptors submitted first, 4 4th acceptors submitted first, and 1 5th acceptor submitted first.
This study suggests at least two interesting properties of MT:
First, time of day does not seem to matter all that much – turkers took a similar amount of time to respond at 5am EST as they did at 5pm EST.
Second, if you want something done quickly, you may need to recruit multiple turkers, as any one turker may not reliably complete a HIT expeditiously. The turker who accepts first will also likely not be the first to submit it.
As we move forward, we plan to try more active techniques at getting turkers to answer more quickly. How much would you have to be willing to spend over a 24 period to guarantee that you could always get an answer to a question like this in less than 10 seconds? How would you do it? Hopefully, we’ll have an answer!
You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.