- Total Cost: about $40
- Running Time: 10-30 minutes
- 6 HITs with 165 Assignments total
- Payment per Assignment: $0.01, $0.25, $0.50
- Not Iterative
- TurKit code for this experiment: wait-for-letter.js
- TurKit version 0.1.42
- Flex code for this experiment: WaitForLetter.mxml (compiled version WaitForLetter.swf)
For many collaborative uses of MTurk (like the Etherpad experiment in the last post), it would be desirable to have a number of turkers interacting at the same time. This is an unusual use of MTurk; most HITs are designed to be done separately and asynchronously by each turker.
This experiment continues our exploration of what it might take to get a group of turkers to be working on a HIT at the same time. (See also Are Turkers Willing to Wait for a Task?)
The HIT used here is a simple letter-transcription task: a picture of a letter (A-Z) is shown, and the turker must type it into a textbox. The catch is that the letter won’t appear until some time in the future, typically a few minutes away. The deadline is the same for all turkers who pick up the HIT. The deadline is stored internally in GMT, but displayed in the turker’s local timezone. Before the deadline, the instructions for the HIT look like this:
- At the time shown below, an image of a letter (A-Z) will appear. In the box at the bottom, please type the letter you see.
- Early answers and wrong answers will not be approved.
- The letter will be visible for 10 seconds.
- If you keep this HIT open in your browser, then a chime will sound while the letter is visible, to remind you to look at it.
- If you don’t see a time below, don’t accept this HIT, because you won’t be able to do it.
When the deadline is reached, the letter appears for 10 seconds, and a chime sound plays to alert the turker to pay attention to it:
You can see an example of what turkers saw when they waited for a letter (with only a 6-second deadline).
Because all turkers have the same deadline, and the letter is only visible for 10 seconds, we can assume that all the turkers who submit the correct answer were viewing and interacting with the HIT within that 10-second interval. So the group was synchronized. A synchronized group could be used to chat with each other, play a game, or pound on a website.
On a Friday evening (11pm-midnight EST), I ran several small pilot experiments to test this approach for collecting synchronized groups of turkers. In the experiments below,
n = number of synchronized turkers desired (number of assignments in the HIT)
reward = payment for the HIT
deadline = time from posting of the HIT when the letter would appear
For n = 10, reward=$0.01, deadline=2 minutes, the HIT obtained 4 synchronized turkers. One cent is evidently not enough to entice turkers to wait around that long.
Increasing the reward to $0.50 produced 6 synchronized turkers. Two minutes may not be enough time to collect 10 turkers; the flow of new turkers interested in this task isn’t large enough.
Increasing the deadline to 10 minutes produced 9 synchronized turkers, which is close to the goal.
Finally, for n=30, reward=$0.50, deadline=10 minutes, we obtained 25 synchronized turkers.
In all cases, every turker provided the correct letter, which was different for each HIT. We also recorded the turker’s total wait time from accepting the HIT to finally submitting it. For the final 10-minute experiment, the wait time averaged 5 minutes 21 seconds, and the longest waiter took 10 minutes and 21 seconds, and the fastest one waited only 12 seconds. This suggests that that the HIT was close to saturating the flow rate of turkers who discovered this HIT and were willing to do it at this time of day and this price point.
Update 1: I reran the experiment (Saturday night between 9pm and 10pm EST) with n=30, reward=$0.25, deadline=10 minutes, and obtained 21 synchronized turkers. Halving the reward still produced about the same number of turkers, which may be around saturation at this time of day.
Another point of curiosity: even though Amazon guarantees that these turkers are all using different MTurk accounts, it seems possible that the same turker could be using multiple accounts at once (perhaps created on behalf of friends and family). Consulting the server logs, however, we find that all 21 turkers came from different IP addresses, which makes it more likely that they were 21 unique people.
Update 2: One more run of the experiment (Sunday night, 8pm-9pm), with n=75, reward=$0.25, deadline=30 minutes, produced 33 synchronized turkers.
In this run, two turkers gave wrong answers early on, rather than wait for 30 minutes (they were not considered synchronized, so were not counted in the 33 above). One turker produced a correct answer 30 seconds before the deadline, probably because their computer’s clock was fast. A flaw in the experiment is that it assumes the turker’s own clock is accurate; a better implementation would measure local clock skew and adjust the deadline accordingly.
In this run, I also watched the server access logs to see how many turkers previewed the HIT. 74 unique IP addresses previewed the HIT, very smoothly distributed over the course of its 30-minute duration. Since 10-minute HITs on Friday and Saturday night obtained roughly 25 turkers, and a 30-minute HIT at least drew the attention of roughly 75 (even though the price wasn’t enough to make them stick around), this suggests that the flow rate is roughly 2.5 turkers per minute (for this task and price on weekend evenings EST).
Of the 74 turkers who previewed the HIT, 58 accepted it, according to Amazon’s requester report. Since only 33 successfully submitted, many of the others may have missed the 10-second window.