Brainstorming Company Names

• Total Cost: $33.86
• 357 HITs
• TurKit code for this experiment: code.js, all files
• TurKit Version: 0.1.37

We asked turkers to brainstorm company names, both iterative and non-iteratively. We used an experimental design based on Image Description — Iterative vs Non-Iterative. We kept 6 iterations for each condition in this experiment.

The experiment itself is based on Website Tags and Website Tags — Not Iterative. Instead of generating tags for websites, we are generating names for companies. We also provide separate input fields for people to add names, rather than a single textbox.

We made up a brief description for four fake companies. You can see the description of one fake company in this sample brainstorming HIT:

The HIT asks turkers to come up with at most 10 company names, and supplies 10 input fields. Turkers in the iterative condition were shown “Example names suggested so far”, which shows company names supplied by previous turkers in the iterative process. This text appeared even if there were no names suggested yet. The non-iterative condition did not have this text.

We also created HITs to evaluate the quality of each name. These evaluations were done on a scale of “1: Poor” to “5: Extremely Good”, similar to the image description experiment. Turkers were not allowed to rate any suggestions for company X if they supplied any suggestions for company X.

We intended to show turkers in the iterative condition the ten best previous suggestions, but due to a bug, we only showed them the suggestions from the turker right before them. The suggestions were still sorted from best to worst.

Note that turkers in the iterative condition were allowed to contribute more than once. This has always been the case, even in previous blog experiments of this sort. Probably this shouldn’t be allowed, and our future investigations will probably prohibit this. With that said, this only happened once in these experiments (i.e. one turker, in the iterative process for one fake company, did the brainstorming HIT twice).

Results:

You can see all the names generated for each fake company here, in both the iterative and non-iterative conditions.

Here are the results for the “productivity tools” fake company:

Discussion:

The most striking observation is that the non-iterative condition generates significantly more company names than the iterative condition: 47.8 vs 34.3 (paired t-test p < 0.002). The highest possible number of names for either condition is 60, since there are 6 iterations, and each iteration asks a turker for up to 10 names.

This suggests that turkers will generate fewer names if they are shown some examples of other people’s names. Possible explanations for this include:

• Seeing other people’s names biases turkers toward those names, and they must think harder to come up with different names.
• Seeing other people’s names suggests that other people are making progress toward the goal, and so it’s not as important to do a good job on the task, since other people will pick up the slack.
• Seeing fewer than 10 example names may cue turkers into thinking it’s ok to provide fewer than 10 names (In fact, turkers shown 10 example names provide 7.2 names on average, while turkers shown between 1 and 9 example names provide 3.3 names (p < 0.01)).

The average quality of names generated in each condition seems to be the same. The average is 2.85 for the iterative condition, and 2.82 for the non-iterative condition. A t-test gives a p-value of 0.57, which is not significant.

Conclusion: Showing people other people’s ideas doesn’t seem to increase the quality of suggestions, and seems to encourage fewer suggestions. It would be nice to figure out some way to increase the quality of suggestions. Perhaps if we show people words that are associated with the company — like “love” and “cupid” for the fake online dating site — they can be used as building blocks for names. We could generate these words in a separate brainstorming process.