Writing an Essay on MTurk: Part 3

In part 2, we had turkers vote on the best topic ideas. The top two ideas were:

• How effective is recycling in saving the environment?
• How important is a college education in todays job market?

I originally intended for the essay to be persuasive, so I wanted the topics to be statements rather than questions. Here is a statement version of each question:

• Recycling is effective in saving the environment.
• A college education is important in today’s job market.

We took all four of these topics — both the question and statement versions of the top two topics — and asked turkers to think of points in favor of them, or possibly against them if the topic was a question. We used the brainstorming algorithm from part 1, and paid each turker 1 cent to think of 3 new points. Each turker saw all the ideas suggested so far for the given topic.

Results:

Here are the points supplied from the first turker for each topic:

• A college education is important in today’s job market.
  • many of the highest paying professions like doctor require a degree
  • employers see a degree as proof of education because high schools don’t teach well
  • employers seek specialized knowledge that high school does not teach, so a college degree is required
  • …
• Recycling is effective at saving the environment.
  • Recycling saves energy it would take to make a new product from scratch.
  • Recycling cuts down on waste in the trash dumps, freeing up more land to be used more productively.
  • Recycling makes one stop and think about the environment each time you look for a bin – it makes you more conscientious and could encourage you to think of other ways to save the environment.
  • …
• How effective is recycling in saving the environment?
  • False if the process of recycling involves using more chemicals to make the item reusable.
  • False – if “recycling fervor” doesn’t encourage people to reduce their consumerism.
  • True when items are reused instead of put into the trash dumps.
  • …
• How important is a college education in todays job market?
  • Give you more time for jobs to open up
  • Provides more job opportunities
  • Puts you into debt
  • …

Raw results: json, html

Discussion:

Some observations:

• If the first turker uses a clear style guide for their points, this guide tends to be followed. One turker began all their points with “Recycling X”, and this pattern was followed by all subsequent turkers. The person who began all their points with “False if X” or “True if X” influenced most subsequent contributors. I’m not sure yet whether this has any effect on quality, pro or con. I suspect it probably has a positive influence, since it gives people a framework within which to frame their ideas (which would be one less thing to think about in coming up with an idea). This may be something to test.
• A couple forms of cheating: at least two users copied/pasted blurbs of text from the internet. Another user copied/pasted points directly from the list of previous points. These both seem easy to catch — if the text is abnormally long, flag it as being copied from the internet, and look for inputs identical to existing inputs — however, I’m not sure it is worth worrying about. Both of these forms of cheating will probably get weeded out anyway in the next step when we decide which points to keep in our essay — we’ll see.

Next Step:

The next step is to find the best supporting points for each topic (or for whichever one we decide to concentrate on). Part of this task may involve clustering the points into similar ideas, so that our top points aren’t all essentially the same.

Code:

• TurKit code for this experiment: code.js, all files
• TurKit Version: 0.1.42