Upon reflecting on user studies, as prompted by a discussion at UIST, these are three main user study strategies. When are these tests most valuable?
USER STUDY STRATEGIES
Formative User Studies - “What should I do?”
This happens before any substantial design work and can be used to identify needs and problems for a population. This can help us, as researchers, to find real world needs instead of only identifying what we think our needs are. If we don’t understand what future users’ needs are, ethnographies and other formative user studies can be valuable, though time consuming. Time may be better spent looking at and drawing upon others’ previous research. Even unrelated work can help point us toward problems users face regardless of specific applications. However, doing studies like this can help to put researchers into contact with a user base who may be perfect for later evaluations.
One issue with this kind of study is that it can narrow your scope to the point that you eliminate the possibility of discovering serendipitous user bases. This benefit has arisen in several projects to come out of the UID group, and leads us to focus more on the following types of studies. We, as computer scientists, may, after all, be better suited to solve problems instead of finding them.
Iterative User Studies - “What should I change?”
This can be useful in testing prototypes. Steve Krug’s Don’t Make Me Think asserts that with three people in a study, especially one on web design, the researcher can ’squint’ and see most major areas of interest - like finding bugs or ‘falling into usability pitfalls.’ However, this may not be enough people; Nielsen and Landauer use a ‘magic number’ of 5. While there may not be an actual magic number, these small numbers can help find big usability bugs. Larger-scope ‘things’ may not be able to be handled by three people, but “if you’re testing all of Photoshop at once you’re doing something wrong anyway.” Constant iteration and testing of prototypes can help you determine if your device or design can even fit into someone’s life.
Summative User Studies - “How did it turn out?”
Testing and evaluating at the end of development as a method of evaluation is incredibly common and a valuable way to reflect on the benefits of a project as a whole.