Qualitative User Studies

Übung Mensch-Maschine-Interaktion 1, Wintersemester 2008/09


Topics Today

Why User Studies?

Goal of a User Study

Subject to evaluation is the system usability

What exactly is evaluated depends on the stage of a project:


Qualitative vs. Quantitative User Studies

Qualitative Quantitative
observe user measure performance
generate insight generate statistical data
used e.g. to
  • find problem areas,
  • find conceptual errors,
  • find missing functionality
used e.g. to
  • verify performance benefits of new input/output devices or interaction techniques
  • determine differences between user groups,
most useful for formative evaluation most useful for summative evaluation


Use the term 'participants' instead of 'subjects'

Ideal number of participants:

Participants should be representative of the user group

In most cases your team members are not representative!

Planning a User Study

  1. Set goals
  2. Design the experiment
  3. Do a pilot study
  4. Schedule users
  5. For each user, typically:
    • Inform the user
      • About the procedure (amount of time, breaks, …)
      • What the study is about in general
    • Consent form
    • Do a survey on
      • Demographics
      • Questions related to the experiment (e.g. fun, handedness)
    • Give instructions on the task
    • Let the user do the tasks and observe / measure
    • Be available for questions and (informal) feedback
  6. Analyze the results

Reporting Results of a User Study

(based on https://apps.lis.uiuc.edu/wiki/download/attachments/2654987/User+study.ppt)

Qualitative User Studies


Goal: Find usability problems in an online music shop.

Tasks the participants might have to complete:

A qualitative user study is only as good as its test tasks!

The following guidelines are based on Rolf Molich's tutorial "Creating Good Test Tasks" at CHI 2008

General Rules

Pretending and Hidden Clues


Scenarios vs. Tasks

Realistic Tasks

Make scenarios and tasks as realistic as possible

Humor and Bollywood

Instructor- vs. participant-defined tasks

Open vs. Closed Tasks

System-oriented vs. User-oriented Tasks

Subjective vs. Objective Answer

Comparative vs. contained within the application/website

Some ideas for test tasks for KA/PI