Topics Today

• Why User Studies?
• Qualitative User Studies
• Exercise 8: Qualitative Evaluation of KA/PI

Why User Studies?

• get a scientific statement (instead of personal opinion)
• learn more (because of surprising results)
• ensure quality in product development
• compare solutions
• provide quantitative figures

Goal of a User Study

Subject to evaluation is the system usability

What exactly is evaluated depends on the stage of a project:

• Ideas and concepts
• Designs
• Prototypes
• Implementations
• Products in use

Approaches:

• Formative evaluation
• throughout the design
• helps to shape a product
• Summative evaluation
• quality assurance of the finished product

Qualitative vs. Quantitative User Studies

Qualitative	Quantitative
observe user	measure performance
generate insight	generate statistical data
used e.g. to find problem areas, find conceptual errors, find missing functionality	used e.g. to verify performance benefits of new input/output devices or interaction techniques determine differences between user groups,
most useful for formative evaluation	most useful for summative evaluation

Participants

Use the term 'participants' instead of 'subjects'

Ideal number of participants:

• qualitative: ~5 participants (number of usability problems found)
• quantitative: ~14 participants (statistical significance)

Participants should be representative of the user group

• Age
• Background (e.g. technical vs. not technical)
• Skills
• Experience

In most cases your team members are not representative!

Planning a User Study

1. Set goals
2. Design the experiment
3. Do a pilot study
4. Schedule users
5. For each user, typically:
   • Inform the user
     • About the procedure (amount of time, breaks, …)
     • What the study is about in general
   • Consent form
   • Do a survey on
     • Demographics
     • Questions related to the experiment (e.g. fun, handedness)
   • Give instructions on the task
   • Let the user do the tasks and observe / measure
   • Be available for questions and (informal) feedback
6. Analyze the results

Reporting Results of a User Study

• Anonymize participants
• Background of participants
• Details of tasks, exact wording
• What did they do?
• Why did they do it?
• What didn't they do?
• What is interesting?
• What was surprising to you?

(based on https://apps.lis.uiuc.edu/wiki/download/attachments/2654987/User+study.ppt)

Qualitative User Studies

Example:

Goal: Find usability problems in an online music shop.

Tasks the participants might have to complete:

• create an account
• order a certain CD
• search for music by a certain artist
• find out shipping rates
• ...

General Rules

• "Don't test your own baby"
• Create an initial set of tasks without looking at the product.
• Involve developers / product managers in task creation.
• Get feedback from test participants (realism, mising tasks)

Pretending and Hidden Clues

• "You are a salesman for a bio-engineering company."
• "You are a Spanish-speaker trying to find out about trips to Hawaii."
• Do not have users pretend to be someone else!

• "Use the 'add contact' wizard to enter Frank's contact detail's."
• "Find John Smith in the Hotmail Membership Directory"
• Avoid hidden clues!

Tasks

• You are testing the usability of the system, not the user.
• Make this clear every time!
• Make the user feel as comfortable as possible.
• The first task should be very simple
• Break down the tasks into small sub-tasks to give participants frequent successes
• Give basic tasks first, then specialized tasks
• Have tasks for novice, intermediate, advanced users

Scenarios vs. Tasks

• A scenario puts a background setting around one or more tasks
• Task:
  "Set a reminder for Friday, 11. January, 13:00"
• Scenario:
  "You friend Tom has invited you for lunch next Friday at 13:00. Make sure that you don't forget it."
• Use scenarios instead of single tasks - they are more realistic and offer more freedom.

Realistic Tasks

Make scenarios and tasks as realistic as possible

• don't leave out parts of a task
  • have participants really buy items at an online store.
  • have participants really send an e-mail
• let participants use their own account/data
• use a real-world collection of documents or addressbook data
• simulate e.g. network problems, full disk, ...

Humor and Bollywood

• Scenario:
  • "Your girlfriend has been kidnapped by Don Giovanni. Find out his address and get driving instructions to his home."
  • "Enter contact information for "Frank N. Stein, PhD, Horror Street 123, 90210 Gotham City."
  • "Use the spray can tool to draw an upside down cross."
• No funny/dramatic/controversial scenarios or test data
  • distracts from test
  • can hurt feelings

Instructor- vs. participant-defined tasks

• defined by instructor:
  "Buy Frank Sinatra's 'In The Wee Small Hours', ASIN: B00000DRCZ from the online store."
• defined by participant:
  "Buy any CD you like from the online store. We'll reimburse you for up to 30 EUR."
• Prefer participant-defined tasks - participants will be motivated more.

Open vs. Closed Tasks

• Open:
  "Look around in the online shop for 10 minutes. You can buy a CD if you want."
• Closed:
  "Find an CD by Elvis Presley and buy it."
• Prefer open tasks - they show you problems you would not have thought of.

System-oriented vs. User-oriented Tasks

• System-oriented:
  "Search the online help for the 'color' function."
• User-oriented:
  "Match the color of the diagram to the corporate color scheme (grey-blue-white). You may look in the online help."
• Prefer user-oriented tasks - they are more realistic and involving.

Subjective vs. Objective Answer

• Objective:
  "Find the cheapest CD in the online store."
• Subjective:
  "Find a CD you really like in the online store."
• Use both task types depending on your goals.

Comparative vs. contained within the application/website

• Contained:
  "Crop the image and convert it to grayscale"
• Comparative:
  "Crop and convert the image to grayscale using PhotoShop and Gimp. Which do you prefer?"
• Prefer comparative tasks - they provide more balanced feedback.

Some ideas for test tasks for KA/PI

• add multiple phone numbers
• set/edit contact categories
• search for contacts/groups of contacts
• create/modify views
• change columns in list view