Topics Today

• Why Evaluations?
• Heuristic Evaluations
• Exercise 7: Heuristic Evaluation of SIP-Communicator

Evaluations

Evaluation is systematic determination of merit, worth, and significance of something or someone using criteria against a set of standards. (Wikipedia)

American Evaluation Association Guiding Principles for Evaluators

1. Systematic Inquiry: Evaluators conduct systematic, data-based inquiries.
2. Competence: Evaluators provide competent performance to stakeholders.
3. Integrity/Honesty:  Evaluators display honesty and integrity in their own behavior, and attempt to ensure the honesty and integrity of the entire evaluation process.
4. Respect for People:  Evaluators respect the security, dignity and self-worth of respondents, program participants, clients, and other evaluation stakeholders.
5. Responsibilities for General and Public Welfare: Evaluators articulate and take into account the diversity of general and public interests and values that may be related to the evaluation.

Why Evaluations?

• ensure functionality (the target task can be completed)
• ensure performance (the target task can be performed within resource limits, like time/system resources)
• customer/user acceptance
• identify problems for specific tasks or users

Approaches

• Inspections
• Heuristic evaluation (check if UI violates a set of rules)
• Consistency inspection (check if UI is consistent)
• Cognitive walkthrough (perform specific tasks)
• Controlled user studies
• Quantitative / Comparative
• Qualitative
• Usage observations
• Protocol findings of an observation on paper, video, log files
• Let the user keep a diary style protocol on a notebook
• Field studies
• Scientific observation in a natural environment (as opposed to a laboratory)

Reality Check

• Usability evaluations seldomly performed
• Reasons
  • Developers are not aware of the techniques
  • Fear of extra cost for evaluation
  • Developers run out of time
  • Necessity is neglected (“The product works”)
  • Teams often think what they understand is understood by everyone
• Usability Evaluation is crucial for almost every product
• Most evaluation techniques are cheap and fast
• Discount Usability Engineering
• Paper Prototyping
• Heuristic Evaluation

Heuristic Evaluation

Proposed by Nielson and Molich (1990) (see also www.useit.com/papers/heuristic/)

• Widely accepted: Google yields 76,000 hits in 2008
• Usability inspection method for an iterative design process
• A user interface is evaluated by a small number of experts
• Evaluators judge the user interface by its compliance to a set of rules or guidelines – the heuristics
• Biggest pitfall: One evaluator can hardly find all problems alone
• → several evaluators inspect a product independently
• Aggregate the findings after everybody is done

Steps of an Heuristic Evaluation

1. Preparation

• Define or agree on heuristics
• Prepare a complete usage scenario (walkthrough) that each evaluator goes through
• Decide on some way to capture the evaluation session and results
• Form/questionnaire
• Videotaping
• Observing and written assessment

2. Evaluation

• All evaluators go through the complete scenario – at least twice or until no more problems are found
• Take notes on problems found and document them carefully

3. Analysis

• Aggregate list of usability problems from individual forms
• Rate problems by severity

Usability Heuristics

• Bruce Tognazzini's 'First Principles'
• List of checklists/tools
• Heuristics for online help
• Heuristics for documentation
• Research-based Usability Guidelines  
• much smaller (16MB), local version (PDF)

Ten Usability Heuristics by Nielsen

1. Visibility of system status
2. Match between system and the real world
3. User control and freedom
4. Consistency and standards
5. Error prevention
6. Recognition rather than recall
7. Flexibility and efficiency of use
8. Aesthetic and minimalist design
9. Help users recognize, diagnose, and recover from errors
10. Help and documentation

Severity Rating

• Used to prioritize problems
• Helps in deciding whether a product is ready for release or not

• Three influencing factors:
• Frequency (often occurring problems are more severe)
• Impact (Can users overcome the problem or find a workaround)
• Persistence (Does the problem occur repeatedly or only under certain conditions)
• Scale to rate problems:
  0 = I don't agree that this is a usability problem at all
  1 = Cosmetic problem only: need not be fixed unless extra time is available on project
  2 = Minor usability problem: fixing this should be given low priority
  3 = Major usability problem: important to fix, so should be given high priority
  4 = Usability catastrophe: imperative to fix this before product can be released

Usability Report

• Write a usability report containing all problems identified
• Structure by severity and by functionality groups
• Give also suggestions for improvement
• Give constructive criticism
• Give an assessment of the products usability

Sources & Recommended Reading

• Nice Overview (cached)
• Nielsen on Heuristic Evaluation
• Nielsen on Severity Rating