Topics Today

Quantitative User Studies

based partially on I. Scott MacKenzie's tutorial at CHI 2008

Results of a Quantitative User Study

Result, which is valid for all people
Test users must be representative
Descriptive statistics:

Tables

Diagrams

Means, Medians, etc.

Inductive statistics:

Ensure that the results are generizable

How To Conduct a User Study

Think	Literature Research Discuss Theory Conduct Pre-studies
Design	State Hypothesis Create Prototype Plan Experiment
Measure	Conduct Experiment Collect Data Document Test Conditions
Analyze & Publish	Review collected data Apply Significance Tests Interprete Results Document Study and Results

Example:

You have developed a new alternative keyboard layout which you think is superior to QWERTY
How can you scientifically prove that your layout is better?

Research Questions

Very weak

Is the new layout good?

Weak

Is the new layout better than QWERTY?

Better

Is the new layout faster than QWERTY

Better still

Is the text entry speed (wpm) for this layout higher than for QWERTY after one hour of use?

Hypothesis

Prediction of the result of a user study
States how a change in the independent variables will effect the measured dependent variables
The hypothesis is either proved or disproved by study results
State hypothesis before conducting the user study

proper scientific way

do not read meaning into measurements afterwards

if you find surprising results, do not adjust hypothesis, but state and test a new one

A null hypothesis predicts that independent variables do not have any effect on dependent variables

Specification

The experiment should be set up to be reproducible
write a specification describing everything which is important for others to reproduce the experiment:

Hard- and software in use
Circuit schematics of self-built prototypes
Environmental conditions

Light conditions

Noise

Skills of the test users, e.g.

„All participants have to be professional designers”
“The candidates should have no experience on using eye-trackers”

Internal vs. External Validity

Internal Validity

How much are the observed effects caused by the test conditions?
Increased by rigorous control of test conditions

External Validity

How generizable are the results to other people or situations?
Increased by real-life tasks and test conditions

There's always a tradeoff between internal and external validity
Solution: Pose multiple, narrow questions that cover the whole range of a more general question
Example: a text input method that is easier to learn, faster, and less error-prone than another one is probably generally better.

Variables

Independent variables

Independent of participant
Manipulated by experimenter
"What shall be compared?"
"Level": number of different values
Example: readability of two fonts (Arial, Times New Roman) on three different backgrounds (red, white, blue) -> six experimental conditions

Dependent variables

Should only depend on the independent variables
Measure participant's performance/views.
"What is the effect of the independent variables?"
Objective: task completion time, error rate, etc.
Subjective: ease of use, fun, preferred options, etc.

Variables

Control variables

Do not vary throughout the whole experiment
Controlled by experimenter

You get higher internal, lower external validity

Examples: lighting conditions, background noise

Random variables

Vary randomly

You get lower internal, higher external validity

Example: age, gender, time of day, network load

Confounding variables

Vary systematically with independent variables

May lead to wrong conclusions

Try to eliminate them

Examples: prior experience, environment, learning effects

Randomization / Counterbalancing

Learning effect is a frequently occuring confounding variable
Randomization/Counterbalancing mitigates the effect of confounding variables
Control order of test conditions so that learning effects are equalized
Example:

User A: first enter text using a physical keyboard, then using an on-screen keyboard

User B: first enter text using an on-screen keyboard, then using a physical keyboard

Problem of naive randomization:

Order could actually be the same for all participants

Solution:

Counterbalance order of test conditions using Latin Squares

Experimental Method

Within subjects / repeated measures

All subjects do the tasks under all different conditions

Randomize the order of conditions to avoid ordering effects
Example: each participant tries out all three text input methods

Between subjects

Multiple groups
Each group does the test only under one conditions
Careful selection of groups is essential

	pro	contra
within subjects	fewer participants required (n), variance across conditions should be the same	learning/interference effects
between subjects	learning/interference effects avoided	more participants required (n * [# conditions]), usually harder to show significance

Latin Squares

Used to counterbalance test conditions
Each condition only occurs once in each row and column (like in Sudoku)

A	B	C
B	C	A
C	A	B

A	B	C	D
B	C	D	A
C	D	A	B
D	A	B	C

A	B	C	D
B	D	A	C
D	C	B	A
C	A	D	B

3x3 Latin Square

4x4 Latin Square

4x4 balanced Latin Square

In a balanced Latin Square each condition is preceeded/followed by each other condition

One-of vs. Comparative Research Questions

One-of:

determine properties of a user interfaces (e.g. overall typing speed)

Comparative:

compare user interface to an alternative design
alternative can be variation of the new design,
or an established design (QWERTY, mouse, Windows audio mixer)

Prefer comparative questions - they can be verified better

Finito