Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to

Evaluation Methods - Summary

How to chose a method?

Stage of study– formative, iterative, summative

Pros & cons Metrics

– depends on what you want to measure Qualitative vs. quantitative Research perspective

– CS vs. psychology vs. sociology

Pros & Cons

Realism Precision Generalizability Time & cost Researcher expertise

Methods

Survey Interview Controlled-lab

experiment In-lab observation Controlled field

experiment Field observation study Heuristic Evaluation

Cognitive Walkthrough Experimental simulation GOMS Computer simulation Formal theory

Survey

Pros– Easy to get a large number of responses.– Quick and easy to conduct.– Highly generalizable.

Cons– Self-selection.– Participants often only offer enough information to answer the

question.– Can miss details.– Low in realism and precision.

Interviews

Pros– Quick and easy to conduct.– Gives designer quick feedback on a range of ideas.– Can get a person’s initial reaction to an idea.– Can get detailed information from a person.

Cons– Often takes place away from natural setting.– Question wording or interviewer “body language” can bias

answers.– High probability of false positives and missed problems (e.g., users

may not have a clear idea of how an app will be used).– Can miss details if interviewer doesn’t know what issues to draw

out.

Controlled Lab Experiment

Pros– Provides precise, quantifiable data.– Easier to draw inferences from data.– Relatively quick.– Can get a medium-sized number of participants.

Cons– Short duration of a lab experiment may not be enough to allow

users to become accustomed to an app.– Not a natural setting – interaction may not be normal.

In-lab Observation

Pros– Relatively quick.– Can get a medium-sized number of participants.

Cons– Observations are subjective and error prone.– Short duration of lab observation is not enough time

for user to get accustomed to using the interface.– Not a natural setting – interaction may not be

normal.

Controlled Field Experiment

Pros– Less intrusive than most other evaluation methods.– Provides more precise data than field observation.– Can observe natural behavior of user (though some

part of the system will be controlled/unnatural).

Cons– More intrusive than field observation.– Less natural than field observation.

Field Observation Study

Pros– Only way to observe natural behavior of user & interaction

between user & tools.

Cons– Difficult and time consuming.– Hard to get permission to observe people.– Observations are subjective and error prone.– Cannot make strong interpretations from observations.– Not very generalizable.

Heuristic Evaluation

Pros– Quick and easy.

Cons– Nielson’s heuristics may not be as relevant to non-

GUIs.– Results in false positives in missed problems,

especially when experts are not part of target audience.

Cognitive Walkthrough

Pros– Quick and easy.

Cons– Results in false positives and missed problems

when evaluator is different from target audience.

Experimental Simulation

Pros– Still fairly precise.– More realistic than in-lab experiment.

Cons (same as lab exp.)– Short duration of a lab experiment may not be

enough to allow users to become accustomed to an app.

– Not a natural setting – interaction may not be normal.

GOMS

Pros– Predict human performance before committing to a

specific design in code or running user studies– Many studies have validated the model (it works)

Cons– Assumes error-free, skilled user behavior– No formal recipe for how to perform analysis– Significant time investment

Computer Simulation

Creating a complete & closed system that models the operation of the concrete system without users.

Example:– geophysical process going on in connection with the

eruption of Mount St. Helens– Simulation of pilot during plane crash

Computer Simulation

Pros– Supposedly high in realism (depends on accuracy of

data/system replication)

Cons– Low in precision & generalizability

Formal Theory– This is a dissertation

Formulating general relations (propositions, hypothesis, or postulates) among a number of variables of interest.

Pros– Relatively generalizable

Cons– Not realistic or precise

How to chose a method?

Stage of study Pros & cons

– Realism– Precision– Generalizability– Time & cost

Researcher expertise Metrics Qualitative vs. quantitative Research perspective

Methods




Cognitive Walkthrough Automated observation

user study Experimental simulation GOMS Computer simulation Formal theory

Early Stage – these are the best






Iterative & Summative Stages – these are the best






Realism – these are the best






Precision – these are the best






Generalizability – these are the best






Time & Cost – these are the best






Researcher Perspective – Whoever is doing will choose whatever they want






Metrics: examples

Traditional GUIs:– efficiency (time to complete task)– accuracy (# of errors)– simplicity

Peripheral Displays:– awareness (recall)– distraction (dual-task behavior)– aesthetics

Documents

Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to