Upload
daniel-ryan
View
213
Download
0
Embed Size (px)
Citation preview
Research Design and Evaluation
Colin Ware
Goals for empirical research
Uncover fundamental truths and test theories (late stage science).
Discover the nature of the world. Find out about how people interact with information artifacts. Find out how people respond to a particular display. (survey of recreational boaters).
Does theory apply in practice. E.g. good continuity applied to flow vis.
Compare two (or more) display or interaction methods. Is this method adequate?
Continued
Compare two entire systems. (GIS 1, vs GIS 2) Measure task performance (e.g how fast can we find a
certain class of patterns). Determine user preferences._________________________________
Non-empirical contributions
Invent a better interface for some particular domain (Social network analysis, interactive flow visualization) Proof of concept prototype Constructivist analysis
Invent a better evaluation method Invent a new “theory” (information foraging
theory)
Methods for evaluation Measure errors (how accurate are we)
Target detection (rapid response) Binary data Method of adjustment
Measure time to perform a task How much work can we do per unit time.
Rating scales (Likert scales) Turn judgments into numbers.
Task analysis + Structural analysis. A cognitive theory is constructed with observation as evidence
(constructivism). (e.g. narrative). Goes hand-in-hand with proor of concept applications
Semi-stuctured interviews. Expose the user to a problem + solution in a
systematic way. (+questions) Wizard of Oz Cognitive walkthroughs.
Have the user perform a task and talk aloud as they do it.
Spiral design method
Must documentthe design process
Spiral design
One way of proceeding is though a spiral design method.
Construct a rapid prototyping environment (tool kit) – demo or die
Construct a prototype Evaluate (and
document findings) Refine prototype
Evaluate (and document findings)
Methodological Issues
If you want to measure time/error data Best to have a simple representative task Justify it.
If you want to get opinions (ratings, interviews). Domain experts often ideal. But they may have established ideas – they like the status quo.
Information interface design
Determine data and problem
Determine tasks
Create mappings from data to representation such that important patterns are “transparent”.
Create interfaces for rapid task performance
Create support for hypothesis
Things to consider
Aim for important targets. Keep it simple Maximize payoffs for (your) time expended Be sure to have control conditions.
Defacto standard Alternative display methods.
Choose dependent variable ranges. Remove all un-necessary variables (e.g.
background). Have sufficient observations.
Statistics
Means. Standard deviations. Standard error of the mean = /sqrt(N);
2se ~ 95% confidence intervals.
ANOVA e.g. color and motion.
The End
Velocity glyphs
CurvedCompactShow direction clearlyEasily readable: more looks “greater”Capable of 100 levels
Visualization design pattern
Patterns of common low-level tasks
Relating patterns- e.g. bioinformatics.
Pattern searching Finding paths Finding groups Finding target symbols