Upload
craigmmacdonald
View
999
Download
2
Tags:
Embed Size (px)
DESCRIPTION
HCI has always focused on designing useful and usable interactive systems, but usability has dominated the field while research on usefulness has been largely absent. With user experience (UX) emerging as a dominant paradigm, it is necessary to consider the meaning of usefulness for modern computing contexts. This paper describes the results of an exploratory study of usefulness and its relation to contextual and experiential factors. The results show that a system’s usefulness is shaped by the context in which it is used, usability is closely linked to usefulness, usefulness may have both pragmatic and hedonic attributes, and usefulness is critical in defining users’ overall evaluation of a system (i.e., its goodness). We conclude by discussing the implications of this research and describing plans for extending our understanding of usefulness in other settings. Paper presented at the 2014 ACM Conference on Designing Interactive Systems (DIS 2014).
Citation preview
What Does it Mean for a System to be Useful? An Exploratory Study of Usefulness Craig M. MacDonald, Ph.D. Pratt Institute, School of Information & Library Science
Michael E. Atwood, Ph.D. Drexel University, College of Computing & Informatics
25 June 2014 Designing Interactive Systems | Vancouver, BC2
HCI has evolved
2
Web, mobile, and personal technologies have drastically altered how, where, and
why people use computers. (Bødker, 2006)
User Experience (UX), which treats human-
computer interaction as a cognitive-emotional process, is emerging as a new
paradigm for the field. (Bargas-Avila & Hornbæk, 2011)
"The 21st Century Concert Experience" by Al Case is licensed under CC BY-NC-ND 2.0
Usability – is it the only factor?
3
"Tricycle" by Aslak Raanes is licensed under CC BY 2.0
“If ease of use was the only valid criterion, people would stick to tricycles and never try bicycles.”
- Douglas Engelbart
Usability = the gold standard Usability research has dominated HCI over the
past 40 years. It has yielded tremendous insights into how to
design systems that are easy to use and easy to learn.
But, little time has been spent on determining
whether systems are actually useful.
4 Source(s): Hornbæk (2010); MacDonald & Atwood (2013)
Usefulness is not new… Designing useful systems has long been cited as one of
the primary goals of user-centered design.
5
…but it’s mostly a mystery Studies of usefulness are largely absent from
HCI research. There is no empirical evidence about the relationships
between usefulness and other factors. There is no widely accepted definition of
usefulness in HCI. When people talk about usefulness, we can’t be certain
they’re talking about the same thing. This research aimed to address these two areas by
(1) defining usefulness and (2) studying usefulness in a controlled experiment.
6
So, what is usefulness? Many different definitions or uses of the term
usefulness, each of which covered one or more of the following: 1. The functions provided by the system.
e.g., “the functions people really need in their work” (Gould & Lewis, 1985)
2. The tasks users are trying to complete. e.g., “the role of the technology in accomplishing the user’s relevant tasks” (Maryniak-Nelson & Caldwell, 1992)
3. The goals users are trying to achieve. e.g., “whether system can be used to achieve some goal” (Nielsen, 1993)
4. The context in which the system is being used. e.g., “how technology can fit into users' actual social and/ material environments” (Nardi, 1996)
7
A working definition: Usefulness is “the extent to which a system’s
functions allow users to complete a set of tasks and achieve specific goals in a particular context of use.”
This may sound similar; according to the ISO:
Usability is “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specific context of use.”
8
What is usability? There is surprisingly little consensus about a
precise definition of usability. (van Welie, van der Veer, & Eliëns, 1999)
But there is broad agreement that usability
refers to effectiveness, efficiency, and satisfaction. (Chen, Germain, & Rorissa, 2009; Hornbæk 2006; Hornbæk & Law, 2007; ISO 9241-11; Nielsen, 1993)
9
Our (imperfect) distinction Usefulness
a system’s appropriateness for a specific context
Usability
its effectiveness, efficiency, and satisfaction within that context
10
Method We designed and conducted an exploratory
experiment* to: 1. examine the effects of context on usefulness;
and 2. explore the relationships between usefulness
and three other UX attributes: usability, aesthetics, and enjoyment.
*Yes, a laboratory study of context seems like an oxymoron but we believe
it is a good starting point for studying usefulness.
11
What is context? Many ways of defining context; for this study,
context is defined simply as:
12
USER TASK
TOOL
ENVIRONMENT
Source(s): Suchman (1987); Bannon (1990); Schilit & Theimer (1994); Hutchins (1995); Kuutti (1995); Nardi (1996); ISO (1997); Dey, Abowd, & Salber (2001); Dourish (2004); Räsänen & Nyce (2006); Connolly, Chamberlain, & Phillips (2008)
(diagram adapted from Shackel, 1991)
Controlled: User The user dimension can depend on many
possible sub-dimensions. So, we tried to control for this factor by
limiting the study population to a group likely to share common interests, knowledge, and interaction styles/preferences.
13
Independent Variable: Task-Env Three written scenarios: one “no scenario” as a
control and two scenarios that differed along the task and environment dimensions:
14
Task-Environment 1 Task-Environment 2 Task:
EXPLORATION (action mode): find reputable resources with facts, pictures,
charts/graphs, etc., to include in presentation about Climate Change.
Task: RETRIEVAL (goal mode): find whether
the name of the first computer was “IPECAC” and whether it was invented in
1928 at Drexel University.
Environment: Early on a weekday morning In the library computer lab
Class assignment
Environment: Late on a weekday evening In a student dormitory room
A wager with a friend
Independent Variable: Tool
15
Educational information portals were chosen because they: 1. offer a range of potential use
cases; 2. are relatively easy to learn;
and 3. are familiar to the study
population.
Dependent Variables Variable Scale Items Source
USEFULNESS 1. The website provides the right functions. 2. I am able to use the website to complete my task(s). 3. I am able to use the website to fulfill my goal(s). 4. The website fits my current situation. 5. The website is useful.
Developed for this research
USABILITY 6. The website is easy to use. 7. I feel in control when I am using this website. 8. The website requires little effort to use. 9. Using the website is effective.
De Angeli, Hartmann, & Sutcliffe (2009)
AESTHETICS 10. Everything goes together on this website. 11. The color composition is attractive. 12. The layout appears professionally designed. 13. The layout is pleasantly varied.
Moshagen & Thielsch (2010)
ENJOYMENT 14. I find using the website to be enjoyable. 15. The actual process of using the website is pleasant. 16. I have fun using the website.
van Shaik & Ling (2011)
GOODNESS 17. I judge the website to be: bad—good Hassenzahl (2004)
16
Study Design We used an incomplete repeated measures
design in which participants were exposed to all three levels of each variable exactly once.
17
Post-Study Questionnaire First, participants responded verbally to the
following question: What makes a website useful to you? What criteria
do you look for? Next, participants answered questions
regarding: Experience with the websites used in this study Knowledge of the Internet (iKnow) (Potosky, 2007)
Basic demographic information (age, gender) Background (major, class level, number of HCI
courses)
18
Participants 36 undergraduate students enrolled at Drexel
University in the Information Technology, Information Systems, or Software Engineering programs. All sophomores or above. Nearly all (35; 97%) had taken at least one HCI course.
Other characteristics: Median age range: 21 years old Gender: 86% male, 14% female iKnow (0-70): ranged from 52 to 70 (average = 62.81)
19
Data Analysis: Marginal Models A special case of Linear Mixed Models that
include: Fixed effects: can be interpreted similarly to
ANOVA Estimates of regression coefficients: can be
interpreted similarly to traditional regression analysis
20
Parameter Estimate t Sig.
USABILITY 1.386 16.834 < 0.001 **
AESTHETICS -0.633 -5.613 < 0.001 **
AESTHETICS * Climate Ch. 0.472 3.600 0.001 **
AESTHETICS * First Comp. 0.100 0.836 0.405
AESTHETICS * Control 0.000 . .
iKNOW * Climate Ch. -0.025 -2.986 0.004 **
iKNOW * First Comp. 0.002 0.370 0.712
iKNOW * Control 0.015 2.342 0.021 *
Marginal Model: USEFULNESS
21
Source F Sig.
USABILITY 283.391 < 0.001
AESTHETICS 29.856 < 0.001
TASK-ENV * AESTHETICS 6.961 0.001
TASK-ENV * iKNOW 5.513 0.002
Final Marginal Model
Coefficient Estimates
Higher ratings of USABILITY associated with higher ratings of USEFULNESS.
Higher ratings of AESTHETICS associated with lower ratings of USEFULNESS (but not as much under Climate Change scenario).
Higher iKNOW scores associated with lower ratings of USEFULNESS under Climate Change scenario.
Higher iKNOW scores associated with higher ratings of USEFULNESS under Control scenario.
Marginal Model: GOODNESS
22
Parameter t Sig. TOOL: ipl2 -0.557 0.580 TOOL: RefSeek 2.559 0.013 * TOOL: Awesome Lib. -2.544 0.013 * USEFULNESS 5.464 0.000 ** USABILITY -0.241 0.810 USEFULNESS * Climate Ch. 1.036 0.303 USEFULNESS * First Comp. -3.667 0.001 ** USEFULNESS * Control . . AESTHETICS * Climate Ch. 4.600 < 0.001 ** AESTHETICS * First Comp. 5.699 < 0.001 ** AESTHETICS * Control 0.684 0.497 ENJOYMENT * Climate Ch. -3.071 0.003 ** ENJOYMENT * First Comp. 0.898 0.372 ENJOYMENT * Control 2.850 0.006 ** iKNOW * ipl2 0.015 0.988 iKNOW * RefSeek -2.266 0.027 * iKNOW * Awesome Lib. 2.589 0.012 * USABILITY * ipl2 4.783 < 0.001 ** USABILITY * RefSeek 2.804 0.007 ** USABILITY * Awesome Lib. . . AESTHETICS * ipl2 -4.085 < 0.001 ** AESTHETICS * RefSeek -3.309 0.001 ** AESTHETICS * Awesome Lib. . .
Source F Sig.
TOOL 4.986 0.004
USEFULNESS 38.452 < 0.001
USABILITY 10.059 0.002
TASK-ENV * USEFULNESS 11.397 < 0.001
TASK-ENV * AESTHETICS 12.169 < 0.001
TASK-ENV * ENJOYMENT 6.350 0.001
TOOL * iKNOW 4.463 0.007
TOOL * USABILITY 12.015 < 0.001
TOOL * AESTHETICS 10.339 < 0.001
Final Marginal Model Coefficient Estimates
All variables had some effect on ratings of GOODNESS, but USEFULNESS was the only variable to have a significant relationship across all systems and contexts.
Higher ratings of USEFULNESS were associated with higher ratings of GOODNESS under all three scenarios (but the effect was weaker in the First Computer scenario).
Qualitative Analysis Four themes:
23
Appropriateness to Context
Simplicity and Ease of Use
Pleasurable Interaction
Visual Attractiveness
CODES: - Suitable for Purpose/Goal - Right Functionality - Appropriate Content
CODES: - Easy to Use/Navigate - Speed/Efficiency in Use - Organized/Uncluttered - Streamlined/Simple Design
CODES: - Pleasing to the Eye - Craftsmanship - General Attractiveness
CODES: - Familiarity - “It” Factor - Irritation-Free - Customizability
Usefulness
n=28 77.8%
n=17 47.2%
n=27 75.0%
n=17 47.2%
Usefulness is shaped by context The quantitative analysis showed a significant
effect of contextual factors (the iKnow x Task-Environment interaction) on perceptions of usefulness.
The qualitative analysis revealed that many participants defined a useful website in terms of contextual factors (whether it provides the “right” functions and access to the “right” information).
Conclusion: it is highly likely that the usefulness of
a system depends on the context in which it is used.
24
Usability is linked to usefulness The quantitative analysis showed that higher
ratings of usability were associated with higher ratings of usefulness.
The qualitative analysis revealed that major aspects of usability (efficiency/speed, effectiveness, irritation-free, etc.) were considered aspects of a useful website.
Conclusion: usability seems to be an integral
component of usefulness, but more data is needed (especially at the boundaries).
25
Usefulness may be hedonic? The quantitative analysis showed that websites
seen as more attractive were seen as less useful and that usefulness was not influenced by perception of enjoyment.
The qualitative analysis showed that almost 50% of participants cited some aspect of aesthetics and/or experience in their definitions of a useful website.
Conclusion: the beauty dilemma1 is powerful, and
the experiential aspects of usefulness may be more nuanced than “enjoyment.”
26 Source: 1 Diefenbach & Hassenzahl (2009)
Usefulness matters – a lot The quantitative analysis showed that
usefulness was the only variable with a significant effect on ratings of goodness (regardless of contextual factors). However, other factors (usability, aesthetics,
enjoyment) were also important depending on the tool and/or task-environment.
Conclusion: usefulness is a necessary (but
maybe not sufficient) dimension of a good system.
27
Implications for Research/Practice 1) Evaluators should explicitly address
issues related to usefulness. Probing for usefulness is not uncommon, but it
is infrequent, informal, and without a coherent connection to evaluation goals. (Nørgaard & Hornbæk, 2006)
We encourage evaluators to incorporate questions of usefulness into evaluation plans and purposefully address usefulness during user testing.
28
Implications for Research/Practice 2) Evaluators may consider varying
evaluation contexts. Potential variants: random sampling of tasks,
allowing users to modify test environments, holding tests in various locations, etc.
Doing so would jeopardize the validity of any experiment, but there is plenty of evidence that usability lab experiments are not valid anyway. (Gray & Salzman, 1998; Lindgaard & Chattrichart, 2007; Nørgaard & Hornbæk, 2006; )
29
Limitations & Future Work This study was a laboratory experiment with a
relatively small and tightly controlled group of mostly male tech-savvy undergraduate students and a highly specific type of interface.
We plan to address these limitations in future
work, and encourage other researchers to do so as well.
30
Thank You Craig M. MacDonald [email protected] http://www.craigmacdonald.com @CraigMMacDonald
31
Acknowledgements: We thank Susan Weidenbeck, Michelle Rogers, Denise Agosto, and
Kasper Hornbæk for their guidance in shaping and directing this research