Embed Size (px)
DESCRIPTION
Evaluating User Experience given by Alistair Sutcliffe & Jenny Hart at Manchester NUX
Citation preview
Evaluating User Experience
Alistair Sutcliffe & Jenny Hart
Manchester Business School University of Manchester Manchester M15 6PB,
UK
NUX July 2011 with thanks to Ons AlShamueli & Rabia Khan
Presentation Outline
1. Background to UX research
2. A framework for UI ‘attractiveness’ and user engagement
3. Some experiments on users’ perceptions of design quality
4. Defining User Experience
5. Evaluating User Experience
So What is User Experience (UX) ?
“A person’s perceptions and responses that result from the use or anticipated use of a product, system or service”. ISO 2010
“The effect and affect produced by aesthetic experience, the meaning we attach to the product, and the feelings and emotions produced”
No real agreement on definitions
Essentially beyond the functional- aesthetics, attractiveness – design that excites and holds our attention- user engagement
UX Research- the quants
• Affordances, aesthetics and emotion key factors in design (Norman 2004)
• What is beautiful is usable (Tractinsky 2000, 2004); expressive & classic aesthetics, pleasure,
• Beauty, goodness, pragmatics & hedonics- Attractdiff (Hassenzahl 2004, 2006, 2010)
• Aesthetic perception and interaction (Lingaard 2006, 2009, Hartmann, Sutcliffe & de Angeli 2008) priming effects
UX Research- The Quals (or contextualists)
• Jordon 1998, Pleasure in Products
• Mc Carthy & Wright 2005 Technology As Experience, 2010 User Experience
• Dourish 2004 Where the Action Is Embodied Interaction
• Cockton 2008, Worth maps in UI design
• Hallnäs and Redström,2005, Presence and user experience
• Designers of User Experience- Gaver, Sengers, Forlizzi and many others
UX Research- Viewpoints
1. There are fundamental cognitive constructs by which we perceive User Experience- and hence can evaluate it (Hassenzahl, Tractinsky)
2. User Experience can only be understood in context, each experience is unique. It can only be understood by case study analysis and heuristics (Mc Carthy & Wright + contextualists)
3. User Experience is best understood by design- cultural probes, etc (Gaver and other designers)
4. UX is a cognitive process which can be modelled and measured (Sutcliffe, de Angeli)
The Manchester Attractiveness Framework
Attractiveness
Content / Services
Reputation / Identity
Usability
Aesthetics
Customisability
Attractiveness “Pleasing or appealing to the senses, arousing interest” OED
UX Experiments
• User perceptions of Aesthetics and UI Design qualities - Comparing web sites +/- attractive design features (interactive metaphors, dynamic media) - Sutcliffe & de Angeli INTERACT 05, DIS 06
• Framing effects and Customisation on UX - subject background, task scenario effects, customisation - Hartmann, Sutcliffe & de Angeli CHI 07,08
• Avatars, Immersion and UX - attractive chatterbots & persuasion - comparing UX in immersive v. standard environments (Khan, Sutcliffe & de Angeli, Sutcliffe & AlRayes 2010)
Menu Design style
Both sites- equivalent information, different UI styles
Summary of Differences
Dimensions Engaging style Traditional style Usability - + Aesthetic classic - + Aesthetic expressive
+ -
Information quality
- +
Engagement + - Memory
Overall Preference
Content-based
14 (50%)
Interface-based
14 (50%)
User Judgement: Effect of Customisation
Mobile phone News feed application
2 versions
Good/poor aesthetics
Customised or General content
Results
aesthetics manipulation content-fit manipulation
Measures Aesth++ Standard customised generic
Aesthetic classic + - n/a n/a
Aesthetic
expressive
+ - n/a n/a
Engagement + - n/a n/a
Usability = = = (+) = (-)
Look and Feel + - + -
Customisability = = + -
Content = = + -
Overall
Preference
+ - + -
Summary- Components of Attractiveness
• Attractiveness is a complex mix of factors, but content and services probably more important
• Judgement and choice very dependent on user background and task
• Usability is important but defects will be tolerated if overall experience is positive (halo effect)
• Aesthetics is important but only in context <user background>
• Content and customisation important components in overall preference, but if equal then other criteria come into play
Experiments on Avatars/ Virtual Characters
• Avatars (human like characters) influence preferences
- make information more credible - persuade people more effectively than text/speech alone
• Attractive avatars are more effective than less attractive
- younger, females - similarity to target audience
• Leverages “Computer as Social Actor” effect – we treat representations of people on line like real people (Reeves & Nass 1996)
Khan, Sutcliffe & De Angeli 2010
Attractiveness in Agents
Small differences in appearance make a huge differences in attractiveness and persuasion
Avatars in virtual worlds increase engagement (the Second Life experience)
more interaction improves UX ratings
• UX in the wide
- Experience throughout the Product life cycle - Initial contact (aesthetics) - Use (functionality, content) - Customisation - Support
• UX in the small (interaction, engagement)
-Presence- user as interactor
-Immersion in the interactive world
-Flow and engagement in the interactive world
-Social presence
Components of User Experience
Engagement and Interaction
Interaction with avatars is more engaging if … - They adapt to the user - Use surprise and are occasionally unpredictable
Interaction
User Experience & Engagement
Arousal
Emotion
Media
Pace Flow
Complexity Change
Content Functionality
Presence
Immersion Human Image & Voice
3D Worlds Avatars
increases
promotes
+ve exp induces +ve
Cost Benefit
Need / Specificity
Engagement and Attractiveness- revised framework:
Content services
Reputation Identity
Customis- ability
Aesthetics
Engagement
Usability
Design Quality
Presence Metaphor Interaction
Flow Continuum
All task / context / use dependent
High level impression
Interactive experience
UX over time
Compared 3 web sites – IKEA avatar virtual shop assistant – NIKE animation and customisation – ALDI baseline
Tasks search + interact with features
Same tasks and sites 3 visits separated by 1 week
Hypotheses - character will improve engagement - interaction (customisation) will improve
engagement - effects stronger after more experience
IKEA- virtual character
NIKE customisation
ALDI Ad pop ups
Preferences & ratings
• No significant difference Nike- IKEA
• Preference and rating order the same 1,2 weeks later
• But rating of criteria changed aesthetics, usability earlier, content, brand later
• Avatar (IKEA) not engaging but animation was • Customisation (Nike) not engaging but animation was • ALDI pop ups disliked
Evaluating User Experience (Sutcliffe & Hart 2011)
• Observation (within session) - user activity - body posture, gestures, facial expressions - attention
• Questionnaire (post session) - existing scales- expressive aesthetics + new scales Presence (from VR), Flow-engagement, Media, and immersion - memory recall- salient features, episodes
• Physiological Measures (in session) - heart rate - GSR - pupilometry & eye tracking
Observation
• Activity - system logs or sampling: pace, critical incidents & breakdowns, task v error repair actions
• Attention - gaze on screen/in world v elsewhere - could supplement with eyetracking but expensive
• Non Verbal Communication - post session analysis from video records - rate posture for arousal/excitement - facial expression for emotion
Questionnaires I
General Engagement • Rate your general mood after using the application (positive happy
… negative depressed) • Rate the strength of your feelings/ emotions: Pleasure, Joy,
Surprise, Sadness, Anxiety, Worry, Fear, Frustration, Disgust.
Interaction/Flow • Rate the pace of interaction (too slow, about right, too fast) • How challenging was operating the interface? (too easy, about right,
too difficult)
Media CASA • Did you notice any images of people? (not at all … very much) • How attractive were the images used in the application?
Questionnaires II
Presence • How natural did your interactions with the application feel? • How aware were you of events occurring in the real world around
you?
Immersion • How compelling was your sense of moving inside the interactive
world? • How natural did the interactive world appear to be?
Social Presence • How aware were you of the person you were communicating with? • How well did the application communicate the identity of other
people?
Summary & Conclusions
• User Experience is multi-faceted and will change over time
• It can be measured/systematically evaluated
• UX is context (domain) dependent
• Our judgement of UX suffers from biases and framing effects
• UX components - in session engagement: avatars, virtual worlds and user as
actor- motivate and attract - across session: personalisation, utility, challenge and
adaptation
Thanks for your attention For more information
• Sutcliffe, A.G. (2003). Multimedia and virtual reality: Designing multisensory user interfaces. Mahwah NJ: Lawrence Erlbaum Associates.
• Sutcliffe, A. G. (2009). Designing for user engagement: aesthetic and attractive user interfaces. In Carroll, J.M. (Ed), Synthesis lectures on human centered informatics. San Rafael CA: Morgan Claypool.
• Sutcliffe A.G. & de Angeli A., (2005), Assessing interaction styles in web user interfaces. In Proceedings of Human Computer Interaction INTERACT 05, Eds Costabile M.F and Paterno F., Rome Sept, 2005, Springer Verlag. pp 405-417.
• De Angeli, Sutcliffe A.G. & Hartmann J. (2006) Interaction, usability and aesthetics: What influences users’ preferences? In Proceedings of DIS 2006, Designing Interactive Systems, ACM Press.
• Hartmann J., Sutcliffe A.G. & de Angeli A. (2007), Investigating attractiveness in web user interfaces. in CHI07, Human Factors in Computing Systems, San Jose, CA, ACM Press.
• Hartmann, J., Sutcliffe, A. G., & De Angeli, A. (2008). Framing the user experience: Information biases on website quality judgement. In Proceedings of CHI-08. New York: ACM Press.
• Hartmann, J., Sutcliffe, A. G., & De Angeli, A. (2008). Towards a theory of user judgment of aesthetics and user interface quality. ACM Transactions on Computer-Human Interaction, 15(4), 15-30.
