Upload
keylimeinteractive
View
6.480
Download
0
Embed Size (px)
Citation preview
Exploring Eye Tracking for Games User Research! A case study of lessons learned
Ania Rodriguez, M.S.CEO/FounderKey Lime Interactive@KeyLimeInteract
Karl Steiner, Ph.D.Manager of UsabilityTHQ, Inc@THQ
© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI
2
Overview
Introductions
Player Research Goals, Techniques and Challenges
Think Aloud vs. Debrief
Our Game Plan
Steps to Success
Q&A
© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI 7/26/2010
3
About Key Lime Interactive Experienced market research and usability
professionals Founded January 2009 Expertise in both Eye-Tracking in-lab studies as
well as remote usability testing techniques
Cross-Industry Expertise Ability to scorecard against competition Ability to suggest cross-industry
recommendations to improve ease of use
Independent Third Party Impartial with no-hidden agenda Ability to work with business team and
developers to “get everyone onboard”
© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI
Clients:
7/26/2010
4
About THQ
THQ Inc. (NASDAQ: THQI) is a leading worldwide developer and publisher of interactive entertainment software. The company develops its products for all popular game systems, personal computers and wireless devices. Headquartered in Los Angeles County, California, THQ sells product through its global network of offices located in North America, Europe and Asia Pacific.
© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI 7/26/2010
7
Usability ≠ Playability
For games, the user experience is the primary purpose of the product Games must not only be usable… But also enjoyable
Part of the appeal of (many) games is overcoming a level of challenge Games should be difficult… But not too difficult
Games employ a variety of devices, used by a variety of players, in a variety of physical settings Despite common stereotypes, gamers represent an
increasingly diverse group Gameplay styles and techniques that are
appropriate for one audience in one setting playing one type of game may not be acceptable in other contexts
© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI 7/26/2010
8
Player Research Goals & Techniques Player Research Goals
Describe and understand common players behaviors within a game
Determine where and why players encounters issues or difficulties when playing a game
Assess when and how players enjoy the experience of playing the game
Player Research Techniques Qualitative Techniques
Lab-based “usability” style tests Quantitative Techniques
Surveys and logs of player interactions
© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI 7/26/2010
9
Using Observation and “Think Aloud” to diagnose player difficulties in a 3D Game
We can observe if player is having difficulties Fails a task Takes too long / too many attempts Gets visibly frustrated
We require additional insights to determine why player is having difficulties May observe player taking incorrect actions or using incorrect
controls Player may verbally describe incorrect assumptions about their
goals or incorrect plans for achieving objectives (Think Aloud or Post-Session Debrief)
© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI 7/26/2010
11
Objects in 3D World
EnemiesEnemiesEnemies
BuildingsEnemiesBuildingsTerrain
© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI 7/26/2010
12
Heads-Up Display (HUD)
Score / Timer
Target Reticle
Mini-mapTarget ReticlePlayer Status
© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI 7/26/2010
13
Issues With Think Aloud for Player Research in 3D Games
Many gameplay issues are related to player awareness and understanding of in-game cues and objects
However, player observation and think-aloud techniques do not always provide adequate information regarding player awareness and understanding of these elements
Players may notice details but fail to understand or act on the information (making observation difficult)
Players may not mention details they noticed while they play
Players may not remember details if they are queried after play
© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI 7/26/2010
14
Our Game Plan
Take a 3D game that has already had been tested with Think Aloud Protocol and rerun the study with Eye-Tracking
Gather additional insights into player gaze and awareness, such as: What objects in the 3D world are noticed? When are they noticed? Do
players comment on or remember these objects? What objects in the 3D world are missed? What elements in the HUD are noticed? At what points are they noticed?
Do players comment on or remember these HUD elements? What elements in the HUD are missed? Do overall gaze patterns change in different areas of the game?
© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI 7/26/2010
How we did itLessons learned from running our first eye-tracking study in Gaming
© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI
16
Step#1: Computing Power
You need to network the eye-tracker to a separate PC or gaming console.
Pre-release games may not have been tuned for performance, so high-end hardware is typically required. If testing on a PC, consider: CPU – A high-end recent CPU (e.g. Intel Core 2 Quad) Video Card – Dedicated video card (laptops with integrated video or low-
performance cards will likely not be sufficient) Video output – make sure that your hardware supports the necessary game
output (e.g. HDMI, VGA, etc.) and is compatible with the eye-tracker hardware
RAM – 4GB (8GB is always better)
Our gaming computer: Gateway FX6831-01 PT.GAH02.004 Desktop PC - Intel Core i7-860 2.8GHz,
8GB DDR3, 1.5TB HDD, DVDRW, ATI Radeon HD5850, HDMI, Display Port, eSATA, FireWire, Windows 7 Home Premium
© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI 7/26/2010
17
Monitor #1 with
SMI EyeTracker
XboxController
Monitor #2 with
SMI ExperimentCenter Software
Laptop runningSMI iView X™ Software
The Gaming Machine
Step#2: Double PC Setup
© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI 7/26/2010
18
Step#3: Testing Protocol
In a game study, some players may not volunteer much information using just a Think Aloud.
The game itself may demand their attention at times, and describing the experience can introduce additional cognitive load or interruptions to game play
Our test scenarios included a natural stopping point midway through the study
Our suggestion: Use ‘debrief’ rather than Think Aloud because: Ability to observe what the participants would naturally do
without disrupting how they would play if they were on their own Adding Think Aloud to this experience adds more cognitive load
to the task than normally would be present
© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI 7/26/2010
19
Step#4: Clarify Goals and Terminology
Clearly stated test goals and plans are important for any domain
Fleeting game events and unique game terminology makes prior planning and preparation even more important
Our suggestion: review test goals with project sponsors in front of the game Game events may appear quickly – observers need to know what to
look for and at what time Game terminology can often be arcane and very specific. For
example, an observer may need to know the visual differences between: Space Marines and Chaos Marines M-16s and AK-47s Atomic Drops and a Facebreaker DDTs
© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI 7/26/2010
20
Step #5: Run Study
8 gamers with third-person shooter experience 7 males, 1 female
1 day of testing; 60 minute sessions
30 minutes between subjects
Session overview: Scene 1 (10-15 mins) Debrief (5-10 mins) Scene 2 (10-15 mins) Debrief (5-10 mins)
One participant, 1 moderator
2 Observers/Note Takers
© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI 7/26/2010
21
Step #5: Analysis
Fixations are generally an inadequate way of measuring subject attention when viewing dynamic media. This is because a subject, when tracking a moving target (like a
soccer ball), is actually performing smooth pursuit. This is similar to fixation, at least cognitively, but it is not detectable via normal fixation detection algorithms.
Instead, we recommend dwell time. This is the summary of time spent within an area whether in fixation, saccade, or smooth pursuit, and is a better way to quantify attention on a moving object.
Glance time is a related and equally useful measure.
© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI 7/26/2010
22
Definitions
Dwell Time: Dwell time starts at the moment the AOI is fixated and ends at the moment the last fixation on the AOI ends(equal to the sum of durations from all fixations and saccades that hit the AOI)
Glance Duration = Saccade duration for entering the object + sum of all fixation durations and saccade durations before the eyes begin to leave the AOI(equal to the dwell time + duration of saccade entering AOI)
7/26/2010© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI
23
Step #6: Gridded AOIs [Dwell Time]
© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI
After 30 seconds End 60 seconds
Example: Gridded Areas of Interests (AOIs) for E3 Promotional Video
7/26/2010
24
Step #7: AOIs and KPIs
© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI
Create AOIs
• AOIs can be • Any shape or form• Made invisible (turn on/off)• Enabled/Disabled
7/26/2010
25
Step #8: Example of Dwell time on Custom AOIs
© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI
Pick object(e.g. skull)
7/26/2010
26
Step #9: Create Report
Create typical usability report with findings Executive Summary Methodology Findings and Recommendations Detailed Findings and Metrics Appendix
Present Include everyone you can (e.g., usability team, senior
leadership, developers, project managers, etc.)
© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI 7/26/2010
27
Step #10: Highlight Videos
© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI
Show Sample VideoEye-tracking playback of the online publically available promotional video on THQ’s site
7/26/2010
© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI
Ania Rodriguez, M.S. Key Lime Interactive Founder [email protected] @keylimeinteract
Karl Steiner, Ph.D. Manger of Usability Research