Upload
others
View
9
Download
0
Embed Size (px)
Citation preview
UX Research test scenarios
KP Ludwig John
UX Researchtest scenarios
UX Research test scenarios
KP Ludwig John
Usability TestPreconditions
Already existing
• PersonasDefinition of target groups
• Context of usewhere, when, how long, equipment
• Test goalsWhat do we want to learn?!
UX Research test scenarios
KP Ludwig John
• What do you want to learn by the test?Which elements of the IA design have been discussed controversial?Which elements seam to be critical for the general success of the application?
• Which elements of your application have to be coveredMain functions / use casesCrucial functions of the applicationUse Cases with many Alternate Flows
Usability-Test PredefinitionsTest goal
UX Research test scenarios
KP Ludwig John
Usability-Test
Comparison of 3 print Layouts Brochure with information about companies
IMS.UX - 2017
Test goal
Source: Team UXperts, IMS.UX 2017
Which Layout is best?
UX Research test scenarios
KP Ludwig John
Usability-Test
Source: Team UXperts, IMS.UX 2017
Aspects to consider
Extracting Information How easy is it to get the information? What information can you get?
Memorizing InformationCan you memorize the information after reading? Is it easy to remember
ClarityIs the structure obvious to the user? (Naming, Positioning, Design of elements)
ComprehensibilityDoes the user understand the content?
Readability - Is the text easy to follow?
Look & Feel - How is the first impression? Do you have a good feeling reading the paper?
Which Layout is best?
UX Research test scenarios
KP Ludwig John
Usability-TestAspects to consider
How well is information communicated?
Which Layout is best?
How fast can information be perceived and processed?
How much attention does relevant information get?
Measuring units
Related questions
?
UX Research test scenarios
KP Ludwig John
Usability-Test
Source: Team „Group Title“, IMS.UX 2017
Aspects to consider
How well is information communicated?
Which Layout is best?
How fast can information be perceived and processed?
How much attention does relevant information get?
Measuring unit: Time to complete tasks.
Measuring unit: Time a test subject’s focus rests on relevant information.
Measuring unit: Amount of misunderstandings and missed information
Measuring units
UX Research test scenarios
KP Ludwig John
paper based software based
Usability-Test
• Which functionality has your prototype to providein order serve your test goals?
PredefinitionsTest object
• What kind of Prototype is needed in order serve your test goals
UX Research test scenarios
KP Ludwig John
Usability-Test
• Diary study (remote) • Thinking aloud (with note taking) • Video observation • Screen recording • Eye tracking • Post test interview
PredefinitionsTest methodWhich test method do you want to apply?
UX Research test scenarios
KP Ludwig John
Usability-Test
• What kind of equipment serves best the test goals? • Consider testing under of real context of use conditions
(Screen size, Space, time, mobile, distraction, sound)
PredefinitionsTest equipment
UX Research test scenarios
KP Ludwig John
Usability-Test PredefinitionsTest scenario
Create a test scenario which defineseach single step of the test in detail,
including
•Introduction •Tasks •and Expression of Thanks
UX Research test scenarios
KP Ludwig John
Test location • Where do you want to test? • Availability, Influence of Weather or Light?
Usability-Test PredefinitionsTest space
Test equipment • Where do you get it? • How to handle it?
UX Research test scenarios
KP Ludwig John
• Test facilitator • Technical manager • Observer • Analyst
Usability-Test PreparationTest team
UX Research test scenarios
KP Ludwig John
• Where can the test persons be recruited from? (do they represent the target group, are they available (time) ?) • Do they get some expense allowance (Aufwandsentschädigung)? (which?) • recruit test persons
Usability-Test PreparationTest persons
Find test persons make appointments and a test schedule
• How many Test persons are needed? (about 6-7 per Test version (!) )
UX Research research methods
overview
KP Ludwig John
UsabilityTest persons
UX Research test scenarios
KP Ludwig John
It’s best to test with representatives of your main target group.But don’t be obsessed about it!If its difficult to recruit them, test with people you can get hold of. Take it in account when evaluating the test results:Would our users have that problem? Or was it only a problem because the participant wasn’t familiar with the field?
Rather few but frequentlystart early in the development processand repeat and repeat and repeat
Who do the test with?
Usability Test Test persons
How many test persons?
UX Research test scenarios
KP Ludwig John
Test personsUsability Test
source: www.nngroup.com/reports/how-to-conduct-eyetracking-studies PDF, page 19
UX Research test scenarios
KP Ludwig Johnreference:Nielsen 2018: https://www.youtube.com/watch?v=RhgUirqki50
Test personsUsability Test
Jacob Nielsen N/N Group
UX Research test scenarios
KP Ludwig John
reference:Nielsen (2000) https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/
Test personsUsability TestHow many test persons?
The best results come from testing no more than 5 users and running as many small tests as you can afford.
UX Research test scenarios
KP Ludwig John
Quelle: Steve Krug, Don‘t make me think! Redline, 2006
Test personsUsability Test
Test with 8 usersfound problems: 5 major ones
How many test persons?
UX Research test scenarios
KP Ludwig John
Quelle: Steve Krug, Don‘t make me think! Redline, 2006
Test personsUsability TestHow many test persons?
2 Tests with 3 users each = 6 Test personsfound problems: 5 major + 4 minor = 9 problems
UX Research test scenarios
KP Ludwig John
Test personsUsability Test
Each point represents a single set of randomly sampled users. The horizontal lines show the mean for each group of 100.
The effect of adding users on reducing variance in the percentage of known usability problems found.
Source: https://link.springer.com/content/pdf/10.3758/BF03195514.pdf
How many test persons?
UX Research test scenarios
KP Ludwig John
Test personsUsability Test
Faulkner (2003): Beyond the five-user assumption
Benefits of increased sample sizes in usability testing Source: https://link.springer.com/content/pdf/10.3758/BF03195514.pdf
UX Research test scenarios
KP Ludwig John
Variables …, such as types of test users available or accessible to the practitioner, … or the potential consequences of any particular usability problem, can have a profound impact on the number of test users required to obtain accurate and valid results.
Test personsUsability Test
Faulkner (2003): Beyond the five-user assumption
Benefits of increased sample sizes in usability testing Source: https://link.springer.com/content/pdf/10.3758/BF03195514.pdf
Despite practitioners’ love of the 5-user assumption, the answer to the question, “How many users does it take to test the usability of an interface?” remains a vague, “It depends.”
How many test persons?
UX Research test scenarios
KP Ludwig John
Most significant problems will be discovered by small number of test users already.
Finding three participants is easy to handle:test and debrief (Nachbesprechen) the same day! Design small tests with focussed tasks.Learn, move on and test again.
Test personsUsability Test
Reference: Steve Krug: „Rocket Surgery Made Easy“, New Riders 2010, chapter 5
How many test persons?
Can you use the same participants again?Not for the same task or application.They know too much already. ?
UX Research test scenarios
KP Ludwig John
Test scenarios in detail
UX Research test scenarios
KP Ludwig John
1. Test person works on given tasksoften under direct observation(screen recording software, video, human observer)
2. Interview with the userto better understand critical situations during the test.Possibility to discuss improvements together
Usability-TestTwo Steps
UX Research test scenarios
KP Ludwig John
Content and style
Test scenario
Identify at least 3 essential activitiesto be covered by the test
Create 1 or more tasks per activity to be followed by the test person
Example:
„Become a new member of this online communityand create your individual account!“
„Which of the dishes offered at Mensa next Monday you prefer? Note it on paper.“
UX Research test scenarios
KP Ludwig John
Content and style
• complex enough (not to simple) • not too complex either (shouldn’t take too long to work on each single task) • cover all important issues of your test
Test scenario
Tasks have to be
UX Research test scenarios
KP Ludwig John
Content and style
• create realistic tasks Users should try to do the same things in Lab as they would do at home or at work
Test scenario
Tasks determine looks!
Nielsen/Pernice: Eyetracking Web Usability, New Riders, Nielsen Norman Group 2010
Important!
UX Research test scenarios
KP Ludwig John
Tasks determine looks!The Monkey Business IllusionSelective attention test by Daniel Simons 2010
https://youtu.be/IGQmdoK_ZfY
Count how many times the players wearing white
pass the ball
UX Research test scenarios
KP Ludwig John
Tasks determine looks!The Monkey Business IllusionSelective attention test by Daniel Simons 2010
Count how many times the players wearing white
pass the ball
?Result
UX Research test scenarios
KP Ludwig John
Tasks determine looks!The Monkey Business IllusionSelective attention test by Daniel Simons 2010
https://youtu.be/IGQmdoK_ZfY
Count how many times the players wearing white
pass the ball
UX Research test scenarios
KP Ludwig John
Tasks determine looks!
Reference: http://www.theinvisiblegorilla.com
These experiments reveal two things:
• we are missing a lot of what goes on around us
• we have no idea that we are missing so much
The Monkey Business IllusionSelective attention test by Daniel Simons 2010
A variation of one of the best-known experiments in psychology
Book first published in 2010
The Invisible Gorilla experiment from 1999
UX Research test scenarios
KP Ludwig John
Test scenario Tasks determine looks!
Same content
Nielsen/Pernice: Eyetracking Web Usability, New Riders, Nielsen Norman Group 2010
Website of Usability Week Conference
Task 1:Decide which of two conferences rather to attend
Task 2:Decide which seminar to choose
from that conference
UX Research test scenarios
KP Ludwig John
Test scenario Tasks determine looks!
Nielsen/Pernice: Eyetracking Web Usability, New Riders, Nielsen Norman Group 2010
People look differently at the same information,depending on what they are trying to do.
In both cases: • Heatmap visualisation of Eyetracking Recordings
(objective measurement) • identical content
But: • Different users interest (determined by the task)
Same content - Different Tasks
UX Research test scenarios
KP Ludwig John
Test scenario Tasks determine looks!
NN-Group 2017https://www.youtube.com/watch?v=XU1-Rz2Q7-E
UX Research test scenarios
KP Ludwig John
Test scenario Tasks determine looks!
source: www.nngroup.com/reports/how-to-conduct-eyetracking-studies PDF page 145
Task: Find the US population and that of the state Texas Looks: Menus, content headings, population clock at the right
Screenshots from us census bureau (statistics)Same content - Different Tasks
UX Research test scenarios
KP Ludwig John
Test scenario Tasks determine looks!
source: www.nngroup.com/reports/how-to-conduct-eyetracking-studies PDF page 146
Task: What do you think of this page? Comment on it! Looks: All over … This will not happen in real user situations!
Screenshots from us census bureau (statistics)Same content - Different Tasks
UX Research test scenarios
KP Ludwig John
Test scenario Tasks determine looks!
source: www.nngroup.com/reports/how-to-conduct-eyetracking-studies PDF page 147
If testing just one page
Don’t open the page and ask the user to “Please look at the page and tell us what you think.”This encourages the user to explore a page in a way they probably never would.
! Do ask the user to do a task, then launch the page
Don’t open the page, let the user examine it, then give him a task. Doing these steps in this order would encourage a user to examine the page when he might have never done this on his own.
UX Research test scenarios
KP Ludwig John
Test scenarioAvoid Priming
source: https://www.nngroup.com/articles/priming/
Fill in the blanks to form an english word:
B_ _ T_RWhat comes to your mind spontaneously??
UX Research test scenarios
KP Ludwig John
Test scenarioAvoid Priming
source: https://www.nngroup.com/articles/priming/
Fill in the blanks to form an english word:
B_ _ T_RWhat comes to your mind spontaneously?
“The aroma of warm bread spread tantalizingly out of the bakery.”
Read the following sentence:
UX Research test scenarios
KP Ludwig John
Test scenario
source: https://www.nngroup.com/articles/priming/
The mention of “bread” is likely to have activated concepts that are associated to it and thus may have made you think of the word “butter” over the many other fitting alternatives.
… “barter,” “banter,” “beater,” “bitter,” or even “better.” …
Avoid Priming
Chances are that your word was “butter.”
B_ _ T_R
UX Research test scenarios
KP Ludwig John
Test scenario
source: https://www.nngroup.com/articles/priming/
Exposure to one stimulus (initial sentence about the aroma of bread)
influences a subsequent response to a different stimulus (template to fill in)
Avoid Priming
UX Research test scenarios
KP Ludwig John
Test scenario Avoid Priming
NN-Group 2019source: https://www.youtube.com/watch?v=LL_8ItbPN8Q
UX Research test scenarios
KP Ludwig John
Test scenario
source: https://www.nngroup.com/articles/priming/
A question such as “What do you think this button does?” can warn users that the indicated UI element is a button, although they may not have realized that it was active.
The facilitator can elicit (hervorlocken) a certain behavior by demonstrating it to the user.
Facilitator’s behaviorAvoid Priming in User Research
In a children study, we redirected a lost user back to the browser’s homepage by pressing the browser’s Back button several times.
Although before that event the child had never used the Back button, afterwards she used it in almost every task.
UX Research test scenarios
KP Ludwig John
First steps
Determine:
What do you want to find out?Which elements of the application are crucial?
Formulate the tasks straight forward, short and precise
Each task should have a defined goal(that defines, when the task is done)
Test scenario Content and style
!Start with simple, easy to handle tasks!More difficult (or more complex) tasks are possible,but not good at the very beginning of the test)
UX Research test scenarios
KP Ludwig John
Test scenario Content and style
UX Research test scenarios
KP Ludwig John
Test conductor although positioned next to the proband,remains silent / interferes only in critical situations.
Don’t get into conversations with the test person during the test itself. That can be done after the test.
Test scenario Content and style
Let the test person act rather independently and on its own
UX Research test scenarios
KP Ludwig John
Test scenario Content and style
Try out the tasks with team members or random usersbefore starting the official test
Set up test conditions close to the real context of use
UX Research test scenarios
KP Ludwig John
Test scenario Structure
Source: Team „Group Title“, IMS.UX 2017
• My name is Martin and I am guiding you through this test • First, we would like to thank you for participating in our test, we really appreciate your help and the time you took for this.
• Very important before we get started: We are not going to test you in person, we try to test different layouts for a magazine, so don’t worry, you can’t do or say anything wrong.
1. Welcoming and short introduction by test leader
UX Research test scenarios
KP Ludwig John
Test scenario
Source: Team „Group Title“, IMS.UX 2017
• If there are any questions or misunderstandings, just ask.
• Vivi, Kate, Daniel and Chris are here to observe the whole testing sequence. Ruben is our technical manager.
• You will have to do some tasks here in the UX-Lab, which I am going to explain later in detail.
• Bastian is a member of Academica. He will guide you to another room where you are going to get your last task,which is answering some questions in a questionnaire.
2. Introduction of the team members and their functions
Structure
UX Research test scenarios
KP Ludwig John
Test scenario
Source: Team „Group Title“, IMS.UX 2017
For the following test here in the UX-Lab, my team members will be observing the test.
We would like to do audio recording during the test. We are not going to provide this data to 3rd parties, but if you don’t want your voice being recorded, feel free to tell us.
3. Declaration of consent
Structure
UX Research test scenarios
KP Ludwig John
Test scenario
Source: Team „Group Title“, IMS.UX 2017
There will be 3 different PDFs which contain several information.
• You have to do some tasks, they will be displayed on the screen. Pressing space on the keyboard will start the first question,
• We would like to encourage you to tell us what you are thinking while doing the tasks. It helps us analyzing your final answers …
• … you can start by pressing the SPACE key
• After a short calibration of the eye-tracking software the actual test is going to start immediately.
4. Definition of tasks
Structure
UX Research test scenarios
KP Ludwig John
oder:
Testszenarien entwickeln
Aufbau
Beispiel: IA6 2013 A Serious Game (Kinect)
Begrüßungstext
UX Research test scenarios
KP Ludwig John
Tasks
Source: IA6 2018, Team Gravecare
Test scenario Content and style
Each task should have a clear goal(that defines, when the task is done)
More specific = better
UX Research test scenarios
KP Ludwig John
Tasks
Source: IMS.UX 2017, Compare 3 Print-Layouts, Team UXperts
Test scenario Content and style
What’s the address of the company?Layout is shown, subject starts task.Subject gives answer
Does the company offer topics for Bachelor Thesis?Layout is shown, subject starts task.Subject gives answer
How many employees work at the company?Layout is shown, subject starts task.Subject gives answer
Each task should have a clear goal(that defines, when the task is done)
More specific = better
UX Research test scenarios
KP Ludwig John
Test scenario
source: https://www.nngroup.com/articles/priming/
When tasks use terms already present in the interface, participants are primed by the task formulation and tend to look for those exact terms.
When asking people to search for an “iPad keypad.” Although the word “keypad” is not as frequent as “keyboard”, most users typed “keypad” instead of “keyboard” in the search box.
… the test may have yielded the wrong conclusion that the product was easy to find.
Task wording
Content and style
UX Research test scenarios
KP Ludwig John
Tasks
Source: IMS.UX 2017, Compare 3 Print-Layouts, Team Xperience
Test scenario Content and style
Questions indicates meaning of UI element
Look for the rating points in the pdf. Find one criteria which has a bad rating and tell us.
Subject gets 2-3 minutes to look closer at the layout.We ask the user to talk about the parts he is looking at.
- Some critical examples
No clear goal, unnatural situation
Why critical?
UX Research test scenarios
KP Ludwig John
Expression of Thanks
Post test Interview to collect impressions and to discuss critical situation observed during the test
probably:
Test scenario Content and style
or probably
Thank you very much for participating in our test.
We would like to offer you this little gift to thank you.
After completing the tasks
UX Research test scenarios
KP Ludwig John
Test scenarioDuration
Rule of Thumb:Between 10 -30 minutes per person is OK
• less than 10 minutes seems rather short as well to the test person as for the team
• longer than 30 minutes requires a lot of effort from both sides
depends on test goal and circumstances
UX Research research methods
overview
KP Ludwig John
PracticePrepare your test
Formulate your test scenario in detail!
Plan and organize your test:
Appoint Functions and responsibilities in the test team Find and appoint test persons Schedule the test (incl. equipment and location)
UX Research research methods
overview
KP Ludwig John
PracticePrepare your test
• Test goals • Criteria of Success and Measurements for that criteria • Target group • Prototype • technical equipment • scenario • post test interview (if applicable)
Summarize your test design into one document, including
Be prepared to discuss it in detailnext week
hs-augsburg.de/homes/john