2/16/05 Usability Testing for ISD - Leslie Carter 1
Instructional Technology & Usability
MATTERS
Leslie Carter, Ph.D.
Instructional Technology & Usability
1. How important is usability?
2. What do we mean by “usability?”
3. How do we evaluate usability?
2/16/05 Usability Testing for ISD - Leslie Carter 3
According to Instructional Designers
• 99% Agree: Usability is essential to e-Learning design.
• 96% Agree: e-Learning components should always be tested for usability.
• 96% Agree: User-centered design enhances the effectiveness of e-Learning.
From the E-Learning Guild’s “Usability and e-Learning” survey reported 2/27/2004.
2/16/05 Usability Testing for ISD - Leslie Carter 4
According to Instructional Designers
What are the "Top 5" features you would recommend that someone consider when licensing a live e-learning product?
Ease of Use was cited by more e-learning professionals than any other factor.
From Brandon-Hall.com winter 2004 survey.
Instructional Technology & Usability
1. How important is usability?
2. What do we mean by “usability?”
3. How do we test for usability?
2/16/05 Usability Testing for ISD - Leslie Carter 6
Usability is“The effectiveness, efficiency, and satisfaction with which specified users achieve specified goals in particular environments.” (ISO 9241)
1. Effectiveness - Accuracy & completeness.
2. Efficiency – Speed & ease of use.
3. Satisfaction - Comfort & acceptability.
2/16/05 Usability Testing for ISD - Leslie Carter 7
What contributes to usability?1. Visual
Design
2. Information Architecture
3. Navigation Design
4. Interaction Design
5. User Interface Design
A. The overall process of designing how a user will interact with a software application or web site, including: requirements analysis, information architecture, interaction design, visual design, user testing, documentation, & help system design.
B. The design of the graphic layout, & the images, icons, fonts selected to support the informational content.
C. Information organization that should help people find & use information efficiently. It’s concerned with things like content chunking, formatting, categorizing, labeling, placement & navigation.
D. The design of the workflow through an application or site, including the dialog between the user & the system, what user inputs are needed & when, as well as, what feedback is appropriate & when.
E. The design and placement of menus & options used to find specific areas of a software application or site.
2/16/05 Usability Testing for ISD - Leslie Carter 8
What contributes to usability?1. Visual
Design
2. Information Architecture
3. Navigation Design
4. Interaction Design
5. User Interface Design
A. The overall process of designing how a user will interact with a software application or web site, including: requirements analysis, information architecture, interaction design, visual design, user testing, documentation, & help system design.
B. The design of the graphic layout, & the images, icons, fonts selected to support the informational content.
C. Information organization that should help people find & use information efficiently. It’s concerned with things like content chunking, formatting, categorizing, labeling, placement & navigation.
D. The design of the workflow through an application or site, including the dialog between the user & the system, what user inputs are needed & when, as well as, what feedback is appropriate & when.
E. The design and placement of menus & options used to find specific areas of a software application or site.
2/16/05 Usability Testing for ISD - Leslie Carter 9
Guidelines for Good Usability
Learning these will help you:• Recognize the cause of problems found
in usability testing • Find solutions to those problems• Reduce the number problems found in
future usability testing
A Comparison of Heuristics from Several “Gurus”
Review the usability heuristics (guidelines) published by researchers & practitioners.
2/16/05 Usability Testing for ISD - Leslie Carter 10
Role of Usability in Evaluation
• Increased cognitive load• Wasted time• Frustration & reduced motivation
Poor usability leads to:
Great training content can lead to lousy training results when poor usability is involved!
Instructional Technology & Usability
1. What does the beginning of the field of usability have in common with instructional technology?
2. What do we mean by “usability?”
3. How important is usability?
4. How do we test for usability?
2/16/05 Usability Testing for ISD - Leslie Carter 12
Conducting a Usability Test
• Determine goal of test • Select evaluation criteria• Select participants• Write a task scenario• Write a testing protocol• Conduct the test• Track the observations• Report the results
2/16/05 Usability Testing for ISD - Leslie Carter 13
Testing Goals1. Determine how to improve a developing
product– Formative: product under construction, could be
even be a prototype– Informal: quantitative measure not required,
qualitative observations needed
2. Compare two different products or versions of same product– Summative: products are mature & ready for the
marketplace– Formal: quantitative measures required
2/16/05 Usability Testing for ISD - Leslie Carter 14
Evaluation Criteria• Most common
– No. of tasks completed– Time to complete task– Errors made in performance– Satisfaction
• Other possible measures– Proportion of time spent determining what to do
instead of doing it– No. of times consulted help or user guide– No. of positive or negative comments from user
2/16/05 Usability Testing for ISD - Leslie Carter 15
Usability is“The effectiveness, efficiency, and satisfaction with which specified users achieve specified goals in particular environments.” (ISO 9241)
1. Effectiveness - The accuracy & completeness.
2. Efficiency – Speed & ease of use.
3. Satisfaction - Comfort & acceptability.
2/16/05 Usability Testing for ISD - Leslie Carter 16
Participants
• Similar to your audience – training, experience…
• Availability – when & for how long.
• Where – your place or theirs, theirs is better if like that of your target audience. Consider remote testing if you can’t get to them.
2/16/05 Usability Testing for ISD - Leslie Carter 17
Tasks Scenario• Select tasks: critical &/or frequent• Develop a realistic scenario for sequencing
those tasks• Start with easy tasks to build user’s
confidence• For each task give user a goal to achieve
instead of steps to perform• Just enough info for user to complete the
task without more info from observers• Make it clear to user when each task is
compete
2/16/05 Usability Testing for ISD - Leslie Carter 18
Sample Scenario Outline
Task 1 - View a webshopTask 2 -Create a webshopTask 3 -Revise a webshopTask 4 -Create a course
(several webshops put together)Task 5 - Revise a course
2/16/05 Usability Testing for ISD - Leslie Carter 19
Sample Tasks
• A page with one large graphic (Image-1large.jpg). Title this page “My First Page.” Add this description “my webshop.” Check to see how this page will look to the site’s viewers before starting on the next one.
• A page with a video clip (TinyMovie3.mov), and an Acrobat PDF file (acrobat6matrix.pdf). Title this page “My Second Page.”
Task 1 – View a WebshopStating from the T/TAC home page, find “Leslie’s Webshop” & view the entire webshop.
Task 2 - Create a WebshopYour goal for our first task is to create a simple webshop made up of several kinds of media. Name this webshop with your name and the date, e.g., “Leslie’s 3/1/04 Webshop.” This webshop will include the sample media we brought & consist of 2 pages:
2/16/05 Usability Testing for ISD - Leslie Carter 20
Testing Protocol
• Intro for participants• Reminders to observers• Copy of the task scenario• Steps required to perform each task• Make a separate copy for each
observer & for each test
2/16/05 Usability Testing for ISD - Leslie Carter 21
Sample Protocol – Participant Intro
• We are not testing your ability to use this site; we are testing the sites’ ability to meet your needs.
• Your feedback will help us ensure that T/TAC Online will meet the needs of T/TAC professionals.
• The site is designed to allow T/TAC staff to upload training content for presentation over the Internet. There are two kinds of presentations we’ll try today. One is a webshop which is a presentation designed to last 15-30 minutes, something a viewer might complete in one sitting. The other is a course which is a set of webshops combined in one course.
• If you would like to take a break at anytime, just let us know.• Please try to think out loud as you use the site. Let us hear
the questions in your mind. We need to know if you can’t find something or if something is not absolutely clear to you. And, let us hear about the features you like about the site.
2/16/05 Usability Testing for ISD - Leslie Carter 22
Sample Protocol – Conducting the Test
Task Steps
1 – View a WebshopStating from the T/TAC home page, find “Leslie’s Webshop” & view the entire webshop.
1. Click on webshop2. Click Continue button
until returned to start
Task 2 - Create a WebshopYour goal for our first task is to create a simple webshop made up of several kinds of media…
1. Online Training tab2. Add a Webshop button3. Enter profile info4. Template button
Reminders:• Don’t tell user the steps, unless they’re very frustrated• Don’t give away the names of the buttons or links• Note start & end times, step where error occurs• Get positive & negative quotes from user
2/16/05 Usability Testing for ISD - Leslie Carter 23
Tracking Results
• It’s very easy to forget or loose site of useful observations
• Need a system for tracking all the issues you want to address
• Track who noticed the issue, when noticed, what kind of issue, what screen, how important, & when fixed.
2/16/05 Usability Testing for ISD - Leslie Carter 24
Reporting on a Test
• MUCH stronger impression is communicated if you have sample user comments from videotapes or Camtasia screen recordings.
• There is an industry standard for usability test reporting from NIST. Very complete. The checklist is great. http://zing.ncsl.nist.gov/iusr/documents/cifv1.1b.htm
2/16/05 Usability Testing for ISD - Leslie Carter 25
Usability Testing in Kuniavsky
1. One day before a test is not enough for the practice test. (p.264)
2. Excellent discussion of dealing with participant recruiters (p.266)
3. Don’t test features. Test user goals. (p.268)
4. Quantitative measures – what are the goals of your test? Don’t forget organizational politics.
2/16/05 Usability Testing for ISD - Leslie Carter 26
Usability Testing in Kuniavsky (cont.)
1. Make it absolutely clear that it’s the interfaces, not the person being evaluated. (p.279)
2. Discount usability testing – Don’t need one-way mirror or cameras. You can do it with remote software & screen capture utilities. (p.286)
3. Define severity ratings to match the bug codes & enter them in the bug tracking system. (p.294)
2/16/05 Usability Testing for ISD - Leslie Carter 27
Resources
http://immersion.gmu.edu/portfolios/lcarter3/resourcesusab.html