Embed Size (px)
DESCRIPTION
Usability of any product, whether it is software application, a website, or technical gadget is an important consideration. Sherry Marcy, Science and Technology Market Strategies will talk, from a business perspective, on how Pfizer improved its software tools for chemistry. She will share important lessons about design considerations, and user testing that will be relevant for businesses managers (start-up and established), entrepreneurs and product development teams.
Citation preview
Usability Testing: A Story
LA2M
12 May 2010
Sherry MarcyScience & Technology Market Strategies LLC
On linkedin, twitter, facebook
734-368-7806
Case Study – Situation at Start
• Multiple brilliant scientific computing
tools
• None of them used by chemists
• No understanding of usability in Pfizer
Informatics
Usability Testing and the
Technology Transfer Process
1. Design
consistent “look and feel”
2. Production Environment
no development glitches
3. Testing
a) alpha b) beta c) usability testing
4. Training
WBT, viewlets, business-led courses…
5. Support
context-sensitive help, cheat sheets
6. Communications
flash sessions, invitations, balloons!
Understanding
Task
Examples
User
Groups
User Experience EngineeringContextual Design by Beyer & Holtzblatt
Building Software that Works for Users
Contextual InquiryWork Modeling
Consolidating
Brainstorming
Visioning
StoryboardingPaper Prototyping
Usability Testing
Vision
Design
Software
WebDock / Docking Study
• Chemists Participating in WebDock Phone Interviews
id Ed At Pfizer WebDock
Training?
WebDock
Use?
Project
U1 PhD 2 years No No PI3-K
U2 BS 5 years Yes Yes FXa
U3 BS 4.5 years No Yes PI3-K
U4 MS 2 years Yes No CDK4
U5 PhD 4 years Yes Yes FXa
U6 MS 2 months Yes No CDK4
U7 PhD 3 years Yes Yes CDK4, …
id Ed At
Pfizer
WebDock
Training?
Recent
Real*
WebDock
Use
Project Use Test Observers
U1 PhD 2 years No Never PI3-K 11/7/01 2:00 Sherry Marcy, David
Wild, Rob Goulet,
David Dunn
U3 BS 4.5
years
No 2 months
ago (just
“to try”)
PI3-K 11/8/01 9:00 David Wild, Rob
Goulet, David Dunn
U4 MS 2 years Yes Never CDK4 11/8/01 1:00 David Wild, Rob
Goulet, David Dunn
U5 PhD 4 years Yes 1 day ago FXa 11/7/01 8:00 Rob Goulet
U7 PhD 3 years Yes 5 months
ago
CDK4,
…
11/7/01 10:30 Rob Goulet
You are a Medicinal Chemist on the Protein Factor Xa project at
Pfizer. The Modeler assigned to your project has created a 3D
model of the protein and set up a WebDock job to allow you to try
out docking your own ligands.
For your series, you have several ideas for variations of the lead
you want to try out to see if they will improve the binding. You
want to use WebDock to dock your ideas to the protein in the 3D
model created by the modeler.
Your specific task is:
Use WebDock to see how your ideas (found in Figures 1 &
2) might bind to the protein Factor Xa.
WebDock Usability Testing
• Chemists
• Task
• Facility
• Facilitator
• Equipment
• Observers
Usability Testing Output
• Stakeholder’s Direct Observation of
Users’ Experience
• Assessment of Relative Quality of Use
• Identification of Obstacles and Insight
into Responses
Quality of Use
• Quality of Use
– Ease of Use
– Ease of Learning
– Reliability
– Validity
– Performance
• Other Aspects of Software Quality
– Maintainability, Extensibility,…
Task
1 2 3 4 All
User 1 73 65 95 70
2 94 85 78 82
3 81 96 80 85
All 82
Task
1
User 1 73
Quality of Use MetricDetermination
Hard |--------------------------------------------------| Easy
Frustrated |--------------------------------------------------| Satisfied
Failure |--------------------------------------------------| Success
Time:
More |--------------------------------------------------| Less
than As than
Expected Expected Expected
Task
1 2 3 4 All
User 1
2
3
All
Quality of Use MetricInterpretation
100
90
80
70
60
0
most users will not accomplish their tasks and will reject
most users will easily accomplish their tasks
users will accomplish their tasks, but many rough edges will impact users
many users will accomplish their tasks, but many will not and will reject
users will consistently accomplish their tasks, will sometimes exceed users' expectations
Quality of Use Metric
results we’ve seen
(before & after)100
90
80
70
60
0
The full range
Dramatic improvements
Quality of Use
correlates with other indicators
How much of the time
Is there room
for a hydroxyl
group in the
pocket?
How do I
find the distance
between two
atoms?
• instructors are describing how to leverage the
tool rather than how to operate it
• users are thinking about their work rather than
the tool
Quality of Use vs Use
100
90
80
70
60
0
Quality of Use predicts Use Over Time
WebDockNovember 2001
WebDock QofU
100
90
80
70
60
0
Most chemists will usually be able to dock
their ligands with WebDock, but many rough
edges will impact them, causing some
chemists to not use it
•Success
•Obstacles: Opportunities
Chemists’ Experience with
WebDock
Where’s WebDock?
People don’t read…
Chemists’ Experience with
WebDock
Chemists’ Experience with
WebDock
Chemists’ Experience with
WebDock
Scroll!
This ChemDraw
link does nothing!
Chemists’ Experience with
WebDock
Enter every time!
Backspace makes
the page go away!
Why a filename?
How many?
Ctrl-z for Undo
doesn’t work!
How is
stereochemistry
handled?
I can’t reuse my
previous sketch!
Chemists’ Experience with
WebDock
Chemists’ Experience with
WebDock
How do I zoom
and translate?
It’s a pain to
switch off all the
bindings!
Are the check
boxes on or off?
Those little boxes
are hard to hit!
Things come back
on when I don’t
expect! (Protein
on causes H’s on)
Chemists’ Experience with
WebDock
WebDock Launch(September 13, 2001)
Holiday Season(Dec, 2001 – Jan 2, 2002)
Lithium Launch(January 29, 2002)
(As of January 5, 2003)
0
10
20
30
40
50
60
70
80
90
100
Holiday Season(Dec, 2002 – Jan 2, 2003)
Metrics of WebDock Use
0
10
20
30
40
50
60
70
80
90
Metrics of WebPK Use
WebPK Launch
(October 9, 2001) Holiday Season(Dec, 2001 – Jan 2, 2000)
Original Usability Testing Score of 85
What’s Quality of Use Improvement
Worth?Value proposition:
• Save development costs
• Save development time
• Reduce maintenance costs
• Save redesign costs
• Attract usage (increase appeal)
• Retain users (frequency of use)
• Decrease training time and costs
• Decrease support costs
• Increase trust in systems
• Drive productivity!
100
90
80
70
60
0
Better Decisions
Sooner
Tools to Think!
Sherry Marcy
