32
Usability Testing Medical Devices Dan Berlin Experience Research Director @banderlin [email protected] Chris Hass SVP of Experience Design @chrishass [email protected] UPA 2012 Las Vegas, NV

Usability Testing Medical Devices

Embed Size (px)

DESCRIPTION

 

Citation preview

Page 1: Usability Testing Medical Devices

Usability Testing Medical Devices

Dan Berlin

Experience Research Director

@banderlin

[email protected]

Chris Hass

SVP of Experience Design

@chrishass

[email protected]

UPA 2012

Las Vegas, NV

Page 2: Usability Testing Medical Devices

Usability activities are NOT clinical Trials

Generally the process will be less stringent:

•  More loosely defined, faster

•  Will likely involve a relatively small number of people per activity

•  Initial studies stress simulation, with no medicinal agents or clinical

impact.

•  Dovetail with interaction design, visual design, and marketing activities

Usability  Tes,ng  Medical  Devices   2  

Page 3: Usability Testing Medical Devices

It’s going to feel like a clinical trial

•  Extremely detailed “Usability Specification” documents

•  “Usability Validation” (Risk assessment and amelioration)

•  “Usability engineering file” / “Design Efficiency File”

•  Know Federal and International regulations:

•  FDA Quality System Regulation

•  HE: 75 2009

•  ISO/IEC 62366:2007

•  Knowledge of (and inspiration for)

emerging standards

Usability  Tes,ng  Medical  Devices   3  

Page 4: Usability Testing Medical Devices

Medical Device Manufacturers may be unfamiliar

with your practices and lack of regulations

Usability  Tes,ng  Medical  Devices   4  

Page 5: Usability Testing Medical Devices

Safeguarding participants may be more challenging

•  Your safety protocols must be exacting, fully defined,

piloted, and effective

•  Even simulated product use may put participants at risk

•  Target audiences may be physically or cognitively

vulnerable

•  Activity environments may be unusual

•  Work with client and medical care specialists

Usability  Tes,ng  Medical  Devices   5  

Page 6: Usability Testing Medical Devices

Reporting responsibilities, guidelines, formats, and

requirements will be more complex

•  Reporting guidelines, formats, and requirements will likely

be extensive and complex

•  Industry reporting standards (CIF)

•  Specialized reporting formats

•  Raw data may be reviewed again and again and again. . .

Usability  Tes,ng  Medical  Devices   6  

Page 7: Usability Testing Medical Devices

Research locales may be more challenging

Device use environments may be:

•  Unfamiliar

•  Sterile

•  Stressful

•  Germ-laden

•  Emotionally distressing

•  Gross

•  Secure

•  Rarely observable

•  Complex

•  or otherwise unusual to non-medical personnel

•  (And fainting can be a factor!)

Usability  Tes,ng  Medical  Devices   7  

Page 8: Usability Testing Medical Devices

Client Relationship

•  A robust set of client team members is essential –

as is knowing their roles:

–  Legal

–  Medical advisor

–  Safety advisor

–  Compliance

–  Adverse Event Specialist

–  Physician is helpful, but not enough

–  The usual suspects

•  Clients may ask for proof of insurance

–  But too much insurance could be a bad thing

Usability  Tes,ng  Medical  Devices   8  

Source: http://robot6.comicbookresources.com/2010/07/take-a-look-at-the-avengers-assemble-painting-by-alex-ross/

Page 9: Usability Testing Medical Devices

FDA Relationship

•  If the study is for FDA approval – learn about

the client’s relationship with the FDA:

–  What are the relevant regulations? –  Who is the client’s liaison to the FDA? –  What is the client’s relationship with the FDA? –  Has the client met with the FDA? –  Have they received comments from the FDA? –  Does the client understand usability testing? –  Are we allowed to talk to the FDA?

Usability  Tes,ng  Medical  Devices   9  

Source: http://news.petpardons.com/fda-denies-rumors-over-possible-discovery-of-chicken-jerky-toxin/

Page 10: Usability Testing Medical Devices

Working with an IRB

•  IRB = Institutional Review Board

•  For FDA approval, typically work with an independent IRB

•  Their job: to ensure that research conducted on human subjects is carried

out correctly and humanely

•  Not as scary as you may think

–  We are NOT evaluating medical efficacy or safety

–  But we DO have to be conscientious

•  Have your ducks in a row and it will be a smooth process

Usability  Tes,ng  Medical  Devices   10  

Source: http://www.bioedge.org/index.php/bioethics/bioethics_article/who_is_watching_the_irb_watchdogs_asks_nature

Page 11: Usability Testing Medical Devices

Working with an IRB

•  Adds about $2000-$5000 to the cost of the study

•  Adds a day or two to the project for dealing with

the IRB

•  Be cognizant of any non-disclosure agreements

•  Prior to the study, signature pages should be

signed-off, circulated, and returned in a timely

manner

•  Everyone on the team will take this very

seriously!

Usability  Tes,ng  Medical  Devices   11  

Source: http://www.bioedge.org/index.php/bioethics/bioethics_article/who_is_watching_the_irb_watchdogs_asks_nature

Page 12: Usability Testing Medical Devices

IRB Documents - Forms

•  Indemnification Agreement: So the IRB doesn’t get sued

•  Financial Disclosure: The researchers don’t have conflicts of interest

•  Investigator Statement: The researcher code of conduct

•  Satellite Site Application: May be needed if the study involves multiple

locations

•  Shipping & Invoice: Where documents are sent

•  Study Application: Detailed information about the study

•  Curriculum Vitae for all researchers

•  Other forms that your IRB requires

Usability  Tes,ng  Medical  Devices   12  

Source: http://blog.pennysaverusa.com/tag/tax-tips/

Page 13: Usability Testing Medical Devices

IRB Documents – Study and Device Info

•  Recruitment Screener: Be very precise

•  Recruitment Flyer: To hang in doctor’s office to aid with recruiting

•  Moderator’s Guide: A second guide may be needed if there are child

participants

•  Consent and asset forms: For adults and children, respectively

•  Safety Concerns: Outline potential risks

•  Device Background: Detailed explanation of the device including

indication and new features

•  Device Instructions for Use: If they are also included in the study

Usability  Tes,ng  Medical  Devices   13  

Source: http://healthcareit.etisbew.com/

Page 14: Usability Testing Medical Devices

IRB Documents – Validation Plan

•  A detailed explanation of how the study will validate that the device is usable

–  If you’re used to winging it, you can’t do that here

–  Lay out every question and how it will be interpreted

–  Lay out a rationale for every metric

–  Break down the pieces of the tasks and what will be observed

–  A detailed explanation of how the will be analyzed and presented

•  Quantitative measures (asked and observed)

–  Rating scales

–  Yes/No observations

Usability  Tes,ng  Medical  Devices   14  

Source:  hDp://www.vnet5.org/pub/approach/uva.html  

Page 15: Usability Testing Medical Devices

Quantitative Measures

•  Rating Scales

–  Asked

•  Ease of use

•  Instructions for

Use clarity

•  Design

•  Ergonomics

–  Observed

•  Usability

•  Help needed

•  Number of

errors

•  Yes/No observations

–  Did the participant do this, that, AND the other thing?

Usability  Tes,ng  Medical  Devices   15  

Chart  Color  

%  of  chart  that  is  red  %  of  chart  that  is  black  

Ra,ng   No  Help  Needed   LiDle  Difficulty   Some  Help     Much  Help   Task  Failure  Explana,on   Completed  with  no  help   1-­‐2  small  errors   3-­‐5  errors   Moderator  gets  involved   Failure  

Small  errors:  Doesn't  press  the  green  buDon  Doesn't  remove  cap  properly  Etc.  

Page 16: Usability Testing Medical Devices

Recruiting

•  Give your recruiter a heads up LONG before the study starts –  And start recruiting early

•  Make a flyer to be hung in doctor’s offices and distributed electronically which includes: –  The purpose of the study

–  Why they would want to participate

–  Compensation

–  Recruiter contact information

•  During the project, any screener changes need to be re-approved by the IRB

•  Many institutions can’t “promote” a study that is under their auspices

Usability  Tes,ng  Medical  Devices   16  

Source: http://www.toptenz.net/top-10-wwii-propaganda-posters.php

Page 17: Usability Testing Medical Devices

Sample Flyer

Usability  Tes,ng  Medical  Devices   17  

Page 18: Usability Testing Medical Devices

Recruiting

•  Don’t be shy – makes friends with local hospitals

and clinics

–  Nurses and admins are the gateway

–  Have the sanitized IRB approval in-hand

•  Find groups that can be a gateway to your desired

audience

–  Healthcare institutions

–  Local and national support groups

–  Federal agencies

–  Community support groups

–  Craigslist & Social Media

•  Plan for surprises – like extensions to the timeline

Usability  Tes,ng  Medical  Devices   18  

Source: http://www.toptenz.net/top-10-wwii-propaganda-posters.php

Page 19: Usability Testing Medical Devices

Working with Children

•  From a legal standpoint, a child is someone under 18 years old

•  Children cannot consent to participate in the study, they can only assent –  An assent form explains the details of the study, in the

first person

•  You will treat a 17 year old different than an 8 year old

–  Be cognizant of the protocol to ensure you are consistent in your data capture

•  The parent conundrum: should they participate?

–  At what point does it become co-discovery and is that okay?

Usability  Tes,ng  Medical  Devices   19  

Source: http://evilstaring.com/2011/06/08/amazingly-evil-child-3/

Source: http://cautionmagnet.blogspot.com/2011/05/confused-kids.html

Page 20: Usability Testing Medical Devices

Working with Children

•  Put the child at ease:

–  Explain the lab (to a point)

–  Ask if s/he would like a break

–  Give reminders that s/he is doing a great job in the study

–  Structure the study so that it isn’t drudgery

–  Don’t make them remember rating scales

–  Not all children like to be spoken to like a child

•  Be extra vigilant for signs that the participant is telling you want you

want to hear

–  As well as signs of fatigue and confusion

•  You don’t need to be their friend, but don’t scare them away

Usability  Tes,ng  Medical  Devices   20  

Source: http://lovefunnypictures.com/funny-pic-bush_funny_face.php

Page 21: Usability Testing Medical Devices

Moderator’s Guide

•  Be precise with a tightly written protocol

•  Must be well-defined in advance and piloted

–  No changes to the protocol without revisiting the IRB, the client, the FDA

approval minutes, etc.

•  Clients may be used to clinical trials and view usability methods as “thin”

–  We’re not running a clinical trials, but that’s no excuse for not adequately

preparing

•  Adhere to the protocol!

–  Any changes require IRB approval

Usability  Tes,ng  Medical  Devices   21  

Source: http://tips.tntdental.com/?tag=website-navigation

Page 22: Usability Testing Medical Devices

Moderator’s Guide

•  The moderator’s guide should capture:

–  The tasks

–  Steps in the tasks

–  What should be observed

–  The questions to be asked of the participants

–  Observer rating scales

•  Basically, a shortened version of the Validation Plan

•  Be precise in the metrics and methods

–  They cannot vary

–  Data reporting requirements will be significant

–  Gaps in the reporting will be very visible

Usability  Tes,ng  Medical  Devices   22  

Source: http://tips.tntdental.com/?tag=website-navigation

Page 23: Usability Testing Medical Devices

Notes Grid

•  We are using “micro-tasks” and want to

capture precise information

•  The notes grid should facilitate the analysis

yet be easy to use

–  Every step and question on its own row

–  Have clear study sections (introduction, etc.)

–  Grey out areas that can be ignored

•  Team note taking: moderator concentrates on

certain items, note taker does the rest

Usability  Tes,ng  Medical  Devices   23  

Source: http://beta.armagetronad.net/design/tron.html

Page 24: Usability Testing Medical Devices

Notes Grid

Usability  Tes,ng  Medical  Devices   24  

Page 25: Usability Testing Medical Devices

Lab Preparation

•  Capture the participants from two

different angles

–  A close-up view of the device interaction

(over-the-shoulder)

–  Straight-on to capture the entire field

•  Have all device supplies at the ready

–  Alcohol wipes, extra needles, batteries,

etc.

–  You’ll use more supplies than you expect

•  Get a first aid kit and nitrile gloves

–  Have a first aid plan and practice

Usability  Tes,ng  Medical  Devices   25  

Source: http://www.ebaumsworld.com/pictures/view/218095/

Page 26: Usability Testing Medical Devices

Participant Safety

•  As with any usability, participant comfort and safety is the

top priority

•  Must be mindful of participants with health conditions

–  Will they need accessibility accommodations?

–  Will they have an aide/caretaker?

–  How will the condition affect their participation?

•  What you can do to ensure participant comfort and safety:

–  Have a robust first aid kit (and possibly medical personnel)

–  Provide water, juice, sugary, and non-sugary snacks

–  Ensure they know that they will be working with a placebo

–  Accessible food

Usability  Tes,ng  Medical  Devices   26  

Source: http://en.wikipedia.org/wiki/File:Sign_first_aid.svg

Page 27: Usability Testing Medical Devices

Participant Safety

•  Warn participants of safety hazards while using the device, if it’s

part of the protocol

•  Have a plan of action if someone is injured

•  Establish a path of notification if a participant is injured

–  Reporting may have many phases: reporting it internally, then to the client, to the FDA, and the IRB

•  Educate yourself about the condition

•  Ensure the protocol does not exacerbate a disease or condition

•  Be vigilant about fatigue, confusion, disorientation, dehydration, &

stress

Usability  Tes,ng  Medical  Devices   27  

Source: http://en.wikipedia.org/wiki/File:Sign_first_aid.svg

Page 28: Usability Testing Medical Devices

Adverse Effects

•  An adverse effect (AE) event is when a participant tells you about a side effect or other experience while using a device or drug

•  This is critical information for the client and must be reported, typically immediately

•  Some clients may require you to go through AE training

•  Before the study begins, determine to whom AEs are reported

•  AEs can be complex to identify, but are VERY important

Usability  Tes,ng  Medical  Devices   28  

Source: http://www.cartoonstock.com/directory/d/Din.asp

When in doubt,

report!

Page 29: Usability Testing Medical Devices

Adverse Effects

•  In the course of a usability study for a medication website, a participant mentions that she stopped taking the drug. She explains that the blister packs and pills sometimes arrived crushed.

•  While testing an infusion pump, the participant mentions that the tube always falls out of his previously used pump, which is the client’s product.

•  During a study, a participant mentions that her arms started to itch when she stopped taking the client’s medication.

•  A male participant mentions that his wife got pregnant two weeks after he started taking the client’s medication, even though she used the birth control pill.

Usability  Tes,ng  Medical  Devices   29  

Page 30: Usability Testing Medical Devices

Analysis

•  The setup from the validation plan, moderator’s

guide, and notes grid should make analysis easy

•  Dealing with gaps in the data can be tricky

–  Did the question get asked?

–  Was the behavior observed?

–  Do we have to go back to the recordings?

•  Make sure you HAVE the recordings!

•  Rely on quantitative measures

–  What percentage of participants passed the task?

–  What ease of use ratings did participants give?

–  How many participants committed the most

common errors?

Usability  Tes,ng  Medical  Devices   30  

Source: http://www.webseoanalytics.com/blog/how-to-conduct-market-research-competitor-analysis/

Page 31: Usability Testing Medical Devices

Analysis

•  Remember to think on a micro-task scale – every little step of using the device should be observed –  It’s difficult to watch and report on every step of a

complex interaction and document it precisely, especially live

–  Tallying needs to be specific, yet practical

•  Be mindful of capturing and organizing the important qualitative data –  This can be especially tough when dealing with many

participants

–  Summarize qualitative data directly after the session – don’t wait

•  Know what you’re reporting – there may be data to help marketing

•  Sometimes the goal of these studies is to identify marketing claims, which can raise concerns

Usability  Tes,ng  Medical  Devices   31  

Source: http://www.webseoanalytics.com/blog/how-to-conduct-market-research-competitor-analysis/

Page 32: Usability Testing Medical Devices

Thank You!

•  Conducting a usability study for a medical

device is similar to a typical study, except

with much stricter protocols

•  Your safety and the safety & comfort of the

participants are paramount

•  Use quantitative measures and report

percentages of participants that did or

didn’t perform an action

•  It’s all about proper planning!

Usability  Tes,ng  Medical  Devices   32  

Questions/Comments?

Dan Berlin

[email protected]

@banderlin

Chris Hass

[email protected]

@chrishass