Landis Lewis, Z. et al.:Measuring Efficiency of Use in a Web-based EMR Developed for Malawi: Lessons Learned From Performing a Preliminary CogTool Analysis

• This slideshow, presented at Medicine 2.0’08, Sept 4/5th, 2008, in Toronto, was uploaded on behalf of the presenter by the Medicine 2.0 team

• Do not miss the next Medicine 2.0 congress on 17/18th Sept 2009(www.medicine20congress.com)

• Order Audio Recordings (mp3) of Medicine 2.0’08 presentations at http://www.medicine20congress.com/mp3.php

Measuring Efficiency of Use in a Web-based EMR Developed for Malawi: Lessons Learned From Performing a Preliminary CogTool Analysis

Zach Landis Lewis, MLISGerald Douglas, MSISValerie Monaco, PhD, MHCI

University of PittsburghDepartment of Biomedical Informatics

Pennsylvania Malawi

Area119,282 km2 118,484 km2

Population

(2007 est.)12.4 million 13.6 million

Background: Malawi

Life expectancy at birth in years

77.8 43.5

Number of people living with HIV/AIDS

18,000

(0.15%)900,000 (14.2%)

Background: Baobab Anti-Retroviral Therapy system (BART)

    

         

Objective

Our research objective is to determine how efficiently novice users complete tasks using the touchscreen interface of the EMR.

Methods

1. Predict skilled task performanceo Select taskso Use CogTool software application to generate prediction

 2. Measure novice task performanceo Collect timestamp data from user interface events

(e.g. pressing a button )o Repeat each task three times

 3. Compare prediction with results of novice performance

Methods: CogTool

Q: What is predictive human performance modeling?A: A method for predicting how long a skilled user will take to complete a task

• validated and used in the field of Human Computer Interaction • over 100 papers validating or using human performance modeling for evaluation or design of interfaces

Examples of real-world applications: - Web pages and browsers - Telephone operator workstations - Space operations database system - Television control system

- Intelligent tutoring system - IRS office automation system - Police in vehicle systems - Firefox tab feature

    

         

Methods: CogTool

Methods: CogTool

Methods: CogTool The five clusters of colored bars

represent all the button presses

required to perform this task,

separated by thinking time.

“5”“8”

“.”

“3”“Next”

Methods: CogTool

This is the final hand movement operator for

pressing the button labeled “5”.

This pane shows a close-up view of a

sequence of cognitive resources being used. Here we see the activities for pressing the button

labeled “5”

Methods: CogTool

This is a “trace” of production rules fired by the ACT-R

production rule system during the task performance

The highlighted production rules correspond with cognitive activities occurring while a user is pressing

the button labeled “5”.

Results: CogTool

1. Selected 31 routinely performed tasks in BART2. Used CogTool to predict skilled task performance3. Predicted performance times in seconds for each task

Results: Novice Performance

Rate of errors:• Errors are any deviation from the optimal

sequence of steps required to complete a task

• 77% (286) of task performances were error-free and were compared with CogTool predictions

• 4 of the 31 tasks were performed without error by all subjects on all repetitions

Results: Comparison of CogTool Prediction with Novice Performance

Discussion

1. CogTool allowed us to rapidly generate predictions of skilled performance

2. Novice subjects demonstrated a low error rate

 3. Novices performed faster than CogTool predictions on average:• Tasks were modeled independently, but users interleaved

some tasks• CogTool's assumptions for inserting "Think" events may

not be applicable for wizard format interfaces

Discussion, continued

 4. Unexpected findings:

• Pittsburgh subjects occasionally used more than one hand to manipulate the interface – (but we haven’t observed that in Malawi… yet)

• Communication time varied greatly between tasks, sometimes resulting in prolonged dialog rather than a single question and answer

Future Work

1. Update the CogTool model to reflect current, more sophisticated understanding of tasks and user actions

- We are working with the CogTool team to be able to adjust the models and CogTool itself to fit the assumptions to our tasks and users

2. Characterize the use of the system in a real-world setting• Collect anonymized user interface event data in Malawi from a

representative group of users• Measure system use by novices and skilled users

Acknowledgements

The National Institutes of Health and the National Library of Medicine, USA - Grant # 5T15LM007059-22 for funding this research

Bonnie John, PhDThe CogTool Project - http://www.cs.cmu.edu/~bej/cogtool/Greg Cooper, MD, PhDMike McKayJoe Rauch, DDSYolanda DiBucciMargaret Henry