Upload
harding-sweeney
View
46
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Innovative Use of Multiple Methods to Assess Health Informatics System Standards APHA 2013 -- Boston, MA November 6, 2013. Linda Piccinino, Andrew Westdorp, Jennifer Kwan, Lydia Rogers, Jean Wilson, & Sue Griffey. Presentation Outline. Background Purpose Framework - PowerPoint PPT Presentation
Citation preview
Innovative Use of Multiple Methods to Assess Health Informatics System Standards
APHA 2013 -- Boston, MANovember 6, 2013
Linda Piccinino, Andrew Westdorp, Jennifer Kwan, Lydia Rogers, Jean Wilson, & Sue Griffey
2
Presentation Outline
• Background• Purpose• Framework• Evaluation Data Collection• Results• Conclusion
4
NCS Vanguard Study
• The National Children’s Study (NCS)– Large-scale, dynamic health study – 20-year time horizon
• NCS Vanguard Study– Pilot study for the NCS main study– Uses multiple information management systems
(IMS) to support one research data collection effort
5
NCS Vanguard Study IMS Evaluation
• Assessment of IMS to see how well meeting the study’s needs– Current needs– Future projected needs
7
IMS Evaluation
• Develop and implement an evidence-based evaluation approach
• Answer fundamental questions:– Does each IMS support the study’s basic
functional requirements?– Does each IMS perform as expected?
• Leverage evaluation framework and methodology
8
Evaluation Phases
• Phase 1 – Short-term: 2-3 months– High level assessment of basic functionality of
IMS solutions – Completed in Fall 2012
9
Evaluation Phases (continued)
• Phase 2 – Longer-term: 9-12 months– Identify/refine preferred practices, processes,
and functionality of an IMS solution used to support NCS Vanguard protocol
– Identify gaps in IMS solutions with respect to evaluation criteria
13
Phase 1 IMS Evaluation
Benchmarking System Characteristics
Real-Case Scenario-Testing
• Reviewed system and user documentation
• Performed system stand up for available solutions
• Developed standardized scenarios
• Conducted testing using host-provided environment
14
Example: Role-Based Scenario Functions Tested
Administrator Data Collector and Supervisor
Supervisor/Manager
Perform Study Center setup
Assign user roles Preload pregnancy
screeners
Consent mother and child
Conduct Pregnancy Visit 1 event
Complete participant record of visit
Enter Operational Data Elements
Enter linkages
Assign cases Manage field work Provide time and
expense information View staff
certifications View and edit
instrument data Obtain basic
cooperation rates Obtain basic refusal
rates Obtain rates by
disposition code
15
Phase 1 Overview: IMS Functional Requirements Tested
IMS Functional RequirementsDoc.
ReviewScenario Testing
Architecture 1. Is the IMS system open-source? √ √2. Does the system utilize open architecture and use of modern non-proprietary programming languages? √ √
3. Does the system have an open API for easy integration with other products? √
4. Does the system have any special hardware requirements? √ √
5. Does the system have the support 2-factor authentication? √
Case Management 1. Cases can be flagged for supervisor review √2. System allows for creation of new person records to be associated with each case √
17
Benchmarking
• Benchmarking results showed where systems were strongest and where they needed additional functionality
18
Review of Documentation on IMS Characteristics
Category IMS 1 IMS 2 IMS 3Documentation Attributes All All All
General System Characteristics ≥ ½ All < ½
Architecture< ½ ≥ ½ ≥ ½
Case Management≥ ½ All ≥ ½
IMS Performance< ½ ≥ ½ < ½
IMS Data Collection< ½ All ≥ ½
Data Management< ½ ≥ ½ ≥ ½
CATI< ½ ≥ ½ ≥ ½
Sample Tracking*< ½ < ½ < ½
All All characteristics documented
≥ ½ Half or more characteristics documented
< ½ Less than half of characteristics documented
*Few sites collecting biospecimens at the time.
19
Real-Case Scenario Functional Testing
• Results provided:– Objective information on functionality– Qualitative input from IMS testers
20
Data Collector User Role Scenario Testing
Functionality IMS 1 IMS 2 IMS 31. Can understand how to enter data and use system
based on documentation available?Y Y Y
2. Can view prior events? Y Y Y3. Can view contact records? Y Y Y4. Next event needed is indicated in system? Y Y N5. Can schedule events? Y Y Y6. Requires Pregnancy Screener records to allow
consent entry? N N n/a
7. Can enter Study consent information for mother?Y Y Y
8. Can view Study consent information for mother?Y Y Y
21
Functionality of IMS Solution Characteristics
> ½ More than half of characteristics tested were functional
≤ ½ Half or fewer of characteristics tested were functional
N Characteristics were not observed
IMS 1 IMS 2 IMS 3
Case Management > ½ > ½ > ½
IMS Performance > ½ > ½ > ½
IMS Data Collection > ½ > ½ > ½
Data Management > ½ N ≤ ½
22
Translating Results into Use
• Results used to:– Identify high-level areas already meeting
requirements – Identify specific gaps or areas needing
improvement– Inform Phase 2 assessment
24
NCS Vanguard Study IMS Evaluation
• Framework – Grounded in theory
• Collaboration– Involving and communicating with multiple
stakeholders– Critical when testing systems
• Innovation – Moved beyond traditional IT evaluation to include
aspects of program evaluation– Implemented ways to test systems, not users, using
protocol-driven real-life standardized scenarios
25
Thank You!
Authors• Linda Piccinino, Senior Evaluator, SSS, [email protected]• Andrew Westdorp, Center Director, SSS, [email protected]• Jennifer Kwan, Evaluation Team Co-Lead, Program Office,
National Children’s Study, Eunice Kennedy Shriver National Institute of Child Health and Human Development, [email protected]
• Lydia Rogers, Project Director, SSS, [email protected]• Jean Wilson, Senior Study Director, SSS, [email protected]• Sue Griffey, Vice President and Director, Evaluation Center,
SSS, [email protected]