Benefits Evaluation of Health Information Technology Projects

Evaluating the Benefits of Health Information Technology Projects

HITenlightenment.comShawn Quait, RN, BScN

Objectives

• Understand what Benefits Evaluation is and why it is important

• Understand how project benefits fit into organizational strategy and strategic priorities

• Understand the Benefits Evaluation process

HITenlightenment.com

What is Benefits Evaluation and why is it important?


What is Benefits Evaluation?

Activity in which the impacts of an intervention or change are measured and analyzed

Pre-Implementation Compare Post-Implementation

Benefits


Why is Benefits Evaluation Important?

• Inform decisions of current/future projects

• Engage stakeholders• Advance investment in IM initiatives • Enhance end user adoption • Focus project efforts • Demonstrate achievement of project

objectives • Improve outcomes of future projects • Inform solution design HITenlightenment.com

How do project benefits fit into organizational strategy and strategic priorities?


Benefits Evaluation Framework

BenefitsNetBenefitsGovernment /

Ministry Initiatives

System Quality

Information Quality

Service Quality

Use

User Satisfaction

Quality of Care

Access

Productivity

Organizational Strategy and

Strategic Priorities


What is the Benefits Evaluation process?


Benefits Evaluation Process Overview

Initiation Planning Support Delivery Build and Test Design

Plan Benefits Measurement (I)

Pre-Implementation Data Collection and Analysis (III)

Post-Implementation Data Collection and

Analysis (III)

Evaluate Technology Benefits (IV)

Define Indicators (II)

Operationalize Indicators (V)

Initiation (I) Planning (II) Closing (IV)Executing (III)

Monitoring and Controlling

Go Live

Benefits Evaluation

IT Project Lifecycle

PMBOK







Analysis (III)






Go Live

Benefits Evaluation


PMBOK


Plan Benefits Evaluation

• Benefit Evaluation Planning • Scope, budget, schedule, deliverables and resources

• Governance structure • Stakeholders and audience for report• Evaluation indicators and studies

• Plan is drafted in initiation, refined in planning, and finalized in the design phase


Benefits Evaluation Lead

• Each project will have a benefits evaluation lead

• Responsibilities• Stakeholder identification and relations• Planning and coordination of the benefits evaluation

• Formation of benefits evaluation working group • Status reporting







Analysis (III)






Go Live

Benefits Evaluation


PMBOK


Define Indicators

Defining Indicators consists of 4 steps1. Create Benefits Description List2. Prioritize Benefits3. Create Indicator Definitions4. Secure Approval for Indicator

Definitions List


Benefit Reduction in medication incidents. Benefit Classification

Domain: Drug Information System ProgramMajor Benefits Dimension: Net BenefitsCategory: Quality Sub-Category: Patient Safety

Ranking:5

Definition Number of Medication Incidents (numerator) per X number of medications ordered (denominator). Benchmark or Target

100% Reduction in medication incidents.

Benefit Realisation Time Frame

Immediately post implementation.

Method / Source

Measurement Phase

Numerator: Number of Medication Incidents

Denominator: Total number of medications ordered

Pre Implementation

Measurement/Analysis Responsibility: Benefits Evaluation Lead Data Capture: Medication incident reports. Assumptions / Constraints: Only medication Incidents that have resulted from errors in the medication ordering and transcription processes (including unclear handwriting) will be counted eliminating those due to adverse drug reactions. However, those due to drug interactions should be included. The present medication incident report should carry adequate information to determine the cause.

Measurement/Analysis Responsibility: Benefits Evaluation Lead Data Capture: Through the pharmacy information system Assumptions / Constraints: The total number of medications, rather than medication orders will be considered to eliminate discrepancies in comparisons between units due to variations in the number of medications within each order.

Post Implementation

Data Capture: Medication incident reports. Assumptions / Constraints: Same as in Pre-Implementation.

Data Capture: Through the pharmacy information system. Assumptions / Constraints: Same as in Pre-Implementation.

Indicator Definition Example

Mandatory Indicators

• Patient Satisfaction with system (If applicable)

• User Satisfaction with system implementation

• System Utilization


Qualitative Data?

• Try to make data collection and objective and quantitative wherever possible

• Example: Collecting data using a question on a survey “% of respondents who felt that unclear medication orders were significantly reduced”, rather than actually measuring it “Number of unclear orders per 100 medications ordered”






Analysis (III)






Go Live

Benefits Evaluation


PMBOK


Data Collection and Analysis

The Data Collection and Analysis phase consists of 4 steps: 1. Develop Data Collection Process2. Trial Data Collection3. Data Collection Proper4. Data Analysis







Analysis (III)






Go Live

Benefits Evaluation


PMBOK


Evaluate Technology Benefits

Benefits Evaluation Report• Benefits Achieved• Benefits Not Achieved• Negative Impacts• Lessons Learned• Evaluation Sustainability Plan• Recommendations formed







Analysis (III)






Go Live

Benefits Evaluation


PMBOK

Operationalize Indicators

• Evaluation continues after solution is operational

• Benefits indicators selected for ongoing measurement become performance measures

• Information Management Monthly report will display the performance measures as appropriate


Ben

efi

ts E

valu

ati

on

S

um

mary



Develop Preliminary Benefits Evaluation

Plan-Define scope of benefits evaluation-Define stakeholders-Define audience for final report-Define indicators and evaluation studies- Consider ethics of measurement plan -Create first draft of Benefits Evaluation Plan-Review initial plan with governance structure

Pre-Implementation Data Collection and Analysis (III)1. Develop data collection process2. Trial data dollection3. Conduct data collection proper4. Analyze data


Analysis (III)1. Trial data collection2. Conduct data collection proper3. Analyze data-Administer User Satisfaction Survey-Administer Patient Satisfaction Survey (If applicable)


-Complete Benefits Evaluation Report - Present benefits evaluation results to governance structure as part of project closureDefine Indicators (II)

1.Create Benefits Description List 2.Prioritize benefits 3.Create Indicator definitions4.Secure approval for Indicator definitions from governance structure


-Identify operational performance measures - Develop reporting strategy for operational performance measures



Go Live

Benefits Evaluation


PMBOK

Refine and Finalize Benefits Evaluation Plan-Augment Benefits Description list with workflow benefits-Refine and finalize Benefits Description List-Refine and finalize indicator definitions and evaluation studies-Refine and update Benefits Evaluation Plan-Review final Benefits Evaluation Plan with governance structure and sign off

Questions/Feedback?

Connect with me on LinkedIn/shawnquait

HITenlightenment.comShawn Quait, RN, BScN

http://www.linkedin.com/in/shawnquait



Health & Medicine

Benefits Evaluation of Health Information Technology Projects