RAVASMARTSOLUTIONS - TECH TIPS

Executive Summary ROI (Guthrie MDTI) Purpose This handout is to demonstrate how a technology project is properly managed by conducting a ROI (Return On Investment). Key point to note: If you do not isolate your technology initiative from the other factors of impact, your ROI is not accurate. In simple terms, you need to ensure that what gains or losses that occur are from your initiative, not some other initiative. Enjoy! Rick Rava

Guthrie Healthcare System EXECUTIVE SUMMARY

Return on Investment

Management Development Technology Initiative Lead by Rick Rava

Submitted by the Return On Investment Team Revised: November 21, 2009

Page 3 of 26

Table of Contents TABLE OF CONTENTS............................................................3 BACKGROUND...................................................................4 WHAT IS ROI?.................................................................5 DATA COLLECTION PLAN.........................................................6 EVALUATION LEVELS............................................................7 LEVEL 1 DATA RESULTS.........................................................8 LEVEL 2 DATA RESULTS........................................................10 LEVEL 3 DATA RESULTS........................................................11 LEVEL 4 DATA RESULTS........................................................12 THE TOTAL 18-MONTH SAVINGS ADJUSTED BY CONFIDENCE LEVEL FOR ALL PERFORMANCE INDICATORS:.................................................................13 ISOLATING THE EFFECTS OF THE MDTI...........................................14 COSTS.......................................................................15 LEVEL 5 ROI ANALYSIS........................................................16 INTANGIBLE RESULTS..........................................................17 HOW HAS THE MDTI PROMOTED THE LEARNING ORGANIZATION?........................18 SUMMARY.....................................................................21 WHAT’S NEXT.................................................................25

Page 4 of 26

Background

he Management Development Technology Initiative (MDTI) is a part of Guthrie Healthcare System's plan to propel the organization to higher performance using information technology as a strategy to accomplish goals

and objectives. As part of the MDTI thrust, Guthrie Healthcare System managers received technology, training, and support to improve their performance. For example, the MDTI will give managers the tools, training, and support necessary to improve the communication of clinical and business information throughout the Guthrie Healthcare System. The Initiative was rolled out in two phases with each group of managers getting the technology, training, support, and infrastructure needed to start them on their journey to higher performance. This report is based on data from Phase I implementation; therefore, the return on investment was measured on the first group of managers participating in the MDTI.

T

Page 5 of 26

What is ROI?

OI (Return On Investment) is simply the answer to the question “Did you spend your money wisely?” Although answering that question seems simple, calculating ROI involves a detailed analysis of

program/project benefits to program/project costs. A typical ROI study would include the following steps:

1. Developing a needs assessment 2. Collecting Data 3. Isolating the effects of the program 4. Evaluating the findings (at all five levels) 5. Calculating the ROI 6. Summarizing the findings

In an effort to demonstrate a positive return on our investment, the ROI Team was interested in answering the following questions related to the MDTI:

1. Did we experience the savings we anticipated? 2. Did managers' productivity increase? 3. Did customer satisfaction increase? 4. Did it help prioritize or roll out the GHS IT Plan

initiatives? 5. Did managers improve their technology skills?

R

Page 6 of 26

Data Collection Plan

he MDTI Data Collection Plan consisted of a process for evaluating participants using a widely-accepted education/training evaluation model developed by Donald L. Kirkpatrick. The model measures programs at four

levels: reaction, learning, behavior change, and business results. These levels are explained in detail on pages 7-11.

The following data collection tools were used in this project:

• Paper questionnaires • Electronic questionnaires • Interviews • Tests • Skill exercises • Participant reaction forms

Audience and Return Rate

• The data was collected from the first group of MDTI managers after they had utilized the technology for at least one year. The second group of managers was not included in this year’s ROI calculations.

• A total of 65 managers participated in Phase I of the MDTI.

• The return rate of the surveys was 77 percent. To increase our rate of return, we used an electronic survey, which included a cover memo introducing managers to the ROI concept.

T

Page 7 of 26

Evaluation Levels

valuation of all five levels of the MDTI was used to calculate the total return on investment. Below is a summary of what each level of evaluation data reflects:

• Level 1 - The reaction of the participants in regard to

their satisfaction with the technology, training, and support provided.

• Level 2 - How well participants mastered the skills

required for competency. • Level 3 - The application of newly learned skills to the

job. • Level 4 - Whether the changes in behavior had a positive

impact on business results. • Level 5 - The return on investment from the MDTI net

program/project benefits divided by the MDTI program/ project costs.

E

Page 8 of 26

Level 1 Data Results

evel 1 data results demonstrated that reaction to the MDTI was very positive. The majority of managers rated the support (advisement, training, help desk, technical/PC) as excellent. The following chart illustrates

these findings:

RATE THE FOLLOWING COMPONENTS OF THE MDTI?

L

0%10%20%30%40%50%60%70%80%90%

Satisfaction Level

Page 9 of 26

In addition, participants preferred hands on learning sessions (instructor-led) over the many other learning options available. Presenting course content in shorter time blocks of 90 minutes enabled participants to master smaller chunks of information at a time and allowed flexibility in scheduling training. By limiting the number of participants in each class, the instructor was able to provide individualized attention to participants as well as relate the course content to application on the job.

Page 10 of 26

Level 2 Data Results

evel 2 data results demonstrated that managers learned many new skills as a result of the MDTI, such as utilizing multiple windows, composing electronic mail, copying and pasting, and many more. The following chart illustrates

one of these findings:

UTILIZING THE MULTIPLE WINDOWS FEATURE TO SWITCH TO DIFFERENT APPLICATIONS:

The managers' understanding of concepts taught during class were assessed using oral and written methods of evaluation.

L

To a Large Extent19%

To Some Extent62%

Do Not Use17%

Not Applicable2%

Page 11 of 26

Level 3 Data Results

evel 3 data results demonstrated that newly learned computer skills were transferred or applied to the job as a result of the MDTI. For example, the majority of managers were using the skills learned to improve

communication of information throughout GHS. In addition, managers expressed the ability to organize and report analytical information more effectively. The majority of managers utilize skills that have improved the communication of information within Guthrie. The following chart illustrates these findings:

PERCENTAGE OF MANAGERS EXPERIENCING IMPROVEMENT

(SPEED, COLLABORATION, COMMUNICATION):

L

68% 69% 70% 71% 72% 73% 74% 75% 76% 77%

Communication

Collaboration

Speed of InformationExchange

Page 12 of 26

Level 4 Data Results

evel 4 data results demonstrated that the MDTI had a positive impact on business results such as reducing the costs of doing routine jobs within Guthrie. The following calculations illustrate these findings:

HOW SAVINGS WERE CALCULATED FOR ONE

SPECIFIC PERFORMANCE INDICATOR:

MEETING HOURS

Average monthly decrease as a result of MDTI 3.57hrs

Average savings per manager per month $141

Total savings for 18-month period $164,910

Total savings adjusted for confidence level* $95,262

* The confidence level represents a conservative adjustment

made by each manager to his/her savings. For this project, the average confidence level was 58%; therefore, the savings were reduced by 42%.

L

Page 13 of 26

The total 18-month savings adjusted by confidence level for

all performance indicators:

SAVINGS

Meeting savings $95,262

Travel time savings $24,226

Travel mileage savings $1,580

Job task savings $150,284

Reworked tasks savings $180,669

Telephone time savings $45,886

Paper usage savings $1,318

Supply savings $4,830

TOTAL SAVINGS $504,055

The long-term savings projected over a 5-year period are

$1,025,175** ** These savings are an annualized figure that adjusts the

savings downward (more conservative as time goes on).

Page 14 of 26

Isolating the Effects of the MDTI

n order to identify how much improvement in performance was a result of the initiative and not related to other factors, we asked participants how much of their change/improvement was related to this initiative. A section

of the questionnaire was designed to collect this data: Although multiple strategies may be used to isolate the effects of a program, many could not be implemented due to the following issues:

• Control groups could not be utilized because of the diversity

of the participants' computer skills and the geographic spread of the participants.

• Levels 3 and 4 evaluations represented a change in the way

most managers think in regard to evaluating programs/ projects; although many could communicate application to the job, many had trouble expressing in terms of results.

• The scope of the MDTI was so complex that it made

evaluating at level 5 difficult.

• A needs assessment was not done up front; therefore, no baseline information was available. The omission of this needs assessment step made it more difficult in the long run to complete the ROI process.

• ROI expertise did not exist within GHS in order to effectively

conduct an ROI study on a project of this magnitude.

I

Page 15 of 26

Costs

he following figures represent the costs incurred as a result of the MDTI, which not only included the cost of technology, but also all development costs such as the cost of training materials, managers’ time attending

training, implementation and program management, advisement (instructors' and managers' time), and support. COSTS Development Costs 11,481 Training Sessions Instructor 7,157 Managers 12,795 19,952 Assessment Sessions Instructor 2,154 Managers 3,209 5,362 Committees MDTI Committee 6,198 Education Subcommittee 1,674 ROI Subcommittee 2,511 10,383 Equipment 149,426 Technician Time 54,183

TOTAL COSTS $250,787

T

Page 16 of 26

Level 5 ROI Analysis

ROI Calculation: Savings - Costs = ROI % Costs 504,055- 250,787 = 101% 250,787 What Does the 101% Indicate? A 101% ROI indicates that during this 18-month period for every $1 spent on Phase I of the MDTI, we realized $1.01 return on our investment.

Please Note: The costs and savings were calculated using each manager's salary versus an average, which gave us a more accurate calculation.

Page 17 of 26

Intangible Results

s a result of the MDTI, there have been many intangible benefits that cannot be converted to monetary values; nevertheless, they are critical to the evaluation process. Listed below are examples of these intangible benefits:

• Managers possess an expanded knowledgebase,

thereby, facilitating the acquisition of more advanced concepts and applications.

• Managers’ comfort levels using technology have increased, thereby, giving them the ability to deal more effectively with technological change.

• The MDTI has increased systems-oriented thinking by management. The technology has provided access to groupware, thereby, enhancing collaborative work on projects by managers throughout the system.

• Managers are more equipped with the tools to communicate with individuals and groups throughout the system, thereby, creating a more system-centered focus.

• The MDTI set the stage for moving the GHS IT plan to the next level. For example, the MDTI facilitates the application of other systems such as PeopleSoft that will be rolled out to managers.

Perhaps one of the most important intangible benefits of the MDTI is that many managers have significantly increased their computer literacy skills. Managers now have access to PC technology at their desktops. This initiative has helped to reduce the "fear of technology" that many people experience when using computers. In addition, the initiative has helped to "level the playing field" by giving all managers in Phase I (and eventually Phase II) the same basic but powerful tools to support them in getting their work accomplished.

A

Page 18 of 26

How has the MDTI Promoted the Learning Organization?

n 1994, Ralph Meyer created a new vision for GHS, "To become a high-performance organization, within the cultural context of a learning organization."

By definition a learning organization is one in which people at all levels, individually and collectively, are continually increasing their capacity to produce results. With Guthrie's continued commitment to become a learning organization, the MDTI clearly exemplified the concepts and disciplines of the learning organization in action. Furthermore, our commitment to calculating the financial return on our investment forced us to look at those results in terms of the bottom line. The five disciplines essential to a learning organization and the role of the MDTI in embracing these concepts are outlined below:

• PERSONAL MASTERY – In order to implement a training strategy aligned with the needs of participants in the MDTI, we assessed each manager's technology skills against a defined set of competencies. Once assessed, we could begin closing the technology/learning gap by creating an Individual Technology Development Plan (ITDP) that would put them on a learning path to higher performance. Furthermore, the MDTI created an environment that encouraged all managers involved to develop the necessary skills that would produce results. By applying newly learned technology skills to their job on a daily basis, managers mastered new skills and continued to expand their capacity to learn and apply the technology in more sophisticated work processes.

I

Page 19 of 26

• TEAM LEARNING – The MDTI team combined the collective talents of many. The team was composed of cross-functional representation throughout the System and relied on each other to address problems, challenge assumptions, and receive feedback. In addition, managers participating in the MDTI learned from each other by sharing how they were applying the technology and training on the job as well as partnering with other managers to facilitate the learning process.

• SHARED VISION – From conception through imple-

mentation, the MDTI Team shared the same vision of using technology as a vehicle to propel the organization to higher performance. Their vision was communicated and shared throughout the System via entity-specific meetings, department head meetings, leadership forums, and assessment and training sessions. This shared vision fostered commitment and support that contributed to our vision becoming reality.

• MENTAL MODELS – The MDTI challenged us to adopt

different mental models that would enable us to clarify and improve our ability to produce results. By answering questions such as "How do we link training to the job," "How do we ensure learning has taken place," "How do we ensure transfer of training to the job," "How do we create results," and "How do we ensure that our investment impacts the bottom line," the MDTI team was challenged to develop new and different strategies to accomplish our goals and objectives and help our vision become a reality.

Page 20 of 26

• SYSTEM THINKING – The MDTI forced us to focus on the "big picture" and the interrelationship of the entire System rather than cause and effect chains. It facilitated access to software such as groupware that would enhance collaboration and communication throughout the System. Using technology as a vehicle to enhance performance fostered an environment of teamwork, possibility thinkers, and people who were beginning to learn, recognize, and understand the system in which they operate and the role they play in making it successful.

Page 21 of 26

Summary

uthrie’s commitment to high performance, learning, responsibility, and accountability led this team on their journey to calculate the ROI on the Management

Development Technology Initiative. Throughout this journey, the team faced many challenges as well as developed insight into the ROI process and how critical the process is to demonstrating bottom-line results and monetary payoffs. Although we have traditionally and consistently evaluated programs/projects to measure satisfaction and learning, we never conducted evaluations at the ROI level. In order to begin collecting data we recognized that a chain of impact needed to occur through the levels. Therefore, in addition to evaluating for satisfaction and learning, managers were asked to identify how they were applying newly learned technology skills to the job as a result of the initiative. In so doing, we quickly learned that in order to take credit for the initiative’s impact a bench level of what that performance was before we launched the initiative should have been established. As a strategy to isolate the effects of the initiative, we asked managers to estimate what percentage of improvement was attributable to the initiative. It was during this stage in the process that the group uncovered barriers that would add to our list of lessons learned. Issues such as culture change, project scope, lack of needs assessment information, and the group’s limited background in the ROI process clearly helped us identify what

G

Page 22 of 26

could be done differently with future projects. Although we recognized that managers found it difficult to calculate these estimates, we also realized the advantages of using the participants’ estimates that were provided. Factors such as its simplicity, credibility, and its low cost to implement all made this strategy attractive to the team. Capturing the costs of the MDTI was a challenge. In our effort to provide accurate, reliable, and realistic figures we included costs such as all costs related to course development including assessment and evaluation, cost of materials, equipment, salaries of managers, instructors, and committee members, as well as any costs related to supporting the initiative. Every effort was made to ensure that all costs were captured. We quickly learned that tracking and documenting all costs throughout the process is critical to calculating a realistic ROI. Timesavings, improved productivity, improved quality, and improved performance were measured and converted to monetary values in order to determine program benefits. For example, the group recognized that having access to technology has helped facilitate teamwork, cooperation, decision making, and communication. In addition, an increase in job satisfaction as well as a reduction in stress levels has also been noted. Although intangibles such as those outlined may not be perceived as being as valuable as monetary measures, they nevertheless are an important piece in the overall evaluation effort. As the ROI Committee members worked their way through the various stages of the ROI process many lessons were learned that reinforced our commitment to the process. First, a comprehensive needs assessment on the front end is critical to the success of any ROI project. Second, in order to increase the credibility of our work in the future more than one strategy for isolating the effects of training will be used. Third, we recognize

Page 23 of 26

how important the role of educating others involved in the process is to acquiring reliable and credible data. Last but not least, we learned the importance of selecting the appropriate projects to conduct ROI analysis. The significant financial investment up front for such an initiative can camouflage program benefits especially if long-term benefits are not considered as part of the initial implementation strategy. One thing is certain ~ the 11 Guthrie employees involved in the original ROI training are, as a result of that training and continued work, convinced of its potential value and are committed to its adoption as an ongoing practice at GHS. The biggest challenges we have faced involve our own inexperience with the process and the time commitment required for implementing the process from beginning to end. Team members have felt frustration, which may have manifested itself in the project selected. The scope of the MDTI may have been too large, involving an entire initiative rather than a particular training program. The team, however, has found support in each other and in our own belief in the value of ROI to keep us moving forward. As we develop our expertise and see our projects through to fruition as well as continue to work and partner with managers in the ROI process, we feel confident that our enthusiasm and past success will help sustain us through system implementation.

Page 24 of 26

In moving forward, our goal is to affect the Guthrie Healthcare System culture in such a way that the understanding, acceptance, and adoption of the ROI process becomes routine. We plan to work with senior staff on selected ROI projects in addition to working on increasing managers’ comfort levels with the ROI process. This will be accomplished by educating managers on the five levels of evaluation and a variety of approaches to calculating ROI. In addition, we will assist managers in conducting needs assessments and partner with them in subsequent projects.

Page 25 of 26

What’s Next

lthough the ROI process will not be applied to Phase II of the MDTI, we will incorporate lessons learned from this experience to improve the usefulness of the information

that we obtain from future ROI projects. The ROI process reinforces the process of conducting a thorough needs assessment because of the evaluation problems that can develop without one. Since the ROI team just recently attended the Needs Assessment workshop we did not conduct one on the MDTI. However, we will now apply that knowledge on future projects to determine training needs. The quality of this data will most likely lead to well designed interventions that will solve real performance problems and impact business results. Furthermore, evaluation after the program will become easier because the criteria for each level will be determined before the interventions are designed. The ROI team recognizes that conducting an ROI analysis in the future should be reserved for programs/projects that meet the following criteria:

• Involve a large target audience • Are expected to be viable for a long time • Are aligned with the organization's strategic objectives • Are costly to implement • Are highly visible • Have a comprehensive needs assessment

A

Page 26 of 26

As a result of the ROI study that was done on the MDTI, managers have an appreciation for the ROI process and the performance indicators that impact organizational results. Our intent is to reinforce ROI concepts through education and application in the workplace. Furthermore, members of the ROI team will partner with managers throughout the System to assist them in future ROI studies that meet the criteria defined above.