Entry to Brandon-Halltigerstaffdev.pbworks.com/f/2008 Kindred Healthcare T… · Web viewThe Nursing Center Division is a big proponent of web based training (WBT) for application

Entry to The Tiger Collaborative 2008 Staff Development and Continuing Education Case Studies

January 2008

CSI: Critical Success Indicators

Submitted byKindred Healthcare

Darlene Thompson, VP Clinical Information Systems and TrainingDavid Ellis, Director Training and QA

Gina Harsanyi, Manager Training Development 680 South Fourth Avenue

Louisville, Kentucky [email protected]

[email protected] [email protected]

502.596.2640502 596 6495502 596 7618

mailto:[email protected]

mailto:[email protected]

mailto:[email protected]%0D

Kindred Healthcare 2008 The TIGER Collaborative Case Study Entry2

Link to Business Priorities

This case study addresses Kindred Healthcare’s challenge of implementing an electronic ordering system for food and supplies (DSSI) using an e-learning training strategy to a target audience whose dynamics included a population with little familiarity or comfort with computer technology. There was also the added complication of field resistance to a standard approach to ordering items from approved national vendors, rather than local companies. Finally, Kindred’s consultant and the vendor were resistant to the training strategy.

Kindred trained the end-users and achieved a 95.3% completion rate by 850 learners, completing almost 8,800 lessons over the duration of the six-month rollout. This entry will describe our training strategy, indicators for success, and measurement processes and tools.

Setting

Kindred Healthcare, Inc. provides long-term healthcare services primarily through the operation of nursing centers and hospitals. Kindred’s Nursing Center division operates 285 nursing centers in 32 states and a rehabilitation therapy business.


The Nursing Center Division is Kindred’s largest organization regarding number of nursing centers, geographic diversity and number of staff. The DSSI project allowed the Division to standardize purchasing practices across nursing centers, as well as capitalize on volume discounts through national vendors. The application vendor, DSSI, provided these opportunities through contracts with various suppliers and exception reporting capabilities.

There were four distinct roles for application usage. Each role was assigned a subset of required training content. These roles spanned both the Center and District/Region levels in the organizational hierarchy. At the Center level the roles included Supply Order Approver, Central Supply Clerk and Dietary Orderer. The District/Region role was a combination of Dietary Orderer and Formulary Manager.

The Supply Order Approver role was assigned to the Administrator and Business Office Manager job titles at each Center. There is not an actual job title of Central Supply Clerk at Kindred, so each nursing center identified a person who would fulfill that role in the application. In many cases the user performing the Central Supply Clerk duties had no, or limited, computer experience. Typically, this role was assigned to either a unit ward clerk or a CAN to be performed for a few hours each week.

The Dietary Manager at each nursing center was trained to order all food purchases from Sysco. Dietary Managers could also select and submit supply orders (non Sysco items) for approval. District and regional nutrition personnel could perform the duties of each center’s Dietary Manager as well as additional formulary and reporting capabilities.

So, in many cases we had three to five learners (including backups), located in 285 nursing centers in 32 states requiring training, a significant number of which had no computer experience or comfort. An additional consideration is most of these positions experience turnover in the 50-60% range.

Design and Delivery

The Nursing Center Division is a big proponent of web based training (WBT) for application implementations and turnover training, particularly since the successful SAP upgrade in 2002. There were two main concerns with using WBT as the training strategy for this implementation. The first stemmed from the anticipated percentage of new computer users inherent in the Central Supply Clerk population and how to train them using unfamiliar technology. Historically, application WBT had been used on a target audience whose job duties required the daily use of a computer and those skills were incorporated into the job description and hiring requirements. The Central Supply Clerk role would be filled by individuals who had previously ordered supplies using a paper-based system and placed orders via the telephone. Also, since this role was a description of duties, rather than a specific job title, computer literacy was not a part of the job description or hiring requirements.


The second concern stemmed from DSSI’s past implementation experience with other clients and anecdotal experience by the independent consultant Kindred hired to facilitate the implementation. In short, there was resistance to using WBT as the primary delivery method for implementation. Kindred’s position was that WBT would prepare the end users for the new system, as well as being fiscally responsible. There was agreement by all parties that WBT would be the best method for turnover/new hire training. Discussions were ongoing throughout the planning phase of the project to identify the perceived differences between rollout and turnover training and why different delivery methods were appropriate and necessary.

The training strategy was written to incorporate the Training Department’s recommendation of WBT as well as the fallback delivery method for live instructor led classroom training. The vendor and the consultant were very vocal about their lack of confidence in an e-learning delivery method due to another client’s previous attempt. The client was evidently very similar to Kindred from an organization standpoint, with a number of nursing centers dispersed over a large geographic area and involved a number of Sysco distribution centers. Its attempt to use WBT differed from Kindred’s plan due to the design of the modules. Kindred’s plan was based on using short modules with interactive simulations, rather than “page turners” with few graphics and no opportunity for hands on practice in either simulations or a training database.

The decision was made to pilot WBT with stringent measurements and anecdotal feedback from pilot end users. We especially needed feedback from novice computer users, since the pilot sites would be trained exclusively by computer.

Define Specific Criteria for Success

The team developed criteria for success in conjunction with the business sponsors and the consultant hired for the project. The main purpose of the criteria was to address the specific concerns raised by the consultant. They included time commitment, completion and retention as well as logging and classifying support calls. Specifically they were:

Kindred Healthcare 2008 The TIGER Collaborative Case Study Entry 5

Indicators of WBT SuccessWt Criteria Activities Specifics/parameters Food Supply

Core roles took and passed WBT courses

Knowledge checks and simulation scores meet minimum passing requirement

1) Determine passing score for each lesson and simulation.

2) Determine acceptable percentage of learners that complied with training schedule.

1) 80%

2) 80%

1) 80%

2) 80%

Pilot scenarios completed in DSSI training environment

Develop scenarios that trained users will enter into DSSI training site prior to go-live.

1) Who will monitor?

2) Can DSSI assist with reporting?

3) What is acceptable percentage of compliance?

1) Implementations Project Leader2) Yes

3) 100

1) Implementations Project Leader2) Yes

3) 100

Time commitment for learners to take WBT courses

Track average (incl. Max) duration of learners to complete course map/courses.

1) Determine acceptable average duration for learners to complete required curriculum

1) 8 hours for full curriculum, including PC Nav

1) 10 hours for full curriculum, including PC Nav

Track type and content of questions received during pilot related to DSSI

Identify common questions where information needs to be enhanced/added in WBT modules.

1) Who takes calls from pilot learners?2) Establish categories for questions:

a) Education (how to)b) Policy & Procedurec) System (I.e. login will not work)d) Validation (learner knows next step, just looking for confirmation)

1) Implementations will take all calls from learners for tracking and triage

2) See categories at left

1) Implementations will take all calls from learners for tracking and triage

2) See categories at left


The above chart identifies the four main success criteria and the method or tools used to measure their success (Activities column). The specific considerations or responsible parties for gathering the information (Specifics/parameters column) and specifically defined success indicators (Food and Supply columns) are also identified. The weight column was included to identify “show stoppers”, however the weights were never assigned, since all of the success indicators were met or exceeded. Each of the indicators and its measurement tool will be discussed.


Implementation

Core Roles Took and Passed WBT CoursesTraining was managed by Implementation resources at the Corporate office using two main tracking tools: data and reports from the LMS database and tracking available within the vendor’s training site. The reports developed for this implementation reflected only passing training course completions. If a learner took a course but did not pass, the learner did not get credit on the report, and so the totals did not reflect unsuccessful passing attempts. Here is a sample from the second pilot training wave:

The report snapshot above shows the full results for the Atlanta Sysco House and the individual results for two of the five Kindred nursing centers associated with it. The entire Sysco House completed and passed 87% of their assigned training. Each nursing center is also listed with a completed and passed percentage. Athena Rehab of Clayton, for example, completed and passed 96% of its assigned courses. As shown in the Indicators of WBT Success chart, success was measured at 80%, so there was absolutely no question the criteria had been met for both the Sysco House average and each nursing center shown. The only nursing centers that did not perform at 80% either


had open positions for a key role(s) or newly hired resources who completed their training after the “wave” went live on the system.

At the detail level, each learner, its specific training requirements based on role, and individual completion and passing success are also reported so that struggling learners could be identified and assisted. Other reports were created using the LMS database to provide simulation scores for each learner, lesson and simulation within the assigned WBT modules. The minimum passing score for a simulation is 80%. There are no restrictions on the number of simulation attempts for a learner and the feedback we received is that many learners repeated the simulations until they performed at 100%. This was very welcomed feedback, since the purpose of the simulation was to provide system experience and reinforce the ordering and approving processes in a safe environment.

FULLNAME Lesson Name Simulation ScoreEllison, Ronald Creating Routine Orders Order Basket Method simulation. 90Ellison, Ronald Routine Orders Add to Order simulation. 90Ellison, Ronald Routine Orders Create Routine Order simulation. 90Ellison, Ronald Routine Order Functions Routine Order Functions

simulation. 90

Ellison, Ronald Routine Order Functions Print Routine Orders simulation. 90Ellison, Ronald Find It Feature Find It Feature simulation. 90Ellison, Ronald Invoice and General Ledger Spending

ReportsGL Reports simulation. 90

Ellison, Ronald Go To Feature Go To Feature simulation. 90Ellison, Ronald View and Submit Orders Deleting Items simulation. 90Ellison, Ronald Routine Orders Create Routine Order simulation. 90Ellison, Ronald Creating Routine Orders Scratch Method simulation. 93Ellison, Ronald Food Categories Food Categories simulation. 93Ellison, Ronald Order Entry Order Entry simulation. 95Ellison, Ronald Routine Order Functions Routine Order Functions

simulation. 100

Ellison, Ronald Routine Order Functions Print Routine Orders simulation. 100Ellison, Ronald Invoice and General Ledger Spending

ReportsGL Reports simulation. 100

Ellison, Ronald Invoice and General Ledger Spending Reports

PO-Invoice Lookup simulation. 100

Ellison, Ronald Invoice and General Ledger Spending Reports

PO-Invoice Lookup simulation. 100

Ellison, Ronald PO Summary Reports PO Summary simulation. 100Ellison, Ronald View and Submit Orders Edit GL simulation. 100Ellison, Ronald View and Submit Orders No Edits simulation. 100Ellison, Ronald View and Submit Orders Edit Qty simulation. 100Ellison, Ronald Product Categories Product Categories simulation. 100Ellison, Ronald Order Entry Order Entry simulation. 100Ellison, Ronald Routine Orders Add to Order simulation. 100Ellison, Ronald Average 95.24


Pilot Scenarios Completed in DSSI Training EnvironmentThe vendor provided a training environment for pilot learners to place and approve mock food and supplies scenarios. The business owners and the project consultant created these scenarios to provide an opportunity to track the application of the knowledge and skill transfer that occurred during the training. Learner results were compared to expected results to validate learner competency.

Due to issues at the vendor, we were unable to capture this information and the original indicator was modified to monitoring activity within the system by the Implementation Project Leaders.

Time Commitment for Learners to Take WBT CoursesThe project team determined that the average time for learners to complete the WBT curriculum could not exceed the time instructor led training would take to complete each “track”: Food or Supply. Using those parameters, the Food curriculum could not average more than eight hours and Supply could not exceed an average of ten hours. Both curriculums included the PC Navigation training, so in reality the times identified were for computer novices.

Information was downloaded to a spreadsheet for calculation and analysis. The spreadsheet below represents the average duration of each WBT module that was started and completed in the same day. Since it was a guided, self-paced delivery method, it was possible for a learner to begin a module on one day and finish on another. The Number of Learners column represents the number of learners who started and completed the identified courses on the same day. This number was used to determine the average duration time for each course.

The Dietary Manager or Food curriculum acceptable average for completing the WBT was set pre-pilot at eight hours for the Food and PC Navigation modules only and did not include any of the Supply courses. Mid-first wave (pilot 1) identified a gap in the Dietary Manager’s training and the Central Supply Clerk’s training curriculum was added to the Dietary Manager’s role. The 9.7 hours represents the average time to train on both the Food Module and the appropriate Supply Module courses. The business owners determined this was an acceptable time and counted it as an indicator met.



Type and Content of Questions Received During Pilot Related to DSSIThe calls made by the field to the Implementations Project Leaders were tracked in the Support Desk tracking system. The following categories were used:

1. Education 2. Policy & Procedure3. System 4. Validation

The Education and Validation categories were really the only two that reflected on or affected the training. Over the course of both pilot waves, totaling twenty nursing centers and eighty users, the number of Education calls was 32 and there were 7 validation calls where the learner knew the steps to perform but needed a voice on the phone for confirmation and encouragement. Questions received in the Education category were resolved either via telephone or, if more involved, the Implementation Project Leader conducted a one-on-one distance learning session with the learner using both demonstration and reverse demonstration.

Anecdotal Feedback from the CentersPilot Center feedback was also an integral part of implementing the system using an e-learning training strategy. It was decided that the best way to obtain the required feedback was via conference calls with all four learners from each pilot site. The Corporate representatives included the Training Department, Implementations Department, Project Management and the business owners. Excerpts from the summary feedback report presented to the entire project team are below and focused on two main concerns:

Did the courses and lessons prepare end users to perform necessary job functions in the DSSI system

What improvements can be made

The results of the pilot are as follows:

Did the courses and lessons prepare end users to perform in the DSSI system?

All participating nursing centers were able to perform their ordering tasks in the DSSI system. Only one nursing center indicated that they would have preferred

face to face training at a Regional Training Center, however the nursing center indicated that the WBT courses did prepare the end users for ordering via DSSI.


Training Period Duration The nursing centers recommended between two and three weeks for

the actual training time frame. 80% of comfortable computer users felt that two weeks was long

enough for completing the courseware, but recommended providing the course maps three weeks out, in order to provide time for scheduling the courses.

Computer novices asked for a week prior to actual DSSI training to become familiar with the computer. Two of the three Seattle nursing centers with computer novices

made use of Business Office Managers and/or other computer savvy employees within the nursing centers.

The Baraboo nursing centers were assisted by the Regional Dietitian for basic computer skills training and setup of Lotus Notes.

The Atlanta nursing centers were assisted by the Regional IS Liaison for basic computer skills training.

What improvements can be made?

User Manuals 50% of the nursing centers requested a hard copy manual on site.

This request was in response to the “what could we improve” question during the call. There is a manual that will be placed online, however providing one

hard copy per nursing center is an option. The recommendation would be to time the delivery of the manual at the time of Go-Live and after the WBT courses have been completed.

Obtain a price per manual to determine the additional cost to the project.

Training Documentation The nursing centers also requested a “what’s happening when”

document or checklist that helps manage the training. A draft has been included.

Pilot Conclusions and Recommendations

Web based training is a successful delivery method for DSSI system training. Users experienced difficulty with distractions from everyday nursing center

duties. Further emphasis should be placed on utilizing the training PC at each nursing center, or if that is unavailable, to seek out a quiet computer workstation in the nursing center or go to another nursing center to take the WBT.


One hard copy of the DSSI user manual will be made available for each nursing center.

An Implementation Management Checklist covering the milestones to achieve in order to begin to use the live DSSI system, including scheduling and taking the WBT courses, will be provided to each nursing center administrator prior to the executive briefing.

A Basic Computer Skills Training Checklist is attached to assist power users when preparing novice PC users for training and job duties in the DSSI system.

Post Implementation SurveyThe business owners also developed and conducted a post implementation survey using SurveyMonkey.com. The survey requested anonymous end user feedback on a number of areas, including training. The training component results are based on a four-point scale and the excerpt below shows the questions and results related to training and implementation:

What is your role? Response Total

Central Supply Approver 14Central Supply Orderer 20Dietary Orderer 21

Total Respondents 55(skipped this question) 0

Please answer the following statements:Strongly Disagree

Disagree Agree Strongly Agree

Response Average

I was able to schedule courses with the "Course Map" prior to taking a course.

1 5 32 13 3.12

I was able to complete my courses in the time allotted.

2 3 31 16 3.17

The web-based training prepared me to use the DSSI system.

1 6 32 13 3.1

The DSSI system is easy to use for ordering materials for my nursing center.

2 7 28 15 3.08

Total Respondents 52(skipped this question) 3


Respondents were also offered an opportunity to provide additional comments about any and all aspects of the project, but those comments addressed system feedback, rather than training or implementation, and so are not included.

Business Impact

Obviously, the implementation of the electronic ordering system, DSSI was a success and WBT was a proven and accepted delivery method, even for computer novices. Kindred also emerged from the project with an additional benefit. The beauty is it was totally unintended and therefore all the more valuable. It stems directly from the methodology. The DSSI project, its Indicators for Success, and measurement processes and tools formalized a repeatable practice that Kindred now applies to all system based projects utilizing an e-learning training strategy. A modified version is applied to traditional delivery methods and those with a blended approach.

Documents

Entry to Brandon-Halltigerstaffdev.pbworks.com/f/2008 Kindred Healthcare T… · Web viewThe Nursing Center Division is a big proponent of web based training (WBT) for application