Optimal Resume at WALDEN UNIVERSITY CAREER ... · Web viewDevelopers of four major health technology information models commonly used were created by innovators Rogers (diffusion

Running head: EVALUATION PROJECT PART 6 1

Evaluation Project Part 6: Evaluation Plan

Steven LeBeau

Walden University

System Design, Planning and Evaluation (NURS - 6431 - 2)

February 6, 2015

EVALUATION PROJECT PART 6 2

Evaluation Project Part 6: Evaluation Plan

Constructing an evaluation plan has many stages; six to be exact in this plan. Detailed

explanations that were collected from the six individual papers will be used to illustrate each

stage of the evaluation process. The progression will start by selecting a very refined PICO

question that will be used to guide the research study and help develop measurable evaluation

goals. Using research skills, a methodology framework will be chosen that will guide the

evaluation plan and assist in implementing the correct research design plan that will best answer

the research question. Once the research question, methodology, and design are chosen an

evaluation tool must be found that will collect data to answer the research question. A research

tool must be able to demonstrate that it is reliability and has validity so that other researchers

may produce similar finding using your methodology, design and tool. When either reliability or

validity are compromised, then a limitation exists in a study and must be reported. Limitations

are vital to report at the end of the research in order for other researchers to extend your research

by eliminating or accounting for the limiter and prove their research question true void of the

limitations. Later in this paper ethical issues and concerns will be discussed along with what

criteria is used for defining a success in a project. Finally this research paper will outline the

limitations found in this research project.

The six assignments have been completed and now the results shall be aggregated into

one cohesive document for you the reader to better enjoy the process. The purpose of this paper

is to educate the reader about the six stages of evaluating a health information project through

aggregating the findings from six scholarly researched papers.


Part 1: Evaluation Plan Focus

Evaluation models and theories form frameworks for informaticists to plan ahead of a

major implementation project thereby increasing the development plan to fit the host (Coplan &

Masuda, 2011). Starting with a sound framework is always a good way to get organized. My

scenario is to implement an electronic health record (EHR) system at both an acute care hospital

and its ambulatory center. Implementing an electronic health record is considered by many

project managers to be a major project due to the management of the many different systems,

goals, and stakeholders involved during installs. The key is to have a knowledgeable

informaticists that has training and knowledge about the technology evaluation process being

implemented so that the correct model can be chosen to drive the framework of organizing the

implementation.

Most recently I conducted a writing assignment on four different theories of evaluating a

project. Innovators in informatics research have been working in the field since 1959.

Developers of four major health technology information models commonly used were created by

innovators Rogers (diffusion of innovation), Christensen (disruptive innovation), Davis

(technology acceptance model), and multiple developers Trist, Bamforth, and Emery

(sociotechnical theory.) There is no one complete model that will work on all implementations,

but using frameworks to implement EHR systems will move informatics closer to finding a

consistent implementation program with rigorous approaches to data collection, measurements

development, and theory building (Rippen, Pan, Russell, Byrne, & Swift, 2013). Most models

were established in the mid 60’s and 70’s and have been in use for many years. The fact that a

theory can exist as a foundational model this long gives great praise to these theories and

theorists. The models work but as you will see later, the question of methodology, tools, and


validity play a large part in why some research is considered standards and others are considered

pretenders.

I chose Technology Acceptance Model (TAM) as my model in order to be proactive in

predicting end users beliefs, usefulness, and attitudes towards technology. Hyun, Johnson,

Stetson, and Bakken (2009) researched the evaluation process of TAM theory looking at the end-

users viewpoint. Hyun et al. noted that attitudes by the staff about the perceived ease of use

(PEOU) and perceived usability (PU) of the system to enrich their working environment could

equate into predictable behaviors. The connection is that every unit will have different attitudes

about the technology but by identifying those with negative perceptions to the technology, the

TAM model has techniques that can help move workers out of one stage into a more productive

stage.

My goal for this course project is to define, test, evaluate, and approve a usability

framework that will assist me in implementation projects. In order to keep the scope of this

project under control, one viewpoint from the nursing perceptive will be used. End-user shall

refer to medical surgical nurses which will serve as my population of clinical area of

documenting on patient care. Now I need a research question that will generate data for my TAM

theory.

Part 2: PICO Question for Evaluation Project

Stating a research question should be treated the same way you might tell a stranger in

your city how to get to an exact location. The traveler needs details and the data must be accurate

or the traveler will get lost along his way. Stating the research question is one of the most

important steps in the research journey (Polit, & Beck, 2012). I wanted to know how nurses’


would adjust to upgrading to a new EHR system. What is needed for a great research question is

what Polit and Beck noted as a foreground question that involves thinking of all the information

about that subject and converting that data into a question that can be answered with research

evidence. Drilling down to the most specific level of researching a subject is a good starting

point. The PICO acronym: P = population, I = intervention or issue, C = comparison of interest

O = outcomes Y = time are used to form the research question. LeBeau (2015) original research

question developed with much help from my classmates and Dr. Wilson:

From the viewpoint of a medical surgical nurse, what is the perception by the nurse

concerning the ease of use and usability during the first initial assessment at the start

of their shift between the old Cerner system and the newly implemented iView Cerner

documenting system? (p. 1).

Part 3: Literature Review

From the viewpoint of a medical surgical nurse, what is the perception of the ease of use

(PEOU) and perceived usability (PU) when transitioning between the old EHR system and the

newly implemented documenting system? Many earlier EHR system have already become

obsolete and need upgrading to achieve the meaningful use requirements (Pfoh, Abramson,

Zandieh, Edwards, & Kaushal, 2012). The question above has relevance because of the limited

research done on transitioning between EHR systems and the possibilities of unique challenges

not experienced yet by researchers (Zandieh, Abramson, Pfoh, Yoon-Flannery, Edwards, &

Kaushal, 2012). Key new challenges for successful upgrade implementations of EHR system

may collide with the technology acceptance model (TAM) of PEOU and PU because of prior

computer experience (Zandieh et al., 2012).


Throughout the literature review, I really started to become knowledgeable about this

new phenomenon. At least I was collecting some evidence but not all of the research question

were working. I could not find any good research on Cerner upgrades, PEOU, or PU information

and the medical surgical population was really making the searches harder to find evidence that

met all the criteria. I had to modify my research question in order to collect reliable information

on upgrading to a new EHR system. I changed my research question slightly by removing the

Cerner information and the first assessment comparisons between the old system and the new

system. LeBeau (2015) revised PICO:

From the viewpoint of a medical surgical nurse, what is the perception of the

ease of use (PEOU) and perceived usability (PU) when transitioning between

the old EHR system and the newly implemented documenting system? (p.1).

Not as foreground of a question as before but a much better snapshot of the situation that will

enable me to produce much more evidence through other researchers invested time. The use of a

summary table was utilized during the collection stage of the literature review. I have attached it

as an object below. Double click on the icon and please review how the information collected

assisted me in writing my literature review using comparisons and contrast methods. The review

table adds a little more time but now equipped with more experience I can actually use the table

as an effective tool for supplying evidence for and against a topic.

Double click object


Finally, the overall statistical appeal results for transition from the old EHR to the new

EHR were favorable (Zandieh et al., 2012). The author noted again the novelty of transitioning

between old EHR systems to new EHR systems and recommended a few observations about

potential barriers: (1) Some nurses will continue to hold onto their old systems just because of

the familiarity (2) assess nurses functionality on the old system before converting over to the

new system to assure quality, safety and efficiency (3) even with lower levels of functionality the

majority of nurses still liked the new upgraded EHR (81%) better than the old EHR (61%) as the

author noted that not all useful functionalities are used by each nurse and starting on a new

system gave these nurses a chance to start over with some experience thereby giving them a fair

chance on the playing field (4) hospital informatics, leaders, and vendors should work together to

identify what are the most desired features the staff would like to see on these transition and

make them work. This single act could ensure overall satisfaction among the whole staff while

the less glamorous functionalities could be taught later (5) lastly, manage practitioner’s

expectations early is noted as a critical step thereby decreasing the let down from too high of

expectations posted by the leadership team. These are new finding from all the other graduate

research studies I have done on EHR implementations.

Part 4: Evaluation Methodology

Good documentation of methodology along with accurate tools will produce evidence.

Researchers need to be able to follow the road map just as the traveler in the research question.

How can someone recreate a research study with laissez faire documentation that leaves many

questions unanswered? Proper documentation involves detailed explanations about the steps, the

time frames, and the methods used to recreate the plan with a similar scenarios (Polit, & Beck,


2012). Methods are like a cook book in some ways. The exact ingredients have to be involved in

order to recreate that special moment. Additionally, this author can now see the importance of

selecting studies that mirror my research question, my population, my surroundings, my

limitations, and evaluating the methodology explanation for an exact plan and the reliability and

validity of the tools used to collect the information. All of the tools and processes seem daunting

now but once conditioned to conduct proper research the skills of performing a proper research

will be an exciting journey.

In this section of the evaluation plan the methodology becomes the main focus in

determining the study design, the measurement methods to collect data, and the data analysis

procedure used to extract meaningful use information (Friedman & Wyatt, 2010). The evaluation

process is concerned with how well a system operates and also provides a systematic approach to

evaluating the systems input performance and output deliverables (Centers for Disease Control

[CDC], n.d.).

Selecting the correct candidate for collecting the data falls to the research team. In

general that person should be a neutral party with excellent character (Polit & Beck, 2012). A pre

and post transition plan would be developed similar to the Zandieh, et al. (2012) survey. Initially

a letter with the surveys link site would be delivered by mail along with a statement letter for the

surveys mission. Polit and Beck (2012) noted that one notice is often not adequate for a complete

return of survey questionnaires and therefore a secondary mailing would follow two weeks later

along with two additional emails over the next two weeks. The pre-transitional period would

start approximately one month out from the go-live transition. After transition and console with

the informatics team, at the three month mark, an email follow-up re-evaluation survey would


arrive to the same participants. The exact same manner of delivering the email and the exact

same reminder dates would be used to maintain some consistency for the subjects. The post-

transition period would last approximately ten months so all subject s had ample time to

participate.

Part 5: Evaluation Tool

Conducting the literature review provided assertions about the PICO question and

generated new information that deserves consideration of inclusion into the research project by

way of the research tool. The question researchers will want to know is the tool reliable and

reproducible and is the tool valid, accurate, and measuring what it is supposed to measure

(Laureate, n.d.f.). If the researcher picks the incorrect tool for their research then scholars will

have a problem with the evidence and researchers may not want to conduct further studies using

possibly flawed set-up methodology and tools (Polit & Beck, 2012).

The Likert scale is an excellent tool used to identify declarative items that express a

viewpoint on a topic (Polit & Beck, 2012). The particular survey tool came from the AHRQ

website and was a health IT survey compendium selection chosen over several other survey

tools. The name of the tool is the Employee / Staff Pre-Go-Live Expectations / Perceptions

Clinical Information Systems Survey (Agency for Healthcare Research and Quality [AHRQ],

n.d.). Selecting a survey tool from the AHRQ site assures the researcher that the survey tools

have been scholarly assessed for their reliability and validity (AHRQ, n.d.). The technology

acceptance model (TAM) was developed by Dr. Fred Davis in 1985 and is still recognized as one

of the best tools for evaluating and predicting factors that are qualitative in nature such as

internal beliefs and attitudes. The model is concerned with connecting attitudes with predictive


behaviors analysis outcomes (Kowitlawakul, 2011). In my evaluative plan one of the main

criteria was the view point of the medical surgical nurse on perceived ease of use (PEOU) and

perceived usability (PU). This scale is particularly made for providing ranges of granularity from

strongly agreeable to strongly disagreeable on many topics (Polit & Beck, 2012).

The last stage of the research process after data collection is data analysis. A trained team

of data collectors would have already been established such as end-users, informatics nurses,

project manager, and other stakeholders that would assist in the interruptions of the evaluation

data into themes (Polit, & Beck, 2012). Goals for the research project of accurate information,

inclusion criteria, rigor of the study’s methodology and tools are reviewed and finally the

statistical reliability and external and internal validity measurement would be calculated (Polit, &

Beck, 2012). Trends found in PEOU or PU would be further analyzed by demographics

information, years of service, length of time at the facility and then all the information is

combined with the Likert scale results to form a complete assessment. All this information

develops a profile that will be plugged into the TAM theory with instructions of moving staff

members from one area to the desired area Thoroughly testing the evaluation data of the survey

will produce statistical results that will explain the validity of the data, limitations, consensus for

what is working, what is not working, and what has to be done to make the system right.

Limitations

Limitations of a study’s design or instrument is a system wide bias that the researcher

could not control and may very well render the research useless (Price & Murnan, 2004).

Developing an effective evaluation plan involves understanding the stakeholders, how the new

electronic health record (EHR) will change their environment, and establishing criteria for


defining success (Coplan & Masuda, 2011). Limitations are thought by some as a weaknesses in

the planning stages of a project or failures in processes by others. Polit and Beck (2012) noted

that through the identification process of listing the limitations by researchers, new key findings

can then result in eliminating the limitations and pushing the study forward.

Evaluation is a process like a running stream that relies on ethical documentation of the

studies limitations to keep the waters of wisdom flowing. Bumps and chips are a natural order

that will eliminate the unusable methods or information. What eventually happens is the a new

system emerges that has been polished for the next group of talented researchers to build upon

the original study by slowly purifying and developing the evaluation process into a refined and

reliable tool for developing predictable results with great validity. In a more general way,

Friedman and Wyatt (2010) explained the evaluation process in informatics as a method of

looking at a range of different types of informational resources, both human and technical, and

evaluating their effects on people and their organization.

Ethical Issues and My Solution

Data exchange across hospital pathways and in some instances across interchanges to

consulting doctor’s office for reviews of personal information about patient’s health concern is

going to be the standard and not the exception. Informatics must find secure means of

communicating the information in a safe environment for ethical reasons and for confidentiality

reasons. Friedman and Wyatt (2010) note that one approach of maintaining confidentiality is

“anonymizing” the data by removing key identifiers. Berner (2008) noted that because information

technology is transferring information across domains that we may have little knowledge about the

recipient’s intention with that technology and the patient’s valuable personal information. Perhaps a new

national unique numbering system that is assigned at random to each patient and is used to tract a patient


until that record is returned to its original resting place which could be a highly technical database ware

house for large data; One Place Only This could be where files will live from now on; Not lying around

the hospital’s or in the internet but deactivated in a large database storage area that all have access to its

information and all information is entered into the same record. When chosen, the access number is

relayed to the forwarding hospital or insurance provider, providing all workers have a clearance pass.

Having the instantly generated access number will allow that record to be downloaded and accessed with

names and hospitals accountability and noted throughout the process. This way the record will follow the

patient across the street to another physician’s office and have the most updated notes or the patient could

be on a cruise in the keys and get sick. The record is accessed from the one storage space and the entries

are made and we now have a truly exchangeable EHR system

Criteria for Success

1. The AHRQ site has an EHR matrix implementation evaluation kit (McGowan, Cusack, &

Poon, 2008).

2. Outcome goals are formulated by key stakeholders, developed in the project plan during

the planning stages early in the project. Deliverables in project charter are evaluation

goals and should match the end-users goals which should match EHR implementation

goals (McGowan, Cusack, & Poon).

3. Choosing the metrics for evaluation, should be chosen by the implementers. Evaluation based on

the areas of formative evaluation, clinical outcome measures, clinical process measures, provider

adoption and attitude measures should be in the matrix (McGowan et al., 2008).

4. McGowan et al. noted other issues addressed by the Evaluation Toolkit include a discussion

about barriers and facilitators to the evaluation, the differences between qualitative and

quantitative assessment methods and findings, sample size and power. Some of these topics can

be resolved through statistical analysis.


5. The study design questions of how will the data be obtained, who will do the work, and what

budget exists to support the formative evaluation are included.

6. Patient knowledge and attitude measures, workflow impact measures, and financial impact

measures provide critical information for informaticists.

7. Finally, comparing various types of HIT projects for their criteria for success in the Evaluation

Toolkit that match the facilities implementation.

I would disseminate the findings first at an open auditorium announcement with refreshments

and awards. The initial evaluation plan would be discussed from a hospital point of view and

then from a nursing point of view. From there a committee would be formed by a governance

group that would meet every month to look at unit problems with the new EHR and find

solutions through the informatics department.

Outlined Limitations of Project Scope

1. Internal Validity

a. Accurately measures what it intends to measure?

i. I believe we captured perceptions of PEOU and PU plus developed new

trends as listed in the literature review (5) new trends

b. Is the research design or instrument free from bias:

i. The collectors were screened and trained for proper facial expression

during questioning, voice tone, and responses to questions, A Hawthorne

effect would exist during observations of nurses performing their jobs.

I need more knowledge on bias through statistical analysis and I am sure

this has statistical limitations


c. Is the study free of systematic error

i. Limitation existed because of my inexperience at conducting an evaluation

methodology study; TAM procedure followed precisely.

d. Respondents do not respond truthfully to items on an instrument?

i. Yes I believe the staff did a great job of being truthful. This would need

stats analysis

e. inadequate literature review

i. Yes this is a limitation - We only choice 5 articles in the end for criteria

selection. I definitely feel like our sample size was limited because of

convenience sampling. Picking the medical surgical unit as the best

representation of the hospitals workflow limited the other units

f. Attrition

i. My target goal would be 12.5 – 18% (Polit, & Beck, 2012).

Summary

Technology changes by the minute and because of the many different types of EHR

systems no one silver bullet exists to service all EHR systems. Zandieh et al. (2012) noted that

because of the accelerated rates of implementing poorly designed EHR systems to meet

government incentives the backlash of updating EHR systems has already started in order to

meet meaningful use requirements. Informatics must develop better evaluation methods and tools

to capture the needed information through research. The evaluation plan, PICO question,

literature review, evaluation methodology, evaluation of tools and finally the evaluation plan is

the complete process of developing an evaluation research. Make sure your PICO question

generates a specific foreground question because the rest of the research is built around the


research question. The literature review is a quick way to become an expert on a subject.

Literature reviews will generate many different points of views on your subject and it is much

easier to have a literature review template that will organize the information and assist the

researcher in becoming the expert on the field. The idea with a literature review is quality and

quantity. The evaluation methodology is a unique project at this time for informatics because of

the lack of and sometimes poor documentation of just how a study was done. We as nursing

informatics must answer the call to develop better evaluation methodology plans. Also more

evaluation tools are needed by informatics nurses to assist in the evaluation process on such

subject as upgrading new EHR systems and how do nurses perceive the change this time

compared to last EHR implementation. Thank you for your time.

PowerPoint Presentation Below Please Click and View

References


Agency for Healthcare Research and Quality. (n.d.). Health IT survey compendium. Retrieved

fromhttp://healthit.ahrq.gov/portal/server.pt/community/health_it_tools_and_resources/

919/health_it_survey_compendium/27874

Berner, E. S. (2008). Ethical and legal issues in the use of health information technology to

improve patient safety. Journal on Hospitals' Ethical and Legal Issues, 20(3), 243-258.

doi:10.1007/s10730-008-9074-5

Coplan, S., & Masuda, D. (2011). Project management for healthcare information technology.

New York, NY: McGraw-Hill.

Friedman, C. P., & Wyatt, J. C. (2010). Evaluation methods in biomedical informatics (2nd ed.).

New York, NY: Springer Science+Business Media, Inc.

Hyun, S., Johnson, S. B., Stetson, P. D., & Bakken, S. (2009). Development and evaluation of

nursing user interface screens using multiple methods. Journal of Biomedical

Informatics, 42(6), 1004–1012. Retrieved from the Walden Library databases.

Laureate Education (Producer). (n.d.f). Reliability and validity. Retrieved from CDN database. (NURS 6431)

LeBeau, S. C. (2015). Evaluation plan part 1: Perspective of usability from the nursing

viewpoint. Unpublished manuscript, Walden University.

McGowan, J., Cusack, C., & Poon, E. (2008). Formative evaluation: a critical component in

EHR implementation. Journal of The American Medical Informatics Association, 15(3),

297-301.

Pfoh, E. R., Abramson, E., Zandieh, S., Edwards, A., & Kaushal, R. (2012). Satisfaction after the

transition between electronic health record systems at six ambulatory practices. Journal

http://mym.cdn.laureate-media.com/2dett4d/Walden/NURS/6431/01/mm/reliability/index.html

http://healthit.ahrq.gov/portal/server.pt/community/health_it_tools_and_resources/919/health_it_survey_compendium/27874

http://healthit.ahrq.gov/portal/server.pt/community/health_it_tools_and_resources/919/health_it_survey_compendium/27874


of Evaluation in Clinical Practice, 18(6), 1133-1139. doi:10.1111/j.1365-

2753.2011.01756.x

Polit, D. F., & Beck, C. T. (2012). Nursing research: Generating and assessing evidence for

nursing practice (Laureate Education, Inc., custom ed.). Philadelphia, PA: Lippincott

Williams & Wilkins.

Price, J. H., & Murnan, J. (2004). Research limitations and the necessity of reporting them.

American Journal of Health Education, 35(2), 66-67, DOI:

10.1080/19325037.2004.10603611

Rippen, H. E., Pan, E. C., Russell, C., Byrne, C. M., & Swift, E. K. (2013). Organizational

framework for health information technology. International Journal of Medical

Informatics, 82(4), e1-e13. doi:10.1016/j.ijmedinf.2012.01.012

Zandieh, S. O., Abramson, E. L., Pfoh, E. R., Yoon-Flannery, K., Edwards, A., & Kaushal, R.

(2012). Transitioning between ambulatory EHRs: A study of practitioners’ perspectives.

Journal of the American Medical Informatics Association, 19(3), 401–406.

doi:10.1136/amiajnl-2011-000333

Documents

Optimal Resume at WALDEN UNIVERSITY CAREER ... · Web viewDevelopers of four major health technology information models commonly used were created by innovators Rogers (diffusion