Evaluation Report Nelson

8/11/2019 Evaluation Report Nelson

1/10

NIGHTINGALE

COLLEGE INTERACTIVE

MODULE EVALUATION

REPORT

This is an official evaluation submitted by Aaron Nelson for the stakeholders

of Nightingale College on May 9, 2014.

An Evalu at io n

Report for Quality

and Participation

within Nursing

Fundamentals 200.

By Aaron F Nelson


2/10

ContentsSummary .......................................................................................................................................................

Description of the program evaluated .......................................................................................................... 3

Program Objectives ................................................................................................................................... 4

Program Components ............................................................................................................................... 4

Evaluation Method ........................................................................................................................................ 4

Participants ............................................................................................................................................... 4

Nightingale Evaluation Team .................................................................................................................... 5

Procedures ................................................................................................................................................ 5

Data Sources ............................................................................................................................................. 6

Results ........................................................................................................................................................... 7

Discussion...................................................................................................................................................... 9

Project Cost ................................................................................................................................................. 10

Appendices .................................................................................................................................................. 10


3/10

Summary

Nightingale College is transferring into a competency based curriculum. As part of thistransformation the Curriculum management team have implemented interactive modules to

replace its old way of delivering lectures through taped video casting through Blackboard-Collaborate. The school started the interactive program a year ago. After much research, theteam decided on Storyline by Articulate. The developers picked Storyline because of the ease ofbuilding a course, the quality of the interactive modules, and the ease of implementation into theLMS. Each module consists of a weekly unit that includes objectives and outcomes for eachspecified chapters of the nursing curriculum. The objectives are aligned with the NCLEX boardexams, which nurses must pass to become registered nurses.

The interactive modules have been implemented into Nursing Fundamentals for a year. Theywere created by Karen Burton, the Colleges curriculum manager and the e-development teamwhich is headed by Blake Halladay, the Colleges e-Developer. The modules include crosswordpuzzles, games, quizzes, character animations, videos, and other media that provided a uniqueway of introducing basic nursing fundamentals.

The purpose of the evaluation report is to give the curriculum management team andstakeholders of the College information on how well we are doing in the quality of our product,and participation among the students in the interactive modules. The major steps of theevaluation included meetings with course developers, students, and administrators. Evaluationwas gleaned from data that was provided from surveys and information reports.

The evaluation found that there are many factors that need to be improved in order for theobjectives to be met. The College will take the information it has received from the evaluationreport and prepare a plan of action. The results of the evaluation will be detailed at the

conclusion of the evaluation report.

Description of the program evaluated

The intended users of the interactive modules include every course and student at Nightingale

College. Currently, the student body is around 100 students. The College has twenty-seven

students enrolled in the Fundamentals Nursing course, where we will conduct the evaluation.

On average, students participate in 15-18 interactive modules per course. They will be graded on

participation and evaluation, which will be included at the end of every interactive module.

Those students who excel at the interactive modules (an overall grade of 80%) will be given theoption of skipping the final examination.

The reason why the College decided to pursue interactive modules in place of webcast lectures is

because the College wanted more participation and involvement with course content. The

instructors are given data concerning how much time the students spend in the course, how long

it takes for the student to do the interactive module, and test scores for the unit reviews of each

interactive module. These reports are available for instructors through Moodle.


4/10

Program Objectives

There are two program objectives:

The first objective is to evaluate the quality of fifteen interactive modules in Fundamental

Nursing. The design team is interested to see if the modules are easy to navigate, have

good designs, graphics, and videos, and meet the objectives outlined for each unitchapter.

The second objective is to evaluate participation in the modules. Since participation in

the interactive modules counts toward a large portion of the students grade, the College

wants to know the amount of time students are participating in the interactive modules,

and how they are influencing overall grades.

Program Components

The intended purpose of Storyline by Articulate is to provide students with polished modules that

students are able to use. The modules may include quizzes, media videos, screen recordings, anddrag-and-drop interaction. The developers are also able to easily build complex interactioncharacters using a host of templates, characters, and animations. For the Nursing Fundamentalscourse, fifteen modules have been built. Before the program, instructions will be given to thebeginning nursing students on how to navigate the modules.

Evaluation Method

Our goal is to have an independent and neutral evaluation process. We will be using the goal-

based model for evaluation. The evaluation method includes three partsplanning, evaluation,

and presentation.

Here is a description of the evaluation method:

1. Aaron Nelson, the head of the evaluation team met with the Director of Curricula to

decide on participants of the team. Objectives were decided on, surveys were written,

and dates were planned.

2. The information will be analyzed, surveys will be handed out, and the evaluation report

will be written.

3. We will deliver our findings to Mikhail Shneyder, the CEO of the College.

ParticipantsStudent body: The Fundamentals Nursing 200 course is made up of first year students in the

AND (Associate Degree Nursing) program. Before entering into Nightingale College, the

students need to complete fifteen general education credits in order to participate in the program.

The current course includes twenty-four participants. Most of these students have previous

experience in the nursing field; however, the College does allow students who are inexperienced.


5/10

Its also important to point out that many of students have little to no experience using

technology. Extensive training and support is provided for help with technology issues.

Nightingale Evaluation Team

The following people will be the main evaluators and respondents for the official final evaluation

report:

Aaron Nelson (Director of Library Services): Aaron Nelson has over three years of library

experience. He received his BA from Weber State University and an MLS from Emporia State

University. Mr. Nelson has been a part of Nightingale University for three years. Currently, Mr.

Nelson is pursuing MET (Master in Educational Technology from Boise State University. He

has over six months experience in evaluation experience.

Karen Burton (The Director, Nursing Curricula and Academics):, Dr. Karen Burton, PhD,

RN, CNE is responsible for the design, development, and consistent implementation of nursing

curriculum and academic programs across the current and future programs at the College. She iscurrently collaborating with the faculty to transition the ADN Program curriculum from

traditional to concept-based. Dr. Burton reports to the President and CEO, and manages all

library and instructional design efforts at the College. She currently holds her Certified Nurse

Educator (CNE) certification and received both her masters and doctoral degrees specifically in

nursing education. She has extensive experience with curriculum design and teaching

methodologies. Dr. Burton holds a doctorate in philosophy (PhD) with a major in nursing

education from the University of Northern Colorado; Greeley, Colorado.

Sue Jero (ADN Program Manager): Ms. Jero has full responsibility for the day-to-day

operations of the ADN Program. Further, Ms. Jero is a member of American Organization ofNurse Executives (AONE), National League of Nursing (NLN), the Academic Leadership

Committee (UONL), and is a member of Ogden Chamber of Commerce. Ms. Jero has been with

the College since inception.

Blake Halladay: Mr. Halladay holds an associate degree with a major in nursing and has an

extensive background of professional experience in new partner acquisition and key partner

relationship management roles. Mr. Halladay comes to the College from one of the largest and

most respected corporate education programs in the country, the Learning and Development

group at Fidelity Investments, a trillion dollar financial corporation. During his tenure at the

Learning and Development group, Mr. Halladay operated in various capacities that includedsmall group education in a physical classroom, presentations to large groups of employees,

project management, relationship management, instructional design, and course development.

Procedures

As this was the first time implementing Storyline by Articulate, there were many obstacles that

needed to be overcome. The e-Developer (Blake Halladay) had to be trained on the tool. The


6/10

students had to be trained on how to navigate through the modules. The faculty had to be trained

on how to design the content that the e-Developer put into the interactive module. Naturally,

there was a learning process that was new to the College.

The evaluation portion of Storyline began in February and concluded in May. This gave the

Nightingale Evaluation team plenty of time to have enough modules developed. It also gave thestudents enough time to become familiar with the content. Below is a timeline of the evaluation

procedures that the team followed in order to develop the evaluation report.

Procedure DatePlanning: Aaron Nelson, the head of the Evaluation Team met withKaren Burton, the director over the Curriculum/Design Team.During this meeting they will decided what other members toinclude on the Evaluation Team. They will also discuss roles, data,

objectives, evaluation report, and other pertinent data required forthe report.

February 3, 2014

The Evaluation Team met to discuss objectives for the interactivemodules. The Evaluation Team decided on two objectives that dealwith quality and participation. These two objectives were decidedupon after much discussion.

February 24, 2014

Evaluation: The Evaluation Team and e-Development team metwith class ambassadors to discuss the new interactive modules. Theambassadors went through the first 5 unit modules and discussed thequality, videos, interface, games, and other factors that they liked ordisliked about the interactive modules.

March 10, 2014

The Evaluation Team meets to create survey questions. Its decidedthat the survey will be a short five question survey that will bedesigned to answer the Evaluation Teams objectives. Its alsodecided that the survey will be handed out with the other coursesurveys during the last two weeks of the course semester startingApril 14.

March 27, 2014

The Course surveys are created inwww.surveymonkey.comandlinks are sent to the students in the Nursing 200 course.

April 14, 2014

The evaluation report is written by Aaron Nelson. Once the report isexamined for his EDTECH 505 course at Boise State University, itwill be rewritten with any advice from the instructor.

April 21-May 9 , 2014

Presentation: The final evaluation will be delivered to MikhailShneyder, Campus CEO, along with any members of the designteam.

June 16t , 2014

Data Sources

The College employed a few data tools to gather information on the factors that are stated in its

objectives. Below is a table that will show each factor, including the tool that we used to
http://www.surveymonkey.com/http://www.surveymonkey.com/http://www.surveymonkey.com/http://www.surveymonkey.com/


7/10

measure each factor. An appendix of all of the information gathered can be found at the

conclusion of the report.

Factor Information Tool

Performance (Test Reviews) Storyline, SCORM, Moodle. The evaluation team used these

three tools to measure performance. At the end of every unit testscores are evaluated using a template from Storyline. Using atechnology tool called SCORM; the scores are then transferredover to Moodle, where the team is able to evaluate test reviewsand scores in Excel spreadsheets.

Participation Moodle. The LMS tracks how often students use the interactivemodule. If a sign of Not Attempted or Incomplete shows up,

it clearly shows that the student didnt participate, or partially

participated in the unit. Also, it shows when the student startedand ended each unit. If the student started the module a daybefore the module was due, it clearly shows that the student

wasnt actively engaged in the interactive module.Quality Survey Monkey. For the quality factor, the students are evaluatedusing surveys created on www.surveymonkey.com. The surveysquery the students on graphics, videos, interactive games, etc.The evaluation team decided on using questions in the LikertScale form.

ResultsThe survey gave results for both objectives. The survey was designed according to the Likert Scale. All

27 students participated in the survey. Here is a summary of these findings:

Survey Question Findings

1. Overall, I enjoyed the interactivemodules for my Nursing 200course.

The survey scored a 7.67. Overall, students enjoyed themodules compared to lectures.

2. It was easy for me to log intoMoodle so that I could do theinteractive module.

The survey scored 7.56. There were very few problemsthat were pointed out by students. Most of them had aneasy time logging into Moodle.

3.

The graphics, pictures, and mediacontent helped me understand thecontent for Nursing 200Fundamentals.

The survey scored 7.67. Overall most students liked thegraphics and quality of the interactive modules.

4.

The games (crossword puzzles,matching, etc) were very helpful andhelped me retain the nursing content:

The survey scored 7.73. Most students liked theinteractive content created by the development team.

5.

I prefer the traditional way oflecturing (BlackBoard Collaborate)over the new interactive modules.

The survey scored 3.33. Most students do not prefertraditional online lectures to interactive modules.


8/10

The Moodle Reports measured participation and gave results for our second objective:

Unit Dates

Taken

Incomplete/Did Not

Take

Taken Received a Failing

Score on Module

Received a Passing

Score on Module

Unit 1 01/07-01/12

8 19 17 Students 10 Students

Unit 2 01/13-

01/19


Unit 3 01/20-

01/26


Unit 4 01/27-

02/02


Unit 6.3 02/10-

02/19


Unit 7 02/20-

02/26


Unit 8 02/24-

03/03


Unit 8.1 03/04-

03/11


Unit 9 03/10-

03/16

- - - -

Unit 10 03/10-

03/23


Unit 11 03/17-

03/23


Unit 12 03/24-03/30


Unit 13 03/31-

04/06


Unit 14 04/07-

04/13


Unit 15 04/14-

04/20



9/10

Discussion

The purpose of the reports and survey were to gather information on how well the College is

meeting its objectives for quality and participation. According to the survey, the students are

satisfied with navigation, the graphics, videos, and accessibility. They are finding little to no

difficulties logging into the LMS, viewing the interactive modules, and doing the interactive

module.

The Moodle Reports give valuable information on participation. They show when students

access the interactive modules, grades, and participation. They were easy to access and analyze.

The sources reveal that there were also many problems. It shows that many students didnt

realize the importance the interactive modules were to their overall grades. As this was a pilot

program, this was taken into account for final grades. Its also important to note that the reports

were confusing to read since the e-developer had added many inactive students and faculty

members to the modules.

The results for the Moodle Reports clearly give an indicator of low participation. There weremany students who consistently had incomplete or did not take the modules. There were also

many students who consistently failed the modules. On the other hand, there were also other

students who consistently took the modules and received passing grades. Since the modules are

based on readings within the textbooks, there is a clear indication that some students never read

the textbooks. Not surprisingly, those who fully participated in the interactive modules, also

show a clear indication of passing our end of level HESI exams (our Nursing exams). The

reports also indicate that there were problems rolling out the program. For instance, Units 8.1

and 9, show that there was close to no participation in the interactive modules.

The College must make the importance of doing the interactive modules clear to the studentsbeginning with the first day of class. After speaking with the instructors and designers, it was

revealed that grades and indicators for doing the interactive modules werent shown to the

students on Canvas (the other LMS system that students use to look at grades.) The reason they

put the modules on Moodle is because Canvas isnt SCORM compatible. Its highly suggested

to the College that they do away with Canvas. Having two different LMSs will greatly confuse

students and instructors.

The data also suggest that there needs more time devoted to preparing the interactive modules

into Moodle. The weeks where there were no modules prepared had an effect on the overall

participation. A suggestion would be to have a meeting to reevaluate how the developersimplement and create the interactive modules. Students also need to be taught how the

interactive modules work and who they need to contact if there is a problem with the modules.

Overall, the data showed that the quality of the modules is good. The best indicator for this will

be in the final Fundamentals HESI examinations for the College. (Test score data will not be

available until the middle of May.) If HESI examination scores are low, the College must re-


10/10

evaluate interactive modules in place of traditional lectures. Based on student surveys, they

highly prefer the modules over the traditional lectures on BlackBoard Collaborate. With a few

corrections in the way that the College implements the program at the beginning of the semester,

the hope is that participation levels will increase.

Project Cost

The cost for the evaluation report is based on a daily rate of $500.00 per a daily rate. Belowshows the dates that Aaron Nelson worked on the evaluation report:

February 3, 2014

February 24, 2014

March 10, 2014

March 27, 2014

April 14, 2014April 21-May 9 , 2014June 16 , 2014

Full Cost = $10,500 for services rendered. (21 Days X $500= $10,500)

Expenses=$4,000

Total Cost = $14,500

The expenses include travel, copies of report, consultations, etc. For a full version of expense

report, please view receipts, reports, etc. in Appendices.

Appendices

Appendix 1=Survey Summary for the 27 Students.

Appendix 2-14=Moodle Reports.

Appendix 3=Expense Reports.

Documents

Evaluation Report Nelson