EDCI 528 HPT Solutions & Evaluation Plan, Reception Desk/Call Answering Inefficiency

Running head: EVALUATION PLAN EDCI 577

1

Instructional Product Evaluation Plan

Level 1 Introduction to Food Safety and Hygiene Online Course

Danielle L. Moffat

Purdue University

EVALUATION PLAN EDCI 577

2

Executive Summary

Purpose of the Evaluation

This evaluation plan has been devised for the purpose of gauging the effectiveness of the

recently developed Level 1 Introduction to Food Safety and Hygiene online course that is intended

to be fully implemented in due course as part of a new staff online induction program hosted in

BB’s LMS in mid-2016. Given that it is a mandatory legal requirement for all staff to have

successfully passed the training course it is important that the course design and content achieve

the desired learning outcomes. Similarly, significant failures by staff to display the target food

safety and hygiene behaviors have the potential to result in serious legal and financial penalties

being incurred by BB. A trial and evaluation of the training course will provide for a detailed

assessment of the above factors in order that recommendations to be made for course

improvements can be made and actioned prior to its full implementation.

Evaluation Objectives

The primary objective of this evaluation is to measure the effectiveness of the training

course in terms of achieving the desired levels of participant learning and target food safety and

hygiene related behaviors in the workplace. The evaluation also aims to quantify the positive

business impacts relating to implementation of the training course across BB’s 4 venues.

Evaluation Plan Summary

The evaluation plan set out in this document utilizes Kirkpatrick’s (2006) four-levels of

evaluation methodology. Participant reactions, learning gains and changes in behavior towards the

desired states shall be measured with reference to a group of approximately 30 participants

comprised of anticipated new hires over the next 3 months. Business results will be assessed

following full implementation of in mid-2016. Data capture, analysis and reporting procedures in

respect of each of Kirkpatrick’s (2006) four levels of evaluation have been set out in detail within

the main body of the accompanying Evaluation the online induction program which shall be

administered to all staff across BB’s organization Plan Report.

Evaluation Results and Key Recommendations

Due to the fact that the course trial has not yet taken place, evaluation data is not yet

available. However, the board of directors will be provided with a Level 1 & 2 Evaluation Report

following the course trial which will provide a detailed analysis of course participant reactions and

learning gains as well as set out full recommendations for course improvements, where necessary,

to be actioned prior to full implementation of the online staff induction program. Approximately

6 months thereafter a further Level 3 Evaluation Report will be provided which will detail changes

in behavior achieved within the participant group as a result of attendance on the course.

Recommendations have been made within this Evaluation Plan Report in relation to steps to be

taken in advanced of the course trial order to create a workplace environment that promotes display

of the target behaviors. A final Level 4 Evaluation Report will be provided approximately 9 months

following full implementation of the online induction program due to the fact that the business


3

impacts of the course are not capable of being assessed on the basis of the small group of course

trial participants. The final Level 4 Evaluation Report will analyse existing documentation relating

to food safety and hygiene, including accident report forms, from across BB’s 4 venues in order

to gather evidence of positive business impacts related to full implementation of the course.

The board of directors can expect to review various types of evaluation data within the

various Reports referred to above including: learner reaction survey results, data relating to the

achievement of learning outcomes including knowledge, skills and hazard perception test scores,

course participant behavioral interview transcripts and pre – post course implementation safety

record comparisons.


4

Evaluation Plan Report:

Level 1 Introduction to Food Safety and Hygiene Online Course

Instructional Product Overview

Brightwing Breweries (“BB”) is a small but rapidly expanding group of gastro-inns, at this

time comprising 4 venues located across the Southwest of England. In light of the group’s plans

to double its number of venues within the next 18 months, a board-level decision was made early

this year to implement a formal, group-wide staff professional development program in order to

promote uniformity of training and standards across the organization as well as to ensure full

compliance with relevant regulations and legislation. The new professional development program

forms one of a number of digitization initiatives currently being rolled out across the group

involving utilization of cloud-based technologies with the aim of improving and streamlining BB’s

business operations.

BB have engaged the services of DM Consult Ltd (“DMC”), a specialist e-learning

solutions company, to design and implement a bespoke LMS to serve as the hosting platform for

the online aspects of the professional development program including e-learning courses,

personalized professional development plans and progress tracking, resources and knowledge

sharing functions for supervisors and management including online discussion forums.

In initial consultations, BB identified staff induction training – which is presently provided

on an informal, face-to-face basis at each venue as part of a one day induction session – as a critical

concern. A recent in-house review of the current induction training has revealed that it fails to fully

comply with current European and UK Health & Safety regulations and legislation, a situation

which has the potential to result in serious legal penalties being incurred by BB. Accordingly, it

was agreed that DMC would prioritize the development of a new staff induction training program,

comprising a number of standalone e-learning courses covering a variety health and safety

induction topics, to form part of the LMS implementation pilot.

This report sets out the proposed evaluation plan in relation to one of the health and safety

e-learning induction courses, Level 1 Introduction to Food Safety and Hygiene, which is intended

to be trialed during the upcoming 3-month LMS implementation pilot. It is intended that, in due

course, all new staff who have not undergone equivalent training within the last 5 years will be

compelled to complete the Level 1 Introduction to Food Safety and Hygiene regardless of

experience level, in accordance with current legislative requirements. A summary of the course’s

structure and instructional content is outlined in Table A1 at Appendix A.

Purpose of the Evaluation

Given that all new relevant staff will be required to undertake the Level 1 Introduction to

Food Safety and Hygiene course, the purpose of this evaluation is to determine the overall

effectiveness of the course and to identify the ways in which it might be improved before full

implementation of the online staff induction training program via the LMS takes place in mid-

2016.


5

Evaluation Objectives

The following evaluation objectives form the basis for the evaluation data capture

procedure by setting out the specific factors that must be addressed during the course of the

evaluation in order to ascertain the effectiveness of the course and make recommendations for any

improvements that might be made thereto, if appropriate.

DMC recommends undertaking the course evaluation by means of Kirkpatrick’s (2006) 4

Levels of Evaluation model, which stresses evaluation of the chronically staged levels of (1)

reaction, (2) learning, (3) behavior, and (4) results. Accordingly, the following table sets out the

evaluation objectives in respect of each level together with an outline of the evaluation instruments

and resources that shall be utilized to measure for each objective.

Table 1

Evaluation Objectives, Questions and Criteria/Measures/Resources

Level Objectives Criteria/Measures/Resources

1 To assess participant attitudes towards the

course.

Course reaction survey

2 To quantify learning gains experienced by

participants as a result of undertaking the

course.

Pre-post course comparison of

knowledge and skills test results.

Assessment of achievement of required

pass mark in posttests.

3 To assess the level to which the course

promotes the display of desired food safety

and hygiene-related behaviors in program

participants.

Participant interviews at 3 and 6 months

post-course.

4 To calculate and/or describe the monetary

and/or intangible business impacts of the

course.

Accident report forms/HSE reports.

Complaints records.

HSE food safety & hygiene inspection

records

Audience and Context

Key stakeholders. Fitzpatrick, Sanders & Worthen define stakeholders as, “the individuals

and groups who have a direct interest in and may be affected by the program being evaluated or

the evaluation’s results” (2004, p. 9). Accordingly, the following groups comprise the key

stakeholders in respect of the Level 1 Introduction to Food Safety and Hygiene course:

Board of Directors: the board of directors is comprised of 5 members who are joint equity

partners in the business. They are the sponsors of the professional development program. As such,


6

they will review the results of the evaluation and have authority for making key decisions regarding

the course’s future.

HR Departments, Managers and Supervisors: this group comprises those persons within

the organization who have direct responsibility for the course (Fitzpatrick et al., 2004). The HR

Departments will be involved in the administration and co-ordination of the course as well as

assisting the evaluators with some elements of the data capture and collation process. Supervisors

will be involved in administering the pre- and post-course skills tests, and both Managers and

Supervisors will be instrumental in providing feedback regarding the effectiveness of the course,

in terms of transfer of the target knowledge and skills in to workplace practice.

Participants: this group comprises the intended beneficiaries of the course (Fitzpatrick et

al., 2004). The participants, being the target learners, comprise all new staff joining BB including

entry-level to experienced floor, bar and kitchen staff, supervisors and managers who are selected

for participation on the course for the purposes of this evaluation.

The target learner group demographics are presumed to fall in line with the most recently

published UK hospitality and tourism labor market statistics: 57% of the workforce is female; 44%

are under the age of 30 (with 31% of the same being aged between 16-24) with the average age of

restaurant managers and bar managers being 39 and 44 years respectively; 85% of the workforce

describes their ethnicity as white and 15% as being of black and minority ethnic background

(People 1st, 2013). The majority of the target learner group are likely to be relatively young.

Participants below supervisory level will tend to be under 30 years of age, with the majority of that

group falling within the age range of 16 – 26. Of that majority group, those between the ages of

16 – 19 will not be advanced learners, having been educated to high school level only.

Context. Due to the fact that the e-learning course is hosted in BB’s cloud-based LMS it

is accessible from any computer via each learners’ individualized professional development

profile. Learners will undertake the course on dedicated training computers provided in the

administrative offices of each of BB’s venues and they will be provided with a pencil and paper

before taking the course for note taking. New staff will spend their first two days at work in

induction, with the first of those days being devoted to undertaking the online induction courses.

Prior to taking the online courses, the learners will receive informal face-to-face training

with their induction training supervisor on how to log in to and use the LMS as well as undertaking

a short training module on how to navigate and use the e-learning courses and interactive activity

types. Therefore prior knowledge in that regard is presumed.

The Evaluation Process

The Level 1 Introduction to Food Safety and Hygiene online course will be trialed during

the upcoming 3-month LMS implementation pilot. BB have advised that, due to the nature of the

hospitality industry, there is a relatively high level of staff turnover across the group. It is projected

that somewhere in the region of 30 new hires will be made during the pilot period, which will

make up the target learner group for the purposes of the evaluation of Levels 1 - 3.


7

It has been agreed that a Level 1 and 2 Evaluation Report will be produced following the

initial trial of the online course, which will provide an early assessment of the effectiveness of the

course in attaining the required level of learning and allow for recommendations to be made and

actioned in respect of course improvements and refinement prior to full implementation of the

online induction program in mid-2016. A longer-term view will be taken in respect of evaluating

for Levels 3 and 4 as sufficient time must be allowed for changes in behavior to take place as a

result of acquiring the target knowledge and, subsequently, for the overall organizational impact

of such changes to become apparent (Kirkpatrick, 2006). The Evaluation Sampling Plan at

Appendix B and Gantt chart at Appendix C together set out the recommended project plan and

timelines. Evaluation instruments, data capture and reporting procedures in respect of each level

are explained in detail in the following sections of this Report.

Level 1: Reaction

Evaluating participant reactions will provide information that will enable an assessment of

the level of the course participants’ motivation to learn. It is recommended that participant

reactions are obtained by means of an electronic post-course reaction survey (Appendix D)

forming part of the online course, to be administered at the culmination of the learning and

electronic test elements of the same in accordance with the data capture procedure for Levels 1

and 2 set out at Table 2 below.

Motivation has a direct and significant impact on learning – low levels of motivation will

inhibit learning while highly motivated learners will be involved in the initiation of, and

persistence in, learning activities. Accordingly, Level 1 evaluation results provide insight in to and

clarification of Level 2 results (Kirkpatrick, 2006). Where Level 1 data reveals a significant

negative participant reaction to the course then it can be inferred that any corresponding low levels

of learning highlighted in the Level 2 data are contributed to by factors affecting the participants’

motivation to learn. In such event, participants’ written comments and suggestions gathered from

the open comment sections of the reaction survey will serve to pinpoint specific issues affecting

the participants’ learning motivation levels, such as for example not being appropriately

challenged by the course content, which can be used to make improvements and refinements to

the course before full implementation. Recommendations for capture of further clarification data

may be recommended within the Level 1 and 2 Evaluation Report in the event that a significant

negative participant reaction is found.

The raw data in relation to the reaction survey will be stored in the LMS which has been

designed to provide reporting functions enabling specific data to be extracted and presented in

graphical or numerical table format. Given that the course trial has not yet been undertaken, actual

Level 1 results are not presently available. In accordance with Kirkpatrick’s (2006)

recommendations, acceptable standards of success have been pre-agreed. Given that passing the

Level 1 Introduction to Food Safety and Hygiene is a mandatory legal requirement, and the pass

grade has been set at 80%, it has been agreed that a similarly high standard will be applied to the

acceptable standards in relation to the Level 1 results. A minimum mean response per survey

question of 4/5 will be considered the acceptable standard. Appendix E sets out the form in which

the results shall be reported to the board of directors within the Final Evaluation Report, which


8

shall be accompanied by a written Level 1 analysis summary and DMC’s recommendations for

further action, if appropriate.

Level 2: Learning

BB have requested that all new hires undertake the trial course during the pilot period due

to the fact that (1) the food safety and hygiene training is a mandatory legal requirement, (2) the

current training regime in that regard is deficient, and (3) possessing the target knowledge and

skills is a pre-requisite to successful carrying out of workplace duties. It is therefore recommended,

for practical purposes, that the evaluation employs a one group pre-assessment – post assessment

design, whereby the entire target learner group is assessed prior to and following the instruction to

determine the learning gains achieved as a direct result of undertaking the course (Dow Chemical

Company, 1992).

Table 2, below, sets out the data capture procedure in relation to the evaluation of Level 1

and 2. The Level 2 learning instruments referred to within Table 2, namely the pre- and post-skills,

transfer and knowledge tests, are set out in detail at Appendix F, G and H respectively.

It should be noted that the pre- and post-test multiple choice test comprises the same 20

questions in order that the result can be readily compared. No feedback will be provided to the

target learners’ responses in the pre-test and questions will be randomized in both tests so as to

avoid skewed results due to participants simply learning the correct responses to the test questions

rather than acquiring the target knowledge.

Table 2

Data Capture Procedure for Levels 1 and 2

Evaluation Instruments Timeline Administration

Skills pre-test (handwashing

demonstration) – Level 2.

Measuring prior skill level.

Administered immediately prior

to undertaking online course to

target learners.

Administered by induction

training supervisor. Grade

assigned using handwashing

rubric (Appendix F). Raw

data entered in skills pre-test

spread sheet.

Knowledge pre-test

(electronic 10 question

multiple choice test) – Level

2. Measuring prior

knowledge. Appendix H,

Table H1.

Online test administered

immediately prior to commencing

the course.

Raw quiz scores stored in

LMS.

Knowledge pre-test

(identification of hazards, 8

possible hazards) – Level 2.

Measuring prior knowledge.

Appendix G.

Module 1 hazard awareness

activity involving clicking on

potential food safety hazards

within an image.

Raw scores stored in LMS.

Hazard perception post-test

(identification of hazards, 8

Hazard awareness in Module 1

administered in final Module 8.



9

possible hazards) – Level 2.

Measuring transfer.

Appendix G.

Knowledge post-test

(electronic 10 question

multiple choice test) – Level

2. Measuring post-course

knowledge. Appendix H,

Table H1.

Administered immediately at the

end of the course.


Post-course reaction survey

– Level 1. Measuring

attitudes to course and

overall course satisfaction.

Appendix D.

Administered immediately

following the knowledge post-test

as a mandatory requirement for

course completion to ensure

100% response. Participants

provided the opportunity to

provide written feedback and

comments in text boxes.

Raw data stored in LMS.

Skills post-test

(demonstration

handwashing) – Level 2.

Measuring post-course skill

levels.

Administered on the second

induction day after participant has

had the opportunity to practice

handwashing techniques at home

using printable step-by-step guide

provided in Module 3.

Administered by training

supervisor. Grade assigned

using handwashing rubric

(Appendix F). Raw data

entered in skills post-test

spread sheet.

In accordance with Kirkpatrick’s (2006) recommendations, the level of learning that has taken

place on the course will be analyzed as follows:

A comparison of mean pre – post course test scores in the multiple choice and food hazard

perception (knowledge) and handwashing (skills) tests. No or low gains between the scores

achieved in the pre- and post-tests will indicate that the training has been ineffective while

substantial gains between pre- and post-tests will signify course efficiency.

A comparison of mean pre – post test scores for individual questions in the multiple choice

knowledge test will provide specific information in relation to learning in the different

modules that can be used highlight areas where learning has been less effective, signaling

a need for improvement.

As with Level 1 results, the raw data in relation to the electronic pre- and post-course

knowledge and hazard perception (transfer) tests will be stored in the LMS and extracted for

analysis at the end of the trial period. The raw data in relation to the pre- and post-course skills test

will be manually input in to a designated spreadsheet at each venue by the training coordinators,

which will then be forwarded to DMC by the HR Departments for consolidation and analysis at

the end of the trial period. Acceptable standards in respect of Level 2 results have been pre-agreed


10

at a minimum result of 80% for all participants in all tests, in line with the pass mark they are

required to achieve in order to successfully complete the mandatory course.

The Level 2 results will be reported to the board of directors in the Level 1 and 2 Evaluation

Report in the form of pre- and post-course test comparison tables and graph, as set out at Appendix

I, together with an accompanying analysis summary detailing the evaluation findings (namely, the

extent of the learning gains in each area and a conclusion as to course efficacy in light of the same).

In relation to the comparison of average scores for individual questions in the pre- and post-course

multiple choice knowledge tests, the analysis summary will, if appropriate, highlight areas for

improvement and/or refinement, making specific recommendations in that regard to be actioned

prior to full implementation of the online induction course if appropriate.

Level 3: Behavior

Evaluating changes in participant food safety and hygiene behavior as a result of attending

the course will provide information that will enable an assessment of the level of transfer of the

target skills and knowledge in to the workplace, namely the level to which the participants go on

to display the desired food safety and hygiene behaviors they learned about on the course during

the course of their every day work (Kirkpatrick, 2006). Given the fact that BB stand to incur

potentially serious legal or financial penalties in the event that its members of staff fail to

adequately display the desired food safety and hygiene behaviors, DMC recommends taking the

following steps prior to the trial course in order to create a work environment that promotes transfer

of the target skills and knowledge:

DMC shall meet with supervisors and managers shortly before the commencement of the

course trial to explain its purpose and importance, present an abridged version of the online

course’s content and set clear expectations in terms of their role in supporting and encouraging

display of the target behaviors.

DMC shall meet with the Head of HR in order to devise and appropriate reward system in

respect of the display of desired health and safety behaviors generally. This forms part of the

wider professional development program development remit and, as yet, remains to be fully

laid out. Therefore, this point is not dealt with in detail here except to draw attention to the

principle that incentivizing staff through a rewards system will encourage display of the desired

behaviors within the workplace (Kirkpatrick, 2006).

BB advises that the majority of the participants will be hired on the full-time basis in which

they will be expected work at least the statutory minimum of 35 hours per week. Accordingly, it

is assumed that the participants will have the opportunity to display the majority of the desired

behaviors within a relatively short period of time. Given that the majority of the new hires will

comprise entry-level candidates it is considered inappropriate in these circumstances, from a time

and costs point of view, to evaluate for pre-existing target behaviors prior to administration of the

trial course. It is recommended that participants are evaluated for changes in behavior at 3 and 6

months following attendance on the course by means of a patterned interview, the form of which

is set out at Appendix J. It is felt that there will be sufficient time for the training supervisors to


11

conduct the interviews with all of the course participants across BB’s 4 venues, which will provide

a full view of the changes in behavior. The rationale for evaluating for Level 3 by means of the

more time-consuming method of patterned interviews, as opposed to using surveys, is that the

results obtained will provide greater detail regarding the behavioral changes as well as provide the

interviewing training supervisors with the opportunity to discuss relevant issues arising as well as

to identify areas in which additional ongoing support in relation to display of the target behaviors

may be provided, either on an individual or organizational basis.

The completed interview forms will be submitted via email to DMC by the respective

training supervisors in order that the Level 3 results can be tabulated. Quantitative data in relation

to the degree to which behavioral changes have occurred as a result of course attendance will be

set out for review by the board of directors within the Level 1 to 3 Evaluation Report in the tabular

and graphical forms set out at Appendix K. Specifically, the degree of behavior change will be

analyzed as follows:

An assessment of increases or decreases in the participants’ ability to identify the desired

food safety and hygiene behaviors encouraged by the course at 3 and 6 months following

the course. Any increase in that regard between the first and the second interview will be

assumed to be due an increased level of engagement in display of those target behaviors

leading to an enhanced awareness of the same.

A comparison of participants’ responses in relation to questions aimed at gathering data

relating to their eagerness to display the target behaviors at 3 and 6 months. Participants

will be provided with the opportunity to provide open comment on the reasoning behind

their choice of the response options. Such comments will be useful in assessing the

underlying reasons for any negative trends found in the results. Relevant participant

comments in that regard will be set out within the Level 1 to 3 Evaluation Report together

with DMC’s recommendations for further action, if necessary.

An analysis of the participants’ perceptions relating to their ability to display the desired

behaviors, and a comparison of such responses at 3 months and 6 months. Where

participants have been unable to engage in the desired behavior they will be given the

opportunity to identify, out of a list of response options, the reason for this inability. Such

information shall be compared between 3 months and 6 months in visual graphical format,

and shall provide important information that can be used to identify and, where appropriate,

eliminate environmental and other factors that may be inhibit display of the desired

behaviors. Participants will be provided with the opportunity to provide further comments

where they have been unable to display the target behaviors for reasons other than the

available response options. Such comments will be reviewed by DMC in detail and may

result in recommendations in the Level 1 to 3 Evaluation Report for further investigations

or capture of additional clarifying data where one or more significant factors affecting

performance not previously considered are identified within the Level 3 results.


12

Level 4: Overall Results/Business Impact

Given the relatively small size of the target learner group in contrast to BB’s employee

base as a whole, it has been agreed that it will be inappropriate to measure for Level 4 results in

relation to the course trial. Full implementation of the online induction program is due to take place

in mid-2016, shortly following which the majority of the existing staff are intended to undertake

the new online induction program. Accordingly, it has been agreed that Level 4 results will be

evaluated 1 year following full implementation of the program, essentially allowing a period of 9

months for changes in behavior to take place and any subsequent Level 4 results to become self-

evident.

Level 4 results are notoriously difficult to measure for and it should be noted that findings

in this regard will generally only provide evidence that implementation of the training course has

resulted in the desired business results being achieved rather than representing proof beyond

reasonable doubt of the same (Kirkpatrick, 2006). Therefore any intervening factors that may arise

in the interim, such as substantive changes to workplace practices, policies or procedures, will

have to be identified and considered in the analysis of the Level 4 results.

It is recommended that Level 4 results are measured as follows:

Comparison a pre – post course implementation accident report forms. A significant

reduction of accident reports relating to food safety and hygiene practices following

implementation of the training course will provide evidence that food safety and hygiene

practices have improved.

Comparison of pre – post course implementation food safety and hygiene related customer

complaint records. Similarly, a significant reduction in relevant customer complaint

records, including those relating to objects found in food, spoiled food and poor employee

hygiene, following implementation of the training course will provide evidence that food

safety and hygiene practices have improved as a result of the same.

Comparison of pre – post course implementation Health & Safety Executive food safety &

hygiene inspection ratings. Enhanced food safety and hygiene inspections ratings

following full implementation of the course will provide evidence of improved food safety

and hygiene practices.

Comparison of pre – post course implementation food spoilage and wastage financial

records. Significant reductions in costs related to food spoilage and wastage following full

implementation of the course may be attributable to increases in food safety and hygiene

practices if other intervening factors are not identified such as physical improvements to

kitchen storage facilities.

BB has advised that the abovementioned documentation required for identification and

analysis of the Level 4 of business impacts resulting from implementation of the Level 1

Introduction to Food Safety and Hygiene online course are created and retained at each venue as

part of the ordinary course of business and therefore will be readily available for review. HR


13

Departments at each venue will be responsible for gathering together the relevant documentation

and forwarding the same by email to DMC for compiling, coding, analysis and interpretation.

DMC will conduct telephone interviews with each venue manager in order to ascertain any relevant

intervening factors before providing a final Level 4 Evaluation Report to the board of directors

summarizing the business impact results in relation to each venue as well as the organization as a

whole.


14

References

Dow's Systematic Evaluation Model - (Dow Chemical Company. (1992). Evaluation of training:

The Dow process. [Manual]. Midland, Michigan: Author.)

Fitzpatrick, J. L., Sanders, J. R. & Worthen, B. R. (2004). Program evaluation: alternative

approaches and practical guidelines (3rd ed.). Boston: Pearson/Allyn and Bacon.

Kirkpatrick, D., & Kirkpatrick, J. (2006). Evaluating training programs: The four levels (3rd ed.).

San Francisco, CA: Berret-Koekler.

People 1st (2013). State of the Nation Report 2013: An analysis of labour market trends, skills,

education and training within the UK hospitality and tourism industries. Retrieved from

http://www.people1st.co.uk/getattachment/Research-policy/Research-reports/State-of-

the-Nation-Hospitality-Tourism/SOTN_2013_final.pdf.aspx

http://www.people1st.co.uk/getattachment/Research-policy/Research-reports/State-of-the-Nation-Hospitality-Tourism/SOTN_2013_final.pdf.aspx

http://www.people1st.co.uk/getattachment/Research-policy/Research-reports/State-of-the-Nation-Hospitality-Tourism/SOTN_2013_final.pdf.aspx


15

Appendix A

Level 1 Introduction of Food Safety and Hygiene Course Outline

Table A1

Summary of Course Structure and Instructional Content

Module Instructional Content

1 Importance of Food Safety Introduction narration – learning objectives

Food safety info drop-down list: preventing food

poisoning, quality control, legal responsibilities

Activity: click areas in image to identify potential hazards

(no feedback)

2 Contamination Contamination narration

Food contamination sources info drop-down list: micro-

organisms, chemical, physical

Activity: drag and drop examples of chemical, physical and

viral contaminants in to correct category (immediate

feedback)

Contamination prevention scroll-over list: physical, viral,

chemical

Cross-contamination narration

Cross-contamination info drop-down list: food handlers,

utensils, equipment and surfaces, raw food, rubbish, insects

and rodents

3 Handwashing Handwashing narration

Handwashing demonstration instructional video

Printable step-by-step handwashing guide image,

instructions to practice handwashing technique

Follow-up activity: demonstrate proper handwashing

technique (administered by supervisor on-the-job)

4 Pests Pest narration

Commercial kitchen pests info drop-down list: rats and

mice, flies, cockroaches

Activity: drag and drop evidence of pest infestation in to

correct pest category (immediate feedback)

Evidence of pest infestation scroll-over: additional images

5 Cleaning Cleaning narration

Effective surfaces and equipment cleaning techniques

demonstration instructional video

Activity: effective cleaning techniques quizlet (immediate

feedback)

6 Personal Hygiene Personal hygiene narration

Personal hygiene best practices info drop-down list: keep

yourself clean, prepare for work, unsanitary habits


16

Activity: drag and drop personal hygiene poster completion

(immediate feedback)

7 Cooking and Chilling Cooking and chilling narration

Critical temperatures info timeline list: cooking, hot

holding, reheating, cooling

Activity: critical control points matching exercise in form

of timed memory game (immediate feedback)

8 Conclusion Recap narration – learning objectives

Activity: click areas in image to identify potential hazards

(same activity as in module 1, no feedback)


17

Appendix B

Evaluation Sampling Plan

Table B1

Evaluation Sampling Plan for Levels 1 – 4

Task Start Date

# Days Required Percent Complete

Location Assignment

Evaluation Planning

Draft evaluation plan 9/5/15 7 100 HQ. DMC

Review/edit evaluation plan 9/12/15 14 100 HQ. DMC

Develop instruments 9/26/15 15 100 HQ. DMC

Finalize/submit evaluation plan 9/26/15

15 100

HQ. DMC

Preliminary Matters

Pre-trial training session with managers and supervisors 10/16/15

1 0

BB Head Office

DMC

Course Trial

Collect Level 1 & 2 data 10/19/15

90 0

Venues DMC/HR/ Supervisors

Preparing and Reporting Level 1 & 2 Data

Collate Level 1 & 2 data 1/19/16

7 0

HQ./Venues DMC/ Supervisors

Analyze data, prepare/submit Level 1 & 2 Report 1/26/16

14 0

HQ. DMC/BoD

Actioning Level 1 & 2 Recommendations

Make course amendments as per agreed recommendations 2/9/16

30 0

HQ. DMC/BoD

Level 3 Evaluation

3-month participant interviews 1/19/16

90 0

Venues Supervisors

6-month participant interviews 4/19/16

90 0

Venues Supervisors

Collate Level 3 data 7/19/16 7 0 HQ. DMC

Analyze data, prepare/submit Level 1 - 3 Report 8/2/16

14 0

HQ/ BB Head Office

DMC/BoD

Full Course Implementation

Existing staff attend online induction program 9/3/16

90 0

Venues HR/ Supervisors

Level 4 Evaluation

Collect pre-course L4 data 9/3/16 30 0

Venues HR/ Managers


18

Gather post-course L4 data 9/3/16 365 0

Venues HR/ Managers

Collate and forward pre-course L4 data 10/3/16

30 0

Venues HR/DMC

Collate and forward post-course L4 data 9/3/17

14 0 Venues HR/DMC

Analyze data, prepare/submit L4 Report 9/18/17

14 0

HQ/ BB Head Office

DMC/BoD


19

Appendix C

Gantt Chart

Figure 1C. Gantt chart setting out the projected project timelines in respect of the evaluation plan.


20

Appendix D

Post-course Reaction Survey

Reaction response scale:

Strongly Disagree Strongly Agree

1 2 3 4 5

Brightwing Breweries is committed to providing you with the very best training. Please take some

time to complete survey with your honest feedback about your experiences on this course which

will help us to think about how we might improve it for the future. Thank you!

Course Content

1. I feel that this course is relevant to my job position at Brightwing Breweries.

2. I believe that the topics covered will help me to perform my job well.

3. The course activities helped to reinforce my understanding of the topics covered.

4. I felt engaged throughout the training.

5. Please provide us with any comments or suggestions you may have regarding the

instructional content in the text box below.

Course Format and Materials

6. I was able to navigate through the course content without any problems.

7. The images and other visual elements helped to reinforce my understanding of the topics

covered.

8. I felt that the modules were presented in a logical, clearly organized format.

9. Please provide us with any comments or suggestions you may have regarding the course

format and/or materials in the text box below.

Overall satisfaction

10. Overall, I was satisfied with this course.

11. Please provide us with any other comments or suggestions you may have for improving

the course and/or your learning experience within the text box below.


21

Appendix E

Level 1: Results Report, Analysis and Recommendations Format

Table E1

Results Report, Analysis and Recommendations Format

Question Mean Score / 5 Relevant Participant Comments and Suggestions

Course Content

1

2

3

4

Course Content Section Mean Score / 5:

Analysis and Recommendations:

Course Format and Materials

6

7

8

Course Format and Materials Section Mean Score / 5:


Overall Satisfaction

10

Overall Satisfaction Section Mean Score / 5:



22

Appendix F

Handwashing Rubric

See step-by-step

guide below

Poor: 1 point Fair: 2 points Good: 3 points Excellent: 5

points

Followed Steps Failed to

demonstrate any

of the proper

handwashing

steps

Missed three or

more of the

proper

handwashing

steps

Missed one or

two of the proper

handwashing

steps

Correctly

demonstrated all

of the proper

handwashing

steps

Order Failed to

demonstrate the

proper

handwashing

steps in order

Demonstrated up

to six of the

proper

handwashing

steps in order

Demonstrated up

to nine of the

proper

handwashing

steps in order

Correctly

demonstrated all

of the proper

handwashing

steps in order

Time Washed hands

for less than 10

seconds

Washed hands

for up to 20

seconds

Washed hands

for up to 30

seconds

Washed hands

for 40 seconds or

more

Total Score / 15


23

Appendix G

Hazard Perception (Knowledge Pre-test/Transfer Post-test) Screenshot


24

Appendix H

Knowledge Multiple Choice Pre-test and Post-test Questions

Table H1

Pre- and Posttest Multiple Choice Questions and Answers

Question Answer options Answer

1. Foods unlikely to support

bacterial growth are:

A: Raw meat and poultry

B: Seafood and shellfish

C: Dried pasta and noodles

D: Milk and cream

C

2. Rinsing your hands under

hot water will remove

bacteria.

True/False False

3. What is the main reason for

pest control in commercial

kitchens?

A: Pest-free kitchens are more

efficient.

B: Customers will get upset if

they see a pest.

C: Staff might trip over a pest

and hurt themselves.

D: Pests carry bacteria that

can contaminate food

D

4. Poor food hygiene might

cause a few upset stomachs

but it won’t cause serious

illness.

True/False False

5. How can you tell that a food

has been infected with

harmful bacteria?

A: You can’t, it will look and

smell normal.

B: It will smell bad.

C: It will taste “off”.

D: It will change color.

A

6. Micro-organisms are…? A: Only found in the sea.

B: Bacteria, mold, viruses and

other organisms too small to

see with the naked eye.

C: Are really small but you

can see them if you squint.

D: Bacteria.

B

7. Practicing good personal

hygiene is only important

when you work in areas where

customers can see you.

True/False False

8. What jewelry is acceptable

for a member of kitchen staff

to wear during service?

A: Nose ring

B: Bracelet

C: None

D: Wedding band

C


25

9. Cooked food should be

cooled overnight at room

temperature before being

refrigerated.

True/False False

10. Which of the following is

true of bacteria?

A: All bacteria gives people

food poisoning

B: Bacteria and viruses are the

same thing.

C: Bacteria multiplies and

grows faster in warm, moist

environments.

D: Bacteria thrives in

temperatures above 60°C

C

11. Food is safe when it is free

from visible contamination.

True/False False

12. Why must food handlers

wash their hands thoroughly

after handling raw food?

A: So they can take a break.

B: To prevent cross-

contamination.

C: To stop them from getting

dirt on their clothes.

D: To prevent food wastage.

B

13. Which of the following is

an example of a physical

contaminant?

A: Mold growing on a piece of

bread.

B: Pesticide residue on a piece

of fruit.

C: A food handler sneezing

over raw chicken they are

preparing.

D: A piece of jewelry found in

some food.

D

14. Always use a disposable or

clean cloth to wipe down

disinfected surfaces.

True/False True

15. Cross contamination of

food will occur when:

A:Washing hands before

preparing food

B: Cleaning down work

surfaces and equipment

C: Food is frozen

D: Cockroaches walk across

food

D

16. A clean, healthy person

can carry food poisoning

causing bacteria without being

ill themselves.

True/False True


26

17. The temperature danger

zone is between 40°C and

60°C.

True/False False

18. How often should kitchen

rubbish bins be emptied?

A: Frequently and before the

rubbish bin is completely

filled.

B: At the end of each service.

C: Twice a day.

D: When the rubbish bin is

full.

A

19. What should you if you are

feeling sick?

A: Get a good night’s sleep.

B: Take some medicine go to

work as usual.

C: Go to your doctor for a

checkup immediately and

report to your supervisor.

D: Let nature take its course –

you’ll feel better in a couple of

days.

C

20. Food poisoning bacteria

found in raw foods can

survive cooking if the food is

stored in the temperature

danger zone.

True/False True


27

Appendix I

Level 2 Results Reporting Format

Table I1

Comparison of Mean Pre – Post Test Scores

Mean Pretest Score Mean Posttest Score

Learning Gain

(%)

Knowledge Multiple

Choice Test

/10

/10

Hazard Perception

Test

/8 /8

Handwashing Skills

Test

/15 /15

Table I2

Comparison of Pre – Post Knowledge Multi-choice Test Scores Per Question

Question Times Correctly Answered

in Pretest / 30 Participants

Times Correctly

Answered in Posttest / 30

Participants

Learning Gain (%)

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20


28

Figure I1. Column graph that shall be utilized to report the number of participants successfully

attaining the required pass mark in respect of each posttest versus the number of participants failing

to achieve the required pass mark.

0

5

10

15

20

25

30

Knowledge Multi-choice Handwashing Skills Hazard Perception

Posttest Pass Rates

Achieved 80% Pass Mark Did Not Achieve 80% Pass Mark


29

Appendix J

Patterned Interview Form

Instructions for Interviewer: Clarify the purpose of the interview, which is to evaluate the

effectiveness of the trial online course in terms of how far the participant has applied the

encouraged behaviors to their job, and explain to the interviewee that all information they give

during the interview will remain anonymous so they are free to give frank and honest answers to

all of the questions posed. Please ensure that all questions are answered.

…………………………………………………………………………………………..

1. Q1 Instructions for Interviewer: Without showing the interview form to the interviewee use

the table below to elicit and review the food safety and hygiene behaviors that the trial course

encouraged. Discuss each encouraged behavior, asking the interviewee to identify whether or

not they have had the opportunity to display the same in the course of their work to date which

should be confirmed by the interviewee providing examples in that regard. Complete the

relevant table elements during the discussion.

Desired Behavior Identified by Interviewee

during discussion (Y/N)

Interviewee has had the

opportunity to display the

behavior (Y/N)

1. Taking steps to avoid

contamination and cross-

contamination – bacterial,

physical, chemical, viral

2. Correct handwashing

technique

3. Washing hands at

appropriate times

4. Identifying and reporting

pests

5. Keeping work spaces clean

6. Using appropriate cleaning

techniques

7. Maintaining appropriate

level of personal hygiene

8. Wearing appropriate work

clothes

9. Cooking, hot holding,

reheating and cooling foods

to appropriate temperatures

2. Since you completed the course how eager have you been to display the encouraged food safety

and hygiene behaviors we have just discussed?


30

____ Very eager ____ Quite eager ____ Not eager

Comments:

3. How well equipped have you been to behave in accordance with good food safety and hygiene

practices as suggested on the course during your every day work?

____ Very ____ Quite ____ Little ____ None

4. If you are not doing some of the things you were encouraged and taught to do, why not?

How Significant?

Desired Behavior (#) Very Somewhat Not

a. It wasn’t practical for my

situation

b. My boss discourages me

from changing.

c. I haven’t had the chance to do

it yet.

d. I tried it but it didn’t work.

e. Other reasons

Other reasons comments:

Appendix K


31

Level 3 Results Reporting Format

Table K1

Assessment of Retention Rates

Desired Behavior Number of

interviewees

identifying desired

behavior at 3 months /

30

Number of

interviewees

identifying

desired behavior

at 3 months / 30

% increase or

decrease

1. Taking steps to avoid

contamination and

cross-contamination –

bacterial, physical,

chemical, viral

2. Correct handwashing

technique

3. Washing hands at

appropriate times

4. Identifying and

reporting pests

5. Keeping work spaces

clean

6. Using appropriate

cleaning techniques

7. Maintaining

appropriate level of

personal hygiene

8. Wearing appropriate

work clothes

9. Cooking, hot holding,

reheating and cooling

foods to appropriate

temperatures


32

Figure K1. Sample comparison pie charts showing the division of total participant answers to

response options between 3 month and 6 month evaluations, which form is to be utilized to report

findings in relation to interview questions 2 and 3.

Q.2 Responses (3 months)

Very eager Quite eager Not eager

Q.2 Responses (6 months)

Very eager Quite eager Not eager


33

Figure K2. Sample comparison bar graph showing the division of total participant answers to

response options between 3 month and 6 month evaluations, which form is to be utilized to report

findings in relation to interview question 4. Separate graphs will be produced for each significance

category – very significant, somewhat significant and not significant.

0

1

2

3

4

5

6

7

8

Not Practical Boss Discourages No Opportunity Tried But Didn'tWork

Other Reasons

Nu

mb

er o

f P

arti

cip

ants

Very Significant

Desired Behavior 1:

Taking steps to avoid contamination and cross-

contamination – bacterial, physical, chemical, viral

3 months 6 months

Documents

EDCI 528 HPT Solutions & Evaluation Plan, Reception Desk/Call Answering Inefficiency