Upload
abidin-mokhlas-sdn-bhd
View
1.227
Download
1
Embed Size (px)
DESCRIPTION
Citation preview
KMS2014 DESIGN & MANAGEMENT OF TRAINING
PROGRAMME
Evaluation of Training Programmes
Objectives:At the end of this unit, students will be able
to:• Explain why evaluation is important.• Identify and choose outcomes to
evaluate a training program.• Discuss the process used to plan and
implement a good training evaluation.• Discuss the strengths and weaknesses of
different evaluation designs.• Explain the process of conducting a cost-
benefit analysis for a training program.
IntroductionIntroduction• Training effectivenessTraining effectiveness
– the benefits that the company and the trainees receive from training
• Training outcomes or criteriaTraining outcomes or criteria– measures that the trainer and the company use to
evaluate training programs
• Training evaluationTraining evaluation– the process of collecting the outcomes needed to
determine if training is effective
• Evaluation designEvaluation design– from whom, what, when, and how information
needed for determining the effectiveness of the training program will be collected
Reasons for Evaluating Training Reasons for Evaluating Training • Companies are investing millions of dollars
in training programs to help gain a competitive advantage
• Training investment is increasing because learning creates knowledge– This differentiates between those
companies and employees who are successful and those who are not
• Because companies have made large dollar investments in training and education and view training as a strategy to be successful, they expect the outcomes or benefits related to training to be measurable
Training evaluationTraining evaluation provides the data needed to
demonstrate that training does provide benefits to the
company.
Types of evaluation• Formative evaluation
• Summative evaluation
Formative EvaluationFormative Evaluation• The evaluation of training that takes place
during program design and development
• Helps to ensure that:– the training program is well organized and runs
smoothly– trainees learn and are satisfied with the
program
• Provides information about how to make the program better
Pilot TestingPilot Testing• The process of previewing the training
program with potential trainees and managers or with other customers
• It can be used: – as a “dress rehearsal” to show the program to
managers, trainees, and customers – for formative evaluation
Summative EvaluationSummative Evaluation• Evaluation conducted to determine
the extent to which trainees have changed as a result of participating in the training program
• May also measure the return on investment (ROI) that the company receives from the training program
Why Should A Training Program Be Evaluated?Why Should A Training Program Be Evaluated?• To identify the program’s strengths
and weaknesses
• To assess whether content, organization, and administration of the program contribute to learning and the use of training content on the job
• To identify which trainees benefited most or least from the program
Why Should A Training Program Be Evaluated?-ctd.Why Should A Training Program Be Evaluated?-ctd.• To gather data to assist in marketing
training programs• To determine the financial benefits
and costs of the programs• To compare the costs and benefits of
training versus non-training investments
• To compare the costs and benefits of different training programs to choose the best program
The Evaluation ProcessThe Evaluation Process
Conduct a Needs AnalysisConduct a Needs Analysis
Develop Measurable Learning Outcomes Develop Measurable Learning Outcomes and Analyze Transfer of Trainingand Analyze Transfer of Training
Develop Outcome MeasuresDevelop Outcome Measures
Choose an Evaluation StrategyChoose an Evaluation Strategy
Plan and Execute the EvaluationPlan and Execute the Evaluation
Training Outcomes: Kirkpatrick’s Four-Level Framework of Evaluation CriteriaTraining Outcomes: Kirkpatrick’s Four-Level Framework of Evaluation Criteria
Level Criteria Focus
1 Reactions Trainee satisfaction
2 Learning Acquisition of knowledge, skills, attitudes, behavior
3 Behavior Improvement of behavior on the job
4 Results Business results achieved by trainees
Outcomes Used in Evaluating Training ProgramsOutcomes Used in Evaluating Training Programs
AffectiveOutcomes
ResultsReturn onInvestment
CognitiveOutcomes
Skill-BasedOutcomes
• Cognitive outcomesCognitive outcomes– determine the degree to which trainees are
familiar with the principles, facts, techniques, procedures, or processes emphasized in the training program
– measure what knowledge trainees learned in the program
• Skill-based outcomesSkill-based outcomes– assess the level of technical or motor skills– include acquisition or learning of skills and use
of skills on the job
• Affective outcomesAffective outcomes– include attitudes and motivation–reaction outcomes – trainees’ perceptions
of the program including the facilities, trainers, and content
• ResultsResults–determine the training program’s payoff
for the company
• Return on Investment (ROI)–comparing the training’s monetary
benefits with the cost of the training• direct costs• indirect costs• benefits
Determining Whether Outcomes Are GoodDetermining Whether Outcomes Are Good
Good training outcomes need to be:
• Relevant
• Reliable
• Discriminative
• Practical
Good Outcomes: RelevanceGood Outcomes: Relevance• Criteria relevance – Criteria relevance – extent to which
training programs are related to learned capabilities emphasized in the training program
• Criterion contamination Criterion contamination –– extent that training outcomes measure inappropriate capabilities or are affected by extraneous conditions
• Criterion deficiency Criterion deficiency – failure to measure training outcomes that were emphasized in the training objectives
Criterion deficiency, relevance, and contamination:Criterion deficiency, relevance, and contamination:
Outcomes Measured in Evaluation
Outcomes Identified by
Needs Assessment and
Included in Training
Objectives
Outcomes Related to Training
Objectives
Contamination Relevance Deficiency
Good Outcomes (continued)Good Outcomes (continued)
• Reliability –Reliability – degree to which outcomes can be measured consistently over time
• Discrimination –Discrimination – degree to which trainee’s performances on the outcome actually reflect true differences in performance
• Practicality – Practicality – the ease with which the outcomes measures can be collected
Training Evaluation PracticesTraining Evaluation Practices
Training Program Objectives and Their Implications for Evaluation:Training Program Objectives and Their Implications for Evaluation:
Reactions: Did trainees like the program?
Did the environment help learning?
Was material meaningful?
Skill-Based:
Ratings by peers or managers based on observation of behavior
Cognitive: Pencil-and-paper tests Affective: Trainees’ motivation or job attitudes
Skill-Based: Performance on a work sample Results: Did company benefit through sales, quality, productivity, reduced accidents, and complaints?
Performance on work equipment
Outcomes
Learning Transfer
Objective
Evaluation Designs: Threats to ValidityEvaluation Designs: Threats to Validity• Threats to validityThreats to validity refer to a factor that
will lead one to question either:– the believability of the study results
(internal validity)(internal validity), or– the extent to which the evaluation
results are generalizable to other groups of trainees and situations (external (external validity)validity)
Threats to ValidityThreats to Validity
Threats to Internal Validity
• Company• Persons• Outcome Measures
Threats to External Validity
•Reaction to pretest•Reaction to evaluation•Interaction of selection and training•Interaction of methods
Methods to Control for Threats to ValidityMethods to Control for Threats to Validity
Pre- and PosttestsPre- and Posttests
Use of Comparison Use of Comparison GroupsGroups
Random Random AssignmentAssignment
Methods to Control for Threats to Validity-ctd.Methods to Control for Threats to Validity-ctd.• Pretests and Posttests – to improve the
internal validity of the study results:– pretraining measure– posttraining measure
• Use of comparison groups– Hawthorne effect
Types of Evaluation DesignsTypes of Evaluation Designs
•Posttest – only
•Pretest / Posttest
•Pretest / Posttest with Comparison Group
•Time Series
•Solomon Four–Group
Factors That Influence the Type of Evaluation DesignFactors That Influence the Type of Evaluation Design
Factor How Factor Influences Type of Evaluation Design
Change potential Can program be modified?
Importance Does ineffective training affect customer service, product development, or relationships between employees?
Scale How many trainees are involved?
Purpose of training Is training conducted for learning, results, or both?
Organization culture Is demonstrating results part of company norms and expectations?
Expertise Can a complex study be analyzed?
Cost Is evaluation too expensive?
Time frame When do we need the information?
Importance of Training Cost Information - ROIImportance of Training Cost Information - ROI• To understand total expenditures for
training, including direct and indirect costs• To compare costs of alternative training
programs• To evaluate the proportion of money spent
on training development, administration, and evaluation as well as to compare monies spent on training for different groups of employees
• To control costs
To calculate return on investment (ROI), follow these steps:To calculate return on investment (ROI), follow these steps:1. Identify outcome(s) (e.g., quality,
accidents)
2. Place a value on the outcome(s)
3. Determine the change in performance after eliminating other potential influences on training results.
4. Obtain an annual amount of benefits (operational results) from training by comparing results after training to results before training (in dollars)
To Calculate Return on Investment (ROI), follow these steps: ctd.
To Calculate Return on Investment (ROI), follow these steps: ctd.
5. Determine training costs (direct costs + indirect costs + development costs + overhead costs + compensation for trainees)
6. Calculate the total savings by subtracting the training costs from benefits (operational results)
7. Calculate the ROI by dividing benefits (operational results) by costs
The ROI gives you an estimate of the dollar return expected from each dollar invested in training.
Determining Costs for a Cost-Benefit Analysis:Determining Costs for a Cost-Benefit Analysis:
DevelopmentCosts
OverheadCosts
Compensationfor
Trainees
Direct Costs Indirect Costs
Measuring Human Capital and Training ActivityMeasuring Human Capital and Training Activity• Expenditure per employee• Learning hours received per employee• Expenditure as a percentage of revenue• Cost per learning hour received• Percentage of expenditures for external services• Learning hours received per training and
development staff member• Average percentage of learning activities
outsourced• Average percentage of learning content by
content area (e.g. basic skills, customer service, executive development)
• Average percentage of learning hours provided via different delivery methods (instructor led, technology based)
What did I learn from here?