Chapter 12 designing and conducting summative evaluations

DESIGNING AND CONDUCTING SUMMATIVE EVALUATIONS

PRESENTED BY JAMES HENDERSON

OBJECTIVES

• DEFINE THE PURPOSE OF SUMMATIVE EVALUATION • DESCRIBE THE TWO PHASES OF SUMMATIVE EVALUATION AND THE

DECISIONS RESULTING FROM EACH PHASE • DESIGN A SUMMATIVE EVALUATION TO EXAMINE ORGANIZATIONAL

BENEFITS OF INSTRUCTION THEY HAVE IMPLEMENTED • CONTRAST FORMATIVE AND SUMMATIVE EVALUATION BY PURPOSE

AND DESIGN

COMPARISONS

•FORMATIVE SUMMATIVE EVALUATION IS DEFINED AS THE DESIGN OF EVALUATION STUDIES AND THE COLLECTION OF DATA TO VERIFY THE EFFECTIVENESS OF INSTRUCTIONAL MATERIALS WITH TARGET LEARNERS.

•SUMMATIVE EVALUATION IS DEFINED AS THE DESIGN OF EVALUATION STUDIES AND THE COLLECTION OF DATA TO VERIFY THE EFFECTIVENESS OF INSTRUCTIONAL MATERIALS WITH TARGET LEARNERS.

EVALUATORS

THE PRIMARY EVALUATOR IN A SUMMATIVE EVALUATION IS RARELY THE DESIGNER OR DEVELOPER OF THE INSTRUCTION. EVALUATOR IS OFTEN UNFAMILIAR WITH THE MATERIALS, THE ORGANIZATION REQUESTING THE EVALUATION, OR THE SETTING IN WHICH THE MATERIALS ARE EVALUATED.

PREFERRED BECAUSE THEY HAVE NO PERSONAL INVESTMENT AND WILL LIKELY BE MORE OBJECTIVE

TWO MAIN PHASES OF SUMMATIVE EVALUATION

•EXPERT JUDGEMENT PHASE•FIELD TRIAL PHASE

EXPERT JUDGMENT PHASE EXPERT JUDGMENT PHASE PURPOSEDO THE MATERIALS HAVE THE POTENTIAL FOR MEETING THIS ORGANIZATION’S NEEDS?THERE ARE SEVERAL ACTIVITIES THAT DECIDE WHETHER THE CANDIDATE INSTRUCTION IS PROMISING.

1. CONGRUENCE ANALYSIS 2. CONTENT ANALYSIS 3. DESIGN ANALYSIS 4. UTILITY AND FEASIBILITY ANALYSIS 5. CURRENT USER ANALYSIS

EXPERT JUDGMENT PHASE

CONGRUENCE ANALYSIS

ANALYZING THE CONGRUENCE AMONG 1.AN ORGANIZATION’S STATE NEEDS AND GOALS AND THOSE ADDRESSED IN CANDIDATE INSTRUCTION 2.AN ORGANIZATION’S TARGET LEARNERS’ ENTRY SKILLS AND CHARACTERISTICS AND THOSE FOR WHICH CANDIDATE MATERIALS ARE INTENDED 3.AN ORGANIZATION’S RESOURCES AND THOSE REQUIRED FOR OBTAINING AND IMPLEMENTING CANDIDATE INSTRUCTION

EXPERT JUDGMENT PHASE

CONTENT ANALYSIS

DURING THIS ACTIVITY, AN IDENTIFIED EXPERT IS USED TO JUDGE MATERIAL FOR ACCURACY AND COMPLETENESS TO DETERMINE IF THEY ARE INLINE WITH THE ORGANIZATION’S STATED GOALS. AN INSTRUCTIONAL ANALYSIS OF THE STATED GOAL IS A VERY COST EFFECTIVE METHOD.

EXPERT JUDGMENT PHASE DESIGN ANALYSIS THE DESIGN ANALYSIS IS AN EVALUATION OF THE ADEQUACY OF THE COMPONENTS OF THE INSTRUCTIONAL STRATEGY INCLUDED IN THE CANDIDATE MATERIAL. CHECKLIST ARE GREAT DURING THIS ACTIVITY.

EXPERT JUDGMENT PHASE

UTILITY AND FEASIBILITY ANALYSIS

FACTORS SUCH AS THE AVAILABILITY OF A LEARNER GUIDE OR SYLLABUS AND AN INSTRUCTOR’S MANUAL ARE TAKING INTO CONSIDERATION DURING THIS ACTIVITY. THIS IS ALSO THE TIME WHEN YOU GET INFORMATION FOR THE PEOPLE THAT DETERMINE THAT THE EVALUATION WAS NECESSARY.

EXPERT JUDGMENT PHASE CURRENT USER ANALYSIS

THIS FINAL ANALYSIS SEEKS TO GET INFORMATION ABOUT THE CANDIDATE MATERIAL FROM THE ORGANIZATIONS THAT ARE EXPERIENCED IN USING THEM. THE NAMES OF CURRENT USERS CAN OFTEN BE OBTAINED FROM THE PUBLISHERS OF THE MATERIALS.

FIELD TRIAL PHASE PURPOSE:

ARE THE MATERIALS EFFECTIVE WITH TARGET LEARNERS IN THE PRESCRIBED SETTING?

OUTCOMES ANALYSIS: 1.IMPACT ON LEARNERS 2.IMPACT ON JOB 3.IMPACT ON ORGANIZATION

FIELD TRIAL PHASE (CONTINUED)

MANAGEMENT ANALYSIS:

1.ARE INSTRUCTOR AND MANAGER ATTITUDES SATISFACTORY?

2.ARE RECOMMENDED IMPLEMENTATION PROCEDURES FEASIBLE? 3.ARE COSTS RELATED TO TIME, PERSONNEL, EQUIPMENT, AND RESOURCES REASONABLE?

CHANGE OF AGENT

AS A CHANGE OF AGENT EVALUATIONS KEEPS THE CHECKS AND BALANCES FOR EVERYTHING. WHEN IT COMES TO INSTRUCTIONAL STRATEGY, EVALUATIONS MAKE SURE THAT LEARNING GOALS ARE MET.

JAMES HENDERSONJAMES.HENDERSON178@GMAIL.COM