August 2007 FFP 1740 1

www.floridastatefirecollege.org

Testing and Evaluation Testing and Evaluation Techniques Techniques

Chapter 7Chapter 7

Florida State Fire College

Ocala, Florida

August 2007 FFP 1740 2

Testing and EvaluationTesting and Evaluation

Note this chapter is reviewed because you will be using some of the techniques involved

When you follow-up with Course Design, the chapter will be covered in much more detail

August 2007 FFP 1740 3

Terminal ObjectiveTerminal Objective

The student will be able to construct, administer, and evaluate an assessment instrument

August 2007 FFP 1740 4

Enabling ObjectivesEnabling Objectives

The student shall be able to:Define the four levels of evaluationDifferentiate between summative and formative evaluationDefine the different kinds of testsDiscuss the difference among the various types of testsList various sources for tests

August 2007 FFP 1740 5

Four Levels of Evaluation Four Levels of Evaluation

Level I – reaction

Level II – learning

Level III – transfer

Level IV – business results

August 2007 FFP 1740 6

ReactionReaction

How did the student react to the class?

Evaluation form

Focus groups (rarely used)

Required by most academic institutions

August 2007 FFP 1740 7

LearningLearning

What has the student learned?

Oral examination

Written test

Skills assessment

August 2007 FFP 1740 8

TransferTransfer

How much did the student retain?

Measured six weeks to six months post coursework – on the job

Based upon tests, observations, surveys, and interviews

Not regularly done within the fire service

August 2007 FFP 1740 9

Business ResultsBusiness Results

An assessment of the financial impact of the training or the return on investment.

Six months to two years post coursework

Most difficult to measureTraining courses do not have business

objectives – ie: reduction in accidentsMethodology for assessment is not yet refined

August 2007 FFP 1740 10

EvaluationEvaluation

“An evaluation is a process of making a value judgment based upon one or more sources”

Evaluation processes examine two components:

Instruction from the teacher

Performance of the student on objectives

August 2007 FFP 1740 11

Purpose of EvaluationPurpose of Evaluation

Provides feedback to students

Provide students gratification and motivation

Measures the effectiveness of the instructor

Measures the effectiveness of the program in meeting objectives

August 2007 FFP 1740 12

Formative EvaluationFormative Evaluation

Ongoing evaluation to change or adapt the program

Compares the objectives to the testing strategy

Occurs during development of course or testPilot testing is formative evaluation

August 2007 FFP 1740 13

Summative EvaluationSummative Evaluation

Typically performed at end of programProvides students feedback on mastery of subjectProvides feedback on effectiveness of teaching strategySummative toolsCourse evaluation formsFinal exams

WrittenPractical

August 2007 FFP 1740 14

TestsTests

Instructor’s last chance to emphasize the important information the student needs to retainWrittenPractical Oral

August 2007 FFP 1740 15

Written TestsWritten Tests

Multiple Choice

True/False

Matching

Completion or fill in the blank

Essay

August 2007 FFP 1740 16

Multiple Choice TestsMultiple Choice Tests

Common method for formal and informal evaluationNormally used for national and state certification examinationsEasy to grade and be objective

Test construction miscuesUsing previous questions informationNegatively worded stemsFill in the blank in middle of stemUsing ‘all of the above” or “none of the above”

August 2007 FFP 1740 17

True/False TestsTrue/False Tests

Limited to two answers, no gray area

Difficult to construct in positive voice

Avoid always or never statements

Useful tool as a study guide

August 2007 FFP 1740 18

Matching TestsMatching Tests

Works best with definitions and terms

Difficult to design

Cautious of multiple matches

Test directions must be clear

August 2007 FFP 1740 19

Completion TestsCompletion Tests

Fill in the blankStatements must be clear as to intent of questionNeed to be grammatically correctBe aware of size of blankAvoid having blank at beginning of sentence

August 2007 FFP 1740 20

Essay TestsEssay Tests

May require long or short answerTime consuming and difficult to gradeRecommended to grade in a group formatRubric is a useful grading toolHand written exams must allow sufficient timeShotgun approach.. As much information as possible in hopes of hitting the target

August 2007 FFP 1740 21

Oral ExamsOral Exams

Requires verbal answers by studentsAdvantagesEvaluate quick reaction of studentAssesses the student thought process

DisadvantagesLimited number of students examined at one timeDifficult to standardizeTime consuming and labor intensiveUnexpected distractionsUnfair emphasis on repeated mistakes

August 2007 FFP 1740 22

Practical ExamsPractical Exams

SituationalDemonstration of a skill in the context of a

scenarioFFII Exam for forcible entry

Rote“Demonstration of steps of performing a skill”FFII Exam donning of SCBA

August 2007 FFP 1740 23

Test ResourcesTest Resources

Written examination resources:NFPAPublishers’ test banksFire textbooksFire textbook instructor guidesTextbooks of practice certification examinationsOn-line and computer-based practice certification testsFire Internet sites

August 2007 FFP 1740 24

Test ResourcesTest Resources

Practical examination resources:NFPAFire Internet sitesFire textbooksFire continuing education programs

August 2007 FFP 1740 25

Test ResourcesTest Resources

Oral examination resources:NFPAFire Internet sites

August 2007 FFP 1740 26

ConclusionConclusion

As part of the learning process, the instructor needs a mechanism to evaluate the

student’s learning and identify whether or not the student is achieving the objectives

and goals of instruction.

August 2007 FFP 1740 27

ConclusionConclusion

EvaluateFour LevelsSummativeFormative

TestsKinds of testDifferences among the various typesSources for testing materials

August 2007 FFP 1740 28

Multiple choice tests should avoid what?“Which choice is not correct”“All of the choices are correct”“None of the choices are correct”Maybe I just broke the rule!!!!!!

August 2007 FFP 1740 29

Which type of test is most commonly used for state or national certifications?True/FalseMatchingMultiple choiceFill in the blank

August 2007 FFP 1740 30

Unexpected distractions would likely be a disadvantage in what type of testing?RoteSituationalWritten Oral

August 2007 FFP 1740 31

Of the four levels of evaluation, which one would likely look at job performance 6 months to a year down the road?Reaction Business resultsLearningTransfer