Review: IntroductionReview: Introduction

Define EvaluationDefine Evaluation

How do formal/informal evaluation differ?How do formal/informal evaluation differ?

What are two uses of evaluation in education?What are two uses of evaluation in education?

What are the pros/cons of using an external What are the pros/cons of using an external evaluator?evaluator?

Alternative Approaches to Alternative Approaches to EvaluationEvaluation

Dr. Suzan Ayers Dr. Suzan Ayers

Western Michigan UniversityWestern Michigan University(courtesy of Dr. Mary Schutten)(courtesy of Dr. Mary Schutten)

Alternative ApproachesAlternative Approaches

StakeholdersStakeholders: individuals and groups who have a : individuals and groups who have a direct interest in, and may be affected by, direct interest in, and may be affected by, evaluation; should be involved early, actively & evaluation; should be involved early, actively & continuouslycontinuously

ProgramProgram: activities that are provided on a continuing : activities that are provided on a continuing basis; typically what is evaluatedbasis; typically what is evaluated

There are a variety of alternative, often conflicting, There are a variety of alternative, often conflicting, views of what evaluation is and how it should be views of what evaluation is and how it should be carried outcarried out

Why so many alternatives?Why so many alternatives?

The way one views evaluation directly impacts The way one views evaluation directly impacts the type of activities/methods usedthe type of activities/methods used

Origins of alternative models stem from Origins of alternative models stem from differences in:differences in:Philosophical & ideological beliefsPhilosophical & ideological beliefsMethodological preferencesMethodological preferencesPractical choicesPractical choices

Philosophical & Ideological BeliefsPhilosophical & Ideological BeliefsEpistemologiesEpistemologies (philosophies of knowing)(philosophies of knowing)

ObjectivismObjectivism (social science base of empiricism; replicate)(social science base of empiricism; replicate)SubjectivismSubjectivism (experientially-based; tacit knowledge)(experientially-based; tacit knowledge)

Pros/Cons of each?Pros/Cons of each?Principles for assigning value Principles for assigning value (parallel obj/subj)(parallel obj/subj)

UtilitarianUtilitarian: focus on group gains (avg scores); greatest : focus on group gains (avg scores); greatest good for the greatest numbergood for the greatest number

Intuitionist-pluralistIntuitionist-pluralist: value is individually-determined: value is individually-determined Room for both or are these dichotomous?Room for both or are these dichotomous?

Philosophical purists are rare (impractical?)Philosophical purists are rare (impractical?)Choose the methods right for THAT evaluationChoose the methods right for THAT evaluationUnderstand assumptions/limitations of different Understand assumptions/limitations of different

approachesapproaches

Methodological PreferencesMethodological Preferences

QuantitativeQuantitative (numerical)(numerical) Qualitative Qualitative (non-numerical)(non-numerical)

Evaluation is a transdiscipline; crosses paradigmsEvaluation is a transdiscipline; crosses paradigms ““Law of the instrument” fallacyLaw of the instrument” fallacy

With hammer/nails, all appears to need hammeringWith hammer/nails, all appears to need hammering

Identify what is useful in each evaluation approach, Identify what is useful in each evaluation approach, use it wisely & avoid being distracted by approaches use it wisely & avoid being distracted by approaches designed to deal w/ different needsdesigned to deal w/ different needs

Practical ConsiderationsPractical Considerations Evaluators disagree whether/not intent of evaluation is Evaluators disagree whether/not intent of evaluation is

to render a value judgmentto render a value judgment Decision-makers or evaluator render judgment?Decision-makers or evaluator render judgment?

Evaluators differ in views of evaluation’s political roleEvaluators differ in views of evaluation’s political role Authority? Responsibility? Authority? Responsibility? These dictate eval styleThese dictate eval style

Influence of evaluators’ prior experienceInfluence of evaluators’ prior experience

Who should conduct the evaluation and nature of Who should conduct the evaluation and nature of expertise needed to do soexpertise needed to do so

Desirability (?) of having a wide variety of evaluation Desirability (?) of having a wide variety of evaluation approachesapproaches

Classification Schema for Classification Schema for Evaluation ApproachesEvaluation Approaches

Conceptual approaches to evaluation, NOT techniquesConceptual approaches to evaluation, NOT techniques Objectives-orientedObjectives-oriented: focus on goals/objectives & : focus on goals/objectives &

degree to which they are achieveddegree to which they are achieved Management-orientedManagement-oriented: identifying and meeting : identifying and meeting

informational needs of decision makersinformational needs of decision makers Consumer-orientedConsumer-oriented: generate information to guide : generate information to guide

product/service use by consumersproduct/service use by consumers Expertise-orientedExpertise-oriented: use of professional expertise to : use of professional expertise to

judge quality of evaluation objectjudge quality of evaluation object Participant-orientedParticipant-oriented: stakeholders centrally involved in : stakeholders centrally involved in

processprocess See figure 3.1 (p. 68)See figure 3.1 (p. 68)

Objectives-oriented ApproachObjectives-oriented ApproachPurposes of some activity are specified and Purposes of some activity are specified and

then evaluation focuses on the extent to which then evaluation focuses on the extent to which these purposes are achievedthese purposes are achieved

Ralph W. Tyler popularized this approach in Ralph W. Tyler popularized this approach in education (criterion ref test)education (criterion ref test)

Tylerian modelsTylerian modelsMetfessel & Michael’sMetfessel & Michael’s paradigm paradigm (enlarged vision of (enlarged vision of

alternative instruments to collect evaluation data)alternative instruments to collect evaluation data)Provus’s Discrepancy Evaluation ModelProvus’s Discrepancy Evaluation Model (agree on stds, (agree on stds,

det if discrepancy exists btwn perf/std, use discrepancy info to decide to det if discrepancy exists btwn perf/std, use discrepancy info to decide to improve, maintain, terminate program)improve, maintain, terminate program)

Logic modelsLogic modelsDetermine long-term outcomes & backtrack to todayDetermine long-term outcomes & backtrack to today

Objectives-oriented StepsObjectives-oriented StepsEstablish broad goals or objectives tied to Establish broad goals or objectives tied to

mission statementmission statementClassify the goals or objectivesClassify the goals or objectivesDefine objectives in behavioral termsDefine objectives in behavioral termsFind situations where achievement of Find situations where achievement of

objectives can be shownobjectives can be shownSelect/develop measurement techniquesSelect/develop measurement techniquesCollect performance dataCollect performance dataCompare data with behaviorally stated Compare data with behaviorally stated

objectivesobjectives

Objectives-oriented Pros/ConsObjectives-oriented Pros/Cons

StrengthsStrengths: simplicity, easy to understand, follow : simplicity, easy to understand, follow and implement; produces information relevant and implement; produces information relevant to the missionto the mission

WeaknessWeakness: can lead to tunnel vision: can lead to tunnel vision Ignores outcomes not covered by objectivesIgnores outcomes not covered by objectivesNeglects the value of the objectives themselvesNeglects the value of the objectives themselvesNeglects the context in which evaluation takes Neglects the context in which evaluation takes

place place

Goal Free EvaluationGoal Free Evaluation

This is the opposite of objectives-oriented This is the opposite of objectives-oriented evaluation, but the two supplement one another evaluation, but the two supplement one another Purposefully avoid awareness of goals; should not Purposefully avoid awareness of goals; should not

be taken as given, goals should be evaluatedbe taken as given, goals should be evaluatedPredetermined goals not allowed to narrow focus of Predetermined goals not allowed to narrow focus of

evaluation studyevaluation studyFocus on actual outcomes rather than intendedFocus on actual outcomes rather than intendedEvaluator has limited contact with program manager Evaluator has limited contact with program manager

and staffand staff Increases likelihood of seeing unintended outcomesIncreases likelihood of seeing unintended outcomes

Management-oriented ApproachManagement-oriented Approach

Geared to serve decision makersGeared to serve decision makers Identifies decisions administrator must makeIdentifies decisions administrator must makeCollects data re: +/- of each decision alternativeCollects data re: +/- of each decision alternativeSuccess based on teamwork between evaluators Success based on teamwork between evaluators

and decision makersand decision makersSystems approach to education in which Systems approach to education in which

decisions are made about inputs, processes, decisions are made about inputs, processes, and outputsand outputs

Decision maker is always the audience to Decision maker is always the audience to whom evaluation is directedwhom evaluation is directed

CIPP Evaluation ModelCIPP Evaluation Model(Stufflebeam)(Stufflebeam)

ContextContext Evaluation: planning decisions Evaluation: planning decisionsNeeds to address? Existing programs?Needs to address? Existing programs?

InputInput Evaluation: structuring decisions Evaluation: structuring decisionsAvailable resources, alternative strategies?Available resources, alternative strategies?

ProcessProcess Evaluation: implementing decisions Evaluation: implementing decisionsHow well is plan being implemented? Barriers to How well is plan being implemented? Barriers to

success? Revisions needed?success? Revisions needed?ProductProduct Evaluation: recycling decisions Evaluation: recycling decisions

Results? Needs reduced? What to do after program Results? Needs reduced? What to do after program has ‘run its course’?has ‘run its course’?

CIPP StepsCIPP StepsFocusing the EvaluationFocusing the EvaluationCollection of InformationCollection of InformationOrganization of InformationOrganization of InformationAnalysis of InformationAnalysis of InformationReporting of InformationReporting of InformationAdministration of Evaluation (timeline, staffing, Administration of Evaluation (timeline, staffing,

budget etc…)budget etc…)

ContextContext Evaluation Evaluation Table 5.1 Table 5.1

Objective: Objective: define institutional context, target define institutional context, target population and assess their needspopulation and assess their needs

Method: Method: system analysis, survey, hearings, system analysis, survey, hearings, interviews, diagnostic tests, Delphi technique interviews, diagnostic tests, Delphi technique (experts)(experts)

For deciding upon the setting to be served, the For deciding upon the setting to be served, the goals associated with meeting needs and goals associated with meeting needs and objectives for solving problemsobjectives for solving problems

InputInput Evaluation Evaluation

Objective: Objective: identify and assess system capabilities, identify and assess system capabilities, procedural designs for implementing the strategies, procedural designs for implementing the strategies, budgets, schedulesbudgets, schedules

Method: Method: inventory human and material resources, inventory human and material resources, feasibility, economics via literature review, visit feasibility, economics via literature review, visit exemplary programsexemplary programs

For selecting sources of support, solution For selecting sources of support, solution strategies in order to structure change strategies in order to structure change activities, provide basis to judge implementationactivities, provide basis to judge implementation

ProcessProcess Evaluation Evaluation

Objective: Objective: identify or predict defects in the process or identify or predict defects in the process or procedural design, record/judge procedural eventsprocedural design, record/judge procedural events

Method: Method: monitoring potential procedural barriers, monitoring potential procedural barriers, continual interaction with and observation of the continual interaction with and observation of the activities of the staffactivities of the staff

For implementing and refining the program For implementing and refining the program design and procedure (a.k.a., process control)design and procedure (a.k.a., process control)

ProductProduct Evaluation EvaluationObjective: Objective: collect descriptions and judgments of collect descriptions and judgments of

outcomes and relate them to CIP, interpret worth/meritoutcomes and relate them to CIP, interpret worth/merit

Methods: Methods: measure outcomes, collect stakeholder measure outcomes, collect stakeholder information, analyses of datainformation, analyses of data

For deciding to continue, terminate, modify, or For deciding to continue, terminate, modify, or refocus an activity and to document the effects refocus an activity and to document the effects (whether intended or unintended)(whether intended or unintended)

Uses of Management-oriented Uses of Management-oriented Approaches to EvaluationApproaches to Evaluation

CIPP has been used in school districts, state CIPP has been used in school districts, state and federal government agenciesand federal government agencies

Useful guide for program improvementUseful guide for program improvement

AccountabilityAccountability

Figure 5.1 (p. 94)Figure 5.1 (p. 94)Formative and summative aspects of CIPPFormative and summative aspects of CIPP

Management-oriented Pros/ConsManagement-oriented Pros/Cons

StrengthsStrengths: appealing to many who like rational, : appealing to many who like rational, orderly approaches, gives focus to the orderly approaches, gives focus to the evaluation, allows for formative and summative evaluation, allows for formative and summative evaluationevaluation

WeaknesswsWeaknessws: preference given to top : preference given to top management, can be costly and complex, management, can be costly and complex, assumes important decisions can be identified assumes important decisions can be identified in advance of the evaluationin advance of the evaluation

REVIEW/QsREVIEW/Qs

Why are there so many alternative approaches Why are there so many alternative approaches to evaluation?to evaluation?

What two conceptual approaches to evaluation What two conceptual approaches to evaluation did we discuss tonight? What are their +/-?did we discuss tonight? What are their +/-?

Which, if either, of these approaches do you Which, if either, of these approaches do you think will work for your evaluation object?think will work for your evaluation object?

Identify your most likely evaluation objectIdentify your most likely evaluation object