Decision-Oriented Evaluation Approaches

Presenters:

Chris, Emily, Jen & Marie

Why did decision-oriented evaluation approaches emerge?

Decision-oriented evaluations emerged in the 1970’s

Due to the fact that the evaluations were being ignored and had no impact

Evaluators wanted to help people who make decisions actually make decisions

CIPP Program CIPP Program is the concept Mr. Stufflebeam Most staying evaluation model “Evaluation’s most important purpose is not

to prove but to improve” pg. 176 Core Values - ideals held by a society that

must be taken into consideration pg. 177

C. I. P. P.

C - Context Evaluation: What is the desired goal?

I - Input Evaluation: How are you going to implement the program?

P - Process Evaluation: How can you improve/modify the process?

P - Product Evaluation: Did it work? What now?

CIPP uses both formative and summative evaluations

Developmental Stages of a Program(p. 174-176)

Focusing the Evaluation Identify the major levels of decision making. For each level, project the decision situations

to be served and describe each one in terms of locus, focus, criticality, timing, and composition of alternatives.

Define criteria for each decision situation by identifying variables to aide in judging alternatives.

Define policies the evaluator must operate within.

Developmental Stages of a Program

Collection of Information Specify:

Source of information Instruments and methods of collecting information Sampling procedure Conditions and schedule for collecting

information

Organization of Information Provide a format for the collected information. Designate a means for performing the analysis.

Developmental Stages of a Program

Analysis of Information Select the analytical procedures. Designate a means for performing the

analysis. Reporting of Information

Define the audience. Specify the means for providing the

information. Specify the format. Schedule the report.

Developmental Stages of a Program

Administration of the Evaluation Summarize the evaluation schedule. Define staff and resource requirements and

plans for meeting those requirements. Evaluate the potential for providing

information that is valid, reliable, credible, timely, and pervasive.

Specify and schedule the means for periodic updating.

Provide a budget.

How can decision-oriented evaluation help at each stage? Stages are advantageous to the user because it

forces reflection Evaluators must think of evaluation as

cyclical Similar to formative assessment in education

What is the Personal Factor?

According to Patton it is the: identification of “one or more stakeholders who

care about the evaluation and are in a position to use it” (p 179).

“presence of an identifiable individual or group of people who personally care about the valuation and the findings it generates” (p 179).

central element of Utilization-Focused Evaluation (UFE)

What are the key factors of UFE?

1) Identify the intended users Consider interest in the study Consider power in program to be evaluated Ideally – both would be high However, interest is more important than

power

What are the key factors of UFE?

2) Involving primary users in the conduct of the study

Help to identify focus of study, how to use info obtained, design of study, and data collection

Ensures they have a thorough understanding of the methodology

3) Involve primary users in interpreting results and making decisions about judgments, recommendations, and dissemination

Advantages of Decision-Oriented Evaluation Oldest and still frequently used. Still being written about and used as guides to

design individual evaluations or evaluation systems.

Use various means. Stages of development, personal factor, consider

ongoing information needs Provide information that helps people make

decisions.

Disadvantages of Decision-Oriented Evaluation Focus on decisions Tend to neglect stakeholders with less power Do not address social equity and equality Focus on managers and their information needs

Evaluator can become the manager’s “hired gun” Occasionally, the evaluator can not respond to

critical issues if they do not match the issues of the decision maker.

Assume that the important decisions and information to make them can be clearly defined in advance and the program will remain stable while the evaluation is conducted.

What are the strengths and limitations of individual approaches?

Strengths WeaknessesLong standing programHelps people make decisions quickly

Decision-making is left to one personHarder to get stakeholders to buy-in because they do not play a part in the evaluation