How does your program work?

Using program theories to guide your evaluation

Mary S MarczakResearch and Evaluation Specialist,

University of Minnesota Extension

Theory of Life: Calvin as the Evaluator

Do you have a program theory?

All programs whose objectives are to cause changes in participants are inherently theory

based.

Wilder Research

Poll #1

To what extent do you agree/disagree with the following statement? “The term theory as used by program developers and implementers typically is used to mean a vague notion or hunch not usually based on social science information.” 

A) Strongly Disagree

B) Tend to Disagree

C) Tend to Agree

D) Strongly Agree

Program theory and evaluation

• Building relationships and coming to a shared understanding

• Test the program theory (Camp K, MN 4-H)• Formative evaluation (performance indicators,

activities-theory fit, implementation quality)• Understand why programs do or do not work

(theory failure? Or implementation failure?)• Attributing outcomes to the program• What else?

Program theory: what is it?

A plausible and sensible

model of how a program is supposed to work

L. Bickman, 1987

A systematic delineation of cause and effect relationships

M. Scheirer, 1987

Identification of resources, activities, and outcomes of a

program and the causal assumptions that connects these

J. Wholey, 1987

Scientific explanations of how social change is possible

C. McClintock, 1987

How should theory be constructed? Two camps• Program theory should be constructed by

key program stakeholders (Wholey, Patton)

• Program theory should stress social science theory and knowledge (Chen, Rossi)

Poll #2

In which “camp” do you put yourself as an evaluator?

A) Theory should be constructed by program stakeholders B) Theory should be constructed based on social science theory and

knowledge

C) Some of both but more heavily influenced by stakeholders

D) Some of both but more heavily influenced by social science theory

E) They are equally important

Pulling it together: Huey-tsyh Chen

Two categories of theories that should be explicit in any program to be evaluated. Each have a role in terms of what we might evaluate.

• Descriptive

• Prescriptive

Descriptive theories

Assumptions about the nature of the problem and the population affected, limiting conditions, and what aspect of the problem should be solved.

Descriptive theory example:

http://eci.ext.msstate.edu/info/theory.pdf

Prescriptive theoriesHow to go about solving the problem or

bringing about change; how should activities be constructed? How should activities/intervention be implemented? What is the causal connection to social change?

Prescriptive theory example:

J Baur and S Son, 2009 Powerpoint

Pulling it all together example:

Danes, 2009 “Staying Resilient in Times of Change” http://cehd.umn.edu/fsos/Centers/RuralMNLife/

Defining the constructs- what are the key components, theory behind the construct, measures.

Word Documents

References• L. Bickman (ed.) Using program theory in evalulation. New Directions for Program

Evaluation, no. 33. San Francisco: Jossey-Bass, Spring, 1987. (multiple authors)• Chen, H-T. Issues in constructing program theory. New Directions for Program Evaluation,

no. 47, Fall 1990.• Chen, H-T and Rossi, P. H. The theory driven approach to validity. Evaluation and

program Planning, 1987, 10, 95-103.• Patton, M. Q. A context and boundaries for theory-driven approach to validity. Evaluation

and Program Planning, 1989, 12, 375-377. • Wilder Research Center. Program theories and logic models. St. Paul, MN: Wilder

Research Center. http://chicano.umn.edu/pdf/resourcesProgram%20Theories%20Example%20Logic%20Model.pdf