YOUTH & FAMILIES AGRICULTUREHEALTHECONOMYENVIRONMENTENERGY COMMUNITIES Evaluation Based Program Development Based on work by Lydia B. Blalock, Ph.D., Youth

YOUTH & FAMILIES AGRICULTURE HEALTH ECONOMY ENVIRONMENT ENERGY COMMUNITIES

Evaluation Based Program Development

Based on work by Lydia B. Blalock, Ph.D., Youth Development Specialist

New Brunswick, New Jersey


From Assessment to Impact: Day 1

Karen Barale, Pierce CountyRenee Overath, Kitsap County

Christy Price, Grant County

Introduction

• Why are you here?• When I hear Program Evaluation

I think . . .• Education Philosophy

– Experiential: I can’t teach anything, but I can provide opportunities to learn!

– There’s no such thing as a stupid question!

How We Got . . . Here

• Various backgrounds• Program Evaluation Philosophy

– We can’t develop or perform an evaluation if the program wasn’t built upon a strong foundation!

• “Train the trainer” Evaluation-Based Program Development

Process Objectives

If I:• Develop and present seminar on EBPD• Develop and distribute appropriate handouts• Engage participants in group activities• Provide participants an opportunity to

practice • Encourage questions

THEN

Outcome Objectives

• Program Development–Understand how to clearly define issue program will address.

–Use program theory to plan activities.

–Write clear outcome and process objectives.

–Use EBPD Model to plan programs.

Participants will:

Outcome Objectives

•Program Evaluation– Develop skills to evaluate programs.– Select appropriate evaluation

methods and designs.– Select criteria to evaluate objectives.– Use EBPD Model to plan evaluation.

Participants will:

Outcome Objectives

• Scholarship of Engagement– Understand scholarship opportunities in

EBPD process. – Identify several avenues to share work. – Use EBPD to plan scholarly activities.

• Cultural Aspects of EBPD– Understand influence of culture on EBPD

(personal & audience).– Understand importance of learning culture of

target audience/s.– Select evaluation methods appropriate for

diverse audiences.

Participants will:

Commit to putting new knowledge and skills into practice!

BUT FIRST: Is it a Program or a program

program/s

event/s

activity/ ies

PROGRAM

PROGRAM

program/s event/sactivity/ ies

•Stakeholder leadership•Faculty leadership•Multi-disciplinary•Sustainable•Academic department •Student component•Research, teaching, outreach

PROGRAM

program/s event/sactivity/ ies

Group Assignments

• Form groups of 4• Sit with new group• Select bag• Develop program based upon

contents of bag• Develop 10-15 minute (tops)

program presentation

Program Development Project: Requirements

• Define problem– What method/s did you use– Identify stakeholders/collaborators

• Develop outcome objectives– Target audience– Resources needs– Short, medium, long term objectives

Program Development Project: Requirements

• Plan program activities (process objectives)

• Develop evaluation plan– Success indicators– Research design and data collection

methods• Identify scholarship opportunities

5. Implementation6. Dissemination & Scholarship 4

Evaluation-Based Program Development: By the Numbers

1. Define the Problem4. Plan Program Evaluation

1. Define the Problem

4. Plan Program Evaluation

5. Implementation

6. Dissemination & Scholarship 4

3. Plan ProgramÑ

2. Develop Outcome Objectives Ï

3. Plan ProgramÑ2. Develop Outcome Objectives Ï

Step 1. Define the Problem

• Describe Current Situation – How does the world look today?

• Why a Problem– Clearly articulate why a problem – Ask why until get to root of

problem– Symptom vs. Cause


• Identify Stakeholders– Population: Who is affected? How

many?– Stakeholder Leadership: interest in

solving problem?– Potential Collaborators: Already

working on problem?


• How: Evidence? Learn more? References & Facts– Learn all you can about the

issue/problem – • “prove” the problem exists• Identify potential solutions

• “Needs” Assessment – Need: difference between actual and

target “state”– A.K.A.: Asset, Gap, etc.

Small Group Activity:Needs Assessment• Purpose of “needs” assessment is to

clearly define or clarify issue.• What kinds of information might you need?• Who or what can provide the information?• What methods could you use to gather

information?• Where can you gather or find the

information?

Needs Assessments

Needs Assessments

• How will we gather the information?– Surveys – Interviews– Focus groups– Card sort– Archival/secondary data

• Criteria for selecting methods– Consider characteristics of target and

respondent groups– Choose combo of methods to gain

different types of info• When

– Develop target dates and timelines

Using Results

• Set Priorities– Factors that influence priorities

• Difference between actual and target states

• Degree of difficulty in addressing needs• Cost of implementing solutions• Political and other factors

– Derive priorities from several data source

– Priorities without resources

Using Results

• Select Solution Strategies to Meet Needs– Consider alternative solutions for

each need– Evaluate alternatives – Select solutions to implement– Scholarship Opportunities?

Table 1. Define the ProblemIdentify who, what, where, when, why.

1. Describe Current Situation How does the world look today?

2. Why a Problem? 3. Identify Stakeholders

Clearly articulate why current situation is a problem. Ask why until you get to root of problem.

Population. Who is affected by problem? How many people are affected?

Stakeholder Leadership. Who’s interested in solving problem?

Potential Collaborators. Who’s interested in solving problem and/or already working on problem?

What is the evidence? How can I learn more? Provide references and facts.

15 Minute Break

Step 2. Develop Outcome Objectives

• Who do you want to change? • What do you want to accomplish? • Do you have resources to

accomplish changes?


• Describe Ideal Situation– How would world look w/o

problem (perfect world)?• Define Target Audience/s

– Who/what can we change (youth, families, policies, etc.)?

– What do we know about culture/s of audience?


• Identify Outcome Objectives– What changes can I influence

in my target audience/s?• Identify Resources

– What do we need?– What do we have?


2. Develop Outcome ObjectivesÏWhat can we change in audience?

Define Target Audience

Who can we change to solve problem?

Identify ResourcesWhat resources are required, available,

needed?

Short Term LEARNING& Awareness, Skills, Attitudes, Opinions,

Knowledge Aspirations, Motivations

Medium Term ACTIONS·Behavior, Practice, Decision-making,

Policies, Social Action

Long Term CONDITIONS þSocial, Economic,

Civil, EnvironmentalIMPACT?

Writing Objectives

• Specific– Will increase knowledge vs. Will know

3 sources of Vitamin C• Time delineated

– At the end of the program, in 6 weeks, etc.

• Measurable– 75%, all, etc.

Writing Objectives

• Example– Participants will increase knowledge

of nutrition.– 75% of program participants will

know 3 sources of Vitamin C 6 weeks after the program.

Table 2. I dentify Outcome Objectives

1. Describe I deal Situation How would the world look if the problem did not exist (the perfect world)?

3. I dentify Outcome Objectives What changes can I influence in my target audience/s?

2. Define Target Audience/ s Who or what can we realistically change (youth, families, farmers, policies, etc.)? What do we know about the culture/s of the audience? Do we need to learn more?

Learning Objectives

Short term changes in awareness, skills,

knowledge, attitudes,

aspirations, motivation. List for

each target.

Action Objectives

Medium term changes in

behavior, policies, practice, decision-

making, social actions. List for

each target.

Condition Objectives

Long term changes in social, civic, economic, environmental

conditions. Impact?

Target 1:

Target 2:

4. Identify Resources Resources Required Revenue Enhancement Development

Opportunities

Step 3. Plan Program

Plan ProgramÑ(Activities)

Develop Process ObjectivesÑWhat will you do to facilitate changes in

target audience/s?

Use “Program Theory” to develop process objectives!

Program Theory: The Key to Unlocking Change

IF we

do this . . .

THEN they

will change . . .IF:THEN Statements

IF we ABC: THEN XYZ

where ABC = the events, programs, activities you will conduct, deliver, develop, train, provide, access, facilitate, etc.

and whereXYZ = participant outcome objectives previously developed

ABC will become your process objectives.

Process Objectives

The actions you will take to produce the desired

outcomes (write curriculum, conduct

training)

Outcome Objectives

Changes in target population as result of

your actions(teachers adopt

curriculum, kids are respectful)

PROCESS:OUTCOME Statements

Delivery Modes

Projects and ActivitiesResultsPRODUCE

Small Group Activity:Process Outcome Challenge

1. Outcome Objectives Insert Outcome Objectives from Table 2.

2. Process Objectives Develop Program Theory (IF:THEN) If we = Conduct, deliver,

develop, train, provide, etc. Then = Outcome

objectives from Table 2

Learning Objectives

Short term changes in awareness, skills,

knowledge, attitudes, aspirations, motivation.

List for each target.

Action Objectives Medium term changes in behavior, policies, practice, decision-

making, social actions. List for each target.

Condition Objectives

Long term changes in social, civic, economic,

environmental conditions.

Impact? If we (insert activities ABC on lines below)

Then (insert Learning Objectives)

Then (insert Action Objectives)

Then (insert Condition Objectives)

3. Plan Program Outputs Deliverables

Program, Curriculum, Emergency Kits Scholarship Go to Table 6!

Template: Process Objectives

WORKING LUNCH1.Define the problem2.Outcome objectives3.Process objectives (program

activities)

Step 4. Plan Program Evaluation


Specify Process Indicators ÑHow will you know if you

successfully implemented program?

Specify Outcome Indicators ÏHow will you know if the audience achieved the specified outcomes?

Why Evaluate?

• I dunno• I think I knew why–once• They make me do it

– Department chair– Funding organization– My mom

• I don’t

Why Evaluate?

• Demonstrate success• Understand failure• Provide feedback for program

improvement– Process– Outcome

• Nip problems in the bud

Why We Don’t Evaluate

• Too hard• Not qualified• Costs too much• Takes too much time• Don’t think about it “in time”• No one ever reads evaluation

reports• Everyone “knows” it’s a good

program

Small Group Activity: Setting Criteria

• Establish criteria– What are criteria (benchmarks,

indicators) for judging your item?• Construct standards

– How well should your item perform on benchmarks (standards)?

• Develop criteria rating system

Small Group Activity:

• AFTER developing criteria: measure performance and compare with standards.

• How well does each type of item measure up against criteria?

Template: Develop Success Indicators

1. Develop Success Indicators NOTE: May have multiple indicators for each objective.We will know (insert Learning Objectives) If (insert criteria for success)

The chocolate chip cookie meets our standard of excellence for aroma

It receives a 3.5 average score for “aroma” from our test panel.

Process Indicators

• How will you know you’ve successfully accomplished your process objectives?

Step 5. Implement Program and Conduct Evaluation

• You planned your work.– Now is the time to work the

plan.• If all goes well – great!• If problems arise – check plan,

revise as needed

15 Minute Break

Cultural Aspects of EBPD

• What is culture?– “Behaviors and beliefs characteristic

of a specific social, ethnic, or age group.”

– Product of our culture– Culture influences (unconsciously)

what we think of people from other cultures.

– Recognize cultural biases towards participants and strive to eliminate them.

Cultural Aspects of EBPD

• Neutralize cultural biases and prevent embarrassing (or worse) EBPD design flaws is:– Know your audience (Step 2 – Develop

Outcome Objects)• Be aware of how differences between

cultures may impact EBPD– Develop cross-cultural efficacy!

Cross Cultural Efficacy

Cultural Competence vs. Cross-cultural Efficacy

Cultural competence I can deal with them.

My culture is superior to their culture. (It’s all about me!)

Cross-cultural efficacy

I can effectively interact with people from other cultures.

I can benefit from our interactions. Neither culture is superior to the other. (It’s all about us!)

Small Group Activity:Cultural Influences

Step 6. Dissemination of Results -Extension Scholarship



3. Plan ProgramÑ


5. Implementation


Step 6. Dissemination of Results - Extension Scholarship

• Tenure track faculty engage in appropriate scholarly activities

• Scholarship – Development and implementation of

creative concepts . . . – . . . broadly communicated and

validated by appropriate peers

Step 6. Dissemination of Results - Extension Scholarship

• Forms– Publications

• peer-reviewed journals, books, proceedings, abstracts

• approved extension publications– Peer-validated presentations – Creation of other forms of durable

scholarly products • includes digital and electronic media• that have been scrutinized by peers and

widely communicated. • Scholarship not simply act of doing

creative work . . .• . . . must be peer validated and

communicated

Examples of Outreach Scholarship

• PowerPoint Presentation• Original Handouts and Templates• Professional presentation

(abstract, talks, papers) • Participant Workbook• Curriculum• Train the Trainer Curriculum

Impact Reporting: A Core Competency

• Evaluate & clearly document programmatic outcomes & impacts

• Short, intermediate, long-term• http://ext.wsu.edu/pd/

documents/GuidelinesforDocumentingOutcomesFinal.pdf

Table 6. Dissemination and Scholarship How will you share your work?

Evaluation-Based Program Development

1. Identify Scholarship Activities

2. Identify Products & Deliverables

Table 1. Define Problem

Table 2. Identify Outcome Objectives and Resources

Table 3. Plan Program

Table 4. Plan Program Evaluation

Table 5. Implement Program, Conduct Evaluation and Document Impact

Table 6. Dissemination and Scholarship

Report Results

Demonstrate Impact on Profession

Attain Recognition

Small Group Activity: Extension Scholarship

Day 1: It’s Over!!!

Day 2 Overview• 8:30 Sign-in, Continental Breakfast • 9:00 Review, Questions• 9:15 Part I Program Evaluation• 10:15 Break• 10:30 Part II Program Evaluation• 11:15 Program Development Template• 12 Working Lunch• 1:00 Questions• 1:30 Group Projects • 3:35 Day 2 Post-Assessment, Day 3 Overview

Thank You! Have a Good Evening



Day 2

Welcome

• Evaluation Models, Approaches and Design

• Program Template• Group Projects

Step 4. Plan Program Evaluation





Select Evaluation Design Select Data Collection Method

Evaluation Design 101

• One-Shot– Data collected following program,

• Retrospective Pretest– Data collected following program; participants

recall behavior prior to and following program. • One Group Pretest-Posttest

– Data collected prior to and following program


• Time Series– Data collected prior to, during, after program

• Pretest-Posttest Control-Group– Data collected from 2 different groups prior to

and following program.– Treatment group: program– Control group: no program

• Post-Only Control-Group– Data from 2 different groups collected following

program only


Evaluation Design Pretest Posttest # Times Assessed Follow-up Control

Group

One-Shot Design X 1

Retrospective Pretest X X 1

One Group Pretest-Posttest X X 2

Time Series X X 3 (min) X

Pretest-Posttest Control-Group X X 2x per group X

Post-Only Control-Group X 1x per group X

Data Collection Methods

• Face to Face Surveys– Respondent completes survey in

presence of evaluator– Evaluator may record responses and is

able to answer questions• Self Administered Questionnaires

– Respondent given survey to complete on his/her own.

– Respondent’s responsibility to return survey.

Data Collection Methods

• Personal Interviews– Respondent asked pre-determined

questions. – Evaluator records responses, can ask

respondent to clarify answers.• Focus Groups

– Several respondents meet with evaluator.

– Evaluator facilitates discussion around pre-determined questions.

– Discussion recorded.

Small Group Activity: Evaluation Designs and Methods

Data Collection Template

Learning Objectives Short term changes in awareness, knowledge, skills, attitudes, aspirations, motivation

1. Develop Success Indicators NOTE: May have multiple indicators for each objective.

2. Select Evaluation Design*

3. Select Data Collection Method*

We will know (insert Learning Objectives)

I f (insert criteria for success) Type Pro/Cons Method Pros/Cons

5. Implementation6. Dissemination & Scholarship 4

Evaluation-Based Program Development: By the Numbers

1. Define the Problem4. Plan Program Evaluation



5. Implementation


3. Plan ProgramÑ


3. Plan ProgramÑ2. Develop Outcome Objectives Ï

WORKING LUNCHPlease return by 1:30

5. Implementation(Program & Evaluation)

1. Define the ProblemWhat is issue or problem? Why* is this a problem?

Who is affected by or interested in solving it?




Specify Outcome Indicators ÏHow will you know if the audience achieved the specified outcomes?Develop Process ObjectivesÑ

What will you do to facilitate changes in target audience/s?

3. Plan ProgramÑ(Activities)



IF:THENIF we (conduct, provide, train, etc.): THEN we can expect these outcomes




target audience/s?



Civil, Environmental




needed?











EBPD: Full Model






needed?






The Template

Day 3 Overview

• 8:00 – 9:30 Sign In, Working Breakfast (projects)

• 9:30 – 11:00 Presentations• 11:00 – 11:15 Debriefing• 11:15 – 12 :00 Lunch (non-

working!)• 1:00 – 3:00 Planning Session:

Discuss Training Strategies• 3:00 Travel Home Safely!

Small Group Activity:Develop Your Program



Day 3

Group Presentations

Resources

• Extension Websites– www.cyfernet– www.sammie.osu.edu– http://www.uwex.edu/ces/pdande/evaluat

ion/index.html

• Journals– Journal of Extension– Youth Development Journal– American Evaluation Association

http://www.cyfernet/

http://www.sammie.osu.edu/

http://www.uwex.edu/ces/pdande/evaluation/index.html

http://www.uwex.edu/ces/pdande/evaluation/index.html

Resources

• Databases– Discipline Journals– PsycLit– OVID

• Evaluation Specialists • Faculty with assessment & evaluation

experience

Evaluation-Based Program Development: Process Objectives

If I, as facilitator:• Develop and present seminar on EBPD• Develop and distribute appropriate

handouts• Engage participants in group activities• Provide participants an opportunity to

practice • Encourage questions THEN

Outcome Objectives

• Program Development–Understand how to clearly define issue program will address.

–Use program theory to plan activities.

–Write clear outcome and process objectives.

–Use EBPD Model to plan programs.

Participants will:

Outcome Objectives

•Program Evaluation– Develop skills to evaluate programs.– Select appropriate evaluation

methods and designs.– Select criteria to evaluate objectives.– Use EBPD Model to plan evaluation.

Participants will:

Outcome Objectives

• Scholarship of Engagement– Understand scholarship opportunities in

EBPD process. – Identify several avenues to share work. – Use EBPD to plan scholarly activities.

• Cultural Aspects of EBPD– Understand influence of culture on EBPD

(personal & audience).– Understand importance of learning culture of

target audience/s.– Select evaluation methods appropriate for

diverse audiences.

Participants will:

Commit to putting new knowledge and skills into practice!



IF:THENIF we (conduct, provide, train, etc.): THEN we can expect these outcomes




target audience/s?







needed?












5. Implementation


3. Plan ProgramÑ


The Last Word: Begin with the End in Mind

If you don’t know where you’re going, how are you gonna’ know when you get there? Yogi Berra

Alice asked: Would you tell me please which way I ought to go from here? That depends a good deal on where you want to get to, said the Cat. I don’t much care where . . . said Alice. Then it doesn’t matter which way you go, said the Cat. From Alice’s Adventures in Wonderland

THE END

• Thank You!• Final Questions• Post-Assessment


The “Logic” Behind Evaluation Based Program Development

A Rose by Any Other Name . . .Can Still Have Thorns!

3. Plan ProgramÑ 2. Develop Outcome Objectives Ï2b.

ResourcesÏ2a. Target

Audience Ï

4. Plan Program Evaluation 5. Implementation 6. Dissemination & Scholarship 4


Documents

YOUTH & FAMILIES AGRICULTUREHEALTHECONOMYENVIRONMENTENERGY COMMUNITIES Evaluation Based Program Development Based on work by Lydia B. Blalock, Ph.D., Youth