Starr Evaluation Workshop

8/2/2019 Starr Evaluation Workshop

1/48


2/48

Questions to Answer Today

1. Why evaluate social projects or programs?2. What types of evaluation are there?3. What are outcomes?4. How do I identify outcomes?5. How do I measure outcomes?6. Where does my data come from?


3/48

Why evaluate social projects orprograms?


4/48

Purposes of Evaluation

Improve Program Implementation Pilot-testing, tracking, mid-course correction, identify

variation or problems

Better planning, better structured programs This is called Formative Evaluation


5/48

Improve Program Implementation Evaluate Program Performance

Did the program meet its intended goals? Should wecontinue to administer program? Should the program beexpanded?

This is called Summative Evaluation

Purposes of Evaluation


6/48

Purposes for Evaluation

Improve Program Implementation Evaluate Program Performance Political or Administrative Reasons Required by law Required by funder

Delay action on a program Promotional


7/48

What types of evaluation are there?

Formative Evaluation What is being implemented? Adequate?

Consistent with design and expectations?

How do clients experience the program?

Summative Evaluation

What results (or net results) can be

attributed to the program? Any behavioral

change directly related to program?

Unintended or unforeseen consequences?


8/48

What are outcomes?

Inputs

Activities Outputs

Outcomes


9/48

Definitions

Program Inputs: Resources or personneldedicated to a program or project Program Activities: Service, treatment,

assistance, or training administered toachieve program goals

Program Outputs: Measures of programactivity consumption

Program Outcomes: Status transformationexperienced by individuals or entities during

or after program participation


10/48

Input, Activity, Output,

or Outcome?


11/48

Access to job postings


12/48

Volunteer


13/48

Long-term mentoring

relationship


14/48

Bridge repair


15/48

Improved Health


16/48

Food rescue


17/48

Funding via specialty

license plates


18/48

Increased confidence


19/48

Housing referral


20/48

Starr Fellowship


21/48

Increased competence

in providing care


22/48

Program Sustainability


23/48

Hip Hop curriculum


24/48

Increased engagement

in philanthropy


25/48

Equitable access to

food


26/48

Mentoring


27/48

Why Collect Information on

Outputs and Outcomes? Track performance Facilitate comparisons

Assess equity and efficiency Refine program or implementation theory Evaluate employees

Strategic planning Improve communication Strengthen the organization


28/48

How do I identify outcomes?

Ask for input from staff, participants orsimilar programs Construct a logic model


29/48

What is a logic model?

A map that outlines your theory ofchange and how you plan to assess it

Key components of a logic model:

Inputs Outputs

Activities Outcomes


30/48

New approachesto social change

Student

competence and

confidence to

effect social

change

Students pursuing

projects post-

graduation

Sample Logic Model

Outcomes

(intended)

Inputs Activities Outputs

Funding

Staff

Food

(primarily

Oreos)

Workshops

Discussions

Mentoring

with alumni

Peer

critiques

Immersion

projects

Completedprojects

Elevator

pitches

Evaluation

reports Sustainability

plans


31/48

31110624-Generation Citizen II TOC (DRAFT)TBG

Generation Citizen II/TOC

GC providescurriculum and

teacher training

GC persuades

districts/principals to

implement/fundGC program

GC provides opportunities forstudents to continue to engage(after-school programs, election

campaigns, ambassadorprogram)

Mentorsprovide

capacity toteachers, and

inspire

participants tobecome engaged

Studentparticipants

haveknowledge,

skillsnecessary to

civicallyengage

Studentsmore

engaged incommunity

in short-term (as

measured byvolunteering,following the

news,participating

incampaigns)

Participantsmore

civicallyengaged inlong term

(as measuredby voting,

volunteering,following the

news,participating

incampaigns,becoming

politicians)

GC recruits andtrains qualified,

motivatedcollege mentors

Teachersdeliver GC

curriculum 2x/wk

Teachersintegrate actioncivics principles

into broader

classroompedagogy

DRAFT

Short-term outcome

Long-term outcome

GC activity

Non-GC activity

Mentors use GCexperience as

impetus to tackleeducation and

political inequityin future jobs


32/48

Logic Model

Annie Wu | Starr Fellowship Retreat | May 8, 2011

Bettergrades

Volunteers timeand effort

Inputs Outputs Outcomes

Money

The LIAProgram

Servicing

Short Term

Long Term

By providing learning opportunities and academic guidance to underprivileged Liberian youth, wecan improve their academic performance, social adjustment, and self confidence.

Lowerincidence ofbehavioralproblems

Increasedconfidence

Betterequipped

Volunteersknowledge and

talents

Homeworkhelp

Learning &fun

A love forlearning!

UnderprivilegedLiberian youth

Books and otherresources

Lower dropout/ pregnancyrate


33/48

Metrics and Measurement

Annie Wu | Starr Fellowship Retreat | May 8, 2011

Number ofstudents

Outputs Outcomes

The LIA ProgramProgress

Measurables

Incidence ofBehavior Problems

Academicassistance

Mentoring

GradesLearningOpportunities


34/48

Inputs

Our time and C.V. Starr funding

11 undergraduate interns from BrownUniversity, Middlebury College, Barnard

College, 1 graduate student intern from theUniversity of Michigan School of NaturalResources and Environment

$2,750 from each intern to cover internshipcosts (including a direct contribution to theeducation fund)


35/48

Activities

Internship training and home-stay

Recruit new host families

English teaching curriculum and materials

English lessons and tutoring Millennium School Feasibility Assessment

Cultural Conservation

Oral History archive

Library of articles and books Raise money for education fund

Sponsor local artists and artisans


36/48

Outputs

Interns are trained as advocates for Secoya

Generate income for Secoya through grassrootstourism and jewelry

Create a framework for future volunteer Englishinstructors to work with Secoya

Help Secoya learn English

Oral history archive and library

Money for education fund


37/48

Outcomes

Interns gain motivation and knowledge to lead

Esperanza projects, raise money

More sustainable community-based tourism

operation geared towards students/volunteers

English instruction enables Secoya to overcome

proximal barriers to education

Oral history archive and library lay the foundationfor culturally-based curriculum

Increased educational opportunities for Secoya

youth through education fund


38/48

Logic Model: Inputs

-coordinators time

-volunteers time

-shareholder money-partnerships with farmers; Farm Fresh Rhode

Island; Brown Dining Services


39/48

Logic Model: Activities

-food purchasing

-market pickup

-shareholder community outreach (emails, recipe

sharing, etc.)-farm participation for shareholders (potlucks,volunteering, etc.)

-publicity and website construction

-budget reporting, compiling of previous programrecords


40/48

Logic Model: Outputs

-500 shareholders will receive food through the

program

-sustainability plan/clear business model

-how-to guide on campus/worksite marketshares model


41/48

Logic Model: Outcomes

1. local equitable food access for Brown

community

2. financial stability for farmers/dollars divertedfrom global food system

3. partnership of Brown community with Rhode

Island agricultural network


42/48

Assumptions

All logic models are based onassumptions

What assumptions have we made in theStarr Fellowship?

What assumptions have you made indesigning your project?


43/48

What is the intended outcome(s) ofyour project?

Small Group Discussion


44/48

How do I measure outcomes?

Difficult to measure something as broad asawareness or as long-termstudents

stay in school through graduation

Each outcome has one or more indicators: The specific observable, measurable

characteristic or change that will represent

achievement of the outcome The specific statistic you will calculate to

summarize level of achievement


45/48

Where does my data come from?

Record extraction

Questionnaires

Interviews Observations

Expert ratings or evaluation


46/48

Issues in Outcome

Measurement

Cost-benefit

Validity external and internal Cause Effect

Negative Externalities


47/48

Starr Deliverable Evaluation Plan

due 5/11. Logic Model of your venture

2. Three assumptions that underlie your model

3. Three outputs to be measured this summer

Metric, target, timeline

4. Three outcomes to be measured this summer

Intermediate or long-term?

Indicators this summer

Target, timeline


48/48

Next meeting small group

next week Bring a draft of your logic model to

share with a small group

Documents

Starr Evaluation Workshop