2

Using Program Evaluation to Assess Your Return on Investment

Schenectady City School DistrictPatty Paser, Assistant to the Superintendent

Kimberly Lewis, District Director of Business and Finance

3

What is Program Evaluation?

Major Types of Evaluation Plans• Objectives-based Evaluation (Tyler, Popham, Steinmetz, Provus)

• Curriculum Evaluation Model (Popham)

• Formative and Process Evaluation Model (Royse, Tyler, & Padgett)

• Participatory-oriented Evaluation (Guba & Lincoln, Patton, Stake)

• Decision/Accountability-oriented Evaluations (Stufflebeam –CIPP)

4

• “The basic purpose of decision-and accountability-oriented studies is to provide a knowledge and value base for making and being accountable for decisions that result in developing, delivering, and making informed use of services that are morally sound and cost-effective . . . .”

• Evaluation’s more important purpose is not to prove but to improve . . .

• Approach is applied both formatively and summatively.” (Stufflebeam & Shinkfield, 2007, p. 199).

Decision/Accountability-Oriented Evaluation (Stufflebeam –CIPP)

5

What is Program Evaluation?

CIPP ModelThe CIPP Model is a social systems model applied to program evaluation. CIPP stands for:

• Context evaluation

• Input evaluation

• Process evaluation

• Product evaluation

These components are or can be viewed as separate forms of evaluation, but they can also be viewed as steps or stages in a comprehensive evaluation.

6

Program Evaluation Implementation Process

• Grouped programs based upon duration• Less than 12 months• 12-36 months• Greater than 36 months

• Older programs – What were the goals? What if they have changed?

• What is a success?

• Do involve multiple stakeholders

• Don’t skip questions

7

Program Evaluation Program: Evaluator:· What problem was the program designed to correct or address?· What need does the program exist to serve?· Why was the program initiated?· What are its goals?

· Whom is it intended to serve?

· Has the program been evaluated previously? If so, what were the findings? How were they used?

· Is the program doing what it proposed or promised to do?· What evidence exists that shows whether the program is doing what is proposed to do?· What evidence exists that shows that the program is effective?· Data collection and analysis· Resources · FTE’s· Training or Ongoing training· Ongoing expense· How cost-effective is the program?

Ø AROI· Recommendations:

Ø Continue Ø ModifyØ Refocus the program or resources

8

Example 1Studio in Art

• Implemented in 2016-17• Goals• participation in at High School level.

• enrollment in Studio in Art Middle School level

• equity among the students enrolled in MS Studio in Art

• Results• 64% of students enrolled in Studio in Art requested a higher level

art class in 2017

• Studio in Art enrollment – 67 in 2016-17; 103 in 17-18

• Recommendation• Modify – maintain 2 sections per each middle school to ensure

classes are running at maximum enrollment.

9

Example 1 ROI Calculation

Full capacity:2 sections/3

schools

Actual:Students who

took higher level art in 9th grade

New goal:Maximize

enrollment

Budget amount 1.0 FTE with benefits

$90,000 $90,000 $90,000

Program .9 FTE $81,000 $81,000 $81,000

Enrollment 103 144

Number of students who took Higher Level Art

n/a 66 92

Cost per success $563 $1,227 $880

10

Example 2 Read 180

• Implemented in 2010 (we think)• Goals• Accelerate reading levels for students in reading – Tier II, III and special

education classrooms.

•Results• When program is implemented with fidelity, 90 minutes of intervention daily,

then results are positive• 2016-17 21.7% students had accelerate growth• 2017-18 12% students had gains

•Recommendation• Increase fidelity - Reduce number of licenses• Target assistance to teachers • Hire teaching assistants to implement program• Target use with special education staff• Recent change – upgrade to software and material will require significant outlay

11

Example 2 ROI Calculation

2016-17 2017-18 2018-19

Annual license costsStaffing costs

$35,000$190,000

$35,000$342,000

$204,000$342,000

Total $225,000 $377,000 $544,000

FTE 10 19 ?

Number of students served 445 349 ?

Number of students who had accelerated growth

93 42

12

Example 3 English Language Learners – Co-Teacher Model

• Implementation - On-going mandate•Goals• To increase the passing rate of ELLs in core content

classes and on state Regents exams

•Results• 2016 Cohort Regents Graduation rate – 23%

graduation v 47% non-ELL• 43% drop-out rate

•Recommendation• Drastic overhaul of services provided to ELL

13

Example 3ROI Calculation

2016-17

Budget amount 1.0 FTE with benefits

$90,000

Staffing Costs $270,000

ELL in Cohort 27

Estimated graduation 6

Cost per success $45,000

14

Example 4 Cleaning model to decrease absenteeism

• To Implement in 2018-19• Goals• If we use a new cleaning product designed to curtail the flu virus, we

can decrease student and staff absenteeism.

• Baseline• Need to obtain student and staff sick leave usage, by building, during

the months in 2017-18 in which we know there were flu related absents.

• Plan• Develop a plan concerning the frequency and location within the

schools (cafeteria, self-contained classrooms, elementary classes, nurse’s office, main office)

• Need to calculate outlay for equipment

• Need to calculate the cost for the supplies

15

Program Evaluation Takeaways

• Identify your problem

• Design the program to address that goal

• Identify your desired outcome and your measures

• Identify how you will know you are successful