Copyright © 2010

American Institutes

for Research

All rights reserved.

Washington 21st Century Community Learning Centers : From State to Local Program EvaluationJune 2011

2

The Evaluation Team

American Institutes for Research

• Recent merger with Learning Point Associates• Responsible for the development and

maintenance of the Profile and Performance Information Collection System (PPICS)

• Demonstrated 21st CCLC and afterschool content knowledge

• Other statewide evaluations of 21st CCLC in New Jersey, Texas, and Oregon

3

The Evaluation Team

David P. Weikart Center for Youth Program Quality Developers of the Youth and School Age PQAs Working to build program quality systems in

20 states, including 12 statewide 21st CCLC implementations, including Washington

Rigorously tested intervention strategy for improving the quality of youth-serving programs

4

Evaluation Objectives

Provide an Assessment of the Current State of 21st CCLC Program Impact

Support the PPICS Reporting Process and the Collection of Student Level Data

Document the Extent to Which 21st CCLC Programs Are Meeting Local, State, and Federal Targets and Goals

Identify Characteristics Associated With High-Performing Programs

Increase the Capacity of Grantees to Meet Their Program Improvement and Evaluation Obligations

5

Overview

• Provide an overview of the 21st CCLC program at the state level

• Provide an overview of the 21st CCLC program at the local level

• Recommendations for future directions

6

Assessment of the Current

State of 21st CCLC• Define the characteristics of the current

21st CCLC programs in Washington

• Assess program outcomes with participating students

• Examine local evaluation reports and alignment with state level data

• Examine the quality of local evaluation reports to support state guidance

7

Data Sources

• 21st CCLC Profile and Performance Information Collection System (PPICS) Grantee and center level

From APR 2010

• 45 local evaluation reports prepared for the 2009-10 reporting period

8

Characteristics of 21st CCLC Programs in Washington

• 46 active 21st CCLC grantees (21 new, 23 mature, 2 sustaining)

• A total of 172 centers • Median first-year award: $333,886• 54 percent of grants has a school

district as a fiscal agent• 96 percent of centers located in

schools• 34 percent offer summer

programming

9

Characteristics of 21st CCLC Centers in Washington

During Academic Year:• Average of 9.9 hours of programming

after school each week• Average of 4.4 days per week over 32

weeksDuring Summer:• Average of 20 hours of programming

per week • 4.4 weeks of programming

10

Characteristics of 21st CCLC Centers in Washington

• 68 percent of students were regular attendees

• Multiple grade levels served, 38 percent are “elementary only”

• Of the staff, 25 percent are paid school teachers, 35 percent are volunteers.

• Multiple types of activities, 39 percent of the centers were “mostly enrichment “, 19 percent were “mostly homework help”

11

Characteristics of 21st CCLC Centers in Washington

12

Characteristics of 21st CCLC Centers in Washington

13

Characteristics of 21st CCLC Centers in Washington

14

21st CCLC Program Student Outcomes

1. Is higher levels of attendance in 21st CCLC programming related to the desired academic and behavioral outcomes?

2. Are particular center and student characteristics associated with student academic and behavioral improvement?

15

State Assessment Outcomes

16

Attendance & Program Outcomes• Significant positive relationship between

# of days in the program and improved behavior (based on teacher surveys).

• Higher level of program attendance was not significantly related to increased performance in state assessments in reading and mathematics in 2009-10 (among students who scored below proficiency in 2008-09).

17

Program Characteristics & Outcomes

School-based centers :

• More likely to be associated with teacher reports of higher improvement rates.

• More likely to demonstrate significant improvement in elementary students’ mathematics proficiency level.

• Students that scored in the lowest proficiency category in the previous year (e.g., Well Below the Standard) were more likely to demonstrate improvement than higher performing students.

18

Program Characteristics & Outcomes

• In centers classified as “mostly tutoring”, elementary students’ mathematics assesment scores were more likely to improve.

• In centers classified as “mostly recreation” teachers rated lower rates of improvement in student behaviors.

• In centers staffed by “mostly teachers” teachers reported higher levels of improvement in motivation, attentiveness and motivation, and homework completion and quality.

19

Local Evaluation Reports

1. What type of student outcomes do the programs target, and how do they measure these program outcomes?

2. What evidence can be obtained from the local evaluation reports regarding how programs may impact student outcomes?

20

Student Attendance

• Grantee goals: All grantees took attendance but only 20 identified increasing attendance and retention as a program objective.

• Attendance was highest in programs serving elementary school students and averaged 80 percent or higher.

• Many programs seeking to achieve a high percentage of regular attendees were not able to achieve their objectives.

21

Academic Performance

• Grantee goals: All grantees aimed to increase academic performance.

• PPICS teacher survey and change in state assessment scores are the most commonly used measures of academic performance.

• Many reports stated objectives to achieve a specific percentage of increase in student achievement but only few reported the percent change in achievement.

22

Academic Performance

• Almost all reports shared findings descriptively.

• Findings were more positive for teacher reports of improvement as compared to reports of change in state assessment scores.

• A large number of reports shared only the percentages of students that improved while ignoring those whose behaviors declined or those who did not need to change.

23

Student Behavior and AttitudesGrantee goals: • Increase positive attitudes towards and

sense of connection to school (most common)

• Decrease referrals and negative behavior• Increased student skills• Student exposure to enrichment and

community involvement activities• Exposure to career and post-secondary

education opportunities.

24

Student Behavior and Attitudes• Data collected from student surveys,

teacher surveys, student focus groups, parent surveys.

• Teacher reports suggested that the majority of students improved although significant variations were commonly reported.

• Evaluations that targeted more specific skills and behaviors reported positive change (e.g., reduced discipline referrals, fewer missed days, increased knowledge of careers).

25

Parental Involvement

Grantee goals:• Increase parental involvement in

children’s education• Parent attendance in program

activities• Increasing parent skills and

knowledge in English, literacy, and community resources

26

Parental Involvement

• Data collected through parent surveys, interviews, or informal feedback but primarily reports of parent attendance in program activities.

• Only a small number of programs achieved their goals on parental involvement. Mostly reports of attendance in social events. Few reported increased parent involvement

with student’s education Few reported increased level of skills and

knowledge in parenting, ESL, GED.

27

Analysis of Local Evaluation Reports

• Largely descriptive analysis to assess program impact

• Assessment of outcomes based on PPICS teacher surveys and district-level assessments

• Lack of meaningful assessment of student behavioral growth common

• Assessment of parental involvement based on attendance

• Results not rigorously compiled

28

Quality of Local Evaluation Reports

Outcome Average

Above Standards

Acceptable

MarginalQuality

Below Standards

4 3 2 1Clarity of Presentation

3.0 18 13 13 1

Comprehensiveness of Content

2.8 9 21 13 2

Relevance of Content to Goals and Objectives

3.0 14 17 13 1

Rigor of Evidence

2.4 1 18 25 1

Report Requirements

3.4 18 27 0 0

29

Quality of Presentation

• Average is 3.0• A large number of reports were rated

as Above Standards ; Clearly organized Materials presented in an order that described

what the program goals and objectives are Activities described in a clear manner to

provide an overview of the program.

30

Comprehensiveness of Content

• Average is 2.8• A large number rated Acceptable or

Marginal Quality Varying degrees of detail Inconsistent reporting of data tools

31

Relevance of Content to Goals & Objectives

• Average is 3.0• A large number rated as Acceptable

or Marginal Quality Did not reflect program action theory Objectives are not presented by SMART

criteria Assessments were not aligned with

program goals and intended outcomes.

32

Rigor of Evidence

• Average is 2.4• More than half of the reports are rated

as Marginal Quality Did not triangulate findings from different

resources to support evidence. Descriptive reporting provide little evidence that

outcomes are due to program participation. Limited information on number of respondents

(e.g., survey response rates, focus group participants)

Little insight into program implementation and quality

33

Alignment with Report Requirements

• Average is 3.

• In general, reports followed state requirements and guidelines.

• Weakest aspect is meaningful discussion of findings and recommendations for future decision making.

34

Recommendations on Evaluation Design

• Identification of evaluation questions• Developing a logic model to guide

program evaluation• Consider the most rigorous evaluation

design that resources allow• Align data collection tools with goals

and objectives rather than what’s available Need to use measures that better assess

student behavioral outcomes.

35

Recommendations on Program Implementation and Quality

• Incorporate program monitoring into annual evaluation

• Adoption of a self-assessment tool on program supports and quality of activities

• Developing an evaluation team

36

Recommendations on Reporting

• Use a template to report findings Developed and disseminated by OSPI

• Require more insightful recommendations in evaluation reports that will guide action plans.

37

State-Level Recommendations

• Consider the adoption of one or more measure(s) to assess social-emotional functioning and other behaviors related to academic functioning (e.g., task persistence, organizational skills, etc.)

• Leveraging local assessment data to both (a) inform the design and delivery of programming and (b) assess student growth and development

• Consider adopting indicators that ask programs to measure within-year student growth on formative assessments employed by the districts they are working with

38

State-Level Recommendations

• Further examine the relationship between student recruitment and enrollment policies and the achievement of desired outcomes

• Find ways to connect the leading indicators being developed currently with local evaluation efforts

39

Activity

- Pair or group with the participants sitting next to you.

- Reflect on the findings that were presented. - Share with group 1. How can the local evaluations supplement

state data rather than duplicate the findings?2. What are the most feasible short-term and

long-term recommendations for improving local evaluations?

3. How can the state can provide guidance to implement the short-term and long-term recommendations?

40

Manolya Tanyu , Ph.DResearcher

American Institutes for Research

P: 312-288-7611

E-Mail: mtanyu@air.org

20 North Wacker Dr. Suite 1231

Chicago, IL 60606

General Information: 800-356-2735Website: www.air.org

Neil NaftzgerPrincipal Researcher

American Institutes for Research

P: 630-649-6616

E-Mail: NNaftzger@air.org

1120 East Diehl Road, Suite 200Naperville, IL 60563