Team Read Team Read Improving Literacy in the Seattle Improving Literacy in the Seattle

School DistrictSchool District

PA 590C – Program EvaluationPA 590C – Program EvaluationFall 2007Fall 2007

Anne DebuyserieAnne Debuyserie

“...a Seattle School district tutoring program dedicated to increasing the reading skills of elementary students through year-long coaching by high school students ... The program's goals for participating high-school tutors are to develop work experience, a sense of responsibility and accomplishment, and to learn the rewards of community service.”

(Team Read Mission Statement, 1998)

What is Team Read?What is Team Read?

Private-public partnership to improve Private-public partnership to improve literacy in Seattle Public Schoolsliteracy in Seattle Public Schools

Cross-age tutoring program using Cross-age tutoring program using high school student as tutorshigh school student as tutors

Started in March 1998 with a three-Started in March 1998 with a three-phase implementationphase implementation

June 1999: 10 schools, 335 students, June 1999: 10 schools, 335 students, 300 coaches300 coaches

June 2000: addition of 7 schoolsJune 2000: addition of 7 schools

Team Read- Org ChartTeam Read- Org Chart

Team Read Stakeholders Team Read Stakeholders

Craig McCaw (and wife Susan): BusinessmanCraig McCaw (and wife Susan): Businessman John Stanford: Seattle Schools SuperintendentJohn Stanford: Seattle Schools Superintendent Joan Dore: Seattle Schools Reading SpecialistJoan Dore: Seattle Schools Reading Specialist Tricia McKay: Team Read Program MgrTricia McKay: Team Read Program Mgr Team Read Advising Board: Seattle School Team Read Advising Board: Seattle School

District Staff, Alliance for Education pgm District Staff, Alliance for Education pgm representatives, Mc Caws, community representatives, Mc Caws, community representatives.representatives.

Additional investors joined by year 2.Additional investors joined by year 2.

Who is Team Read?Who is Team Read? CoachesCoaches

– Strict selection guidelinesStrict selection guidelines– High expectations from Team Read High expectations from Team Read

leadersleaders– CompensationCompensation

ReadersReaders– School selection comes firstSchool selection comes first– Student selection based on reading levelStudent selection based on reading level– Eligibility based on needEligibility based on need– First-come, first-serve basisFirst-come, first-serve basis– Low socio-economic backgroundLow socio-economic background

Who is Team Read?Who is Team Read? Site CoordinatorsSite Coordinators

– Part-time, teachers or school staffPart-time, teachers or school staff– Meet informally with Tricia McKay for Meet informally with Tricia McKay for

feedbackfeedback VolunteersVolunteers

– Needed due to amount of workNeeded due to amount of work– Partnership with VISTA/Americorps Partnership with VISTA/Americorps

and U. of Washington Pipelineand U. of Washington Pipeline– Assist site coordinatorsAssist site coordinators

Team Read SessionTeam Read Session Two 1 hr session per week during school Two 1 hr session per week during school

yearyear Standardized session format and structureStandardized session format and structure

– 3.00 PM: coaches arrive3.00 PM: coaches arrive– 3.15 PM: snack with reader3.15 PM: snack with reader– 3.30 pm: reading session begins3.30 pm: reading session begins– 4.15 PM: reading session ends/ coaches fill out 4.15 PM: reading session ends/ coaches fill out

paperworkpaperwork– 4.30 PM: reading coaches leave4.30 PM: reading coaches leave

Opportunity for site coordinators to be Opportunity for site coordinators to be creativecreative

Team Read Evaluation ResultsTeam Read Evaluation Results 11stst year evaluation: not as positive as year evaluation: not as positive as

expectedexpected 22ndnd year: promising but Team Read’s year: promising but Team Read’s

impact not as promising as hopedimpact not as promising as hoped Tricia McKay disappointed by results Tricia McKay disappointed by results

and wonders if findings provide her and wonders if findings provide her with the information needed for with the information needed for improvementsimprovements

Questions about the accuracy of Questions about the accuracy of results.results.

Team Read EvaluationsTeam Read Evaluations Evaluation conducted by Margo Jones, Evaluation conducted by Margo Jones,

statistician by trade.statistician by trade. 17 schools in the program by June 2000 17 schools in the program by June 2000

but only 10 used in the evaluation.but only 10 used in the evaluation. 3 main objectives to show Team Read’s 3 main objectives to show Team Read’s

Impact:Impact:– Goal#1: Do the reading skills of the student

readers improve significantly during their participation in the Team Read program?

– Goal#2: How does the program affect the reading coaches?

– Goal#3 What is working well, and what can be improved?

Evaluation Methods Goal #1Evaluation Methods Goal #1 Goal #1: Reading skills. Two-fold Goal #1: Reading skills. Two-fold

approachapproachGradeGrade Pre-TestPre-Test Post-TestPost-Test Post-Test MetricPost-Test Metric

22 DRA Fall 1999DRA Fall 1999 DRA Spring DRA Spring 20002000 Test LevelTest Level

33 DRA Spring DRA Spring 1999*1999*

ITBS Spring ITBS Spring 2000*2000* NCENCE

44 ITBS Spring ITBS Spring 19991999

WASL Spring WASL Spring 20002000 Scale ScoreScale Score

55 WASL Spring WASL Spring 19991999

ITBS Spring ITBS Spring 20002000 NCENCE

Apples and orangesApples and orangesGradGrad

ee Pre-TestPre-Test Post-TestPost-Test Post-Test Post-Test MetricMetric

22

33 NCENCE

44

55

Evaluation Methods Goal #1, Evaluation Methods Goal #1, Cont’dCont’d

22ndnd method: determine proportion of method: determine proportion of Team Read participants who moved Team Read participants who moved from below grade level test score to from below grade level test score to at-or above level at-or above level compared to all compared to all students from entire districtstudents from entire district

Note: one of the selection criteria to Note: one of the selection criteria to be in Team Read is to be within be in Team Read is to be within bottom bottom 25% of district reading 25% of district reading test scoretest score

Evaluation Methods Goal #1Evaluation Methods Goal #1 Both methods for goal 1 are flawed.Both methods for goal 1 are flawed. Note from Jones in report confirming problems Note from Jones in report confirming problems

with different tests.with different tests. “The question then becomes ‘Is it reasonable to

assume that the pre- and post-tests measure the same skills?’ The evaluator adopted a correlation criterion for answering this question. If the correlation between pre- and post-tests was near or above .8 (see footnote 5) (as was the case for the 2nd grade, where pre- and post-tests were essentially the same), the pre- to post-test change score was interpreted as a gain score. If the correlation coefficient was much less than .8, no analysis was performed.”

Evaluation Methods, Goals #1- Evaluation Methods, Goals #1- AssessmentAssessment

Visible problems with tests:Visible problems with tests:– Pre and post-tests do not measure the Pre and post-tests do not measure the

same skills except in 2same skills except in 2ndnd grade grade– All Team Read participants are within All Team Read participants are within

the bottom quartile of the reading the bottom quartile of the reading test scores and can’t be compared test scores and can’t be compared with students district-wide.with students district-wide.

– Analysis could not be carried out for Analysis could not be carried out for some grades (3some grades (3rdrd) due to ) due to incompatibility of reading testsincompatibility of reading tests

Evaluation Method, Goal #2Evaluation Method, Goal #2

Method: questionnaire given to coaches Method: questionnaire given to coaches during their last week of coaching. during their last week of coaching.

Goal: to assess the impact of Team Goal: to assess the impact of Team Read on coaches.Read on coaches.

Identify 3 main areas of job satisfaction:Identify 3 main areas of job satisfaction:– How positive is their experience with TR?How positive is their experience with TR?– Extent to which the coaches feel that TR Extent to which the coaches feel that TR

helps students readershelps students readers– How supportive is TR’s site coordinators How supportive is TR’s site coordinators

and staff?and staff?

Evaluation Method, Goal #2 - Evaluation Method, Goal #2 - AssessmentAssessment

Jones results showed that coaches were Jones results showed that coaches were satisfied with their job and felt a sense satisfied with their job and felt a sense of pride and accomplishment.of pride and accomplishment.

Test performed by Jones was successful Test performed by Jones was successful in meeting its goals, unlike in Goal #1.in meeting its goals, unlike in Goal #1.

Some questions provided answers as to Some questions provided answers as to what to improve in the program.what to improve in the program.

Test is useful to the program managerTest is useful to the program manager

Evaluation Method, Goal #3Evaluation Method, Goal #3

Method: interviews with major Method: interviews with major stakeholders, one site visit, review stakeholders, one site visit, review of last 5 years’ research literature of last 5 years’ research literature on cross-age tutoring and last 3 on cross-age tutoring and last 3 years research on best practices.years research on best practices.

Goal: assess the overall progress Goal: assess the overall progress of the program.of the program.

After the EvaluationAfter the Evaluation Margo Jones makes her Margo Jones makes her

recommendations based on her recommendations based on her findings to improve the efficacy of findings to improve the efficacy of Team Read.Team Read.

Tricia McKay reads the evaluation Tricia McKay reads the evaluation but fail to see what the results are but fail to see what the results are telling her about the programtelling her about the program

Evaluation Methods - SummaryEvaluation Methods - SummaryGoalsGoals Methodology

Reading improvementamong elementarystudents

Analyzed standardized test scores for 10 of 17 Team Read schools. Rate of increase on pre- and post- tests. Percent of students moving from below reading level to at/above grade level. Collected data by grade level and by school. Compared Team Read results to district scores.

Coach development Surveyed coaches by questionnaireProgram implementation Interviewed stakeholders

Observed Team Read sessionReviewed other studies

Evaluation ProblemsEvaluation Problems Academic v. Practitioner approach.Academic v. Practitioner approach. Result presentation.Result presentation. Research design flaws.Research design flaws.

– Not all the data was analyzedNot all the data was analyzed– Comparison with dissimilar itemsComparison with dissimilar items– Statistical analysis stretchedStatistical analysis stretched

Evaluation only covers 3 goals.Evaluation only covers 3 goals. Lack of communication between the Lack of communication between the

evaluator and the program evaluator and the program manager.manager.

What Could Have Been Done What Could Have Been Done Better?Better?

Impact: 2005-2006 Reading Results

From Team Read’s 2006 Annual Report.http://www.teamread.com/downloads/team_read_106903_report_proof.pdf

Team Read Update - 2006Team Read Update - 2006 Program Evaluation Results for 2005-2006

School Year   – 70% of 2nd graders and 52% of 3rd graders were

reading at/above or approaching grade level – 54% of 2nd graders and 35% of 3rd graders

gained greater than 1.5 grade levels in reading – 98% of the parents of 2nd & 3rd graders and 92%

of their teachers reported increased reading skills as a result of participation in Team Read

– 76% of 2nd & 3rd graders said that reading was more fun since joining Team Read