Evaluating the NYC Core Knowledge Early Literacy Pilot: Year 3 Report

11

Evaluating the NYC Core KnowledgeEarly Literacy Pilot: Year 3 Report

Research and Policy Support GroupFebruary 2012

2

Executive Summary – Quantitative AnalysisCore Knowledge Reading (CKR) students had significantly greater gains in

Year 3 than comparison school students on nearly all measures. Overall Achievement

Spring scores for 2nd grade CKR students were greater than that of comparison students on all tests.*

High vs. Low Performers The CKR intervention had an impact for all students, regardless of their incoming fall

reading scores, but the effects were strongest for students with lower incoming scores.

CKR Program Exposure Students new to the CKR program this year (who also had lower incoming fall scores)

displayed greater increases on the WJ Brief Reading than those who had been in the program longer. CKR students who had been in the program for multiple years displayed the highest spring scores.

Post- Pilot CKR Participation CKR Kindergarten and 1st grade classes implementing the CKR program this year

scored significantly higher than comparison schools in the spring on TerraNova Reading.

*All differences were statistically significant except TerraNova Reading

3

Surveys The teacher and administrator surveys indicate high levels of satisfaction with the

CKR Program and a preference for the program over other 2nd grade reading curricula.

Most teachers found that students were very engaged with the content of the curriculum and that it was successful at sparking enthusiasm for reading.

Site Visits Fidelity to the program was quite high. The main difficulty noted was in allotting an

uninterrupted 60 minutes for program implementation.

Most of the observed classrooms fully utilized the materials provided and students were largely attentive during lessons, participated in discussions, and were curious about the content of the materials.

In surveys and interviews, teachers noted difficulty with differentiation of instruction using the provided materials for students with special learning needs, such as English Language Learners and Students with Disabilities.

Executive Summary – Surveys & Site Visits

4

Methodology: A multi-method, longitudinal research designPROGRAM YEAR 3: 2ND GRADE

Assessments of 2nd Grade Students (at 10 CKR schools & 10 comparison schools)*

• Pre- and post-test of literacy skills • Tests of science and social studies skills at the end of the year• Additional test of content-specific literacy skills (the Core Knowledge Assessment for

CKR students only) at the end of the year

Teacher and Administrator Surveys (at 10 CKR schools):• Assesses satisfaction with and impact of CKR

Case studies (at 4 CKR schools):• Classroom observations, administrator & teacher interviews

Hypothesis: Students taught with the Core Knowledge Reading (CKR) Program will gain reading competencies and content knowledge (science and social studies skills) at a faster rate than their peers.

Focus of the Evaluation

• In Year 1 and 2, Reading First schools were used as a subgroup in the evaluation. However, these schools were no longer Reading First in Year 3. ** Kindergartners and 1st graders in CKR pilot schools no longer received full CKR program supports and two schools did not test additional students.

ADDITIONAL GRADE LEVELSAssessments of Kindergarten and 1st Grade Students** (at 9 CKR schools & 9 comparison schools)

• Tests of literacy, science and social studies skills at end of the third year

12.6

70.1

82.0

39.8

6.89.5

78.2

88.0

39.1

5.7

ELL Free/ Reduced Lunch

Black/ Hispanic Level 3/4 Students ELA

School Size (in Thousands)

Demographics: While the comparison schools for the evaluation were initially picked for their similarities to the pilot schools, analyses control for demographics to account for any demographic differences that have emerged over the past two years.

5

CKR Students (N = 700)

Comparison Students

Analyses in all slides exclude students with disabilities. Data from 08-09 was used to select comparison schools (data presented here are from the 2010-11 school year). * Difference between CKR and Control groups statistically significant. ** ELA percents based on total number of students who took the exams in the schools in this evaluation.

Percent of 2nd Grade Students

(N = 357)*

* N = the number of students for whom both fall and spring data were available for any WJ test.

A random sample of half of the students in the comparison schools were selected for testing. All students in CKR schools were tested.

*** *

Evaluation of

Achievement Gains

6

7

Quantitative Results Overview:CKR students displayed higher scores and gains on nearly all tests.

Woodcock-Johnson III TerraNova

Test W-J Brief Reading

W-J Word Attack

(Decoding)

W-J Spelling of Sounds

(Written Spelling)

TerraNova Reading

(Oral Reading Comprehension,

Vocabulary, Basic Reading,

Decoding)

TerraNova Social Studies

TerraNova Science

W-J Letter Word

Identification (Basic Reading

Skills)

W-J Passage Comprehension

(Oral Reading Comprehension)

Spring Scores at

CKR Schools

Compared to Comparison

Schools

CKR

Significantly Greater

Spring Scores

CKR Significantly

Greater Spring Scores

CKR


Spring Scores

CKR

Significantly Greater Spring

Scores

XNo Significant

Difference

(But CK students had

higher scores)

CKR


Spring Scores

CKR


Spring Scores

Change in Fall to Spring

Scores at CKR

Schools Compared toComparison

Schools

*CKR

Significantly Greater Fall to

Spring Change

CKR Greater

Significantly Fall to Spring Change

CKR Greater Significantly Fall to Spring

Change

CKR Greater Significantly Fall to Spring

Change

Not Administered in

Fall

Not Administered

in Fall

Not Administered

in Fall

Note: All analyses control for student demographic characteristics.

*Significant only at one-tailed level. Two-tailed p=.067

Quantitative Results Overview: CKR students’ literacy gains were more than double the gains of students at demographically similar comparison schools.

8

Average Fall-Spring Gainin Scale Score Points

Woodcock-Johnson (Brief Reading Test)CKR Students

Displayed ASignificantly Greater

Change FromFall to Spring

Than Comparison School Students

(N = 683) (N = 352)

Fall Spring

97.4

100.7

101.9

104.5104.4

105.2

1 Year/2nd Grade Only (N=86)2 Years/1st and 2nd Grade (N=162)3 Years/All Grades (N=435)

9

Impact of time in program: CKR had the largest impact among students who were new to the program (who had lower starting scores)However, the highest ultimate (spring) scores were found among those who had been in the program the longest.

Average Fall and Spring Scale Score PointsBy Years in CKR Program in Year 3 Woodcock-Johnson (Brief Reading)

+ 3.2 pts

+0.8 pts

+2.6 pts

10

Impact of time in program: The longer second grade students had been in the CKR program, the higher their TerraNova Reading scores were.

Average Spring Scale Score PointsBy Years in CKR ProgramTerraNova (Reading Test)

3 Years/All Grades (N=447) 2 Years/1st and 2nd Grade (N=169)

1 Year/2nd Grade Only (N=129)

540.0

550.0

560.0

570.0

580.0

590.0

600.0

610.0

594

582577

50th percentileNationwide (606)

Scores are not adjusted for demographics.

10th percentileNationwide (553)

11

Effects of CKR Following Participation in the Pilot: Kindergarten students in schools implementing CKR showed higher TerraNova Reading spring scores than students in comparison schools in both the year they participated in the pilot and the current year.

Spring ScoresTerraNova Reading Test

Scores are not adjusted for demographics.

CKR Kindergartners in 2008-2009Kindergarteners in Year 1

CKR Kindergartners in 2010-2011Kindergarteners in Year 3

CKR Schools (n=693)

Comparison Schools (n=307)

510.2

501.9

CKR Schools (n=547)


512.2

501.7

12Scores are not adjusted for demographics.

CKR Schools (n=698)


560.6

555.0

CKR Schools (n=600)


557.6

541.0

Spring ScoresTerraNova Reading Test

CKR First Graders in 2009-2010First Graders in Year 2

CKR First Graders in 2010-2011First Graders in Year 3

Effects of CKR Following Participation in the Pilot: First grade students in schools implementing CKR showed higher TerraNova Reading spring scores than students in comparison schools in both the year they participated in the pilot and the current year.

13

Administrator and Teacher Surveys

13

1414

Administrators reported satisfaction with the CKR program.

No

7Yes

2

Will your 2nd grade classrooms be using the CK

Reading Curriculum next year?

(n = 9)

Would you recommend the CK Reading

curriculum to other school administrators

you know?

(n = 9)

1No

Not Sure

7Yes

Administrators’ overall

satisfaction with CK Reading

(n = 9) *

The writing connection is missing. We don't feel there is enough aligned to the new CCSS and although the children are able to get the Phonics, and Phonemic Awareness down; the comprehension and independent thinking skills are lacking…

VerySatisfied

SomewhatSatisfied

Neutral

We feel that it is not preparing them for the state standards for reading and Writing.

Each school is unique. I have a significant amount of ELL’s and SWD's other schools may not. As I have said before, children gain in Phonics and Phonemic Awareness. They need more basic vocabulary development and comprehension skills and making a writing connection.

1

* Administrators had five options for answering this question: Very Satisfied; Somewhat Satisfied; Neutral; Somewhat Dissatisfied; Very Dissatisfied .

33.3%38.5%

38.9%38.5%

1515

Teachers’ overall satisfaction with CK

Reading(n = 18)a

Percent of Respondents

77.0%72.2%

Much BetterSomewhat Better

Teachers’ overall opinion of CK Reading compared with other

reading programs(n =13)b

Very Satisfied

Somewhat Satisfied

Other Answers: Question a Neutral=2 (11.1%); Somewhat dissatisfied=2 (11.1%); Very Dissatisfied=1 (5.6%); Question b About the Same=0; Somewhat Worse=2 (15.4%); Much Worse=1 (7.7%). In Year 2, Overall satisfaction was 82.7% very/somewhat satisfied, while overall opinion of CK relative to other programs was 50% much/somewhat better

Teachers’ Views:“The students enjoyed learning new concepts

which sparked their reading and the need to know. Although their skills were limited they were always ready for a new concept or a new presentation of skills. Because this was a new program for me as a teacher it sparked my interests as well.”

“Over more than twenty years of teaching I have experienced a number of reading programs. Phonics, spelling, vocabulary and writing and comprehension skills were integrated throughout and the sequence made sense. Enrichment suggestions and remediation suggestions were presented throughout the materials.”

Most teachers were satisfied with CKR and reported that CKR was better than other reading programs.

Note: Calculations were based only on responses that selected an answer other than N/A, which resulted in a lower overall N count.

62.5% 50.0%50.0%

25.0%

25.0% 25.0%

87.5%

75.0% 75.0%

1616

Goals of lessons are clear

(n = 16)

Neutral: N=2 (12.5%)Disagree: N=1 (6.3%)Strong y Disagree: N=2 (12.5%)

I have enough time to complete

daily lessons(n = 16)


Students find activities engaging

(n = 16)

Neutral: N=2 (12.5%)Disagree: N=1 (6.3%)Strongly Disagree: N=2 (12.5%)

Somewhat Agree

Strongly Agree

Goals of lessons are

clear(n = 16)


I have enough time to complete

daily lessons(n = 16)


Students find activities engaging

(n = 16)


Skills Strand Listening and Learning Strand

Percent of Respondents

Teachers indicated that both CKR strands had clear goals, were engaging, and that they had sufficient time to complete the lessons. In Year 2, teachers had rated the Skills Strands higher on these measures.*

* In Year 2, those that rated Strongly Agree or Somewhat Agree: Goals of lessons are clear =96.6%;Students find activities engaging=78.6%; I have enough time to complete daily lessons= 72.4%. Note: Calculations were based only on responses that selected an answer other than N/A, which resulted in a lower overall N count.

68.8%50.0%

62.5%

12.5%

18.8%6.3%

81.3%

68.8% 68.8%

23.1%30.8% 30.8%

45.5%

25.0%

61.5%38.5% 38.5% 9.1%

25.0%

84.6%

69.2% 69.2%

54.5%50.0%

Somewhat Better Much Better

About the Same: N=1 (7.7%)Somewhat Worse: N=0Much Worse: N=1 (7.7%)

About the Same: N=1 (7.7%)Somewhat Worse: N=1 (7.7%)Much Worse: N=2 (15.4%)



About the Same: N=3 (25%)Somewhat Worse: N=2 (16.7%)Much Worse: N=1 (8.3%)

Compared to other programs they taught, most teachers found CKR much or somewhat better at teaching content/background knowledge.

Teachers’ Views:“I like the content reading, the

students really enjoyed learning about different areas and different people. I always got the question-what are we going to learn next?

17Note: Calculations were based only on responses that selected an answer other than N/A, which resulted in a lower overall N count.

Teachers believed the CKR program was better than other programs they have previously used at accommodating different learning needs and teaching vocabulary.

Teaching vocabulary(N=13)

Accommodations for special education

students(N=9)

Accommodations for English language

learners(N=11)

Teaching writing skills(N=13)

23.1%

55.6% 54.5%

23.1%

53.8%

11.1% 9.1%

15.4%

76.9%

66.7% 63.6%

38.5%

Somewhat Better Much BetterTeachers’ Views:“The writing component was weak and

did not really teach students how to write a multi paragraph writing. Most of your writing pieces were quick and easy. They didn't really show the writing process or teach students how to write extended responses.”

18

About the Same: N=0Somewhat Worse: N=2 (15.4%)Much Worse: N=1 (7.7%)

About the Same: N=0Somewhat Worse: N=2 (22.2%)Much Worse: N=1 (11.1%)



Note: Calculations were based only on responses that selected an answer other than N/A, which resulted in a lower overall N count.

19

Site Visits and

Interviews

19

20

Measuring fidelity to the CKR curriculum

• Examining implementation fidelity allows us to better determine:1) whether achievement gains can be attributed to the CKR program2) which components of the CKR program teachers are struggling to

implement and those they are implementing successfully

• We took several different approaches to measuring fidelity in the CKR Pilot schools:

Site visits with classroom observations• Site visits were conducted at 4 randomly selected Pilot schools• 11 classrooms were observed

• Interviews with administrators and teachers• Interviews were conducted with 16 teachers and 3 administrators

• Additional teacher and administrator survey questions

20

Examples from Checklist

• Check to see that CKR visuals are posted and that CKR materials are present and being used effectively

• Verify that the daily schedule allows for both the Skills and the Listening and Learning Strands to last a full 60 minutes each

• Whether or not students are familiar with language introduced in the domain

• Teacher engaging all students in exercises, practice, class discussions, and games

• Whether or not students are attentive to the teacher during instruction

• Whether or not students are eager to participate in the daily activities of the program

• Whether or not students are generally on task and complete assignments in a timely fashion

• Evidence of differentiated instruction and assessment techniques

21

During the site visits, we used a classroom observation protocol developed from the Core Knowledge Reading Pilot Observation Form

These components were taken from the Core Knowledge Reading Pilot

Observation Form

These components were added as additional measures of student engagement

21

22

10

11

9 8

1

1 2 3

10

Students are attentive to

their teacher during

instruction

Students respond

appropriately to their teacher’s

directions, questions, and assignments

Teachers have Skills Strand

materials posted and used Word Walls effectively

Small group and/or

individual practice time

Full 60 minutes is allotted for

both the Listening and Learning and

Skills Strands*

Number of Classrooms (N=11)

Observed During Visit

Not Observed During Visit

Nearly all classrooms demonstrated high fidelity to the CKR program. In particular, we observed high levels of student engagement in lessons.

* 60 minutes for the Strands typically not allotted uninterrupted.

23

Overall, teachers and administrators reported high fidelity to the program. One area where they reported struggling to implement the program was in finding time for daily small group instruction.

Fidelity Survey Questions Spring 2011 Teacher Responses

Please indicate your overall level of satisfaction with the CK Reading Curriculum so far. 72.2% Very Satisfied or Somewhat Satisfied

Typically, I find I have enough time to complete the daily Skills Strand Lesson 81.3% Strongly or Somewhat Agreed

Typically, I find I have enough time to complete the daily Listening and Learning lesson 75% Strongly or Somewhat Agreed

Core Knowledge Reading engages students and sparks enthusiasm for reading 69.2% Strongly or Somewhat Agreed

The Skills Strand materials are developmentally appropriate 75% Strongly or Somewhat Agreed

The Listening and Learning materials are developmentally appropriate 62.5% Strongly or Somewhat Agreed

The CKR curriculum accommodates different learning needs 54.5% Strongly or Somewhat Agreed

24

• Comparison of 3rd grade 2012 NY State ELA test scores among:• Students who participated in the CKR Pilot as second graders in the

2010-2011 vs. students who were assessed at comparison schools in that year.

• Students who had previously participated in the CKR Pilot as second graders (one year in the pilot) versus those who had previously participated in the CKR Pilot for multiple years.

• Exploration of associations between 3rd grade scores on the 2012 NYS ELA test and previous performance on tests of reading comprehension (WJIII, TerraNova) while participating in the pilot as second graders.

NYCDOE/RPSG’s Year 4 EvaluationTrack student achievement through 3rd grade on the NYS English

Language Arts (ELA) test

Next Steps: Year 4

Documents

Evaluating the NYC Core Knowledge Early Literacy Pilot: Year 3 Report