11

Impact of the California High School Exit Exam (CAHSEE): A Snapshot of

Effects

Wade Buckland, Sheila Schultz, and Becky Dvorak

June 21, 2010

Presentation toThe Council of Chief State School OfficersNational Council on Student Assessment

2

Background

• CAHSEE Background– Exit exam assesses English language arts (ELA) and mathematics– HumRRO has been the independent evaluator since 1999– A requirement to graduate since 2006

• Purpose of instruction study– Assess changes in curriculum and instruction associated with CAHSEE

requirement– Identify effective intervention/remedial programs and practices that help

struggling students pass CAHSEE requirements

• Online surveys administered end of 2008-09 school year– Principals– ELA and mathematics department heads– ELA and mathematics teachers– Teachers of English learners (EL)– Teachers of students with disabilities (SWD)

3

Survey Sample

• Targeted 400 schools

• At least one faculty member responded from 271 different schools

Faculty Position Respondents Notes

Principals 136

ELA Dept. Head 138

Math Dept. Head 162

ELA Teacher 159 (1-6 surveys per school)

Math Teacher 178 (1-5 surveys per school)

EL Teacher 95 (1-3 surveys per school)

Teacher of SWD 126 (1-4 surveys per school)

Total 994

Survey Sample by Position

4

Presentation Focus

• Teacher Quality: How do teacher experience and teaching credentials relate to student performance on the CAHSEE?

• Standards Integration: How have California’s content standards been integrated?

• Student Remediation: Have students had access to remediation? If so, what impact has the remediation had on student performance?

• Impact on Instruction: How has the CAHSEE requirement influenced the California school system?

5

Teacher Quality: Years Teaching

• Teachers were relatively experienced.

- The majority of Department heads reported that most of their teachers had at least 5 years of experience for ELA and Math

-More common that most teachers teaching primary supplemental courses had at least 5 years experience than teachers of intervention programs

Primary or Supplemental Courses

Intervention Programs

ELA (n=105)

Math (n=135)

ELA (n=104)

Math (n=135)

Most 67 67 59 59

Almost Half 19 24 14 14

Few 11 8 13 13

Not Applicable 4 2 15 14

Teachers with Five or More Years of Experience

6

Teacher Quality: Student CAHSEE Performance

• ELA: Years of teaching experience was positively related to student CAHSEE performance for primary and intervention courses.

• Math: No significant relationships.

• Further analyses found no significant relationship for ELA or math between teaching experience and gain in CAHSEE score from Grade 10 to Grade 11 (p>.05), regardless of course type.

Years of Teaching Experience: Primary and Supplemental Courses

Years of Teaching Experience: Intervention Programs

ELA Math ELA Math

Statistical Finding f(2, 96)= 4.10, p<.05

f(2, 127)= 1.71, p>.05

f(2, 84)= 4.85, p<.05

f(2, 111)= 2.39, p>.05

Significant Relationship Yes No Yes No

Relationship of Teaching Experience to CAHSEE Performance of

2009 10th Grade Students

7

Teacher Quality: Credentials

• ELA classes tend to have more teachers working with an appropriate credential than math classes.

• Intervention programs are slightly more likely to be staffed by teachers not holding an appropriate credential than a typical ELA or math class.

Department Head Response

ELA and Math Teachers Who Worked With a Full

Credential

Intervention Programs Taught by Teachers With a Full

Credential

ELA (n=105) Math (n=136) ELA (n=104) Math (n=132)

Nearly or All (more than 90%)

74 45 68 54

Most (75-90%) 14 29 11 10

Some (25-74%) 7 22 10 17

Only a few (less than 25%)

3 3 5 11

None 2 2 7 8

Full Teaching Credential Prevalence

8

Teacher Quality: Student CAHSEE Performance

• ELA: Having appropriate teaching credential was positively related to student CAHSEE performance for primary and intervention courses.

• Math: Having appropriate credential was positively related to performance for primary courses; no significant relationship for intervention courses.

• Further analysis found no significant relationship between teaching credential and gain in CAHSEE score from Grade 10 to Grade 11.

How many teachers in your department have a full ELA/math credential?

How many of your intervention programs/courses are taught by fully credentialed ELA/math teachers?

ELA Math ELA Math

Statistical Finding

f(5, 97)=5.69, p<.05

f(5, 127)=5.20, p>.05

f(5, 96)=2.99, p<.05

f(5, 123)=1.20, p>.05

Significant Relationship

Yes Yes Yes No

Relationship of Teaching Credential to CAHSEE Performance of 2009

10th Grade Students

9

Standards Integration: CAHSEE Blueprints

• In 2005 and 2009, reported integration was mostly complete or complete.

• In 2009, a slightly higher percent reported integration was complete compared to 2005.

Principals Reporting Completion of School’s Integration of California

Content Standards Contained in Blueprints Adopted for CAHSEE by

Year

Response Percentage

ELA Math

2004/05 (n=227)

2008/09 (n=135)

2004/05 (n=227)

2008/09 (n=135)

Complete (96-100%) 47 57 49 53

Mostly Complete (61-95%) 47 39 43 41

Partially Complete (40-60%) 6 2 7 4

Little (less than 40%) 0 1 1 1

Do not know 0 2 0 2

10

Student Remediation: Courses Offered

• Almost half of responding principals indicated their schools offer remediation courses targeted to juniors and seniors who have struggled to pass the CAHSEE requirements.

• Approximately 11% of responding principals indicated their schools do not offer remediation courses.

Response Percentage (n=130)

Yes, for sophomores, juniors, and seniors 39

Yes, for juniors and seniors 48

Yes, for seniors only 2

We do not have CAHSEE intervention courses 11

CAHSEE Intervention/Remediation Courses Offered, According to

Principals

11

Student Remediation: Impact on CAHSEE Scores

• About two-thirds of responding principals indicated their schools’ remediation efforts have been at least minimally effective in improving students’ CAHSEE scores.

Response Percentage (n=120)

Minimally Effective 26

Effective 42

Very Effective 21

Extremely effective 12

Effectiveness of District in Improving CAHSEE Scores for At-risk

Students, According to Principals

12

Student Remediation: Impact on Drop-out

Response Percentage (n=115)

Not at all 6

Slight extent 24

Moderate extent 37

Great Extent 29

Very Great Extent 4

Impact of CAHSEE Intervention/Remediation Courses on Preventing

Students from Dropping out of School, According to Principals

• Of responding principals whose schools offer remediation, almost 90% said those courses have had a positive impact on preventing student drop-out.

13

Impact on Instruction: Student Preparation

Extent Increased Focus of Using CAHSEE-like Tasks, Matching Content of Instruction and Content of CAHSEE, and Covering Test-Taking Skills to Prepare Students for CAHSEE

Response Using CAHSEE-like Tasks

Matching Content of Instruction and

Content of CAHSEE

Covering Test-Taking Skills

ELA %

(n=153)

Math %

(n=169)

ELA %

(n=153)

Math %

(n=169)

ELA %

(n=153)

Math%

(n=169)

Not at all 5 11 5 11 1 4

Slight Extent 27 26 21 25 21 22

Moderate Extent 38 37 43 30 48 36

Great Extent 29 26 31 34 29 38

• While increased focus on using CAHSEE-like tasks, matching instruction to CAHSEE content, and covering test-taking skills to prepare students was fairly consistent, ELA teachers tended to increase their focus slightly more than math teachers, especially in matching instruction to CAHSEE content.

14

Impact on Instruction: Instigate Change

Teacher’s Use of Assessment Results

Response Teacher Type

ELA %

(n=159)

Math %

(n=178)

Provide feedback to students 87 89

Modify lesson plans 87 85

Practice differential instruction 70 69

Assign to supplemental/remedial classes 30 31

Provide feedback to district 40 29

• CAHSEE assessment results were used at a high frequency to provide feedback to students, modify lesson plans, and practice differential instruction.

• Less than half of respondents indicated that CAHSEE assessment results were used to assign students to supplemental/remedial classes and provide feedback to districts.

15

Impact on Instruction: Instigate Change

Ways in Which ELA and Math Teachers Collaborated With Other

Teachers

Response

Teacher Type

ELA %

(n=159)

Math %

(n=178)

By aligning instruction across courses 57 62

By planning coverage of CAHSEE standards 50 48

By assessing individual student needs 62 51

By sharing ideas about teaching strategies 88 83

I collaborate with other teachers in another way 13 11

I do not collaborate with other teachers 3 7

• Most teachers reported they collaborate with other teachers in some way, especially by sharing ideas.

• At least half of responding teachers reported collaborating by aligning instruction across courses, assessing individual student needs, and planning coverage of CAHSEE standards.

16

Impact on Instruction: Instigate Change

Extent to Which CAHSEE Is Used to Change School’s Instruction,

Assessment, and Overall Improvement

Response School’s Instruction School’s Assessment Overall School Improvement

ELA %

(n=153)

Math %

(n=169)

ELA %

(n=152)

Math %

(n=169)

ELA %

(n=153)

Math %

(n=170)

Not at all 13 22 16 21 12 12

Slight Extent 35 34 34 37 27 35

Moderate Extent 40 32 43 28 43 41

Great Extent 12 12 7 14 18 13

• Most teachers reported CAHSEE is used to a slight or moderate extent to change their school’s instruction, assessment, and overall improvement.

• CAHSEE was used slightly more to make overall improvements to the school than to change instruction or assessments.

17

Summary

• Teaching experience was sometimes related to student CAHSEE performance– Years of experience for ELA teachers significantly related to ELA student

CAHSEE performance

• CAHSEE content standards have commonly been integrated into course instruction

• Remediation/Intervention programs commonly present in schools– Although most believe these programs at least somewhat improve

performance and decrease drop out rates, there is room for improvement

• CAHSEE found to impact classroom instruction– Teachers reported increasing their focus on using CAHSEE-like tasks and

matching the content of instruction with that of the CAHSEE

18

Further Reference

• Full Reports:

http://www.cde.ca.gov/ta/tg/hs/evaluations.asp

• Specific questions:

wbuckland@humrro.org