Using the 2011 NECAP Science Results

New England Common Assessment Program

2

Dr. Kevon Tucker-SeeleyAssessment Specialist – NECAP Manager

Office of Instruction, Assessment & Accountability1-401-222-8494

Peter McLarenScience and Technology Specialist

Office of Instruction, Assessment & Accountability1-401-222-8454

Jennifer GoleniaScience and Technology Specialist

Office of Instruction, Assessment & Accountability1-401-222-8406

Lindsay WepmanAssessment Specialist

Office of Instruction, Assessment & Accountability1-401-222-8254

Welcome and RIDE Introductions

NECAP

Service Center

1-877-632-7774

Elliot Dunn

NECAP Science Program Manager

603-749-9102 ext. 2126

dunn.elliot@measuredprogress.org

Harold Stephens

NECAP Program Director

603-749-9102 ext. 2235

stephens.harold@measuredprogress.org

Carole Soule

NECAP Program Manager

603-749-9102 ext. 2450

soule.carole@measuredprogress.org

3

Measured Progress Introductions

NECAP Service Center: 1-877-632-7774

Kellie Beaulieu: NECAP Program Assistant603-749-9102 ext. 2156

beaulieu.kellie@measuredprogress.org

Mellicent Friddell: NECAP Program Assistant603-749-9102 ext. 2355

friddell.mellicent@measuredprogress.org

Alison Cady: NECAP Program Assistant603-749-9102 ext. 2149

cady.alison@measuredprogress.org

4

Measured Progress Introductions

• Review the different types of NECAP Science reports and share State results

• Demonstrate a simple data analysis protocol using NECAP Science Data

• Share some observations from the NECAP benchmarking process

5

Goals of the Workshop

6

Workshop Reference Materials

7

Types of NECAP Science Reports

Student Report (Confidential)Information for Parents

Item Analysis Report (Confidential)School level by student

Results Report (Public) School and District level

Summary Report (Public) District/State level

Student Level Data Files (Confidential)Excel/csv files by grade on district and school confidential site

8

http://iservices.measuredprogress.org

Accessing Your Confidential Reports

Select “Interactive”

to view Interactive

Reports

Select “Reports” to view Static Reports

9

Accessing Your Confidential Reports

NECAP Analysis and Reporting System (NARS) Account Creation Hierarchy

Password Assistance: Use the following list to determine whom to contact for assistance with your User Name and Password:Superintendents – Contact NECAP Service Center at 877-632-7774Principals – Contact NECAP Service Center at 877-632-7774Teachers – Contact School Principal

http://www.ride.ri.gov/assessment/Results.aspx

Using NECAP Science Data

• Focus is improving student learning– The goal is to increase student achievement

• Engage in collaborative discourse about data– Using relevant, timely data to influence educational

decisions is considered a best practice

• Discussion of data requires a structured approach– Protocols, ground rules, and shared talk time

• Remember, NECAP Science data is for assessing school-level achievement, NOT individual student achievement – Look for trends and patterns

NECAP Science Data

Today we’ll take a look at:

• Demo District Results

• Demo School Results

• Test Items and Item Analysis Reports

• State Results

Phases of Collaborative

Inquiry

Adapted from N. Love

Using NECAP Science Data

Using Data―A Simple Approach

1. Observe the data

– Look at the data with your table.

– Make objective statements about the data. Avoid statements like, “It seems…” or “I think…”

– What do you notice?

Using Data―A Simple Approach

2. Discuss

– Talk with your colleagues at your table.

– Why do think the data might look this way?

Using Data―A Simple Approach

3. Action steps

– What are some other sources that could confirm or refute this data?

– What does the data mean for my school regarding curriculum and instruction?

– How can I dig deeper?

Using Data―A Simple Approach

4. Reflection (time permitting)

– What did you learn from this activity?

– Are there new things that you’d like to bring back to your school?

Tying it Together

• Limitations – This is one source of data. You can’t make major

programmatic changes from this alone.

– Look to other sources of data to confirm or refute.

– Think about curricular and instructional action steps.

• Look for professional development around data use in the 2012–13 and 2013–14 school years.

Percentage of Rhode Island Students by Achievement Level

Grade Year Subject SBP PP P PwD P+PwD CHGAverage

SS

4 10/11 SCI 18 38 43 1 44 0 438

4 09/10 SCI 20 36 43 1 44 4 437

4 08/09 SCI 20 40 40 <1 40 --- 437

8 10/11 SCI 29 46 24 1 25 3 833

8 09/10 SCI 31 47 22 <1 22 5 832

8 08/09 SCI 38 45 17 <1 17 --- 831

11 10/11 SCI 25 49 25 1 26 5 1134

11 09/10 SCI 32 48 20 1 21 1 1133

11 08/09 SCI 36 45 19 1 20 --- 1132

10/11 Comparison of NECAP States by Achievement Level

Grade State Subject SBP PP P PwD P+PwD CHGAverage

SS

4 RI Sci. 18 38 43 1 44 0 438

4 NH Sci. 8 36 54 1 55 1 441

4 VT Sci. --- --- --- --- --- --- ---

8 RI Sci. 29 46 24 1 25 3 833

8 NH Sci. 21 52 27 1 28 1 835

8 VT Sci. --- --- --- --- --- --- ---

11 RI Sci. 25 49 25 1 26 5 1134

11 NH Sci. 26 47 26 1 27 3 1134

11 VT Sci. --- --- --- --- --- --- ---

Note: Vermont’s data is embargoed until 9/28. RI will do a public release on 9/27

Benchmarking Short Answer and Constructed Response Items

•Short answer items receive a score from zero to two.

•Constructed response items receive a score from zero to three or zero to four.

•Zeros are employed when a student produces some work, but the work is totally wrong or irrelevant or if he or she leaves the item blank.

•For purposes of aggregating item results, blanks and zeros both count as zero points towards a student’s score.

Preparation for Benchmarking

The work in preparation for scoring student responses included:

• Development of scoring guides (rubrics) •Content specialists from the NH, RI, and VT Departments of Education •Measured Progress’s test developers

•Selection of “benchmark” responses•Examples of student work at different score pointsfor each item•Used in training and continuous monitoring of scorer accuracy

Scorer Training

Scorer training consisted of:

•Review of each item and its related content and performance standard

•Review and discussion of the scoring guide and multiple sets of benchmark responses for each score point

•Qualifying rounds of scoring in which scorers needed to demonstrate a prescribed level of accuracy.

NECAP Benchmarking Process

Lessons Learned from NECAP Benchmarking

• What does it mean – to identify?– to describe?– to explain?

• How does this look over grade levels?

• What does it mean to compare and contrast?

Lessons Learned

Grade 4

• What is a characteristic?

• What observations of information can be gained from charts and diagrams?

• What patterns can be distinguished from graphs?

• How can students better use evidence from data in their explanations?

Lessons Learned

Grade 8

• When should line graphs be used? When should bar graphs be used?

• How can students better support or refute their predictions/hypotheses using evidence from data?

• Why is it important to use multiple trials in an investigation?

Lessons Learned

Grade 11

• What are typical sources of error? – How can they affect the outcomes of an investigation?

• How can skills in the analysis of data be demonstrated more deeply?

• By 11th grade, students should know about the use of proper units of measurement.

• Development of Assessment Targets

• Participation in annual test item review committee & bias and sensitivity review committee

• Use of classroom teacher judgment data

• Participation in standard setting panels

• Technical Advisory Committee work

29

Involvement of Local Educators in NECAP Science

3030

Use the data• Share progress with parents• Identify gaps in curriculum, instruction, and assessment

Identify students in need of accommodations• Anticipate types of accommodations• NimbleTools will be an option – Spring 2012

Plan for administration now• Review what went well and what went not so well during the

2011 administrationBegin preparing your students

• Share the reference sheets• Released Items documents• Preparing Students for NECAP:

Tips for Teachers to Share with Students

Preparing for 2012 NECAP Science

3131

Questions?

Peter McLarenScience and Technology Specialist

Office of Instruction, Assessment & AccountabilityPeter.mclaren@ride.ri.gov

1-401-222-8454

Jennifer GoleniaScience and Technology Specialist

Office of Instruction, Assessment & AccountabilityJennifer.golenia@ride.ri.gov

1-401-222-8406