Guide to Using the 2006 NECAP & NH-Alternate Reports: Companion PowerPoint Presentation

1

Guide to Using the 2006

NECAP & NH-Alternate

Reports:

Companion PowerPoint Presentation

February 2007

New Hampshire Alternate Assessment

New Hampshire Statewide Assessment

2

• Introductions•NHDoE Updates•Background•NH-Alternate Assessment

Reporting•Short break•NECAP Reporting

Agenda

3


Welcome & Introductions

Tim Kurtz

Director of Assessment

NH Department of Education

Gaye Fedorchak

Alternate Assessment Coordinator

NH Department of Education

https://reporting.measuredprogress.org/NECAPReportingVT

4

Measured Progress Service Center: 1-877-632-7774

Measured Progress Offices: 1-800-431-8901


Welcome & Introductions

Harold StephensNECAP Program [email protected]

Amanda BreitmaierNH Alt. Assessment Program [email protected]

Amanda SmithNECAP Program Manager – Grade [email protected]

Mellicent FriddellNECAP Program [email protected]

Josh EvansNECAP Program Manager – Grades [email protected]

Tina HaleyNECAP Program [email protected]

Elliot ScharffNECAP Program Manager – [email protected]

Jennifer VarneyNECAP Program [email protected]

5


NHDoE Contacts

Tim KurtzDirector of Curriculum and AssessmentPhone: (603) 271-3846E-Mail: [email protected]

Gaye FedorchakAlternate Assessment CoordinatorPhone: (603) 271-7383E-Mail: [email protected]

Susan Morgan - ELL Access SpecialistPhone: (603) 271-3719E-Mail: [email protected]

Carol AngowskiAssessment Program SpecialistPhone: (603) 271-8392E-mail: [email protected]

Linda Stimson - ELA CoordinatorPhone: (603) 271-2035E-mail: [email protected]

Rich Andrusiak - Mathematics CoordinatorPhone: (603) 271-3844E-Mail: [email protected]

Jan McLaughlin - Science CoordinatorPhone: (603) 271-3856E-Mail: [email protected]

Ken Relihan - Social Studies CoordinatorPhone: (603) 271-6151E-mail: [email protected]

Deb Fleurant - Bias and Sensitivity and Title I

Phone: (603) 271-3838E-mail: [email protected]

http://www.ed.state.nh.us/assessment

mailto:[email protected]







6

AssessmentNECAP

Piloted Reading, Math, and Writing in NovemberNo student, school, or district scores will be producedFirst operational test – October 2007

NH-Alt2006-2007 Grade 10 portfolios in Reading, Mathematics, and

Writing2007-2008 In addition to above, Grade 11 portfolios in science

AccountabilitySchools – AYP based on Graduation rate onlyDistricts – AYP based on May 2006 NHEIAP and 2005-2006

Alt

Update: High School Assessment

7

AssessmentNECAP

Pilot Science May 14-24, 2007Two Sessions – about 90 minutes eachNo student, school, or district scores will be producedFirst operational test (three sessions) – May 2008Workshops during the week of April 30 – May 4

NH-AltGrade 11 Science Portfolios will be added in school year

2007-2008

AccountabilityNone at this time, but NCLB reauthorization proposals

include it starting in school year 2007-2008.

Update: Science Assessment

8

i.4.see Workbench• Hub for district submitted student level data• Many new build-it checks on submitted data

i.4.see Dashboard• Check summative student demographic files that affect

AYP• Ability to change incorrect data• Notice will go to Superintendents, Special Education

Directors, i.4.see coordinators, and Principals• Hope to go live on Monday, February 12, 2007• Deadline on changes has not been set yet

Follow the Child Assistance Center• Phase I – April/May• Phase 2 – early summer

Update: Student Demographic & Assessment Data

New

9

Any questions about Updates?

Updates

10


Purpose of the Workshop

• To assist schools and districts in using the variety of NH Statewide Assessment reports and support material in conjunction with the release of results

• To provide school and district personnel an opportunity to ask questions regarding the reports and support materials

11

• Development of Grade Level Expectations (NECAP)

• Development of Alternate Achievement Standards Linked to Grade Level Expectations (NH-Alt)

• Test Item Review Committees (NECAP)

• Bias and Sensitivity Review Committees (NECAP)

• Classroom Teacher Judgment Data (NECAP)

• Standard Setting Panelists (NECAP and NH-Alt)

• Technical Advisory Committee (NECAP and NH-Alt)

• NH-Alt Advisory Task Force (NH-Alt)

Local Educator InvolvementNew Hampshire Alternate Assessment

12

When:• NECAP: Oct 2006• NH-Alternate: School year 2005-2006

What:• Reading and Mathematics –

• NECAP: Grades 3-8 (tests previous year’s GLEs)

• NH-Alt : Grades 2-7, 10 (tested during instruction last year)

• Writing – • NECAP: Grades 5 & 8 (tests previous year’s

GLEs) • NH-Alt: Grades 4 & 7 (tested during

instruction last year)

Test BasicsNew Hampshire Alternate Assessment

13

NECAPVariety of Item Types

– Reading: multiple choice and constructed response– Mathematics: multiple choice, short answer 1 point, short answer

2 points, and constructed response– Writing: multiple choice, constructed response, and extended

response

Common and Matrix Items– Common items used to generate student scores– Matrix items used for equating and field testing

Basics of Test DesignNew Hampshire Alternate Assessment

14

NH Alternate Portfolio Assessment:

What is an “Item” in NH-Alt? – In each content area, two standards-based targeted skills are

identified that address student learning needs– Baseline data are collected for each targeted skill– Work samples are gathered over the course of the year to show

how the student is progressing on these two targeted skills– The collection of work samples and data presented in the

portfolio are scored as a single item– Progress data and student work samples are collected across the

school year (Sept – April) on each targeted skill as the student is learning the skill.

Reading: 2 Targeted Skills Mathematics: 2 Targeted Skills, Writing: 2 Targeted Skills


15

NH Alternate Portfolio Assessment

Targeted skills are represented by a variety of work samples collected over the course of the year.– Collections include multiple samples of actual student work,

which typically show the student solving problems in many ways, i.e. multiple choice responses, short answer responses, and longer responses that look similar to constructed response problems. Responses often involve use of manipulatives, communication systems, or other instructional technology.

– Authentic classroom constructed response and short answer tasks provide the best methods for allowing students with severe disabilities to show their work in the way they are able to produce it.

– Provides richer evidence of access to standards-based content and student progress.


16

Norm-Referenced: What is the student’s status or rank in a designated group?

Criterion-Referenced (Standards-Based): Has the student learned the skills or achieved the standard?

Both NECAP and NH Alternate Assessments are Criterion-Referenced Tests.

Types of Educational TestsNew Hampshire Alternate Assessment

17

Questions about the purpose of the workshop

or NECAP or NH-Alternate

background?

Questions and AnswersNew Hampshire Alternate Assessment

18


Student Report (Confidential) with Information for Parents and Report Interpretation Guide

Student Roster Report (Confidential) School and District levels by student

District Student Level Data Files (Confidential)

Disaggregated Results by Content Area: (Public) School, District, & State reported separately for each grade

State Summary Reports (Public)

NH-Alt Students included in NECAP reports as “NT-Approved”

Basics of Test Design

19


20

21

22

23


• Names of Achievement Levels are the same as in NECAP -

Level 4: Proficient with Distinction

Level 3: Proficient

Level 2: Partially Proficient

Level 1: Substantially Below Proficient

• But - describe performance of grade-linked standards-based skills reduced in depth, breadth, and complexity for the student.

NH-Alt Achievement Levels

24

NH-Alt: Two Scoring Dimensions


1. The Performance Dimension –

Most heavily weighted in score and includes: • Student Progress (from start of year baseline)

• Connections and Access to the General Curriculum (Student work samples must show linkage to grade-level content at reduced level of depth, breadth, complexity)

25

NH-Alt: Two Scoring DimensionsNew Hampshire Alternate Assessment

2. The Program Dimension -

Less heavily weighted in scoring, and includes: • Generalized Performance (Student uses skill

across different settings and situations)

• Self-Determination (Student attempts to self direct and monitor work)

• Supports (Does level of assistance provided support growth of independence and match student need?)

26

Sample Portfolio Score CalculationNew Hampshire Alternate Assessment

Scoring Sub-Dimension:

Base Points

(Range 1-4 pts.)

WeightGiven

WeightedSub-Score

Student Progress 4 pts X 4 16

Access to Curriculum 2.5 pts X 4 10

Skill Generalization 2 pts X 3 6

Self-Determination 4 pts X 1 4

Supports 4 pts X 1 4

Calculate Content Area Raw Score:

16 + 10 + 6 + 4 + 4 = 40

27

Raw Score To Achievement Level Conversion Chart


If - Total Raw Score for the content area is:

Then -Achievement Level for the content area is:

47-52 Level 4: Proficient with Distinction

38-46 Level 3: Proficient

29-37 Level 2: Partially Proficient

13-28 Level 1: Substantially Below Proficient

28

Questions about

NH-Alternate Reporting?

Questions and AnswersNew Hampshire Alternate Assessment

29

• Student Report• Item Analysis Report• School/District Results Report• School/District Summary Report• District Student Level Data Files

NECAP: Types of Reports

30

The Family Educational Rights and Privacy Act (FERPA)

• Access to individual student results is restricted to:• the student• the student’s parents/guardians• authorized school personnel

• Superintendents and principals are responsible for maintaining the privacy and security of all student records.

• Authorized school personnel shall have access to the records of students to whom they are providing services when such access is required in the performance of their official duties.

• FERPA website: http://www.ed.gov/policy/gen/guid/fpco/ferpa/index.html

Family Educational Rights and Privacy Act

http://www.ed.gov/policy/gen/guid/fpco/ferpa/index.html

http://www.ed.gov/policy/gen/guid/fpco/ferpa/index.html

31

National Council on Measurement in Education

Educating Others

Evaluation and Research

Interpretation and Use

Scoring

Development

Marketing and Selling

Selection

Administration

www.ncme.org

http://www.ncme.org/

32

Student Report

• Two copies mailed to the school (February 7)• color copy for parents/guardians• black and white copy for local use

• Dual sided: Parent Letter & Student Report

• Spanish version of Parent Letter and Student Report shell available for NECAP(available at: http://www.ed.state.nh.us/NECAP)

33

34

35

36

37

Questions about the Student Report?

Questions and Answers

38

• Rationale• Inclusion of students in “Teaching

Year” reports

Teaching School vs. Testing School

39


Schools Can View Reports for Testing Year (2006-07)

40


Or Teaching Year(2005-06)

41

Testing Year

Teaching Year

42

http://iservices.measuredprogress.org

43

New thisyear

Please note:

You must always log in through this page to view confidential

reports and data

44

45

46

47

Item Analysis Report

• Available at the school level via password protected website (User ID and passwords sent to Principals and Superintendents in January)

• One report for each content area/grade level combination

• Issued in both Testing Year and Teaching Year formats

• Use in conjunction with Released Items documents

48

49

50

51

Released Item Documents

52

Released Item Documents

• New information provided in the Released Item Support Materials - documents include:

• For all content areas, providing the multiple-choice items and the GLE code and text that each item is linked to

• For math, indicating which items were included in session 1 of the math test by adding a “no calculator” symbol above the items

• For reading, listing the passage name and type of text• For writing, listing passage name and mode of writing

53

Questions about the Item Analysis Report or the Released Item documents?


54

• Available on-line (January 30) athttp://reporting.measuredprogress.org/NECAPpublicNH/

• Grade Level Report• One for each grade in each school and district• Issued in both testing year and teaching year

formats

• Summary Reports• One for each school or district• Issued in both testing year and teaching year

formats

School/District Results

55

56

Page 1: Title PagePage 2: Grade Level Summary ReportPage 3: Reading ResultsPage 4: Disaggregated Reading ResultsPage 5: Mathematics ResultsPage 6: Disaggregated Mathematics ResultsPage 7: Writing Results (Grades 5 and 8 only)Page 8: Disaggregated Writing Results (ditto)

Grade Level Results Report

57

58

Questions about the Grade LevelSummary Report page?


59

• Pages 3, 5, and 7 of the NECAP School/District Results Report

• Provides current and historical summative information about achievement in tested content areas

• Provides information about achievement in subtopics of tested content areas

Content Area Results

60

Content Area Results - Historical Data

• Data is included for two years and is combined in the Cumulative Total row

• In the Cumulative Total row for the achievement level columns:• The N columns are summed• The % columns are found by dividing the total

number of students in the level by the total number of students tested

• The Mean Scaled Score column is a weighted average

61

62

Content Area Results – Subscore Results

• Results based on both common matrix equating items

• Many Reading items are included twice – once under Type of Text, and once under Level of Comprehension

• Standard Error Bars are included to help determine if differences between scores are statistically significant

63

64

Comparisons of NECAP Scores Across Years

• School and District-Level Scaled Scores and Achievement Levels

• Student-Level Scaled Scores and Achievement Levels

• Content Area Subscores

65

Number of Scaled Score Points Denoting Minimally Statistically Significant Difference for Average Group Results*

Grade SubjectNumber of Students Tested in Group (Class, School etc.)

10 25 50 100 200

3Reading 6 4 3 2 1

Mathematics 5 3 2 2 1

4Reading 6 4 2 2 1


5

Reading 5 3 2 2 1


Writing 7 4 3 2 2

6Reading 6 4 3 2 1


7Reading 6 4 3 2 1


8

Reading 6 4 3 2 1


Writing 6 4 3 2 1

*Standard error of the mean difference with assumption n1=n2 and s1=s2

66

Percentage Difference in Student Achievement Level Classification Denoting Minimally Statistically Significant

Differences for Group Results*

Percentages of Students in Achievement Level(s)

Number of Students Tested in Group (Class, School etc.)

10 25 50 100 200

10 9 6 4 3 2

20 13 8 6 4 3

30 14 9 6 5 3

40 15 10 7 5 3

50 16 10 7 5 4

60 15 10 7 5 3

70 14 9 6 5 3

80 13 8 6 4 3

90 9 6 4 3 2

*0.05 level of statistical significance

67

Questions About the Content AreaResults Page?


68

• Pages 4, 6, and 8 of the NECAP School/District Results Report

• Provides information about achievement for different groups in a school/district

• Testing year reports have detailed information about all students enrolled in October 2006

• Teaching year reports include only those students with NECAP scores from Oct 2006.

• Detailed information for districts and the state may be found in the district and state reports.

• CSV files containing all this data will be posted to the NHDoE website in mid February.

Disaggregated Content Area Results

69

70

Questions about the Disaggregated Content Area Results page?


71

• Separate report for each school and district• Provides summative information for each

grade level within the school or district• Provides summative achievement level data

for whole school or district• Provided in both teaching year and testing

year formats• CSV files containing all this data will be

posted to the NHDoE website in mid February.

School/District Summary Report

72

Identifying the Report

• The first line designates the year the test was administered

• The second line is the name of the report• The third line differentiates between

“teaching year” and “testing year”

73

74

Questions about School/District Summary Reports?


75

Student Level Data Files

• CSV files available at the district level via password protected website (User ID and passwords sent to Superintendents in January)

• All demographic information for each student that was provided by the state

• Provided in both teaching year and testing year formats• The scaled score, achievement level, and subscores earned

by each student in all content areas tested• Also contain:

• Performance on released items• Student questionnaire responses• Optional reports fields data

76

77

Questions about Student Level Data Files?


78

Supporting Materials

Materials available now• Guide to Using the

2006 NECAP Reports

• 2006 Reporting Workshop Ppt

• Grade Level Expectations

• Released Item documents

• GLE Support Materials

Materials available soon• Roster of school and district

results (teaching year and testing year formats) (csv)

• Disaggregated Data for school & districts by grade level (teaching year and testing year formats) (csv)

• Student work samples without scores or annotations (pdf)

• Release Item distracter stats (pdf)

• 2006 Technical Manual