1

Regents Scanning Pilot –CFN 201

Amy Reynolds, Assessment Liaison Damarine Reyes, Data/IT Liaison

22

Agenda

> Overview of new Regents data requirements & DOE’s plan to meet these requirements

> Scanning Process Key points for each step in the process Method

> Next Steps: Using Regents data to inform instructional decisions ATS reports overview Activity

> Discussion / Questions > Support

3

Regents Exam Data New York State Department of Education: New Requirement: Beginning in June 2011, NYSED will require that ALL Districts submit item-level data for select Regents exams. This includes the following subjects:

• English Comprehensive (not included for January pilot)• Integrated Algebra• Earth Science• Living Environment• US History & Government• Global Studies & Geography

Impact: Student responses need to be scanned to efficiently capture the required item-level data without impacting the turnaround time for results required by the schools.

Note:

•Teachers must still score constructed response portions of exams

•Other Regents exams and RCTs will still use the traditional scoring process

4

DOE Data Capture Solution To leverage existing technology and processes currently in use in schools to capture item-level information & provide real-time results. This includes:

• Answer Documents: Generated & printed at school-level• Scoring: Teachers to score Constructed Response items only• Scanning: Using schools’ attendance scanners, locally scan documents on-site• Reporting Results: Automated transfer from ATS to STARS

Benefits:• Increased automation of time-consuming manual processes

• Less overall time required to complete scoring by teachers• No further data entry requirements of aggregate scores

• Reduction in clerical scoring errors• eliminate manual scoring of MC items• adding totals from different sections• application of conversion tables

• Districts/schools/teachers will be able to better utilize item-level results to help inform instruction, similar to Grade 3-8 exams

STARS ATSAttendance

Scanner

ATSSTARS

(and beyond)

1. Schedule students for Regents Exams in STARS

2. Generate & Print answer documents for each scheduled student

3. Answer documents printed for each scheduled student

6. Scan answer documents for upload to ATS

7. ATS receives data, applies scoring key & conversion chart for scale score instantly.

8. ATS passes data nightly to STARS for Report Cards and Transcripts and for reporting to NYSED

Exam Day

Scoring

4. Student uses pre-slugged answer document

5. Scoring committee marks Open-Ended items and bubbles in score on answer document.

Attendance Scanner

Re-scanning (if needed)

Scanning Process Overview

5

STARS – (Review. Nothing has changed)

Schedule students for Regents exam in STARS• Who is responsible for STARS scheduling?

• Program chair of each school usually handles STARS scheduling process

• STARS scheduling process remains exactly the same as before

• Major Consideration: Should I schedule one section or multiple sections for each Regents exam?• One section:

• Shorter scheduling process • Use “Master Schedule” function

• Multiple Sections:• Allows you to break students up into certain groups, assign each group

to a certain room, print attendance rosters by exam room, and print exam invitation reports

• Use “Master Schedule Builder” functionFor support with using STARS, contact your network’s Data/IT person

6

ATSPrint answer documents from ATS to attendance scanner

• Who is responsible for ATS printing?• Pupil accounting secretary

• Print answer documents up to a week before Regents week

• There are three printing options: Regents Scanning Handbook / ATS Screenshots, page 20

1. “Print scheduled students not previously printed”- these students have been scheduled in STARS

Print all answer sheets that have not been printed yet2. “Print by student ID”

Print an answer sheet for any student one at a time. This option could be for another public school student taking the exam at your school

3. “Print by name” Print an answer sheet for a non-public school student who is taking the exam at

your school

For support with using ATS, contact your network’s Data/IT person.

7

Log in to ATS.

On the screen that appears below type: REGT

8

ATS Screenshots

TAB to RGPR and press RIGHT/ CTRL/RED ENTER

9

ATS Screenshots

Exam DayUse printed answer documents for test taking & scoring

• Students use pre-slugged answer documents on the day of the exam to answer multiple choice questions

• Students and Teachers need to use PENCIL for marking responses on answer documents• Responses are only to be marked in the new answer document

• Proctors ensure that student answer documents are signed and bubbled in appropriately.

• Scoring team bubbles in their scores for the open-ended questions on the teacher portion of the answer document • Teacher portion for all January Regents exams is page 2, except for the

one-page Integrated Algebra form

10

Attendance scanner Scan answer documents for upload to ATS (p.27 Handbook)

• Do not scan until you’ve received notice from the DOE

• Use your school’s attendance scanner

• Scan the answer documents face up & bar code to the left (bottom first)

• Scan both pages for 2-page answer documents • Batch page 1 documents separately from page 2 documents

• Scanner will print a report of how many documents were scanned successfully; for those not scanned successfully the document will be reprinted with an error message on it

• Rescan answer sheets if necessary• Audit trail captured of re-scans and changes to the item responses

• Scanning must be complete by the end of the Regents window

11

For support with using the attendance scanner, contact your network’s Data/IT person or call the DIIT Helpdesk at (718) 935-5100

ATS

ATS processes scanned data and makes it available for use

• ATS receives data, applies scoring key and conversion chart for scale score instantly

• If both pages of a 2-page answer document are not scanned, scale score will not be displayed (Incomplete Record)

• ATS passes data nightly to STARS for Reports Cards and Transcripts, and for reporting to NYSED, etc. 

12

Accessing Results

ATS Reports Available for Printing

• Document Status Report• Ensure that answer documents have been properly scanned.

• Item Report• Analyze the Regents results for individual students and inform

decisions on how to move these particular students forward.• Review error report – multiples and omissions

• Item Distribution Report• Analyze the Regents results for the student group as a whole in

order to inform instructional decisions for these students as well as future students.

13

Document Status ReportAllows you to ensure that answer documents have been properly

scanned.

14

Student Name and ID

Raw and Scale Score

Scan Date and Time

# of Answer Sheets Generated

# of Answer Sheets Scanned Errors Warnings

Item ReportAllows you to analyze the Regents results for individual students and

inform decisions on how to move these particular students forward.

15

Student Name

Student ID #Raw and Scale score

• Question number• Answer key • Student answer choices• Open-ended question

scores

Dash = correct answer

Item Distribution ReportAllows you to analyze the Regents results for the student group as a whole

in order to inform instructional decisions for these students as well as future students.

16

Question Number

For each answer choice, number and percentage of students that selected it

Total number of responses for each question

Data Analysis Activity

Examine sample data using worksheets…

Distribution Report

Item Report

Reflect on…

Uses for the reports

Next Steps

Share…

17

Using data to inform instructional decisions

ITEM REPORT

• Questions this report could help you to answer• Which students scored the highest?• Which students scored the lowest?• Which students will have to retake this Regents exam?• Which questions did individual students have trouble with?• Are there any patterns in terms of what skills these questions focus on?

• Next Steps• Plan instructional interventions for students who will need to retake the

Regents• Support students who passed the Regents with further mastery of the areas

that they had trouble with**Keep in mind that you will need to have a copy of the corresponding

Regents exam to see the actual questions which correspond to the data.

18

Quality Assurance – Data Validation

• 5% manual verification• Schools need to manually verify the scores of 5% of the

answer documents for each exam have been scored properly.

• Identify two people at school level perform the data verification

• CFN Assessment Liaison will collect signed document from each pilot school• Document will indicate that the school has performed the 5%

manual verification

Review Scanning Process & Regents Scanning Implementation Checklist

Pre-Exam - Regents Scanning Implementation Checklist

• STARS – Schedule students for Exams

• Supplies: Paper, toner

• Printing answer documents

Exam Day - Regents Scanning Implementation Checklist

• Personnel available to printout extra answer sheets if necessary

• Students use PENCIL on the answer documents

Post-Exam - Regents Scanning Implementation Checklist

• Complete score / open-ended questions.

• Wait for official notice from DOE to begin Scanning

• Accessing reports

• Quality Assurance – Complete Data Validation Documentation

20

21

Ongoing Support for Assessment Liaisons

Technical Support•Scheduling, ATS and STARS

> Contact your Data/IT Liaison•Scanner and printer

> Contact the DIIT helpdesk at (718) 935-5100

Overall Process Support•Contact your Borough Assessment Implementation Director (AID)

•Queens – Barbara Marcisak

Training Documents

•Access documents from today’s training in ARIS Connect’s Summative Assessment Implementation Community: http://tinyurl.com/regents-scanning