1

June 2012 Regents ScanningTraining for Charter School Assessment Coordinators

Version 3.01 – May 2012

22

ObjectivesThe purpose of today’s session is to educate you on the Regents scanning procedures, and empower you to turn-key the process at the school level.After the training, you will understand:

> The overall scanning process to be performed by schools, and step-by-step procedures for each stage;

> How schools can avoid common pitfalls in the scanning process by using best practices;

> Process updates for the June 2012 administration; and> What documents and resources are available for training and

support, including use of the demo IT environment to train schools

33

Process Changes for June 2012 Regents Scanning> Key differences for June 2012 Regents Exams include:

Schools must now scan all 10 Regents examinations. Exams that will be scanned for the first time include Algebra 2/Trig, Chemistry, Geometry, and Physics.

A scan confirmation page will now be automatically generated after every batch of answer documents is scanned to report the scanning status of all answer documents and inform any necessary follow-up.

Use of “did not meet lab requirements” is now available for all 4 science exams

No final score will be calculated until an answer document is fully scored and scanned; this now includes one page answer documents scanned with missing teacher scored responses

44

SED Regents RequirementsBeginning in June 2012, the SED requires all districts to collect and submit item-level data for all ten Regents exams.

*RCTs will continue to use the traditional administration and scoring process and will not be scanned.

> Beginning in June 2012, all ten Regents exams will be scanned, including Algebra 2/Trig, Geometry, Chemistry, and Physics.*

Pre-Planning STARS ATS &

Scanner

ATS

1. Schools receive training and begin preparations for Regents Scanning process.

2. Schedule students for Regents and produce exam invitations in STARS.

3. Schools test scanner and generate and print pre-slugged answer docs in advance.

6. Schools scan answer docs for upload to ATS.

7. ATS instantly applies scoring key & conversion chart for scale score. Results available immediately.

Exam Week & Scoring

5. Schools administer exams. Answer documents for walk-ins are printed on-demand. Scoring committee marks Constructive Response items and bubbles in scores on answer documents.

Attendance Scanner

Re-Scanning (Only As Necessary)

Regents Scanning from A-Z: Process Overview

5

Logistics Planning

4. Schools conduct Regents week planning activities for logistics, including assignment of roles, and performing checklist items in the week before exams are administered.

6a. Re-scan docs.

66

Timeline of Key StepsDates School Activity

Now – June General Preparation

Now – June Clean/test scanner, and ensure adequate supplies (toner, paper, etc.).

5/29 – 6/12 Schools begin to print answer documents in ATS

6/13 – 6/21 Regents administration

6/13 – 6/25 Regents scanning window

77

Step 3: Printing Answer Docs in ATS> There are three main options for printing answer docs:

1. RGPS (Batch printing answer documents)Batch printing option for schools who are unable to register students in STARS. This is the preferred option for charter school printing.

2. RGPR Print by Student ID Prints an answer sheet for any student, one at a time. (Use SBIO

screen to look up student ID.) (RGPR)3. RGRR Reprint Answer Documents

Reprint previously printed docs by student ID or answer-document page number. (RGRR)

88

Step 3: Printing Answer Docs in ATS (RGPS)> Use RGPS screen for batch printing option.

99

Step 3: Printing Answer Docs in ATS (RGPR)

Sorting Options

> Use RGPR screen for printing individual student answer documents.

1010

Step 3: Printing Answer Docs in ATS (RGRR)> Use RGRR screen for reprinting individual student answer

document.

Sorting Options

1111

ATS-Generated Answer Documents Regents ScannableOld Answer Document

> Schools will no longer receive answer sheets from NYSED for Regents exams

1212

Step 3: Printing Answer Docs in ATS> Best practices for printing answer documents

Whenever possible, print answer documents using the ATS printer/scanner you intend to use for scanning. If this is not possible, print answer docs only on supported printers (Lexmark T640 or later), not older or incompatible printers

After printing batches of answer documents, a “test page” will print. Scan the test page to confirm the ink quality for a batch. Trailing test pages are specific to each exam to better diagnose scanning issues ahead of testing period. Make sure to scan the test pages and with enough time to reprint answer document as necessary.

Answer documents must be printed on white paper only. Use the same stock paper as is normally used for attendance forms. Using colored paper or heavy stock may result in scanning errors.

Ensure only one answer document is printed per student per exam to avoid possible data errors.

**If a school is having trouble printing its answer documents, it should contact Bert Wyman at Bwyman@schools.nyc.gov.

1313

Step 4: Logistics Planning for Regents Week> Main tasks:

Distribute test rosters and page one of answer docs to appropriate staff in advance.

Train exam proctors on new procedures and policies. Ensure adequate supply of toner, paper, pens/pencils.

> Best practices: Roles and responsibilities are clear and communicated to all

relevant staff before Regents Week. Ensure the scanner is clean; a dirty scanner may result in

inaccurate results. See the Tips for Scanning appendix of the Regents Scanning Handbook here (Password June2012) for instructions on cleaning.

Arrange for servicing of scanner(s) as necessary prior to scanning. For assistance, contact the helpdesk at 718-935-5100.

1414

Step 5: Exam Week & Scoring> Main tasks:

Proctors emphasize proper technique - • Completely fill in bubbles and erase stray marks; • Remind students to sign student declaration when finished; and• Ensure accommodations are properly accounted for on the

answer sheet> Best practices:

Students use pencil for MC, pen for essays and DBQ questions, and pencil or pen for student attestation.

Teachers carefully fill in bubbles (using only pencil) when scoring Constructive Response questions.

Ensure no teacher scored items are left blank and teacher names are filled in on the answer sheet.

1515

Step 5: Exam Week & Scoring> Administering Exams to English Language Learners:

Schools should ensure ELL accommodations are correctly recordedBubble ELL Accommodation Eligibility

11 Time Extension Schools may extend the test time for English language learners. Principals may use any reasonable extensions, such as “time and a half” (the required testing time plus half that amount), in accordance with their best judgment about the needs of the English language learners.

12 Separate Location Schools are encouraged to provide optimal testing environments and facilities for all students. They may administer Regents Exams to ELLs individually or in small groups in a separate location.

13 Bilingual Dictionaries and Glossaries

ELLs may use bilingual dictionaries and glossaries when taking Regents Exams in all subjects. The bilingual dictionaries and glossaries may provide only direct translations of words. Bilingual dictionaries or glossaries that provide definitions or explanations are not permitted.

14 Simultaneous Use of English and Alternative Language Editions

For Regents Exams for which the Department provides written translations, ELLs may use both an English and an alternative language edition of the test simultaneously. However, they should be instructed to record all of their responses in only one of the two editions. The alternative language edition used by the student should be so indicated on the student’s answer sheet.

15 Oral Translation for Lower Incidence Languages

Schools may provide English language learners with an oral translation of a Regents Exam when there is no translated edition provided by the Department. This accommodation is permitted for State exams in all subjects except English. All translations must be oral, direct translations of the English editions. No clarifications or explanations may be provided.

16 Writing Responses in the Native Language

ELLs making use of alternative language editions or of oral translations of Regents Exams may write their responses to the open-ended questions in their native language. Scoring the tests is the responsibility of the school.

18 Reading of Listening Selection

Proctors may read the listening passage of the Regents Comprehensive Examination in English a third time to English language learners.

1616

Step 5: Exam Week & Scoring> SED Rescoring Policy:

As of June 2011, schools are no longer permitted to rescore any of the open-ended questions on any Regents exam after each question has been rated the required number of times (as specified in the rating guide), regardless of the final exam score.

Specifically, exams receiving a final score of 60-64 or 50-54 are not permitted to be re-scored.

Principals and other administrative staff in a school or district do not have the authority to set aside the scores arrived at by the teacher scoring committee and rescore student exams.

1717

Step 5: Exam Week & Scoring> Teacher scoring of their own students’ exams:

In June 2012 exams, teachers can still score their own students’ exams, although NYSED requests principals should make every effort to prevent teachers from rating their own students’ responses, where possible.

Under guidance from the New York State Education Department, starting in the 2012-13 school year, teachers will no longer be able to score their own students’ responses.

1818

Step 6 and 6a: Scanning Answer Documents> Main tasks:

After notification from the DOE, you may use the attendance scanner to begin scanning answer documents. Documents scanned before notification will not be processed and will have to be rescanned.

Documents with errors in reading bar codes will be reprinted by the scanner with an error message. Documents with bubble read errors will be indicated on confirmation sheet(s) and may be investigated in ATS. Re-scan affected documents as necessary.

Review REDS reports for “warnings” noted to inform of scan read errors, including multiple or omitted responses, or stray marks.

Review the automatically generated SCNS report, which provides information on scanning status and whether ATS successfully received scanned data.

In order to preserve answer sheet quality for audit purposes, answer documents should be scanned only once except if there are missing or incorrect teacher scores on the document, or if the “did not meet lab requirements” bubble is filled.

1919

Step 6 and 6a: Scanning Answer Documents> “Did not meet lab requirements” bubble:

Addition of a “Did not meet lab requirements” bubble on all science exam answer documents, including Chemistry and Physics.

This designation is used only for students who sat (but were ineligible) for either exam, and should be used only on completed and scanned documents. Un-schedule students who did not sit for the exam in STARS and do not scan these documents.

If this bubble is filled when scanned, it will remove the score; no grade will appear on report cards or transcripts. The score will not be reported to the State.

2020

Step 6 and 6a: Scanning Answer DocumentsCode Use Case Method of Entry Effect on exam

recordAbsent (ABS) Student is absent Scanned answer

document or RGTU Score is ABS

Did not meet lab requirements

Student mistakenly allowed to sit for exam; exam was mistakenly scanned and a score generated.

Re-scanned answer document

Score not reported on transcript or report card (data sent to SIF)

Invalidated (INV)

Confirmed case of student cheating

Scanned answer document (with principal initials)

Score of INV

Mis-administration(MIS)

School error (ex. missing accommodations, document was created for the wrong student)

RGTU Score not reported on transcript or report card

Do not scan completely blank (unused) answer documents

2121

Step 6 and 6a: Scanning Best Practices> The scanner will automatically print a confirmation page after every

scanned batch indicating pages with scanning errors, and will reprint pages for which the bar code could not be read.

> If only one page of a two-page document is scanned, it will register in ATS as an incomplete scan until the second page is processed. No score will be calculated until both pages of a two-page document or if all teacher responses are blank on a one-page answer document.

> Schools may scan the first page of two-page documents immediately, before scoring is complete for page two.

> When scanning two-page documents, it is recommended to batch all page one documents and scan separately from page two documents for faster scanning and upload.

> File the scanned batches together with the DFS confirmation page. This will assist in retrieving papers as needed in the future.

> Scanning is open from 6 a.m. – 9 p.m. on weekdays, and from 8:30 a.m. – 4 p.m. on weekends. Please allow time for the whole batch to process.

2222

Step 6 and 6a: Scanned Images> Scanning captures

an item-level data string as well as images of answer documents (and any ancillary re-scans) for an audit trail.

> Central users have access to scanned image capture.

2323

Step 7: ATS Reports: REDS (Status Report)> REDS reports (available immediately after scanning):

Use Document Status Report to ensure that all answer docs have been properly scanned and to flag warnings.

Use Item Report to see students’ scores of individual students. Review report for “warnings/multiples/omits” by using “Include only W/M/O” option.

Use Item Distribution Report to inform decisions on present and future instruction.

> REDS report changes:• Scores of Misadministration, Did not meet lab requirements,

Exam invalidated, and Absent will all override scaled score from responses.• Score of “INC” will be shown when only one page (of a two-

page answer doc) has been scanned or when all teacher score responses on a one-page answer document are unfilled.

> Given the length (and width) of the REDS report, it is best viewed when printed, as all data may not be visible on the ATS screen. Schools may download ATS reports into Excel for further analysis.

2424

Step 7: ATS Reports: REDS (Status Report)> Report uses:

Displays score, accommodations, warnings, for each student. Shows summary of documents scanned, outstanding, and warnings. Use this report to track outstanding unscanned and incompletely

scanned document.

Accom.Scores Scan-specific info

Document statuses

2525

Step 7: ATS Reports: REDS (Item Report)> Report uses:

Displays line-item data for each student. Shows specific questions to review (warnings). Use to flag Teacher section questions to investigate (e.g. omits, multiples).

TEACHER SECTIONSTUDENT SECTION

Correct answers

Multiple answersIncorrect answers CR raw scores

2626

Step 7: ATS Reports: SCNS (Scan Statistics Report)> Report uses:

This report is automatically printed, but can be run separately for an exam. This report provides information on scanning status including the number of pages scanned, and whether the data was accepted or rejected by ATS.

The total number of answer documents scanned, with and without errors.

2727

Step 7: ATS Reports: RADI (Item Dist. Report)> Report uses:

Shows school-level aggregated data for each question. Use data to flag specific subject matter and question types for instruction.

‘*’ denotes correct answer

2828

Step 7: ATS Reports: RGTU (Update Regents Answers)> The RGTU screen is used to update student and teacher responses not

accurately picked up by the scanner, including incomplete erasures, selections lightly bubbled in, or stray marks. Process:

• After a school confirms a scanner error, the school will use RGTU to update individual student or teacher section responses.

• Changes made in RGTU that do not cause a record to move from a failing score to a passing score will be processed automatically (e.g. an 80 82 or 49 52). » Requested changes that cause a record to move from a failing score to a

passing score will be held for review and approval by DAPS, and will cause the record to be locked until the document status is resolved.

• If a school attempts to rescan a locked record, the scan sheet will be successfully scanned and sent to ATS—but ATS will reject the record. The error will appear on the SCNS report, and the record will not be updated in ATS

• Principals will be able to review the status of requests in ATS. Most requests will be reviewed within 48 hours unless further information is needed.

Appeal process• Principals (and Data/IT Liaisons) will be directly notified via email if a change

requested has been denied, or if further information is needed to justify the change. Assessment and/or Data/IT Liaisons will work with DAPS to clarify any such requests.

2929

> Best practices for use of the RGTU screen: Do not attempt to use the RGTU screen on an incompletely scanned

answer document. Use RGTU for scanning errors, such as stray marks or lightly bubbled

responses. Do not erase or re-bubble and then rescan any student responses.

Under no circumstances may teachers or administrators mark, erase, or otherwise modify any part of the student section of the answer document.

If a teacher fails to properly record a score in the Teacher section, then the RGTU screen should be used to correct the error. If a Teacher section answer is omitted, the answer document should be bubbled in with the correct response and then rescanned.

In the event a document cannot be scanned, schools will need to recreate the student answer document and scan (and retain the original) in order to access the RGTU screen.

Schools may mark an exam record as misadministered on this screen. Each school must retain a written explanation of each change in RGTU for

audit and review purposes.

Step 7: ATS Reports: RGTU (Update Regents Answers)

3030

> Note corrections to Student and Teacher sections are made separately in the RGTU screen

Step 7: ATS Reports: RGTU (Update Regents Answers)

31

Step 7: ATS Reports: RGTU (Update Regents Answers)

> Example change to student section:

32

Step 7: ATS Reports: RGTU (Update Regents Answers)

> Example change to student section: Note that the response to Question 10 registers as “M” (multiple).

33

Step 7: ATS Reports: RGTU (Update Regents Answers)

> The online edit changes the student’s response to Question 10 from “Multiple” to “1”, leading to a new raw and final score.

3434

> Example Change to Teacher SectionStep 7: ATS Reports: RGTU (Update Regents Answers)

3535

> Change to Teacher Section: Note that Question 34 has been left blankStep 7: ATS Reports: RGTU (Update Regents Answers)

3636

> Change to Teacher Section: Question 34 is now filled in, and the raw and final scores have changed.

Step 7: ATS Reports: RGTU (Update Regents Answers)

3737

> If an online edit results in a “critical change” (i.e., a change in score below 65 to a score of 65 or more) DAPS will review the requested online edit in conjunction with the scanned image(s), and make a determination either to accept or reject the edit.

Step 7: ATS Reports: RGTU (Update Regents Answers)

3838

> Schools can view the status of their requested online edits that have been held for DAPS review in the RGDR screen in ATS:

Step 7: ATS Reports: RGDR (Approval Status)

3939

> Schools can view the status of their requested online edits that have been held for DAPS review in the RGDR screen in ATS:

Step 7: ATS Reports: RGDR (Approval Status)

> To identify the status of all RGTU requests at your school, you can you can simply press F2 twice.

4040

> Schools should check RGDR during Regents scanning period to observe the status of pending online edit requests

Step 7: ATS Reports: RGDR (Approval Status)

4141

Step 7: Score Verification> Main tasks:

Although optional, schools should verify a sample of scanned results for accuracy - approximately 5% of answer documents.

Schools should look in REDS reports for students with incomplete scores (INC) as well as students with a score of zero, which may result from scanning a blank answer document.

> Best practices: Verify scan data is correct prior to close of scan window; data

errors identified after this time can only be corrected centrally.

4242

State Testing Calendar

4343

On-Going Support & Resources

> Documentation and updates: Regents Scanning Handbook (Click Here Password June2012)

> Technical support (ATS): Bert Wyman CSO: Bwyman@schools.nyc.gov

> Scanners and printers: DIIT Help Desk, 718-935-5100

> Overall process support: Borough Assessment Implementation Directors (BAID)

The following resources are available to support implementation and execution of Regents scanning.

4444

Thanks for your participation.