Welcome to the Spring 2017 Post-Test Administration DTC ... · Welcome to the Spring 2017 Post-Test...

Welcome to the Spring 2017

Post-Test Administration DTC Meeting

Agenda

Post Administration Review 9:00 AM – 12:00 PM

Testing window overview

What went well?

Opportunities for improvement

Testing Materials

Computer-based testing

Next Steps

CCRA Preliminary Results

District Advisory Committee Meeting 1:00 PM – 3:00 PM

State Department of Education

Dr. Jeanene Barnett, Deputy Superintendent of Assessment &

Accountability

Craig Walker, Executive Director of State Assessments

Angela Bilyeu, Assistant Executive Director of State Assessments

Maria Harris, Director of Assessment and Data Literacy

Sarah Owens, Mathematics Assessment Specialist

Elizabeth Warren, ELA & Social Studies Assessment Specialist

Cora James, Science Assessment Specialist

Rebecca Logan, Executive Director of NAEP & International

Assessments

Kurt Bernhardt, Executive Director of Education Technology

Measured Progress and eMetric

Measured Progress

Tammy Bullock, Director Client Services

Julie DiBona, Portfolio Manager, Oklahoma

Sharman Lyons, Senior Program Coordinator

eMetric

William Kinnison, Project Manager

General Discussion

What went well? What are our

opportunities for

improvement?

What was new this year?

Manuals - Redesign

MetaMetrics Linking Study – Embedded Items

Spanish Audio Files

Breach vs. Equivalent (EQ) Forms

Materials Return Process Poster

All materials being returned together

Automatic UPS pickup days

New eMetric features

Test Materials

Service Desk Calls

Initial Orders

Manuals

Test Administration

Return Materials

Total calls (4/3/17 – 4/28/17) – 3,738

Service Desk Calls

Top 5 Call Categories1. Student

Enrollment2. Online Testing3. Shipping4. Content5. Technical issues

Initial Orders

High schools that showed enrollments for

elementary school test materials

Online testing schools that showed orders

for paper materials

Missing students in precode led to fewer

PSTGs than expected

Manuals

New look and feel

Bookmarks were great for electronic devices

Challenges with combining sample items;

especially for grade 3

Missing information

Header names

Explanation of the “Form” bubble

Implications that CBT section two had sample

items

Test Administration

Repeated lettering of answer choices

Items for the linking study looked out of place

Calculator concerns

Breach forms available to be assigned at the

beginning of the test window

Returning materials

The process is going more smoothly this year!

Fewer district RS labels were issued this year

Common return issues or errors:

Photocopies of Class ID Sheets

Mixed grades in scored material envelopes

Security forms from different schools mixed into one district box

Oklahoma School Testing Program Online Assessments

OSTP Online AssessmentsOnline Testing Summary

Students Scheduled for Test

Tests CompletedTotal Tests Completed Spring 2017: 493,050Total Tests Completed Spring 2016: 699,347

Total Number of Students Scheduled 2017: 231,855Total Number of Students Scheduled 2016: 299,801

OSTP Online AssessmentsGrades 6-8 Online Testing Counts - Operational

OSTP Online AssessmentsGrades 6-8 Online Testing Counts - Breach

Grade 10/History Online Testing Counts - Operational

OSTP Online Assessments

*2016 includes Spring English II, English III, Algebra I, Algebra II, Geometry and Biology

Grade 10/History Online Testing Counts - Breach

OSTP Online Assessments

*2016 includes Spring English II, English III, Algebra I, Algebra II, Geometry and Biology

OSTP Online AssessmentsWeek 1

Week 2

Fri Thurs Wed Tues Mon

20,000

20,000

30,00030,000

OSTP Online Assessments

10,000

30,000 30,000

12,500

Fri Thurs Wed Tues Mon

Week 3

Week 4

OSTP Online AssessmentsOperating Systems Used

20162017

• Site Readiness Testing

• 1898 Orgs

• 809 Participated

• 482 Certified

• 0 Minutes Datacenter Downtime

OSTP Online AssessmentsRisk Mitigation

OSTP Online AssessmentsNew Features for Spring 2017

• Bulk Upload of Student Accommodations• Export Students by Test Completion Status• Dashboard• Bulk Add and Edit Users• Deactivate/Reactivate User Accounts• Bulk Class Creation• Test Administrator Technology Guide• Add DOB and Middle Initial to Test Tickets• iOS Automatic Assessment Configuration• Default to Operational Program• Bandwidth Test Tool Tip• Students Sorted by Last Name

DTC Request!

DTC Request!

DTC Request!

DTC Request!

DTC Request!

DTC Request!

OSTP Online AssessmentsBulk Upload of Student Accommodations

OSTP Online AssessmentsExport Students by Test Completion Status

OSTP Online AssessmentsDashboard

OSTP Online Assessments

Deactivate/Reactivate User AccountsBulk Add and Edit Users

OSTP Online AssessmentsBulk Class Creation

OSTP Online AssessmentsKnown Issues and Proposed Resolutions

Component Issue Resolution

Kiosk Slow download Under investigation

Kiosk Notes and bookmarks not saved on pause

Investigating options

Kiosk Shading on items difficult to see Support guides, content, hardware

Kiosk Installing & launching ChromebookApplication in kiosk mode

Training, online tutorial

Portal Grade vs course level class Investigating options

Portal Student transfers in the middle of a test

Investigating options

Reporting Key DatesGrade 3 RSA reports 7/19/17

Final reporting 10/11/17

I don’t understand, why is reporting so late this year?What is standards setting and why does this delay reporting?

Standard Setting

Newly developed Oklahoma Academic Standards

were instructionally implemented in 2016

necessitating the need for assessment standard

settings.

The primary goal of the standard setting is to

determine the knowledge, skills, and abilities

(KSAs) that are necessary for students to

demonstrate in order to be classified into each of

the performance levels.

Standard Setting

Standard setting for Grade 3 RSA will take place in June, followed by a larger standard setting for OSTP Mathematics, ELA and Science in August. A total of 11 panels for the 18 grade/content combinations with 11 members each will be convened

Upon completion of the standard setting meeting, Measured Progress will submit a report to the SDE that documents all aspects of the standard setting process. Includes: Process information

Cut scores

Reporting data implications

Panelist evaluation information

Cut Score Approval

Panelist recommendations will be presented to the

Commission on Educational Quality &

Accountability (CEQA) for final consideration.

SDE is planning to send a letter to schools

(separate from the reports) and develop other

tools to explain to parents the new level of

expectations for mastering our state standards

and new performance expectations on the

statewide annual assessments.

CCRA Preliminary Results

CCRA Program 2017

Preliminary Data as of 5/10/2017

Preliminary Participation Counts• Estimated CCRA Counts:

o Total # Juniors (11th): approx. 44,919

o Minus 1% (449): 44,470

o Answer Documents Scored: 35,181

o Percentage that participated as of

5/10/2017: 79%

• Increased by 3% from SY 15-16

• Make-up Testers (not included in

preliminary reporting)

o SAT – 283 students

o ACT – 1200 students

• Total Answer Documents

Scored as of May 10, 2017:o 35,181

• 17, 287 – Male

• 17, 667 – Female

• 227 – Unknown (ACT only)

Preliminary Participation Counts• Total Answer Documents

Scored as of May 10, 2017:o 33,753

• 16,597 – Male

• 16,929 – Female

• 227 – Unknown

• Estimated ACT Counts:o Total # Juniors (11th): approx. 42,614

o Minus 1% (449): 42,188

o Answer Documents Scored: 33,753

o Percentage that participated as of

5/10/2017: 79%

50%49%

1%

Totals

Males Females Unknown

Preliminary Results –ACT Composite Scores

• 3 out of

33,753

received a

36 (perfect

score)

• Average

Composite

Score: 18.9o SY 15-16: 18.9

0

500

1000

1500

2000

2500

3000

5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35

Average Scores

per Content

Total Possible Points per Section:

36

Avg. English Section Score:

SY 16-17 17.8

SY 15-16 17.6

Avg. Math Section Score:

SY 16-17 18.5

SY 15-16 18.7

Avg. Reading Section Score:

SY 16-17 19.7

SY 15-16 19.6

Avg. Science Section Score:

SY 16-17 19

SY 15-16 18.9

Benchmark Scores*

• These college courses

include:o English Composition,

o College Algebra,

o Introductory Social Science

courses, and

o Biology

• Scores on The ACT ®

subject-area tests that

represent the level of

achievement required

for students to have a o 50% chance of obtaining a “B”

or higher or

o about a 75% chance of

obtaining a “C” or higher in

corresponding credit-bearing

first-year college courses

*Based on a sample of 214 institutions and more than 230,000 students from across the United States, the Benchmarks are median course placement values for these institutions and as such represent a typical set of expectations.

College & Career Readiness Benchmarks

• The ACT ® English

Subtest Benchmark: 18

• College Course:

English Composition

• Met or Above

Benchmark: 46%

• Below Benchmark: 54%

• SY 15-16o Met or Above: 48%

o Below: 56%

46%54%

Students

Met or Above Benchmark Below Benchmark

College & Career Readiness Benchmarks

• The ACT ® Mathematics

Subtest Benchmark: 22

• College Course:

College Algebra

• Met or Above

Benchmark: 25%

• Below Benchmark: 75%

• SY 15-16o Met or Above: 23%

o Below: 77%

25%

75%

Students

Met or Above Benchmark Below Benchmark

College & Career Readiness Benchmarks

37%

63%

Students

Met or Above Benchmark Below Benchmark

• The ACT ® Reading

Subtest Benchmark: 22

• College Course: Social

Sciences

• Met or Above

Benchmark: 37%

• Below Benchmark: 63%

• SY 15-16o Met or Above: 35%

o Below: 65%

College & Career Readiness Benchmarks

24%

76%

Students

Met or Above Benchmark Below Benchmark

• The ACT ® Science

Subtest Benchmark: 23

• College Course:

Biology

• Met or Above

Benchmark: 24%

• Below Benchmark: 76%

• SY 15-16o Met or Above: 23%

o Below: 77%

College & Career Readiness Benchmarks

• The ACT ® STEM

Benchmark: 23

• Met or Above

Benchmark: 8%

• Below Benchmark: 92%

8%

92%

Students

Met or Above Benchmark Below Benchmark

Preliminary Participation Counts• Total Answer Documents

Scored as of May 10, 2017:o 1,428

• 690 – Male

• 738 – Female

• Estimated SAT Counts:o Total # Juniors (11th): approx. 2,305

o Minus 1% (449): 2,282

o Answer Documents Scored: 1,428

o Percentage that participated as of

5/10/2017: 63%

48%52%

Totals

Males Females

Preliminary Results –SAT Total Scores

• Average

Composite

Score: 938

• Scale:

400-1600

0

10

20

30

40

50

60

400

540

610

650

690

730

770

810

850

890

930

970

1010

1050

1090

1130

1170

1210

1250

1290

1330

1370

1410

1450

1490

Chart Title

Series 1

Average Scores per Content

Score Range per Section

200 - 800

Avg. Score: Evidence-based Reading & Writing

480.4

Avg. Score: Math 457.6

Score Range per Test

10 – 40

Avg. Score:Reading

24.4

Avg. Score: Writing & Language Arts

23.7

Avg. Score: Math 22.9

College & Career Readiness Benchmarks Defined

• Students are considered college- and career- ready

when their SAT section scores meet both the Math

and the Evidence-Based Reading and Writing

(EBRW) benchmarks.

• Students with an SAT score that meets or exceeds

the benchmark have a 75% chance of earning at

least a C in first-semester, credit-bearing college

courses.o SAT Math Section: Algebra, Statistics, Pre-Calculus, or Calculus Classes

o SAT EBRW Section: History, Literature, Social Sciences, or Writing Classes

*

• How Color-Coded Score Ranges Are Computedo The values for green test and sub-score score ranges are based on the

average test scores and sub-scores of all test-takers who met or exceeded

the corresponding benchmark for their grade.

o Yellow and red score range values are based on scores of test-takers who

did not meet the corresponding benchmark for their previous grade.

• Their average test scores and sub-scores determine the highest score in

the red range.

• Students who score above this average and below the green range fall

in the yellow range.

College & Career Readiness Benchmarks Indicators

Green The section score meets or exceeds the benchmark.

Yellow The section score is within one year’s academic growth of the benchmark.

Red The section score is below the benchmark by more than one year’s academic growth.

College & Career Readiness Benchmarks

• SAT Evidence Based

Reading & Writing

Section Benchmark:

480

• College Course: History,

Literature, Social

Sciences, or Writing

Classes

• Met or Above

Benchmark: 45%

• Below Benchmark: 55%

45%55%

Students

Met or Above Benchmark Below Benchmark

College & Career Readiness Benchmark Indicators

• SAT Evidence Based

Reading & Writing

Section Benchmark

Indicators

• Green: 45%

• Yellow: 8%

• Red: 47%

45%

8%

47%

Students

Green Yellow Red

College & Career Readiness Benchmarks

• SAT Mathematics

Section Benchmark:

530

• College Course:

Algebra, Statistics, Pre-

Calculus, or Calculus

Classes

• Met or Above

Benchmark: 23%

• Below Benchmark: 77%

23%

77%

Students

Met or Above Benchmark Below Benchmark

College & Career Readiness Benchmark Indicators

• SAT Mathematics

Section Benchmark

Indicators

• Green: 30%

• Yellow: 8%

• Red: 62%

30%

8%62%

Students

Green Yellow Red

11th Grade Benchmarks

• SAT Evidence Based

Reading & Writing

Section Benchmark:

460

• College Course: History,

Literature, Social

Sciences, or Writing

Classes

• Met or Above

Benchmark: 53%

• Below Benchmark: 47%

53%47%

Students

Met or Above Benchmark Below Benchmark

11th Grade Benchmark Indicators

• SAT Evidence Based

Reading & Writing

Section Benchmark

Indicators

• Green: 53%

• Yellow: 13%

• Red: 34%53%

13%

34%

Students

Green Yellow Red

• SAT Mathematics

Section Benchmark:

510

• College Course:

Algebra, Statistics, Pre-

Calculus, or Calculus

Classes

• Met or Above

Benchmark: 29%

• Below Benchmark: 71%

29%

71%

Students

Met or Above Benchmark Below Benchmark

11th Grade Benchmarks

11th Grade Benchmark Indicators

• SAT Mathematics

Section Benchmark

Indicators

• Green: 29%

• Yellow: 10%

• Red: 61%

29%

10%61%

Students

Green Yellow Red

CCRA Contacts

Tony Cortez

ACT Account Executive

tony.cortez@act.org

918-527-9948

Melany Franklin

Director

State & District Partnerships

mfranklin@collegeboard.org

Office:512-721-1822

Cell: 512-667-0151

2017-2018 DRAFT Testing Calendar

It’s all about

student learning.

Period.

We appreciate the opportunity to serve you

and the students of Oklahoma!