Cornelia Orr, Executive DirectorNational Assessment Governing Board

12th Grade NAEP Preparedness Research Plans and Progress

The Board’s Work on The Board’s Work on Preparedness Has Been Preparedness Has Been

Underway for Nearly a DecadeUnderway for Nearly a Decade

2002

12th Grade Commission Established

12th Grade Commission Established

2005

Board Resolution on Reporting12th Grade PreparednessBoard Resolution on Reporting12th Grade Preparedness

2006

Grade 12 Reading& Mathematics Frameworks Revised

Grade 12 Reading& Mathematics Frameworks Revised

2009

Board Approves Preparedness Research PlanBoard Approves Preparedness Research Plan

Conduct Research Studies and Develop Preparedness ReportConduct Research Studies and Develop Preparedness Report

2010-2012

The Governing Board has demonstrated a rigorous, comprehensive, and methodical approach

to research and has not presumed the equivalence of college and career preparedness. This

research will help inform the definition of preparedness in terms of NAEP.

2July 26, 2011 Cornelia Orr, NAGB

Key Considerations for the Governing Board

Statements in NAEP reports must be shaped and supported by validity evidence (validity framework)

Recognize NAEP has limitations

Reporting on NAEP 12th grade preparedness may or may not prove to be feasible

Consider additional Grade 12 academic areas in the future

3July 26, 2011 Cornelia Orr, NAGB

Purpose: Technical Panel on 12th Grade Preparedness Research

Identify technically appropriate and valid ways to use grade 12 NAEP for reporting preparedness

Assist in planning research and validity studies

Recommend research designs and priorities for implementation in 2008, 2009, and thereafter

Focus on 12th grade reading and mathematics

4July 26, 2011 Cornelia Orr, NAGB

Technical Panel on 12th Grade Preparedness Research: Members

Michael Kirst (Chair)Stanford University

John CampbellUniversity of Minnesota

David T. ConleyUniversity of Oregon

Michael KaneNational Conference of

Bar Examiners

Mark David MillironCatalyze Learning International

Robert MislevyUniversity of Maryland

George C. Thornton, IIIColorado State University

5July 26, 2011 Cornelia Orr, NAGB

Panel’s Key Conclusions

Reporting on 12th grade preparedness seems feasible

Use multipronged approach to collect validity information

Avoid representing 12th grade NAEP as the single authoritative definition or conception of preparedness

Consider the national context related to preparedness

6July 26, 2011 Cornelia Orr, NAGB

Preparedness: Working Definitions

Preparedness for College: reading and mathematics knowledge and skills to qualify for placement into entry level college credit courses that meet general education requirements, without the need for remedial coursework in math or reading

Preparedness for Workplace Training: reading and mathematics knowledge and skills needed to qualify for a job training program without remediation in math or reading

“Preparedness” versus “Readiness”

7July 26, 2011 Cornelia Orr, NAGB

Types of Research Studies

Content alignment

Judgmental standard-setting

Statistical relationship

Survey of Postsecondary Institutions

Benchmarking Studies

8July 26, 2011 Cornelia Orr, NAGB

500

391

367

336

327

313

297

286

277

256

243

0

346

302

265

Statistical LinkingStudies

JudgmentalStudies

(Expert Panels)

Study Interrelationships

Higher EducationSurvey

Content AlignmentStudies

9July 26, 2011 Cornelia Orr, NAGB

Data for Preparedness Research

2009 NAEP 12th grade Reading and MathematicsNational and state level data 11 states voluntarily participated

Linkages to state data systems (only 1 in 2009)Advance planningConfidentiality assurance processesFollow-up for post secondary college/career status

Planning for 2013

10July 26, 2011 Cornelia Orr, NAGB

Grade 12 NAEPResults for 11 states

11July 26, 2011 Cornelia Orr, NAGB

Research Studies

Preparedness Study Status

Content Comparisons (SAT, ACT, Accuplacer, Work Keys)

Complete

Judgmental Standard Setting Underway

Statistical Relationships /Linking(SAT, Accuplacer, ACT)

Underway

Survey of Higher Ed Institutions Underway

Benchmarking Planned

12July 26, 2011 Cornelia Orr, NAGB

500

391

367

336

327

313

297

286

277

256

243

0

346

302

265

Reference Points on the NAEP Scale (Scenario 1)

College Preparedness Standards: A

College Preparedness Standards: B

Exemplar Job 1

Exemplar Job 2

Exemplar Job 3

Exemplar Job 4

Exemplar Job 5Placement in entry-level credit-bearing course

Advanced

Proficient

Basic

Below Basic

July 26, 2011 13Cornelia Orr, NAGB

500

391

367

336

327

313

297

286

277

256

243

0

346

302

265

Reference Points on the NAEP Scale (Scenario 2)

Exemplar Job 1

Exemplar Job 2

Exemplar Job 3

Exemplar Job 4

Exemplar Job 5

College Preparedness Standards: A

College Preparedness Standards: B

Placement in entry-level credit-bearing course

July 26, 2011 14Cornelia Orr, NAGB

Reference Points on the NAEP Scale (Scenario 3)

500

391

367

336

327

313

297

286

277

256

243

0

346

302

265

College Preparedness Standards: B

College Preparedness Standards: A

Placement in entry-level credit-bearing course

Exemplar Job 4

Exemplar Job 1

Exemplar Job 5

Exemplar Job 3

Exemplar Job 2

July 26, 2011 15Cornelia Orr, NAGB

Next Steps

Develop summary report of validity evidenceValidity Framework – external reviews

Develop preparedness reportsBased on national data analyses

Research-focused report

Public-focused report

Issue Preparedness Report in 201216July 26, 2011 Cornelia Orr, NAGB

Questions and Discussion

What are the results from the preparedness research so far?

The content of NAEP is similar to the content assessed on common indicators of college preparedness, that is, the ACT, SAT, and ACCUPLACER.

Some content on NAEP is similar to WorkKeys, though the content on NAEP is generally broader than the content on WorkKeys and WorkKeys measures some content that is not in NAEP.

The linking of NAEP to SAT: NAEP proficient is about at 500 on the SAT for both reading and mathematics, the college board readiness benchmark.

How can states use the NAEP preparedness research?

As a resource for state and/or consortia standard setting

To provide reference points for state readiness/preparedness definitions

17July 26, 2011 Cornelia Orr, NAGB

Other Questions?

Dr. Cornelia S. Orr

Executive Director

National Assessment governing Board

800 North Capitol Street NW, Suite 825

Washington, DC 20002-4233

www.nagb.org

18July 26, 2011 Cornelia Orr, NAGB