PARCC Research Framework ConsiderationsPARCC Research Framework ConsiderationsPARCC Research Framework ConsiderationsPARCC Research Framework ConsiderationsValidityValidity

Presented at: CCSSO National Conference on Student Assessment 2014New Orleans LA

Presenter : Arthur Thacker, HumRRO

June 26, 2014

Human Resources Research Organization (Human Resources Research Organization (HumRROHumRRO))66 Canal Center Plaza, Suite 700 Alexandria, Virginia 2231466 Canal Center Plaza, Suite 700 Alexandria, Virginia 22314--1578 | Phone: 703.549.3611 | Fax: 703.549.9661 | 1578 | Phone: 703.549.3611 | Fax: 703.549.9661 | www.humrro.orgwww.humrro.org66 Canal Center Plaza, Suite 700 Alexandria, Virginia 22314-1578 | Phone: 703.549.3611 | Fax: 703.549.9661 | www.humrro.org

New Orleans, LA

First, Some Terminology

Theory of Action (TOA) indicates intended uses and expected impact of the assessment system.

Theory of ActionTheory of Action

The TOA can inform testable claims related to theThe TOA can inform testable claims related to the interpretation of test scores.

2

Terminology (Continued)

These testable claims represent the interpretive argument.

TOA

Interpretive Argument

Every use or interpretation of an assessment score relies on meeting specific claims and themeeting specific claims and the various assumptions that justify them.

3

Terminology (Continued)

The evidence supporting those assumptions represents the validity argument.

TOA

Th fTheory of

Interpretive Validity ArgumentArgument Argument

4

Organization of This Presentation

Arranged in seven stages

1. Design2 D l t2. Development3. Implementation4. Administration5. Scoringg6. Reporting7. Use

Each stage includes interim goals toward end goal:Each stage includes interim goals toward end goal:Scores provide an indication that students are, or are not, college and career ready (CCR), or on track to b CCR f t d tbecome CCR for younger students

5

Stage 1: Design

Test design claims, assumption, and evidence should be developed and documented to demonstrate effective test design. Eleven claims comprise the design phase:

1. Connects to CCSS2. Determines Whether Students Are CCR or On Track to be CCR3. Indicates How Academic Achievement in U.S. Compares with Other p

Countries4. Measures Full Range of Student Performance, Including High- and Low-

achieving Students5 Provides Data for Accountability Including Measures of Growth5. Provides Data for Accountability, Including Measures of Growth

6

Stage 1: Design (Continued)

6. Incorporates Innovative Approaches, Including Technology, That Are Interactive and Engaging

7. Provides Multiple Measures of Student Achievement8 I l d R f It T8. Includes Range of Item Types9. Captures Critical-thinking, Problem-solving, and Communication Skills10. Accommodations are Appropriate for SWD and EL Students11. Assessments are Accessible to SWD

Now, let’s work through one of these claims in more detail.

7

Claim 1: Connects to CCSS

8

Claim 1: Connects to CCSS

Assumptions Evidence TimelineTest items link to CCSS Item to Standards

Assumptions Evidence TimelineTest items link to CCSS content

Item-to-Standards Alignment Study Results

Summer 2014

Test Items are written to assess the depth and Item Development

Development Procedures C l A il bl

Test items link to CCSS content

Item-to-Standards Alignment Study Results

Summer 2014

Test Items are written to It D l t

Development Procedures assess the depth and breadth described by the CCSS

Item Development Procedures and Item-to Content Maps

Currently AvailableItem-to-Content Maps Updated Annually

assess the depth and breadth described by the CCSS

Item Development Procedures and item-to content maps

e e op e t ocedu esCurrently AvailableItem-to-Content Maps Updated Annually

Correlational Evidence with other CCSS Measures (state tests)

Summer 2014 (Field Test) Updated Annually

p y

Correlational Evidence with other CCSS Measures (state tests)

Summer 2014 (Field Test) Updated Annually

Test Blueprints-to-CCSS standards comparisons

Summer 2014Measures (state tests) Updated AnnuallyTest Blueprints-to-CCSS standards comparisons

Summer 2014

9

Claim 1: Connects to CCSS

10

Stage 2: Development

● Three claims under this stage relate to how the development of the PARCC assessments function to help PARCC achieve its interim goal of developing tests in aPARCC achieve its interim goal of developing tests in a rigorous and valid manner:1. Subject Matter, Presentation, and Language Use is Free of Potential Bias

and is Acceptable to Students Parents and Other Community Membersand is Acceptable to Students, Parents, and Other Community Members2. Items are of Sufficient Quality and Rigor3. Schools Have the Resources and Infrastructure to Implement and

Administer the Assessments as Intended

11

Stage 3: Implementation

Three claims are related to having the necessary infrastructure in place to implement the tests as intended:1. Communication plan clearly conveys to stakeholders the policies and p y y p

practices essential for effective implementation of assessment system2. Students understand the format of the assessments and how to use the

technology3. Test Vendors, Departments of Education, School Districts, and Schools

are Coordinated in the Assessment Process

Includes aspects of the assessment such as ensuring that test proctors (teachers) are trained to administer the test using standardized procedures and that students can use the technology required to answer test questions.

Effective implementation of a testing system is a prerequisite for ensuring that tests are appropriately administered and that they eventually generate test scores that are valid for their intended uses.

12

y g

Stage 4: Administration

● Four claims undergird the operational administration of the PARCC assessments. The claims under this stage relate to administering the assessments as they are intended—that is,

ith fidelitwith fidelity. 1. Administrators and teachers are prepared to administer the assessments

as intended2 Teachers have the resources and supports to administer the assessments2. Teachers have the resources and supports to administer the assessments3. Technology improves and facilitates the assessment experience4. Students respond to items as intended

A i t t t d i i t ti i i it f i th t● Appropriate test administration is a prerequisite for ensuring that the data collected from students can be accurately scored and reported, and that the scores can eventually be used in valid waysways.

13

Stage 5: Scoring

● Five claims under this stage relate to ensuring that students’ scores on the tests are accurate and meaningful. 1. Rubric is diagnostic1. Rubric is diagnostic2. Scores accurately and reliably reflect student achievement on the

assessed content3. Growth model provides accurate information on student growthp g4. Scores from multiple PARCC assessments provide both unique and

complementary information. 5. Cut score is indicative of college- and career-readiness

● Accurate scoring of test data is a prerequisite for ensuring that score reports are useful and that the scores can be pinterpreted in valid ways.

14

Stage 6: Reporting

● The single claim under this stage relates to ensuring that score reports are clear and provide useful information:1. Score reports are clear and easily understood by stakeholders1. Score reports are clear and easily understood by stakeholders

● Clear and useful reporting of students’ test results is a prerequisite for ensuring the information provided in theprerequisite for ensuring the information provided in the score reports can be used in valid ways.

15

Stage 7: Use

● The “use” stage of the TOA deals primarily with impact and utility of the test scores, beyond their validity as measures of the intended constructs.

● These claims and their underlying assumptions are vital to ensuring that PARCC succeed and they should be investigated routinely to determine if the assessments areinvestigated routinely to determine if the assessments are having the intended impact on students, teachers, schools, and other stakeholders.

● A series of impact studies informed by the ongoing● A series of impact studies, informed by the ongoing collection of validity evidence, should be undertaken as part of the PARCC research agenda. See the Program Evaluation Memorandum!Evaluation Memorandum!

16

Creation of an Annually Updated Validity Argument Document

• The evidence gathered from the studies and documentation referenced in this memorandum must be gathered into a single cohesive narrative.

• This validity argument should present the evidence and summarize the extent to which it represents support for the various associated

tiassumptions.

• If the support for an assumption is weak or the evidence inconclusive, then that aspect of the testing system must be improved or additionalthen that aspect of the testing system must be improved or additional evidence gathered.

• An annual update to the validity argument is recommended toAn annual update to the validity argument is recommended to document the continuing collection of validity evidence for the PARCC assessments.

17

Outline for Validity Argument DocumentationV lid ti h d1. Validation research agendaa. Compile and evaluate major findings from completed researchb. Indicate critical additional validity evidence needed and provide a timeline for

collecting that evidencec. Indicate evidence that will be collected during routine procedures that should

be included in the validity argumentd. Indicate which topics will require periodic validation and recommend a

schedulee. Indicate supplemental or additional research to be completed as needed or as

lower priority studiesf. Recommend procedures for reacting to unanticipated challenges to PARCC

validityR ti /C i ti2. Reporting/Communicationa. Revise and expand validity argument figureb. Recommend branching system for reporting (via website and for technical

reports)3. Create a provision for external verification of results

18