Andrew C. PorterVanderbilt University

Measuring the Content of Instruction:

Uses in Research and Practice

Tools

Teacher surveys of instructionDaily logsEnd-of-semester or end-of-year surveys

Content analyses ofStandardsTestsCurriculum materials

Alignment indices—e.g, alignment between assessment and standards

Categories of Cognitive Demand

TopicsMemorize Perform

ProceduresCommunicateUnderstanding

Solve Non-Routine

Problems

ConjectureGeneralize

ProveMultiple

StepEquations

Inequalities

LiteralEquations

Lines /Slope andIntercept

Operationson

Polynomials

QuadraticEquations

Content Matrix

Instruction

Vertical and Horizontal Alignment

Achievement

Assessment

Assessment Standards

StandardsDistrict

State

Example Matrices to Measure Alignment

.3 0 .10 .1 00 .2 .10 .1 .1

.2 0 .10 .2 0.1 .2 .10 0 .1

Assessment Standards

Topics

Cognitive Demand

Alignment Index =1 -X=Assessment Cell ProportionsY=Standards Cell Proportions

∑ |X-Y|

2

Alignment of Assessments with Standards7th-Grade Math:Goals Study

Standards Assessment States B D E F

B .37 .39 .37 .45 D .35 .37 .36 .40 E .36 .33 .43 .31 F .32 .35 .30 .41

NCTM .34 .40 .33 .47

Average Within-State Alignment = .40Average Between-State Alignment = .39Average State-Test-to-NCTM Alignment = .39

Alignment of Instruction with Assessment8th-Grade Math:SCASS Study

Instruction AssessmentState H J K L E O NAEP

H 0.35 0.22 0.19 0.28 0.21 0.04 0.38J 0.34 0.21 0.18 0.25 0.20 0.05 0.38K 0.42 0.28 0.21 0.29 0.25 0.05 0.39L 0.36 0.24 0.19 0.29 0.22 0.05 0.40E 0.39 0.26 0.17 0.26 0.24 0.04 0.38O 0.35 0.21 0.16 0.26 0.20 0.05 0.38

Average Within-State Alignment = .22Average Between-State Alignment = .23Average State-to-NAPE Alignment = .39

States H J K L E O G I M N

HJ .73

K .59 .66L .56 .64 .67E .65 .71 .78 .70

O .71 .80 .63 .65 .70G .71 .81 .66 .67 .71 .84I .73 .82 .63 .66 .68 .79 .80M .68 .77 .61 .62 .66 .73 .76 .79N .62 .69 .58 .61 .62 .71 .70 .67 .65

Average Alignment = .69

Alignment of Instruction with Instruction 8th-Grade Math: SCASS Study

7th Grade StandardsState E State F NCTM

Quality of Data Response rates Interrater agreement for content

analyses Validity of teacher self-report

Explaining between-teacher variance in alignment to NAEPPredicting student achievement gains

[Note: The need for a reform-neutral language]

Uses of Tools Describing Instructional Practices

ResearchServe as dependent variable in teacher decision-making researchDescribe the implemented curriculum Measure implementation of new curriculaAssess the validity of transcript studies

PracticeInform teacher reflections on their own instructional practices

[Note: Should not be used for teacher accountability.]

Uses of Tools Describing Instructional Materials

ResearchResearch effects of textbooks on instructionAssess the breadth and depth of content in instructional materials

PracticeBuild testsWrite content standardsDevelop national, state, or district indicator systems

Uses of Indices of Alignment Research

Serve as a control variableServe as a dependent variable Serve as a descriptive variable

PracticeAlign state tests to state standardsAlign instructional materials to standards or course frameworks

Increasing Validity and Value Getting the content languagecontent language right Using time samples to describe instruction for an entire

school year Replicating the finding that alignment predicts student

achievement gains Identifying contexts in which teacher self-reportteacher self-report on the

content of instruction is more or less accurate Improving the level and consistency of interrater interrater

agreementagreement in content analyses Understanding the distributional propertiesdistributional properties of the

alignment statistics Building powerful professional developmentpowerful professional development programs

for data-based decision making Developing a content language for reading

Conclusions Much progress has been made in recognizing the

importance of instructional content as a variable in education research.

Some progress has been made in building tools for including content in education research.

There have been several innovative uses of these new tools in both research and practice, and more are on the horizon.

But there is much more work to be done.

7th-Grade Standards--Close ViewState E State F NCTM

Number Sense and Numeration

7th-Grade Standards--Close ViewState E State F NCTM

Data Analysis and Probability

Response Rates for Survey

Eisenhower LongitudinalWave 1 75%Wave 2 74%Wave 3 75%

Eisenhower Cross-Sectional 72%Reform Up Close 75%

Interrater Agreement

Assessment Mean Range Goals Study .51 .77 to .34 CCSSO Study .47 .60 to .37

Standards Goals Study .48 .59 to .33

[Note: In each study, there was one outlier rater.]

Eisenhower Longitudinal Study

42% of variance explained by level (elementary, middle, high school) and subject

27% of variance explained by teachers in the same school

0% of variance explained by between school or between years

Longitudinal data on instruction alignment to NAEP yielded:

Alignment to Predict Achievement Gains

Index Cross-Sectional Gain

Longitudinal Gain

Level .336 (p=.02) .238 (p=.11) Level and Configuration .425 (p=.01) .341 (p=.07)

Level x Configuration .438 (p=.00) .304 (p=.04)

Cell Correlation .335 (p=.02) .240 (p=.11)

Total Configuration .312 (p=.04) .335 (p=.02)

Index Intercorrelations

L LXC CELL TOTAL

Level 1.0

Level x Configuration .69 1.0

Cell Correlation .62 .72 1.0

Total Configuration .77 .59 .86 1.0