Integrated Document Analysis Workshop

Integrated Document Analysis Workshop

Peter Worth, Assessment Standards & Development Services, WestEd, CAJohn L. Smithson, Director, Measures of the Enacted Curriculum

Wisconsin Center for Education Research, University of Wisconsin-Madison

Washington, DC May 5-7, 2008

Describing the Content & Language Demands of

Standards, Assessments & Student Work

The intended curriculum: State content standards—What students should learn

The enacted curriculum: What teachers teach

The learned curriculum: Student outcomes based on school learning

The assessed curriculum: State (and other) assessments—tested learning

Analyzing Alignment

Linquistic Complexity by Language Demands

A neutral linguistic taxonomy

A neutral content language

Topics by Cognitive Demand

Content and Language Coding Form

Focus: (A = Academic/S = Social) Language Demand: (A-P, Z) Density: (L = Low/H= High)

Mode: Rec. = Receptive (LI=Listening; RD= Reading) Construction: (S=Simple/C=Complex)

Pro. = Productive (SP=Speaking, WR=Writing) Overall Complexity: (1= lowest, 4=highest;

X= Complexity cannot be determined)

Content Codes

Language Codes

Strand ID

TPC1

CGD1

Focus

(A/S)

Mode Language Demand

(A-P, Z)

Density

(L/H)

Construction

(S/C)

Overall Complexity

(1-4, X)

Comments

Rec. Pro.

1

2

3

4

5

6

Content and Language Coding

Grade 8 mathematics item

Content and Language Coding Form

Focus: (A = Academic/S = Social) Language Demand: (A-P, Z) Density: (L = Low/H= High)

Mode: Rec. = Receptive (LI=Listening; RD= Reading) Construction: (S=Simple/C=Complex)

Pro. = Productive (SP=Speaking, WR=Writing) Overall Complexity: (1= lowest, 4=highest;

X= Complexity cannot be determined)

Content Codes

Language Codes

Strand ID

TPC 1

CGD1

Focus

(A/S)

Mode Language Demand

(A-P, Z)

Density

(L/H)

Construction

(S/C)

Overall Complexity

(1-4, X)

Comments

Rec. Pro.

1 16 506 E A RD D L C 2

2 204 C E

3 K

4

5

6

K-12 Mathematics Content

K-12 Mathematics ContentBasic Algebra

The Goal

To render quantitative descriptions of instruction, standards, and assessments using a common language in order to facilitate comparisons and analyses of the three domains of a standards-based approach to education reform and their relationship to one another.

To Facilitate Comparisons

Academic Content

ELL Lang. Acquisition

Academic Content

ELL Lang. Acquisition

Academic Content

ELL Lang. Acquisition

The Two Dimensions Of Content

What students should know[Topics]

And…

Be Able to Do [Expectations for student performance]

Describing Instructional Content

SEC utilizes a two-dimensional taxonomy based on:Topic

by Cognitive Demand

1

2

3

4

5

B C D E F

Coarse Categories of Cognitive Demand

Content Areas

Memorize Perform Procedures

Demonstrate Understanding

Conjecture, Hypothesize

Non-routine problems

Number Sense

Operations

Measure-ment

Basic Algebra

Advanced Algebra

Geometric Concepts

The Content Matrix

Fine Categories of Cognitive Demand

Topics Memorize Perform

Procedures Demonstrate

Understanding Conjecture, Hypothesize

Non-routine problems

Place Value

Whole Nbrs.

Fractions

Decimals

Percents

Ratio & Proportion

The Content Matrix

… adding levels of relative emphasis yields a 3-D construct

Coarse Categories of Cognitive Demand

Content Areas

Memorize Perform Procedures

Demonstrate Understanding

Conjecture, Hypothesize

Non-routine problems

Number Sense

Operations

Measure-ment

Basic Algebra

Advanced Algebra

Geometric Concepts

Describing the Cognitive Domain

3

4

5

6

How Many Categories?

Bloom’s

SEC

DOK (Webb)

SCASS Science

Dimensions of Knowing & Inquiry

Acquire Use Extend

(From: Dimensions of Knowing and Inquiring about Science, State Collaborative on Assessments & Student StandardsScience Project, Council of Chief State School Officers, 1997)

Depth of Knowledge

* Webb, N. 1999. Alignment of Science and Mathematics Standards in Four State. NISE Research Monograph #18. Madison:Wisconsin Center for Education Research.

Skill/Concept

Recall

StrategicThinking

ExtendedThinking

Level

1

2

3

4

Skill/ConceptRecallStrategicThinking

ExtendedThinking

Acquire Use Extend

Exploring Cognitive Demand

Cognitive Demand (or Expectations for Student Performance)

Skill/ConceptRecallStrategicThinking

ExtendedThinking

Acquire Use Extend

MemorizePerform

ProceduresDemonstrate

Understanding

Conjecture,Generalize

Prove

Solve non-routine/ makeconnections

MemorizeConduct

InvestigationsCommunicateUnderstanding

AnalyzeInformation

Apply concepts/make

connections

RecallPerform

ProceduresAnalyze/

InvestigateEvaluate

Generate/Demonstrate

Cognitive Demand (or Expectations for Student Performance)

Skill/ConceptRecallStrategicThinking

ExtendedThinking

Acquire Use Extend

MemorizePerform

ProceduresDemonstrate

UnderstandingAnalyze

InformationEvaluate/Apply

Recall Understanding Application Analyze Evaluate Create

Practice Session #1

ExploringCognitive Demand

But first….

Let’s take a 10 minute break!

Exploring Cognitive Demand

CgD Immersion Activity• Organize into Groups/Tables• Each Table w/ CgD Pie• Each Person w/ Cgd Descriptors

Step 1: Place CgD cards on Pie Slices face-down

Step 2: Turn cards over: ID agreements e.g. 2 cards w/ same descriptor in same sliceif Group Agrees ... discuss key wordsif not … Discuss … operational definition to distinguish

Step 3: Discuss disagreementsif consensus reached put in envelope / if not, set aside

Content Analysis Materials

• Cognitive Demand List• Topics Lists• Comments & Suggestions Worksheet• Coding Forms• Documents to be analyzed

Cognitive Demand Lists

• Five categories of cognitive demand• Slightly different for each subject• Each category is defined by a list of descriptors• The list of descriptors are not exhaustive

• Each category stands on its own• Each category has an associated letter (B-F)

Dimension A: Content Topics

Topics List (In your packet of material)

Organized at two levels:Content Areas

(16 for Mathematics)(27 for Science)(14 for ELAR)

Topics (identified by number)(182 Mathematics Topics)(211 Science Topics)(114 ELAR Topics)

Plus: non-specific & other

Topics Lists

• Topics Lists• Mathematics• Science• English Language Arts & Reading

• Cover grades K-12• Organized into Content Areas• Topics & Content Areas have an associated #

Comments & Suggestion Worksheet

• One for each reviewer - more available• Use to:

• Record coding conventions/decision rules• Suggest/identify additional topics not listed• Suggest/identify additional CGD descriptors• Provide other comments & suggestions

• Be sure to turn in at end of workshop (and with mail-in materials, as necessary).

Coding Procedures

TPC CGDContent Descriptions arerecorded using pairs ofcodes to identifyparticular intersectionsTopic & CognitiveDemand Categories.

B200203

C

Cognitive Demand Categories arerepresented by letters B-F.

Topics are identifiedusing numbers.

D290

TPC CGD

Large Grain Topic Descriptions:To reference vague content descriptions thatcould cover most topics in a given contentarea, use the content area code (ends in 00)to indicate the relevant content area. (Thereare 7 content areas in K-8 Mathematics and25 content areas in K-8 Science.)

200Grain Size can berepresented using thefollowing conventions:

Coding Conventions

Should none of five categories of cognitivedemand listed on the Cognitive DemandCoding Sheet appear relevant, (e.g. thestandard statement makes no reference tocognitive demand, use the letter ‘Z’. Thisshould only occur rarely, if at all.

Non-SpecificCognitive Demand:

Coding Conventions

TPC CGD200 Z

Coding Conventions

Non-specific Content TPC CGD

0

Z

999 Z

Non-specific target subject 0

EOut of subject area

Cognitive Demand only

(*Please make note on your comments form.)

Non-specific Topics: TPC CGDB290

293 DOther: (rater-defined*)

Other: (topic not listed)

Coding Conventions

The Content Analysis Process

‘Coding’ Teams of 4-5 Content Experts

Independent Coding by each Analyst w/ Group Discussion

Goal for Process: Generalizability not Inter-rater Reliability

Should not be necessary to discuss every item – select by team

Pick-up and return documents / coding sheets to Alissa

Sign & return to Alissa non-disclosure forms

Practice Exercise #1 (Math)

Practice Exercise #2 (ELAR)

A neutral content language

Topics by Cognitive Demand

The intended curriculum: State content standards—What students should learn

The enacted curriculum: What teachers teach

The learned curriculum: Student outcomes based on school learning

The assessed curriculum: State (and other) assessments—tested learning

Content Analysis Workshop

Coding Forms• Assessment Coding Forms• Standards/Benchmark Coding Forms• Each is used to record content descriptions• Each content description consists of

• A topic number• A cognitive demand category letter

• The Integrated Analysis Coding Form• Combines Content & Linguistic Information

in a single data collection form

Rater: ________ Subject:__NC Math Gr 3 Benchmark Test (SF)_ Form: Test 1 Page of

Itm. Desig./Nbr. Itm. Desig./Nbr.TPC1 CGD1 TPC2 CGD2 TPC3 CGD3 TPC1 CGD1 TPC2 CGD2 TPC3 CGD3

1 40

2 41

3 42

4 43

5 44

6 45

7 46

8 47

9 48

10 49

11 50

12 51

13 52

14 53

15 54

16 55

17 56

18 57

19 58

20 59

21 60

22 61

23 62

24 63

25 64

26 65

27 66

28 67

29 68

30 69

31 70

32 71

33 72

34 73

35 74

36 75

37 76

38 77

39 78

Content Code 2 Content Code 3Content Code 1 Content Code 2 Content Code 3 Content Code 1

Assessment Coding Forms

Rater: ________ Document:__NAEP Grade 8____Itm. Desig./Nbr. Itm. Desig./Nbr.

Number PropertiesNumber Sense TPC1 CGD1 TPC2 CGD2 TPC3 CGD31a 4a

1b 4b

1d 4c

1e 4d

1f

1g 5a

1i 5b

1j 5c

Estimation TPC1 CGD1 TPC2 CGD2 TPC3 CGD3 5d2a

5e2b

5f2c

Measurement

2d Measuring Physical Attributes TPC1 CGD1 TPC2 CGD2 TPC3 CGD3

1bNumber Operations TPC1 CGD1 TPC2 CGD2 TPC3 CGD3

3a 1c

3d 1g

3e 1h

3f 1j

3g 1k

Properties of Number & Operations

TPC1 CGD3

CGD3CGD1 TPC2

CGD1 TPC2 CGD2 TPC3

CGD2 TPC3

Content Code 1 Content Code 2 Content Code 3 Content Code 1 Content Code 2 Content Code 3Ratios & Proportional Reasoning

TPC1

Standards Coding Forms

Standards Coding Forms

Nature of ScienceScience & Technology

Sci., Health, & Envrnmnt.Measurement in Science

Components of Living Sys.Biochemistry

BotanyAnimal Biology

Human BiologyGeneticsEvolution

Reproduction & Dev.Ecology

EnergyMotion & Force

ElectricityWavesKinetics & Equilibrium

Properties of MatterEarth Systems

AstronomyMeteorology

Elements & The Per. Sys.Chem. Form. & Reactions

Acids, Bases, & SaltsOrganic ChemistryNuclear Chemistry

Other:

All Content Areas

-0.018--0.008 -0.008-0.002 0.002-0.012 0.012-0.0220.022-0.032 0.032-0.042 0.042-0.052 0.052-0.0620.062-0.072 0.072-0.082 0.082-0.092

Mem

orize

Perform

Comm

unicate

Analyze

Apply

Nature of Science

Science & Technology

Sci., Health, & Envrnmnt.

Measurement in Science

Components of Living Sys.

Biochemistry

Botany

Animal Biology

Human Biology

Genetics

Evolution

Reproduction & Dev.

Ecology

Energy

Motion & Forces

Electricity

Waves

Kinetics & Equilibrium

Properties of Matter

Earth Systems

Astronomy

Meteorology

Elements & The Per. Sys.

Chem. Form. & Reactions

Acids, Bases, & Salts

Organic Chemistry

Nuclear Chemistry

Other:

All Content Areas

-0.018--0.008 -0.008-0.002 0.002-0.012 0.012-0.0220.022-0.032 0.032-0.042 0.042-0.052 0.052-0.0620.062-0.072 0.072-0.082 0.082-0.092

Gr. 5-8 National Science Standards