CADRECOMMUNITY FOR ADVANCING

DISCOVERY RESEARCH IN EDUCATION (CADRE)

A Review of Research Instruments for STEM Teacher Practices, PCK, and

Content KnowledgePRESENTED BYDAPHNE MINNERALINA MARTINEZAbt Associates

JUNE 15, 2012DR-K12 PRINCIPAL INVESTIGATOR ANNUAL MEETING

ARLINGTON, VA

CADREDR K-12 Portfolio Overview

Purpose• Describe important characteristics• Provide evidence of progress toward goals• Identify areas for targeted studies• Inform CADRE’s support activities

Questions• What are the characteristics of projects? • What types of projects are being funded?• What research is being conducted?• Are projects evaluating their work?• How are projects going to disseminate their work?

June 15, 2012

CADRE

Data from 248 Projects

Cohort 4(71 Projects)

Cohort 1(75 Projects)

Cohort 3(51 Projects)

Cohort 2(51 Projects)

Includes projects that submitted at least a proposal

Sources: Provided by PIs

voluntarily in response to requests

Found in the public domain

June 15, 2012

CADRE

Project Materials were Reviewed and Coded

Project materials were reviewed and

coded

Coded information was recorded in a

database

Descriptions of DR K-12 portfolio characteristics

were compiled for the annual report

June 15, 2012

CADRE

RFP Guides Innovation in STEM Education

Grand Challenges (2006)

• K-12 math and science assessments• Elementary grade science• Cutting-edge STEM content in K-12

Challenge Strands (2008)

• Contextual challenges• Frontier challenges

Challenge Areas (2009, 2010, 2011)

• Assessments• Opportunity to learn for all students• Ability of teachers• Promising innovations• Cyber-enables learning (2011 only)

Strands (2012)

• Assessment• Learning• Teaching• Scale-up and sustainability

June 15, 2012

CADREDisciplines Addressed in Projects

34%

41%

19%

3%

2%

1%

MathematicsOther

Engineering

Multi-disciplinary

Sciences

Computer

June 15, 2012

CADRE

Target Segment of the Education Population

0%

20%

40%

60%

80%

100%

80%75%

13%5% 4%

Students

OtherHigher Education Faculty

Teachers

School Adminis-trators

June 15, 2012

CADRE

Outcomes Studied

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

66% 65%

14%10%

6% 5%

Implementation Fidelity

Administrators

TeachersStudents

Research Quality, Progress

Other & Unknown

June 15, 2012

CADRE

Types of Student and Teacher Outcomes Studied

Students

0%10%20%30%40%50%60%70%80%90%

100%

62%

32%

18%

Attitudes

Behavior

Teachers

0%10%20%30%40%50%60%70%80%90%

100%

49%

35% 34%

23%

Attitudes

Classroom Practices

Pedagogical Content

Knowledge

Content Knowledge

June 15, 2012

Achievement

CADRETeacher Instrument Study

Overview

• Rationale

• Methods

June 15, 2012

CADRE

75 projects assessed teacher outcomes with existing instruments

Number of Teacher Instruments Used

Number of Projects

1 35

2 21

3 9

4 65 4

June 15, 2012

Outcome Domain Number of Projects

Practice 32

PCK 14

PCK & Practice 8

Content 7Practice & Content 6PCK & Content 4All three 4

Table A Table B

CADRE

82 unique instruments identified

42 used to assess Practices Reformed Teaching Observation Protocol (modified)—15 studiesInside the Classroom Observation and Analytic Protocol—8Surveys of Enacted Curriculum (modified)—5

24 used to assess ContentViews of Nature of Science (Form C)—3 studiesNational Assessment of Educational Progress (modified)—3Praxis content tests/ Earth & physical science (modified)—2TIMSS content tests (modified)—2

June 15, 2012

CADRE82 instruments identified

22 used to assess Pedagogical Content Knowledge

Mathematical Knowledge for Teaching—14 studies Knowledge of Algebra for Teaching (modified)—2Science Teaching Efficacy Belief Instrument—2Reformed Teaching Observation Protocol (modified)—2

June 15, 2012

CADRE

More Refined Look at Constructs

• Instruction practices• Content knowledge• Instructional beliefs; teaching self-efficacy• Class management• Assessment practices/ skills• Classroom social context• Classroom physical context• Administrative or policy context• Student / teacher demographics• Instructional planning

June 15, 2012

CADRE

Instructional PracticesAFM Assessment of the Facilitation of Mathematizing obs PK-6 M

EMCO Early Mathematics Classroom Observation obs PK METAP EdTech Assessment Profile survey - Tech

ISCOP Instructional Strategies Classroom Observation Protocol obs mid SC

LFCPO Lesson Flow Classroom Observation Protocol obs mid SC

LoFTI Looking for Technology Integration obs - Tech

O-TOP OCEPT-Classroom Observation Protocol obs post SC, M

STIR Science Teacher Inquiry Rubric obs elem SC

TIDES Transforming Instruction by Design in Earth Science--Teacher assignment quality rubrics rubric - SC

Scoop Notebook Artifact rubric mid SC

The Quality of Instruction Measure rubric - M

June 15, 2012

CADREInstructional Practices plus

CETP-COPThe Collaboratives for Excellence in Teacher Preparation core evaluation classroom observation protocol

Physical context, demo obs, inter K-16 SC,M

EQUIP Electronic Quality of Inquiry Protocol demo obs K-12 SC,M

KAT Knowledge of Algebra for Teaching content survey, assessment Mid, high M

ICOT ISTE Classroom Observation Tool tech use obs K-12 Tech

MQI Mathematical Quality of Instruction content obs Elem, mid M

PRAXIS Praxis Teaching Foundations: Science Class mgmt, assessment survey Mid, high SC

PRISM Preschool Rating Instrument for Science and Mathematics Physical context obs PK SC,M

SESAME Self-Evaluation of Science and Math Education planning - PK SC,M

SIOP Sheltered Instruction Observation Protocol planning obs, rubric elem Gen

TIMSS Third International Mathematics and Science Video Study (TIMSS) social, demo obs, survey mid SC,M

Ohio Middle Level Mathematics and Science Education Bridging Study - Teacher Questionnaire

content, demo survey K-12 SC

June 15, 2012

CADRE

Teachers’ Content KnowledgeDTAMS Diagnostic Teacher Assessments in

Mathtematics and Science PCK test mid SC,M

DTAMS Diagnostic Mathematics Assessments for Elementary Teachers--algebra PCK test elem M

M-SCAN The Mathematics Scan instruction, assessment obs elem M

ACT American College Testing St. test post GENAP Anvanced Placement St. test high GEN

CST Content Specialty Test Earth Science for New York Teacher Certification test K-12 SC

FACETS Diagnoser Tools St. test Mid, high SC,M

FCI Force Concept Inventory assessment St. test High, post SC

GRE Graduate Record Exam test post GEN

ITBS Iowa Test of Basic Skills St. test Elem, mid GEN

MAP Missouri Assessment Program St. test Elem, mid SC,M

MKT Mathematical Knowledge for Teaching test Elem, mid M

June 15, 2012

CADRE

Teachers’ Content KnowledgeMOSART Misconceptions-Oriented Standards-Based Assessment Resources

for Teachers test K-12 SC

NAEP National Assessment of Educational Progress St. test K-12 GEN

PISA Program for International Student Assessment St. test high GEN

PRAXIS content tests/ Earth & physical science (modified) test post SC

Regents New York State Regents exam St. test K-12 GEN

SAT Stanford Achievement Test St. test high GEN

TAGLIT Taking a Good Look at Instructional Technology test post TECH

TIMSS content tests St. test Elem, mid SC,M

West-E Washington Educator Skills Test-Endorsements test K-12 GEN

WESTEST Science WESTEST St. test Gr. 4-9 SC

American Chemical Society Division of Chemical Education Examinations Institute test post SC

Classroom Test of Scientific Reasoning St. test - SC IL Certification Testing System Study Guide-Science: Biology test post SC

June 15, 2012

CADRE

Instructional BeliefsIMBS Indiana Mathematics Beliefs Scale survey - M

MTEBI Mathematics Teaching Efficacy Belief Instrument survey preservice M

PSI-T Principles of Scientific Inquiry-Teacher survey high SC

SETAKIST Self-Efficacy Teaching and Knowledge Instrument for Science Teachers survey elem SC

STEBI Science Teaching Efficacy Belief Instrument survey elem SC

TBI Teacher Belief Interview interview high SC

TSES Teachers' Sense of Efficacy Scale survey PK-6 GEN

TSI Teaching Science as Inquiry survey elem SC

VNOS-C Views of Nature of Science Form C survey - SC

June 15, 2012

CADRE

MultidimensionalSYSTEM-WIDE REFORM FOCUSED

Practices

Content

Beliefs

Management

Assessment

Social

Physical

Admin context

Demographics

planning

CIP Inside the Classroom Teacher Interview Protocol COP Inside the Classroom Observation and Analytic

Protocol CTRI Coaching /Teacher Reflection Impact Surveys FFT Danielson's Framework for Teaching Domains LSC LSC Core Evaluation Classroom Observation

Protocol SEC Surveys of Enacted Curriculum SII Study of Instructional Improvement

TIMSS-R TIMSS-R Science Teacher Questionnaire TIMSS-R TIMSS-R Mathematics Teacher Questionnaire

Inside the Classroom Teacher Questionnaire: Math or Science version

June 15, 2012

CADRE

System-wide Reform FocusedCIP Inside the Classroom Teacher Interview Protocol interview K-12 SC,M

COP Inside the Classroom Observation and Analytic Protocol obs K-12 SC,M

CTRI CoachingTeacher Reflection Impact Surveys survey K-8 M

FFT Danielson's Framework for Teaching Domains rubric K-12 GEN

LSC LSC Core Evaluation Classroom Observation Protocol obs K-12 SC,M

SEC Surveys of Enacted Curriculum survey K-12 SC,M,LIT

SII Study of Instructional Improvement survey elem M, LIT

TIMSS-R TIMSS-R Science Teacher Questionnaire survey elem, mid SC,M

TIMSS-R TIMSS-R Mathematics Teacher Questionnaire survey elem, mid SC,M

Inside the Classroom Teacher Questionnaire: Math or Science version survey K-12 SC,M

June 15, 2012

CADRE

June 15, 2012

DIOLOGUE FOCUSEDPractices

Content

Beliefs

Management

Assessment

Social

Physical

Admin context

Demographics

planning

CLASS The Classroom Assessment Scoring System CLO Classroom Lesson Observation Instrument COEMET Classroom Observation of Early Mathematics

Environment and Teaching DAISI The Dialogic Activity in Science Instruction EAS The Emergent Academic Snapshot ELLCO The Early Language and Literacty Classroom

Observation IQA Instructional Quality Assessment ISIOP Inquiring into Science Instruction Observation

Protocol RTOP Reformed Teaching Observation Protocol SPC Standards performance continuum Classroom Snapshot Mathematics Classroom Observation Protocol Science Classroom Observation Guide (NCOSP)

CADRE

Dialogue FocusedCLASS The Classroom Assessment Scoring System obs K-12 GEN

CLO Classroom Lesson Observation Instrument obs post GEN

COEMET Classroom Observation of Early Mathematics Environment and Teaching obs PK-6 M

DAISI The Dialogic Activity in Science Instruction obs elem M,LIT

EAS The Emergent Academic Snapshot obs PK-6 GEN

ELLCO The Early Language and Literacty Classroom Observation obs, interview PK-6 LIT

IQA Instructional Quality Assessment obs, artifact elem M,LIT

ISIOP Inquiring into Science Instruction Observation Protocol obs, survey mid, high SC

RTOP Reformed Teaching Observation Protocol obs K-16 SC,M

SPC Standards Performance Continuum obs elem, mid GEN

Classroom Snapshot obs K-12 GEN

Mathematics Classroom Observation Protocol obs, rubric elem, mid M,LIT

Science Classroom Observation Guide (NCOSP) obs K-12 SC

June 15, 2012

CADRE

Reliability Information

June 15, 2012

• Reliability type– missing information (19, 38%)– internal stability alpha (11, 22%) (6 acceptable, 5 good)– interrater, % agreement (7, 14%) (2 low, 2 accept., 3 good)– interrater, Kappa (2, 4%) (acceptable)– Interrater, Spearman rank-order correlation (1, good)– Other (2, good and N/A)– More than one type (8, 16%) (all acceptable to good)

• Level– missing information– <.59 or equivalent: low– .6-.79 or equivalent: acceptable– > .8 or equivalent: good

CADRE

Validity Evidence

June 15, 2012

• Validity Type– Missing information (28, 54%)– Content/ face (11, 21%)– Criterion/ predictive (9, 17%)– Construct (8, 15%)– Concurrent/ convergent (5, 10%)– Discriminant /divergent (2, 4%)

CADRE

Trends• Instructional practices (n=22)

– 11 instruments predominantly observation– 11 instruments with 1-2 other constructs,

observation supplemented with survey methods• Teachers’ content knowledge (n=25)

– primarily assessed domain-specific conceptual or procedural knowledge by multiple choice test

– majority of these instruments being used for teachers were originally designed for students

– 8 science, 3 math, 1 technology-specific tests

June 15, 2012

CADRE

Trends cont.• Instructional beliefs (n=9)

– Predominated by survey methods– 6/9 were science specific with only 2 for math

• Multidimensional (n=23)– System-wide reform (predominantly surveys) (10)– Dialogue focused observation instruments to

capture classroom climate and social aspects (13)• Making reliability and validity evidence more

readily available with information about instruments is essential

June 15, 2012

CADRE

Discussion

• Surprises?• Gaps?• Strategies?• Resources?

June 15, 2012

CADRE

Contact InformationAbt Associates

Daphne Minner

Daphne_Minner@abtassoc.com

(617) 520-2454

Alina Martinez

Alina_Martinez@abtassoc.com

(617) 349-2312

June 15, 2012