View
226
Download
1
Category
Tags:
Preview:
Citation preview
Compendium of Research Instruments for STEM Education
PART I : Teacher Practices , PCK and Content Knowledge
Written for CADREby Abt Associates (Daphne Minner, et. al.), 2012
In 2011, the National Science and Technology Council reviewed how 13 federal agencies spent $3.4 billion in fiscal year 2010 to support STEM education.
NSF was found to have made the largest investment in STEM education, and its DRK-12 program had the largest budget of its 6 educational research and development programs.
The compendium reviewed here (Part 1 of 2) focuses on 5 cohorts of DRK-12 projects (2008-2012) that utilized instruments designed to assess teacher practices, pedagogical content knowledge, and content knowledge.
The purpose of this compendium is to provide an overview on the current status of STEM instrumentation commonly being used in the U.S and to provide resources useful to research and evaluation professionals.
Synopsis
Research Question: What are the instruments, constructs, and methods being used to study teacher outcomes within the DR-K12 portfolio?
Only extant, named instruments (as opposed to instruments being developed as part of a current proposal) were included.
Two Phases:◦ Phase 1: A review of all proposals funded by DRK-12 2008-
2012 revealed 295 eligible projects.
◦ Phase 2: Data collection was conducted for instrument-specific information about reliability and validity evidence, development and piloting, accessibility of the instrument, administration, and constructs measured.
Methods
Since CADRE is funded as a cooperative agreement rather than a contract, they were unable to access Fastlane files and relied on materials provided by PIs.
For 36 projects, materials were unavailable.
For 8 of the 57 PCK instruments, the actual instruments were unavailable.
6 instruments required purchasing.
Limitations
75 projects proposed to measure teacher practices, PCK, or Content: 71% measured only 1outcome, 24% measured 2 outcomes, and 5% measured all 3.
Instruments Identified
◦Practices: 42
◦PCK: 24
◦Content Knowledge: 27
Cross-Cutting Comparisons (Leah)
5 Categories of Instruments
1. Instructional Practices (Appendix A)
2. Instructional Practices plus Additional Constructs (Appendix B)
3. Instructional Beliefs (Appendix C)
Multidimensional
4. System-wide Reform Focused (Appendix D)
5. Discourse Focused (Appendix E)
Cross-Cutting Comparisons
Need to be more cognizant about providing relevant psychometric information on the tools used and developed in order for others to reliably implement the tools in their own projects.
Instruments developed must go through rigorous reliability and validity
Initial step towards the systematic assessment and improvement of STEM research tools.
Cross-Cutting Comparisons: Conclusions
Eleven instruments that primarily assessed classroom instructional practices:
Seven observation protocols, Three rubrics, One survey
Predominantly designed for pre-k through middle school teachers (6, 55%)
More focused on science (5, 45%) than mathematics (3, 27%) or technology (2, 18%).
The three science observation protocols capture variables ranging from the lesson’s temporal flow and percentage of time students spend in different types of groupings, to the extent of opportunity for students to engage in the various phases of the investigation cycle.
Instructional Practices (Pete)
The two science scoring rubrics are intended to be applied to lesson artifacts and instructional materials that the teacher provides students. They contain codes for student grouping, structure of lessons, use of scientific resources, hands-on opportunities through investigation, cognitive depth of the materials, encouragement of the scientific discourse community, and opportunity for explanation/justification, and connections/applications to novel situations.
Across these eleven instruments, one had low reliability evidence, and four (36%) had acceptable or good evidence. For only two instruments was the team able to find validity evidence.
Instructional Practices
Two Examples Related to Science Instructional Strategies Classroom
Observation Protocol ◦ Identifying sense of purpose; asking account of student
ideas; engaging students with relevant phenomena; developing and using scientific ideas; promoting student thinking about phenomena, experiences, and knowledge
Scoop Notebook – Artifact rubric◦ Portfolio assessment that captures: grouping, structure of
lessons, use of scientific resources, hands-on, inquiry, cognitive depth, scientific discourse community, explanation/justification, assessment, connections/applications
11 instruments that measure instructional practices in addition to one or two other constructs, meaning:◦ physical context◦ demographics◦ teacher content knowledge ◦ an aspect of classroom management
This more comprehensive nature is also reflected in the subject domains being assessed—◦ 2 each, mathematics and science◦ 5 both mathematics and science ◦ 1 technology◦ 1 general teaching skills
Instruments to Determine Teacher Practices Plus One or Two other Constructs (Michele)
Exist for many subjects. Middle School version tests all sciences very generally, whereas high school breaks it apart by specific domains
Sit-down test, 4 hours, 50 multiple-choice questions and 2 constructed-response questions
The test is designed to provide evidence that an examinee has a basic working knowledge of teaching foundations
Instruments to Determine Teacher Practices Plus one or two other Constructs
Ratings are made after at least 3 hours of observation
Ratings for each item are made on a 7-point scale. Behavioral descriptors are present at the 1, 3, 5, and 7 levels.
Assesses the materials and instructional supports for math and science learning present
Instruments to Determine Teacher Practices Plus one or two other Constructs
Instruments to Determine Teacher Instructional Beliefs (Yann)
Name What’s measured Type Validity
EvidenceGRADE LEVEL
Mathematics Teaching Efficacy Belief Instrument (MTEBI, 2000)
Modified from STEBI Riggs (California State U) & Enochs (Kansas State U),1990
Personal math teaching efficacy (13 items) & math teaching outcome expectancy (8 items)
Extent to which teachers believe they have the capability to positively affect student achievement
Survey (Likert 5-point scale)
Construct
MTEBI:21 items, PRESERVICE(STEBI:25 items, ELEMENTARY)
PS: Why is VNOS –C included?
Instruments to Determine Teacher Instructional Beliefs
Name What’s measured Type Validity
EvidenceGRADE LEVEL
Principles of Scientific Inquiry-Teacher (PSI-T)
Campbell, Chapman (Utah State U) & Abd-Hamid (U of Iowa), 2010
Teacher (& student) perceptions of frequency of occurrence when students are responsible for each of 5 principles of scientific inquiry (NRC)
Extent to which students are experiencing inquiry in science classrooms
Survey (5-point, 20 items)(teacher & student version)
ContentConstruct
HIGH
Instruments to Assess System-wide Reform Efforts
Name What’s measured Type Validity
EvidenceGRADE LEVEL
Local Systemic Change Classroom Observation Protocol (LSC)
Inside the Classroom: Teacher Interview Protocol
Overall quality of observed math/science lesson: lesson design; implementation; math/science content; classroom culture; likely impact on students' understanding
Teachers' perceptions of factors that influenced selection of lesson content and pedagogy
Observation (5-point scale)
Content (Good inter-rater % reliability)
K-12
Interview
Horizon Research Inc., 2000
Thirteen instruments looking at instructional practices and social aspects of classroom community (including class management).
Observation protocols Six are non-domain specific Three are math-specific Three are science-specific One measures both Seven demonstrated more than one type of
validity (more than other categories)
STEM Discourse (Matt)
Three scales◦ Lesson design implementation◦ Content ->PCK->Propositional and Procedural◦ Classroom culture (e.g., egalitarian s-t
relationship) High interrater % agreement High Validity
◦ Construct◦ Content◦ Predictive
Reformed Teaching Observation Protocol (RTOP)
Three domains◦ Emotional◦ Classroom organization◦ Instructional support
High internal consistency High interrater % agreement Content validity
Classroom Observation of Early Mathematics Environment and Teaching (COEMET)
25 Content Tests, 12 General tests, 8 Science, 3 Math, 1 Science and Math, 1 Technology.
◦ General Tests: American College testing, GRE, ITBS-Iowa Test of Basic Skills, NAEP, PISA, PRAXIS, WEST-E.
◦ Science: MOSART, FACETS, IL Certification Testing System Study Guide-Science, FCI Force Concept Inventory Assessment, DTAMS-science: Diagnostic Science Assessment for Middle School Teachers, Classroom Test of Scientific Reasoning (Lawson).
◦ Math: MKT, M-SCAN, DTAMS-math: Diagnostic Math Assessment for Middle School Teachers.
◦ Science and Math: TIMSS
◦ Technology: TAGLIT: Taking a Good Look At Instructional Technology
12 of them Student Test, 9 Teacher Test, 2 Survey, 1 observation Tool, 1 Student test and Teacher Tools.
7 K-12 level, 4 elementary and middle, 4 Postsecondary, 3 high school, 2 middle, 1 grades4-9, 1 elementary, 1 middle and high, 1 high and postsecondary, 1 no level indicated.
Survey Instruments to determine Teacher Content Knowledge (Arzu)
Each assessment is composed of 25 items—20 multiple-choice and 5 open-response.
Paper-and-pencil format Pre- and post-tests before and after workshops To determine growth in teachers' content knowledge To be completed by test-takers within an hour. Each assessment has 3-4 science sub-domains. Available for use free of charge. Scored for a fee of $10 per teacher per assessment -includes scores on
individual items, on each science sub-domain in the content area, and on four different knowledge types (memorized, conceptual understanding, higher-order thinking, pedagogical content knowledge)
• http://louisville.edu/education/centers/crmstd/diag-sci-assess-middle - http://louisville.edu/education/centers/crmstd/diag-sci-assess-middle
DTAMS-Science
Free and can be accessed after completion of four online tutorials that explain test design, use, scoring, and interpretation of results.
A set of multiple-choice items include K–12 physical science and earth science content, and K–8 life science content in the NRC NSES as well as to the research literature about misconceptions concerning science concepts.
MOSART:Misconceptions-Oriented Standards-Based Assessment Resources for Teachers
ASW – Analysis of Student Work: A rubric is used to score teachers’ evaluations of a standardized set of video cases of student problem solving.
LoU – Levels of Use Interviews: An interview determines how a change is being implemented in the classroom.
SEPUP - Group Interaction and Communication of Scientific Information Rubrics: Rubrics are used to grade student work on a variety of measures including how they design and conduct an investigation, analyze data, understand concepts, evaluate evidence and identify tradeoffs, communicate scientific information, and work cooperatively in a group.
Unclassified Instruments (Jenay)
Detailed access information can be found for each instrument in Appendices H & I of the compendium.
Part 2 of the compendium (not covered here) details measurement of students’ content knowledge, reasoning skills, and psychological attributes.
Conclusion
Recommended