Comparing the Efficacy of Reform-Comparing the Efficacy of Reform-Based and Traditional/Verification Based and Traditional/Verification Curricula to Support Student and Curricula to Support Student and

Teacher Learning Teacher Learning

E.M. Granger and T.H. Bevis, E.M. Granger and T.H. Bevis,

Office of Science Teaching ActivitiesOffice of Science Teaching Activities

College of Arts & SciencesCollege of Arts & Sciences

Florida State UniversityFlorida State University

  

Y. Saka and S.A. SoutherlandY. Saka and S.A. Southerland

School of Teacher EducationSchool of Teacher Education

Florida State UniversityFlorida State University

Recent Reform Efforts’ Recent Reform Efforts’ Emphasis on Learning Emphasis on Learning

Student understanding of the discipline’s core Student understanding of the discipline’s core concepts, theories, and models; concepts, theories, and models;

Student understanding, in general, of how Student understanding, in general, of how scientific knowledge is generated, tested, and scientific knowledge is generated, tested, and accepted; and accepted; and

Students being able to use the first two to extend Students being able to use the first two to extend understanding into new areas.understanding into new areas.

To achieve this, students’ learning of science should To achieve this, students’ learning of science should echo the way science is conducted by scientists. echo the way science is conducted by scientists.

(Karplus & Thier, 1967; Bybee, 1997)(Karplus & Thier, 1967; Bybee, 1997)

National Reform EffortsNational Reform Efforts

Have produced promising, well-designed Have produced promising, well-designed curricula, yet have largely failed to shift curricula, yet have largely failed to shift classroom practices as classroom practices as

reforms require fundamental shifts in the way reforms require fundamental shifts in the way teachers understand both their content and the teachers understand both their content and the appropriate ways to teach this contentappropriate ways to teach this content

new practices are perceived by many teachers new practices are perceived by many teachers and administrators as being in conflict with and administrators as being in conflict with student achievement on standardized testsstudent achievement on standardized tests

Rationale of the ResearchRationale of the Research

Central issues for successful enactment of Central issues for successful enactment of science education reformscience education reform

Teacher professional developmentTeacher professional development Well-designed instructional and curricular Well-designed instructional and curricular

materialsmaterials Knowledge of how best to combine this Knowledge of how best to combine this

professional development and curriculum to professional development and curriculum to achieve desired student outcomes achieve desired student outcomes

This Research Aims toThis Research Aims toExplore the influence of professional Explore the influence of professional

development in combination with the development in combination with the enactment of reform-based science enactment of reform-based science curriculum on students’ learning and on curriculum on students’ learning and on teachers’ knowledge and beliefs about teachers’ knowledge and beliefs about science and the teaching and learning of science and the teaching and learning of science. This paper will report preliminary science. This paper will report preliminary results on what we have learned from the results on what we have learned from the data collected in the first year of a two-data collected in the first year of a two-year study.year study.

The StudyThe Study

Design: A randomized field trial (Quasi-Experimental Design)Design: A randomized field trial (Quasi-Experimental Design)

Participants (year one): Participants (year one): 32 Treatment teachers (10 432 Treatment teachers (10 4thth grade, 22 5 grade, 22 5thth grade) (PD and GEMS) grade) (PD and GEMS)

696 students from their classes696 students from their classes 29 Control teachers (10 429 Control teachers (10 4thth grade, 19 5 grade, 19 5thth grade) (District’s Text) grade) (District’s Text)

482 students from their classes482 students from their classes Control/Experimental group matching within schools if possible, based on Control/Experimental group matching within schools if possible, based on

student demographics where not possiblestudent demographics where not possible

Time: 2007-2008 school year Time: 2007-2008 school year

Location: 18 participating schools in a district located in central FloridaLocation: 18 participating schools in a district located in central Florida

Curricula:Curricula: Experimental: GEMS Experimental: GEMS Space Science Curriculum SequenceSpace Science Curriculum Sequence

Control: District Adopted Text’s Space Science UnitControl: District Adopted Text’s Space Science Unit

Professional DevelopmentProfessional Development

A four day, intensive professional development prior to school yearA four day, intensive professional development prior to school year

• activities, discussions, and reflections centered around the GEMS activities, discussions, and reflections centered around the GEMS curriculum and the pedagogies that underpin itcurriculum and the pedagogies that underpin it

• explicit/reflective experiential approach throughout with respect to explicit/reflective experiential approach throughout with respect to both the curriculum and the pedagogy (pedagogy included: learning both the curriculum and the pedagogy (pedagogy included: learning cycle approach to science instruction, questioning/discussion cycle approach to science instruction, questioning/discussion strategies, evidence circles, assessment strategies, nature of strategies, evidence circles, assessment strategies, nature of science teaching strategies)science teaching strategies)

Three-hour follow up immediately prior to teaching the GEMS unitThree-hour follow up immediately prior to teaching the GEMS unitThree-hour mid-teach meetingThree-hour mid-teach meetingRegular observations and/or videotapingRegular observations and/or videotapingClassroom support by a science coach as requested Classroom support by a science coach as requested

Curriculum: Experimental GroupCurriculum: Experimental Group

Great Exploration in Math and Science (GEMS) Great Exploration in Math and Science (GEMS) Space Science Curriculum Sequence Space Science Curriculum Sequence (2007)(2007)

Designed to address the age-appropriate core concepts in space Designed to address the age-appropriate core concepts in space science (NSES, 1996) and common misconceptions that students science (NSES, 1996) and common misconceptions that students harbor about them.harbor about them.

Inductive, activity based approach to space science concepts Inductive, activity based approach to space science concepts (learning cycle format lessons).(learning cycle format lessons).

Explicit focus on the role of models and evidence in science.Explicit focus on the role of models and evidence in science. Students encouraged throughout to evaluate alternative explanations, Students encouraged throughout to evaluate alternative explanations,

to use evidence to support explanations, and to critique the merits of to use evidence to support explanations, and to critique the merits of an explanation. an explanation.

Clear directions for the teacher (material preparation, background Clear directions for the teacher (material preparation, background content information, lesson scripts, pedagogy notes).content information, lesson scripts, pedagogy notes).

Teachers instructed to adhere strictly to curriculum.Teachers instructed to adhere strictly to curriculum.

Curriculum: Control GroupCurriculum: Control Group

District adopted textDistrict adopted text Features more didactic presentation of space science Features more didactic presentation of space science

concepts including direct instruction, reading of text, concepts including direct instruction, reading of text, with students answering very focused questions. with students answering very focused questions.

The activities included in the text typically are verification The activities included in the text typically are verification of the content already presented in the text or, more of the content already presented in the text or, more commonly, activities that are peripherally associated with commonly, activities that are peripherally associated with the topic, but that do not address the core concepts. the topic, but that do not address the core concepts.

Teachers asked to adhere to the text and to this Teachers asked to adhere to the text and to this instructional model (even if this is not their normal instructional model (even if this is not their normal science teaching practice)science teaching practice)

Overall Research QuestionsOverall Research QuestionsDoes the use of such curricula plus professional Does the use of such curricula plus professional development work? development work?

How does it work, that is, what mediating variables can How does it work, that is, what mediating variables can we identify that indirectly transmit the effect of the we identify that indirectly transmit the effect of the treatment? treatment?

For whom does the use of such curricula plus For whom does the use of such curricula plus professional development work, that is, what student professional development work, that is, what student moderator variables can we identify from the data moderator variables can we identify from the data collected? collected?

Under what conditions do such curricula plus professional Under what conditions do such curricula plus professional development work, that is, in this case what teacher development work, that is, in this case what teacher moderating variables can we identify from the data moderating variables can we identify from the data collected? collected?

Construct to be Measured Tools for Data Collection and Analysis

Space Science Content MOSART resources (Sadler, Coyle, Cook-Smith, & Miller, 2007) http://www.cfa.harvard.edu/smgphp/

mosart/site_map.html

Attitudes toward science Homerton Science Attitudes Survey (Warrington, Younger, & Williams, 2000)

Models and evidence content knowledge

Knowledge about Models & Evidence (Granger & Bevis, designed for this study, 2007)

Science Inquiry Views of Scientific Inquiry—Elementary (VOSI-E) (Schwartz, Lederman, & Lederman, 2008)

Data AnalysisData Analysis

Exploratory statistical analysis without Exploratory statistical analysis without controlling family-wise error.controlling family-wise error.

Hierarchical Linear Modeling (HLM-6) was Hierarchical Linear Modeling (HLM-6) was required to account for all the inter-required to account for all the inter-dependencies of student outcomes within dependencies of student outcomes within teachers (Raudenbush & Bryk, 2002; teachers (Raudenbush & Bryk, 2002; Raudenbush et al., 2004)Raudenbush et al., 2004)

Findings—Student DataFindings—Student Data

Outcome UnstandardizedGEMS Effect

Standard Error

p-value StandardizedGEMS Effecta

Post Measures

Achievement 0.629* 0.206 0.004 0.22*

Attitude 1.862* 0.995 0.067 0.12*

Delayed Post Measures

Achievement 0.340 0.213 0.116 0.12

Attitude 1.282 1.075 0.239 0.08

GEMS Main Effects for Student Outcomes

* Statistically significant at the 0.10 level.a Standardized effects have been obtained by dividing the raw score GEMS coefficient by the outcome standard deviation

FindingsFindings

Pre Teacher Achievement

0 2 4 6 8 10 12

Pre

dict

ed S

tude

nt P

ost

Ach

ieve

men

t

1.6

1.7

1.8

1.9

2.0

2.1

2.2

2.3

GEMS Group

Control Group

Discussion Student FindingsDiscussion Student Findings

WWell-designed, reform-based curriculum combined with ell-designed, reform-based curriculum combined with professional development centered around that professional development centered around that curriculum can support the development of elementary curriculum can support the development of elementary students’ content knowledge (cognitive outcomes). students’ content knowledge (cognitive outcomes).

Well-designed, reform-based curriculum combined with Well-designed, reform-based curriculum combined with professional development centered around that professional development centered around that curriculum can support elementary students’ improved curriculum can support elementary students’ improved attitudes about science (affective outcomes). attitudes about science (affective outcomes).

Students in the classrooms of Students in the classrooms of teachers who were low on teachers who were low on their content knowledge test score their content knowledge test score before beginning the before beginning the study showed a significant increase in their post content study showed a significant increase in their post content test scores.test scores.

How do we explain our results in light How do we explain our results in light of mixed results from previous studies?of mixed results from previous studies?

Strength of curriculum: GEMS Strength of curriculum: GEMS SSCS SSCS is is certainly a well-designed and easy to enact certainly a well-designed and easy to enact curriculum.curriculum.

Tight control of curriculum and instructional Tight control of curriculum and instructional methodology of both experimental methodology of both experimental and and control groups.control groups.

Monitoring and assessment of fidelity to Monitoring and assessment of fidelity to the assigned curriculum and instructional the assigned curriculum and instructional methodology.methodology.

ImplicationsImplicationsOur results suggest that reform-based instruction Our results suggest that reform-based instruction using a specified curriculum in combination with using a specified curriculum in combination with professional development results in positive professional development results in positive effects on student cognitive and affective effects on student cognitive and affective outcomes as compared with more traditional outcomes as compared with more traditional instruction. instruction.

We further suggest that at least some of the We further suggest that at least some of the mixed results found in previous research may be mixed results found in previous research may be due to experimental design, especially in due to experimental design, especially in assessing fidelity to the curriculum and/or assessing fidelity to the curriculum and/or instructional methodology in instructional methodology in bothboth the treatment the treatment andand control groups. control groups.

Findings:Findings:

Teacher StudyTeacher Study

Construct to be Measured Tools for Data Collection and Analysis

Space Science Content Collaboration for Astronomy Education Research (CAER) (1999). Astronomy Diagnostic Test.

Self-efficacy/Outcome expectancy/Attitude about reform practices

Teaching Science as Inquiry (TSI) (Smolleck, Zembal-Saul, Yoder, 2006).

Attitudes toward science Test of Science-Related Attitudes (TOSRA) (Fraser, 1981)

Teaching beliefs/concepts Beliefs About Reformed Science Teaching and Learning (BARSTL) (Sampson & Benton, 2006)

Data AnalysisData Analysis

Exploratory statistical analysis without Exploratory statistical analysis without controlling family-wise error.controlling family-wise error.

Standard single-level multiple regression Standard single-level multiple regression models on the effects of the treatment on models on the effects of the treatment on five teacher outcomes from three of the five teacher outcomes from three of the teacher assessments (Space Science teacher assessments (Space Science Content, TSI, and TOSRA)Content, TSI, and TOSRA)

FindingsFindings

FindingsFindings

Discussion Teacher FindingsDiscussion Teacher Findings

WWell-designed, reform-based curriculum combined with ell-designed, reform-based curriculum combined with professional development centered around that professional development centered around that curriculum can support the development of elementary curriculum can support the development of elementary teachers’ content knowledge as well as their beliefs teachers’ content knowledge as well as their beliefs about teaching. about teaching.

The GEMS curriculum materials and professional The GEMS curriculum materials and professional development were more effective in supporting teachers development were more effective in supporting teachers who had more to learn--those with lower self efficacy and who had more to learn--those with lower self efficacy and outcome expectancy at the outset of the study. This outcome expectancy at the outset of the study. This effect lessened for teachers who had high self efficacy effect lessened for teachers who had high self efficacy and outcome expectancy at the outset. and outcome expectancy at the outset.

ImplicationsImplications

Our results suggest that what teachers Our results suggest that what teachers learn from professional development and learn from professional development and teaching with the curriculum varies teaching with the curriculum varies according to what their knowledge, beliefs, according to what their knowledge, beliefs, and/or attitudes are entering the and/or attitudes are entering the experience. experience.

Future ResearchFuture ResearchFor which student groups (i.e., SES, primary language, etc.) For which student groups (i.e., SES, primary language, etc.) does this combination of curriculum and professional does this combination of curriculum and professional development work best?development work best?

What teacher characteristics influence the success of the What teacher characteristics influence the success of the application of this combination of curriculum and professional application of this combination of curriculum and professional development?development?

Are these same increases in student learning using similar Are these same increases in student learning using similar curricula and professional development found in secondary-curricula and professional development found in secondary-school classrooms (where teachers usually have stronger school classrooms (where teachers usually have stronger content background than that of the elementary teachers in content background than that of the elementary teachers in this study)?this study)?

What is the effectiveness of implementing the curriculum What is the effectiveness of implementing the curriculum without professional development on teachers’ knowledge without professional development on teachers’ knowledge and beliefs.and beliefs.

How best to monitor and assess fidelity?How best to monitor and assess fidelity?

Copies of the papers may be Copies of the papers may be downloaded from:downloaded from:

http://ret.fsu.edu/papers_09.html http://ret.fsu.edu/papers_09.html