The Aggie-STEM Center

Aggie Science, Technology, Engineering & Mathematics

Supporting Teachers’ Implementation of STEM Project Based Learning: An Observation

Instrument

Inside • Introduction

• Project-Based Learning

• STEM PD

• Observation of Teachers

• STEM PBL Classroom

Observation Instrument

• Summary

• Further Reading

1

Introduction

Teaching is a complex activity that requires making ongoing multiple decisions and sporadic, responsive actions all while performing preplanned prescribed tasks. Certain aspects of teaching are best assessed with a well-designed observation instrument. This policy brief reviews the need for a classroom-based observation instrument, the use of an instrument for observation and using data collected during observation to inform decision-making. Project-Based Learning STEM Project-Based Learning (PBL) has been explained as a “well-defined outcome with an ill-defined task” (Capraro & Slough, 2006, p. 3) within an interdisciplinary framework. These ill-defined tasks can be complex and messy by nature. With ill-defined projects, students start to investigate interdisciplinary, rigorous real-world topics (Chin & Chia, 2006) usually stemming from a driving question (Blumenfeld, 1991; Krajcik, Blumenfeld, Marx, & Soloway, 1994). STEM PBL integrates engineering design-principles within the K-12 curriculum (Capraro & Slough, 2006). These STEM PBLs can be a model “for classroom activity that shifts away from the classroom practices of short, isolated, teacher-centered lessons and instead emphasizes learning activities that are long-term, interdisciplinary, student-centered, and integrated with real-world issues and practices” (Holbrook, 2007, Internet). When a school’s curriculum is focused on STEM PBLs, those projects foster student understanding by encouraging them to make connections between the content taught in various classes (Pfeiffer, Overstreet, & Park, 2010).

Policy Brief 2011 vol. 1 Issue 1

Copyright© 2011, Aggie STEM Center Disclaimer: The opinions expressed here are the views of Aggie STEM and do not necessarily reflect the views and opinions of the any funding agencies.

Mary Margaret Capraro Robert M. Capraro James R. Morgan James J. Scheurich

Linda Stearns

2

2

Thus, incremental improvements might be gained by guiding teachers to better implementation of STEM projects (PBLs) because improved student understanding of STEM may lead more students to STEM careers (Feller, 2011). STEM Professional Development Professional development (PD) is considered essential for educators to keep up with current reforms. Teachers cannot just be shown a new idea or practice but need experimentation and a culture of learning to fully implement a new practice or idea (Franke, Carpenter, Levi, & Fennema, 2001). PD was shown to be effective over the course of three years when interspersed with classroom observations that tracked targeted instructional behaviors (VanTassel-Baska et al., 2008). Thus a prolonged and sustained PD model should include estimates of assimilation of new ideas introduced through PD into classroom teaching practices. This would require researchers to study and assess transfer of new ideas into the classroom Observation of Teachers In order to improve STEM education classes that are designed to encourage conceptual development (i.e. PBLs), teachers need feedback and support. “There has been considerable evidence from different studies suggesting that how teachers behave in the classroom and the instructional approaches they employ, significantly affect the degree to which students learn” (Van Tassel-Baska, Quek, & Feng, 2007, p. 85).

3

Observations can be either peer or professional in nature, but the observer needs to provide feedback to the educator so he or she may evaluate and adjust their teaching to benefit students (Patrick, 2009). Follow-up Professional Development After the observation process, an iterative planning and implementation phase should be undertaken for designing future PD. The observations should be aggregated across teachers (analysis) with attention to commonalities in both strengths and weaknesses. Information should then be provided back to the teachers during conferences (discussions) and the information should be used as a basis for planning subsequent PDs (needs assessment). Analyses: While an individual observation can serve as the basis for personalized feedback and estimating a single teacher’s growth toward some school or district identified target set of skills, it should not function as a basis for determining school or district-wide PD. A carefully structured observation schedule of all teachers can inform the design of PD. The thoughtful analysis of all teachers’ observations allows the stakeholders to examine the information and build ownership of the pedagogical behaviors and recognize the need for change based on those aggregated observations (Corcoran, 1995). Discussion: It is important to encourage discussion of the aggregated analyses among stakeholders because without this discussion teachers would not have a clear understanding of their degree of assimilation of new ideas or how their lessons may appear to students (Franke et al., 2001). One important aspect of the discussion phase is in differentiating between systemic and idiosyncratic strengths and weaknesses, and teacher reflection on their teaching (Franke et al., 2008). Without these discussions, teachers would not know which topics their colleagues could help them with or what would best be done in large-scale sustained PD. The results of the discussion allows for stakeholders to categorize their needs into teacher specific, school-site, or large scale PD. Needs Assessment: Once teachers and administrators reach a shared understanding they can begin to prioritize which needs should be addressed first and how to best plan for future PD. Follow-up PD for the

3

4

selected needs should be addressed by the district/school in concert with the PD provider, allocating five or more PD days in which all teachers are encouraged to attend (Corcoran, 1995). STEM PBL Classroom Observation Instrument Based on the need to inform the field and to support STEM teaching and learning, an observation instrument was developed by a team of professors and graduate students from The Aggie STEM Center at Texas A & M University to inform on-going and sustained PD. This instrument was specifically created to evaluate observable teaching and learning objectives when teachers develop and implement PBLs activities in their classrooms. Teachers who were evaluated with this instrument participated in sustained PD (10 full days) focusing on STEM PBL. The PD focused on each of the measured objectives. Both the observers and the teachers were trained on the components and purposes of the instrument. Description of Instrument The instrument contains twenty-two items organized by six objectives. The objectives include: (a) PBL structure, (b) PBL Facilitation, (c) Student Participation, (d) Resources, (e) Assessment, and (f) Classroom Learning Environment. The number of indicators under each objective varies. Each indicator was evaluated on a scale ranging from 1 (no evidence) to 5 (to a great extent) with the observer justifying each score assigned to an item. Occasionally, an item will not apply to what is taught during a particular observation. In this case, or when the observer is only present for a part of a PBL activity, a well-documented lesson plan can provide insights and further details. The observer may still choose to indicate that a particular behavior was not applicable or not observed during the class period. Use of Instrument and Aggregating Observation Data It is important for both observers and those observed to understand that scales are not intended to be used as a grading tool, but rather to indicate progress toward an ideal. Therefore, while it may be tempting to use the scale as a ratio, caution should be used. A low score on the instrument is an indicator of need for continued PD but should never be used as part of any

5

teacher’s formal evaluation process. For the scale to function well, the observers should be well trained, have a solid understanding of what the ideal is, what it means to not see a particular item on the scale, and reliably be able to rate particular events similarly to peer observers. Inter-rater reliability is only achieved by multiple observers observing the same teaching event and then discussing their ratings and justifications. In the identification of systemic issues it is important to aggregate across observations and observers, campus boundaries, and subject areas within each of the six objectives. Data can be aggregated using the mode. However, in the presence of adequate inter-rater reliability and substantial training the scale can be used to compute means and standard deviations. In either case, larger numbers indicate greater progress toward the ideal goal and lower numbers indicate greater opportunity for systemic PD. The graph displayed below (Figure 1) provides a depiction of fidelity of PBL implementation at one high school. As indicated by the graph, the teachers demonstrated significant variance in PBL implementation at the beginning of the study. By the end of the study, the teachers in this HS exhibited much less variance in PBL implementation with an

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

2007 2008 2009 2010

High School Longitudinal Fidelity of Implementation

Structure

Facilitation

St. Participation

Resources

Assessment

Learning Environemnt

Figure 1. High school longitudinal fidelity of implementation for teacher observations.

4

1

Further Reading

Blumenfeld, P., Soloway, E., Marx, R., Krajcik, J., Guzdial, M., & Palincsar, A. (1991). Motivating project-based learning: Sustaining the doing, supporting the learning. Educational Psychologist, 26, 369-398.

Capraro, R. M., & Slough, S. W. (Eds.). (2009). Project-based

learning: An integrated science, technology, engineering, and mathematics (STEM) approach. Rotterdam, The Netherlands: Sense.

Chin, C., & Chia, L. (2006). Problem-based learning: Using ill-

structured problems in biology project work. Science Education, 90(1), 44-67.

Corcoran, T. (1995). Helping teachers teach well: Transforming

professional development (CPRE Policy Brief). New Brunswick, NJ: Consortium for Policy Research in Education. Retrieved from http://www.cpre.org/Publications/rb16.pdf

Darling-Hammond, L., Wei, R. C., Andree, A., Richardson,

N., & Orphonos, S. (2009). Professional learning in the learning profession: A status report on teacher development in the United States and abroad. Palo Alto, CA: National Staff Development Council and The School Redesign Network at Stanford University.

2

Franke, M. L., Carpenter, T. P., Levi, L., & Fennema, E. (2001). Capturing teachers’ generative change: A follow-up study of professional development in mathematics. American Educational Research Journal, 38, 653-689.

Holbrook, J. (2007). Project-based learning with multimedia.

Retrieved from http://pblmm.k12.ca.us/PBGuide/WhyPBL.html

Krajcik, J. S., Blumenfeld, P. C., Marx, R. W., & Soloway, E.

(1994). A collaborative model for helping middle grade science teachers learn project-based instruction. The Elementary School Journal, 94, 483-496.

Patrick, P. (2009). Professional development that fosters

classroom application. Modern Language Journal, 93, 280-287.

Pfeiffer, S. I., Overstreet, M., & Park, A. (2010). The state of

science and mathematics education in state-supported residential academies: A nationwide survey. Roeper Review, 32, 25-31.

Van Tassel-Baska, J., Quek, C., & Feng, A. X. (2007). The

development and use of a structured teacher observation scale to assess differentiated best practice, Roeper Review, 29(2), 84-92.

VanTassel-Baska, J., Feng, A. X., Brown, E., Bracke, B.,

Stambaugh, T., French, H., . . Bai, W. (2008). A study of differentiated instructional change over 3 years. The Gifted Child Quarterly, 52, 297-312.

6

overall high-level of fidelity. Summary This observation instrument is intended to assess the enactment of the essential elements of implementing a STEM PBL activity in classrooms and is not suitable for assessing broad or generalized PBL, or PBL without the requisite Aggie STEM professional development series. We believe the usefulness of the instrument lies in its ability to distinguish high fidelity implementations and to assist in pinpointing areas for improvements. The target areas can be the focus of ongoing discussion and decisions; sporadic, responsive actions; as well as the preplanned prescribed tasks associated with implementing STEM PBL. The type of evidence provided by the instrument with regard to teaching is well correlated with improved student learning (Darling-Hammond, Wei, Andree, Richardson, & Orphonos, 2009).