Inquiry-Based Instruction in Iowa: A Report on the ... · The focus of the ELI professional development is to promote inquiry-based instruction. This approach, articulated in the

Inquiry-Based Instruction in Iowa: A Report on the Implementation of Every Learner Inquires in Year 2

July 2008

Submitted to Kathy McKee

Iowa Department of Education

Prepared by Jonathan Margolin, Ph.D., Principal Investigator

Megan Brown Shazia Miller

20 North Wacker Drive, Suite 1231 Chicago, IL 60606-2901 800-356-2735 312-288-7600 www.learningpt.org

2646_07/08

Contents Page

Executive Summary ........................................................................................................................ 1

Student Learning ....................................................................................................................... 2

Teacher Practice ........................................................................................................................ 3

Organizational Support ............................................................................................................. 3

Teacher Learning and Understanding ....................................................................................... 4

Reactions to Professional Development Workshops ................................................................ 4

Recommendations ..................................................................................................................... 5 Introduction ..................................................................................................................................... 7

What Is Inquiry-Based Science Instruction? ............................................................................. 7

Program Design ........................................................................................................................ 9

Program Goals ........................................................................................................................ 10

Design of Professional Development ...................................................................................... 11

Evaluation Overview .............................................................................................................. 12 Methods......................................................................................................................................... 15

ELI Workshop Reaction Data ................................................................................................. 15

Teacher Survey ....................................................................................................................... 15

Student Survey ........................................................................................................................ 17

Instructional Logs ................................................................................................................... 18

Classroom Observations ......................................................................................................... 19

Interviews With Principals and Teachers ............................................................................... 20 Section 1: Workshop Feedback Summary .................................................................................... 22

Workshop Attendance ............................................................................................................. 22

Workshop Topics and Activities ............................................................................................. 23

Ratings of Quality and Utility ................................................................................................. 24

Summary of Findings .............................................................................................................. 32 Section 2: Impact on Teacher Understanding ............................................................................... 35

Level of Understanding ........................................................................................................... 35

Self-Efficacy ........................................................................................................................... 36

Summary of Teacher Learning and Understanding ................................................................ 37

Section 3: Impact on Teaching Practice ........................................................................................ 39

Inquiry Instruction .................................................................................................................. 39

Classroom Culture .................................................................................................................. 44

Learning Cycle ........................................................................................................................ 47

Summary of Impact on Teaching Practice .............................................................................. 50 Section 4: Organizational Support ................................................................................................ 51

Administrator Support ............................................................................................................ 51

Professional Learning ............................................................................................................. 53

School Resources .................................................................................................................... 54

Coherence ............................................................................................................................... 55

Implementation Concerns Related to Organizational Support ............................................... 56

Summary of Organizational Support ...................................................................................... 57 Section 5: Impacts on Student Learning ....................................................................................... 59

Impact on Student Understanding of Scientific Concepts and Science Inquiry ..................... 59

Impact on Student Engagement and Self-Efficacy ................................................................. 60

Addressing the Needs of All Subgroups ................................................................................. 62

Summary of Impact on Student Learning ............................................................................... 62 Conclusions and Recommendations ............................................................................................. 63 References ..................................................................................................................................... 65 Appendixes

Appendix A. Survey Methodology ......................................................................................... 67

Appendix B. Teacher Survey Frequency Tables. ................................................................... 76

Appendix C. Instructional Log and Observation Frequency Data. ......................................... 88

Appendix D. Interview Protocols ........................................................................................... 91

Executive Summary Every Learner Inquires (ELI), which runs from August 2006 through June 2010, is a statewide K–12 professional development effort to improve science learning in Iowa schools by promoting inquiry-based instruction. This program, currently involving teachers from four schools in four districts, provides a summer institute and five workshops per year; participants are expected to engage in job-embedded professional learning during the time between workshops. The goals of the initiative are to improve science learning for all students, build teacher leadership and content expertise in the area of science, provide teachers with the content and skills necessary to implement inquiry-based instruction, and establish a structure for sustained implementation. Under contract from the Iowa Department of Education, Learning Point Associates is conducting a mixed-method evaluation of the program. This report provides formative feedback on the outcomes during Year 2, based on findings from interviews with teachers and principals, a survey of teachers, observations of science lessons, and instructional logs. In addition, this report provides formative feedback on the professional development workshops offered during the winter and spring of Year 2. A separate report, to be delivered in spring 2009, will present findings from an analysis of student achievement data to describe student learning outcomes. The design of the ELI initiative calls for the training of teachers from four case study schools along with regional teams based in area education agencies (AEAs) who will expand the program statewide in Years 3 and 4. All science teachers in the four case study schools participate in the workshops, implement the instructional strategies presented therein, keep logs of their implementation, and use inservice time to support implementation. Eleven AEA leadership teams are participating in the program to build their capacity to deliver professional development in inquiry instruction; in the third and fourth years of the initiative, AEA teams are expected to begin to provide professional development for schools in their respective areas, building on the lessons learned from the case study schools. Each AEA team includes AEA science education staff and at least two teacher leaders drawn from schools within the jurisdiction of the given AEA. The focus of the ELI professional development is to promote inquiry-based instruction. This approach, articulated in the National Science Education Standards (National Research Council, 1996), provides students direct experience with the scientific inquiry process. Although the skills and understandings related to scientific inquiry vary according to grade levels, the Standards articulate the following essential features of inquiry instruction for all grade levels:

• Learners are engaged by scientifically oriented questions.

• Learners give priority to evidence, which allows them to develop and evaluate explanations that address scientifically oriented questions.

• Learners formulate explanations from evidence to address scientifically oriented questions.

• Learners evaluate their explanations in light of alternative explanations, particularly those reflecting scientific understanding.

• Learners communicate and justify their proposed explanations.

Learning Point Associates Every Learner Inquires Year 2 Evaluation—1

Inquiry-based instruction is supported by a student-centered classroom culture in which teachers serve as facilitators of learning, promote discourse and discussion, encourage curiosity and questioning, and respect the diversity of their students. In light of the fact that teachers need guidance and structure to enact the Standards, the ELI program provides a pedagogical model for lesson design. This model, called the 5E Learning Cycle, articulates five phases of inquiry, and each phase corresponds to a different learning goal. During Year 2 of the ELI program, teachers were expected to implement two full learning cycles. The findings of the Year 2 evaluation are organized by questions aligned to five levels of program outcomes: student learning, teacher practice, organizational support, teacher learning and understanding, and reactions to professional development (Guskey, 2000). Student Learning 1. To what extent have students acquired abilities and fundamental understandings about

scientific inquiry?

2. What is the impact of the ELI program on student enthusiasm and self-efficacy toward science learning?

3. Does the ELI program address the needs of all student subgroups?

4. Are the scientific understandings and abilities of students improving? Most teachers reported that inquiry instruction has improved student understanding of both scientific concepts and the scientific inquiry process, with the number who reported the latter outcome growing since last year. Several teachers stated that inquiry-based instruction has improved problem-solving skills in particular. Teachers and principals reported that inquiry instruction has improved the level of student engagement and enthusiasm in science; they mentioned that students appear to be more excited about attending science class than before. Elementary students themselves typically reported that they are enthusiastic toward science. However, high school students typically reported that they are not enthusiastic about science class. The overall level of enthusiasm toward science has not increased since Year 1. Regarding self-efficacy toward science, about half of elementary students and one fifth of high school students rated themselves as being capable to accomplish various tasks involving their science class. The ELI program appears to address the needs of all student subgroups. As reported last year, teachers and principals indicated inquiry instruction supports the learning of special education students and English language learners. Respondents described the program as reaching students by meeting students at their own level and teaching students through hands-on activities. The improvement of scientific understandings and abilities of students during Year 2 will be evaluated through an analysis of student achievement data to be completed in spring 2009.


Teacher Practice 5. To what extent has teaching practice changed to reflect an inquiry-based approach? The evaluation focused on the extent to which teaching practice reflected inquiry-oriented instruction, the extent to which the classroom culture became more student centered, and the extent to which teachers adapted their lessons to the 5E Learning Cycle, a pedagogical model for lesson design. Teachers reported that these three aspects of inquiry are occurring frequently in their classrooms. Moreover, most teachers have completed at least three learning cycles. Although the frequency of the inquiry-related activities has not increased since Year 1, several teachers stated that their ability and comfort level have increased over the past year. However, the level of implementation appears less than comprehensive because it is largely focused on hands-on data collection and analysis. Other aspects of inquiry learning do not appear to be prevalent, such as designing studies, writing reports of findings, critically assessing procedures, or evaluating findings. Several aspects of a student-centered classroom culture are prevalent, but their implementation is also less than comprehensive. On the one hand, students frequently engaged in discussion, and teachers typically encouraged active participation of all students and acted as facilitators. On the other hand, lecturing remained very prevalent, teachers infrequently encouraged students to generate conjectures or find alternate solutions, and student questioning did not drive the focus of the lesson. Organizational Support 6. To what extent does the school administration advocate, facilitate, and support

implementation?

7. To what extent are teachers participating in collaborative planning for inquiry? The evaluation focused on several aspects of organizational support, including the level of principal support for the program, material resources, coherence with other initiatives, and opportunities for job-embedded learning (also referred to as professional learning communities). In general, teachers reported low levels of support both from their principals and from the professional learning community in their schools. Both of these ratings were more negative than in Year 1. The involvement of principals was generally limited to providing resources and enabling teachers to attend professional development but did not extend to strongly advocating for the program or providing instructional leadership in the form of monitoring or feedback on instruction. Principals reported that they support the program by providing time for collaboration, but teachers typically did not report this. Although the ELI workshops themselves provided team meeting time, most teachers felt they lacked sufficient time to plan lessons or work with other teachers during the school day. Teachers did not regularly receive feedback on inquiry instruction, although they appreciated feedback when they did receive it. Several teachers cited this lack of support from principals and other teachers as a concern about implementation. Underscoring the importance of organizational support, teachers who reported higher levels of job-embedded learning tended also to report greater frequency of inquiry instruction.


Most teachers expressed concern about the lack of time for planning and implementing inquiry-based lessons. Teachers typically reported that their material resources were adequate. They expressed appreciation for science kits, mentioning that the kits make it easier for them to incorporate inquiry in their classrooms. In general, teachers reported strong levels of program coherence, with most teachers reporting that the program was consistent with their goals for their own professional learning. Some teachers, however, cited their difficulty with aligning inquiry instruction with existing curriculum requirements as a concern about implementation. Teacher Learning and Understanding 8. To what extent are teachers gaining the knowledge and skills to implement inquiry-

based science instruction? Teacher learning was evaluated with survey items focusing on understanding and self-efficacy. As a whole, teachers indicated that they understand inquiry instruction and the learning cycle and that the workshops were successful in explaining what both should look like in the classroom. Regarding self-efficacy toward different aspects of inquiry instruction, findings are mixed. About three fourths of teachers felt well prepared to implement inquiry-based instruction, and the proportion of such teachers increased from last year. About two thirds of teachers expressed confidence regarding their ability to align their curriculum units to the learning cycle. Just over half of teachers expressed confidence regarding evaluating student work in inquiry lessons; this proportion has increased somewhat since last year. Fewer than half of teachers felt prepared to differentiate instruction or align lessons to local standards, and these ratings did not change from Year 1. Elementary teachers felt more prepared than secondary teachers to implement inquiry instruction, differentiate instruction, and evaluate student work. In summary, it appears that most teachers understand what inquiry instruction looks like and have self-efficacy toward teaching lessons in this manner. However, teachers have less self-efficacy toward several related instructional tasks (evaluation, alignment to local standards, differentiated instruction) that are essential for full implementation. Reactions to Professional Development Workshops 9. What were the perceptions of the quality, utility, and effectiveness of the workshops?

10. In what ways should the professional development be improved? Attendance at the three winter/spring 2008 workshops varied by about 10 percentage points, with about 60 percent attendance in January, 70 percent in February, and 50 percent in April. The attendance rate of case study participants was 25 percent in January, and the same as the overall rate in the other two months. Quality of Presentation Overall, spring workshop participants considered the workshops to be of good quality. The overwhelming majority of respondents agreed that the content of the session was well organized, materials were clear, questions and concerns were addressed, and information was presented understandably. The February session was rated as particularly high regarding both quality and utility. Participants valued the opportunity to share and discuss real-world examples through


modeling (as in the February session) and lesson study (in April). Many respondents wanted to see more of these types of activities at future workshops. Respondents would like to receive more information in advance of the workshop, including agendas, meeting details, and a list of which materials to bring. Utility of Workshops Overall, respondents found the workshops useful, particularly in regards to the relevance of information and for providing tools and strategies for the classroom. However, different groups of teachers viewed the workshops as too basic and too advanced, respectively. That is, some respondents who believed that they were already familiar with inquiry saw the sessions as repetitive of what they had already learned. Other teachers who were new to inquiry would have liked an even slower pace to allow themselves to understand inquiry instruction more fully. Respondents found team-based planning useful, and the time provided for this activity in the latter two sessions seemed about right to them. Finally, respondents wanted more guidance for implementation. AEA teams in particular sought more guidance for their efforts to provide professional development to schools in their respective regions during Year 3. Ratings for the utility of team planning time and clarity of “next steps” for implementation improved between January and the final two workshops. Effectiveness Participants perceived the workshops as being most effective at increasing their understanding of how students learn and of the learning cycle. However, several teachers reported not yet clearly understanding the differences among the stages of the learning cycle, and they stated that they would like continued instruction on this topic. The workshops were perceived as less effective at promoting the learning goals of identifying students’ preconceptions and differentiating instruction, which seem to be more challenging instructional tasks. Recommendations Because the Every Learner Inquires initiative will scale up during Years 3 and 4, it is important to be explicit with those continuing and considering participation about what the program is; what its goals and milestones are; and what is required from teachers, principals, schools, districts, AEA teams, and the state. The findings suggest that the explicitness would increase implementation and impact through clear goals and clear expectations about work, contributions, support, and progress. Program leaders should provide more explicit guidelines for local support of the ELI initiative. At the minimum, schools ought to provide teachers with protected time on a monthly to weekly basis to collaborate and plan for implementation. The main recommendation is to promote job-embedded professional learning by setting clearer expectations for how teachers should work together, what they should accomplish, and how principals should support them. To this end, the workshops should provide teachers with a framework in which to engage in this learning, such as lesson study, and should guide participants in setting up an embedded professional development system in their schools. The involvement of outside science consultants also would be helpful in the lesson study process, through coaching and modeling. To enhance the focus on changes in teaching practice, program leaders should map out the goals


and milestones for teacher learning and implementation for the entire year. Participants should receive assignments between workshops to promote opportunities to practice what they have learned and to receive feedback. These expectations should be conveyed to schools continuing and considering participation in the initiative.


Introduction Every Learner Inquires (ELI) is a statewide K–12 professional development effort to improve science learning in Iowa schools by promoting inquiry-based instruction. The program, which runs from August 2006 through June 2010, seeks to promote an inquiry-based approach to science instruction, as outlined in the National Science Education Standards (National Research Council, 1996). This program provides several workshops per year, and participants are expected to engage in job-embedded professional learning during the time between workshops. The goals of the initiative are to improve science learning for all students, build teacher leadership and content expertise in the area of science, provide teachers with the content and skills necessary to implement inquiry-based instruction, and establish a structure for sustained implementation. Under contract from the Iowa Department of Education, Learning Point Associates is conducting a mixed-method evaluation that examines the following anticipated outcomes of the program:

• Teacher reactions to professional development.

• Changes in teacher understanding of inquiry-based instruction.

• Changes in classroom instruction.

• School experiences in supporting the program.

• Student learning outcomes.

The purpose of this report is to provide formative feedback on these outcomes during Year 2, based on findings from interviews, surveys, observations, and instructional logs. These findings are presented by section according to the five outcomes. A separate report, to be delivered in spring 2009, will present findings from an analysis of student achievement data to describe student learning outcomes. What Is Inquiry-Based Science Instruction? Inquiry-based science instruction is at the heart of the National Science Education Standards published by the National Research Council (1996) and of the Benchmarks for Science Literacy published by the American Association for the Advancement of Science (1993). The Standards define inquiry in the following manner:

Inquiry is a multifaceted activity that involves making observations; posing questions; examining books and other sources of information to see what is already known; planning investigations; reviewing what is already known in light of experimental evidence; using tools to gather, analyze, and interpret data; proposing answers, explanations, and predictions; and communicating the results. Inquiry requires identification of assumptions, use of critical and logical thinking, and consideration of alternative explanations. (p. 23)

The Standards for science content indicate that students should learn the process of scientific inquiry and are based on the assumption that students need direct experience with scientific inquiry to gain the understanding and skills necessary to engage in such activities. Moreover, they are based on the assumption that teachers need to introduce students to the logic and


meaning of the process of inquiry, so that students can understand how inquiry activities allow them to arrive at scientific understandings. For these reasons, the Standards articulate both skills and understandings relating to inquiry that students should attain at different grade levels. Across all grade levels, the Standards articulate the following essential features of inquiry instruction:

• Learners are engaged by scientifically oriented questions.

• Learners give priority to evidence, which allows them to develop and evaluate explanations that address scientifically oriented questions.

• Learners formulate explanations from evidence to address scientifically oriented questions.

• Learners evaluate their explanations in light of alternative explanations, particularly those reflecting scientific understanding.

• Learners communicate and justify their proposed explanations. Not every teaching style is equally suited for affording students the opportunity to engage in inquiry. For these reasons, the Standards (in Chapter 3) describe what teachers should know and do in their planning, instruction, assessment, and collegial relationships in order to support inquiry instruction. These standards describe several aspects of a student-centered classroom culture that promote opportunities to engage in inquiry. For example, teachers should serve as facilitators of learning such that they promote discourse and discussion, encourage curiosity and questioning, and respect the diversity of their students. Also, teachers should promote learning communities in which intellectual rigor is respected, students collaborate on projects, and where students’ interests may determine the focus of classroom investigations. These student-centered teaching practices are considered to be necessary conditions to promote inquiry learning. In light of the fact that teachers need guidance and structure to enact the standards, the ELI program provides a pedagogical model for lesson design. This model, called the 5E Learning Cycle, articulates five phases of inquiry, with each phase corresponding to a different learning goal. The model aligns with a student-centered, constructivist approach to science education. To this end, the model describes the teacher and student behaviors at each stage that will promote high levels of student involvement with generating and discussing questions, ideas, and explanations. The five phases in this model are as follows:

• Engage. Students become interested in a problem or a phenomenon and consider what they currently know about it.

• Explore. Students conduct investigations or develop common experiences so they can compare results and share ideas.

• Explain. Students explain concepts and ideas in their own words and use data from their investigations to support their explanations. Students learn to use appropriate scientific terms.

• Elaborate. Students apply what they learned to a new situation and draw conclusions based on evidence.


• Evaluate. Students compare their conclusions to those of others and perhaps revise their explanations. Students generate new questions that lead them into a deeper exploration of the topic.

Program Design Participation in the ELI program is organized by teams. There are two types of teams: case study school teams and area education agency (AEA) leadership teams. Four schools have been chosen for the first cohort, to act as case studies with the purpose of providing formative data about how schools implement and support the initiative. Case study schools agreed to implement the instructional strategies presented in the professional development, keep logs of their implementation, allocate inservice time to supporting implementation, and cooperate with data collection for the evaluation. All science teachers in each case study school are expected to attend the workshops and participate in regular team meetings to discuss assignments that are part of the workshops, as well as other topics related to program implementation. This model of school implementation includes several components that research suggests may promote implementation of inquiry instruction, elaborated as follows:

• Collective participation of teachers from the same school (Desimone, Porter, Garet, Yoon, & Birman, 2002). Case study schools are expected to involve the entire science teaching staff and building principal in the ELI initiative over a four-year period. As part of this commitment, all science teachers and the building principal are expected to participate in the professional development workshops.

• Job-embedded professional learning, such as observations, collaborative lesson planning, mentoring, or study groups (Desimone et al., 2002). As a condition of their participation, case study schools commit to conducting follow-up planning meetings at their school. The extent to which teachers engaged in job-embedded learning outside of these planning meetings was left of to the individual schools.

• Coherence with teachers’ own goals for professional learning and with other programs, standards, and requirements (Penuel, Fishman, Yamaguchi, & Gallagher, 2007). Although there are no requirements per se related to program coherence, case study schools agreed to commit inservice time to this program, and principal participation should encourage coherence with other programs in the school.

The 11 AEA leadership teams also are participating in the full four-year cycle of professional development. These teams include AEA science education staff and at least two teacher leaders drawn from schools within the jurisdiction of the given AEA. The purpose of the participation of these AEA teams corresponds to the Teacher Learning goal of building leadership throughout the K–12 educational system in Iowa. These teams will participate in the ELI program in order to build their own capacity to deliver professional development in inquiry instruction. During the third and fourth years of the initiative, the AEA teams are expected to begin to provide professional development for schools in their respective areas, building on the lessons learned from the case study schools.


Program Goals Several stated goals of the ELI program encompass student learning, teacher learning and practice, and organizational change. These goals were expressed in ELI documents as follows:

• Teacher Practice goal: Implement inquiry-based instruction.

• Organizational goal: Establish a structure that sustains the implementation of ELI.

• Teacher Learning goal: Build teacher leadership and content expertise within the system.

• Student Learning goal: Improve science learning for all K–12 students in the state of Iowa.

The following paragraphs elaborate on these goals, the program’s approach to accomplishing them, and the approach for evaluating their accomplishment. Teacher Practice Goal The Teacher Practice goal is to implement inquiry-based instruction in science classrooms at all school levels. The evaluation focused on the extent to which teaching practice reflected inquiry-oriented instruction, the extent to which the classroom culture supported inquiry instruction, and the extent to which teachers adapted their lessons to the 5E Learning Cycle model. In addition, the evaluation included in its focus the extent to which teachers differentiated instruction in the context of inquiry lessons. Organizational Goal The evaluation focused on several aspects of the Organizational goal, including the level of principal support for the program, material resources, coherence with other initiatives, and opportunities for job-embedded learning (also referred to as professional learning communities). Principal support is the degree to which the principal articulates a vision for inquiry-oriented instruction and advocates and promotes participation in the ELI program as well as changes in teacher practice. Material resources include lab equipment, lab space, aligned curriculum materials, and kits or consumable laboratory supplies. The other two aspects under this goal (i.e., coherence and job-embedded learning) were described above in the discussion of team organization (see Program Design). Teacher Learning Goal In line with the changes in teacher practice outlined above, the evaluation addressed two aspects of teacher learning: understanding and self-efficacy. The evaluation focused on whether teachers had acquired a fundamental understanding of inquiry instruction and the learning cycle. In the context of this program, self-efficacy refers to whether teachers perceive that they are prepared to implement the instructional strategies and approaches that are the intended goals for teacher practice.


Student Learning Goal The Student Learning goal is to improve science education for all learners in Grades K–12 in the state of Iowa. However, this goal may be understood in several different ways. One intended outcome is to increase the scientific understanding of students. A related outcome is to increase the ability of learners to engage in the scientific inquiry process. Other outcomes relate to improved attitudes of students toward science learning, and stronger engagement in learning science. The evaluation examined each of these types of student learning outcomes through two methodologies. This report presents data from a descriptive study of teacher and student opinions regarding these student learning outcomes. The evaluation also has included a statistical analysis of student science achievement over time, as measured by annual standardized tests. This latter study was recently completed for Year 1 of the program and was submitted as a separate report to the Iowa Department of Education in May 2008; the Year 2 achievement report is scheduled to be submitted in April 2009. Design of Professional Development The cycle of workshops during Year 2 was intended to build upon the previous year. As in every year of the program, professional development was delivered through a summer institute and whole-day workshops during the academic year. During the time between these semimonthly workshops, participants were assigned to collaborate on lesson design, reflect on their learning, and implement different aspects of inquiry instruction. Summer Institute and Workshops Most of the interactions between teams and project staff (i.e., the design team) occurred during professional development workshops. There was a four-day “kick-off” workshop in July and five one-day “academic-year” workshops that occurred in September, November, January, February, and April. The September, January, and February workshops were delivered in both elementary and secondary strands, with teachers choosing the strand that corresponded to their grade level. To facilitate travel, each of the academic-year training workshops was presented separately to participants in the eastern and western halves of the state. During the workshops, teachers participated in several types of interactive learning activities that were designed to increase their level of involvement. For example, during some workshops, participants experienced an inquiry-based science lesson from the perspective of a student. Following presentations from the instructors, participants would typically engage in a group or paired discussion. Participants engaged in written reflection several times per workshop. For example, they were asked to enter a “line of learning” in project journals, in which they described their current understanding of a particular topic. They also engaged in “quick writes” to answer a particular question, designed to engage the participants in a particular topic. Postworkshop Assignments Following each workshop, teachers had an assignment to complete during the interval before the next workshop. The purpose of these assignments was to reinforce the learning that had occurred


during the workshop. The major assignment during Year 2 was to design and implement two full learning cycles. School- and Web-Based Meetings Case study teams met with varying frequency to discuss assigned readings or other assignments and to develop lesson plans. Some of the case study teams posted minutes from these meeting to a Web-based discussion board. Each case study and AEA team had its own discussion thread on this Web board. Its main purpose was to facilitate communication among team members. Workshop Topics Each workshop focused on a particular aspect of inquiry instruction. In addition, the workshops provided guidance on different techniques and fundamental understanding related to inquiry. Descriptions of these topics follow. How Students Learn. The professional development introduced research about how students learn in relation to three critical learning needs: (1) addressing preconceptions of learners, (2) building conceptual structures to organize factual information, and (3) fostering self-monitoring of learners. Facilitation Strategies. The workshops included Accountable Talk presentations, a collection of techniques for four group-facilitation purposes: supporting group discussion, supporting accountability to the learning community, supporting accountability to accurate knowledge, and supporting accountability to rigorous thinking. Strategies for Differentiation. The professional development included training on how to address the needs of all learners. Avenues for differentiation included allowing students to choose different assignments and providing background reading according to student reading level. Building Professional Learning Communities. The workshops presented information about characteristics of collaborative groups, in order to build the capacity of each of the teams to engage in collaborative learning activities. For example, during the November workshop, facilitators modeled a process for reflective dialogue about science lessons. Evaluation Overview The evaluation study of Year 2 of ELI focused on the four program goals. Several data-collection methods provided data on progress toward these goals. During site visits conducted by the evaluator, case study schools’ staffs participated in interviews and classroom observations. Case study teachers also completed implementation logs, and students in these schools completed a survey about their science classroom experiences. AEA teachers joined case study teachers in completing a survey about their experiences with science instruction and with the ELI program. The evaluation instruments addressed the specific evaluation questions related to the program



goals. Table 1 aligns each evaluation question with the data source that addresses it; these data sources are described in the Methods section. In addition to the focus on the four program goals, this report provides formative feedback on the professional development workshops offered during the winter and spring of Year 2. (A previous report summarized reactions to the workshops offered during the fall of Year 2.) This feedback is based on surveys filled out by participants of the January, February, and April workshops. It addresses the quality, utility, and effectiveness of the workshops and provides suggestions for improving these workshops in the future.

Table 1. Crosswalk of Evaluation Questions to Data Sources

Evaluation Question Teacher Survey

Teacher Interview

Principal Interview

Obser-vation

Teacher Logs

Student Survey

Reaction Form

ITBS Data*

1. To what extent have students acquired the abilities and fundamental understandings about scientific inquiry? x x

2. What is the impact on student enthusiasm and self-efficacy toward science learning? x x x x

3. Does the program address the needs of all student subgroups? x x x

4. Are the scientific understandings and abilities of students improving? x x x

5. To what extent has teacher practice changed to reflect an inquiry-based approach? x x x x x x

6. To what extent does the school administration advocate, facilitate, and support implementation? x x x

7. To what extent are teachers participating in collaborative planning for inquiry? x x x

8. To what extent are teachers gaining the knowledge and skills to implement inquiry-based science instruction? x x x

9. What were the perceptions of the quality, utility, and effectiveness of the workshops? x x x

10. In what ways should the professional development be improved? x x x

*Iowa Tests of Basic Skills (ITBS) student achievement data for Year 2 will be collected in fall 2008 and is not included in this report.


Methods ELI Workshop Reaction Data This report summarizes participant reactions to three ELI workshops presented in winter and spring of 2008, along with the attendance and topics addressed in these workshops. Workshop Feedback Forms Program leaders distributed and collected workshop feedback forms after each workshop. The purpose of the postworkshop feedback form was to assess participant satisfaction with the professional development sessions. The form asked participants to rate items aligned with four topics: (1) workshop quality, (2) workshop utility, (3) the amount of time spent on different formats of activities, and (4) the extent to which the workshop improved understanding or ability in regards to five learning goals. The form also presented five open-ended questions designed to elicit feedback regarding overall strengths and areas for improvement of the workshop. The facilitators distributed the postevent reaction forms to all participants at the end of the workshop. All participants had the opportunity to complete the feedback form, including AEA science consultant staff, teachers, and administrators. Evaluators completed descriptive analyses of the Likert scale items (multiple choice) from the feedback forms, and they identified themes and tones (positive or negative) in participant comments within and across workshops. Workshop Agendas Workshop agendas were analyzed to categorize the planned activities based on topic and method of delivery. Categories of workshop topic were derived inductively. The categories of delivery method were determined a priori to include the following: lecture, small or large group discussion, planning (including lessons and team planning), and modeling or demonstrations of lessons and instructional methods. Workshop Attendance Participants at each workshop were asked to sign in to indicate their attendance. The attendance was tracked across sessions of the workshops and disaggregated by team type (AEA or case study). Teacher Survey Learning Point Associates developed a teacher survey to address the four goals of the ELI program: teacher learning, implementation of inquiry instruction, organizational support, and student impact. For each construct described in the introduction, the survey included several items to reflect different details or elements of the construct. Psychometric procedures validated that these items could be combined into a single scale score. The following paragraphs describe how each construct was evaluated with the survey. A more extensive description of the instrument and its validation appears in Appendix A.


Teacher Learning The survey included several items to evaluate the extent to which teachers gained the knowledge and skills to implement inquiry-based science instruction. Survey items asked teachers to rate their level of understanding of both the National Science Education Standards and the learning cycle, and the extent to which the workshops provided sufficient information on what inquiry instruction and the learning cycle look like in the classroom. In addition, teachers rated their level of preparation to implement inquiry learning, modify curriculum units to align with the learning cycle, differentiate instruction while doing inquiry, evaluate student work from inquiry-based lessons, and align inquiry lessons to state standards. Organizational Support The survey included four groups of items that were used to create scales addressing professional learning communities, principal support, resources, and coherence. Each scale is described in more detail, as follows:

• Professional Learning Community. Eight items asked teachers to rate their level of agreement with statements about professional learning opportunities in their school, for example, “Science teachers in this school regularly observe each other teaching classes in order to learn how to implement inquiry-based lessons.” This scale was included on the Year 1 survey and was compared across years.

• Administrator Support. Nine items asked teachers to rate the extent to which administrators support inquiry learning. The stem “To what extent does your principal, assistant principal, or department head:” introduced the items. Examples of these items include “understand inquiry-based instruction” and “monitor teacher implementation of inquiry learning.” This scale was included on the Year 1 survey and was compared across years.

• Resources for Inquiry Instruction. Eight items asked teachers to rate their agreement with statements about whether they had adequate resources for inquiry instruction, for example, “I have sufficient laboratory equipment to support inquiry learning.”

• Coherence. Five items asked teachers to rate the extent to which the ELI program is consistent with other programs, reform efforts, standards, and assessments. The stem “To what extent is Every Learner Inquires consistent with each of the following:” introduced the items. Examples of the items include “Other science professional development programs at your school or district” and “State and district standards and curriculum frameworks.”

Teacher Practice Several items examined the frequency with which teachers used certain strategies—or with which students engaged in certain activities—aligned with inquiry-based learning. When answering these items, teachers were directed to describe a particular class of theirs. There were three groups of items of this type:


• Inquiry Instruction Scale. Teachers rated the frequency with which students in their class took part in 15 types of science activities aligned with the essential features of inquiry learning. These items were introduced with the question, “About how often do students in this class take part in the following types of science activities?” Examples of items include “Critically examine the scientific explanations of other students” and “Record, represent, or analyze data.” This scale was included on the Year 1 survey and was compared across years.

• Classroom Culture Scale. Teachers rated the frequency with which they incorporate nine student-centered instructional practices in their science instruction. An example of an item of this type was “Assign students to work in groups on projects.” This scale was not included on the Year 1 survey.

• Learning Cycle Implementation. Three items referred to implementation of the learning cycle. Teachers reported the number of times they taught a complete learning cycle in science during Year 2. Another item asked if they experienced any barriers to teaching a full learning cycle (with response options of Yes or No), and if so, to describe the barriers in an open-ended fashion.

Student Learning Four items asked teachers to rate the extent to which their implementation of inquiry learning has improved student learning outcomes. These outcomes were engaged learning, understanding of science concepts, enthusiasm, and understanding of inquiry. Teachers rated the extent of improvement with the response options of Not at all/Slightly, Somewhat, Moderately, and Very much so. These items reflect different aspects of student learning outcomes and were each analyzed individually. They were included on the Year 1 survey, and responses were compared across years. Response Rate Of the 80 teachers who were contacted, 46 teachers completed the survey, for an overall response rate of 57 percent. The response rate did not vary by team type; 12 of 21 case study teachers and 34 of 59 AEA teachers completed the survey. Student Survey The student survey was administered during April 2008 to students who were taking a science class in the case study schools in Grades 3 and above (see Appendix A for a discussion of response rates along with a more extensive description of the instrument). Learning Point Associates developed the student survey based on items from previously published instruments. All items on the survey had a response scale of Yes, No, and Sometimes. Each of the following constructs was combined into a single scale score (all except the self-efficacy scale were included on the Year 1 student survey):

• Inquiry Instruction. Ten items from the Investigation scale of the What Is Happening in this Classroom (WIHIC) questionnaire (Fraser, McRobbie, & Fisher, 1996) were used to


address the construct of inquiry learning. A sample item from this scale is “I carry out investigations to test my ideas.”

• Classroom Culture. Nine items from the Involvement scale of the WIHIC questionnaire were used to address the construct of classroom culture. A sample item from this scale is “I talk with other students about how to solve problems.”

• Enthusiasm Toward Science. Six items from the Test of Science-Related Attitudes (TOSRA) (Fraser, 1982) were used to measure a student’s enthusiasm toward science. A sample item of this sort is “I look forward to science lessons.”

• Self-Efficacy for Science. Six items asked students to rate whether they can accomplish certain science classroom skills. Each of these items started with the phrase “I can” and was completed with phrases such as “ask a scientific question” and “collect data to answer a question.”

A psychometric analysis comparing responses of elementary and high school students determined that these two groups responded differently to the survey items. For this reason, the scale scores from these two groups are reported separately. Instructional Logs Instructional logs were used to collect teachers’ self-reported data on their use of inquiry-oriented instruction. The logs were revised from last year’s evaluation to reflect feedback from members of the ELI design team. Overall, the log form required teachers to indicate whether different student or teacher activities were present during a particular lesson. For the purpose of this report, the term lesson denotes the instruction occurring in a single class period. Teachers in case study schools completed one log per lesson for 10 consecutive lessons during November 2007. The first section of the form, corresponding to student activities, was adapted from the Local Systemic Change Classroom Observation Protocol (Horizon Research, 2005). This section listed the following four categories of student activities (number of specific activities in each category in parentheses):

• Listened to a presentation. (3)

• Engaged in discussion/seminar. (2)

• Engaged in reading/reflection/written communication about science. (6)

• Engaged in problem solving/investigation. (12) Under each category were a number of specific activities. For example, the category of “Engaged in reading/reflection/written communication about science” included activities such as Read about science and Answered textbook/worksheet questions. The category of “Engaged in problem solving/investigation” included activities such as Worked with manipulatives and Recorded patterns, cycles, or trends. Teachers indicated which student activities occurred in the class session.


The second section of the form listed teacher activities, each of which corresponded to one of the five phases of the learning cycle: Engage, Explore, Explain, Elaborate, and Evaluate. For example, the activity Piqued students’ curiosity and generated interest corresponded to the Engage stage; Provided time for students to puzzle through problems corresponded to the Explore stage; Requested justification (evidence) for students’ explanations corresponded to the Explain stage; Encouraged students to apply what they have learned to a new situation corresponded to the Elaborate stage; and Encouraged students to assess their own progress corresponded to the Evaluate stage. These examples were taken from a description of the 5E Learning Cycle (National Institutes of Health, 2005). Teachers checked each teacher activity that occurred during each lesson, along with the phase of the learning cycle to which the activity belonged. Response Rate The evaluation team sent packets of instructional logs to every classroom science teacher in each of the case study schools (a total of 20 teachers1). A total of 9 teachers completed all 10 forms for a response rate of 45 percent. One of these teachers completed two sets of instructional logs, one for an upper elementary class and one for a lower elementary class. Analysis The items in the first two categories (regarding presentations and discussion) align with the classroom culture construct, and the second two categories (reading/writing and problem solving/investigation) align with the inquiry instruction construct. The category of the activity was considered to have occurred if any of its activities were reported to have occurred. For each teacher, evaluators calculated the percentage of lessons that the teacher used each activity and category of activity. These percentages were then averaged across the 10 teachers. Classroom Observations The evaluation team used an observation protocol that was adapted from previously used instruments. The first two sections of this form were identical to the instructional log in all respects except one: In the section on teacher activities, the observer rated the degree of emphasis each phase of the learning cycle received in the lesson (in the instructional log, teachers simply indicated whether a lesson addressed each phase of the learning cycle or not). The remaining sections of the observation form were identical to the Reformed Teaching Observation Protocol (RTOP) (Piburn & Sawada, 2000). According to the RTOP Training Guide (Sawada et al., 2000), the RTOP is

an observational instrument that can be used to assess the degree to which mathematics or science instruction is ‘reformed.’ It embodies the recommendations and standards for the teaching of mathematics and science that have been promulgated by professional societies of mathematicians, scientists and educators. (p. 1 )

1 One case study teacher who was invited to participate in the teacher survey was not asked to complete the instructional logs on the grounds that this teacher did not teach a regular class.


Using the RTOP, trained observers rate a series of statements about classroom instruction on a 5-point scale, where 0 corresponds to “Never occurred” and 4 corresponds to “Very descriptive.” These statements correspond to three broad topics: (1) Lesson Design and Implementation; (2) Content, with subsections on Propositional Knowledge and Procedural Knowledge; and (3) Classroom Culture, with subsections on Communicative Interactions and Student/Teacher Relationships. The third section aligns with the evaluation construct of the same name. The Content section aligns with the construct of Inquiry Instruction. However, the various items in the Lesson Design and Implementation section align with different evaluation constructs, and each will be reported in its appropriate place. Staff from Learning Point Associates visited all four case study schools during April 2007 and observed a total of 12 lessons (three in each school). There was one observer per lesson. In only one school was it necessary to sample, and in this school, a stratified random procedure was used to ensure that the three observed teachers were at different grade levels. Analysis of Observation Data The analysis of the observation data differed by section of the observation form. For the section on student activities, the results describe the percentage of lessons in which a particular activity was observed. For the section on teacher activities, each phase of the learning cycle was rated on the scale described in Table 2. The analysis of this section involved calculating the percentage of lessons that received a rating of 2 or above. The analysis of the RTOP followed the same procedure.

Table 2. Rating Scale for Level of Emphasis of Learning Cycle Phases

Rating Explanation of Rating (0) Not Observed This phase was not observed.

(1) Slight This phase received isolated use and/or little time in class. Clearly not a focus of the lesson.

(2) Modest This phase received modest time and emphasis in class. A minor focus of the lesson.

(3) Moderate This phase received substantive time or emphasis in the class. A main focus of the lesson.

(4) Extensive This phase received substantive time or emphasis in the class. The major focus of the lesson.

Interviews With Principals and Teachers During site visits to each case study school in April 2007, staff from Learning Point Associates conducted semistructured interviews with the three observed teachers and the principal. The interview questions focused on all five levels of the evaluation (reactions to professional development, teacher learning, change in teacher practice, organizational support, and student learning). The teacher interview included several questions to clarify what took place during the observation (e.g., the lesson’s purpose). After the first site visit, five additional questions were


added to the teacher interview protocol and were presented to the teachers at the remaining three schools. The questions on the principal interview closely paralleled those on the teacher interview, although the latter included more specific and detailed questions about the teacher’s level of learning and degree of implementation. Interview responses were content analyzed to identify the major themes reported by coaches and principals. The interview protocols are included in Appendix D. Analysis of Interviews The analysis of qualitative interview data utilized an inductive approach that incorporates systematic methods of managing data through reduction, organization, and connection (Dey, 1993; LeCompte, 2000). This process relied upon systematic procedures for coding and categorizing the data to recognize patterns within and across schools. The process began by transferring all interview notes to electronic text files. The Learning Point Associates research team inductively analyzed these notes by scanning the data for categories of phenomena and for relationships among such categories.


Section 1: Workshop Feedback Summary The evaluation addressed two questions related to the reactions to the professional development:

• What were the perceptions of the quality, utility, and effectiveness of the workshops?

• In what ways could the professional development be improved? The central program activity for the ELI initiative is the professional development offered during five full-day workshops over the course of the year. These workshops utilized several different learning formats to address various learning goals. Several data sources were used to evaluate the quality, relevance, and utility of these workshops because of their centrality for the initiative. Responses to postevent workshop feedback forms provide data on participant opinions of the workshop’s quality and utility, balance of time devoted to different activities, and amount learned. In the context of each of these items, participants provided feedback on ways the professional development could be improved. Workshop attendance is summarized to indicate the breadth of participation in the professional development within schools. The topics and formats of these workshops also are described based on workshop agendas. Previous reports, submitted in October 2007 and January 2008, summarized feedback regarding these same topics following the 2007 ELI summer institute and the September and November 2007 workshops. Where relevant, this report highlights important changes in participant feedback about the professional development from the 2007 workshops to those in early 2008, though the report mainly analyzes responses of the 2008 workshops. Workshop Attendance Overall attendance rates have remained relatively stable at about 60 percent on average, though the attendance of case study participants varies widely. Response rates increased in the beginning of 2008 but dropped off throughout the following months. Attendance at the January and February workshops varied by about 10 percentage points, with about 60 percent attendance in January, 70 percent in February, and 50 percent in April. The overall attendance rates in the spring were very similar to that of November, in which about 60 percent attended. The attendance rate of case study participants varied widely, with about 25 percent attending in January, 70 percent attending in February, and about 50 percent attending in April. About 50 percent had attended the November workshops. One of the January workshops was canceled due to weather, as well as several in February. The canceled February sessions were made up, but the January session was not. Because of this, several case study participants did not have an opportunity to attend in January. Table 3 presents the attendance at the November workshops and the response rates for the post-workshop feedback forms. It serves as a comparison for the spring 2008 workshop attendance and response rates, which are presented in Tables 4–6.


Table 3. Attendance and Response Rates for November 2007 Workshops

Team Type N Total Attendance

Attendance Rate

Returned Forms

Response Rate

AEA 118 75 64% 38 51% Case study 25 12 48% 6 50% Total 143 87 61% 44 51%

Table 4. Attendance and Response Rates for January 2008 Workshops


Attendance Rate

Returned Forms

Response Rate

AEA 106 72 68% 51 71% Case study 20 4 20% 3 75% Total 126 76 61% 54 71% Note: Nine AEA members attended both January workshops, but they are counted only once. The West secondary workshop was canceled due to weather, so 12 AEA members and 5 case study participants who are secondary only are not included in the total attendance.

Table 5. Attendance and Response Rates for February 2008 Workshops


Attendance Rate

Returned Forms

Response Rate


Note: Twelve AEA members attended multiple workshops, but they are counted only once. The East workshops were canceled due to weather in February, so totals include the attendance at a makeup session in April.

Table 6. Attendance and Response Rates for April 2008 Workshops


Attendance Rate

Returned Forms

Response Rate


Note: Twelve AEA members attended multiple workshops, but they are counted only once. Due to the need to reschedule the canceled February workshop in April, the participants in the East region participated in the “April” workshop (depicted in this table) in May.

Workshop Topics and Activities Table 7 presents the number of hours focused on specific workshop topics. Planned topics included general program (basic information about ELI and upcoming events), the learning cycle, and the planned rollout (expansion of the program to schools throughout the state).


According to the agendas, the use of time varied significantly among workshops. The majority of the time in January and February was spent focusing on the learning cycle, at over four hours per session. In April, time was split between the learning cycle and the upcoming rollout of the program.

Table 7. Number of Hours Devoted to Various Topics, Spring 2008

Topics January February April General program .5 .5 .5 Learning cycle 4.25 4.5 1.5 Scale-up/rollout 0 0 2.5

Table 8 details the types of workshop activities, including lecture, small or large group discussion, planning (including lessons and team planning), and modeling or demonstrations of lessons and instructional methods. The activities also varied among workshops, though slightly less so. The majority of time in January was spent in lecture, at a little over three hours, and about 1½ hour was spent in discussion, whether in small group or large group. No time was spent in planning, and only 30 minutes were spent in modeling activities. In February, the majority of time was spent in a modeling activity; participants also spent just under an hour each in discussion and planning activities. Only 30 minutes were spent in lecture. Finally, in April, time was more evenly split, with about 1½ hour each in planning and modeling, and about an hour in lecture. Only about 30 minutes were spent in discussion.

Table 8. Number of Hours Devoted to Various Activities, Spring 2008

Topics January February April Lecture 3.2 0.5 1.0 Discussion 1.5 0.8 0.5 Planning 0.0 0.8 1.5 Modeling/demonstration 0.5 3.0 1.5

In summary, the amounts of time devoted to specific topics and in specific activities varied by workshop. Organizers generally spent more time focusing on the learning cycle in the beginning two sessions, and then split time between the learning cycle and the rollout for the final session. Organizers planned more lecture and discussion activities in January and then switched to more modeling and planning for the final two sessions. Ratings of Quality and Utility Teachers expressed their opinions on the quality and utility of the workshops on feedback forms and in teacher interviews. On the feedback forms, teachers also rated the effectiveness of the workshops in terms of the degree to which their understanding or ability improved. Because this feedback was provided as a reaction to a particular session, it is presented here. The next section of this report describes overall opinions on the degree of teacher learning.


Workshop Quality On the feedback forms, workshop participants rated their level of agreement with four statements of workshop quality: whether the content of the session was well organized, materials were clear, questions and concerns were addressed, and information was presented clearly. Overall, spring workshop participants were satisfied with the quality of the workshop sessions, although February respondents generally gave higher marks to the sessions and, compared with both January and April, a greater percentage of respondents selected Strongly Agree for each item. The findings, also summarized in Table 9, are as follows:

• At least 90 percent to 100 percent of respondents from each month’s workshops selected Agree or Strongly Agree for each of the four statements.

• For all four items, higher percentages of February respondents selected Strongly Agree than respondents in January and April. In three of the items (content was well organized, materials were clear, information was clearly presented), at least 70 percent selected the highest rating. This was significantly higher than that for January and April respondents; for the four items, about half of January and April respondents selected Strongly Agree.

Table 9. Overall Ratings of Workshop Sessions, Spring 2008

Please rate your level of agreement with the following statements:

Month N Strongly Disagree Disagree Agree Strongly

Agree N/A

The content of the session was well organized.

Jan. 55 2% 2% 42% 55% 0%

Feb. 69 1% 0% 25% 74% 0%

April 46 7% 0% 39% 54% 0%

Materials (manuals, handouts) were clear.

Jan. 55 2% 4% 49% 45% 0%

Feb. 69 1% 0% 29% 70% 0%

April 47 6% 0% 30% 40% 23%

My questions and concerns were addressed.

Jan. 54 2% 2% 57% 39% 0%

Feb. 67 2% 0% 42% 54% 3%

April 46 7% 4% 37% 41% 11%

The information in the session was presented in a clear and comprehensible manner.

Jan. 55 0% 2% 38% 60% 0%

Feb. 70 0% 0% 26% 70% 4%

April 45 7% 0% 33% 53% 7%


These differences in respondents’ perceptions of the quality of the workshops also can be seen in the responses to the question, “How could this workshop have been improved?” Across the sessions, more comments were positive in February and April. Furthermore, in January more specific content suggestions were made. Details follow:

• In general, greater shares of participant comments were positive in February and April, about 40 percent and 50 percent respectively, compared with 13 percent in January. Comments included “I enjoyed this session above all others,” “It was great today,” and “I liked this one.”

• In January, the majority of participants (about 60 percent) made suggestions about content and activities. The two most common suggestions regarded including more time to meet with their team, mentioned by one fourth of respondents, and more opportunities for practice, mentioned by one fifth of respondents. For example, one January participant said the sessions should “somehow incorporate ‘doing’ and learning.’” About 30 percent of respondents in February and April discussed content, although no more than two agreed on a common area for improvement.

• Regarding the February and April sessions, participants made many positive comments about the activity in which one participant shared a video of a lesson and walked the teachers through the activity, after which the group critiqued and discussed the lesson. One wrote, “This is a model workshop that should be used for all teacher workshops in the future of ELI. Let it be teacher led.”

• Compared with January, a greater share of suggestions for improvement in both February and April dealt with logistics than with content. In February and April, about one fifth of respondents respectively mentioned improving room logistics, such as the room being to cold, and improved communication between sessions, such as informing teachers about the agenda and which materials to bring in advance. One teacher explained, “[The] May 6 [workshop] was a total surprise, as was the information that teachers are presenting on May 6. Please be aware of what you have told us and what you haven’t. The Web board doesn’t seem to be used for dissemination as it was supposed to be.” While these types of comments were more common in February and April, it is important to note that logistical issues were mentioned for all months, including about 15 percent of responding participants in January.

In summary, spring workshop participants considered the workshops to be of good quality, though the February session received higher ratings than those in January and April. Feedback in open-ended comments was more positive in February and April than January. February and April participants appreciated opportunities for hands-on learning, especially a video feedback activity in February. The most common suggestion for improvement from January respondents regarded content issues, whereas in February and April the most common comments were generally positive. Respondents from all three months mentioned logistical and communication problems, but greater shares of February and April respondents discussed such issues.


Balance of Time Devoted to Workshop Activities Through feedback forms, workshop participants rated the amount of time spent on four types of workshop activities: lectures, small-group discussions, team-based planning, and participating as a science learner. For each of these, participants indicated whether the amount of time was Much More Than Needed, Somewhat More Than Needed, About Right, Somewhat Less Than Needed, or Much Less Than Needed. Overall, spring workshop respondents felt the amounts of time dedicated to the activities were appropriate, although the percentage of respondents selecting About Right increased with each month. The results, also summarized in Table 10, are as follows:

• To describe the time spent on lectures, small-group discussions, and participating as a science learner, between 80 percent and 100 percent of respondents for each month selected About Right, and the percentages continually increased. As previously discussed, the amount of time spent on these activities changed from January to April, with more time spent in lecture and discussion in earlier sessions, compared with more time spent focusing on planning and modeling/demonstrations in February and April. The change in participant reactions to time may be a reflection of this shift in the types of activities.

• Participants appreciated the time to collaborate and participate in hands-on activities. In teacher and principal interviews, 8 out of 13 respondents commented on the value of seeing practical applications of techniques, such as examples of inquiry lessons or participating in a lesson. When asked how the workshop could be improved, about 10 percent of January workshop participants mentioned collaborating with other workshop attendees, and 20 percent of respondents mentioned the need for more opportunities for hands-on learning. In contrast, more time spent on modeling and demonstrations in February and April may have led to higher percentages of participants rating About Right, and open-ended comments indicate that many appreciated the activities.

• ELI participants appreciated the time to talk with their colleagues. In interviews, nine said one benefit of the workshops is conversations with other professionals, both their school coworkers and their colleagues in other districts. A teacher from Poplar School commented, “I really enjoy getting to talk with teachers who are out of our district and professionals—and just kind of pick their brain.”

• In one area, time dedicated to team-based planning, there were dramatic differences in January compared with February and April. About half of the January respondents indicated that the time on this activity was Somewhat Less Than Needed or Much Less Than Needed, and one fourth of January respondents mentioned the idea in their responses to the question, “How could this workshop have been improved?” The amount of time devoted to planning changed between the three workshops, with no time in January compared with nearly an hour in February and 1½ hour in April. This change may be reflected in participant reactions. The majority of February and April respondents, about 80 percent and 100 percent respectively, felt the amount of time for team-based planning was About Right, and respondents reacted positively to team time in their comments. Among April respondents, 30 percent mentioned meeting with their team as the most important thing from the workshop; one respondent wrote, “Thanks for the team time!!”


Table 10. Ratings of Time Spent on Workshop Activities, Spring 2008 Please rate the amount of time spent in the workshop on the following activities:

Month N

Much More Than

Needed

Somewhat More Than

Needed

About Right

Somewhat Less Than

Needed

Much Less Than

Needed

Lectures

Jan. 55 0% 11% 82% 5% 2%

Feb. 70 0% 6% 93% 1% 0%

April 44 0% 0% 100% 0% 0%

Small-group discussions

Jan. 55 2% 5% 80% 13% 0%

Feb. 70 0% 3% 84% 13% 0%

April 43 0% 0% 100% 0% 0%

Team-based planning

Jan. 51 2% 6% 41% 29% 22%

Feb. 65 0% 2% 82% 15% 2%

April 46 0% 0% 98% 2% 0%

Participating as a science learner

Jan. 55 4% 2% 80% 15% 0%

Feb. 67 0% 0% 94% 5% 2%

April 44 0% 0% 91% 7% 2%

In summary, participants were satisfied with the amounts of time spent on various activities, although respondents were most satisfied with the time for April workshop activities. Respondents expressed a need for working with their teams, and the opportunity to focus on the practical implementation of inquiry is something the February and April participants particularly enjoyed.

Ratings of Workshop Utility On the feedback forms, participants rated four aspects of the utility of the workshops: relevance to teaching needs, providing teaching tools and strategies, team-planning times for implementing what was learned, and clear directions for implementation. Overall, the ratings for these items were high for all of the spring workshops, with respondents in February and April rating the workshops significantly higher in some of the items. In addition, some interview comments provided additional insight into workshop utility.


The findings, also presented in Table 11, are as follows:

• Teachers and principals appreciated information that is relevant for use in the classroom, and for many the workshops provide that information; at least 80 percent or more of respondents selected Agree or Strongly Agree on the relevance of information and providing tools and strategies for the classroom. Many indicated that they appreciated the February video activity and April lesson share because they were relevant. One respondent commented, “This [workshop] is the best one yet. … It has been the most applicable.” In interviews, five respondents said they would like to see more opportunities to focus on lesson practice, such as modeling or sharing lessons. A teacher from Poplar School said, “That’s one of the things [that], as a teacher, you feel like you want to create and it’s already out there, so why spend time to reinvent it when someone else has come up with it.”

• One item with improved ratings from January to February regarded whether the time spent in team planning was useful in preparing for implementation. About half of January respondents selected Agree or Strongly Agree when considering this, compared with about 90 percent of respondents in February and April, and the percentage selecting Strongly Agree doubled in the last two months. (However, about 40 percent of January respondents selected N/A regarding team planning times.) In April, several respondents commented on the value of team time; one respondent said, “It was great! The team time was useful.” Respondents clearly appreciated the previously mentioned increase in the amount of team time.

• At least 80 percent or more of respondents selected Agree or Strongly Agree regarding providing clear direction for implementation, and ratings improved between the workshops. The percentage of respondents selecting Strongly Agree doubled between January and the other two workshops, and 30 percent of April respondents said knowing the plan for the future was the most important thing they learned at the workshop. However, many responses to open-ended questions in all three workshops indicate participants would like more guidance for the upcoming rollout; 20 percent of January and February respondents and 30 percent of April respondents identified the rollout as a topic that remains unclear.

• The workshop survey findings are somewhat at odds with interview comments, in which several respondents indicated the workshops were not highly useful. When asked, “How well did the Every Learner Inquires professional development prepare you to implement inquiry learning?” the majority of respondents responded along the lines of fairly well, and three teachers and one principal were unhappy with the sessions. The reasons for lower ratings varied, depending on previous experience with inquiry. Respondents at two schools where teachers were familiar with inquiry reported that the sessions contained a lot of review, which some find supplements their understanding and others find frustrating. One principal said, “At this point, we really feel like we’re doing a lot of review and spinning our wheels.” On the other hand, teachers who are new to the program reported they are frustrated because they feel they need more review in order to fully understand inquiry instruction.


Table 11. Overall Ratings of Workshop Utility, Spring 2008 Please rate your level of agreement with the following statements:

Month N Strongly Disagree Disagree Agree Strongly

Agree N/A

The information presented in the session was relevant to my teaching needs.

Jan. 55 2% 0% 38% 51% 9%

Feb. 70 0% 0% 26% 70% 4%

April 45 7% 0% 33% 53% 7%

The workshop provided tools and strategies I can use in the classroom.

Jan. 55 0% 0% 38% 45% 16%

Feb. 69 0% 0% 25% 67% 9%

April 47 6% 0% 26% 57% 11%

The team planning times were useful for preparing to implement the learning.

Jan. 51 0% 12% 27% 24% 37%

Feb. 66 0% 3% 33% 58% 6%

April 47 6% 2% 19% 70% 2%

The workshop provided clear direction about next steps for implementation.

Jan. 51 0% 12% 61% 20% 8%

Feb. 68 0% 9% 53% 37% 2%

April 45 7% 7% 38% 47% 2%

Overall, respondents found the workshops useful for their work, although February and April respondents thought more highly of their experiences at the sessions. January respondents wanted more chances to meet with their team and time to discuss the rollout, whereas February and April respondents viewed their experiences in lesson sharing, team time, and planning for the rollout more positively.

Ratings of Teacher Learning Workshop participants rated the degree to which each workshop had improved their understanding and skills in five areas: understanding of how students learn, understanding the learning cycle, adapting curriculum to the learning cycle, identifying students’ preconceptions, and differentiating instruction. As detailed in Table 12, overall, respondents indicated that the workshops were effective in promoting their understandings and skills, although these ratings varied among the different sessions. At least half of respondents for each month selected Moderately or Extremely for each of the five items. February respondents generally rated teacher learning the highest for all items, whereas April and January participants’ ratings varied, with higher scores in January for some items and higher scores in April for others. Some interview


comments provided additional insight into teacher learning. Other relevant findings include the following:

• Respondents appreciate continued instruction on various elements of the learning cycle; participants in all three months gave some of the highest ratings for teacher learning regarding the learning cycle. About 95 percent of January respondents’ comments referred to the learning cycle as the most important thing they learned at the workshop. The February and April respondents appreciated seeing and sharing examples of inquiry lessons, mentioned by about 65 percent of February respondents and about 30 percent of April respondents. In interviews, the majority of teachers indicated learning about the learning cycle made a difference in their teaching.

• However, respondents said they need continued instruction on various elements of the learning cycle. About 40 percent of January comments and 25 percent of February comments mentioned aspects of the learning cycle, such as the distinction between stages, as topics that remain unclear. A few workshop respondents and the majority of teachers and one principal from the spring interviews indicated they would like to discuss evaluation and assessment more. A teacher from Poplar School explained, “Because we do standardized grades on the Internet and then … rubrics for … the writing—that’s the burning question for us: How do you cut this up to make it fair for everybody?”

• Teachers’ ratings for adapting curriculum varied by workshop. In January, about half selected Moderately or Extremely for this item, compared with about 70 percent of April respondents and 80 percent of February respondents. However, some participants identified it as a concern; in January, about 15 percent identified meshing inquiry with curriculum as a potential barrier to the program, and about one fourth of February respondents said they need guidance on integrating content with the learning cycle.

• Ratings of identifying students’ preconceptions also varied by workshop. In January and April, about half selected Moderately or Extremely for this item, compared with about 80 percent of February respondents.

• Participants in all three months rated teacher learning the lowest regarding differentiating instruction. About 60 percent of January and February respondents selected Moderately or Extremely, as did half of April respondents.

• Some teachers also gave lower ratings to workshop impact on identifying students’ preconceptions; 15 percent of January respondents and 11 percent of April respondents selected Not at All.


Table 12. Ratings of Workshop Impact on Teacher Learning Compared to where you were before, to what extent has the workshop improved your:

Month N Not at All Somewhat Moderately Extremely N/A

Understanding of how students learn.

Jan. 55 2% 31% 55% 11% 2%

Feb. 70 4% 19% 49% 29% 0%

April 47 4% 19% 55% 13% 9%

Understanding of the learning cycle.

Jan. 55 2% 24% 53% 20% 2%

Feb. 70 0% 14% 46% 40% 0%

April 47 4% 21% 49% 21% 4%

Ability to adapt your curriculum to the learning cycle.

Jan. 55 0% 31% 42% 13% 15%

Feb. 69 1% 9% 48% 35% 7%

April 47 4% 15% 49% 23% 9%

Ability to identify students’ preconceptions.

Jan. 55 15% 22% 47% 5% 11%

Feb. 69 1% 19% 54% 26% 0%

April 46 11% 26% 41% 13% 9%

Ability to differentiate instruction.

Jan. 54 11% 26% 46% 11% 6%

Feb. 70 13% 26% 44% 17% 0%

April 47 9% 34% 38% 13% 6%

Overall, spring respondents felt the workshops impacted their learning, although February respondents rated teacher learning higher than January and April respondents. Respondents appreciate continued instruction on the learning cycle, especially when coupled with real-world examples. Summary of Findings The overall January, February, and April workshops were varied somewhat in attendance, with about 60 percent of participants attending in January, 70 percent in February, and 50 percent in April. However, the attendance rate of case study participants varied widely, with about one fourth attending in January, 70 percent attending in February, and about half attending in April.


Ratings of Teacher Learning Participants perceived the workshops as being most effective for increasing their understanding of how students learn and the learning cycle. For two areas of learning, some respondents said the workshop did little to promote their learning: identifying students’ preconceptions and differentiating instruction. Quality of Presentation Overall, spring workshop participants considered the workshops to be of good quality. The overwhelming majority of respondents agreed the content of the session was well organized, materials were clear, questions and concerns were addressed, and information was presented understandably. The February session was rated as particularly high regarding the organization of the session, the clarity of the materials, and the clarity of presentation. One reason for this increase in perceived quality may be the inclusion of an activity in which participants watched and discussed a video of a lesson. Utility of Workshops Overall, respondents found the workshops useful. Participants rated the workshops particularly high for the relevance of information and for providing tools and strategies for the classroom. The February and April sessions received the highest ratings of relevance, which may reflect on the quality of the workshop activities. Some ratings changed over time, in two regards:

• Ratings for the utility of team planning time improved dramatically between January and the remaining two workshops.

• The majority of respondents agreed that the workshops provided clear direction for implementation, although the strength of agreement tended to be lower than for other items. Ratings improved steadily from January to April.

Some interviewees expressed frustration about the workshops, though for different reasons. Respondents familiar with inquiry see the sessions as a lot of review, which some find supplements their understanding and others find frustrating; teachers who are new to the concept reported they are frustrated because they feel they need more review in order to fully understand inquiry instruction. Suggestions for Improvement Based on open-ended comments and participant interviews, the following are suggestions for improvement:

• Respondents need continued instruction on various elements of the learning cycle; several respondent comments indicated confusion on aspects of the learning cycle, such as understanding the differences between the phases.

• Respondents appreciate the opportunity to share and discuss real-world examples. A February workshop activity of observing and discussing a teaching video and a lesson


share in April were popular among respondents; many respondents wanted to see more of these types of activities at future events.

• Respondents value time for team-based planning, and the amount provided in the February and April sessions seems to be about right. Several January respondents mentioned the need for more time to work with their teams. However, after participating in more team planning in February and April, very few respondents mentioned the need.

• Respondents want more direction for implementation. Though the majority of spring respondents felt the workshops were useful in this regard, many responses to open-ended questions indicate participants would like more guidance for the upcoming rollout and for how to move to a more advanced level of implementation.

• Respondents would like to receive more information in advance of the workshop, including agendas, meeting details, and which materials to bring.


Section 2: Impact on Teacher Understanding The evaluation addressed one question related to teacher understanding:

To what extent are teachers gaining the knowledge and skills to implement inquiry-based science instruction? The extent of teacher learning about scientific inquiry was evaluated using the survey to measure teacher opinions about their level of understanding of inquiry instruction and their beliefs about their level of preparation to implement it. Survey items asked teachers to rate these opinions regarding not only inquiry instruction but also the learning cycle. In addition, teachers rated their level of preparation to accomplish three more specific aspects of instruction that previous evaluation reports have identified as difficult in the context of inquiry instruction: differentiating instruction, evaluating student work, and aligning lessons to standards. Level of Understanding As described in the introduction, the Standards provide a fundamental description of inquiry learning. Thus, comprehension of the Standards is directly connected to understanding the five essential features of inquiry learning. The following are the major findings, also displayed in Table 13:

• Over 70 percent of teachers participating in the ELI program reported being Moderately Familiar or Very Familiar with the standards (up from about 60 percent in Year 1).

• High school teachers had the greatest familiarity, with over half stating they were Very Familiar with the standards. By contrast, nearly 40 percent of elementary teachers described themselves as Not at All Familiar or Somewhat Familiar.

Table 13. Teacher Level of Familiarity With the Standards

How familiar are you with the National Science Education Standards published by the National Research Council?

N Not at All Familiar

Somewhat Familiar

Moderately Familiar

Very Familiar

Elementary school 21 19.0% 19.0% 38.1% 23.8% Middle school 7 0% 14.3% 71.4% 14.3% High school 15 0% 20.0% 26.7% 53.3% Total 43 9.3% 18.6% 39.5% 32.6%

Teachers also reported their level of understanding of the learning cycle. About 86 percent of teachers stated that they understand it Moderately or Very much so, with no major differences by school level.


Table 14 displays the findings.

Table 14. Level of Understanding of the Learning Cycle (N = 43)

How well do you understand the learning cycle? Percentage Not at all/Slightly 4.7% Somewhat 9.3% Moderately 51.2% Very much so 34.9% Total 100%

Teachers were asked to rate the extent to which the workshops provided sufficient information on what inquiry instruction and the learning cycle should look like in the classroom, which produced the following results:

• Nearly 90 percent of teachers selected Moderate or Very much so to describe the extent to which the workshops provided enough information on what inquiry-based science instruction should look like in the classroom.

• Over 75 percent of teachers selected Moderate or Very much so to describe the extent to which the workshops provided enough information on what the learning cycle should look like in the classroom.

These data are consistent with the reactions of participants following the workshops, as indicated on the workshop feedback forms. As reported in the previous section, teachers generally rated the workshops as highly effective in enhancing their understanding of the learning cycle and improving their ability to adapt lessons to the learning cycle; however, a minority of participants suggested they would like more information on the learning cycle. Self-Efficacy Teachers rated their self-efficacy in terms of the extent to which they are prepared to implement inquiry instruction, align lessons to the learning cycle, and address several related teaching challenges. These findings, displayed in Table 15, are summarized as follows:

• Implementing Inquiry Instruction. A high proportion of teachers perceived themselves as prepared to implement inquiry-based science instruction, as indicated by the 75 percent who selected the responses of Moderately (44 percent) and Very much so (33 percent). This is an increase over the 63 percent who selected Moderately or Very much so in Year 1. Most elementary teachers were confident in their preparation; 90 percent perceived themselves as well prepared compared with 64 percent of secondary teachers.

• Planning the Learning Cycle. In terms of their ability to align curriculum units with the learning cycle, about two thirds of teachers selected the responses of Moderately (42 percent) or Very much so (23 percent) to describe their level of preparation. (This item was not included on last year’s survey.) By the same token, this left about one third of teachers who rated their preparation as Somewhat (30 percent) or Not at all/Slightly (5 percent).


• Differentiating Instruction. Many teachers still reported that they did not feel well prepared to differentiate instruction. To rate their level of preparation for differentiated instruction, 40 percent of teachers selected Moderately (26 percent) and Very much so (14 percent). By the same token, about half selected Somewhat to describe their preparation, and 12 percent selected Not at all/Slightly. This was similar in proportion to the responses of teachers on last year’s survey. On this item, fewer secondary teachers felt well prepared to differentiate instruction, with 27 percent selecting Moderately or Very much so, compared with 53 percent of elementary teachers.

• Evaluating Student Work. A somewhat higher percentage of teachers than last year felt prepared to evaluate student work in inquiry lessons, although this remains an area of low self-efficacy for many teachers. Fifty-four percent of teachers selected Moderately (44 percent) and Very much so (9 percent), an increase over the 40 percent selecting these responses for Year 1. By the same token, nearly half did not perceive moderate preparation (about one third selected Somewhat and 14 percent selected Not at all/Slightly). Again, on this item, secondary teachers rated themselves particularly low, with 36 percent selecting Moderately or Very much so compared with 70 percent of elementary teachers.

• Aligning Lessons to Local Standards. A majority of teachers reported low self-efficacy for aligning inquiry lessons to local standards. Forty-four percent of teachers selected Moderately (23 percent) and Very much so (21 percent) to describe their level of preparation. By the same token, about half selected Somewhat to describe their preparation, and 9 percent selected Not at all/Slightly. This was similar in proportion to the responses of teachers on last year’s survey.

Table 15. Level of Preparation to Implement Inquiry Instruction

To what extent has your participation in Every Learner Inquires prepared you to do each of the following:

Not at All/Slightly Somewhat Moderately Very

Much So

Implement inquiry-based science instruction 0% 23.3% 44.2% 32.6% Modify curriculum units to align with the learning cycle

4.7% 30.2% 41.9% 23.3%

Differentiate instruction while implementing inquiry-based lessons 11.6% 48.8% 25.6% 14.0%

Evaluate student work products from inquiry-based lessons 14.0% 32.6% 44.2% 9.3%

Align inquiry-based lessons to local science standards 9.3% 46.5% 23.3% 20.9%

Note: N = 43. Summary of Teacher Learning and Understanding As a whole, teachers indicated that they understand inquiry instruction and the learning cycle and that the workshops were successful in explaining what both should look like in the classroom. Regarding self-efficacy toward different aspects of inquiry instruction, findings are mixed.


About three fourths of teachers felt well prepared to implement inquiry-based instruction, and the proportion of such teachers increased from last year. About two thirds of teachers expressed confidence regarding their ability to align their curriculum units to the learning cycle. Just over half of teachers expressed confidence regarding evaluating student work in inquiry lessons; this proportion has increased somewhat since last year. Fewer than half of teachers felt prepared to differentiate instruction or align lessons to local standards, and these ratings did not change from Year 1. Elementary teachers felt more prepared than secondary teachers to implement inquiry instruction, differentiate instruction, and evaluate student work. In summary, it appears that most teachers understand what inquiry instruction looks like and have self-efficacy toward teaching lessons in this manner. However, teachers have less self-efficacy toward some related instructional tasks (evaluation, alignment to local standards, differentiated instruction) that are essential for full implementation.


Section 3: Impact on Teaching Practice The evaluation addressed one question related to teaching practice:

To what extent has teacher practice changed to reflect an inquiry-based approach? Changes in teaching practice were examined in relation to inquiry instruction, classroom culture, and the learning cycle. Overall, evaluators found frequent use of certain activities aligned with inquiry instruction, such as hands-on experiments and data collection and student discussions. However, a range of instructional practices related to inquiry are less frequently, and in some cases infrequently, being used. Inquiry Instruction On the teacher survey, teachers rated the frequency with which students in their class participated in several types of inquiry-related activities. These responses were combined into a single scale score to represent overall frequency of inquiry instruction. These scale scores were categorized according to the most likely response to an item of average difficulty. Thus, each teacher was categorized in terms of the typical frequency rating he or she assigned to different inquiry-related student activities. The different categorizations are summarized in Table 16. They indicate that, for items of typical difficulty, slightly more than half of teachers rated the activities as occurring on a weekly basis, and slightly few than half of teachers rated them occurring on a monthly basis. This pattern is very similar to what was observed last year, indicating that there has been no increase in the frequency of inquiry instruction. In the regression model, no teacher characteristics were significantly related to inquiry instruction.

Table 16. Categorizations of Teacher Responses on the Inquiry Learning Scale (N = 46)

Inquiry Instruction Teacher Scale Score Percentage

All or almost all lessons 0% Once or twice a week 56.5% Once or twice a month 43.5% A few times a year 0% Almost never 0%

The teacher survey findings provide an overall summary of retrospective perceptions of inquiry instruction. Data from the instructional logs complement these findings by providing a concurrent record of activities in each day’s class (over the span of 10 consecutive lessons). These logs indicate whether lessons included a variety of different activities related to “problem solving/investigation” (12 activities) or “reading/reflection/written communication about science” (six activities). Overall, these data indicate that teachers used certain inquiry-based practices frequently and others seldom. When looking across all of the items in each of the two categories, 61 percent of lessons (on average) included some type of problem-solving/investigation activity, and about half of lessons


included some type of reading or writing about science. These overall findings are therefore consistent with the survey findings, in that they indicate that inquiry-related activities occur on a frequent basis. However, a look at the specific types of activities within each category reveals great disparities, as will be described for each category. The classroom log data are presented in their entirety in Appendix C. For the category of “Engaged in problem solving/investigation,” the most common activities reported by teachers were as follows: worked with manipulatives (41 percent), did hands-on investigations (36 percent), and recorded and/or analyzed data (36 percent). However, students infrequently evaluated the validity of scientific arguments or claims, critically examined the scientific explanations of other students, or followed specific instructions in an investigation; these activities occurred in 10 percent to 12 percent of lessons. In about one fourth of lessons, students explained findings and conclusions to other students. For the category of “Engaged in reading/reflection/written communication about science,” the most common activities reported by teachers were as follows: wrote reflections in a notebook or journal (occurring in 20 percent of lessons on average), answered textbook/worksheet questions (occurring in 19 percent of lessons on average), and reflected on readings activities or problems (occurring in 18 percent of classrooms on average). Students infrequently wrote a description of a plan, procedure, or problem-solving process or prepared a report of findings, with these activities occurring in fewer than 10 percent of lessons. Overall, data from teacher logs indicated that students are frequently engaged in hands-on investigations and the recording of data, yet infrequently engaged in explaining and evaluating findings. Moreover, students appear to work on worksheets or writing reflections, but infrequently on writing a description of a plan or preparing a written science report. This provides a fuller picture, albeit from the teachers’ perspective, of the frequency of inquiry instruction. Data from the interviews indicate a similar finding. When asked to describe how often and in what respects they are using inquiry instruction, teachers mentioned the following aspects:

• Five teachers mentioned working on questioning techniques. This included both teachers’ questioning of students and students developing scientific questions of their own. A teacher from Poplar School said, “Some of the questioning strategies have really helped try to draw out more of the student responses and, instead of just hearing their answer, try to probe more.”

• Five teachers mentioned using hands-on activities, including at least one teacher from each school. Similarly, four teachers mentioned collecting and/or analyzing data; a teacher from Birch School said, “They’re following somewhat of a procedure, then analyzing the data on their own. Trying to work on pulling evidence for any conclusions they draw. That’s kind of been my big focus this year. Evaluate and analyzing the lab.”

• Three teachers discussed working on students’ writing; a teacher from Poplar School said, “Organization of their writing—trying to get students to write more—I need to do more of that. But, that’s helpful to see students’ answers on paper and them trying to get their thoughts down.”


In summary, teachers described using inquiry practices related to questioning strategies, encouraging students to ask questions, and providing opportunities for hands-on data collection and analysis. This is consistent with the other data sources, inasmuch as these represent some but not all of the instructional strategies aligned with the inquiry approach. What appear to be absent from their description are the more intellectually challenging aspects of inquiry, which require students to take responsibility for their own learning; student activities such as devising studies to test hypotheses, reporting of results, and critical analysis of findings were seldom mentioned. These same aspects of inquiry also were infrequently reported on the instructional logs. The perspectives of students and outside observers will be considered next. Student Perceptions of Inquiry Instruction As with the teacher survey, the student survey also included several items describing classroom activities aligned with inquiry instruction. For each of these, students indicated whether the activity was descriptive of their classroom or not. These responses also were combined into a single scale and then categorized according to the most typical responses. Table 17 shows the distribution of the students across the three response categories. There were large differences based on school level, similar to Year 1. Ninety-four percent of elementary students typically responded Yes (54 percent) or Sometimes (40 percent); these findings were slightly more positive than last year. By contrast, 80 percent of high school students typically responded Yes (31 percent) or Sometimes (49 percent) to items about science investigations; these findings were slightly more negative than last year. By comparison, there were no differences among teachers by school level.

Table 17. Typical Responses of Elementary and High School Students to Inquiry Instruction Scale Items

Inquiry Instruction Student Scale Score

Elementary(N = 623)

High School (N = 350)

Yes 53.9% 30.9% Sometimes 40.1% 49.4% No 5.9% 19.7%

In summary, elementary students are comparable to teachers in regards to their perceptions of inquiry instruction; the clearest way to express this similarity is to note that few members of either group perceive that inquiry is not occurring. A more substantial minority of high school students perceive that inquiry is not occurring, and the majority perceives it occurring at most sometimes. Observer Perceptions of Inquiry Instruction A third perspective on instructional practices is provided by observers who recorded the student activities aligned with inquiry instruction. This section of the observation form was the same as the instructional logs used by teachers. Like teachers, observers found that the general category of problem solving/investigations was prevalent, occurring in 58 percent of lessons. Unlike teachers, however, observers infrequently noted student activities related to reading/writing


about science. The overall findings for both data sources are summarized in Table 18; see Appendix C for a complete summary of observation data.

Table 18. Prevalence of Student Activities, as Indicated by Instructional Logs and Observations

Category of Student Activity Instructional Logs* (N = 10)

Observations (N = 12)

Engaged in problem solving/investigation 61% 58% Engaged in reading/reflection/written communication about science 51% 25%

*Instructional log data represent the mean percentage of teachers’ lessons that used the specific strategy. As with the instructional log data, the observation data indicated that the individual items within each broad category differed greatly in prevalence. For the category of “Engaged in problem solving/investigation” in observations, two activities were prevalent: followed specific instructions in investigations and did hands-on investigations, both occurring in 58 percent of observed lessons. Most of the remaining activities were not observed at all, including those items related to posing questions, explaining data, or critically evaluating findings of others. For the category of “Engaged in reading/reflection/written communication about science,” the most frequent item was this: answered textbook/worksheet questions (occurring in 2 of 12 lessons or 17 percent). All other activities were observed only once (8 percent of the total observation) or not at all. Overall, observation data are consistent with data from the instructional logs in several respects. Both data sources indicate that students frequently do hands-on investigations and record data, and infrequently explain or evaluate findings. Moreover, both sources noted that the most common form of writing about science involved worksheets, and the least common writing involved reports of findings or descriptions of procedures. However, observers seldom observed students engaged in reading or writing about science, whereas teachers recorded this category in about half of all logs. One additional discrepancy between data sources should be highlighted: Observers perceived that in every instance of investigations, students followed specific instructions, whereas teachers indicated that this was the case in only 12 percent of lessons. It is possible that teachers and observers differed in their understanding of this term. Observers also completed additional items regarding inquiry instruction from the RTOP form. As opposed to the student activities sections of the observation form and the instructional logs, the RTOP items require observers to make judgments regarding the degree to which lessons reflected different aspects of reform-oriented instruction. Thus, RTOP data refer to higher-order concepts rather than readily observable behaviors. Overall, the observed lessons rarely reflected different elements of reform-oriented instruction in their design or content. Regarding lesson design, none of the observed lessons encouraged students to seek and value alternative modes of investigation or of problem solving. Regarding lesson content (i.e., the subject matter covered in classes), findings were mixed:


• Two aspects of lesson content were prevalent. Every observed lesson involved fundamental concepts of the subject, and a majority of teachers demonstrated a “solid grasp of the subject matter content” in their responses to student questions.

• Three items were seldom present in lessons. Few lessons promoted connections with other content disciplines and/or real-world phenomena (25 percent) or encouraged elements of abstraction (i.e., symbolic representations, theory building) (17 percent), or promoted conceptual coherence (25 percent).

Lessons were seldom characterized by most of the aspects of procedural knowledge of scientific inquiry:

• Observers most frequently noted that students made predictions or hypotheses and devised how to test them (42 percent).

• Observers infrequently noted that students used a variety of means (models, drawings, graphs, concrete materials, manipulatives, etc.) to represent phenomena (25 percent).

• Observers did not find that any classroom reflected the remaining three aspects of procedural knowledge: critical assessment of procedures; reflection on learning; or intellectual rigor, constructive criticism, and the challenging of ideas.

To summarize these findings, observers noted that science lessons frequently addressed substantial scientific content and that students made scientific predictions fairly often. However, lessons were seldom characterized several hallmarks of inquiry, such as critical assessment of procedures, reflection on learning, or intellectual rigor. Change in Use of Inquiry Instruction As noted, the teacher and student surveys indicated a comparable overall level of frequency of inquiry instruction. A more nuanced finding is provided by the interview data. In general, teachers and principals stated that the use of inquiry has changed since last year. Five teachers said they are using inquiry instruction more often than in the past. Moreover, four teachers and one principal commented that teachers’ ability to use inquiry has improved. A teacher from Poplar School said, “Each year, we keep refining it and we get better at teaching it. … I would definitely say that we’re not stagnant in our curriculum, we are constantly rewriting and revamping, [asking] ‘What can we do to make it better?’” Finally, one principal and three teachers mentioned that teachers are more comfortable with using inquiry. The principal from Chestnut School said, “I do believe [teachers’ use of inquiry has changed], just for the fact that they’re more comfortable with it and it’s a little bit more natural or easier for them to do.” There were some dissenting voices. One principal and two teachers said the way teachers use inquiry has not changed, although each respondent explained it was because the teachers were already using the techniques. Taken as a whole, the data suggest a nuanced conclusion regarding the change in inquiry use. Although frequency of student activities related to inquiry instruction appears to have remained constant, teachers report that their ability and comfort with inquiry has improved. It is possible, that quality of inquiry instruction—if not frequency of particular activities—has changed.


Summary of Inquiry Instruction Overall, classrooms of teachers involved in this project are characterized by frequent use of student activities related to inquiry instruction. The frequency of these activities has remained stable over time, although teachers reported that their ability to use inquiry instruction and their comfort level have increased over the past year. The inquiry-related activities that teachers employ are limited; hands-on investigations whereby students follow a procedure and record data are prevalent, but intellectually challenging activities in which students take responsibility for their own learning are seldom found. For example, students infrequently design studies to test predictions, write reports of findings, or critically assess procedures or evaluate findings. The data suggest that teachers perceive a greater degree of student latitude in designing instructions than do outside observers. It is possible that some teachers have an understanding of inquiry that is defined by hands-on activities and recording data. The reasons for this pattern of results will be discussed in the conclusion when findings from other sections of the report are considered. Classroom Culture On the teacher survey, teachers rated the frequency of student-centered classroom activities. These responses were combined into a single scale score, the Classroom Culture scale, to represent overall frequency of such activities. Scores on this scale were categorized using the same procedure described above for the Inquiry Instruction scale. The different categorizations are summarized in Table 19. In general, teachers reported frequent use of student-centered activities. About three fourths of teachers reported typical activities as occurring on a weekly (41 percent) or daily (33 percent) basis, with the balance perceiving a monthly occurrence. The regression analysis indicates that this finding differs based on classroom type and team type. Elementary teachers report more frequent student-centered activities than do secondary teachers, β = -.65, t = -2.87, p < .01. Whereas nearly half of elementary teachers report that student-centered activities occur on a daily basis, only about one fifth of secondary teachers report them occurring that frequently. The regression analysis also indicated that teachers from AEA teams reported student-centered activities more frequently than did case study team teachers, β = -.30, t = -1.88, p < .10. Whereas all teachers from case study teams were doing these activities on a monthly or weekly basis (and none on a daily basis), fully 43 percent of teachers from AEA teams were doing these activities on a daily basis.

Table 19. Categorizations of Teacher Responses on the Classroom Culture Scale (N = 46)

Classroom Culture Teacher Scale Score Percentage

All or almost all lessons 32.6% Once or twice a week 41.3% Once or twice a month 26.1% A few times a year 0.0% Almost never 0.0%


The teacher survey findings provide an overall summary of retrospective perceptions of classroom culture. Data from the instructional logs complement these findings by providing a concurrent record of activities in each day’s class. These logs indicate whether lessons included a variety of different activities related to “Listened to a presentation” (three activities) or “Engaged in discussion/seminar” (two activities). Overall, these data indicate that classroom discussions are very common, and presentations by teachers are common. Specific findings are as follows:

• On average, teachers indicated that 29 percent of lessons included a presentation by a teacher, whereas presentations by students or outside experts occurred in 6 percent and 3 percent of lessons, respectively.

• On average, 69 percent of lessons included a discussion, with 55 percent of lessons including a whole group discussion and 48 percent including small group discussion.

In summary, the classroom log data show the emphasis on discussions as a student activity. Data from the interviews provide detail about the change toward a more student-centered approach to teaching. Three teachers mentioned using more student-directed activities. For example, a teacher from Chestnut School said the process has shifted more toward the students, such that she and other teachers let the students question things more and figure things out on their own rather than the teachers giving them the answers. At the same time, one of these teachers described some difficulties with using a student-centered approach while maintaining a focus on the curriculum. This teacher said, “I’m good at the questioning part, but trying to get [my students] to develop the question—or when they do have a question, being able to use that question in context into the curriculum, whatever we’re working with—sometimes, that isn’t possible. They have a question, but it doesn’t lead us where we want to go.” On the other hand, four teachers, three of whom were from Birch School, mentioned they still use a fair amount of teacher-directed instruction, such as through lecture or investigations with specific instructions. A teacher from Birch School said, “You can stand and lecture, but you can also have inquiry method during a lecture. I hope there’s inquiry every day in my classroom. But I’ll be the first to admit that there are days there probably aren’t.” All four of these teachers gave various reasons for using the more traditional teacher-oriented approach, and most of the reasons related to the needs of their students. For example, one teacher taught young students who reportedly needed a fair amount of direction in all activities, while another taught an upper-level science course and said more direction in activities were needed to ensure the safety of the students. This paragraph synthesizes the teacher point of view: Teachers perceive frequent use of student-centered activities that support the classroom culture of inquiry, although teachers at case study schools use them somewhat less than teachers on AEA teams. Case study teachers (in their instructional logs) indicate that teacher presentations are common and discussions are very common. Several case study teachers explain that their practice has changed and they put a greater onus on students to figure out answers on their own, although several teachers retained a teacher-centered orientation in part to address the curriculum.


Student Perceptions of Classroom Culture As with the teacher survey, the student survey included several items describing classroom activities aligned with a student-centered classroom culture. For each of these, students indicated whether a particular activity was descriptive of their classroom or not. Table 20 shows the distribution of the students categorized according to their most typical response categories. Students at different school levels responded comparably, with over 55 percent of students typically responding Sometimes and over one third of students responding Yes. In summary, most students at both elementary and high school levels agree that activities that reflect a student-centered classroom culture are occurring at least sometimes. Few believe that they are not occurring.

Table 20. Typical Responses of Elementary and High School Students to Classroom Culture Scale Items

Classroom Culture Student Scale Score

Elementary(N = 627)

High School (N = 330)

Yes 42.1% 34.5% Sometimes 55.3% 58.8% No 2.6% 6.7%

Observer Perceptions of Classroom Culture A third perspective on classroom culture is provided by observation data. Like the instructional logs, these data indicated that teacher presentations and classroom discussions were prevalent. Observers reported that teacher presentations were very prevalent, occurring in 67 percent of lessons; this is much more frequent than the average of 29 percent reported (by teachers) in instructional logs. Similar to these logs, student presentations were observed in 8 percent of classrooms. Classroom discussions were observed in 67 percent of lessons, with whole group and small group discussions occurring in 50 percent and 33 percent of lessons, respectively. In summary, although observers reported a much higher prevalence of teacher presentations than did the instructional logs, the overall pattern is consistent, showing frequent teacher presentations and discussions. Data from the RTOP section of the observation form provide insight based on observer judgments of classroom climate. Overall, observers noted strong student-teacher relationships, although communicative interactions tended to be teacher directed. These findings may be summarized as follows (see Appendix C for a full summary table):

• Regarding student-teacher interactions, in most lessons, teachers demonstrated patience with their students in regards to listening to their full ideas. In a majority of lessons, teachers encouraged active participation of all students and fulfilled the role of “teacher as listener.” In half of lessons, the teacher acted as a resource person to assist students.

• Teachers rarely encouraged students to generate conjectures or alternative solution strategies.


• Regarding communicative interactions, the findings were mixed. In most classrooms, observers noted a high proportion of student talk and a climate of respect. However, student discussions did not typically drive the focus of the lesson, students did not use a variety of means to communicate, and teacher questioning did not lead to divergent modes of thinking.

Summary of Classroom Culture A variety of data sources indicate that classroom culture includes some of the student-centered approaches that promote inquiry instruction. Teachers rated student-centered activities as occurring daily or weekly. Other data sources indicated that students frequently engaged in discussion, and that teachers typically encouraged active participation of all students and acted as facilitators. These elements of classroom culture are more typical for elementary teachers and those on AEA teams. At the same time, however, teacher-oriented instruction remains a prominent feature of many classrooms. For example, lecturing remained very prevalent, teachers infrequently encouraged students to generate conjectures or find alternate solutions, and student questioning did not drive the focus of the lesson. Teachers explained that they rely on teacher-oriented instructional practices for a variety of reasons such as student safety, the developmental stage of the learner, and the demands of the curriculum. This last reason is consistent with the finding, reported in the previous section, that a majority of teachers do not perceive themselves as well prepared to align inquiry to local standards. Learning Cycle On the survey, teachers reported the number of full learning cycles they had completed during the current year. All teachers reported having taught at least one or two learning cycles, and over 70 percent reported having taught three or more, as shown in Table 21. It appears that the number of reported learning cycles differed by school level, with two fifths of secondary teachers reporting five or more full cycles compared with one fifth of elementary teachers.

Table 21. Number of Learning Cycles Completed, by School Level (Teacher Survey)

This school year, how many times have you taught a complete learning cycle in a science unit? School Level N 1 to 2 3 to 4 5 or more Elementary 20 30.0% 50.0% 20.0% Secondary 24 25.0% 33.3% 41.7% Total 44 27.3% 40.9% 31.8%

On the teacher survey, 80 percent of teachers stated that they had experienced some barrier to implementing a full learning cycle. Because most of these barriers relate to organizational support, they will be summarized in that section. During interviews, teachers provided more detail about both the frequency and fidelity to the learning cycle. Eight of nine teachers said they completed at least one learning cycle; the remaining teacher was in the process of completing one. Six of these teachers reported


completing two learning cycles, and two reported completing one with every unit. Several teachers reported having adapted their curriculum to the learning cycle; five of seven teachers who were asked this question reported having done so. The most common (mentioned by three teachers) way teachers adapted their curriculum was through the organization of the lesson based on the five Es (Engage, Explore, Explain, Elaborate, Evaluate). It is evident from survey and interview findings that teachers frequently do not complete the full cycle for a given lesson or unit. One teacher (from Poplar School) explained that the full learning cycle did not always seem to fit with the learning objectives of a given unit. This teacher explained, “I [use] components of it, probably daily, but to incorporate all five Es in a given lesson, it depends on the different unit that we’re on.” If teachers are not implementing the full learning cycle, it is important to understand which phases are more prevalent than others. The classroom logs required teachers to select, from a list of practices aligned to the different phases of the learning cycle, which ones were present in a given lesson. If any of the practices aligned with a particular phase were present, that phase was also considered as having been present. As summarized in Table 22, the analysis indicates that not all phases were equally prevalent. About half of lessons included teacher behaviors aligned with the Explain and Explore phases, between 35 percent and 40 percent of lessons included behaviors aligned with the Engage and Elaborate phases, and 20 percent of lessons included behaviors aligned with the Evaluate phase. The prevalence of different phases varied by school level, with elementary teachers employing each phase more often than high school teachers, with the exception of the Explain phase (where the high school teachers had an advantage).2

Table 22. Proportion of Instructional Logs Including Different Learning Cycle Phases

Case Study School Level N Engage Explore Explain Elaborate Evaluate

Elementary 7 47% 63% 46% 40% 24% High School 3 20% 13% 63% 23% 7% Total 10 39% 48% 51% 35% 20%

The differences in prevalence among the different phases can be connected to findings about inquiry-oriented student activities (also reported in the classroom logs). For example, the high prevalence of the Explore stage is consistent with the finding that students frequently engage in hands-on data collection. The high prevalence of the Explain phase is consistent with the finding that students frequently explain findings and conclusions to other students. The low prevalence of the Evaluate stage is consistent with the finding that students seldom assess procedures or evaluate findings.

2 Although this finding seems at odds with the greater number of learning cycles completed on the secondary level, it is important to keep in mind that the findings from classroom logs are based on three high school teachers compared with 24 secondary teachers (middle and high school) who participated in the survey. Thus, they reflect the experiences of a small number of teachers at a single case study school.


Observers provided a different perspective on the use of the learning cycle. The observation procedure differed from the instructional log in one respect: Rather than simply indicating whether a phase was present or not, the observer rated the extent to which the phase was exemplified by the lesson. However, these ratings were subsequently converted to a binary code of present or absent in order to facilitate the analysis. The percentages are displayed in Table 23. There were no major differences by school level or classroom type. These findings suggest either that observers did not observe a representative sample of lessons, or that observers have a stricter definition of the phases. The findings are consistent with the observation data regarding student activities, which indicated that students seldom engaged in behaviors related to explaining findings, making connections to other topics, or evaluating findings or procedures. The high prevalence of the Explore phase was consistent with the findings of high prevalence of hands-on activities.

Table 23. Percentage of Lessons in Which Different Learning Cycle Phases Were Observed (N = 12)

Phase Percentage Engage 58% Explore 75% Explain 8% Elaborate 8% Evaluate 8%

Two items from the lesson design section of the RTOP aligned closely with the Engage phase of the learning cycle. About 60 percent of lessons included instructional strategies and activities that respected students’ prior knowledge and the preconceptions inherent therein. In about one fourth of lessons, student exploration preceded formal presentation. This latter finding is difficult to generalize because the observed class period was not always at the stage where students would explore prior to formal presentation. In summary, the observation data indicates that the Engage and Explore phases of the learning cycle are prevalent, and the Explain, Elaborate, and Evaluate phases are seldom used. In many lessons, the design of the Engage phase included activities that respected students’ prior knowledge. Summary of Learning Cycle Implementation All teachers report having completed a full learning cycle, with most completing the cycle three or more times. Secondary teachers completed a somewhat greater number of cycles than did elementary teachers. However, teachers often do not teach a full learning cycle, with the earlier phases of the cycle (Engage and Explore) being more prevalent than the latter phases (Explain, Elaborate, and Evaluate). Data from observations and classroom logs (completed by observers and teachers, respectively) differ markedly regarding the prevalence of the Explain phase.


Summary of Impact on Teaching Practice The evaluation focused on the extent to which teaching practice reflected inquiry-oriented instruction, the extent to which the classroom culture became more student-centered, and the extent to which teachers adapted their lessons to the learning cycle. In some respects, all of these aspects are occurring frequently. Teachers report that inquiry-related student activities occur on a monthly to weekly basis, and that student-centered activities occur on a weekly to daily basis. Moreover, most teachers have completed at least three learning cycles (their assignment was to complete two over the course of the year). Although the frequency of the inquiry-related activities has not increased since Year 1, several teachers perceived that their ability and comfort level related to inquiry instruction have increased over the past year. However, the level of implementation appears less than comprehensive, because it is largely focused on hands-on data collection and analysis. Other aspects of inquiry learning do not appear to be prevalent, such as designing studies, writing reports of findings, critically assessing procedures, or evaluating findings. The adoption of a student-centered classroom culture reflects a similar less-than-comprehensive pattern. On the one hand, students frequently engaged in discussion, and teachers typically encouraged active participation of all students and acted as facilitators. On the other hand, lecturing remained very prevalent, teachers infrequently encouraged students to generate conjectures or find alternate solutions, and student questioning did not drive the focus of the lesson.


Section 4: Organizational Support The evaluation addressed two questions related to organizational support:

• To what extent does the administration advocate, facilitate, and support implementation?

• To what extent are teachers participating in collaborative planning for inquiry?

This section reports the findings of the teacher survey and principal and teacher interviews regarding organizational support. Included are the levels of administrator support, the professional learning communities within schools, school resources, and coherence of inquiry instruction with personal and organizational goals. Overall, teachers reported that their administrations do not offer high levels of support for inquiry learning and that they do not have access to many opportunities for professional learning. However, they indicated they generally do have the materials to implement inquiry and that inquiry is aligned with other programs and reform efforts. Interviews and feedback forms provided some insight into the experiences of teachers, as well as some suggestions for supporting teachers in the future. Administrator Support On the survey, teachers rated the extent to which their principals and other administrators supported inquiry instruction. Each teacher was classified according to his or her most likely response to an additional item of “average” difficulty (as described in the Methods section). Table 24 shows the expected distribution of the teachers across the various rating categories. Teachers generally perceived their principals as not being highly involved in the program or offering high levels of support; about three fourths were likely to report that their administrators were Not at all/slightly (34 percent) or Somewhat supportive (41 percent). This was more negative than last year, when about 60 percent selected these two response categories. Although case study teams include the principal in all professional development, there was no difference in the Leadership scale score based on team type (i.e., AEA or case study). The scale score for Administrator Support was not significantly related to inquiry instruction or classroom culture, suggesting that administrator support did not significantly affect teachers’ inquiry-based practices.

Table 24. Teacher Ratings of Administrator Support (N = 44)

Topic Percentage Very much so 6.8% Moderately 18.2% Somewhat 40.9% Not at all/slightly 34.1% Total 100%

The finding of relatively low ratings of administrator support is somewhat at odds with interview comments in which teachers discussed the ways principals support the program. During


interviews, teachers and principals were asked to describe several facets of administrative support. As the responses were generally more positive than in the teacher survey, it may be indicative of the high expectations of teachers and principals regarding support. Principal Expectations Interview respondents discussed the level of priority the principal places on inquiry instruction. Three of the four schools tended to have more similar answers. Seven out of ten teachers, representing those three schools, and all four principals indicated inquiry instruction was a priority at their school. All four principals indicated they have similar expectations for inquiry, namely that it is used, and six out of ten teachers, representing the three schools, mentioned the same expectations. Respondents from these schools indicated the most common ways principals communicate their expectations was through informal conversations, mentioned by three out of eight teachers and three out of four principals, and attendance at the meetings with teachers. While the teachers at three of the schools gave relatively consistent answers, teachers at Birch School had notably different responses, both from each other and the other schools. First, teacher responses regarding the priority of inquiry were dissimilar; one teacher said inquiry was somewhat of a priority, one teacher said the principal is still working that out, and one teacher said the principal gives mixed messages, sometimes giving it a priority and sometimes not. Second, teachers at Birch School reported they are unclear as to the principal’s expectations for the use of inquiry. While the principal reported the expectations are “that it’s used whenever possible and used effectively,” teachers reported otherwise. One teacher explained, “I don’t know what they are exactly. I think [our principal] has confidence in us and our abilities, and if it’s meeting the needs of the students, the district, and the state, then [our principal] supports it.” Monitoring Inquiry Teachers and principals also commented on if and how principals monitor implementation of inquiry in the schools. Six out of ten teachers indicated it was not monitored, though three of these were from Birch School. Of those who indicated implementation is monitored, teachers most often mentioned these methods: performance reviews, informal conversations, and self-monitoring (all three were mentioned by two teachers). Principals did not have any common answers, although one mentioned informal conversations and one mentioned using data. Supporting Inquiry Respondents explained how school administrators at case study schools support inquiry instruction. Nine out of ten teachers agreed that their principal supported inquiry-based instruction in some way or another. At least half of teachers indicated principals support inquiry by providing resources, ensuring teachers can attend professional development, and by being involved with the program, such as attending sessions themselves. The principal from Chestnut School explained, “[I] show a genuine interest in attending those inservices with them ... being part of that group. … [I am] learning with them.” At least one principal mentioned each of the methods of support discussed by teachers, but the most common response by principals was providing collaboration or instructional time for teachers. Thus, there appears to be a difference


in perspective between teachers and principals because teachers did not mention that the principal supports collaborative time to work on inquiry. In conclusion, survey respondents tended to rate principal support on the lower end. While case study participants also rated principal support as low, they indicated principal support was present in several respects, namely through providing resources, accommodating teachers so they may attend professional development, and staying involved in the program. Principals but not teachers mentioned providing collaborative time as one of the ways principals support the program. Professional Learning On the survey, teachers rated the extent to which they have opportunities for job-embedded professional learning, such as instructional coaching. Table 25 shows the distribution of the teachers across the various rating categories. The majority of teachers were likely to disagree that there are such opportunities at their school; about 90 percent either Disagree (56 percent) or Strongly disagree (35 percent). These findings were more negative than last year, when about 70 percent of teachers disagreed or strongly disagreed. There were no differences in the scale score based on team type (i.e., AEA or case study). The scale score for Professional Learning Community was significantly related to inquiry instruction (β = .398, t = 1.92, p < .10), indicating that teachers who reported more opportunities for professional learning tended to report more frequent use of inquiry instruction. This scale score was not related to the Classroom Culture scale score.

Table 25. Teacher Ratings of Professional Learning Opportunities (N = 44)

Response Percentage Strongly agree 2.3% Agree 7.0% Disagree 55.8% Strongly disagree 34.9% Total 100%

During interviews, teachers and principals discussed two of the learning opportunities in their schools: collaborative or individual planning and instructional feedback. Interviewees generally indicated there were opportunities for these two types of professional learning at their schools. Their comments provide details of existing arrangements and some explanations as to the needs of teachers regarding professional learning. Opportunities for Collaborative Planning In interviews, teachers and principals reported on opportunities to collaborate with other teachers; about half said they do have formal times for collaborative planning to accommodate inquiry instruction. For the most part, teachers are able to meet weekly with their coworkers, although the respondents from Birch School said they meet one to two times a month. For two schools, planning time usually occurs on teachers’ own time, either before or after school. In the


other two schools, teachers have time during the school day and during inservices for lesson planning and review; one teacher from Acacia School commented, “I have plenty of time to figure out my lessons, get the supplies, [and] get everything put together. And it would be impossible to do that if you didn’t have time during the day.” On the other hand, one teacher reported that the opportunity to work on inquiry instruction during scheduled professional development days has been curtailed. Despite differences in the setting of collaborative planning, most teachers felt the time they had was not adequate to plan and prepare for inquiry. Opportunities for Feedback or Review Case study participants indicated they have opportunities to receive feedback on their instruction. Eight out of ten teachers and all four principals indicated teachers have received feedback on inquiry instruction, and half of the respondents said the feedback most often comes from other teachers. Other less common sources of feedback included the principal and personnel from the district, the AEA, and the state. Case study respondents indicated feedback is most often followed up with verbal communication and, for two teachers, by written communication. Finally, all teachers who discussed feedback indicated it was helpful; some teachers said it provides encouragement for what they are doing, and two teachers said it helps them understand inquiry instruction better. In summary, the majority of respondents to the teacher survey indicated that opportunities for job-embedded professional development are not available at their schools. While case study participants also gave low ratings to professional development opportunities, their interviews provide some insight into the types of support teachers consider helpful. Their requests for more planning time during the school day and their favorable reaction to instructional feedback each represent a possible way to improve implementation in these and other schools. School Resources On the survey, teachers rated the extent to which they have access to school resources such as materials or lab equipment for inquiry instruction. Table 26 shows the distribution of the teachers across the various rating categories. Teachers indicated they have access to such resources; over three fifths of teachers were likely to select Agree (50 percent) or Strongly agree (14 percent). There was no difference in the School Resources scale score based on team type (i.e., AEA or case study). Due to significant changes in the survey items, this cannot be compared with last year’s ratings. The scale score for school resources was not significantly related to inquiry instruction or classroom culture.

Table 26. Teacher Ratings of School Resources (N = 44)

Response Percentage Strongly agree 13.6% Agree 50.0% Disagree 27.3% Strongly disagree 9.1% Total 100%


In interviews, teachers and principals discussed the availability of resources at case study schools. All four principals felt teachers have the resources they need for inquiry instruction, and, as previously mentioned, 9 of 10 teachers said their science curriculum supports inquiry. The most common type of curriculum support used is kit based, mentioned by teachers at three of the four schools. Teachers appreciated using the kits, as a teacher from Acacia School explained, “It promotes inquiry and makes it very easy to turn some of the cookbook lessons in the curriculum into inquiry lessons because there is that open-ended possibility.” Science kits played an important role for teachers who indicated they were struggling with the program, those who were newer to inquiry or who teach in self-contained classrooms; one teacher expressed they would be lost without them. In conclusion, teachers, in surveys and in interviews, indicated they have access to school resources for inquiry instruction. Case study participants, especially those who may struggle more with inquiry, expressed appreciation for science kits, mentioning they make it easier for them to incorporate inquiry in their classrooms. Coherence On the survey, teachers rated the extent to which inquiry learning is consistent with other programs and reform efforts, including personal goals and state and district curriculum guidelines, and these items were developed into a scale measuring coherence (as described in the Methods section). Table 27 shows the distribution of the teachers across the various rating categories. Overall, teachers indicated that inquiry aligned with other science instruction goals; over three fourths were likely to select Very much so (32 percent) or Moderately (46 percent). This measurement is new this year, so it cannot be compared with last year’s findings. There was no difference in the Coherence scale score based on team type (i.e., AEA or case study). The scale score for coherence was not significantly related to inquiry instruction or classroom culture.

Table 27. Teacher Ratings of Coherence (N = 44)

Response Percentage Very much so 31.8% Moderately 45.5% Somewhat 20.5% Not at all/slightly 2.3% Total 100%

Some interview comments were relevant to the consistency between the ELI program and existing curriculum frameworks. In the interviews, 9 of 10 teachers said their science curriculum supports inquiry. However, some teachers experienced difficulties incorporating inquiry due to a lack of time and space, as well as the need to focus on mathematics and reading basics. A teacher from Poplar School explained, “For the lower elementary, trying to teach all of the subjects, doing these labs, and having the space to do them in—you can’t always do it.”


In summary, teacher survey respondents indicated inquiry is consistent with other programs, reform efforts, and science curriculum. Although some teachers are comfortable meshing inquiry with their science curriculum, some lower elementary teachers reported difficulties doing so. Some teachers expressed the need for more assistance with aligning inquiry instruction with existing responsibilities. Implementation Concerns Related to Organizational Support On the teacher survey, in postworkshop reaction forms, and in interviews, teachers discussed their concerns about implementing inquiry instruction. Time On reaction forms, participants’ most commonly mentioned barrier and concern was time: time to implement, time to plan the rollout, and the time needed for teachers to fully understand the techniques and then be able to implement them. About 30 percent of January comments, 50 percent of February comments, and 70 percent of April respondents identified time as a barrier to implementation. Of the teachers who reported encountering some barrier to completing the learning cycle, about three fourths mentioned the barrier of time. Forty-one percent mentioned a lack of class time to complete a full learning cycle, 14 percent mentioned a lack of planning time, and 22 percent simply mentioned “time,” with no further elaboration. In the interviews as well, several teachers described time as barrier. For example, regarding class time, one teacher from Poplar School explained, “With the time that it takes to go through the entire [cycle] … you do the whole thing and work through it, and the students generate most of the information—that does take a lot of time.” Along these lines, some teachers described how the pressure to cover content interferes with the goal of teaching a learning cycle. One teacher elaborated on this dilemma in an open-ended comment as follows:

Time to implement the lessons is a huge barrier for me. I teach [an AP course], and the primary focus of the curriculum is on content, lots of content. Consequently, there is not time to spend five or six days on a complete learning cycle that requires multiple investigations. Sometimes it is hard to justify the time it would take to even do a portion of the learning cycle. The nature of the course forces me to “cover” new material constantly, rather than developing deeper understanding of a smaller number of concepts.

In interviews and in open-ended survey comments, several teachers discussed the disruptions to instructional time created by schoolwide schedule interruptions, student absences, and reductions of time allotted to science. Two teachers, both from Birch School, mentioned students missing class; one teacher explained as follows:

It’s just being able to get all that in. You have kids leaving for athletics and music competitions. You have snow days. You have class meetings. Assemblies. If you’ve got three classes of physical science but one of those classes completely misses a day or day and a half during the week. We have a regular schedule, but around here we call it our irregular regular schedule.


Resources Another prevalent type of barrier was a lack of material resources, mentioned by 22 percent of teachers who described a barrier on the survey. This barrier included both a lack of laboratory supplies and classroom space. Regarding the former, one teacher explained, “We get 25 dollars a year to buy all our supplies for the whole year for the whole class and must use this money for all our needs. I have 29 students to buy for, so it can get very expensive for important materials.” Support Another common concern expressed on postworkshop reaction forms involved gaining the support and participation of other parties, including their team members, other teachers, parents, and students. About 20 percent in January and February and about 30 percent in April mentioned lack of support as a barrier to implementation in general. Over half of January respondents and 20 percent of April respondents voiced it as a concern for their team’s follow-through after the workshop. Program Direction Several respondents mentioned needing more clear direction on the upcoming rollout. About 20 percent of February respondents mentioned this confusion as a concern for their team’s implementation; one commented, “The roll-out process still seems overwhelming to me; the parameters are vague. How many years? Budget?” In interviews, two out of nine teachers and one principal indicated they want more discussion about the future of the program; a teacher from Poplar School said, “I’m worried that we’re doing this, and after our three years are up or however many years are up, it’s just going to die.” Other Concerns One or two teachers each mentioned the following barriers on the survey: lack of understanding of the learning cycle, student behavior, and limited scientific knowledge on the part of the teacher. Regarding student behavior, one survey respondent explained, “Allowing this much freedom is too much for some kids to responsibly and respectfully handle, so management issues do arise.” In the interviews, two teachers (both from Chestnut School) also mentioned their own lack of understanding as a barrier. One teacher said, “I don’t quite understand the learning cycle and how to adapt things completely. I need more help with that.” Summary of Organizational Support Teachers gave low ratings to administration support and opportunities for professional learning, and they generally gave higher ratings to the availability of materials for inquiry and the alignment of inquiry with other programs and reform efforts. Ratings of principal support and professional learning opportunities were more negative than in Year 1. Principals supported teachers mainly by providing resources, allowing them to attend professional development, and staying involved with the inquiry program. Principals but not teachers also mentioned that they support the program by providing time for collaboration. Teachers expressed a desire for more


opportunities to plan during the school day. Teachers occasionally received feedback on inquiry instruction from other teachers and appreciated these opportunities. Teachers also expressed appreciation for science kits, mentioning they make it easier for them to incorporate inquiry in their classrooms. Some teachers, especially those on the elementary level, expressed the desire for more assistance with aligning inquiry instruction with existing responsibilities. The importance of professional learning opportunities was underscored by its positive relation to frequency of inquiry instruction. Many teachers had concerns about implementing inquiry, which related to organizational support. The most prevalent barriers to inquiry instruction are the amount of time it takes to complete the full cycle, meeting curriculum requirements, lack of local support, and the absence of clear direction for the program. Teachers also cited lack of supplies and space and their own limited understanding of the learning cycle as barriers to implementation.


Section 5: Impacts on Student Learning The evaluation addressed several questions related to student outcomes:

• To what extent have students acquired the abilities and fundamental understanding about scientific inquiry?

• What is the impact on student enthusiasm and self-efficacy toward science learning?

• Are the scientific understandings and abilities of students improving?

• Does the program address the needs of all subgroups? This section reports the findings of the teacher survey, student survey, and principal and teacher interviews regarding student learning. Student achievement data from Year 2 will not be collected or analyzed until fall 2008. Overall, interview and survey respondents indicated inquiry instruction has a positive effect on student learning and engagement and the program is able to reach students from all ability levels. Impact on Student Understanding of Scientific Concepts and Science Inquiry In surveys, teachers rated the extent to which inquiry instruction improved student understanding of science concepts and of the inquiry process. As indicated in Table 28, most teachers reported at least a moderate impact of inquiry instruction on student learning. The following summarizes these ratings:

• Regarding the extent of impact on understanding of scientific concepts, over three fourths of teachers selected ratings of Moderately (44 percent) or Very much so (33 percent). Ratings of this item were similar to responses in Year 1.

• Regarding the extent of impact on the understanding of the scientific inquiry process, nearly 90 percent selected ratings of Moderately (51 percent) or Very much so (37 percent). These ratings were clearly higher than in Year 1, when about two thirds selected Moderately or Very much so.

There were some significant differences between case study teachers and AEA teachers. About 80 percent and 95 percent of teacher from AEA teams selected Moderately or Very much so for understanding concepts and the inquiry process (respectively), compared with about 65 and 70 percent of teachers from case study teams.

Table 28. Teacher Ratings of the Impact of Inquiry Instruction on Student Learning

To what extent do you think your use of inquiry-based instruction has improved the following for your students:

N Not at All/Slightly Somewhat Moderately

Very Much

So Understanding of scientific concepts 45 2% 20% 44% 33% Understanding of the scientific inquiry process 43 0% 12% 51% 37%


In interviews, the majority of teachers and principals discussed if and how the quality of student learning has changed since beginning inquiry instruction. Five out of five teachers and three out of four principals and said student learning has improved; one principal was unsure until the test scores came back. Teachers and principals varied on how they identified the changes. Three teachers from different schools noted an improvement in students’ abilities in problem solving. A teacher from Acacia School said, “I think they’re much smarter, as far as how to problem solve. After doing science for a year, if you see the [students] from the beginning of the year to now, they’ve grown leaps and bounds as far as their problem-solving abilities.” All three of the principals, on the other hand, discussed increases in standardized test scores; the principal from Acacia School said, “For the past two years, our ITBS scores in science have been going in the right direction. I feel students have a better grasp on science concepts.” In summary, most teachers report that their use of inquiry instruction has at least moderately improved their students’ understanding of scientific concepts or scientific inquiry itself. The number who perceive the latter of the two impacts appears to have grown since last year. A greater proportion of teachers from AEA teams reported this level of influence than did teachers from case study teams. Several teachers identified problem solving as the specific area in which students have improved. Impact on Student Engagement and Self-Efficacy Teachers perceive that inquiry instruction has strongly improved student engagement and enthusiasm in science, as shown in Table 29. When rating the extent to which inquiry instruction had improved active engagement, 92 percent of teachers selected Moderately (36 percent) or Very much so (56 percent). When rating the extent to which inquiry instruction had improved student enthusiasm in science, 82 percent of teachers selected Moderately (22 percent) or Very much so (60 percent). These ratings were similar to those from last year. There were some differences by team type, with a greater proportion of teachers from AEA teams than case study teams indicating that inquiry instruction had a moderate to high impact. Regarding active engagement, about 95 percent of AEA team members selected Moderately or Very much so, compared with about 80 percent of case study participants. Similarly, regarding the impact on enthusiasm, about 85 percent of AEA teachers selected those options, compared with about 75 percent of case study teachers.

Table 29. Teacher Ratings of Student Engagement and Enthusiasm

To what extent do you think your use of inquiry-based instruction has improved the following for your students:

N Not at All/Slightly Somewhat Moderately

Very Much

So

Active engagement of students in science lessons 45 0% 9% 36% 56%

Student enthusiasm in science 45 2% 16% 22% 60% During interviews, case study teachers and principals said that, overall, they have seen changes in student attitudes toward science and student learning since beginning inquiry instruction. Five


teachers said students are more engaged and excited in science class; a teacher from Poplar School said, “I think that they enjoy it more. Their enjoyment is there, their interest is there, and if their interest is there, then they’re more likely to learn more from [science class].” Teachers also indicated students enjoy science more and are more confident in their roles as scientists in the classroom. A teacher from Birch School said, “I think after they get used to this way of learning, they appreciate the fact that they’re figuring things out, and you’re not holding their hand all the way through it.” All four principals agreed with teachers regarding student attitudes; two principals mentioned that, compared with before the program, more students want to go to science class in their school. The principal from Acacia School said, “They’re much more engaged and excited about [science]. … I’ve never heard anybody say they didn’t want to go to science. And a few years ago I would’ve heard that.” In surveys, students rated their level of agreement with items on the Enthusiasm towards Science and Self-Efficacy in Science scales. As with the teacher survey, each student was classified according to his or her most likely response to items on each scale (as described in the Methods section). Table 30 shows the distribution of elementary and high school students across the response items for science enjoyment and self-efficacy. Elementary students typically affirmed statements expressing enthusiasm toward science; 65 percent typically selected Yes to such items. High school students typically did not affirm such statements; 18 percent of high school students typically selected Yes. This pattern is similar to the findings reported for Year 1, although both groups of students expressed somewhat less liking this year than last (when about 70 percent of elementary students typically responded Yes, and 56 percent of high school students typically responded Sometimes).

Table 30. Student Perceptions of Enthusiasm and Self-Efficacy in Science

School Level N Yes Sometimes No Enthusiasm Elementary 623 64.7% 30.3% 5.0% High school 329 18.2% 48.0% 33.7% Self-Efficacy Elementary 623 46% 46% 7% High school 329 21% 58% 21%

Regarding self-efficacy toward science, elementary students were roughly split between those who affirmed their self-efficacy and those who did not, whereas high school students typically did not perceive themselves as capable in the context of science instruction. Because the items used to measure science self-efficacy during Year 1 could not be combined into a single scale, it is not possible to compare Year 2 with Year 1 in this regard. In summary, teachers perceive that their use of inquiry instruction improves the level of student engagement and enthusiasm in science. Elementary students typically report that they are enthusiastic toward science, whereas high school students typically report that they are not enthusiastic. The overall level of enthusiasm of science has not changed since last year. About


half of elementary students and one fifth of high school students rated themselves as being capable to accomplish various tasks involving their science class. Addressing the Needs of All Subgroups In interviews, teachers and principals discussed whether the program meets the needs of special education students and English language learners (ELLs). All respondents (four principals and five teachers) said agreed the program meets the needs of both of these subgroups. Respondents also described the response of ELL and special education students to the program. Four teachers said the program reaches students at their own level; a teacher from Poplar School said, “I think it provides entry level for kids of all abilities. Kids who are more intelligent or are perceived as smarter have something to share, and those kids who are maybe struggling with some of the basics are willing to try things and new ways of doing things.” Two principals and one teacher said the hands-on approach helps students of all levels engage in science. The principal from Poplar School said,

They’re definitely the strongest points. Some of our students that are doing the best in science class right now are some of our students that are struggling in the other classrooms. They’re special ed kids, they’re students that struggle with their learning where they have to sit and write … or [they’re] some of our kids who have behavior issues. That’s one of the biggest things with them in science—get up, move around, be part of groups—and I think it most definitely helps.

In summary, teachers and principals indicated inquiry instruction supports the learning of special education and ELL students. Respondents described the program reaching students by meeting students at their own level and teaching them through hands-on activities. Summary of Impact on Student Learning The evaluation examined the perceived impact of inquiry instruction on student learning and attitudes toward science, and whether students of all types benefit. Regarding student learning, most teachers reported that inquiry instruction has improved their students’ understanding both of scientific concepts and of the process of scientific inquiry. The number who perceived the latter of the two impacts appears to have grown since last year. Several teachers identified problem solving as the specific area in which students have improved. Teachers and principals also reported that inquiry instruction has improved the level of student engagement and enthusiasm in science; students appear to be excited about attending science class. Elementary students typically report that they enjoy science, whereas high school students typically report that they do not enjoy science. The overall level of enjoyment of science has not changed since last year. Finally, teachers and principals described the impact of the program on special education and ELL students, indicating that the program is especially effective for those students.


Conclusions and Recommendations This section synthesizes the major findings in this report. Many participants remain optimistic about the promise of the Every Learner Inquires (ELI) initiative to transform science education, but there is more work to be done to support implementation. The professional development sessions were well received, and participants valued the opportunity to share and discuss real-world examples. Many participants expressed greater confidence in their ability to implement inquiry instruction than in the previous year, and participants continued to see inquiry instruction as benefiting students of all ability levels. However, the participants indicated that they need greater support in the context of their schools in order to implement what they have learned in their workshops. Teachers stated that they receive low levels of support from their principals and seldom engage in job-embedded professional learning. Although teachers do meet to discuss implementation, there is not much of an opportunity for job-embedded learning involving observation, modeling, feedback, or lesson study. Participants mentioned this lack of support as one of their concerns about implementation. It is not surprising, then, that teachers also reported difficulty in finding time to plan inquiry or to integrate it with their other curriculum requirements. In addition to greater local support, participants also expressed the desire for clearer direction about the goals and milestones of the program. Some case study participants are uncertain about the direction of the program and whether it will continue. Several AEA participants requested greater direction for how to take on the responsibility of training others in inquiry. This desire for clearer direction was also manifest in the suggestion, made by several participants, that information about workshops be provided in advance. These reactions suggest that participants are not clear about what they are supposed to learn and implement in each year. This finding may explain the fact that the level of implementation of inquiry-based strategies did not appear to increase since Year 1, and that several essential features of inquiry instruction were not prevalent. Perhaps the lack of clarity about what should be implemented, combined with the absence of strong support from the principal or professional learning community, made it difficult for teachers to incorporate the more challenging aspects of inquiry instruction. Because the ELI initiative will scale up during Years 3 and 4, it is important to be explicit with those continuing and considering participation about what the program is; what its goals and milestones are; and what is required from teachers, principals, schools, districts, AEA teams, and the state. The findings suggest that explicitness would increase implementation and impact through clear goals and clear expectations about work, contributions, support, and progress. In order to foster more robust job-embedded professional learning, program leaders should provide clearer expectations for how teachers should work together, what they should accomplish, and how principals should support them. Principal support for the program is essential. Currently, few participants report strong support from their principals. Although principals already are expected to attend the workshops, they also should receive clearer direction of where their responsibilities lie for implementing the program. These responsibilities may include communicating expectations about implementing inquiry learning, helping teachers meet local curriculum guidelines, and providing opportunities for teacher collaboration. At a


minimum, schools ought to provide teachers with protected time on a monthly to weekly basis to collaborate and plan for implementation. Another way to promote stronger job-embedded learning is to provide more guidance for setting up an embedded professional development system in each school. Lesson study is one framework for a professional learning community that appears appropriate for this type of initiative (Loucks-Horsley, Love, Stiles, Mundry, & Hewson, 2003). Under this approach, teachers plan a lesson, observe it in the classroom, and discuss how it can be improved. This framework would provide teachers with opportunities to collaborate on lesson planning and receive feedback on lesson design, instructional quality, and student learning outcomes. Regardless of the framework, research has shown that teachers need extensive practice and feedback in order to be able to incorporate the techniques presented in workshops (Joyce & Showers, 2003). The workshops should continue to provide direction for how to engage in job-embedded professional learning activities such as lesson study. In addition, program leaders could help to coordinate the involvement of science consultants affiliated with AEA teams who could help facilitate lesson study (such as by observing an iteration of the lesson that emerges from this process) and provide coaching and modeling. Hand in hand with more explicit guidelines for local support, program leaders also should provide explicit goals and milestones for learning and implementation in each year of the program. Corresponding to these learning goals, program leaders should develop a plan for the entire year of workshops, specifying their expected learning outcomes and the follow-up assignments that will provide opportunities for both practice and feedback. This would provide a clear sense of direction for what the year of participation in ELI is supposed to produce. The topics and methods of these workshops should correspond to the suggestions for improvement offered by participants, which are summarized in brief as follows:

• Respondents need continued instruction on various stages of the learning cycle.

• Respondents appreciate the opportunity to share and discuss real-world examples.

• Respondents value time for team-based planning, and the amount provided in the February and April sessions seems to be about right.

In summary, the initiative in Year 3 should be structured to provide a coherent plan for workshops, assignments that provide opportunities for practice and feedback, and collaborative structures such as protected time and lesson study to provide job-embedded opportunities for professional learning. These expectations should be conveyed to schools continuing and considering participation in the initiative.


References American Association for the Advancement of Science. (1993). Benchmarks for science literacy.

New York: Oxford University Press. Desimone, L. M., Porter, A. C., Garet, M. S., Yoon, K. S., & Birman, B. F. (2002). Effects of

professional development on teachers’ instruction: Results from a three-year longitudinal study. Educational Evaluation and Policy Analysis, 24(2), 81–112.

Dey, I. (1993). Qualitative data analysis: A user-friendly guide for social scientists. New York:

Routledge. Fraser, B. J. (1982). TOSRA: Test of science-related attitudes handbook. Hawthorn, Victoria:

Australian Council for Educational Research. Fraser, McRobbie, & Fisher. (1996). WIHIC: What is happening in this classroom? Melbourne:

Australian Council for Educational Research. Guskey, T. R. (2000). Evaluating professional development. Thousand Oaks, CA: Corwin Press. Horizon Research. (2005). 2005–06 local systemic change classroom observation protocol.

Chapel Hill, NC: Author. Retrieved July 12, 2008, from www.horizon-research.com/LSC/manual/0506/tab6/cop0506.pdf

Joyce, B., & Showers, B. (2003). Student achievement through staff development (3rd ed.).

Alexandria, VA: Association for Supervision and Curriculum Development. LeCompte, M. D. (2000). Analyzing qualitative data. Theory Into Practice, 39(3), 146–154. Loucks-Horsley, S., Love, N., Stiles, K. E., Mundry, S., & Hewson, P. W. (2003). Designing

professional development for teachers of science and mathematics (2nd ed.). Thousand Oaks, CA: Corwin Press.

National Research Council. (1996). National science education standards. Washington, DC:

National Academy Press. National Institutes of Health. (2005). Doing science: The process of scientific inquiry (Teacher’s

Guide). Retrieved July 12, 2008, from http://science.education.nih.gov/supplements/nih6/inquiry/guide/implementing.htm

Penuel, W. R., Fishman, B. J., Yamaguchi, R., & Gallagher, L. P. (2007). What makes

professional development effective? Strategies that foster curriculum implementation. American Educational Research Journal, 44(4), 929–958.


Piburn, M., & Sawada, D. (2000). Reformed teaching observation protocol reference manual (ACEPT Technical Report No. 1N00-3). Tempe, AZ: Arizona Collaborative for Excellence in the Preparation of Teachers.

Sawada, D., Piburn, M., Turley, J., Falconer, K., Benford, R., Bloom, I., et al. (2000). Reformed

teaching observation protocol (RTOP) training guide. Tempe, AZ: Arizona Collaborative for Excellence in the Preparation of Teachers.


Appendix A Survey Methodology

Teacher Survey Learning Point Associates developed a teacher survey to address the four goals of the ELI program: teacher learning, organizational support, implementation of inquiry instruction, and student impact. The teacher survey included new items written for this evaluation as well as items used in previous evaluations of science professional development. For each construct described in the Introduction, the survey included several items to reflect different details or elements of the construct. A Rasch analysis was used to validate whether these items corresponded to an underlying and unitary construct as expected. Where they did, the survey items were evaluated in terms of a single scale score. The following paragraphs describe how each construct was evaluated with the survey. Teacher Learning

The survey included several items to evaluate the extent to which teachers gained the knowledge and skills to implement inquiry-based science instruction. Survey items asked teachers to rate their understanding of and preparation to implement inquiry instruction and the learning cycle. In addition, teachers rated their level of preparation to implement three related aspects of instruction: differentiating instruction, evaluating student work, and aligning lessons to standards. The specific items are as follows:

• A single item asked teachers to describe how familiar they are with the National Science Education Standards, which they rated by selecting from the following: Not at all familiar, Somewhat, Moderately, and Very familiar.

• One item asked teachers to rate how well they understand the learning cycle by using the following scale: Not at all/Slightly, Somewhat, Moderately, and Very much so.

• Two items asked teachers to rate the extent to which the workshops provided sufficient information on what inquiry instruction and the learning cycle look like in the classroom. Although these items ostensibly are referring to opinions on workshop quality, it was assumed that they would be an effective implicit measure of level of understanding of what inquiry-based instruction looks like. These items were rated with the response options of Not at all/Slightly, Somewhat, Moderately, and Very much so. These items were new this year.

• Five items asked teachers to rate their level of preparation to accomplish several broad instructional tasks. These tasks were to implement inquiry learning, modify curriculum units to align with the learning cycle, differentiate instruction while doing inquiry, evaluate student work from inquiry-based lessons, and align inquiry lessons to state standards. The second of these items was new this year. These items were rated with the response options of Not at all/Slightly, Somewhat, Moderately, and Very much so. These items were not intended to evaluate a single construct and were analyzed individually.


Organizational Support

The survey included four groups of items that were used to create scales addressing professional learning communities, principal support, resources, and coherence. Each scale is described in more detail below.

• Professional Learning Community. Eight items asked teachers to rate their level of agreement with statements about professional learning opportunities in their school. The survey included four of the five items from the scale used last year, along with four new items. The statement “Rate your agreement with the following statements about science teaching in your school:” introduced these items. Teachers rated their agreement with the response options of Strongly Disagree, Disagree, Agree, or Strongly Agree. These items were combined into a single scale score. The items in this scale are as follows (with items from the Year 1 survey designated by an asterisk):

Science teachers in this school regularly observe each other teaching classes in order to learn how to implement inquiry-based lessons.*

Teachers meet to examine student work (e.g., to study the effectiveness of inquiry lessons).*

Teachers receive in-school coaching and/or modeling to support inquiry-based instruction.*

Teachers meet following ELI professional development sessions to discuss or plan implementation.*

Teachers have time during the regular school week to work with each other on implementation of inquiry learning.

Teachers discuss what they learned with other teachers in their school or department who did not attend the particular ELI workshop.

Teachers discuss or share what they learned at ELI workshops with administrators (e.g., principal or department chair).

Teachers in this school communicate with teachers from other schools about inquiry learning.

• Administrator Support. Nine items asked teachers to rate the extent to which administrators support inquiry learning. The survey included all four items from the scale used last year, along with five new items. The stem “To what extent does your principal, assistant principal, or department head:” introduced the items. These items were rated with the response options of Not at all/Slightly, Somewhat, Moderately, and Very much so. These items were combined into a single scale score. The items in this scale are as follows (with items from the Year 1 survey designated by an asterisk):

Understand inquiry-based instruction.*

Communicate expectations about implementing inquiry learning.*

Encourage teacher collaboration to support inquiry learning (e.g., lesson planning, observing lessons, reviewing student work).*


Monitor teacher implementation of inquiry learning.*

Accept the noise that comes with an active classroom.

Encourage the implementation of current national standards in science education.

Encourage innovative instructional practices.

Encourage teachers to make connections across curriculum areas.

Act as a buffer between teachers and external pressures.

• Resources for Inquiry Instruction. Eight items asked teachers to rate their agreement with statements about whether they had adequate resources for inquiry instruction. Teachers rated their agreement with the response options of Strongly Disagree, Disagree, Agree, or Strongly Agree. These items were combined into a single scale score. The items in this scale are as follows:

I have sufficient planning and preparation time to create inquiry-based lessons.

I have sufficient access to computers for science instruction.

I have sufficient laboratory space to support inquiry learning.

I have sufficient laboratory equipment to support inquiry learning.

I have sufficient well-stocked science kits or boxes to support inquiry learning.

I have sufficient laboratory supplies or consumables to support inquiry learning.

I have instructional materials that align with inquiry learning.

I have sufficient funds for purchasing equipment and supplies for science.

• Coherence. Five items asked teachers to rate the extent to which the ELI program is consistent with other programs, reform efforts, standards, and assessments. The stem “To what extent is Every Learner Inquires consistent with each of the follow:” introduced the items. These items were combined into a single scale score. The items in this scale are as follows:

Other science professional development programs at your school or district.

Other school or district reform efforts.

Your own goals for professional development.

State or district standards and curriculum frameworks.

State and district assessments.

Teacher Practice

Several items examined the frequency with which teachers used certain strategies, or the frequency with which students engaged in certain activities, that align with or support inquiry-based learning. When answering these items, teachers were directed to describe a particular class of theirs. There were three groups of items of this type:


• Inquiry Instruction Scale. Teachers rated the frequency with which students in their class took part in 15 types of science activities aligned with the essential features of inquiry learning. The scale included eight items from last year’s survey and seven new items. These items were introduced with the question, “About how often do students in this class take part in the following types of science activities?” Teachers rated frequency with the following response options: Almost never, A few times a year, Once or twice a month, Once or twice a week, and All or almost all lessons. These items were combined into a single scale score. The items in this scale are as follows (with items from the Year 1 survey designated by an asterisk):

Generate their own ideas, questions, and/or propositions.

Share ideas or solve problems with each other in small groups.

Ask scientifically oriented questions about phenomena they observe.

Critically examine the scientific explanations of other students.*

Do hands-on/laboratory science activities or investigations.

Explain their findings and conclusions to other students.*

Follow specific instructions in an activity or investigation.*

Plan investigations to answer scientific questions.*

Participate in fieldwork.

Write reflections in a notebook or journal.

Prepare written science reports.*

Record, represent, and/or analyze data.*

Support their explanations with scientific knowledge.*

Use data to support an explanation.*

Work on models or simulations.

Work on extended science investigations or projects (a week or more in duration).

Make formal presentations to class.

• Classroom Culture Scale. Teachers rated the frequency with which they incorporate nine student-centered instructional practices in their science instruction. The scale included three items from last year’s survey and six new items. These items were introduced with the question, “About how often do students in this class take part in the following types of science activities?” Teachers indicated frequency with the following response options: Almost never, A few times a year, Once or twice a month, Once or twice a week, and All or almost all lessons. These items were combined into a single scale score. The items in this scale are as follows:

Arrange seating to facilitate student discussion.

Assign students to work in groups on projects.

Provide opportunities for student-to-student interaction.


Engage the whole class in discussions.

Engage students in small group or one-on-on discussions.

Let the questions and comments of students guide the focus of the lesson.

Provide differentiated background readings according to student reading level.

Provide students with a choice of different assignments.

Share ideas or solve problems with each other in small groups.

• Learning Cycle Implementation. Three items referred to implementation of the learning cycle. Teachers reported the number of times they taught a complete learning cycle in science during Year 2. Another item asked if they experienced any barriers to teaching a full learning cycle (with response options of Yes or No), and if so, to describe the barriers in an open-ended fashion.

Impact on Student Learning Four items asked teachers to rate the extent to which their implementation of inquiry learning has improved student learning outcomes. These outcomes were engaged learning, understanding of science concepts, enthusiasm, and understanding of inquiry. Teachers rated the extent of improvement with the response options of Not at all/Slightly, Somewhat, Moderately, and Very much so. These items reflect different aspects of student learning outcomes and were each analyzed individually. Teacher Survey Administration In April 2008, Learning Point Associates e-mailed each participating teacher, asking them to complete a survey via the Internet. At the request of Learning Point Associates, the director of the ELI initiative also sent a message to each participant asking them to respond to the survey. The survey remained open for three weeks. Evaluators sent two follow-up messages directly to nonrespondents and also contacted the principals at case study schools to request that they remind all teachers to complete the ELI survey if they had not already done so. ELI project staff also sent a reminder message to nonrespondents encouraging their participation. Of the 80 teachers who were contacted, 46 teachers completed the survey, for an overall response rate of 57 percent. The response rate did not vary by team type; 12 of 21 case study teachers and 34 of 59 AEA teachers completed the survey. Student Survey Learning Point Associates developed the student survey based on items from previously published instruments. All items on the survey had a response scale of Yes, No, and Sometimes. The survey included the following constructs:

• Inquiry Instruction. Ten items from the Investigation subscale of the What is Happening in this Classroom (WIHIC) survey were used to address the construct of inquiry learning. These items were combined into a single scale score. The items in this scale are as follows:


I write science reflections in a journal.

I make science presentations to my class.

I carry out investigations to test my ideas.

I am asked to think about the evidence for statements.

I carry out investigations to answer questions coming from discussions.

I explain the meaning of statements, diagrams, and graphs.

I carry out investigations to answer questions that puzzle me.

I carry out investigations to answer the teacher’s questions.

I find out answers to questions by doing investigations.

I solve problems by using information from my own investigations.

• Classroom Culture. Nine items from the Involvement subscale of the WIHIC were used to address the construct of classroom culture. The items on both of these scales adapted from the WIHIC were the same as last year, although two items from last year’s Classroom Culture scale were moved to the Inquiry Instruction scale to better align with the definition of these constructs. These items were combined into a single scale score. The items in this scale are as follows:

I discuss ideas in science class.

I give my opinions during class discussions.

The teacher asks me questions.

My ideas and suggestions are used during classroom discussions.

I ask the teacher questions.

I explain my ideas to other students.

I talk with other students about how to solve problems.

I work on group science projects.

• Enthusiasm Toward Science. Six items from the Test of Science Related Attitudes (TOSRA) were used to measure a student’s enthusiasm toward science. These items were combined into a single scale score. The items in this scale are as follows:

Science lessons are fun.

School should have more science lessons each week.

Science is one of the most interesting subjects in school.

I really enjoy going to science lessons.

I look forward to science lessons.

Everyone should learn about science.

• Self-Efficacy for Science. Six items asked students to rate whether they can accomplish certain science classroom skills. Each item started with the phrase “I can” and was


completed with phrases such as “ask a scientific question” and “collect data to answer a question.” These items were combined into a single scale score. The items in this scale are as follows:

I can create my own experiment.

I can write up findings from an experiment.

I can ask a scientific question.

I can collect data to answer a question.

I can explain my findings to other students.

I can help other students with science class work.

A psychometric analysis comparing responses of elementary and high school students determined that these two groups responded differently to the survey items. For this reason, the scale scores from these two groups are reported separately. Student Survey Administration The student survey was administered during April 2008 to students who were taking a science class in the case study schools in Grades 3 and above. The evaluator sent paper and pencil survey forms to 12 science teachers to distribute to students in his or her class. Every teacher distributed and returned the surveys. The number of respondents from each school ranged from 77 to 349, with a total of 961 respondents. Each teacher was asked to indicate the total number of students and the total number of students absent or who declined to participate. Nine of the 12 teachers provided this information; among these teachers, there were 577 total students, of whom 526 completed the survey for a response rate of 91 percent. Survey Analysis Plan Responses to the teacher and student surveys were analyzed descriptively. Scale scores were interpreted by categorizing each participant according to his or her most likely response to an item of average difficulty (i.e., the difficulty with agreeing with the item). For example, for the construct of principal support, it was possible to estimate the likelihood that each participant would select the response options of Not at all/Slightly, Somewhat, Moderately, and Very much so for an item of average difficulty. Thus, the distribution of respondents among these categories provides a picture of overall principal support. Survey items that were not part of a scaled score were analyzed by examining the frequency of each response. The analysis of the teacher survey took into account two important distinctions among teachers:

• For items addressing professional development and teacher learning, the responses were disaggregated by classroom type: self-contained or not self-contained. This distinction is important because the latter group is expected to have a greater level of experience with inquiry instruction to begin with, owing to the teachers’ status as full-time science teachers.


• For items addressing organizational support, teacher survey responses were disaggregated by participant type (case study versus AEA teacher leader). Because case study schools have a formal agreement to support the program, this may be an important distinction.

Based on these distinctions, t-tests were performed to examine group differences in responses to the relevant survey scales. For survey items that were not scaled, response frequencies were disaggregated by these groups, as appropriate, to identify possible group distinctions. Statistical Modeling of Teacher Practice Outcome Regression analysis was used to examine the impact of organizational support and teacher characteristics on the outcomes related to teacher practice, namely inquiry instruction and classroom culture. The variables initially selected for this equation and their transformations were as follows:

• Organizational Support Scale Scores Administrator Support scale score

Professional Learning Community scale score

School Resources scale score

Coherence scale score

• School and Classroom Characteristics. The regression analysis included several school and classroom characteristics, reflecting responses to survey items:

Grade level (0 = elementary, 1 = secondary. Grades 7 and above were coded as secondary.)

Team type (0 = AEA, 1 = case study)

Minutes of science instruction per week (0 = low, 1 = high). Teachers described the amount of time they spend teaching science during a typical week by selecting from five ranges of minutes that each spanned 40 minutes (1–40 minutes, 41–80 minutes, etc.). This variable was recoded as high or low, based on a median split. Respondents who selected the highest category of “161 minutes or greater” were recoded as high, and the other respondents were recoded as low.

Class size (0 = small, 1 = large). This variable was recoded as large or small, based on a median split, where classes with 20 students or fewer were classified as small and classes with 21 or more students were classified as large.

Class type (0 = not self-contained, 1 = self-contained). Teachers indicated whether their class was self-contained (i.e., that they teach multiple subjects to the same group of students) or not self-contained (i.e., that they teach the same subject to multiple classes).

Materials type (0 = textbook based, 1 = kit based). Teachers described the type of science program materials they use in their school.

• Teacher Characteristics. The regression analysis included several teacher characteristics, reflecting responses to survey items:


Graduate degree (0 = no, 1 = yes). Teachers indicated the highest degree they have earned by selecting from the following: associate’s, bachelor’s, master’s, or doctorate.

Science certificate (0 = no, 1 = yes). Teachers indicated whether they had a science education certificate by selecting Yes or No.

Teacher experience (0 = low, 1 = high). Teachers selected from five ranges of years to describe the number of years they have been teaching. This variable was recoded as high or low, based on a median split. Teacher with 10 or fewer years experience were classified as low, and teachers with 11 or more years were classified as high.

After examining the intercorrelations among these variables, the science certificate variable was dropped as a predictor because it was almost perfectly correlated with grade level. (Most secondary teachers had science certificates, and no elementary teachers had them.) The relation of the organizational support scale scores to implementation is described in their respective places in the section on organizational support. The relation of teacher characteristics to implementation is described in the section on teacher practice.


Appendix B Teacher Survey Frequency Tables

Table B-1

1. Is this class self-contained or non-self-contained?

Percentage (N = 46)

Self-contained (i.e., you teach multiple subjects to the same group of students) 33%

Non-self-contained (i.e., you teach the same subject to multiple classes of students)

67%

Table B-2

2. What is the total amount of time in minutes that you teach science to this class in a typical week?

Percentage (N = 44)

0 minutes per week 0% 1 to 40 minutes per week 5% From 41 to 80 minutes per week 14% From 81 to 120 minutes per week 11% From 121 to 160 minutes per week 20% 161 or more minutes per week 50%

Table B-3

3. What is the science content area for this class? Check all that apply:

Percentage (N = 46)

Biology/Life Sciences 59% Chemistry 30% Earth/Space Sciences 59% Physics 35% Other, please specify 15%


Table B-4 4. Please indicate the grade level(s) of the students in this class (select all that apply):

Percentage (N = 46)

K 7% 1 2% 2 9% 3 11% 4 11% 5 13% 6 9% 7 7% 8 7% 9 17% 10 24% 11 17% 12 17%

Table B-5

5. What is the total number of students in this class?

Percentage (N = 45)

0 students (I don't teach science)

0%

1–5 0% 6–10 4% 1–15 7% 16–20 27% 21–25 38% 26–30 20% 30 or more 4%


Table B-6 6. Indicate the number of students in this class who are formally classified as limited-English-proficiency students:

Percentage(N = 46)

Not applicable 65% 1 20% 2 7% 3 2% 4 0% 5 4% 6 0% 7 0% 8 0% 9 0% 10 or more 2%

Table B-7

7. Indicate the number of students in this class who are formally classified as special education:

Percentage(N = 46)

Not applicable 24% 1 20% 2 11% 3 11% 4 9% 5 17% 6 4% 7 0% 8 0% 9 0% 10 or more 4%


Table B-8

8. About how often do you do each of the following in your science instruction?

N

Almost Never

A Few Times a Year

Once or Twice a Month

Once or Twice a Week

All or Almost

All Lessons

Arrange seating to facilitate student discussion 46 2% 4% 22% 26% 48%

Assign students to work in groups on projects 46 0% 0% 7% 50% 43%

Provide opportunities for student-to-student interaction 46 0% 0% 4% 24% 72%

Engage the whole class in discussions 46 0% 0% 7% 26% 67% Engage students in small group or one-on-one discussions 46 0% 2% 11% 37% 50%

Let the questions and comments of students guide the focus of the lesson 46 0% 7% 28% 41% 24%

Provide differentiated background readings according to student reading level

46 28% 9% 35% 13% 15%

Provide students with a choice of different assignments 44 16% 36% 27% 20% 0%

Table B-9

9. About how often do students in this class take part in the following types of science activities?

N Almost

Never

A Few Times a Year


Once or Twice a Week

All or Almost

All Lessons

Generate their own ideas, questions, and/or propositions 46 2% 17% 35% 37% 9%

Share ideas or solve problems with each other in small groups 46 0% 2% 15% 43% 39%

Ask scientifically oriented questions about phenomena they observe 45 0% 7% 36% 38% 20%

Critically examine the scientific explanations of other students 43 2% 9% 42% 42% 5%

Do hands-on/laboratory science activities or investigations 43 0% 0% 7% 58% 36%

Explain their findings and conclusions to other students 45 0% 4% 28% 54% 13%

Follow specific instructions in an activity or investigation 46 0% 7% 26% 48% 20%

Plan investigations to answer scientific questions 46 4% 13% 48% 26% 9%


9. About how often do students in this class take part in the following types of science activities?

N Almost

Never

A Few Times a Year


All or Once or Almost Twice a All Week Lessons Participate in field work 45 38% 27% 22% 9% 4% Write reflections in a notebook or journal 46 4% 9% 24% 28% 35%

Prepare written science reports 45 11% 38% 40% 9% 2% Record, represent and/or analyze data 46 0% 4% 22% 59% 15% Support their explanations with scientific knowledge 46 0% 2% 37% 43% 17%

Use data to support an explanation 46 0% 2% 30% 50% 17% Work on models or simulations 46 2% 11% 50% 33% 4% Work on extended science investigations or projects (a week or more in duration) 46 11% 46% 20% 11% 13%

Make formal presentations to class 45 13% 40% 40% 4% 2%

Table B-10

10. To what extent do you think your use of inquiry-based instruction has improved the following for your students?

N Not at

All/Slightly Somewhat Moderately Very Much

So Active engagement in science lessons 45 0% 9% 36% 56% Understanding of scientific concepts 45 2% 20% 44% 33% Enthusiasm in science 45 2% 20% 44% 33% Understanding of the scientific inquiry process 43 0% 12% 51% 37%

Table B-11

11. This school year, how many times have you taught a complete learning cycle in a science unit?

Percentage(N = 44)

0 0% 1 5% 2 23% 3 25% 4 16% 5 7% 6 or more 25%


Table B-12 12. Have you experienced any barriers to teaching a full learning cycle?

Percentage (N = 41)

Yes 78% No 23%

Table B-13

13. Please indicate the title, author, publisher, and publication year of the science textbook/program used most often by students in this class.

43 Responses

Table B-14

14. Which is the best description of the science program you identified in the previous question?

Percentage (N = 42)

Kit-based program 53% Textbook-based program 47%


Table B-15 15. Please rate your agreement with the following statements about science teaching in your school:

N

Strongly Disagree Disagree Agree Strongly

Agree

Science teachers in this school regularly observe each other teaching classes in order to learn how to implement inquiry-based lessons.

43 47% 49% 5% 2%

Teachers meet to examine student work (e.g., to study the effectiveness of inquiry lessons).

42 40% 36% 24% 2%

Teachers receive in-school coaching and/or modeling to support inquiry-based instruction.

43 30% 47% 21% 2%

Teachers meet following ELI professional development sessions to discuss or plan implementation.

43 37% 40% 14% 9%

Teachers have time during the regular school week to work with each other on implementation of inquiry learning.

42 60% 29% 10% 2%

Teachers discuss what they learned with other teachers in their school or department who did not attend a particular ELI workshop.

43 24% 31% 40% 5%

Teachers discuss or share what they learned at ELI workshops with administrators (e.g., principal or department chair).

43 19% 19% 44% 19%

Teachers in this school communicate with teachers from other schools about inquiry learning.

43 33% 28% 35% 5%


Table B-16 16. To what extent does your principal, assistant principal, or department head:

N Not at

All/Slightly Somewhat Moderately Very Much So

understand inquiry-based instruction. 44 18% 36% 27% 18%

communicate expectations about implementing inquiry learning. 44 39% 30% 20% 11%

encourage teacher collaboration to support inquiry learning (e.g., lesson planning, observing lessons, reviewing student work, etc.).

44 43% 32% 14% 11%

monitor teacher implementation of inquiry learning. 44 50% 36% 9% 5%

accept the noise that comes with an active classroom. 44 2% 18% 32% 48%

encourage the implementation of current national standards in science education.

43 14% 9% 47% 30%

encourage innovative instructional practices. 43 12% 16% 30% 42%

encourage teachers to make connections across curriculum areas.

44 14% 16% 34% 36%

act as a buffer between teachers and external pressures. 42 19% 50% 14% 17%


Table B-17 17. Please rate your agreement with the following statements about your teaching resources for inquiry learning:

N Strongly

Disagree Disagree Agree Strongly Agree

I have sufficient planning and preparation time to create inquiry-based lessons. 44 27% 52% 16% 5%

I have sufficient access to computers for science instruction. 44 16% 30% 41% 14%

I have sufficient laboratory space to support inquiry learning. 43 12% 23% 44% 21%

I have sufficient laboratory equipment to support inquiry learning. 44 11% 27% 43% 18%

I have sufficient well-stocked science kits or boxes to support inquiry learning. 44 11% 27% 43% 18%

I have sufficient laboratory supplies or consumables to support inquiry learning. 44 7% 20% 52% 20%

I have instructional materials that align with inquiry learning. 44 7% 20% 52% 20%

I have sufficient funds for purchasing equipment and supplies for science. 44 18% 39% 36% 7%

Table B-18

18. In your opinion, to what extent is Every Learner Inquires consistent with each of the following?

N Not at

All/Slightly Somewhat ModeratelyVery Much

So Other science professional development programs at your school or district 44 14% 25% 30% 32%

Other school or district reform efforts 44 7% 27% 41% 25% Your own goals for professional development 44 0% 9% 27% 64% State or district standards and curriculum frameworks 43 0% 12% 42% 47%

State and district assessments 41 10% 29% 37% 24%

Table B-19

19. To what extent did the ELI workshops provide enough information on the following topics?

Not at All/Slightly Somewhat Moderately Very

Much So

What the learning cycle looks like in the classroom 0% 24% 26% 50%

What inquiry-based science instruction looks like in the classroom 0% 12% 45% 43%

Note: N = 42.


Table B-20

20. To what extent has your participation in Every Learner Inquires prepared you to do each of the following:

Not at All/Slightly Somewhat Moderately

Very Much

So Implement inquiry-based science instruction 0% 23% 44% 33% Modify curriculum units to align with the learning cycle 5% 30% 42% 23%

Differentiate instruction while implementing inquiry-based lessons 12% 49% 26% 14%

Evaluate student work products from inquiry-based lessons 14% 33% 44% 9%

Align inquiry-based lessons to local science standards 9% 47% 23% 21%

Note: N = 43. Table B-21

21. How well do you understand the learning cycle?

Percentage (N = 43)

Not at all/slightly 5% Somewhat 9% Moderately 51% Very much so 35% Not at all/slightly 5%

Table B-22

22. How familiar are you with the National Science Education Standards published by the National Research Council?

Percentage (N = 43)

Not at all/slightly 5% Somewhat 9% Moderately 51% Very much so 35%

Table B-23

23. During which school year did you begin participating in ELI?

Percentage (N = 44)

2006–2007 95% 2007–2008 5%


Table B-24 24. Which of the following workshops did you attend this school year? (Check all that apply.)

Percentage (N = 44)

Summer institute 93% September 96% November 69% January 58% February 27% April 62%

Table B-25

25. What is your gender? Percentage (N = 43)

Male 21% Female 79%

Table B-26

26. How many years have you been teaching? Percentage (N = 43 )

This is my first year 0% 2 to 3 years 2% 4 to 5 years 7% 6 to 10 years 16% 11+ years 74%

Table B-27

27. What grade level do you teach? (Select all that apply.)

Percentage (N = 43 )

K–3 30% 4–6 28% 7–8 19% 9–2 42%

Table B-28

28. Select the highest degree you have earned. Percentage (N = 43 ) Associate’s 0%

Bachelor’s 40%

Master’s 60%

Doctorate 0%


Table B-29

29. Please indicate the subject(s) of your college degree(s). Select all that apply.


Biology/life sciences 35% Chemistry 28% Earth/space sciences 9% Physics 12% Other science 5% Science education (any science discipline) 23% Mathematics/mathematics education 7% Elementary education 53% Other education (e.g., history education, special education) 14% Other, please specify 35%

Table B-30

30. Do you have a science education certificate?


Yes 43% No 57%

Table B-31

31. What is your name?

37 Responses

Table B-32

32. At what school do you teach? Percentage (N = 43 )

Harlan High School (Harlan CSD) 9% Lincoln Elementary School (Washington CSD) 2% North Cedar Elementary School (North Cedar CSD) 12% Perkins Academy (Des Moines CSD) 2% Other, please specify 74%


Appendix C Instructional Log and Observation Frequency Data

Table C-1. Frequency of Student Activities in Instructional Logs and Observations

Student Activity Instructional Logs (N = 10)

Observations (N = 12)

Percent SD Percent SD Listened to a presentation: 38% 0.21 75% 0.45 By teacher (would include demonstrations, lectures, media presentations, extensive procedural instructions) 29% 0.21 67% 0.49 By student (would include informal, as well as formal, presentations of their work) 6% 0.08 8% 0.29 By guest speaker/“expert” serving as a resource 3% 0.05 0% 0.00 Engaged in discussion/seminar 69% 0.25 67% 0.49 Whole group 55% 0.30 50% 0.52 Small groups/pairs 48% 0.29 33% 0.49 Engaged in reading/reflection/written communication about science 51% 0.20 25% 0.45 Read about science 14% 0.12 8% 0.29 Answered textbook/worksheet questions 19% 0.21 17% 0.39 Reflected on readings, activities, or problems (individually or in groups) 18% 0.09 0% 0.00 Prepared a written report of findings (e.g., from an experiment) 6% 0.07 0% 0.00 Wrote a description of a plan, procedure, or problem-solving process 10% 0.11 8% 0.29 Wrote reflections in a notebook or journal 20% 0.25 8% 0.29 Engaged in problem solving/investigation 61% 0.16 58% 0.52 Posed scientifically oriented questions 24% 0.23 0% 0.00 Worked with manipulatives 41% 0.28 0% 0.00 Did hands-on/laboratory science activities or investigations 36% 0.22 58% 0.52 Followed specific instructions in an investigation 12% 0.15 58% 0.52 Had some latitude in designing an investigation 27% 0.25 33% 0.49 Recorded, represented, and/or analyzed data 36% 0.18 17% 0.39 Recognized patterns, cycles, or trends 26% 0.27 17% 0.39 Evaluated the validity of scientific arguments or claims 11% 0.14 0% 0.00 Critically examined the scientific explanations of other students 10% 0.08 0% 0.00 Explained findings and conclusions to other students 26% 0.13 0% 0.00 Supported explanations with scientific knowledge 15% 0.12 0% 0.00 Supported explanations with data 17% 0.13 0% 0.00

Note: Instructional log percentages reflect the mean of the per-teacher percentages.


Table C-2. Frequency of Observed Use of Phases of the Learning Cycle (N = 12)

Phases 0 1 2 3 4 Engage 25.0% 16.7% 25.0% 25.0% 8.3% Explore 16.7% 8.3% 25.0% 16.7% 33.3% Explain 45.5% 45.5% 0% 0% 0% Elaborate 63.6% 27.3% 0% 9.1% 0% Evaluate 90.9% 0% 0% 0% 9.1%

Table C-3. Frequency of Observed Use of Reformed Strategies in ELI Classrooms

Reform-Oriented Strategies 0 1 2 3 4 LESSON DESIGN AND IMPLEMENTATION

The instructional strategies and activities respected students’ prior knowledge and the preconceptions inherent therein.

0% 41.7% 41.7% 8.3% 8.3%

The lesson was designed to engage students as members of a learning community. 8.3% 41.7% 25.0% 25.0% 0.0%

In this lesson, student exploration preceded formal presentation. 33.3% 8.3% 25.0% 25.0% 8.3%

This lesson encouraged students to seek and value alternative modes of investigation or of problem solving. 83.3% 16.7% 0% 0.0% 0.0%

The focus and direction of the lesson was often determined by ideas originating with students. 83.3% 0% 8.30% 0.0% 8.3%

CONTENT Propositional Knowledge The lesson involved fundamental concepts of the subject. 0% 0% 25.0% 33.3% 41.7% The lesson promoted strongly coherent conceptual understanding. 41.7% 33.3% 16.7% 8.3% 41.7%

The teacher had a solid grasp of the subject matter content inherent in the lesson. 0.0% 41.7% 41.7% 8.3% 8.3%

Elements of abstraction (i.e., symbolic representations, theory building) were encouraged when it was important to do so.

50.0% 33.3% 8.3% 0.0% 8.3%

Connections with other content disciplines and/or real-world phenomena were explored and valued. 58.3% 16.7% 25.0% 0.0% 0.0%

Procedural Knowledge Students used a variety of means (models, drawings, graphs, concrete materials, manipulatives, etc.) to represent phenomena.

41.7% 33.3% 16.7% 0.0% 8.3%

Students made predictions, estimations, and/or hypotheses and devised means for testing them. 50.0% 8.3% 25.0% 16.7% 0.0%


Procedural Knowledge (continued) 0 1 2 3 4 Students were actively engaged in thought-provoking activity that often involved the critical assessment of procedures.

100% 0% 0% 0% 0%

Students were reflective about their learning. 100% 0% 0% 0% 0% Intellectual rigor, constructive criticism, and the challenging of ideas were valued. 100% 0% 0% 0% 0%

CLASSROOM CULTURE Communicative Interactions Students were involved in the communication of their ideas to others using a variety of means and media. 91.7% 0% 8.30% 0% 0.0%

The teacher’s questions triggered divergent modes of thinking. 75.0% 8.3% 16.7% 0% 0.0%

There was a high proportion of student talk, and a significant amount of it occurred between and among students.

8.3% 16.7% 58.3% 8.3% 8.3%

Student questions and comments often determined the focus and direction of classroom discourse. 75.0% 8.3% 8.3% 8.3% 0.0%

There was a climate of respect for what others had to say. 0% 25.0% 33.3% 25.0% 16.7% Student/Teacher Relationship Active participation of students was encouraged and valued. 16.7% 25.0% 16.7% 33.3% 8.3%

Students were encouraged to generate conjectures, alternative solution strategies, and ways of interpreting evidence.

50.0% 25.0% 25.0% 0% 0.0%

In general, the teacher was patient with students. 0% 25.0% 50.0% 8.3% 16.7% The teacher acted as a resource person, working to support and enhance student investigations. 16.7% 33.3% 16.7% 25.0% 8.3%

The metaphor teacher as listener was very characteristic of this classroom. 8.30% 33.3% 50.0% 8.3% 0%


Appendix D Interview Protocols

ELI Teacher Interview — Spring 2008

I’m ____________________ and I am one of the interviewers with the Every Learner Inquires evaluation team. Thanks very much for making time to speak with me today. The purpose of this interview is to find out about your experiences with implementing inquiry-based instruction this year. This interview will take 30 to 40 minutes. Your responses will be kept confidential. We don’t identify individual respondents or their schools in any of our reports. I would like to record this interview to be sure that we have a record of all of your comments. Is this all right? If yes: OK, for the record, please state your name and school and indicate your consent to be voice recorded. Teacher Background

1. How many years have you been teaching?

2. How many years have you been at this school?

3. What sort of science certification do you hold?

4. Did your college background prepare you to teach the science content in your course? Post-Observation Follow-Up

5. We have just a few questions to follow up on the lesson we observed. What was the

purpose of the lesson? 6. Was this lesson aligned with part of the learning cycle? If so, please describe.

7. [Clarify any activities that were unclear.]

8. In what ways do you think future instruction will take into account what happened in this

lesson? Areas for Workshop Improvement

9. Which ELI workshops did you attend this year? PROBE: (specify the months)

10. What are the strengths of the workshops offered through the Every Learner Inquires initiative?

11. How should the ELI workshops be improved?


Opportunities for Job-Embedded PD

12. Has anyone at this school or outside of the school provided feedback on your inquiry lessons?

13. If so, who provided the feedback and how did they do so? 14. How helpful has this feedback been?

Administrator Support

15. Does your principal support inquiry-based instruction in your school? If so, how?

16. What are your principal’s expectations about the use of inquiry-based instruction in your school?

17. How often or for how many units does your principal expect you to use an inquiry

approach?

18. What is the level of priority that your principal gives to inquiry-based instruction? In other words how important is it for your principal that teachers are using an inquiry approach?

19. How does your principal communicate his or her expectations about inquiry instruction? 20. How is the use of inquiry-based instruction monitored in this school? PROBE: How

would anyone know if teachers are implementing it or not? School Resources for Inquiry Instruction

21. Since your school began participating in ELI, has your school purchased any materials to accommodate inquiry instruction? If so, please describe.

22. Has your school done anything else to accommodate inquiry instruction? PROBE: For

example, have you made changes to teachers’ schedules? 23. To what extent do your curriculum materials support inquiry-based science instruction?

Preparing for Inquiry Instruction

24. Do you have opportunities to plan and collaborate with other teachers on inquiry-based science instruction? If so, please describe. Probe: When and how often? Is it paid time or your own time?

25. Are these opportunities adequate to plan inquiry-based lessons? Why or why not?


Use of Inquiry Instructional Practices

26. Describe how you are typically using inquiry instruction in your classroom.

27. Has the way you are doing inquiry instruction changed since last year? If so, how? (PROBE: Have you grown in your capacity to do inquiry learning?)

28. How often do your lessons reflect an inquiry approach? PROBE: What proportion of

your lessons is inquiry based?

29. Is this a change from last year? If so, describe.

PD Effectiveness

30. What did you learn this year from ELI, if anything, that has made a difference in the way you teach?

31. How well did the Every Learner Inquires professional development prepare you to

implement inquiry learning? 32. On what topics of inquiry instruction do you feel you need additional training?

Facilitating and Impeding Factors

33. What is supporting your implementation of inquiry instruction? 34. Have you encountered any impediments to implementing inquiry-based science in your

classroom? PROBE: What are these impediments? Use of the Learning Cycle

35. Have you adapted any of your curriculum units to the 5E learning cycle? If so, in what ways?

36. Have you done a complete learning cycle? If so, how many? 37. What impediments, if any, have you encountered in using the 5E learning cycle?

Change in Classroom Culture

38. Since you began using an inquiry approach, have there been changes in the way students work together? Describe.

39. Have there been changes in your classroom role as a teacher? Describe.


Student Outcomes

40. Since you began using inquiry instruction, have you noticed any changes in student attitudes towards learning science? Probe: Are they more or less enthusiastic or confident about science?

41. Has inquiry-based science instruction affected the quality of student learning? If so, in

what ways? Probe: Do you feel the students have a better grasp on science concepts?

42. To what extent does the approach of the ELI program address students at all ability levels (such as special education students and English Language Learners)?

Closing Comments

43. Are there any external factors that are influencing the implementation of inquiry-based science, such as major changes in the school or district or other initiatives? PROBE: In what ways do these affect implementation?

44. Is there anything else that I should know regarding the Every Learner Inquiry initiative at

your school?


ELI Principal Interview — Spring 2008

I’m ____________________ and I am one of the interviewers with the Every Learner Inquires evaluation team. Thanks very much for making time to speak with me today. The purpose of this interview is to find out about your experiences with implementing inquiry-based instruction this year. This interview will take 30 to 40 minutes. Your responses will be kept confidential. We don’t identify individual respondents or their schools in any of our reports. I would like to record this interview to be sure that we have a record of all of your comments. Is this all right? If yes: OK, for the record, please state your name and school and indicate your consent to be voice recorded.

Areas for Workshop Improvement

1. Which ELI workshops did you attend this year?

2. What are the strengths of the workshops offered through Every Learner Inquires?

3. How could the workshops be improved? Opportunities for Job-Embedded PD

4. Has anyone at this school or outside of the school provided feedback to teachers on their inquiry lessons?

5. If so, who provided the feedback and how did they do so?

Administrator Support

6. What is your role in supporting your teachers in implementing inquiry-based instruction?

7. What are your expectations about use of inquiry-based science instruction?

8. What is the level of priority that you give to inquiry-based instruction? In other words how important is it for you that teachers are using an inquiry approach?

9. How do you communicate these expectations and priorities?

10. How do you monitor teacher implementation of inquiry instruction? Probe: How do you

know if teachers are doing inquiry or not?


School Resources for Inquiry Instruction

11. Since your school began participating in ELI, has your school purchased any materials to accommodate inquiry instruction? If so, please describe.

12. Has your school done anything else to accommodate inquiry instruction? PROBE: For

example, have you made changes to teachers’ schedules? 13. Do teachers now have the resources they need to support inquiry instruction?

a. If not, what else do they need?

Preparing for Inquiry Instruction

14. Describe opportunities for teachers to plan and collaborate with other teachers for inquiry-based science instruction? Probe: When and how often?

Use of Inquiry Instructional Practices

15. To what extent are teachers at this school implementing inquiry-based science instruction?

16. What have you seen or heard about that leads you to this conclusion?

17. Has the level of inquiry-based science instruction changed since last year? If so, how?

PD Effectiveness

18. How well did the Every Learner Inquires professional development prepare your teachers to implement inquiry learning?

19. On what topics of inquiry instruction do you feel you need additional training?

Facilitating and Impeding Factors 20. What is supporting the implementation of inquiry instruction in your school?

21. Have your teachers encountered any impediments to using inquiry-based science in your

classrooms? PROBE: What are these impediments? Change in Classroom Culture

22. Since teachers began using inquiry approach, have there been changes in the way students work together? Describe.



23. Have there been changes in the way teachers are carrying out their classroom role? Describe.

Student Outcomes

24. Since teachers began using inquiry instruction, have you noticed any changes in student attitudes towards learning science? Probe: Are they more or less enthusiastic or confident about science?

25. In what ways has inquiry-based science instruction affected the quality of student

learning? Probe: Do you feel students have a better grasp on science concepts?

26. Does the instructional approach of the ELI program address students at all ability levels (such as special education students and English Language Learners)?

Closing Comments

27. Are there any external factors that are influencing the implementation of inquiry-based science, such as major changes in the school or district or other initiatives? PROBE: In what ways do these affect implementation?

28. Is there anything else that I should know regarding the Every Learner Inquiry initiative at

your school?

Documents

Inquiry-Based Instruction in Iowa: A Report on the ... · The focus of the ELI professional development is to promote inquiry-based instruction. This approach, articulated in the