PEEC Cross Program 2004-05 Informal Progress · PDF filePEEC Cross Program 2004-05 Informal Progress Report Prepared for PEEC members representing The Upper Valley Community Foundation,

PEEC Cross Program 2004-05 Informal Progress Report

Prepared for PEEC members representing The Upper Valley Community Foundation, the Forest For Every Classroom program, the CO-SEED program, and the Sustainable Schools Project

Prepared by Michael Duffin, Amy Powers, and PEER Associates, Inc.

September 11, 2005

The Place-based Education Evaluation Collaborative (PEEC) is a unique partnership of organizations whose aim is to strengthen and deepen the practice and evaluation of place-based

education initiatives. PEEC programs (and organizations) include the CO-SEED Project (Antioch New England

Institute); the Community Mapping Program (the Orton Family Foundation, Vermont Institute of Natural Science, and the Institute for Technology Development); the Sustainable Schools

Project (Shelburne Farms, and the Vermont Education for Sustainability Project); and A Forest for Every Classroom Project (Shelburne Farms, The Northeast Natural Resource Center of the

National Wildlife Federation, The Marsh Billings Rockefeller National Historical Park, The Conservation Study Institute, and Green Mountain National Forest).

In addition, the Upper Valley Community Foundation provides funding and support for several of these programs through its Wellborn Ecology Fund, as well as financial, administrative and

staff support for collaborative evaluation and research efforts.

Special thanks to the individual teachers, students, community members, and CMaP staff who so graciously participated in this evaluation.

NOTE:

This report was authored by PEER Associates, Inc. Principals Amy Powers and Michael Duffin can be contacted at

[email protected] or [email protected]

PEEC Progress Report 2004-05 PEER Associates, Inc. p. 1 of 43

EXECUTIVE SUMMARY The purpose of this report is to informally update members of the Place-based Education Evaluation Collaborative (PEEC) on the progress made on the 2004-05 cross-program evaluation agenda. The report presents sections on Extending the Tipping Point Findings, and Benefits of Collaboration, followed by a brief Implications for Practice section.

Extending the Tipping Point Findings In the 2003-04 PEEC evaluation cycle, a school-level “tipping point” hypothesis emerged from analysis of survey data. Our data suggested that place-based education programs that work systematically with a whole school community for multiple years may end up impacting the school culture. During the 2004-05 evaluation cycle, we explored four areas in more detail:

Participant Interviews We interviewed 15 educators at the CO-SEED site at the Beebe School in Malden, MA. We asked them to critically assess PEEC’s 2003-04 survey findings that suggested that CO-SEED may be influencing school culture. We specifically and rigorously probed for alternative interpretations. An overwhelming amount of this interview data was clearly supportive of the assertion that CO-SEED contributes to major changes in school culture. This triangulation of survey and interview data lends substantial strength to the tipping point hypothesis. Further, interviewees at the Beebe School suggested seven different factors that have contributed to the strong school culture at that site:

• PEEC program personnel that are skillful and available; • Facilitated, common planning time for educator teams; • A school-wide environmental and health science theme; • Strong administrative support; • Healthy competition among teachers to do and display good work; • Parental involvement; and • Repetition of themes/projects on a multi-year cycle.

Review of school culture literature A tightly targeted review of research on school culture revealed several themes that are generally consistent with PEEC findings. Culture is central to both school and business success, including some associations with academic achievement, but is only one part of the big picture of school change. Coherence of culture at a school is important, and it operates at deep, affective levels on behavior and attitude. In general, school culture literature aligns with and supports the program logic driving PEEC programs.

Focus group interview with PEEC program staff A two-hour focus group on the topic of school culture tipping points took place during a semi-annual PEEC meeting in June 2005, and served both as an opportunity for


exchange of ideas among collaborative members, and data collection for the evaluation team. The main ideas generated during the focus group session fell into four themes:

• Positive shifts in school culture • Factors necessary for positive shifts • Challenges to influencing school culture shifts • Implications for PEEC professional development models

Continued refinement of quantitative survey data analysis strategies Two important findings emerged from a close look at the limited number of additional surveys that were added to the aggregate data set in 2004-05:

1) Claims based on PEEC survey data are likely to become more compelling after further “factor” analysis of the natural groupings of item-level responses; and

2) PEEC programs appear to be inspiring participants to pursue additional place-based education training.

Benefits of Collaboration In the 2003-04 PEEC Cross-Program Report, we identified our clear sense that PEEC as a whole is much greater than the sum of its parts. This year’s work further strengthened our belief in this regard. A PEEC Long Term Research Agenda document was drafted and serves as an ambitious and compelling road map for continuing PEEC’s work. All survey instruments are now available on-line through the PEEC web site, including direct access to selected survey results for some program staff. Graduate research students generated high quality, professional work, enabling more flexibility and higher productivity across the collaborative. The PEEC web site continues to be highly functional, serving to more efficiently organize and disseminate both the work of PEEC and other related research efforts. The 2003-04 PEEC Cross-Program Report was synthesized into a more concise and technical journal article format, and submitted for peer review.

Implications for practice Three main implications for practice are presented in this report:

1) Secure resources to pursue the Long Term Research Agenda; 2) Increase the priority of dissemination of findings; 3) Continue to investigate school culture.


INTRODUCTION This report is different than previous PEEC cross-program reports. The purpose is to informally update PEEC members on the progress made on the cross-program plans as opposed to formally disseminating more extensive PEEC cross-program findings to the broader public. The report is organized by regrouping the eight “line items” (5a-5g) from the 2004-05 PEEC Cross-Program Evaluation Overview (see Appendix A) into two main headings that reflect themes in the way the work actually unfolded (Extending the Tipping Point Findings, and Benefits of Collaboration), followed by a brief Implications for Practice section. In some cases, much of the substantive content is located in the Appendices, with the main body of the report reserved for more summary-type highlights. This report format also reflects a natural evolution of the PEEC agenda, roughly characterized as a shift from “collecting more data” to “making data more useful.” During 2004-05, the deliverable evaluation products for individual PEEC programs were more informal and/or tailored to more specific uses. At the cross-program level of PEEC, the evaluation plans and activities focused on laying the foundation for the next big leap in scale and impact of PEEC cross-program data. While the 2004-05 cross-program agenda had only about half the budget of the previous year, solid progress was still made on the qualitative, quantitative, and literature review fronts.



EXTENDING THE “TIPPING POINT” FINDINGS In the 2003-04 PEEC evaluation cycle, a school-level “tipping point” hypothesis emerged from analysis of survey data. Our data suggested that place-based education programs that work systematically with a whole school community for multiple years may end up impacting the school culture. This statistical pattern came to light as an incidental byproduct of the primary task of pilot testing the dose-response measurement strategy, but now has set the stage for continued evaluation to provide evidence-based support for long term program funding and design decisions. Here is a quick summary of the tipping point story the survey data seemed to tell. At veteran/exemplar PEEC sites, educators who were newer to a school or less directly involved in the PEEC program reported many of the desired teaching practices on par with educators who had been teaching at the school for several years or who were more heavily involved with the formal elements of the PEEC program intervention. Because we also had other compelling quantitative and qualitative evidence of positive impact of PEEC programs on desired teacher practices at the veteran/exemplar sites, we speculated that PEEC programs had inspired and supported norms of teaching practice that eventually became embedded in the way teaching happens at that school. Thus, the cumulative exposure of the school as a whole to the intervention (i.e., some kind of school-level “dose”) could become a more accurate predictor of desired program outcomes than exposure of the individual educators to the formal elements of the program intervention. It seemed that the school-level dose had crossed a tipping point, and the school culture change became a widespread factor influencing teacher behavior. While this tipping point hypothesis was supported by several statistical and logical tests within existing PEEC evaluation data, the interpretation required a relatively high level of inference, and so had to be treated fairly cautiously. Further critique of this hypothesis, more in depth investigation of the context, and refinement of the quantitative methodology were warranted. They were explicitly laid out as items 5d) and 5e) in the 2004-05 PEEC Cross-Program Evaluation Overview. Additionally, part of the 2004-05 CO-SEED evaluation plan was modified to overlap with this larger PEEC cross-program goal. As a result, during the 2004-05 evaluation cycle, we explored four areas in more detail:

• Interviews with CO-SEED participants to critique the school culture tipping point hypothesis;

• Tightly targeted review of literature related to school culture; • Focus group interview with PEEC program staff on the topic of school culture at

PEEC sites; and • Continued refinement of quantitative survey data analysis strategies.


Participant critique of the school culture tipping point hypothesis On May 19, 2005, we interviewed 15 educators at the CO-SEED site at the Beebe School in Malden, MA. We asked them to critically assess PEEC’s 2003-04 survey findings that suggested that CO-SEED may be influencing school culture. We specifically probed for alternative interpretations. An overwhelming amount of this interview data was clearly supportive of the assertion that CO-SEED contributes to major changes in school culture. Some of the interview data was neutral or inconclusive in this regard, and literally none of it was contradictory or disconfirming. This triangulation of survey and interview data lends substantial strength to the tipping point hypothesis. Evaluators inferred a story about school culture change from statistical analysis of survey data that was interesting and fairly compelling on its own merits but still imposed by people from outside the school context. Several educators who are rooted in the actual daily experience at a key program site independently told a very similar story but with words instead of numbers. Since different lines of inquiry have converged on similar interpretations, confidence in our assertion about this phenomenon of school culture impact is greatly enhanced. Interviewees at the Beebe School suggested seven different factors that have contributed to the strong school culture at that site:

• PEEC program personnel that are skillful and available; • Facilitated, common planning time for educator teams; • A school-wide environmental and health science theme; • Strong administrative support; • Healthy competition among teachers to do and display good work; • Parental involvement; and • Repetition of themes/projects on a multi-year cycle.

The first three factors described above emerged in all or nearly all of the interviews. The last four factors were each mentioned by one or two respondents. Data from this particular CO-SEED site figured prominently in the development of the school culture tipping point hypothesis that we proposed as a 2003-04 PEEC cross-

Quotes from educators at the Beebe School “It starts with the principal, but everybody knows [PEEC program staff people names]. Now it’s just part of the culture of the school. We’re familiar with them.” “People are cooperating, talking to people they didn’t before.” “Collaboration is kind of a built in thing.” “[Veteran teachers] sweep these new people up and into the [environmental] theme, the culture of the school. The new person seems to be able to go with it because of the comfort level.” “I think that the theme is quite embedded in the school.”


program finding. To the extent that the dynamics of place-based education at the Beebe School are representative of the cross-program logic model and working theory of change that PEEC created as a collaborative, this set of interviews strengthens the assertion that key educator practices intended by PEEC-style programs can become embedded in school cultures. Based on the body of evaluation evidence collected at other PEEC sites to date (particularly qualitative interviews and observations), we think the Beebe school is likely to be highly representative of a whole school change model place-based education program implemented well at a site that was ready to “tip.” Further investigation into a more detailed understanding of conditions that encourage or inhibit that tipping point is clearly warranted.

Review of school culture literature As with any research, it is important to draw from existing thought and research into the subject at hand. We sought input from several networks of colleagues in the fields of education and evaluation, and conducted a focused review of the more pertinent literature related to school culture. The overall impression we got from this investigation was that the connection between PEEC programs and the idea of school culture is strong and worth studying. The findings from this focused literature review seemed to resonate strongly with PEEC members as they discussed and reflected upon our presentation of this material to them during their June 2005 retreat in Shelburne, Vermont. A version of the following list of eight questions and two more summary-type topics served as the basis for the mini-workshop. Appendix C consists of slightly reorganized and expanded notes from our presentation. In this main body of the report, only the supporting notes for the final two summary topics are shown.

• Why should we investigate school culture? • How does school culture intersect with PEEC programs’ logic? • What connections do we see between place-based education and school culture? • How is school culture defined in the literature? • What are the major functions of the culture in a school? • What factors contribute to positive school culture change? • What role do parents play in influencing school culture? • How is school culture measured?

• Some overall reflections on the rationale for PEEC programs thinking more explicitly

about school culture. • Emerging implications, recommendations, and considerations for PEEC programs.


Some overall reflections on the rationale for PEEC programs thinking more explicitly about school culture

• Culture is central. “The culture of an enterprise plays the dominant role in exemplary performance. Highly respected organizations have evolved a shared system of informal folkways and traditions that infuse work with meaning, passion, and purpose” (Deal & Peterson, 1999, p. 1).

• Culture isn’t everything. School culture is only one part of the big picture in

school change, but it can be an umbrella for many things. “Simply changing the conditions teachers work under will not lead to changes in teachers’ practice; changes in teachers’ pedagogical goals and beliefs are also necessary…. School culture for teachers does not alone lead to beneficial outcomes for students” (Buck, 1992).

• Culture is associated with achievement. A very recent doctoral dissertation

compared three pairs of matched schools that differed in the amount of improvement they demonstrated over a two year period. In all three cross-case comparisons, the school with the more effective culture was also the school that demonstrated the most growth in student achievement (Schoen, 2005).

• Culture literature aligns with PEEC program logic. Several generalizations about

successfully changing school culture from Schoen (2005, p. 265) are very consistent with the underlying approach of PEEC programs, perhaps especially so for the whole school change models:

the beliefs and norms of faculty have a substantial effect on the success of change efforts

cultural change is a gradual process, it takes years to complete planned change requires much effort, those involved need a lot of

support

• Coherence matters. Schools may have multiple cultures but Schoen (2005) found that multiple simultaneous cultures were only present in the non-improving schools with lower performance scores. All the improving schools had a single unified culture. A negative example of dual cultures: several “old guard” teachers always dueling with a new, progressive administrator.

• Culture operates at deep levels. “Principals in schools that have cohesive cultures

recognize the importance of focused professional development, time for learning and reflection, and shared leadership. Cohesive schools share values and beliefs, support a common mission, and facilitate both student and faculty motivation and enthusiasm” (DiPaola, 2003).


Emerging implications, recommendations, and considerations for PEEC programs 1. Site Selection: Consciously consider school culture factors when developing and

using readiness criteria and site selection tools. All schools have a culture, which do you want to work with?

• Identify schools with strong cultures because the initiative is more likely to succeed. Alternatively, build strategies for transforming school culture into the program.

• Define how existing school culture factors in and plan accordingly. To what extent is it negative, neutral, positive?

• Recognize that leadership is critical to the evolution of school culture. Talk with potential partner schools about their experiences of leadership within the school, and their plans for how school leadership might consciously promote place-based education in the future.

2. Assumptions/Values/Norms: Since these are so important to the development of a positive school culture, PEEC programs may want to help schools to identify, shape, and promote activities and language that make these things explicit. It may well be presumptuous or insensitive for PEEC programs to advocate for specific assumptions, values, and norms. However, intentionally creating space for these conversations to happen in the context of PEEC programs seems like a reasonable consideration. Some examples might be:

• assumptions (i.e. kids are capable of strengthening community; the community is a leaning place);

• values (a deeper sense of what is important, i.e. not just tests but learning; not just succeeding personally, but taking care of others and the environment);

• norms, or “behavioral blueprints” which develop formally and informally in the form of behavior, language, dress (i.e. how one treats parents—are they enemies or collaborators; what teachers talk about over lunch—is it clothing or plant species).

3. Ceremonies and Rituals: One way culture shows up and influences behavior is through regularly occurring events with symbolic import. For instance, many schools have existing rituals such as welcoming ceremonies into which sustainability/ community/ environmental themes could be integrated. These might be the overarching theme for a ceremony, or might be incorporated in seemingly small ways such as offering a local product or natural object as a gift to new students and teachers.

4. Parents: Whenever possible, integrate parents into elements of PEEC programs (including ceremonies) as participants, not just as invitees.

5. Evaluation: Decide whether and where to focus evaluation energy in terms of

measuring and monitoring school culture patterns in PEEC schools. Consider the plusses and minuses of including school culture explicitly in program logic.


PEEC program staff focus group on school culture While quantitative measures provide us with important numerical analyses of a phenomenon that can sometimes be more simply generalized, the qualitative data gathered in tandem with it typically offers a more in-depth and detailed perspective. A focus group with PEEC program staff was designed to explore their experiences with the following evaluation questions:

• Which school-level factors cause a school to “tip”? • What types of PEEC intervention do they see contributing to this tipping? • What conditions were present in schools where program concepts really took

hold or did not take hold? • What recommendations do program staff have for increasing the odds of tipping?

This two-hour focus group took place during a semi-annual PEEC meeting in June 2005, and served both as an opportunity for exchange of ideas among collaborative members, and data collection for the evaluation team. Facilitated by PEER Associates evaluators, the session was transcribed on site and the transcript was analyzed for key themes to help shed light on the identified evaluation questions. The focus group guide is included in this report as Appendix D. Due to timing and context, the participatory activity was replaced with more standard full group discussion. Focus group participants included representatives from: the Wellborn Ecology Fund (1), Antioch New England Institute (2), Shelburne Farms (2). Collectively, these five PEEC members represented three place-based education programs and one foundation. The following is a summary of the main ideas generated during the focus group session, divided into four sections:

• Positive shifts in school culture • Factors necessary for positive shifts • Challenges to influencing school culture shifts • Implications for PEEC professional development models

Positive shifts in school culture Respondents were asked to discuss instances where they observed positive shifts in the school culture in the schools in which they have implemented PEEC programs. Their primary observations include:

• more collaborative planning among teachers; • increased parent involvement in the schools; • a spread of effect to the district level; and • a shift in teachers’ collective confidence and enthusiasm for teaching.

Several respondents talked about witnessing teachers engaged in more collaborative planning, and a subsequent “appreciation for the planning process and reflection


among teachers.” This has increased the level of intentionality teachers bring to their teaching, and has fostered more unified, thematic teaching both within and among grade levels. Respondents also reported witnessing an increase in parent involvement in the schools. Examples included a teacher who has an extremely high “dose” of one PEEC program who reports 100% participation in parent nights. She noted that this is considerably different from her past experience, and different from other classrooms in the school. The teacher attributed the parents’ increased level of engagement to having invited them throughout the school year to be more involved in a growing number of field trip experiences, and engaging them in a “Junior Naturalists” after school program. The staff working closest to her noted that these types of events represented “more points of contact” with parents since implementing PEEC-related activities and approaches. In another case, respondents observed “a group of parents organizing themselves to put together a food committee” as a result of participation in an evening designed to showcase student learning about food systems. Another respondent noted that having students present and facilitate at parents night events was a successful “hook” that drew more parents to out-of-school-time events. One respondent noted that the program seems to be having an impact on district-level culture. The PEEC program has worked from the start of their program with the district’s literacy coordinator, melding the program’s goals and strategies with that of the district’s literacy agenda. Consequently, the literacy coordinator has continued to utilize such strategies as teacher reflection “study circles” at different schools in the district, even though the initial PEEC school has since discontinued that practice. Finally, respondents from two programs discussed the increased confidence they witnessed in teachers who, because of the program’s support and authorization of their passions, felt more comfortable “coming back to good teaching” and also “trying new things.” This sort of renewed enthusiasm for their profession can form the basis for a culture that embraces positive norms about teaching.

Factors necessary for positive shifts Focus group participants were asked to reflect on the existing conditions that were present—and perhaps necessary—when a cultural shift was observed. The most agreed upon factor was an actively supportive school leadership. One example of the benefit of a school leader embracing curricular shifts offered by the place-based education program was a principal who led the teaching staff in coming up with a school-wide “essential question” around which to organize the school under a central umbrella of sustainability.


Another respondent echoed the value of a central leader. As the program started up in one school there were many positive initiatives going on throughout the school. However, it was “almost too much” in that all these strong initiatives were not coherent or centralized. What the PEEC program offered them was an opportunity to “knit together” their efforts in to one, cohesive direction. The school, she reported, had a “strong leader” whose flexibility and support facilitated this shift. Respondents noted that even if a particular principal does not have a strong affinity for place-based education strategies, the leadership function can be fulfilled if that appointed leader paved the way for a strong teacher or other in-school leader to assume that role instead. In other words, positive shifts in the culture were likely to occur as long as some form of leadership emerged to promote place-based education, and as long as a designated leader did not inhibit the spread of effect. Aside from strong leadership coming from within the school, respondents reported several other elements within their place-based education programs that were most successful at promoting cultural shifts:

• offering a program that has a “multiple layered quality” with elements such as professional development institutes for teachers, community-school teams, summer institutes, grade level planning meetings, reflection time, and teacher release time;

• the “presence of the environmental educator” in the school, linking the school to the community;

• an incremental, personalized approach to working with each school; • personalized attention to teachers’ individual needs; one respondent noted that

“one of the most crucial roles of the environmental educator and the program staff person is a polite persistence, in other words, being tediously pushy in a really nice way;”

• taking teachers away from their classrooms to engage in dialogue with their teaching teams; and

• providing teachers time to observe another exemplary program.

Challenges to influencing school culture shifts Focus group participants were asked to discuss the types of barriers to cultural shifts that they observed while attempting to implement their programs. Naturally, the presence of an unsupportive school leader, a general lack of leadership, or a high level of turnover in leadership and school faculty were cited as the strongest barriers. Also mentioned repeatedly was the barrier of time. It takes ample time for teachers to engage in new ways of collaborating, planning, and teaching, and that time is often thinly spread out amongst many responsibilities. One competing interest in particular was mentioned. When a school had other major school wide initiatives “imposed” on the school simultaneous to the PEEC initiative it served as a source of “competition” for


teachers’ time and focus. A specific example offered was a school district which, just after deciding to work with the PEEC program, had accepted Reading First money and mandated the school to embrace the program’s goals, which the PEEC respondent reported had influenced the school’s culture or norms toward literacy instruction. Respondents reported that in schools where school wide meetings were not a part of the culture as a routine way of doing business, their efforts to bring the school together in this manner were not appreciated and/or the meetings were sparsely attended. Likewise, individual teachers with entrenched practices and who were unwilling to consider new ideas were noted as challenges to the process. Many respondents discussed ways in which they had surmounted this challenge by approaching them “while standing side by side at the copy machine,” working with teachers gradually and connecting with their interests. Finally, several respondents concurred that the school building design had, in some cases, been a barrier to their attempts to engender a cultural shift. While some school buildings have a central place where everyone crosses paths at some points during the school day, those that do not tend to impede the spread of ideas and the fluidity of a program.

Implications for PEEC professional development models Not every PEEC program works with schools as a whole. Two PEEC programs are “professional development” models that work with individuals or a group of teachers from various schools. These programs do not explicitly address school culture. To factor this into our preliminary investigation, we posed the questions, “What are the implications of school-level dose for professional development model programs in PEEC?” and “How can the professional development programs target their investments to make the most lasting change?” Respondents speculated that the professional development programs are a “stepping stone” toward successful school wide program impacts. Perhaps, suggested one respondent, professional development programs “can be a gateway to greater impact on school wide culture” by serving as “feeder” programs for the whole school change initiatives. One respondent suggested that we examine this question:

To what extent did our successes with whole school models come from having the support of individual teachers who had previously engaged in professional development related to the whole school initiative’s premise?

A respondent who oversees the development and implementation of several place-based education initiatives, both professional development and whole school change models, articulated the theory behind her organization’s multi-pronged approach. She stated that when you “unleash” a variety of programs, they target different audiences


who are ready for different types of involvement. In some cases it is simply a matter of “keeping teachers in the profession.” She has heard teachers state that they had “been numb but not dead” and that participating in a place-based professional development program revived their enthusiasm for teaching. That teacher, perhaps, brings her renewed energy back to her school and can be a catalyst for broader change, particularly if a whole school change project is waiting in the wings. “At a certain point [the different types of programs] come together and you get that tipping point,” she suggested. Another advantage of professional development programs, noted another respondent, is the notion of “seeding” change around a region. This strategy has a lower price tag—and thus lower risk--associated with it than “digging in deep” with expensive whole school change initiatives in schools that may not even be “ready” for the program. The group concurred that assessing sites (and even teachers and teacher teams) for “readiness” is important. One of the programs has developed a site selection tool with criteria to be used when assessing a site prior to working there. [NOTE: This tool is included in this report as Appendix E.] Several people wondered whether having a cross-program readiness scale was warranted. This tool could be used for referring a school or teacher to a program that may be more appropriate for their level of readiness.

Refinements of quantitative data analysis Last year saw the successful pilot of using surveys to generate useful PEEC findings with a dose-response measurement strategy. The 2004-05 PEEC cross-program survey task consisted mostly of long term planning and building out and/or refining the survey instruments and access to them. These are described later in the Benefits of Collaboration section of this report. The official PEEC cross-program quantitative data analysis was limited to the data from 10 Educator surveys from the FFEC 4 cohort and a quick scan of selected items from 19 Educator surveys from CMP.1 Two important findings did, however, result from a close look at the available numbers:

4) Claims based on PEEC survey data are likely to become more compelling after further “factor” analysis of the natural groupings of item-level responses; and

5) PEEC programs appear to be inspiring participants to pursue additional place-based education training.

1 Last year, the evaluation plans of the four PEEC program generated 355 Educator, 113 Community member, and 1,623 Student surveys. This year, CO-SEED administered far fewer surveys, and those, like the surveys that SSP administered, are slated for analysis next year. CMP generated 19 Educator surveys and 50 Student surveys, but organizational transition factors at the national level of that program meant that CMP was not able to formally participate in the PEEC cross-program effort.


Factor analysis of PEEC survey data is warranted One benefit of the dose-response measurement strategy is that the data set is cumulative. Adding this year’s FFEC survey data in with responses from previous FFEC surveys both affirmed previous findings and led to new insights. In particular, a very interesting pattern emerged: all the strongest index-level outcomes in the FFEC educator survey data to date were about getting FFEC participants (students and adults) out into the community and connecting with local resources. This led us to recombine several survey items into a new “all-star” module, which we labeled connection to community. Subsequent dose-response analysis generated the strongest effect size that we have seen yet for any PEEC outcome ( R2 = .41, p < .00, df = 41). This argument is presented in more detail in the 04-05 FFEC Educator Survey Results Informal Report. Based upon this exciting finding for FFEC, we re-analyzed the educator survey data from all the suitable 2003-04 PEEC program data sets to see if this connection to community module showed a similar pattern at the cross-program level of analysis. In short, the pattern did not persist,2 leaving us with the interpretation that this particular combination of items tells us more about FFEC than it does about PEEC cross-program or any of the other individual PEEC programs.3 The main point we want to make here, though, is that the analysis process reflected above (i.e. the emergence of this connecting to community module for FFEC) may well be the first step toward a developmental advance in PEEC’s quantitative analysis strategy. It may be worth taking a moment to describe the reasoning behind this next layer of statistical analysis. Up to now, our outcome measures have been rationally organized. In other words, we sat down ahead of time and combined the survey items into clusters (“modules”) that we thought made sense based upon our own thinking as evaluators. Rationally derived scales have the advantage of being readily interpretable because the story of their meaning exists prior to data collection. An alternative, potentially complementary strategy for clustering survey items, relies on statistically sifting through survey responses to identify groups of items that tend to vary together. This is called “factor analysis.” The assumption is that groups of items to which people respond in synchronized ways, are likely tapping into some common underlying factor. Even though the meaning of the underlying factor becomes somewhat speculative, letting the pattern of

2 Effect sizes for individual data sets ranged from R2 = -.05 to .17, p > .05, and the effect size for the aggregated cross-program data set was R2 = .08, p < .01, df = 263. 3 On a more technical note, it should be noted that regression statistics are designed to capitalize on chance configurations within a set of data by constructing an equation that gives the best possible results from the data you give it. The acid test of whether this equation represents some “real world truth” is to try it on a data set from which it was not derived. The “all-star” failed that test, and thus may possibly be attributable in large part to chance configurations of the FFEC data. Further quantitative and/or qualitative exploration of this pattern is warranted.


survey responses guide the grouping of items creates empirically derived scales that lend an alternative type of credibility to the use of the instrument. This kind of statistical exploration of our instruments and analysis strategies is embedded in the PEEC Long Term Research Agenda document. [Note: See Appendix F.] The fact that “factor analysis”-type findings are emerging organically from the FFEC data is further evidence of the important contribution that individual programs (in this case, FFEC) make to the overall PEEC data analysis effort.

PEEC inspired additional training This year we modified all PEEC surveys to collect more detailed information about non-PEEC place-based education training. Combining data from this year’s FFEC and CMP Educator surveys has helped us begin to unravel the impacts of PEEC programs versus these other influences. The preliminary descriptive data in the table below shows clearly that about two-thirds of the participants surveyed so far (i.e. 19 out of 29) report that their PEEC program has inspired them to pursue additional place-based education training.

D.1z. Of the non-[PEEC] place-based or environmental education activities you listed in D.1v-y above, approximately what portion of these did you do as a result of being inspired by [PEEC]:

Response Percent

Response Total

a. none 17% 5

b. about a quarter 26% 8

c. around half 17% 5

d. maybe three quarters 7% 2

e. all 14% 4

f. I’m not sure/couldn’t guess 17% 5

Total Respondents 29

In terms of the sequential regression statistical analysis that our dose-response measurement strategy employs, it is unclear just how to include non-PEEC PBE training in the equation. Last year, we controlled for non-PEEC PBE training in the data sets that included that variable. This year’s data (shown above) suggest clearly that this approach was not as accurate as it might be since for most participants, non-PEEC PBE training was an outcome of the PEEC program dose rather than a pre-existing factor that should be subtracted out of the PEEC program dose. Thus, controlling for non-PEEC PBE training under-estimated the effect sizes, effectively “robbing” the PEEC program dosage factor of some of its statistical effect on the outcome. Even though we have not yet figured out exactly how to include these new insights into the sequential regression analysis, this turn of events, like the factor analysis discussion above, is also evidence of the way individual PEEC programs contribute to refining PEEC’s quantitative measurement strategy.


BENEFITS OF COLLABORATION: In the 2003-04 PEEC Cross-Program Report, we identified our clear sense that PEEC as a whole is much greater than the sum of its parts. This year’s work further strengthened our belief in this regard. Below are brief updates and/or reflections on each of the line items from the 2004-05 PEEC Cross-Program Evaluation Overview that have not been reported on thus far in this informal report.

5c) On-going refinement of survey infrastructure (including developing the “one-pager” document) Effort to develop a one-page synthesis of the PEEC cross-program agenda’s present status and future direction resulted in the creation of the PEEC Long Term Research Agenda document that is included as Appendix F in this report. This document can now provide a relatively clear (while continuously evolving) road map for the next steps of this shared research and evaluation journey. The fact that the potential budgets described in this document far outstrip current resources is one indicator of the power of this collaborative. It is much more difficult to imagine that any one PEEC program could even consider such potentially far-reaching research and evaluation projects on its own. As described earlier in this report, most of the cross-program quantitative effort went into survey infrastructure. All PEEC surveys were revised and updated to reflect the many incremental improvements in wording, organization, and format that emerged from last year’s pilot of the instruments. All survey instruments are now available at www.PEECworks.org/PEEC_Inst to be administered on-line as appropriate, or to be downloaded from the PEEC web site and printed out for paper administration. Further, web infrastructure has been developed to allow program staff direct access to summaries and, in some cases, detail views of participant responses. Development of this functionality was driven primarily by CMP, but also replicated for CO-SEED. At this point, this functionality can be replicated for other PEEC programs with relative ease and speed should that become a priority as expressed in evaluation plans for succeeding years.

5a) PEEC meetings PEEC meetings continue to be an invigorating and useful activity for both PEEC members and the evaluation team. In an effort to be more efficient with time commitments, all evaluators did not participate in all PEEC meetings. This strategy is likely to continue for the coming year.

5b) Utilize existing literature review It was deemed to be more useful to spend time preparing and presenting the review of school culture literature than to dig deeper into the findings from last year’s literature


review of educational reform, diffusion of innovations, student achievement, and other topics. This is evidence of what we see as a strategic tension that is likely to persist. Namely, there will always be more research literature available that is interesting, relevant, and informative for PEEC practice than there will be time for in the evaluation plans. That said, it is probably a reasonable goal at this stage of PEEC’s evolution to choose one topic a year to investigate in more depth via a focused literature review.

5f) Student projects The purpose of this line item was to allow PEER and PEEC the flexibility to engage student research assistants in projects that most effectively balanced their skills, project needs, and the ‘moving target’ nature of individual program and cross-program evaluation efforts. This year, those projects ended up being: a review of literature on fellowships for FFEC; developing site-specific reports of descriptive survey results for CO-SEED; investigation into standardized test scores for CO-SEED’s Gorham site; transcription of interviews for SSP, FFEC, and CO-SEED; and web-based and paper-based survey infrastructure work for all PEEC programs. Initial efforts were made to support a doctoral student from another university in doing a qualitative investigation of PBE at CO-SEED’s Malden site, but that turned out not to be a good fit for the participants at that site.

5g) Manage the PEEC web site The PEEC web site at www.PEECworks.org continues to be highly functional. Between January and August of 2005, the site received 3,678 unique visitors, with about 1/3 of those visiting the site just once, and about 1/3 of the total hits coming from the 56 unique visitors spending more than an hour at the site. Presumably this latter group of more in-depth users of the site are PEEC clients and their immediate networks of stakeholders. The most downloaded resources from the site are the PEEC evaluation reports, with 635 separate instances. The single most popular item is the 2003-04 PEEC Cross-Program Report. The most popular entrance point to the site is the page that links users to our growing list of relevant research, followed in popularity by the page that links users to evaluation and research tools. Overall, maintaining this web site seems like an effective and efficient use of PEEC resources.

5h) Write an article for publication in a scholarly journal The 2003-04 PEEC Cross-Program Report was synthesized into a more concise and technical journal article format. The purpose was to broaden the range of venues and networks through which PEEC’s work is disseminated. Ultimately, publication in the peer-reviewed academic literature will be one of the ways in which PEEC’s work gains the credibility and stature that will most effectively support program justification. This will also augment the


spread of the findings by becoming part of the stream of information listed in academic databases which are the primary source of literature review for most researchers. One of the challenges of this publication route, however, is that there is often a long lag time between generation of research and its publication. Typically the process stretches to a year, sometimes less (as in PEEC’s experience last year with its unusually brisk publication in the Journal of Environmental Education), sometimes more. After seeking input from PEEC advisors and from faculty in Michael Duffin’s Ph.D. program, we decided to craft the article for submission in the peer-reviewed journal, Educational Evaluation and Policy Analysis. This journal of the American Educational Research Association is fairly highly regarded in academic circles, and is fairly selective in what it publishes (accepting about 15% of the manuscripts submitted). As our draft was completed, however, we decided to submit it first to a journal with more name recognition value for PEEC clients, the Harvard Education Review, even though it was more of a long shot due to stiffer competition. The article was not accepted by HER, and so was promptly submitted to the original journal, EEPA. As of this writing the article is in the review process for that journal, which claims to respond about publication status within about four months. In the meantime, below is the abstract and concluding paragraph for the manuscript we submitted.

Abstract…

Evaluating Place-Based Education Using a Dose-Response Measurement Strategy This article describes a strategy devised to evaluate educational outcomes across diverse naturalistic settings at modest cost. Employing a measurement strategy more typically associated with natural science applications, we demonstrated significant relationships between educators’ degree of exposure to innovative educational interventions, and their adoption of behaviors the interventions are designed to promote. Evidence of student- and whole school-level effects emerged as well. Findings are based on surveys from 338 educators (grades K-12) and 721 students (grades 4-12), from 55 public schools spanning four different place-based education programs operating mostly in New England. Results are part of an ongoing, multi-program, multi-method evaluation effort conducted by the Place-based Education Evaluation Collaborative (PEEC, 2005a).

…Concluding Paragraph

Place-Based Education as a Viable Strategy The programs of the Place-based Education Evaluation Collaborative are showing defensible evidence that their interventions in K-12 schools are achieving many of the most important intended effects for participating educators, students, and schools. The diversity of the programs sampled for this study suggests that these results may begin to predict the results for other similarly designed and implemented place-based education programs. Although this study was not designed with a primarily summative purpose in mind, the results so far are certainly consistent with a continued investment in the four particular PBE programs evaluated.



IMPLICATIONS FOR PRACTICE What follows are four implications for practice for PEEC to consider as they plan and execute evaluation efforts for 2005-06. The first two are oriented more toward overarching strategy issues, and the second two are more logistically oriented. 1) Secure resources to pursue the Long Term Research Agenda. We include this

implication at the risk of sounding somewhat trite and obvious. However, the need is great for quantitative measures that reflect the less tangible but perhaps more important outcomes of place-based education. Further, the tipping point hypothesis that has emerged from PEEC’s dose-response measurement strategy does hold substantial promise for becoming a key piece of a credible alternative approach to the No Child Left Behind mindset. The collaborative structure of PEEC, coupled with the growing volume and quality of evaluation data gathered to date, seems to place PEEC at the doorstep of opportunity to make a notable impact on the field. The Long Term Research Agenda is a good framework for continuing on this path. Until resources are secured to invest more fully in this agenda, it is important to continue to chip away at it, generating useful findings from even small amounts of new data in much the same way that the FFEC and CMP data this year were able to do.

2) Increase the priority of dissemination of findings. With all the focus on the long term

goals of what PEEC could achieve in the future, it is easy to lose sight of what PEEC has already achieved. At least four major findings over the last few years are definitely worth promoting and sharing within the networks that PEEC cares about. First, individual and cross-program evaluations show strong evidence for positive impact of these programs. That alone is news that deserves to be shared. Second, qualitative cross-program analysis in 2002-03 revealed six common themes for the ways in which PEEC programs are reported to positively impact program participants. Third, 2003-04’s pilot of the dose-response measurement strategy generated an innovative, useful set of analysis tools as well as providing quantitative findings that largely align with existing qualitative data. The tables and graphs tell a similar story to the words we’ve collected. Fourth, in 2004-05, interviews with program participants plainly supported our previous, more tentative tipping point interpretation of last year’s survey data. This adds substantial strength to our assertion that PEEC programs seem to become embedded in a school’s culture.

In the absence of major investments materializing for PEEC’s long term research agenda, investment in dissemination efforts are perhaps the highest leverage use of PEEC resources. This might include presentations at more conferences by individuals from PEEC and/or PEER. Also important to consider would be the development of a handful of more glossy, summary type brochures or “report cards” targeted specifically to key program stakeholders that have the ability to help PEEC move to the next level. To this end, the most valuable role for PEER to play is


probably text writers, as well as consultants or advisors to graphic design experts who are actually creating these dissemination materials, including providing key textual content. The same roles may make sense for the generation of articles targeted for popular press magazines or newsletters. Further publication in the academic literature can probably take a back seat to less scholarly dissemination efforts until bigger pieces of the long term research agenda yield results. An exception to this would be if investigations into student achievement that are being currently carried out mature into clearly presentable studies.

3) Continue to investigate school culture. Existing research literature seems to align

very well with PEEC’s own data about the interplay between place-based education and school culture. An earlier section of this report detailed five recommendations to consider in this regard.

4) Consider adding new members to PEEC. One of the clear strengths of PEEC is the

diversity of programs represented, despite the fact that all the programs are aligned in their pursuit and implementation of common place-based education theories and practices. The formal departure of CMP compromises this diversity somewhat. Bringing on a new program partner or two could add analytic power to PEEC data analysis in addition to enhancing the other benefits of collaboration that have been described in this and last year’s cross-program reports.

PEEC

Pro

gres

s Rep

ort 2

004-

05

PEER

Ass

ocia

tes,

Inc.

p. 2

3 of

43

App

endi

x A

: 200

4-05

PEE

C C

ross

-Pro

gram

Eva

luat

ion

Ove

rvie

w


Appendix B: Malden CO-SEED Site Interview Guide NOTE: Interviews done May 19, 2005 at the Beebe School in Malden, MA. Intro: • I am outside evaluator with PEER Associates, hired by CO-SEED for the last three years to

help improve the program. • Request permission to record, take notes, transcribe. • Your responses are confidential in that names are never used. Quotes are used. Only

evaluation staff will see raw data. • Data from these interviews will be written up in an informal report given to CO-SEED staff

to share as they deem appropriate. Data will also be transcribed and systematically analyzed for possible inclusion in a formal, public report on CO-SEED as a whole during 06. May end up indirectly informing my Ph.D. dissertation as well.

• Main purpose is to critically challenge previous evaluation survey findings that suggest that CO-SEED may be influencing school culture.

• Questions or concerns?

1) Do you experience Beebe as having something that could be called a distinct school culture? If so, how would you describe it? (Additional prompts: Is the culture strong or weak? Consistent or variable? Do you experience a difference between a culture for teachers and a culture for students? How, if at all, are those related? Are there “in” or “out” groups? Do you see your teaching as in line or at odds with the main school culture?)

2) Have you seen or felt any changes in Beebe’s culture in the time you’ve been connected with the school? (Additional prompts: Can you tell me a story about a time you noticed a change? When did it happen? What else was going on at the time? Who was involved?)

3) If you had a magic wand that was only half-strength, what would you do to make the school culture at Beebe the best it could be? (Additional prompts: What do you see as the main things that influence Beebe’s culture?)

4) Take a look at last year’s survey data. In what ways is our interpretation NOT consistent with your personal experience at Beebe? (Additional prompts: Can you think of other explanations for this survey data? To what extent do you think these surveys measure something real?)

5) What are the biggest factors affecting Beebe students, teachers, and community in the last five years that are NOT related to CO-SEED? How do these compare to CO-SEED in terms of the relative size/strength of their influence?

6) What aspects of CO-SEED do you think are going to have the most lasting impact? Why?

7) Is there anything about the idea or experience of school culture, either at Beebe or in general that you are especially curious about? If so, what?

8) Anything else you’d like to share?

THANK YOU!!!


Appendix C: School Culture Lit Review Supporting Materials On June 20, 2005, in Shelburne, Vermont, PEER Associates facilitated a presentation and discussion of school culture literature with the members of PEEC. A version of the list of eight questions and two reflection topics below served as the basis for the presentation. The notes presented here in this report have been slightly reorganized, clarified, and expanded to include a few more details and references for each question/topic in an attempt to maximize the utility of this material.

• Why should we investigate school culture? • How does school culture intersect with PEEC programs’ logic? • What connections do we see between place-based education and school culture? • How is school culture defined in the literature? • What are the major functions of the culture in a school? • What factors contribute to positive school culture change? • What role do parents play in influencing school culture? • How is school culture measured? • Some overall reflections on the rationale for PEEC programs thinking more explicitly

about school culture. • Emerging implications, recommendations, and considerations for PEEC programs.

Why should we investigate school culture?

• Cultural patterns are: 1. highly enduring 2. have a powerful impact on performance 3. shape the way people think, feel, act

• School culture literature intersects with the organizational culture literature, which has become an increasingly powerful tool for understanding and improving businesses.

• A positive school culture has been shown to be an important predictor of effective schools (see reference list at the end of this section).

• “Parents, teachers, principals, and students have always sensed something special, yet undefined, about their schools—something extremely powerful but difficult to describe. This ephemeral, taken-for-granted aspect of schools is often overlooked and consequently is usually absent from discussions about school improvement.” (Deal & Peterson, 1999, p. 2)

• Harmon (1996) studied the extent to which various characteristics specific to individual teachers contribute to differences in professional norms. In this study “interdisciplinary team membership was found to be a significant predictor of professional norms. The length of time a teacher has taught in a particular


building usually was also found to be a significant predictor of these norms. This suggests that the extent to which norms are adopted by teachers may be the result of the interaction of both on-the-job socialization processes and school structure” (p. 67). [NOTE: This finding is particularly interesting in light of the prominence that common planning as an interdisciplinary team played in the CO-SEED Beebe School site interviews described earlier in this informal report].

• The first of Eleven Key Attributes of School Effectiveness (Stoll & Fink, as cited in Earl & Lee, 1998, Chap. 3 in an unnumbered document) refers directly to school culture or climate, and several others of the key attributes are directly tied to it:

1. Climate Setting 2. Vision 3. Joint planning 4. Leadership 5. Involvement and empowerment 6. Partnership 7. Mentoring and evaluation 8. Problem-solving 9. Staff development and resource assistance 10. Adapting management structures 11. Creativity in relation to external mandates

How does school culture intersect with PEEC programs’ logic? The literature we reviewed seems to suggest that school culture is a factor that can be directly impacted by programs or other interventions while it simultaneously interacts in a reinforcing feedback loop with behavior changes of educators and students, perhaps as displayed in Figure 1 below. PEEC programs may or may not want to consider including explicit references to school culture in their collaborative or individual program logic models.

Program in a school

Changes in school culture

Changes in educator practice/ student learning

Longer term outcomes

(e.g. improved community)

Figure 1. School Culture Basic Theory of Change


What connections do we see between place-based education and school culture? • “The culture of an enterprise plays the dominant role in exemplary performance.

Highly respected organizations have evolved a shared system of informal folkways and traditions that infuse work with meaning, passion, and purpose.” (Deal & Peterson, 1999, p. 1)

It is interesting to note the philosophical connection between place-based education and the very idea of school culture. Connecting to the local community, for example, is one way of allowing a school to better access the “folkways” and traditions of the community. Also, the idea that school culture “evolves” rather than just being adopted, is akin to PEEC programs’ efforts to work with a school to evolve the right program for that site.

Much like successful place-based education programs do, the most successful school cultures:

• “blend internal and external features” • “tie the history of the school to the history of the community” • “build bonds with community” (Deal & Peterson, 1999, various pgs.)

Like place-based education’s emphasis on civic engagement, it is important to consider how embedded the idea of school culture is in the tradition of American public schools: Deal & Peterson (1999, p. 30) talk about “re-embrac[ing] the mythology that launched the public school system in USA:

• school should be a place to create a sense of community • each student should be able to realize his/her potential • each student can become a greater American”

Since place-based education and school culture have similar philosophical approaches, an analysis of the two realms may serve for theoretical triangulation, i.e. examining the convergence of the emerging theory of the importance of place-based education and the emerging theory of the importance of school culture. There is a shared purpose in that both honor the importance of schools emerging from isolation, connecting with community, etc. Question for exploration: Does the use of place influence school culture positively? Or does a positive school culture demand the use of local place?

How is school culture defined in the literature? • “…the way we do things around here” (Bower, as cited in Deal & Peterson, 1999,

p. 3)


• “Culture consists of the stable, underlying social meanings that shape beliefs and behavior over time” (Deal & Peterson, 1999, p. 3).

• School culture is “a system of shared orientations (norms, core values, and tacit assumptions) held by members, which holds the unit together and gives it a distinct identity” (Hoy, Totter, & Kottkamp, as cited in Roach & Kratochwill, 2004, p. 12).

• “…the way of being and doing that prevails at a given school. School culture evolves over time and is the product of a convergence of factors both internal and external to the school” (Schoen, 2005, p. 256).

• “We believe the term culture provides a more accurate and intuitively appealing way to help school leaders better understand their school’s own unwritten rules and traditions, norms, and expectations that seem to permeate everything: the way people act, how they dress, what they talk about or avoid talking about, whether they seek out colleagues for help or don’t, and how teachers feel about their work and their students” (Deal & Peterson, 1999, p. 2).

• Four Dimensions of School Culture (in Schoen, 2005) 1. Professional Orientation (the activities and attitudes that characterize the

degree of professionalism present in the faculty) 2. Organizational Structure (the type of leadership, communication and

processes that characterize the way the school conducts its business) 3. Quality of the Learning Environment (the intellectual merit of the

activities in which students are typically engaged) 4. Student-centered Focus (the programs and services offered to support

student achievement)

What are the major functions of culture in a school? Deal & Peterson (1999, p. 7-8) cite six major functions and impacts of culture on schools: “Culture…

• … fosters school effectiveness and productivity • … improves collegial and collaborative activities that foster better

communication and problem-solving practices • … fosters successful change and improvement efforts • … builds commitment and identification of staff, students and administrators • … amplifies the energy, motivation, and vitality of a school staff, students,

and community • … increases the focus of daily behavior and attention on what is important

and valued.”


Helping a school to develop a strong culture will aid in the development of all of these benefits.

What factors contribute to positive culture change? The following factors in building positive school cultures are used to organize the chapters of Deal & Peterson’s excellent summary reference book, Shaping School Culture: The Heart of Leadership (1999). That book provides many examples and citations. A few additional examples and/or elaborations are provided below for each set of factors. Mission, purpose, vision, values, norms

“School leaders, faculty, and students who share common values and beliefs work more effectively together. They share greater trust, respect one another, and are more likely to take risks” (Hughes, as cited in DiPaola, 2003, p. 17).

Examples: Unspoken positive norms like “we encourage creativity amongst students and staff” or “we take care of the school yard and school building” foster a more positive school culture than negative norms such as “it’s okay to complain about the principal all the time” and “follow the rules and don’t ask questions.”

Rituals, traditions, and ceremonies

Example: Principal greets children each day with a handshake or high five. Example: At the beginning of the year, offering rituals such as a welcoming

ceremony in which each new member of school community is honored and welcomed publicly rather than an anonymous integration.

History, Stories

Example: pre-school in-service day that has staff break up into decade groups, recount the history of the school during that decade, present back to the overall group—keep the spirit of the school and community alive over time—not just working in a void.

Within a school culture there are often people who informally take on the role of “storytellers,” sometimes more as “priest/priestess” who uses stories to guide and lead, sometimes more as “gossip” who can either make sure that key information is spread effectively through the culture or reinforce a toxic culture with negativity.

Architecture, Symbols

Example: Deal & Patterson (1999, pp. 15-21) describe how Ganado Primary School building physically embodies the culture. The entry way has a smaller likeness of “spider rock” an important spiritual, natural place in the local community. People are greeted by this familiar symbol each day and it reinforces the connections between school and local culture and community.


Example: Physical spaces can enhance culture. In some schools, the school building is divided into quarters, each of which functions as a neighborhood, or a “family” subculture.

The CO-SEED site in Vinalhaven, ME exemplifies both of the points above, but with lobster boats and granite slabs for the entry, and calling the four sub-communities within the school “islands.”

Collaborative Planning and Reflection

“School faculties that are provided numerous formal and informal opportunities to engage in loosely structured dialogue centered on revising goals and developing a common vision for ‘the way things should be’ have an inherent advantage. Structures that provide teachers with the time and the questions upon which to reflect can go a long way towards establishing a strong Professional Orientation (Dimension I, see above) because the stimulate teachers to think deeper about their practices….” (Schoen, 2005, p. 259).

Example: Collaborative study groups on chosen topics like how to teach poetry.

Example: SSP’s Critical Friends Groups. Example: Grade level planning days that happen as prominent CO-SEED

activities as some sites, especially Beebe, Young Achievers, and Haley School. Strong Leadership

“Successful principals understand the importance of school culture as a key variable in effective change” (Deal & Peterson, Fullan, & NAESP, as cited in DiPaola, 2003, p. 17).

The most successful schools in Schoen’s (2005) study were: those with the most effective school communication patterns; and those whose principals structured teacher time in ways to facilitate a strong Professional Orientation (Dimension I, see above), such as school wide strategic planning.

SSP example of how a change in school culture can self-perpetuate: SSP teachers with a high “dose” of SSP expressed skepticism that SSP-related initiatives would continue when individual teachers retire or move on. They emphasized how critical being part of the dialogue was for them. The school’s principal, however, was more confident because she incorporates an SSP “screen” into the hiring process. She also noted that when she herself retires, the hiring committee will be comprised of like-minded teachers she hired.

What is the role of parents in influencing school culture? Since many people believe that involved parents hold schools accountable, a common goal of many schools around the country today is to increase parent involvement. Parents are a key aspect of school culture. Place-based education initiatives honor and actively seeks out parents' local knowledge which may far exceed their “book”


knowledge and school comfort zone. For example, a father born in the school’s town and working on the local road crew has local knowledge which—if valued and welcomed--can encourage him to engage more in the school, thereby making a difference in the culture. Place-based education offers tools, resources, and mechanisms for involving parents. A study of parent-school relationship in a Northern California inner city showed that parents:

• valued education • wanted to help children with school • wanted to assist in school • wanted to feel comfortable in the school • were pretty sure they’re not always wanted, welcomed or listened to

(Smrekar, as cited in Deal & Peterson, 1999, p. 132)

It is important to honor diversity within communities external to the school. Embracing just one dominant ethnic culture, for instance, is at the exclusion of the minority students, whatever they may be. Be sure that creation of strong internal school culture does not exclude parents’ “other” cultures. It is important to create symbolic bonds across school perimeter to invite families to participate.

How is school culture measured? Quantitative Options • Revised School Culture Elements Questionnaire (RSCEQ)—sent 2 inquiries • Comprehensive Assessment of School Environments (CASE), including the CASE

School Climate Survey (55 items administered to grades 6-12, teachers, and parents to assess their perceptions about 10 dimensions of school climate. Larger package includes Satisfaction Surveys administered to parents, teachers, and students, Teacher Report Forms for collecting information about teachers’ perceptions of school and district leadership, student report forms.

• Organization Health Inventory (OHI) and Organizational Climate questionnaire (OCDQ) both have separate instruments for using elementary, middle and high schools

Qualitative Options • Descriptive-Cross Case Comparisons of school culture (contrasting case studies) • Ethnographic and participant observation methods help to assess participants’

interpretations, what meaning they make (surveys are more descriptive) • Critical Incident Analysis—note significant moments, triangulate perspectives of

observer and interviewees about the incident, use given probing questions; analyze a collection of critical events and present/reflect


• Quality Improvement Tools—examine staff members’ attitudes and beliefs about the school’s work patterns and org structure, complete QITs with focus groups, allows people to probe deeper and collectively form conclusions

Also, Roach and Kratochwill (2004, p. 12) discuss the difference between measuring school climate and school culture. They suggest it may be more reasonable to evaluate school climate which is less abstract than culture. Climate is the “personality” of the school where culture embodies the more complex realm of unspoken norms. This may be worth further exploration.

Some overall reflections on the rationale for PEEC programs thinking more explicitly about school culture

• Culture is central. “The culture of an enterprise plays the dominant role in exemplary performance. Highly respected organizations have evolved a shared system of informal folkways and traditions that infuse work with meaning, passion, and purpose” (Deal & Peterson, 1999, p. 1).

• Culture isn’t everything. School culture is only one part of the big picture in

school change, but it can be an umbrella for many things. “Simply changing the conditions teachers work under will not lead to changes in teachers’ practice; changes in teachers’ pedagogical goals and beliefs are also necessary…. School culture for teachers does not alone lead to beneficial outcomes for students” (Buck, 1992).

• Culture is associated with achievement. A very recent doctoral dissertation

compared three pairs of matched schools that differed in the amount of improvement they demonstrated over a two year period. In all three cross-case comparisons, the school with the more effective culture was also the school that demonstrated the most growth in student achievement (Schoen, 2005).

• Culture literature aligns with core PEEC logic. Several generalizations about

successfully changing school culture from Schoen (2005, p. 265) are very consistent with the underlying approach of PEEC programs, perhaps especially so for the whole school change models:

the beliefs and norms of faculty have a substantial effect on the success of change efforts

cultural change is a gradual process, it takes years to complete planned change requires much effort, those involved need a lot of

support

• Coherence matters. Schools may have multiple cultures but Schoen (2005) found that multiple simultaneous cultures were only present in the non-improving schools with lower performance scores. All the improving schools had a single


unified culture. A negative example of dual cultures: several “old guard” teachers always dueling with a new, progressive administrator.

• Culture operates at deep levels. “Principals in schools that have cohesive cultures

recognize the importance of focused professional development, time for learning and reflection, and shared leadership. Cohesive schools share values and beliefs, support a common mission, and facilitate both student and faculty motivation and enthusiasm” (DiPaola, 2003).

Emerging implications, recommendations, and considerations for PEEC programs

1. Consciously consider school culture factors when developing and using readiness criteria and site selection tools. All schools have a culture, which do you want to work with?

• Identify schools with strong cultures because the initiative is more likely to succeed there OR conscientiously build in ways of transforming culture into the program.

• Define how existing school culture factors in and plan accordingly. To what extent is it toxic, neutral, positive?

• Recognize that leadership is critical to the evolution of school culture. Talk with potential partner schools about their past experiences of leadership within the school, and their plans for how school leadership might consciously promote place-based education in the future.

2. Assumptions/Values/Norms: Since these are so important to the development

of a positive school culture, PEEC programs may want to help schools to identify, shape, and promote activities and language that make these things explicit. It may well be presumptuous or insensitive for PEEC programs to advocate for specific assumptions, values, and norms. However, intentionally creating space for these conversations to happen in the context of PEEC programs seems like a reasonable consideration. Some examples might be:

• assumptions (i.e. kids are capable of strengthening community; the community is a leaning place);

• values (a deeper sense of what is important, i.e. not just tests but learning; not just succeeding personally, but taking care of others and the environment);

• norms, or “behavioral blueprints” which develop formally and informally in the form of behavior, language, dress (i.e. how one treats parents—are they enemies or collaborators; what teachers talk about over lunch—is it clothing or plant species).

3. Ceremonies: Recommend to integrate sustainability/ community/ environment,

etc. into existing rituals. It may be in big, overarching themes or it might be in


seemingly small ways such as incorporating a local product or natural object as a ceremonial gift into annual welcoming ceremony for new students and teachers, or into the start or end of staff meetings.

4. Parents: Whenever possible, integrate parents into elements of PEEC programs

(including ceremonies) as participants, not just as invitees.

5. Evaluation: decide whether and where to focus evaluation energy in terms of measuring and monitoring school culture patterns in PEEC schools. Consider the plusses and minuses of including school culture explicitly in program logic.

References Cited in Presentation Notes Buck, R. (1992, April). Teachers’ goals, beliefs, and perceptions of school culture as predictors of

instructional practice. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA. ERIC Document Reproduction Service No. ED352359.

Deal, T. E., & Peterson, K. D. (1999). Shaping school culture: The heart of leadership. San

Francisco: Jossey-Bass. DiPaola, M. F., & Walther-Thomas, C. (2003). Principals and special education: The critical

role of school leaders. Retrieved August 31, 2005 from http://www.personnelcenter.org/pdf/copsse_principals.pdf

Earl, L. M., & Lee, L. E. (1998). Evaluation of the Manitoba School Improvement Program.

Retrieved August 31, 2005 from http://www.peecworks.org/PEEC/PEEC_Research/0068D419-007EA7AB.0/MSIP_evaluation%201998.pdf

Edwards, J. L. (1996, April). Factor and Rasch Analysis of the school Culture Survey. Paper

presented at the Annual Meeting of the American Educational Research Association, New York, NY. ERIC Document Reproduction Service No. ED401290.

Harmon, M. (1996). Differences in the professional norms of middle grades teachers with and

without middle grades certification/endorsement. Unpublished doctoral dissertation from the University of Illinois at Urbana-Champaign.

Roach, A. T., & T. R. Kratochwill. (2004). Evaluating School Climate and School Culture.

Teaching Exceptional Children 32(1), pp. 10-17.


Rugutt, J. K. et al. (2004, June). Dimensions of School Culture: An Elementary and Secondary School Teacher and Administrator Perspective. Paper presented at the Focus on Illinois Education Research Symposium, Naperville, Illinois. Pp. 11-14 of Compendium of Abstracts retrieved June 5, 2005 from http://ierc.siue.edu/documents/compendium_2004.pdf

Schoen, L. (2005). Conceptualizing, describing, and contrasting school cultures: A comparative

case study of school improvement processes. Unpublished doctoral dissertation from Louisiana State University.

Wendel, T. (2000). Creating Equity and Quality: A Literature Review of school effectiveness

and improvement. Society for the Advancement of Excellence in Education Research Series #6. Retrieved August 31, 2005 from http://education.devenir.free.fr/Documents/CreatingEquity.pdf

Other References to Pursue if Future Study Warrants Earl, L., Torrance, N., Sutherland, S., Fullan, M., & Ali, A. S. (2003). Manitoba School

Improvement Program: Final evaluation report. Retrieved August 31, 2005 from http://www.msip.ca/3_evaluation/report.pdf

Fullan, M. (2001). Leading in a culture of change. San Francisco: Jossey–Bass. Fullan, M., & Stiegelbauer, S. (1991). The new meaning of educational change. New York:

Teachers College Press.

Hoy, W. K., Tarter, C. J., & Kottkamp, R. B. (1991). Open schools/healthy schools: Measuring organizational climate. Newbury Park, CA: Sage. Retrieved September 3, 2005 from http://www.coe.ohio-state.edu/whoy/on-line%20books_4.htm#O

Hughes, L. W. (1999). The leader: Artist? architect? commissar? In L. W. Hughes (Ed.),

The principal as leader (2nd ed., pp. 3-24). Upper Saddle River, NJ: Merrill/Prentice Hall.

Levine, D. U., & Lezotte, L. W. (1990). Unusually effective schools: A review and analysis of

research and practice. Madison, WI: National Center for Effective Schools Research and Development. ERIC Document Reproduction Service No. ED330032.

Marzano. Further explore work on school change Sashkin, M. & Walberg, H. L. (1993). Educational leadership and school culture. Berkeley,

CA: McCutchen Publishing Corp. Sergiovanni, T. J. (2000). The lifeworld of leadership: Creating culture, community, and

personal meaning in our schools. San Francisco: Jossey-Bass.


Sergiovanni, T. J. (1995). The principalship: A reflective practice perspective. Boston, MA:

Allyn and Bacon. Smrekar, C. (1996). The impact of school choice and community: In the interest of families and

schools. Albany: State University Press. Stoll, L., & Fink, D. (1996). Changing our schools: Linking school effectiveness and school

improvement. Buckingham & Toronto: Open University Press. Trumbull, D. (1999). The new science teacher. Teachers College Press. (NOTE: Supposedly this work reports on a longitudinal study examining how school

culture affects the ability of pre-service teachers to enact reforms of science teaching once they land their first job).

Record of Methods for Qualitative Process • Sent query to the Greening Schools Research Network • Sent query to Evaltalk • Queried Chris Koliba and Erica Zimmerman • Identified several KEY resources • Google Scholar searches (key words “school culture” “measuring school culture”

“school climate” etc. • ERIC searches • Identified key questions for focus group with PEEC members • Gathered input from Jennifer Jewiss on literature and focus group preparations

Sample query letter Greetings Greening Schools Research Network My colleagues and I have generated some quantitative program evaluation findings that point to the possible presence of a kind of "tipping point" phenomenon in K-12 schools. In short, our data suggests that place-based education programs that work systematically with a whole school community for multiple years may end up impacting the school culture in such a way that exposure of the school as a whole to the intervention becomes a more accurate predictor of desired outcomes than exposure of the individual teachers to the formal elements of the program intervention. It seems that teachers new to a school exhibit many of the desired teaching practices on par with teachers with teachers who have been exposed to the intervention for years. We're thinking that this may be due to those norms of teaching practice somehow having become embedded in the way teaching happens at that school. As we follow this emergent hypothesis and consider developing and/or refining our measures to look at some kind of whole school "dosage" of the program intervention, we are first conducting a focused review of existing literature on the construct of "school culture." What evidence is there that school's have an identifiable "culture?" If so... How might that culture be measured? Under what conditions does it seem likely that the school culture changes? How important is school culture in affecting what goes on for students? for teachers? for administrators or the local community? Can any of you suggest the top ONE or TWO most important studies or meta-analyses of literature that you know of that could get us started in the right direction on investigating this idea of "school culture?" Thank you. ;-) Peace Michael Duffin and Amy Powers Evaluation Co-Directors, Program Evaluation and Educational Research (PEER) Associates


Appendix D: Focus Group Guide for PEEC Program Staff on School Culture • State the purpose of the focus group—gather qualitative data to further explore last year’s

findings and on-going development of quantitative measures. Your experiences can help to target the quantitative explorations more precisely, and also can elucidate the nuances that may be lost in quantitative data, and may not be as evident to participants. Also, opportunity to share ideas among the group.

• Explain taping and transcribing, and “relative confidentiality.” If necessary for context, quotes may be identified with program name but never staff members’ name. Program participant names will never be used in reporting. Opportunity to review the draft will be provided.

• Offer focus group guidelines: • Be mindful to give everyone a turn, but not everyone needs to speak every time • In the interest of time, may need to move on without hearing from everyone • No one should dominate • Opinions and observations only, no right or wrong answers

• Questions for focus group participants: 1. In the PEEC schools you work with (or are aware of), have you seen positive shifts

in the culture or norms of the school? If so, what does this look like? How do you know it’s happening?

Participatory evaluation practice: divide into groups of two, three (ideally) or four. One plays the role of interviewer, one responds, one transcribes. Opportunity to practice different aspects of evaluation process, and a chance for all voices to be captured as data. Use form provided. Write legibly because data will be used.

2. In your experience, which aspects of PEEC programs do you think are most effective

in contributing to changing school culture? (are some dispensable, necessary for other purposes, but not for building school culture?)

After the breakout session, have each person report briefly to the group the most important thing you heard in your group—maybe it was something repeated over and over, maybe it was only nuanced, maybe it was what you personally contributed. 3. In cases where you have seen program concepts really take hold, in the sense of

shifting school culture, did you observe conditions that were present in the school that you felt were necessary for this transformation? (for instance, you’ve never seen a school shift that didn’t have a strong PTA in place; or a school garden program, or a rural setting)


4. Can you think of conditions within a school that you’ve observed to be barriers to having this cultural shift take place? (for instance, school is located in a rural area; or tragic events have caused widespread depression; or school budget has not passed in 20 years)

This line of inquiry assumes that a program’s change is targeted at a school level. Since FFEC works with individual teachers and teacher teams, whole school change is not officially part of FFEC’s logic. However, given that ultimately, their aim is to create sustainable communities, it does seem to be implicit in their logic. At a very core level the logic behind all of the PEEC programs is that you invest in a few people and change begins to emerge at broader levels such as school and community. This study could be a springboard for the next layer, examining a community-level dose. 5. What are the implications of school level dose for professional development model? 6. How can PD programs target their investments to make the most lasting change? 7. An extension of FFEC did work with one participants’ whole school following her

participation. How did school-level change work with the Ray School? Did you see shifts in culture?

a. Are there other examples of FFEC teachers or teacher teams who brought their newfound skills and knowledge back to their schools to effect change?

8. We want to be sure that we’re pursuing a line of inquiry that is useful to you and

your programs. Stepping back and taking in the information we shared last night, and the discussion we had today, what are the advantages and disadvantages of having the evaluation team pursue the school culture idea?


Appendix E: CO-SEED Site Selection Rating Tool

Revised January 26, 2005 CO-SEED

New Site Selection Rating Sheet Site:________________________________

CRITERIA Rating—

1=weak, 5 =strong

Evidence of strong leadership with a clear vision and good administrative support for this initiative. (weight double)

Evidence that implementation of CO-SEED will have tangible and demonstrable positive impacts on the school’s curriculum.

Potential for PBE approach to become self-sustaining in this school and community, as evidenced by a broad, diverse and deep constituency and the rich web of relationships necessary to sustain the program over time.

Desire to shift to place-based education and project-based learning within the whole school, and an understanding of PBE as differentiated from EE, science education, adventure education, etc.

Some place-based education projects currently underway in the school or community for us to build on – experience with EE or community-based education.

Density of opportunities for new successful school / community projects. Potential for high impact in terms of numbers of students, teachers, community members reached directly or indirectly through outreach.

Demonstrated readiness for change, willingness to remove obstacles and enthusiasm for this initiative, generally.

Need for our assistance in the school and community – inability to pull off a transition to place-based education without us.

Currently engaged in strengthening school-community relationships. Commitment to including CO-SEED in faculty development opportunities. (weight double)

Commitment to carrying out a Vision-to-Action Forum. Ability to gather a diverse SEED Team that will remain committed over time. Demonstrated readiness and commitment for receiving this program in the school on the part of the faculty including an active cohort of teachers ready and willing to start.

Few competing initiatives, though possibly some complementary initiatives. Promise of our work with this site to help CO-SEED evolve as a program in ways that are important to us at the time.

Total (Which serves as a starting point for discussion, not an ending point!)


Appendix F: PEEC Long Term Research Agenda

Potential Long Term Directions for the Cross-Program Research and Evaluation Agenda

of the Place-based Education Evaluation Collaborative (PEEC) Submitted by PEER Associates, October 15, 2005, version 4a

Organization PEEC formed in 2002 to increase their collective program evaluation effectiveness. In addition to conducting evaluations of individual PEEC programs, each year PEEC takes on cross-program investigations which seek to clarify more transferable (perhaps eventually generalizable) characteristics of place-based education. For more details, see the PEEC Overview Paper at http://www.peecworks.org/PEEC/PEEC_Concept_Full_12-11-03.pdf

Past Findings The 2002-03 PEEC cross-program agenda used analysis of interview data to identify changes in teacher practice common across PEEC programs. The 2003-04 agenda successfully piloted a quantitative dose-response measurement strategy. This strategy employed statistical analysis of teacher survey data to explore relationships between exposure to PEEC programs and targeted program outcomes such as desirable teacher practices, student performance, and impact on the local community. Many positive correlations were found. Further, data suggested that place-based methods may diffuse gradually into a school culture until they reach a critical threshold—or “tipping point”—when they achieve the status of a norm or widely supported standard. In 2004-05, confidence in the tipping point hypothesis grew as the interpretation was critiqued by program participants during evaluation interviews. Also, review of research literature on school culture was largely consistent with PEEC findings. Full reports are available at http://www.peecworks.org.

Current Work The PEEC 2005-06 cross-program agenda is taking incremental steps toward evaluation of the “tipping point” hypothesis that emerged from our 2003-04 findings and was supported by our 2004-05 findings. The plan summarized in Table 1 on the following page shows what might be possible should resources and program priorities allow.

Notes 1) The PEEC cross-program findings are more likely to be transferable/ generalizable to broader contexts than findings

from individual PEEC programs because they represent a bigger and more diverse sample of program activities/contexts and participant responses. Thus, the PEEC cross-program research and evaluation agenda requires a commitment to continue supporting the ongoing evaluation efforts of individual PEEC programs. Investment in evaluating individual PEEC programs IS investing in the cross-program agenda.

2) The time and dollar figures provided in Table 1 should not be considered fixed quantities. Rather they are estimates of a middle of the road approach. More investment in any of the items would almost certainly return more useful, thorough, and defensible results. Conversely, incremental progress could likely be made with less investment than noted, though expectations should be ratcheted down accordingly.

3) It is important to note that the direction and substance of this agenda is not set in stone. It can and should continuously evolve to accommodate new findings, opportunities, and ideas that emerge from PEEC’s work together.
