Race to the Top College and Career Readiness Initiatives ... Web viewRTTT C&CR Evaluation Annual Report, September 2012. ... gender and race/ethnicity) ... Respondents’ administrative

Massachusetts Race to the Top College and Career Readiness InitiativesEvaluation Annual Report, September 2012

Prepared for the Massachusetts Department of Elementary and Secondary Education

RTTT C&CR Evaluation Annual Report, September 2012

Acknowledgements

The UMass Donahue Institute extends its sincere appreciation to the many people who supported and collaborated with us on this evaluation. In particular, we want to thank personnel from the Massachusetts Department of Elementary and Secondary Education and from the schools, districts, partners, and vendors associated with the Massachusetts Pre-AP, STEM Early College High School, and MassCore initiatives.

Massachusetts Race to the Top College and Career Readiness InitiativesEvaluation Annual Report, September 2012

Project Staff

Eliot Levine, Senior Research Manager, Project ManagerJeremiah Johnson, Research ManagerPatricia Lee, Research AnalystSteven Ellis, Director, Research and Evaluation

Report Information

This report was prepared by the UMass Donahue Institute, the project evaluator, under contract with the Massachusetts Department of Elementary and Secondary Education.

About the Donahue Institute

The University of Massachusetts Donahue Institute is the public service, outreach, and economic development unit of the University of Massachusetts President’s Office. Established in 1971, the Institute strives to connect the Commonwealth with the resources of the University through services that combine theory and innovation with public and private sector applications.

UMDI’s Applied Research and Program Evaluation group specializes in applied social science research, including program evaluation, survey research, policy research, and needs assessment. The group has designed and implemented research and evaluation projects for diverse programs and clients in the areas of education, human services, economic development, and organizational development.

University of Massachusetts Donahue Institute 413-587-2400 (phone)Applied Research and Program Evaluation Group 413-587-2410 (fax) 100 Venture Way Suite 5 www.donahue.umassp.eduHadley, MA 01035-9462

UMass Donahue Institute Research and Evaluation Group

2


Table of Contents

Acknowledgements......................................................................................................................................2Executive Summary.....................................................................................................................................4Introduction................................................................................................................................................10Pre-AP Training and Alignment..............................................................................................................12STEM-focused Early College High Schools............................................................................................51MassCore Policy and Implementation.....................................................................................................75Appendices..................................................................................................................................................96

Appendix A1: Pre-AP Field Interview Protocol, Teacher or Administrator..................................97Appendix A2: Pre-AP Teacher and Lead Teacher Survey...........................................................101Appendix A3: Pre-AP Administrator Survey...............................................................................112Appendix A4: Pre-AP Vendor Interview Protocol.......................................................................119Appendix A5: Pre-AP Vertical Team Meeting Observation Protocol..........................................122Appendix A6: Pre-AP Teacher Summer Training Observation Protocol.....................................123Appendix B1: ESE Interview Protocol.........................................................................................124Appendix B2: STEM ECHS JFF Interview Protocol...................................................................126Appendix B3: STEM ECHS Pre-Implementing Schools Interview Protocol..............................128Appendix B4: STEM ECHS Implementing Schools Interview Protocol.....................................131Appendix B5: STEM ECHS IHE Partner Interview Protocol......................................................134Appendix B6: STEM ECHS School Personnel Focus Group Protocol........................................136Appendix B7: STEM ECHS Observation Protocol......................................................................138Appendix C1: MassCore Administrator Interview Protocol........................................................139Appendix C2: MassCore District Administrator Survey..............................................................142


3


Executive Summary

The Massachusetts Department of Elementary and Secondary Education (ESE) was awarded a federal Race to the Top (RTTT) grant for the years 2010-2014. The college and career readiness (C&CR) components of the grant were designed to help students develop "knowledge and skills necessary for success in postsecondary education and economically viable career pathways." The RTTT initiatives are intended to provide students with opportunities to participate in quality, upper-level high school coursework and new approaches to assist them with high school completion and transition to higher education and the workforce. These programs are part of a broader effort, as specified in the Delivery Plan of ESE's College and Career Readiness group, to increase the Massachusetts 5-year high school graduation rate to 88.3% and the number of students who complete the MassCore program of study to 85% statewide.

The UMass Donahue Institute is conducting an evaluation of three of the C&CR components of the Massachusetts RTTT efforts – the Pre-AP Training and Alignment Program, the STEM-focused Early College High Schools (ECHS), and the MassCore Policy and Implementation initiative.For each of these three programs, this executive summary provides a brief program description, followed by evaluation findings for the period ending September 30, 2012 (referred to as Year 2). Strategic considerations are also presented for some programs. The full report that follows the executive summary incorporates findings from all evaluation activities conducted to date, including a synthesis of the three previous quarterly evaluation briefings. A companion document, the Technical Supplement – RTTT C&CR 2012 Annual Evaluation Report, presents tables and figures that provide a more complete summary of baseline graduation and dropout data.

Pre-AP Training and Alignment

The aims of the Pre-AP Training and Alignment program are to increase the number of low-income and minority students prepared to participate and succeed in mathematics, science, and English Advanced Placement courses and credit-bearing college-level coursework; to provide teachers in grades 6-12 with high-quality professional development to assist them in developing curricula, instruction, and assessments that prepare students for AP coursework; and to provide an opportunity for teachers to collaborate in horizontal and vertical teams and to network with other teachers in their region for the purpose of improving curriculum and instruction. Technical assistance for the project is based on the Laying the Foundation (LTF) curriculum and is provided by the Massachusetts Math + Science Initiative (MMSI), a project of Mass Insight Education in partnership with the Commonwealth of Massachusetts.

Districts are participating in Pre-AP training in multiple configurations. The configuration for RTTT Goal 4D is to send the same teachers for four-day trainings in three consecutive summers and to create discipline-specific vertical teams, led by trained lead teachers, that meet quarterly to share and deepen their Pre-AP practices. Districts that are using Goal 4A or non-RTTT funds have created several variations of the basic training configuration, and the percentage of all teachers in a given school and district who are being trained varies widely. Districts also selected different disciplines or combinations of disciplines among the mathematics, science, and ELA trainings offered.

The Year 2 evaluation of the Pre-AP initiative included the following activities and data sources: interviews of teachers and administrators; surveys of teachers, lead teachers, and administrators;


4


interviews of ESE Program Manager and Project Director; interviews of technical assistance vendor; observations of vertical team meetings; observations of Pre-AP training sessions; training registration and attendance database; vertical team meeting attendance database; supplemental school data requests; ESE documents and databases; and, related project documents.

The Pre-AP Training and Alignment initiative is well underway. During summer 2011, MMSI provided four-day summer trainings for 561 teachers from 37 districts (including 19 4D districts), and teachers attended an average of 3.6 days (out of 4). Also during summer 2011, 30 teachers participated in lead teacher training, and 25 administrators (from seven 4D districts) participated in administrator training. For summer 2012, more than 900 teachers registered for training; final registration data were not available for this report. The ESE program manager reported that, after summer 2012, Massachusetts will have met the RTTT performance goal of having trained 1,000 teachers. In addition, the technical assistance and support provided by MMSI and ESE were widely seen as helpful and satisfactory.

Lead teachers and administrators generally reported being satisfied with the Pre-AP training events. Administrators and teachers largely agree that the LTF lessons and assessments are well-aligned with the Common Core standards and are examples of high-quality pedagogical practices. Teachers were asked about the impacts of their participation in Pre-AP training, and 70% or more of 4D teachers agreed that, as a result of the Pre-AP training, they now teach more Pre-AP content, use more Pre-AP pedagogical strategies in the classroom, and have greater awareness of the importance of using Pre-AP strategies.

Substantial challenges to Pre-AP implementation were reported by both administrators and teachers with regard to curricular resources, classroom equipment, planning time, pacing guides, and student academic readiness. Such concerns represented a substantial minority in some cases, and a clear majority in others, particularly in relation to the adequacy of planning time and the academic readiness of students for the Pre-AP curriculum. The number of Pre-AP lessons and assessments that teachers are expected to implement varies widely across districts.

During the 2011-12 school year, 4D district teams reported attendance at an average of 3.5 vertical team meetings, with a range of 2 to 4 meetings. The meetings were attended by an average of 63% of summer 2011 trainees, plus an average of four personnel who hadn’t completed Pre-AP training. Some lead teachers were not satisfied with the agendas and activities provided by MMSI to structure vertical team meetings. Some lead teachers made slight modifications to agendas, and others completely restructured agendas to meet their district’s needs. The MMSI program director reported encouraging lead teachers to make changes as needed to meet their school’s goals.

The LTF program philosophy, strongly shared by ESE, is that every student should be considered a Pre-AP student, but evaluation findings indicate diverging voices and practices on this topic in the field. Most teachers and administrators who were interviewed agreed that every student should be considered a Pre-AP student, and multiple interviewees expressed mild to strong opposition to tracking. In contrast, responses to the teacher survey provided more varied responses. The survey asked teachers if they used more Pre-AP materials and strategies with some classes than with others. About half said yes, and most of those indicated that they used more Pre-AP materials and strategies with their honors or advanced-level students. This group of respondents generally agreed that Pre-AP materials were not suitable for all students, such as those in remedial classes, those who had not mastered skills from a previous course, and some English language learners.

Strategic considerations (explained in greater detail in the full report).


5


Expectations for minimum implementation levels should be clarified or reiterated.

Providing additional materials and suggestions for differentiation of LTF lessons may increase classroom implementation.

Further assessments of MMSI trainer quality, and additional training of some trainers, may be indicated.

The level of implementation of Pre-AP practices should be monitored and interpreted in relation to student impacts.

Pre-AP schools may need to budget for additional equipment and professional development.

Clarification of the intended uses of vertical team meeting time might lead to increased Pre-AP implementation.

Sharing of modified vertical team meeting materials could save lead teacher time and enhance meeting productivity.

STEM-focused Early College High Schools

In its Race to the Top proposal, ESE proposed to open six STEM early college high schools to prepare students for productive STEM careers, to reduce achievement gaps, and to provide an accelerated pathway to postsecondary education for underrepresented students by partnering with colleges and providing opportunities to earn up to two years of college credit while still in high school. Six districts were chosen in a competitive process and are currently receiving RTTT Goal 4E funds for this purpose. Eight additional STEM ECHS sites in the state are being established, most with the support of discretionary RTTT funds. These additional sites will be included in future evaluation activities, but ESE has requested that UMDI wait until the work of those sites has progressed further.

Evaluation activities during Year 2 included interviews with the ESE Program Manager for STEM ECHS, technical assistance vendor, administrators, and IHE partners; a focus group with school personnel from one implementing site; two site visits, and analysis of relevant documents and databases. Baseline graduation and dropout data are presented for STEM ECHS schools and districts.

The six Goal 4E STEM ECHS sites are in various phases of project development and implementation. Two sites enrolled students in STEM ECHS activities during the 2011-12 school year, three sites plan to start during the 2012-13 school year, and the last site’s STEM ECHS work has been incorporated into a broader school redesign plan.

Interviewees from five districts described assembling a planning team as a successful aspect of their planning process. Additional factors that contributed to the success of the planning teams were including representatives from diverse stakeholder groups and consistent collaboration across members. Respondents shared that holding planning meetings on a regular basis was difficult, because most team members had many other substantial responsibilities and commitments. Interviewees from all districts cited support from district leadership as essential to the planning process and establishing strong working relationships with their IHE partner.

Informants reported many challenges that arose during the planning phase, including budget management, instruction and curriculum development, staffing, engaging the community, and working with teachers’


6


unions. Several districts also reported challenges related to the quality, frequency, or clarity of communication across stakeholders.

Marlborough is the only site that began full implementation during Year 2, with 189 students in grades 6 and 9. One Marlborough interviewee said that a primary success has been the staff’s ability to encourage and support all students in their efforts to complete a curriculum of five honors-level courses. All teachers said that the majority of STEM ECHS students were having a positive experience and “moving in the right direction.” The interviewees expressed that the following factors have contributed to their successful implementation to date: strong curriculum development, a collaborative partnership with the IHE, and the district’s commitment to providing teachers with time for professional collaboration.

Many challenges were reported by Marlborough teachers. One high school teacher reported that some colleagues working within the main high school resent the increased resources (e.g., structured planning and collaboration time, professional development) and decreased class sizes enjoyed by the STEM ECHS teachers. A middle school teacher reported that building administrators assigned personnel to the STEM ECHS without giving them the opportunity to opt out of the program. STEM ECHS teachers also reported that they had faced many challenges with technology, primarily related to the one-to-one laptop initiative, which was not fully implemented until five months into the school year. Teachers said that it was difficult and time-consuming to integrate the use of laptops into the curriculum at that point in the year.

All districts reported that ESE had consistently provided effective, professional, and timely assistance. Districts noted that ESE personnel were aware of, and sensitive to, the contexts in which the districts were working. Jobs For the Future, working in consultation with ESE, provided technical assistance to the Goal 4E STEM ECHS sites. The support provided to the sites has individual technical assistance, meetings with all six schools, and “just-in-time” problem solving. The majority of districts said that JFF was viewed as a valued partner that was helping to move their programs forward.

All interviewees (administrators, teachers, IHE partners, technical assistance vendors, and ESE) reported that finding the financial resources to sustain these STEM ECHS sites will be a significant challenge. No districts have established a clear plan for addressing this issue, although several said that they anticipated committing increased levels of attention to sustainability issues over time. One interviewee also suggested that, over time, the state could consider adopting policies that would benefit or expedite the development of STEM ECHSs and other programs providing high school students with early college experiences.

Strategic considerations (Explained in greater detail in the full report).

Initial lessons from the early-implementing STEM ECHS schools could benefit the other STEM ECHS schools and inform technical assistance from JFF and ESE.

Continuity of leadership and succession planning appear important to success of the STEM ECHS schools.

Securing the financial resources to sustain STEM ECHS partnerships is a primary concern of all STEM ECHS schools.

Changes to state and local policies may facilitate STEM ECHS development.

Increased connections with the broader STEM community could benefit the STEM ECHS schools.

MassCore Policy and Implementation


7


The Massachusetts High School Program of Studies (MassCore) recommends a set of courses and other learning opportunities which Massachusetts students should complete before graduating from high school, in order to arrive at college or the workplace well-prepared and without need for remedial coursework. The 155 districts that selected the RTTT college and career readiness goal committed to implementing strategies to increase the percentage of their students who complete the MassCore curriculum. The state’s RTTT goal is to increase the statewide MassCore completion rate from the 2010 baseline of 70% to 85% of graduates by 2014, and the state has created a formula-based goal for each district. Each district was also expected to determine areas in which courses or supports needed to be expanded in order to meet the 2014 targets, and to create and implement a plan to improve the accuracy of their reporting of MassCore completion levels.

Evaluation activities during Year 2 included an interview with several RTTT district administrators, a survey of RTTT district administrators, interviews with the ESE Program Manager for MassCore and ESE Project Director for the RTTT C&CR evaluation, and analysis of ESE documents and databases. Baseline graduation and dropout data are presented for RTTT MassCore districts.

Districts proposed many approaches to moving toward greater implementation of the MassCore program of studies. These included structural changes (e.g., adding courses, shifting to block scheduling), changing policies (e.g., permitting after-school sports to fulfill physical education requirements), collaborating with external vendors (e.g., to introduce online credit recovery courses). Districts also reported many challenges. The most common curricular challenge was the required fourth year of mathematics, but lab sciences, physical education, world languages, and fine arts were also mentioned. Additional challenges included inadequate funds and staffing in some curricular areas, course scheduling, student academic readiness, severe student learning disabilities, and the fact that MassCore was not a required curriculum. Districts suggested numerous supports that they believed could facilitate their efforts at increasing MassCore completion rates.

ESE's intention was for MassCore to shift from a recommended to a de facto curriculum for the state during the first year of RTTT. This is still the state's intention, but it has not yet been realized. As a result, although many districts believe that increasing MassCore completion is a worthy goal, are committing RTTT and other resources toward that end, and believe it is a realistic goal for the majority of their students, districts have no clear accountability to the state for achieving progress and may be prioritizing other initiatives that have greater accountability or a higher profile. In addition, substantial validity limitations of the state’s MassCore completion indicator raise serious concerns about the ability to assess changes in MassCore completion accurately at the state and district levels.

Strategic considerations (Explained in greater detail in the full report).

Most districts see substantial value in raising MassCore completion rates, but limited accountability for reaching state MassCore targets may subordinate MassCore among district priorities.

Meaningful assessment of changes in MassCore completion is challenged by serious validity and reliability limitations of the MassCore element of the SIMS database.

A MassCore exemption process, or the promotion of alternative ways to fulfill certain requirements, may reduce some districts’ concerns about conflicting mandates and priorities.


8


MassCore specifies the quantity of courses that must be completed, but questions remain regarding expectations of course quality.

The most common challenges to increasing MassCore completion rates are: building capacity to provide required courses, services, and facilities; inadequate funding; the gap between student skill levels and MassCore expectations; course scheduling; and MassCore completion not being required for graduation.

Additional supports and policy changes may increase district progress toward MassCore completion goals.

More than a quarter of district administrator survey respondents did not agree that their district is using RTTT funds to take actions that they believe will lead to increased rates of MassCore completion.


9


Introduction

The Massachusetts Department of Elementary and Secondary Education (ESE) was awarded a federal Race to the Top (RTTT) grant for the years 2010-2014. The college and career readiness (C&CR) components of the grant were designed to help students develop "knowledge and skills necessary for success in postsecondary education and economically viable career pathways." The RTTT initiatives are intended to provide students with opportunities to participate in quality, upper-level high school coursework and new approaches to assist them with high school completion and transition to higher education and the workforce. These programs are part of a broader effort, as specified in the Delivery Plan of ESE's College and Career Readiness group, to increase the Massachusetts 5-year high school graduation rate to 88.3% and the number of students who complete the MassCore program of study to 85% statewide.

The UMass Donahue Institute is conducting an evaluation of three of the C&CR components of the Massachusetts RTTT efforts – the Pre-AP Training and Alignment Program, the STEM-focused Early College High Schools (ECHS), and the MassCore Policy and Implementation initiative – which include activities at the school, district, and state levels. The evaluation began during the fourth quarter of the RTTT grant’s first year, which coincided with the first major field activities. Year 2 was therefore the first full year of substantial evaluation activity. For ease of explanation, “Year 2” in this report refers to the 15-month period from 7/1/11 through 9/30/12, which includes the final quarter of Year 1 plus all of Year 2. This report encompasses all RTTT C&CR evaluation activities that occurred during this 15-month period.

It synthesizes the three previous quarterly evaluation briefings (presented in December 2011, March 2012, and June 2012) and adds findings from evaluation activities that occurred since the most recent quarterly briefing. This report also presents tables and figures of baseline graduation and dropout rates for the Pre-AP, STEM ECHS, and MassCore programs. A companion document, Technical Supplement – RTTT C&CR 2012 Evaluation Annual Report, presents tables and figures that provide a more complete summary of these same baseline graduation and dropout data, including both average and individual graduation and dropout rates for RTTT C&CR districts and individual schools.

Future evaluation reports will use these baseline data to describe trends in graduation and dropout rates in targeted schools and districts. The data necessary to disaggregate baseline graduation and dropout rates by subgroup (e.g., gender and race/ethnicity) were not available in time for this report, but will be included in future analyses. Future reports will also describe trends in indicators of student engagement and academic achievement for students in targeted schools and districts.

Evaluation Questions

This annual report contains separate sections for each of the three RTTT C&CR programs being evaluated, including versions of the project-wide evaluation questions presented below tailored to each program’s unique goals and characteristics.1 Each program’s section also contains the description of methods used for the evaluation of that program.

1 Structuring the report in this way requires some repetition, but ESE and UMDI agreed that the repetition increased clarity.


10


Process questions.

P1. In what ways have grantees implemented the program components? What are the major challenges to and facilitators of successful program implementation encountered by grantees? What midcourse corrections and attempts to overcome challenges have been undertaken? What additional steps are necessary?

P2. In what ways has ESE implemented the program components described in their grant application? What are the major challenges to and facilitators of program support and facilitation encountered by ESE? How have challenges been overcome and midcourse corrections undertaken? What additional steps are necessary?

P3. How do key project stakeholders rate and explain the quality, relevance, and effectiveness of major program components and services?

P4. What infrastructure, systems, and processes were put in place to aid program sustainability during and beyond the grant period? What are the greatest challenges and barriers to creating sustainability?

Outcome questions.

O1. What progress is being made toward the two top-priority goals of ESE's CCR Delivery Plan – increasing the 5-year high school graduation rate to 88.3%, and increasing the number of students who complete the MassCore program of study to 85%?

O2. To what extent are students in grantee programs achieving improved outcomes in college and career readiness indicators including attendance, credit accumulation and recovery, measures of academic achievement (e.g., MCAS, PSAT, SAT, and AP), enrollment in AP courses, graduation, MassCore completion, dual enrollment accumulation of college credits, and college enrollment and persistence?

O3. At the school and district levels, do observed changes differ across student characteristics such as gender, race/ethnicity, free/reduced lunch status, ELL status, and special education status? Is there evidence that gaps are narrowing? What strategies are associated with the greatest changes for the most disenfranchised students? Are program services reaching students who are at the greatest risk?

O4. To what extent are observed changes in student outcomes attributable to program activities (including combinations of program activities) versus contextual variables or non-RTTT interventions?

O5. What are the major differences in program features and implementation, as well as contextual variables at the school and district levels, across programs whose levels of improvement differ substantially?

O6. What is the relationship between level of program implementation and achievement of targeted student outcomes?


11


Pre-AP Training and Alignment

Background

The aims of the Pre-AP Training and Alignment program are to increase the number of low-income and minority students prepared to participate and succeed in mathematics, science, and English Advanced Placement courses and credit-bearing college-level coursework; to provide teachers in grades 6-12 with high-quality professional development to assist them in developing curricula, instruction, and assessments that prepare students for AP coursework; and to provide an opportunity for teachers to collaborate in horizontal and vertical teams and to network with other teachers in their region for the purpose of improving curriculum and instruction.

Technical assistance for the Pre-AP project is based on the Laying the Foundation (LTF) curriculum and is provided by the Massachusetts Math + Science Initiative (MMSI), a project of Mass Insight Education in partnership with the Commonwealth of Massachusetts. Cohort 1 districts are those who completed their Pre-AP training before the beginning of the 2011-12 school year. Goal 4D districts are those who paid for their Pre-AP training and technical support using RTTT Goal 4D funds and who have agreed to send the same teachers for four-day trainings in three consecutive summers, to create vertical teams of teachers, to hold quarterly vertical team meetings, and to have “lead teachers” who organize and run the vertical team meetings and attend an additional training day for lead teachers. Goal 4D districts from Cohort 1 are the primary focus of this evaluation. Other Cohort 1 districts are using Goal 4A or non-RTTT funds to enroll teachers (hereafter referred to as “non-4D teachers”) in the Pre-AP program, but those districts are not required to commit to vertical teaming and sending teams for multiple years of training.

The state’s RTTT scope of work indicates that the performance goal for Pre-AP is 1,000 teachers trained at the end of the 2011-12, 2012-13, and 2013-14 school years. UMDI has received different explanations over time of what the number ‘1,000’ refers to; our understanding at present is that it refers to the total number of summer training participants, whether 4D or other, and whether participating for multiple years or just one. So, for example, the goal could be met by 200 teachers who each attend for three summers, plus 400 teachers who only attend for a single summer. The evaluation questions specific to the Pre-AP evaluation are listed below.

Process Evaluation Questions

P1. In what ways have grantees implemented Pre-AP program components? What are the major challenges to and facilitators of successful program implementation encountered by grantees? What midcourse corrections and attempts to overcome challenges have been undertaken? What additional steps are necessary?

P2. In what ways has ESE implemented the Pre-AP program components described in their RTTT grant application? What are the major challenges to and facilitators of program support and facilitation encountered by ESE? How have challenges been overcome and midcourse corrections undertaken? What additional steps are necessary?

P3. How do Pre-AP program teachers and administrators rate and explain the quality, relevance, and effectiveness of major program components and services?


12


P4. What infrastructure, systems, and processes were put in place to aid Pre-AP program sustainability during and beyond the grant period? What are the greatest challenges and barriers to creating sustainability?

Outcome Evaluation Questions

O1. To what extent are students served by the Pre-AP program achieving improved outcomes in measures of academic achievement (e.g., MCAS, PSAT, SAT, and AP), enrollment in AP courses, graduation, and MassCore completion?

O2. At the school and district levels, do observed changes in students served by the Pre-AP program differ across student characteristics such as gender, race/ethnicity, free/reduced lunch status, ELL status, and special education status? Is there evidence that gaps are narrowing? Are program services reaching students who are at the greatest risk?

O3. To what extent are observed changes in student outcomes attributable to Pre-AP program activities versus other RTTT program activities and/or measurable contextual variables?

O4. What are the major differences in Pre-AP program implementation and contextual variables across schools and districts whose levels of improvement on student outcomes differ substantially?


Methods

The Year 2 evaluation of the Pre-AP initiative included the following activities and data sources:

Interviews of teachers and administrators. Three teachers and two administrators from 4D districts and who had participated in Pre-AP training in summer 2011 were interviewed during Year 2, either by phone or in-person (see interview protocol in Appendix A1 ). The interviews were also used to inform development of the teacher and administrator surveys.

Surveys of teachers, lead teachers, and administrators. An online survey was sent in February, 2012 to all Pre-AP teachers and lead teachers who had completed at least one day of training during the summer of 2011 (see protocol in Appendix A2). An online survey was sent in May, 2012 to administrators of all Goal 4D, Cohort 1 districts (see protocol in Appendix A3).

Interviews of ESE program manager and project director. Three in-person interviews were conducted during the year, as well as informal email and phone communications throughout the year (see interview protocol in Appendix B1).

Interviews of technical assistance vendor. Two interviews were conducted with MMSI’s director of the Pre-AP program, an in-person interview in November 2011 and a phone interview in August 2012 (see interview protocol in Appendix A4). Additional communications included informal email and phone communications throughout the year, and a group meeting in August 2011with representatives of ESE, MMSI, LTF, and UMDI.

Observations of vertical team meetings. Two vertical team meetings were observed in December 2011 and May 2012, to develop an understanding of this aspect of program implementation (see observation protocol in Appendix A5).


13


Observations of Pre-AP training sessions. Four Pre-AP summer training days were observed to develop an understanding of this aspect of program implementation (see observation protocol in Appendix A6).

Training registration and attendance database. MMSI developed and populated a database that tracked teachers, lead teacher, and administrator registration for and attendance of Pre-AP training. UMDI collaborated with MMSI on developing the database structure, and utilized the database for analyses.

Vertical team meeting attendance database. In collaboration with MMSI and ESE, UMDI developed and populated a database that tracked attendance at vertical team meetings. UMDI created a form for districts to report attendance and sent them monthly reminders to submit data updates as additional vertical team meetings were conducted.

Supplemental school data requests. MMSI is responsible for obtaining student PSAT scores from schools participating in the Pre-AP program, to serve as an interim outcome indicator. MMSI has not yet been able to obtain these scores, but UMDI hopes to incorporate them into future reporting.

ESE documents and databases. The Pre-AP quantitative data presented below and in the “Technical Supplement – RTTT C&CR 2012 Evaluation Annual Report” were developed using ESE’s 4-year and 5-year graduation cohort lists as well as SIMS data from October and June of 2008-11. Findings were also informed by graduation and annual dropout rate information reported by ESE.

Related project documents. Project documents including the Massachusetts RTTT proposal, LTF curriculum materials, and MMSI quarterly reports were reviewed.

Findings

Findings are presented in relation to the process and outcome evaluation questions. Pre-AP questions P1 and P3 are addressed in the sections on district training configurations, training activities, implementation, and factors facilitating and obstructing implementation. Questions P2 and P4 are primarily addressed in the sections on training activities, support, and sustainability. The outcome questions are addressed with the presentation of baseline graduation and dropout rates for Pre-AP Cohort 1 4D districts and a discussion of how outcome data for subsequent project years will be utilized.

Survey Response Rates

The Pre-AP administrator survey was sent to administrators of all Goal 4D, Cohort 1 districts (N=25) in May 2012, and had a 68% response rate (N=17). Respondents’ administrative roles included curriculum coordinator (59%, N=12), program director (18%, N=3), assistant superintendent (6%, N=1), and assistant principal (6%, N=1). Most respondents were district-level administrators (88%, N=15), and the others were affiliated with specific middle or high schools. Some questions on this survey were parallel to questions on the Pre-AP teacher and lead teacher survey, and comparisons between administrator and teacher responses are noted throughout the findings section.

The teacher survey was sent to 347 4D teachers and had a 29% response rate (N=101); it was also sent to 198 non-4D teachers and had a 19% response rate (N=38). The survey included a section for lead teachers, and 21 of the 4D respondents and 10 of the non-4D respondents indicated that they were lead


14


teachers. While a sample of 139 teachers is substantial, these low response rates raise the possibility that the sample of respondents was not representative of the larger population of Pre-AP trained teachers.

Teacher survey respondents were asked their number of years as a certified teacher, and the average was 12 years (range 1 to 40). The academic discipline of respondents’ primary teaching assignment was mathematics (35%), English language arts (35%), biology (8%), general science (6%), chemistry (5%), physics (3%), social studies (2%), and one percent each computer science, life science, ESL, environmental science, elementary, multiple disciplines, and administration. (This totals slightly more than 100% because some teachers responded in multiple categories.) The majority of teachers reported teaching at multiple grade levels, as shown in the table below, with the largest group of teachers being in the high school grades.

Teacher Survey: The Academic Level of Your Primary Teaching Assignment

Survey Item N (%)

5th Grade 5 4

6th Grade 15 11

7th Grade 25 18

8th Grade 27 19

9th Grade 39 28

10th Grade 42 30

11th Grade 39 28

12th Grade 38 27

Special Education 8 6

Classroom aide 1 1

ESL Specialist 2 1

Other 5 4

The responses of 4D and non-4D teachers are reported separately. The text will focus on 4D responses but will indicate when non-4D responses differ by more than 15 percentage points from the reported 4D responses.

District Participation Configurations


15


Districts are participating in Pre-AP training in multiple configurations. The configuration for RTTT Goal 4D is to send the same teachers for four-day trainings in three consecutive summers; to create vertical teams across middle and high school grades of mathematics, science, or English language arts teachers; to hold quarterly, two-hour vertical team meetings to deepen their pre-AP practices; and to have “lead teachers” who organize and run the vertical team meetings and attend a training day for lead teachers.

Eighty-two districts initially designated their intention to be Goal 4D districts, but as of the FY12-FY14 proposals, 47 districts remain in that category. Currently known reasons for attrition include changes in district leadership, difficulty accommodating the program’s structural requirements, and realizing that the district couldn’t afford the training costs. At least one district substantially reduced the number of teachers they intended to train.

Several variations of the basic configuration have occurred which will have implications for which teachers and students are included in analyses and as which cohorts. For example, one large district completed half of its training during summer 2011 and initiated vertical team meetings in December 2011 but did not finish Year 1 training until January 2012. Although they are likely to have begun substantial change at the same time as Cohort 1 districts, ESE requested that they be included in Cohort 2 because of when they completed their training.

The percentage of all teachers in a given school and district who are being trained varies widely. Districts also selected different disciplines or combinations of disciplines among the mathematics, science, and ELA trainings offered. In districts that trained only a portion of the teachers in a given discipline, those who received training may have attributes that are not representative of their discipline as a whole. For example, one administrator said, “We started with people who we knew who would dig into this. Picking the first group carefully is important. It is incredibly powerful to have teachers implement it in their classroom, and share their experiences with other teachers.” In districts where participating in Pre-AP training was voluntary, those who chose to participate had to be willing to invest four days of their summer. In some districts, collective bargaining agreements dictated that teachers could volunteer to participate in the summer training but couldn’t be required to do so.

Training Activities

During summer 2011, MMSI provided four-day summer trainings for 561 teachers from 37 districts (including 19 4D districts), and teachers attended an average of 3.6 days (out of 4). Also during summer 2011, 30 teachers participated in lead teacher training, and 25 administrators (from seven 4D districts) participated in administrator training. For summer 2012, more than 900 teachers registered for training; final registration data were not available for this report. The ESE program manager reported that, after summer 2012, Massachusetts will have met the performance goal of having trained 1,000 teachers.

Teacher training. Results from the teacher survey indicated that prior to the summer 2011 Pre-AP training, just 4% of 4D teachers had already completed one or more days of the MMSI Pre-AP program, and just 5% had participated in MMSI’s AP (Core) program. In contrast, a much higher percentage of non-4D teachers had participated in the MMSI Pre-AP (21%) and AP (24%) training.

As shown in the table below, 73% of teachers reported participating in all four days of the MMSI Pre-AP training. The 14% who had participated for more than four days of training were presumably among those who had reported participating in previous years. Among the non-4D group, 33% had completed more


16


than four days, and 23% had completed exactly eight days, suggesting that this was their second full year of Pre-AP training.

Teacher Survey: How many days of MMSI Pre-AP training have you attended?

# of Days4D Non-4D

% N % N

0 1 1 4 1

1 4 4 0 0

2 2 2 0 0

3 6 5 13 4

4 73 68 50 15

5 10 9 10 3

6 2 2 0 0

7 1 1 0 0

8 1 1 23 7

Almost all 4D teachers (91%) reported that they intended to complete all three years (12 total days) of Pre-AP training, which is the commitment districts make when using 4D funds. A somewhat smaller group of non-4D teachers (82%) made that commitment. The table below summarizes these findings.

UMDI’s observations of two summer training classrooms were striking, because the two classrooms were so different. One science classroom was vibrant, with lively and engaged teachers working on an activity related to distance, time, and motion sensors. A very affirming trainer repeatedly used phrases such as “Love it!” and other teachers followed suit with “That’s cool!” and so on. There was much back and forth about typical problems, such as what to do in the absence of needed technology, or the best ways to describe certain scientific terms to enhance student understanding. The teacher prompted helpful reflection on questions such as “How have the instructional strategies modeled by the trainer enhanced the learning experience?” and offered a variety of ways that teachers could access support from her, LTF, and each other after the training had ended.

The English classroom observed, in contrast, was much less lively, engaged, and reflective. It was also much more facilitator-centered, with far less interplay among teachers. Teachers created poems and other pieces of writing, and after a given poem was read out loud the facilitator in effect gave her own brief assessment of the piece’s qualities without seeking input from others in the room. There was less talk about what happens in actual classrooms and the unique challenges of the participants, and there was little reflection on how the training activities connected back to the LTF model.

In short, while both facilitators were leading teachers through LTF activities, one facilitator seemed to be generating more enthusiasm, creating more of the type of classroom that LTF promotes, and providing


17


high-level reflection and opportunities for mutual support. These findings suggest the importance of how facilitators are selected, trained, evaluated, and given feedback, as well as whether they have the opportunity to observe effective facilitators, either live or recorded.

Changes to MMSI’s teacher training. Teacher survey respondents were asked to suggest changes to the four-day MMSI training they attended, and 38 responded. Thirteen wanted fewer lectures and more group work time to discuss and develop lesson plans with other teachers. Six said that their trainer did not exhibit mastery of the subject area, with one saying, “Having an instructor read a manual aloud is not helpful, and completing the exercises as if we are students is a waste of time. The more teachers talk with one another about lessons, units, philosophy, methods, the more valuable the time becomes.”

Five respondents said that they would prefer training to be spread out over the school year, rather than a four-day intensive training, because being presented with so much information at once was overwhelming. These respondents also said that they would like to have time to digest the information and materials at one training session, to practice them, and then to return to follow-up trainings in order to share ideas. Four respondents stated that the training could be conducted in fewer than four days, considering the slow pace and the amount of repetition, and four requested more training in using Pre-AP assessments.

A total of five respondents commented on the activities and exercises conducted during the training. Three said that actually conducting the activities during training was not an efficient use of time, and one said “The experiments presented, except one, were very common. A welcome change would be new experiments or new twists in the analysis of traditional experiments.”

Three respondents said that the materials presented were too difficult for their low-skilled or middle school students. One said, “Some material was much too advanced and lost my attention, as it was irrelevant to my teaching.”

One teacher suggested that, “instead of sending home giant books with all of the middle and high school lessons combined, perhaps create grade-level guides/manuals for the teachers to use with the appropriate lessons included.” Another suggested greater differentiation of trainings for more and less experienced teachers.

With regard to the Pre-AP math training, the lead teacher would have liked more opportunities to interact with teachers from other districts regarding their classroom procedures and curricular activities. He recommended more “pair-share” types of activities, beyond the problems that LTF/MMSI provided for the training session. For the lead teacher training, he felt that too much time was spent promoting LTF/MMSI, and not enough on developing vertical team activities.

Lead teacher training. When asked about lead teacher training, as shown in the table below, 73% of lead teacher survey respondents agreed that the summer lead teacher training had been helpful in supporting their work as a lead teacher, 64% agreed that the summer lead teacher training was engaging, and 82% agreed that the lead teacher training was well-executed.

Teacher Survey: Summer Lead Teacher Training


18


Survey Item NStrongly Agree

(%)

Agree

(%)

Neutral

(%)Disagree

(%)

Strongly Disagree

(%)

The summer Lead Teacher training session was helpful in supporting what I need to do as a Lead Teacher.

11 27 46 9 18 0

The summer Lead Teacher training session was engaging. 11 27 37 27 9 0

Overall, the Lead Teacher Training session in September was well executed.

11 36 46 9 9 0

Note. Response options also included “Not Applicable”. The percentage of respondents who selected those choices are listed below, but those respondents were not used to calculate the percentages reported in the table. Q1. 4D: 42%, Non-4D: 0%, Q2. 4D: 42%, Non-4D: 0% , Q3. 4D: 39%, Non-4D: 0%.

Changes or improvements to lead teacher training sessions. Respondents were asked to comment on what changes or improvements, if any, they would make to the lead teacher training sessions. The five lead teachers who responded suggested focusing on collaboration to plan vertical team meetings, having one additional day of lead teacher training in September, having all teachers bring laptops to meetings, providing new information during lead teacher training compared to presenting the same information from the first training, and clarifying goals of the meetings.

One lead teacher who was interviewed felt that the lead teacher training was less a practical session than a pep talk about Pre-AP philosophy. She said, “At the lead teacher training it was kind of idealistic, like very philosophical. Like you are going to get them all on board with LTF and ‘this works for kids’ and the numbers are proven statistically, but then the rest was kind of dropped off into our hands as far as what to do [at the vertical team meetings].” This topic is discussed further in the section on vertical teams.

Administrator training. Administrators appreciated the opportunity to observe teachers engaged in LTF activities during the administrator training. One administrator noted that observations of LTF activities “provide a great model of what administrators should be seeing when they walk through schools.” He added, “I would suggest that more time be spent on observing and less time on nuts and bolts. I had done some reading and knew about the approach, so I did not need that information. … I wish we had spent more time in classes. They mentioned an evaluation instrument, but we never actually got to see or use that tool during the training. I wish we had done that. To have practice with that type of tool would have been beneficial.” Some administrators are constrained in using tools such as the UTOP protocol to assess teacher practices related to LTF implementation, because instruments that will be used for performance evaluation must be collectively bargained. However, such protocols may also be used to generate discussion rather than to influence evaluation. Only five administrator survey respondents (29%) reported having attended MMSI’s 2011 Pre-AP summer administrator training, but most responded positively to the experience, as shown in the table below.

Administrator Survey: Pre-AP Training


19


Survey Item NStrongly

Agree(%)

Agree(%)

Neutral(%)

Disagree(%)

Strongly Disagree

(%)

As a result of participating in the training, my awareness of the importance of using Pre-AP strategies has increased.

5 20 60 20 0 0

What I learned in the training has helped me promote the Pre-AP program with district-level personnel.

5 20 60 20 0 0

What I learned in the training has been helpful with my district’s vertical curriculum alignment efforts.

5 20 60 20 0 0

The training session was engaging. 5 20 80 0 0 0The training session was well executed. 5 20 80 0 0 0Note. Response options also included “Don’t Know,” but no respondents chose this option.

Administrator training recommendations. Administrators were asked to make suggestions to improve the MMSI summer trainings. One commented that the long drive to the training location deterred some teachers from attending, and also that teachers were more available to attend trainings at the end of the school year, rather than later in the summer. One recommended that MMSI provide more vertical team materials and resources. Finally, one administrator recommended that MMSI provide more help for developing an implementation action plan and more models for integrating strategies.

Implementation

Tracking. The LTF program philosophy, strongly shared by ESE, is that every student should be considered a Pre-AP student. Personnel from both ESE and MMSI have expressed concerns that some middle schools are predicting that certain students are not bound for college and therefore tracking them away from classes that contain Pre-AP content and pedagogy.

Most teachers and administrators who were interviewed agreed that every student should be considered a Pre-AP student, and more than one expressed mild to strong opposition to tracking. Some teachers also expressed support for anti-tracking practices such as including special education teachers in summer training and vertical team meetings, as well as training all teachers in a given discipline so no student is taught by a teacher who lacks Pre-AP training.

Responses to the teacher survey provided a different perspective on the issue of tracking, and on how Pre-AP materials are used more generally. The survey asked teachers if they used more Pre-AP materials and strategies with some classes than with others, and 52% said yes. Of the 54 teachers who offered explanations for their responses, 78% said that they used more Pre-AP materials and strategies with their honors or advanced-level students. They generally agreed that Pre-AP materials were not suitable for all students, such as those in remedial classes, those who had not mastered skills from a previous course, and some English language learners. Several (N=8) also stated that planning, modifying, scaffolding, and


20


implementing Pre-AP lessons required more time than they had available, in part due to keeping up with district pacing guides.

Teachers were asked to respond to the item, “Compared to my honors-level classes, implementing one LTF activity with my non-honors classes requires [more, less, or the same amount of] time.” Twenty-two percent reported that their school only has one level, and 17% that they only teach one level. Of the remaining teachers, 79% reported that implementing an LTF lesson takes longer with their non-honors classes, and just 1% (i.e., one teacher) reported that implementing an LTF lessons takes longer with their honors-level classes.

Teacher Survey: Honor vs. Non-Honors

Survey Item 4D N More(%)

Less(%)

The same amount of

(%)

Compared to my honors-level classes, implementing one LTF activity with my non-honors classes requires [more, less, or the same amount of] time.

Y

N

61

28

79

86

1

0

20

14

Note. Response options also included “My school has only one level” and “My school has both levels, but I only teach one level”. The percentage of respondents who selected those choices are listed below, but those respondents were not used to calculate the percentages reported in the table above. 4D: School only has 1 level – 22%; Only teach 1 level – 17%. Non-4D: School only has 1 level – 9%: Only teach 1 level – 9%.

Of the 48% of teacher survey respondents who said “no” and offered comments, 37% (N=14) stated that they applied Pre-AP lessons and strategies evenly in all of their classes. Some respondents reported simply modifying lessons and strategies to match class levels, while seven respondents said that, since all of their classes were the same level and followed the same pacing guide, they were also provided with the same Pre-AP materials and strategies. Five respondents said that they distributed the materials evenly because their students all deserved to have access to, and benefit from, the same materials as other students; four of these five respondents were science teachers.

In response to these findings, a MMSI representative emphasized that the Pre-AP activities can be broken down into smaller pieces to make them more accessible for students with lower skill levels. She acknowledged that this reduces the amount of content coverage, but she questioned the value of “getting through” everything if the students can’t access the materials. However, to “cover” less content in exchange for greater depth, as promoted by LTF, teachers need the approval of their administrators. Many teachers are pressured to cover more content, and doing so is tied to their performance evaluation, so without explicit approval they could be reluctant to make the changes that LTF implementation requires.

Expectations for implementation. Administrators were asked how many LTF lessons and assessments each Pre-AP trained teacher in the school/district was expected to implement during the 2011-12 school year, and responses varied substantially. They included: at least one per quarter, at least one per term, at least two, two lessons per quarter pairing them with assessments, two lessons and two


21


assessments, and four from their summer training. Two respondents stated that their teachers did not have a specific number that they were expected to implement. One of these administrators stated:

There was no minimum expectation, but it is apparent through vertical team meetings and conversations with staff that our trained Pre-AP teachers were trying out many of the lessons. We still have one final Pre-AP vertical meeting where we will set the expectation for next year based on our learnings from this year.

The other respondent said,

We are not using this as a standard for our teachers. Rather, we are looking for teachers to use elements found in the LTF lessons and integrate them into learning experiences within the classroom. We feel that it is better to help teachers understand what makes a quality lesson, rather than give them a set of tasks to complete with their students.

In response to this quotation, the MMSI Pre-AP program director emphasized that 4D districts should be implementing specific LTF activities and assessments, not just integrating LTF “elements” into their classrooms, and that their letter of agreement specified this. She added, “I really can’t see how [district representatives] can say that they aren’t putting in any activities. I haven’t seen a [letter of agreement] that didn’t have some activities listed. To me, this would be unacceptable. I would strongly urge the administrator to have the activities implemented, and if not I think they are wasting their money.” She indicated that her “dream” would be to have teachers implement five LTF activities and/or assessments each quarter.

All teacher survey respondents reported having implemented substantially more LTF lessons and assessments than required by their districts (6.4 implemented vs. 2.4 required for 4D teachers; 10.0 implemented vs. 1.1 required for non-4D teachers). This difference – 10.0 vs. 1.1 – was particularly notable for non-4D teachers. Last, teachers reported that, on average, implementing one LTF lesson required 2.4 hours (for 4D) and 1.9 hours (for non-4D).

At present, it appears that the pace of change will be gradual. A strategic consideration is whether the desired level of student impacts can be achieved at this pace, and whether that pace can or should be increased. It appears that teachers are expected to implement only 2-3 LTF activities or assessments per quarter, and that this is what MMSI training promotes as an expectable level of change for the first year, believing that it will gradually gain momentum. Impact may also come through routes other than implementation of specific LTF activities, such as if teachers’ deeper understanding of Pre-AP pedagogical approaches also infuses aspects of their usual classroom activities.

Resources. Some LTF math and science activities require expensive equipment that some schools lack, particularly at the middle school level. Some teachers require substantial professional development and/or preparation time to be ready to use this equipment with students, and some schools need guidance about which equipment to purchase. Several districts have asked MMSI to advise them about what equipment to buy, and MMSI has been able to provide advice as well as referring them to other consultants. Schools who wish to adopt these activities need to budget for the needed equipment and professional development.

Several teacher survey items, shown in the table below, asked teachers about the adequacy of their resources and supports, as well as the preparation of their students. With regard to their ability to


22


implement Pre-AP lessons and/or assessments, 66% agreed that they had adequate curricular resources, 53% that they had adequate classroom equipment and supplies, and 58% that they had long enough class periods. Just 22% reported that they had adequate planning time and 21% that they had sufficient time to share Pre-AP approaches with same-discipline colleagues. Seventy-one percent agreed that they had full administrative support to integrate Pre-AP lessons into their teaching, and 26% agreed that performing Pre-AP activities prevented them from adhering to district pacing guides. Teachers clearly felt strongly that their students were not adequately prepared to participate in Pre-AP lessons and/or assessments targeted to their grade level, as just 30% agreed with this statement.


23


Teacher Survey: Pre-AP Lessons and Assessments

Survey Item 4D NStrongly

Agree(%)

Agree(%)

Neutral(%)

Disagree(%)

Strongly Disagree

(%)

I have access to adequate curricular resources to implement Pre-AP lessons and/or assessments.

Y

N

99

38

16

13

50

53

18

18

12

16

4

0

I have adequate classroom equipment and supplies to implement Pre-AP lessons and/or assessments.

Y

N

99

38

8

8

45

37

21

21

18

29

8

5

I have adequate planning time to prepare new Pre-AP lessons and/or assessments.

Y

N

97

37

2

3

20

22

31

19

36

48

11

8

My school provides adequate planning time for same-discipline teachers to share their Pre-AP approaches with each other.

Y

N

98

38

2

3

19

24

24

8

37

34

18

31

Class periods are long enough to provide adequate time for my students to perform Pre-AP activities.

Y

N

96

37

5

11

53

46

20

5

19

30

3

8

Performing Pre-AP activities with my students prevents me from adhering to my district’s pacing guides.

Y

N

99

38

8

0

18

26

26

26

44

37

4

11

My administrators fully support my integration of Pre-AP lessons and assessments into my teaching.

Y

N

97

38

21

31

50

50

20

16

7

0

2

3

My students have sufficient academic preparation to participate in the Pre-AP lessons and/or assessments that are targeted to their grade level.

Y

N

99

38

5

5

25

24

24

21

36

42

10

8

As shown in the next table, 81% of administrators (compared to 66% of respondents to the teacher survey) agreed that teachers in their schools/districts have adequate curricular resources to implement Pre-AP lessons and/or assessments, and 75% of administrators (compared to 53% of teachers) said that


24


they had adequate classroom equipment to do the same. When asked if teachers in schools/districts have adequate time to prepare new Pre-AP lessons and assessments, 44% of administrators agreed, versus just 22% of teachers. Nineteen percent of administrators said that conducting Pre-AP activities prevents teachers from adhering to their district’s pacing guidelines, as compared to 26% of teachers.

Administrator Survey: Pre-AP Resources and Supports


Agree(%)

Agree(%)

Neutral(%)

Disagree(%)

Strongly Disagree

(%)

Teachers in my school/district have adequate curricular resources to implement Pre-AP lessons and/or assessments.

16 31 50 6 13 0

Teachers in my school/district have adequate classroom equipment to implement Pre-AP lessons and/or assessments.

16 19 56 13 6 6

Teachers in my school/district have adequate planning time to prepare new Pre-AP lessons and/or assessments.

16 6 38 31 25 0

Conducting Pre-AP activities prevents teachers from adhering to my district’s pacing guidelines.

16 6 13 13 37 31

I fully support the integration of Pre-AP lessons and assessments into my school/district’s curriculum.

16 75 19 6 0 0

Students in my school/district have sufficient academic preparation to participate in the Pre-AP lessons and/or assessments that are targeted to their grade level.

16 6 44 44 6 0

Scheduling vertical team meetings is a challenge in my district. 16 13 31 25 25 6

Note. Response options also included “Don’t Know,” but no respondents selected that option.

Alignment of assessments and standards. Compared to the survey items above, responses to questions about the LTF curriculum and its relation to the Common Core showed a greater concordance between administrators and teachers, as shown in the following two tables. Eighty-two percent of teachers (compared to 80% of administrators) said that the LTF curriculum is well-aligned with the Common Core standards, 75% of teachers (compared to 72% of administrators) said that the connections between the LTF curriculum and the Common Core are well specified, and 87% of teachers (versus 81% of administrators) said that the LTF lessons and assessments are examples of high-quality pedagogical practices.


25


Teacher Survey: LTF Lessons and Assessments

Survey Item 4D NStrongly Agree

(%)

Agree

(%)

Neutral

(%)Disagree

(%)

Strongly Disagree

(%)

The LTF lessons and assessments are well-aligned with the Common Core standards for my content area.

Y

N

89

31

22

29

60

58

14

7

3

6

1

0

The connections between the LTF lessons and assessments and the Common Core standards are well specified for my content area.

Y

N

88

31

18

26

57

48

17

13

6

13

2

0

The LTF lessons and assessments are examples of high-quality pedagogical practices in my content area.

Y

N

97

35

29

29

58

57

11

11

2

3

0

0

LTF lessons and assessments provide appropriate supports for differentiation for students across a wide range of skill levels.

Y

N

95

36

46

47

21

22

14

17

6

3

13

11

In order to implement a typical LTF lesson, I need to create supplemental activities to introduce or otherwise scaffold aspects of the LTF lesson.

Y

N

97

37

27

16

55

57

15

19

3

5

0

3

Note. Response options also included “not applicable” and “don’t know”. The percentage of respondents who selected those choices are listed below, but those respondents were not used to calculate the percentages reported in the table. Q1. 4D: 1% NA, 9% DK, Non-4D: 3% NA, 16% DK. Q2. 4D: 1% NA,10% DK, Non-4D: 3% NA, 16% DK. Q3. 4D: 0% NA, 1% DK, Non-4D: 3% NA, 5% DK. Q4. 4D: 0% NA, 2% DK, Non-4D: 0% NA, 5% DK. Q5. 4D: 1% NA, 0% DK, Non-4D: 0% NA, 3% DK.


26


Administrator Survey: LTF Curriculum and the Common Core


Agree(%)

Agree(%)

Neutral(%)

Disagree(%)

Strongly Disagree

(%)

The LTF curriculum is well-aligned with the Common Core standards. 15 40 40 13 7 0

The connections between the LTF curriculum and the Common Core standards are well specified.

15 29 43 21 7 0

The LTF lessons and assessments are examples of high-quality pedagogical practices.

16 62 19 19 0 0

Note. Response options also included “Don’t Know.” The percentage of respondents who selected this choice was: Q1. 6%, Q2. 13%, Q3. 0%. These respondents were not included in the number of respondents and percentages reported in the table.

Changes in teaching practices. Teachers were asked about the impacts of their participation in Pre-AP training, and responses are shown in the table below. In the 4D group, 70% or more of teachers agreed that, as a result of the Pre-AP training, they now teach more Pre-AP content (70%), use more Pre-AP pedagogical strategies in the classroom (74%; 90% for non-4D), and have greater awareness of the importance of using Pre-AP strategies (80%). A smaller but still substantial number agreed that they now use more Pre-AP assessment strategies (47%), that they have changed their teaching philosophy to be more consistent with the Pre-AP program (57%), and that implementing LTF lessons and assessments represents a substantial change to their teaching practice (47%).


27


Teacher Survey: Impacts of MMSI Pre-AP Training


(%)

Agree

(%)

Neutral

(%)Disagree

(%)

Strongly Disagree

(%)

As a result of participating in the MMSI Pre-AP training, I now teach more Pre-AP content.

Y

N

100

38

22

32

48

39

14

24

14

5

2

0

As a result of participating in the MMSI Pre-AP training, I now use more Pre-AP pedagogical strategies in the classroom.

Y

N

99

38

18

32

56

58

15

8

9

2

2

0

As a result of participating in the MMSI Pre-AP training, I now use more Pre-AP assessment strategies.

Y

N

95

37

10

13

37

46

34

30

15

11

4

0

As a result of participating in the MMSI Pre-AP training, my teaching philosophy has changed to be more consistent with that of the MMSI Pre-AP program.

Y

N

98

38

12

18

45

24

31

48

11

5

1

5

As a result of participating in the MMSI Pre-AP training, my awareness of the importance of using Pre-AP strategies has increased.

Y

N

100

36

27

28

53

58

17

8

2

0

1

6

Implementing LTF lessons and/or assessments represents a substantial change to my teaching practice.

Y

N

99

38

11

13

36

40

33

29

16

13

4

5

As shown in the table below, teacher survey respondents reported that collaboration with colleagues to develop Pre-AP lessons and/or assessments was most common with same-department colleagues, whether the colleagues were Pre-AP trained (46% agree or strongly agree) or not (44%). Such collaboration was lower with colleagues in other schools (31%).


28


Teacher Survey: Collaboration With Colleagues

Survey Item 4D NStrongly

Agree(%)

Agree(%)

Neutral(%)

Disagree(%)

Strongly Disagree

(%)

I actively collaborate with Pre-AP trained colleagues in my department to develop Pre-AP lessons and/or assessments.

Y

N

98

38

14

16

32

26

18

18

26

16

10

24

I actively collaborate with colleagues in my department who aren’t pre-AP trained to develop Pre-AP lessons and/or assessments

Y

N

98

38

9

18

35

24

18

21

31

21

7

16

I actively collaborate with colleagues in other schools in my district to develop Pre-AP lessons and/or assessments.

Y

N

98

38

5

0

26

13

23

16

32

42

14

29

The table below reports responses to several teacher survey items regarding Pre-AP implementation during the 2011-12 school year, and some notable findings are highlighted next. First, non-4D respondents, who are not required by the Pre-AP project to participate in vertical team meetings, reported having participated in an average of 1.5 such meetings. More information about vertical team meetings is provided in the next section of this report. Second, non-4D teachers reported accessing LTF online materials, as well as meeting with same-discipline colleagues to jointly develop Pre-AP lessons and assessments, more often than 4D teachers.


29


Teacher Survey: Pre-AP Implementation During the 2011-12 School Year

Survey Item 4D Mean

Number of times participated in vertical team meetings.Y

N

2.3

1.5

Number of times met with other teachers in your discipline to jointly develop Pre-AP lessons and assessments

Y

N

3.2

4.1

Number of times accessed online materials on LTF website

Y

N

5.5

7.2

Number of times participated/reviewed discussions in LTF/MMSI online forum

Y

N

0.8

0.6

Number of times observed another teacher or team of teachers presenting an LTF lesson in their classroom

Y

N

0.5

0.9

How many LTF lessons and assessments has your school or district told you that you must implement during this school year?

Y

N

2.4

1.1

On average, how many hours of class time have you needed to spend in order to implement one LTF lesson?

Y

N

2.4

1.9

Please think about the class (or classes) in which you have implemented the most LTF lessons during the 2011–12 school year. How many LTF lessons have you implemented in that class during the school year?

Y

N

4.8

7.5

For that same class, how many LTF assessments have you implemented this school year?

Y

N

1.6

2.5

Vertical team meetings. One commitment of districts participating in RTTT Goal 4D is to schedule four vertical team meetings per year to be attended by Pre-AP trained teachers and administrators across the middle and high school grades in a common Pre-AP curricular area (i.e.,


30


English, math, and science). Other relevant school personnel are also encouraged to participate in these meetings. The meetings are intended to be led by trained Pre-AP lead teachers with the purpose of advancing implementation of the MMSI/LTF lessons and assessments. LTF provides materials to facilitate these meetings, and MMSI provides technical assistance to the extent permitted by its staffing capacity, and in relation to perceived district need, as well as in response to invitations from districts.

As summarized in the table below, district teams reported attendance for an average of 3.5 vertical team meetings, with a range of 2 to 4 meetings. The meetings were attended by an average of 63% of summer 2011 trainees (range 9% to 100%) as well as an average of four personnel that hadn’t completed Pre-AP training. Attendance data were also examined to identify attendance trends within districts. Two districts showed a notable downward trend in attendance across meetings (Attleboro ELA – 13, 11, 4, 4; Waltham Science – 7, 9, 4, 5), but other districts showed fairly even attendance, or had such low numbers of trainees that substantial percentage changes might reflect changes of only one or two participants.


31


Vertical Team Meeting Attendance by District and Discipline

District Discipline

VT Meetings Reporte

d(N)

Summer 2011

Trainees(N)

Average Attendance of Trainees Per

Meeting(%)

Average Non-Trained Staff

Who Attended Each Meeting

(N)

Attleboro ELA 4 13 62 7

Auburn ELA 4 6 100 1

Boston Math 4 44 23 0

Boston Science 3 22 30 1

Chelsea ELA, Math, Science 2 7 71 13

Chicopee Math 4 62 81 3

Everett ELA 4 11 89 1

Everett Math 4 12 73 1

Fall River ELA 4 25 44 4

Malden ELA 3 26 10 1

Malden Science 4 20 9 7

North Adams ELA, Math, Science 2 1 100 10

Saugus Math 4 7 93 3

Somerset Math 4 7 89 4

Swampscott ELA 3 4 58 4

Swampscott Math 4 4 63 4

Uxbridge Science 4 2 100 3

Waltham ELA 3 14 64 1

Waltham Math 4 13 52 3

Waltham Science 4 14 45 1

W. Springfield ELA 3 9 70 8

W. Springfield Math 2 4 63 13

Average 3.5 14.9 63.1 4.2

Scheduling four two-hour vertical team meetings per year presents districts with multiple challenges, including the following, depending on the district:


32


Attendance is optional unless the meetings are held during contractual time. This can include scheduled professional development days. The stipend was an important motivator for teacher attendance, but still not sufficient for some teachers who preferred not to attend. If schools want to prioritize attendance from all teachers, holding the meetings during contractual time would be important. To accommodate this, at least one district has decided to hold two vertical team meetings on each of two scheduled professional development days, thereby meeting the requirement of four meetings. A strategic consideration is whether such an arrangement achieves or shortchanges the purpose of holding four meetings per year.

Middle school and high school teachers must meet together. For after-school meetings, this can present the challenge of the schools having different dismissal times

District Letters of Agreement, which specify the year’s four vertical team meeting dates and times, were in some cases signed too late for the 2011-12 school year to be integrated into the district schedule, and the first vertical team meetings for most districts were scheduled for November or December. First-year startup issues were part of the problem. An implication for future years is to secure LOAs early and set the first meeting earlier in the school year.

With regard to scheduling of Pre-AP vertical team meetings, four lead teacher survey respondents said that their district does not hold vertical team meetings, perhaps because the survey was conducted before that district’s first vertical team meeting. The remaining lead teachers said that vertical team meetings were held after school (56%), during professional development time (31%), and during teacher planning and/or grade-level or content meetings during the school day (38%). This sums to more than 100% because some respondents indicated multiple times.

Administrators reported participating in an average of 2.9 Pre-AP vertical team meetings (range 1 to 8). They also reported observing a teacher or teachers presenting an LTF lesson in their classroom an average of 2.8 times (range 0 to 15), with four administrators reporting zero observations. Forty-seven percent of administrators agreed that scheduling vertical team meetings was a challenge in their district. (A parallel question was not asked of teachers.)

At one vertical team meeting that was observed, some of the teachers were already Pre-AP trained from past years, but the agendas provided by LTF reportedly assume that teachers in attendance are new to the model. It may be useful to provide differentiation of vertical team meetings under such conditions, particularly as in subsequent years there may increasingly be a mix of new and already-trained teachers.

In one school there was an important divergence of understandings about the binder of materials that LTF provides at lead teacher trainings which suggests activities for the four vertical team meetings.

The administrator was enthusiastic about the binder, felt that the lead teacher’s role was straightforward and relatively undemanding, and believed that the lead teacher was mostly following the provided materials.

The teacher, in contrast, didn’t feel that teachers would find the provided materials engaging, saying “I didn’t want to bore everyone to death after a long day of school.” One activity for the meeting was about writing multiple choice questions, and the lead teacher did substantial advance preparation, finding additional articles on the topic and having teachers review and discuss the articles before rating the quality of different items. She felt under considerable pressure to provide a high-quality meeting because not only were her colleagues all invited, but her curriculum coordinator and assistant superintendent would be present.


33


This suggests that the lead teacher role might be more demanding than some believe, particularly when the role is taken on by a very conscientious teacher. It also suggests the potential value of expanding the materials provided by LTF.

For this particular meeting, which was after school hours, only about 4 out of 20 invited teachers attended. It would not be surprising if a lead teacher became less engaged over time in response to such poor attendance after such concerted efforts. Finding ways to increase attendance seem imperative for that reason, as well as for the more basic reason that low attendance means that potential impacts are lost. Meetings held outside of contractual time may have an extra level of burden to be engaging and useful to maximize attendance.

The MMSI program director has committed to attending many districts’ vertical team meetings, and has forwarded emails to UMDI which indicate that she is collaborating with districts to help them plan their meetings. She confirmed that many districts are not using the materials provided by LTF to guide the vertical team meetings. She also reported that MMSI envisions the vertical team meetings including a greater amount of sharing among colleagues, such as discussing strengths and weaknesses of LTF activities and assessments that they have used, what they plan to try next and why, how they will work together to facilitate the work, and how to improve over time. The MMSI program director also said that vertical team time could be used to learn how to use new technology that is required by LTF activities, such as probes or graphing calculators. She saw this as a shift from vertical team meeting activities that LTF has suggested, such as discussing how to teach text annotation at appropriate levels across several grades. She said,

I actually encourage districts to get rid of those [LTF vertical team meeting agendas] if that wasn’t what they needed to do, and to do what they needed to do instead. It seems to have been helpful advice. The overriding goal of most of the districts right now is alignment to the Common Core in English and in math, and so that is where we are focusing right now… ‘Tell me what your biggest needs are, and we will use Pre-AP to address those needs,’ as opposed to Pre-AP just being another initiative on your plate.

She also said that one good strategy for developing vertical team meeting agendas would be to focus on preparing teachers to implement one or two LTF lessons or assessments:

I would like to see districts…bring in a lesson or two for each of the grade levels [to] present at the meeting, and then do those lessons together and discuss who is going to be responsible for various portions of lesson development and implementation…By doing this through the vertical team meetings, you can effortlessly get eight activities and assessments implemented during the school year…We are talking about lessons that they have already done during summer training, so this would be their second exposure.

Factors Facilitating and Obstructing Implementation

Factors facilitating implementation. The Pre-AP approach was seen by administrators as a good fit for their district's need for greater rigor and higher student performance in mathematics, science, and English language arts. Administrator survey respondents were asked to identify the factors and strategies that were the most helpful in getting the Pre-AP program started, and 7 out of 15 reported that the Pre-AP training was the most helpful. One said, “Having both the middle school and high school teachers participating in the LTF Pre-AP trainings has provided the pedagogy and expectations for


34


teachers to identify and incorporate a greater level of rigor in their classes.” Seven also reported that the support of staff, teachers, and other administrators was helpful, including two who reported that district-level curriculum coordinators have attended all of the district’s vertical team meetings. Six respondents mentioned vertical team meetings, which they said have allowed teachers from different grade levels to exchange ideas and strategies and to foster a greater connection with each other. One administrator said,

The relationships were made stronger; the respect from middle school to high school grew; we broke down boundaries and had a really fun time with this. We never saw this as just a Pre-AP initiative; we saw this as an effort that was in direct correlation with all of the other RTTT initiatives.

Five administrators reported that having coordinators and lead teachers oversee Pre-AP program activities and implementation was helpful, including one who said that this was “crucial” to implementation. Other helpful factors mentioned were financial planning and compensation (N=2), good quality training and resources (N=2), and having worked with MMSI before (N=1). One administrator felt that common planning time is a key resource for helping same-discipline teachers deepen their use of Pre-AP approaches, but that creating that within existing schedules will be a major challenge. He felt that vertical team meetings are a good start but not sufficient for deeply embedding the Pre-AP activities.

Administrator obstacles to implementation. Administrator survey respondents were also asked, “What have been the greatest challenges in getting your Pre-AP program started? Are there changes you have made in response to these challenges?” The most common challenge reported by administrators was time (N=5). Three noted that scheduling vertical team meetings after school is difficult, because middle school and high school end at different times. One said that the district extended professional development time in order for vertical teams to meet, but was still unable to hold the four required vertical team meetings. Two respondents, both from the same district, said that their professional development days had been set aside for category training and NEASC accreditation, so they had little time for vertical team meetings. Finally, one said that time has been a challenge for Pre-AP trained teachers who have attempted to integrate Pre-AP lessons into their curriculum maps.

Three administrators reported additional challenges in relation to vertical team meetings. One said that vertical team meetings have not been very productive for middle school teachers. Two said that lead teachers elicited resistance from their colleagues during vertical team meetings, by instructing teachers on program content rather than presenting the LTF curriculum as a source of reference material from which they could draw. The situation was improved by discussing the issue with lead teachers and receiving guidance from MMSI.

Additional challenges included obtaining staff buy-in (N=2), launching the program when some teachers hadn’t yet been trained (N=2), needing additional funding to purchase nonfiction reading materials and hire additional personnel (N=2), and ensuring that teachers were actually embedding Pre-AP activities into their lessons (N=2). One respondent said that implementation has not been visible, despite close collaboration between lead teachers and department chairs, a challenge which one respondent said should be addressed by encouraging administrators to monitor the classrooms of Pre-AP trained teachers. One administrator reported that it has been difficult to have the vendor understand and integrate their new curriculum demands together with the Common Core. This respondent said:


35


The vendor we must deal with, MMSI, offers PD that is seemingly lesson plans in a can, created by LTF in Texas. Texas is not necessarily the go-to state on the Common Core.2 It has been a bit of a battle to get the vendor to reference their work in terms of the new MA 2011 standards. This is perplexing, given that they are the vendor for the state. It is challenging to support lesson implementation when the lessons appear contrived and are not necessarily on point with the language that transitions us to [the Common Core State Standards]. An internal change is that I have reassigned more of my staff members' time to have oversight and push-back to some of the plans within the vendor’s program.

MMSI noted that some efforts have been made to link LTF materials to the Common Core standards, and to make these connections transparent for teachers using those materials in their classrooms.

Another respondent said that prior to stepping into her current administrator position, the district had already decided to focus only on the English Language Arts strand of the LTF program. She expressed interest in expanding the school’s Pre-AP programs to include science, math, and social studies.3

Teacher obstacles to implementation. The teacher survey asked respondents to describe any other factors that significantly challenged their ability to implement Pre-AP lessons and/or assessments (beyond the ones already asked about in the closed-ended items). Half of the respondents across all three disciplines reported that the Pre-AP content was much too difficult for their low-level students. Seven ELA teachers, five science teachers, and nine math teachers reported that Pre-AP materials were extremely challenging for their students who lacked prior knowledge or still struggled with reading and writing. For the math teachers, a typical sentiment was:

My students' skill level and content knowledge are so low that they are barely able to grasp basic concepts, never mind participate in Pre-AP lessons. As an 8th-grade teacher, I am forced to spend time teaching 5th–7th grade skills that my students have never mastered, such as adding fractions, subtracting decimals, and comparing numbers.

When asked about the difficulty of the LTF lessons and assessments, the MMSI program director said,

We talk about the lessons and we talk about how to scaffold the lessons for the needs of their students. We also talk about scaffolding during training. It is really easy to say that these activities are too hard for students…These lessons are tied to the Common Core standards…[so] we need to get the students to the point they need to be so that they can do these materials. We need to work together to make the teachers comfortable presenting it and introducing [the lessons] to the students. It is certainly not a reason to not do the activities.

Time was the second most common challenge mentioned (N=11). Some teachers also said that they do not have any common planning time with other teachers in the same discipline at their schools, or with teachers at other schools in their districts. Across all three subjects, respondents said that trying to balance Pre-AP lessons and assessments with other competing demands has been difficult, given an already stressful schedule of trying to keep up with pacing guides, implementing district curriculum, distributing tests, and planning lessons. One math teacher said,

2Texas is one of five states that has not adopted the Common Core State Standards Initiative. Reference: In the States. Retrieved June 21, 2012, from http://www.corestandards.org/in-the-states.3 Social studies is not currently an option for MMSI’s pre-AP program in Massachusetts.


36


Need to implement STEM, need to implement Pre-AP, need to implement stronger, broader, and deeper common standards while students still come to class without mastering prior standards, need to implement more summative evaluations, need to pull more time from class to implement increased MCAS testing, and need to cut the homework ‘road to nowhere’. How can we continue to pretend to add curriculum and increase rigor while at the same time increasing testing, cutting total class time and eliminating stress by cutting homework? The LTF material is high quality but extremely time consuming for the content presented and required within a course.

In response to this concern, the MMSI program director acknowledged that time can be a challenge. She said, “We are asking [districts] to look at their curriculum and see that one activity covers multiple standards. If you look at it that way, you are going to free yourself up with time to focus on the activity and consolidate lessons.”

The third most common challenges for respondents were curriculum-related (N=9). They reported that a poorly organized curriculum, a constantly changing curriculum, inconsistent pedagogical approaches used by teachers, and a disconnection between district curriculum and Pre-AP materials significantly challenged their ability to implement Pre-AP lessons and assessments. One respondent said that there was “No explicit bridge or connection between prescribed curriculum and Pre-AP materials. With no flags or alerts tied to the topics identified in pacing guides or standards, finding an appropriate Pre-AP activity becomes a deliberate and tedious act of manual searching – not exactly inspirational for planning.”

Another common challenge was the lack of equipment to support Pre-AP lessons (N=7). Three respondents were science teachers who reported lacking equipment or laboratory spaces. Other teachers noted that they lacked needed calculators, reading materials, or computer technology.

One teacher expressed concern about having conducted a Pre-AP lesson that appeared successful on the surface, because students carried out the intended activities, but that didn’t yield the intended learning. She said, “I have very few students who possess the skills to do even the most basic activities. I used an activity involving rate of change, which I thought was excellent. But after two days they had no idea what they had done, and they did the activity quite well!” A final issue raised by one respondent was contractual problems, saying “The fact that we are working without a contract leads to some reluctance to go above and beyond the time and initiatives that have been agreed to.”

Implementing LTF lessons and assessments requires substantial change for some teachers. While some welcomed the richness of the offerings, others who have well-established routines reported being reluctant to invest the substantial time needed to change. The change is not just in the activities themselves, but in the approach, such as being more student-driven than teacher-driven. Respondents reported that this requires more complex management from teachers, because it’s not just telling students what to do, but rather requires students to be more active, motivated, and generative. Student-driven activities can also require more class periods to complete than a teacher-driven activity that addresses comparable standards. However, a teacher reported that, when implemented well, many students find these types of activities very engaging, and the activities can promote higher-order skills – such as “thinking like a scientist” – that are needed for the AP exams and linked to college and career readiness.

Advice


37


Advice from teachers implementing LTF lessons and assessments. Teacher survey respondents were asked to provide advice for others who are trying to implement the LTF lessons and assessments in their classroom. Out of 61 respondents, 21 advised other teachers not to hesitate to try using LTF materials in the classroom. Some suggested starting small and adding more lessons as their comfort level increases. Others advised teachers to be patient and not to get discouraged. One teacher said, “Just commit and practice as often as you can. We teachers have a tendency to stick to our methods, so really embrace the strategies early in the year and continue to use them as often as possible.”

Respondents (N=21) also said that teachers should modify lessons to fit student levels, and to align lessons with school curriculum. One teacher said, “Be ready for the kids to struggle with it initially, but push them through, guide their attempts, fill in blanks where they don't have the knowledge, and it WILL pay off in the long run. If necessary, scaffold the lessons even further so that kids will have an easier time wrapping their heads around the new material.”

Six respondents advised teachers to plan ahead. Four advised them to complete the lessons and assignments themselves first, before giving them to students. Other advice included using the LTF website (N=3) and working with a partner to discuss lessons (N=3).

Advice from administrators implementing LTF curriculum. Administrator survey respondents were asked to provide advice for others who were trying to implement the LTF curriculum in their schools or district. Eleven respondents provided diverse advice, including to work hard to get teacher buy-in (N=1), to be persistent (N=1), to start small and build trust among the team (N=1), and to get teachers and administrative staff teams trained in implementation and support (N=2). Additional comments included:

Be patient and really take the time to complete the trainings. The trainings are engaging and challenging. The teachers I have worked with walk away feeling more confident themselves about their content and skills.

Seek to collaborate with surrounding districts that are implementing the program. The LTF program is outstanding and truly has impacted our district in a positive way. Collaboration and communication on vertical team meetings and ways in which the program has been implemented in other districts has been very useful.

Make it a priority. Make it content-focused. Make it fun. We had a great time. Our vertical meetings flew by and people often stayed later than expected. Just don't view it (Pre-AP) in isolation. View it as something that can help you reach many goals in your district. I got to know the entire math staff 6-12, we had an opportunity to meet and talk and I was able to listen to their needs and they felt supported.

Those who are trying to implement the curriculum need to make sure that they have the time and capacity to work with the LTF curriculum to make it their own.

Slow and steady growth and implementation. Let word spread. Conduct comparative data between teachers of the same content/grade level to talk about the strategies informally. This will engender buy-in and build cohorts.

Connect to curriculum development. Use professional development strategically to help reach scale. Think about who sits on the cohort team receiving the direct training from LTF and how these folks can be leveraged for support of implementation in schools and at different


38


grade levels. How can these folks be leveraged for the curriculum development work and connecting one to the other? Truly understand the MMSI goals...make them part of your own. Communicate, communicate, communicate.

Support

The collaboration between ESE and MMSI seems to be productive and evolving. It appears that ESE sees MMSI as expending tremendous efforts, and that MMSI has substantial abilities to facilitate work in the field. ESE concerns with MMSI have included:

Insufficient attention to meeting reporting deadlines and tracking required data. (MMSI has attempted to address this concern by hiring additional staff.)

Capacity concerns regarding the substantial increase in the number of teachers to be trained and districts to be supported starting in summer 2012. (MMSI has attempted to address this concern by hiring additional staff, although it was not clear at the time of this report whether that change had been sufficient to address the capacity shortfall.)

Canceling some summer 2011 trainings, asserting that they needed at least 24 teachers in a given subject area to offer a training, but not having stated that in their original agreement with ESE. This left 20-30 teachers unable to receive training.

Providing different training for biology, chemistry, and physics teachers, which makes it difficult to gather enough teachers to hold a training. ESE proposed that LTF might address this challenge by combining the three science trainings into a single training that integrates activities from all three disciplines.

With regard to technical assistance and support, most teacher and administrator survey respondents agreed that the response from MMSI and ESE had been helpful and satisfactory when the school or district had requested support.

Teacher Survey: MMSI Technical Assistance & Support


(%)

Agree

(%)

Neutral

(%)

Disagree

(%)

Strongly Disagree

(%)

When my school or district has requested technical assistance and support from MMSI, the response has been helpful and satisfactory.

Y

N

44

16

20

19

34

44

46

37

0

0

0

0

Note. Response options also included “Not Applicable” and “Don’t Know”. The percentage of respondents who selected those choices are listed below but, those respondents were not used to calculate the percentages reported in the table above. 4D: 0% NA, 52% DK, Non-4D: 0% NA, 29% DK.


39


Administrator Survey: Technical Assistance and Support from MMSI and ESE


Agree(%)

Agree(%)

Neutral(%)

Disagree(%)

Strongly Disagree

(%)

When my school or district has requested technical assistance and support from MMSI, the response has been helpful and satisfactory.

16 77 15 8 0 0

When my school or district has requested technical assistance and support from the Department of Elementary and Secondary Education, the response has been helpful and satisfactory.

8 23 62 15 0 0

Note. Response options also included “Don’t Know.” The percentage of respondents who selected this choice is listed below, but these respondents were not used to calculate the percentages reported in the table. Q1. 19%, Q2. 19%.

MMSI and ESE training and technical assistance. Administrators were asked to identify what training or technical assistance from MMSI or ESE had been most helpful, and 12 administrators provided comments. Half of those responding to this item (N=6) reported that support from MMSI and ESE’s staff and their responsiveness to inquires had been the most helpful. Respondents stated that MMSI quickly responded to calls, attended meetings, and made helpful suggestions regarding the purchase of equipment. Five respondents reported that MMSI provided planning support. These administrators noted that MMSI helped to plan and implement vertical team meetings, prepare for Year 2 cohort training, and develop a Letter of Agreement. One district said, “They assisted in creating the agenda, topics and activities for the teachers, facilitated and mentored the Pre-AP Team Leads, modeled the ideal format for a vertical team meeting, remained flexible based on our needs... there is so much they did to support us every step of the way.” Respondents also reported that digital resources and modules (N=1), follow-up training on Vernier equipment (N=1), and LTF summer trainings (N=1) were helpful.

MMSI reported that some schools want to train extra lead teachers, both to develop leadership within their departments and to provide equity across schools. Under current agreements, this would be expensive for districts and a capacity challenge for MMSI, because MMSI is currently committed to providing 12 hours of support per lead teacher. MMSI said that to accommodate districts that want to train 5-10 lead teachers, MMSI needs to come up with a pricing structure and adequate staffing that allows them to provided needed support.

Additional support from MMSI and ESE. Administrator survey respondents were asked to identify what further training or support from MMSI or ESE would be must helpful, and 13 provided comments. Five asked that MMSI and ESE continue to provide support for “the modules,” the coaching of lead teachers, and the planning and implementing of vertical team meetings. One administrator was unable to attend last year’s administrator training and expressed interest in attending this year. Another administrator said,


40


It would be great if MMSI supported us by letting us make some of our own decisions and followed communication protocol. In the past several months, they have sent out emails to our teachers promising PD and possible stipends (without district permission). This has cost a great deal of energy, as we respond to teachers who have misinformation. They have also seemed to insist that we offer our teachers multiple PD dates when some of the dates are in direct conflict with district PD, yet we have had to open the dates that MMSI wants.

Respondents reported that it would be helpful if MMSI or ESE provided additional materials and resources such as textbooks and technology (N=1), provided modeling and in-district coaching (N=1), followed up on implementation with new lead teachers (N=1), negotiated with ESE about the grant process (N=1), and provided information about funding opportunities beyond the RTTT grant (N=1). One administrator expressed interest in making better use of assessment materials, stating that she/he was less certain of their alignment with the Common Core. Another administrator said, “I would also like to see more about the Speaking and Listening and the Language strands of the CCSS in ELA/Literacy,” and to know more about the “implications of Standards for Mathematical Practice and NGSS – practices and cross-cutting concepts (in service to the content of math and science).”

The teacher survey asked what additional supports from MMSI would be most helpful, and 44 offered comments. Across all three disciplines, 48% (N=21) said that access to more resources and materials would be most helpful. These included reading texts and 9th-grade Pre-AP writing materials for ELA teachers, more interactive lessons for math teachers, and more equipment for science teachers. Two wanted monthly email blasts with new LTF lessons and resource links, and four wanted additional differentiated lessons for their lower-level students. Three teachers requested having LTF materials in Microsoft Word format, so that they could customize them, and two teachers wanted instructional or demonstration videos.

Teacher survey respondents from all three disciplines (N=11) wanted help in learning how to modify their LTF lessons for lower-level students and how to integrate Pre-AP curriculum with their district curriculum. One teacher expressed, “It would be great to actually work with MMSI in fully integrating different strategies into all classes, in targeting specific areas, and in creating Pre-AP lessons to coincide with all curriculum.” Finally, six mentioned wanting more meetings and professional development, including during the school year rather than just during the summer, as well as follow-up meetings with their initial training cohort and other teachers to reflect on Pre-AP lessons they have conducted.

Additional requests included science equipment, graphic calculators, and the layered English text (N=5); training in algebra for middle school students (N=1); another set of LTF manuals (N=1); English lessons that incorporate technology such as Eno boards (N=1); a blog to communicate with fellow trainees (N=1). Many teachers (N=10) also wanted all grade 6-10 teachers to complete Pre-AP training in order to increase AP readiness.

Teacher survey respondents were asked to specify what training or technical assistance from MMSI had been most helpful, and 72 offered comments. Forty-seven percent (N=34) said that access to resources and materials such as lessons, assessments, manuals, and the LTF website had been most helpful. Forty-nine percent (N=35) said that participating in training workshops and professional development activities throughout the school year and during the summer provided them with useful strategies, including opportunities to collaborate with peers and to work through the lessons themselves. Four math teachers cited instruction in using graphing calculators as the most helpful form of assistance.


41


Lead teachers were asked about the support provided by MMSI. As shown in the table below, of those whose districts had held vertical team meetings, 57% agreed that the materials and activities provided by MMSI for structuring a Vertical Team meeting were effective in achieving the goals of the meeting. Of those whose district had held a vertical team meeting, 50% agreed that the support provided by MMSI for structuring and running Vertical Team meetings had helped them develop leadership skills.

Teachers were asked to respond to the item, “When my school or district has requested technical assistance and support from MMSI, the response has been helpful and satisfactory.” Fifty-two percent reported that they didn’t know, which is unsurprising, as teachers are less likely than administrators to be involved in such communications. Of the remaining teachers, 54% agreed, and the rest were neutral; no one disagreed or strongly disagreed.

Teacher Survey: MMSI Support for Vertical Team Meetings

Survey Item N Strongly Agree

(%)

Agree

(%)

Neutral

(%)

Disagree

(%)

Strongly

Disagree

(%)

The materials and activities provided by MMSI for structuring a Vertical Team Meeting are effective in achieving the goals of the meeting.

14 7 50 36 7 0

The support provided by MMSI for structuring and running Vertical Team Meetings has helped me develop my leadership skills.

14 7 43 43 7 0

Note. Response options also included “We have not held a vertical team meeting”. The percentage of respondents who selected those choices are listed below, but those respondents were not used to calculate the percentages reported in the table. Q1. 4D: 5%, Non-4D: 0% Q2. 4D: 26%, Non-4D: 0%.

When asked about the support provided by MMSI, 64% agreed that they were satisfied. When asked if the role of lead teacher required more time than they had available, just 18% agreed.


42


Teacher Survey: MMSI Support Satisfaction and Lead Teacher Time Available

Survey Item NStrongly Agree

(%)

Agree

(%)

Neutral

(%)

Disagree

(%)

Strongly Disagree

(%)

When I have requested support from MMSI, I have been satisfied with the support I received.

14 7 57 36 0 0

The role of the lead teacher requires more time than I have available. 17 6 12 59 23 0

Note. Response options also included “Not Applicable”. The percentage of respondents who selected that choice is listed below, but those respondents were not used to calculate the percentages reported in the table. Q1. 4D: 26%, Non-4D: 50%, Q2. 4D: 11%, Non-4D: 0%.

The MMSI program director expressed concern that districts who are actively requesting assistance may be receiving a disproportionate amount of MMSI’s time, while other districts who may also need support are neither requesting nor receiving it. She said,

What I need to do is create a [process that I use to] reach out… if I haven’t heard from a given district, I need to continually ping them until get some communication with them going. Not everyone needs the same level of support from me…but I can’t assume that if someone hasn’t reached out to me, or hasn’t responded to my offer of help, that they don’t need it.

Sustainability

The ESE program director for Pre-AP reported that there is currently no policy in place to promote the sustainability of Pre-AP programming or practices beyond the RTTT funding period. Data collection in Year 3 will investigate what strategies school or districts are employing to promote sustainability.

Baseline Graduation and Dropout Rates

The following three pages each contain a table paired with a figure that present the following baseline graduation and dropout rate data for high schools in the 28 Cohort 1 districts.4

1. Average 4- and 5-year cohort graduation rates from 2007-08 to 2010-11.

2. Average annual dropout rates for 2007-08 to 2010-11.

3. Average 4- and 5-year cohort dropout rates 2007-08 to 2010-11.

The companion document “Technical Supplement – RTTT C&CR 2012 Evaluation Annual Report” presents tables and figures that provide a more complete summary of these same baseline graduation and dropout data. The Technical Supplement contains tables and figures that present both average and individual graduation and dropout rates for RTTT Goal 4D districts and individual schools. Future

4 Current MMSI registration and attendance records indicate that 37 districts sent one or more teachers to training during the summer of 2011. Of these 37 districts, 28 are currently assigned to Cohort 1. Questions remain regarding the cohort assignments of the remaining 9 districts, which are not included in the analyses presented in this report.


43


evaluation reports will use these baseline data to describe trends in graduation and dropout rates in Pre-AP schools and districts. The data necessary to disaggregate baseline graduation and dropout rates by subgroup (e.g., gender, race/ethnicity) were not available in time for this report, but will be included in future analyses. Future reports will also describe trends in indicators of student performance (e.g., MCAS, attendance).


44


Pre-AP Cohort 1 4D & Non-4D: 4- & 5-Year Graduation Rates

4- & 5-Year Graduation Rates for Pre-AP 4D & Non-4D Districts, 2007-08 to 2010-11

Year

Number of Students 4-Year Graduation Rate

5-Year Graduation Rate

Difference Between 4- and 5-Year Rates

4D Non-4D 4D Non-4D 4D Non-4D 4D Non-4D2007-08 11,353 3,012 67.5% 80.5% 72.3% 83.0% 4.8% 2.5%2008-09 11,283 2,988 68.5% 81.0% 72.5% 84.2% 4.0% 3.2%2009-10 11,056 3,060 69.5% 81.3% 73.5% 84.4% 4.0% 3.1%2010-11 10,432 2,831 71.1% 82.5% NA NA NA NA

4- & 5-Year Graduation Rates for Pre-AP 4D & Non-4D Districts, 2007-08 to 2010-11

2007-08 2008-09 2009-10 2010-1150%

55%

60%

65%

70%

75%

80%

85%

90%

95%

100%

4-Year Rate - Pre-AP 4D5-Year Rate - Pre-AP 4D4-Year Rate - Pre-AP non-4D5-Year Rate - Pre-AP non-4D

School Year

Gra

duat

ion

Rat

e


45


Pre-AP Cohort 1 4D & Non-4D: Annual Dropout Rates

Annual Dropout Rates for Pre-AP 4D & Non-4D Districts, 2007-08 to 2010-11

YearTotal HS Enrollment Number of Dropouts Annual Dropout Rate

4D Non-4D 4D Non-4D 4D Non-4D

2007-08 41,195 10,971 2,692 426 6.5% 3.9%

2008-09 40,289 10,591 2,315 303 5.7% 2.9%

2009-10 39,569 10,398 2,075 316 5.2% 3.0%2010-11 39,443 10,145 2,011 316 5.1% 3.1%

Annual Dropout Rates for Pre-AP 4D & Non-4D Districts, 2007-08 to 2010-11

2007-08 2008-09 2009-10 2010-110%

1%

2%

3%

4%

5%

6%

7%

8%

9%

10%

Pre-AP 4DPre-AP non-4D

School Year

Ann

ual D

ropo

ut R

ate


46


Pre-AP Cohort 1 4D & Non-4D: 4- & 5-Year Cohort Dropout Rates

4- & 5-Year Cohort Dropout Rates for Pre-AP 4D & Non-4D Districts, 2007-08 to 2010-11

Year

Number of Students 4-Year Dropout Rate 5-Year Graduation Rate

Difference Between 4- and 5-Year Rates

4D Non-4D 4D Non-4D 4D Non-4D 4D Non-4D2007-08 11,353 3,012 18.2% 11.1% 18.4% 10.5% 0.2% -0.6%2008-09 11,283 2,988 16.7% 9.4% 17.0% 8.7% 0.3% -0.7%2009-10 11,056 3,060 14.2% 8.2% 15.1% 8.3% 0.9% 0.1%2010-11 10,432 2,831 13.2% 7.7% NA NA NA NA

4- & 5-Year Cohort Dropout Rates for Pre-AP 4D & Non- 4D Districts, 2007-08 to 2010-11

2007-08 2008-09 2009-10 2010-11

-5%

0%

5%

10%

15%

20%

25%

30%

4-Year Rate - Pre-AP 4D5-Year Rate - Pre-AP 4D4-Year Rate - Pre-AP non-4D5-Year Rate - Pre-AP non-4D

School Year

Coh

ort D

ropo

ut R

ate


47


Conclusions

The Pre-AP Training and Alignment initiative is well underway, and districts are participating in Pre-AP training in multiple configurations. During summer 2011, MMSI provided four-day summer trainings for 561 teachers from 37 districts (including 19 4D districts), and teachers attended an average of 3.6 days (out of 4). Also during summer 2011, 30 teachers participated in lead teacher training, and 25 administrators (from seven 4D districts) participated in administrator training. For summer 2012, more than 900 teachers registered for training; final registration data were not available for this report. The ESE program manager reported that, after summer 2012, Massachusetts will have met the performance goal of having trained 1,000 teachers. In addition, the technical assistance and support provided by MMSI and ESE were widely seen as helpful and satisfactory.

Lead teachers and administrators generally reported being satisfied with the Pre-AP training events. Administrators and teachers largely agree that the LTF lessons and assessments are well-aligned with the Common Core standards and are examples of high-quality pedagogical practices. Teachers were asked about the impacts of their participation in Pre-AP training, and 70% or more of 4D teachers agreed that, as a result of the Pre-AP training, they now teach more Pre-AP content, use more Pre-AP pedagogical strategies in the classroom, and have greater awareness of the importance of using Pre-AP strategies.

Substantial challenges to Pre-AP implementation were reported by both administrators and teachers with regard to curricular resources, classroom equipment, planning time, pacing guides, and student academic readiness. Such concerns represented a substantial minority in some cases, and a clear majority in others, particularly in relation to the adequacy of planning time and the academic readiness of students for the Pre-AP curriculum. The number of Pre-AP lessons and assessments that teachers are expected to implement varies widely across districts.

During the 2011-12 school year, 4D district teams reported attendance at an average of 3.5 vertical team meetings, with a range of 2 to 4 meetings. The meetings were attended by an average of 63% of summer 2011 trainees, plus an average of four personnel who hadn’t completed Pre-AP training. Some lead teachers were not satisfied with the agendas and activities provided by MMSI to structure vertical team meetings. Some lead teachers made slight modifications to agendas, and others completely restructured agendas to meet their district’s needs. The MMSI program director reported encouraging lead teachers to make changes as needed to meet their school’s goals.

The LTF program philosophy, strongly shared by ESE, is that every student should be considered a Pre-AP student, but evaluation findings indicate diverging voices and practices on this topic in the field. Most teachers and administrators who were interviewed agreed that every student should be considered a Pre-AP student, and multiple interviewees expressed mild to strong opposition to tracking. In contrast, responses to the teacher survey provided more varied responses. The survey asked teachers if they used more Pre-AP materials and strategies with some classes than with others. About half said yes, and most of those indicated that they used more Pre-AP materials and strategies with their honors or advanced-level students. This group of respondents generally agreed that Pre-AP materials were not suitable for all students, such as those in remedial classes, those who had not mastered skills from a previous course, and some English language learners.


48


Strategic Considerations

Expectations for minimum implementation levels should be clarified or reiterated. While some district administrators reported that their Pre-AP trained teachers were expected to implement several Pre-AP lessons or assessments each quarter, others reported that they do not require their teachers to implement any Pre-AP lessons or assessments. This suggests that participating in the Pre-AP program does not imply participation in a specific set of practices across schools or even classrooms within schools. However, MMSI indicated that 4D districts have clearly committed to implementing Pre-AP lessons and assessments. To the extent that a minimum expectation exists, it should be communicated clearly to program participants.

Providing additional materials and suggestions for differentiation of LTF lessons may increase classroom implementation. Several teachers noted that the time required to modify LTF lessons for differentiation purposes was an obstacle to implementation. Moreover, the MMSI program director affirmed that such differentiation was strongly encouraged, and that appropriate modifications could make LTF lessons more accessible for students with lower skill levels. The more that the LTF lessons and assessments can incorporate materials and suggestions for such differentiation, the more broadly the lessons and assessments may be implemented by teachers.

Further assessments of MMSI trainer quality, and additional training of some trainers, may be indicated. While a formal assessment was not undertaken, UMDI’s informal observations (made by an experienced classroom teacher) suggested that trainers varied widely in terms of the quality of their instruction and the extent to which their instruction modeled LTF’s proposed pedagogy.

The level of implementation of Pre-AP practices should be monitored and interpreted in relation to student impacts. Teachers and districts clearly have many factors in addition to the Pre-AP training that affect their practices and policies, and implementation of Pre-AP lessons and assessments appears so far to represent at most a moderate portion of classroom activities. As the number of Pre-AP trained teachers and their level of experience with Pre-AP practices increases, the pace of change and the potential for student impacts may increase accordingly. Over time it will be important to continue monitoring the extent of implementation, to discuss whether that extent can or should be increased, and to interpret findings regarding student impacts in relation to the extent of implementation.

Pre-AP schools may need to budget for additional equipment and professional development. LTF math and science activities require expensive equipment that some schools lack, particularly at the middle school level. Teachers may also require substantial professional development and/or preparation time to be ready to use this equipment with students, and schools may need guidance about which equipment to purchase. Several districts have asked MMSI to advise them about what equipment to buy, and MMSI has been able to provide advice as well as referrals to other consultants. Schools who wish to adopt Pre-AP activities may need to budget for additional costs.

Clarification of the intended uses of vertical team meeting time might lead to increased Pre-AP implementation. Vertical team meetings were well-launched, but clarifying intended meeting content could be important. Some teachers and administrators have reported that they have substantially modified the LTF curriculum provided for the vertical team meetings in order to better meet their school’s needs, and the MMSI program director has approved of this practice. However, it was unclear from UMDI’s lead teacher interviews and initial observations of vertical team meetings, to what extent districts are using vertical team


49


meeting time to specifically advance Pre-AP implementation versus addressing more general curricular goals. Clarification of expectations for 4D districts might lead to increased Pre-AP implementation.

Sharing of modified vertical team meeting materials could save lead teacher time and enhance meeting productivity. As already noted, Pre-AP lead teachers are modifying the vertical team meeting materials provided by LTF. Collecting and sharing these materials online might provide a resource that reduces the needed time investment and increases the impact of lead teachers’ efforts. Lead teachers could receive reminders of this resource and periodic updates regarding new or timely content.


50


STEM-focused Early College High Schools

Background

In its Race to the Top proposal, ESE proposed to open six STEM early college high schools to prepare students for productive STEM careers, to reduce achievement gaps, and to provide an accelerated pathway to postsecondary education for underrepresented students by partnering with colleges and providing opportunities to earn up to two years of college credit while still in high school. Six districts were chosen in a competitive process and are currently receiving RTTT Goal 4E funds for this purpose. Eight additional STEM ECHS sites are being established with the support of discretionary RTTT funds, and ESE has requested that UMDI postpone evaluation of those sites until their work has progressed further. The evaluation questions specific to the STEM-focused Early College High Schools are listed below.


P1. To what extent have grantees developed and implemented STEM ECHS components? What are the major challenges to and facilitators of successful program implementation encountered by grantees? What midcourse corrections and attempts to overcome challenges have been undertaken? What additional steps are necessary?

P2. To what extent has ESE implemented the STEM ECHS program components described in their RTTT grant application? What are the major challenges to and facilitators of program support and facilitation encountered by ESE? How have challenges been overcome and midcourse corrections undertaken? What additional steps are necessary?

P3. How do STEM ECHS teachers and administrators rate and explain the quality, relevance, and effectiveness of major program components and services?

P4. What infrastructure, systems, and processes were put in place to aid STEM ECHS sustainability during and beyond the grant period? What are the greatest challenges and barriers to creating sustainability?


O1. To what extent are STEM ECHS students achieving improved outcomes in college and career readiness indicators including attendance, credit accumulation and recovery, measures of academic achievement (e.g., MCAS, PARCC, PSAT, SAT, and AP), enrollment in AP courses, graduation, MassCore completion, accumulation of college credits, and college enrollment and persistence (to the extent known during the grant period).

O2. Do observed changes in STEM ECHS students differ across student characteristics such as gender, race/ethnicity, free/reduced lunch status, ELL status, and special education status? Is there evidence that gaps are narrowing? Are program services reaching students who are at the greatest risk?

O3. To what extent are observed changes in student outcomes attributable to STEM ECHS activities versus other RTTT program activities and/or measurable contextual variables?

O4. What are the major differences in STEM ECHS implementation and contextual variables across schools whose levels of improvement on student outcomes differ substantially?


51



Schools and Partners

The six chosen districts will each receive approximately $120,000 of RTTT funds to be spent over four years for school planning, start-up expenses, and full implementation. Two schools opened with small classes of students in fall 2011 (including one that suspended activities during the second semester); one school opened during summer, 2012 (slightly ahead of schedule); and three additional schools are scheduled to open in 2012. A list of funded school districts and their higher education partners is provided in the table below.

STEM ECHS Sites and IHE PartnersDistrict Start

DateInstitute of Higher Education Grades

OfferedMarlborough Public Schools Fall 2011 Framingham State University 6-12Quaboag Innovation STEM early college high school (Quaboag Public Schools) Fall 2011 Holyoke Community College,

Worcester Polytechnic Institute 9-12

Southeastern Regional Vocational Technical (Massachusetts Association of Vocational Administrators - MAVA)

Summer 2012 Northeastern University 9-12

Randolph High School (Randolph Public Schools) Fall 2012 Massasoit Community College 9-12

North High School (Worcester Public Schools) Fall 2012 Quinsigamond Community

College, College of Holy Cross 7-12

Dearborn Middle School (Boston Public Schools) Fall 2012 Northeastern University 6-12

Methods

The Year 2 evaluation included the following activities and data sources:

Interviews with ESE program managers, the technical assistance vendor, STEM ECHS administrators, and IHE partners.

o Three in-person interviews with ESE program managers in September 2011, December 2011, and July 2012 (see interview protocol in Appendix B1).

o Three phone interviews with the technical assistance vendor, Jobs for the Future (JFF), in October 2011 and August 2012 (see interview protocol in Appendix B2).

o Interviews with administrators from the six Goal 4E sites in February and March 2012. These included phone interviews with the four pre-implementing sites, and in-person interviews during site visits to the two implementing sites (see interview protocols in Appendices B3 and B4).


52


o A 1-hour phone interview in June, 2012 with an administrator from Framingham State University regarding their partnership with Marlborough’s STEM ECHS (see interview protocol in Appendix B5).

Focus group with school personnel. A focus group was conducted in June 2012 with two middle school teachers and two high school teachers from the one STEM ECHS site that was fully operational during the 2011-12 school year (see focus group protocol in Appendix B6).

Site visits. Two STEM ECHS sites were visited, one of which was implementing ECHS activities at the time of the visit (see observation protocol in Appendix B7). Each site visit included one or two interviews with key school personnel and brief observations of school activities. In addition, UMDI attended two full-day technical assistance meetings hosted by JFF and ESE.

Document review. UMDI reviewed two sets of reports from a JFF consultant's individual meetings with the six schools, as well as several documents (e.g., proposals, continuation reports, marketing materials) from several STEM ECHS sites. UMDI also monitored use of “Base Camp” – an online resource maintained by JFF which serves as a professional learning community and project collaboration tool.

ESE databases and documents. The STEM ECHS quantitative data presented below and in the “Technical Supplement - RTTT C&CR 2012 Evaluation Annual Report” were developed using ESE’s 4-year and 5-year graduation cohort lists as well SIMS data from October and June of 2008-2011. Findings were also informed by ESE documents including the RTTT application, and graduation and annual dropout rate information reported by ESE.

Supplemental student data request. Supplemental data requests were prepared for STEM ECHS sites, asking them to indicate which students are receiving services, as well as attendance and credit accumulation data that are not included in the state databases but are needed for answering evaluation questions. The deadline for schools to respond is October 2012, so related findings will be incorporated into future reporting.

Findings

Findings are presented in relation to the process and outcome evaluation questions. Process questions P1 and P3 are addressed in the Project Status, Planning, and Implementation sections. Questions P2 and P4 are addressed in sections on Support and Sustainability. The outcome questions are then addressed with the presentation of baseline graduation and dropout rate information for the STEM ECHS, a summary of student participation to date, and a discussion of how outcome data that will be available for subsequent project years will be utilized.

Project Status

The six STEM ECHS sites receiving RTTT support are in various phases of project development and implementation. Two sites (Marlborough and Quaboag) enrolled students in STEM ECHS activities during the 2011-12 school year. Three sites (MAVA, Worcester, and Randolph) are preparing to launch with inaugural cohorts during the 2012-13 school year. Dearborn's STEM ECHS work has been incorporated into a broader school redesign plan. Additional information for each site is provided below.


53


Marlborough. The Marlborough STEM ECHS began serving 6th- and 9th-grade students in fall 2011. District personnel reported that nearly 200 students (115 in grade 6 and 74 in grade 9) were served. Marlborough intends to add two grades per year to the STEM ECHS until full enrollment (grades 6-12) is reached in 2014-2015. The STEM ECHS programs are housed as schools-within-a-school in the middle school and high school buildings.

The implementers for this site originally intended to place a higher priority on serving underachieving students, many of whom were minorities. However, during the first year of program implementation all applicants were accepted, approximately 50% of which were underrepresented minorities. The district held several community forums in order to inform parents/guardians about the launch of the school, its purpose, and the district’s plans for serving students.

Students entering 9th grade participated in an all honors curriculum (Geometry, Algebra, English, History, Physics, and Engineering). District personnel reported that many students were unprepared for the demands of this coursework. In order to accommodate student and teacher requests for increased support, the district reduced teaching loads from five classes to four, and arranged a daily common planning period at the high school. Similar accommodations were instituted for STEM ECHS teachers at the middle school, but were subsequently cancelled because of perceived inequities in course loads. The middle and high school both emphasized 21st-century skill development as well as inquiry- and project-based learning experiences for students in all classes. The school issued laptops to all students to increase technology use and enhance students’ educational experience. Marlborough is working with several business partners to support the laptop initiative.

STEM ECHS administrators and Framingham State University, Marlborough’s IHE partner, are developing a longitudinal study of the STEM ECHS. As a part of this work, a set of surveys and focus groups were developed by FSU faculty and administered to STEM ECHS students and staff. To the extent that these activities provide information relevant to UMDI's evaluation, they will be presented in future reports. FSU has also designated a faculty fellow to provide professional development for high school staff and students in the area of college readiness.

Marlborough has reported that most members of the current 6th and 9th grade STEM ECHS cohorts will be promoted to 7th and 10th grade, respectively. A new group of approximately 135 6th-graders and 70 9th-graders have been selected for the 2012-13 school year. Marlborough’s STEM ECHS middle school received 172 applications for the 135 available seats, so Marlborough selected students by lottery.

Marlborough anticipates that students will begin participating in college coursework for credit during their junior year. There are no STEM ECHS students enrolled in college coursework during the 2011-12 school year, and none are scheduled to do so for the 2012-13 school year.

Quaboag. In the Fall of 2011, Quaboag enrolled 14 juniors and seniors in an electronics course taught by a high school teacher. Students who passed the course and met Accuplacer testing score requirements were eligible for non-transferable college credit at Quinsigamond Community College (Quaboag’s IHE partner). This program was suspended during the spring semester to allow time for staff to receive needed training and to further refine the articulation agreement between Quaboag and Quinsigamond Community College. Quaboag is preparing to re-launch their STEM ECHS during the 2012-13 academic year.


54


MAVA. Representing nine vocational technical high schools, MAVA is preparing to launch a STEM ECHS in the fall of 2012. The nine schools coordinate as one program through a planning committee of school representatives that make all key decisions and appoint subgroups as needed. Participating schools have agreed to make information technology the school's focus. The initial cohort of students was selected at the end of their sophomore year through an application process that was facilitated by school counselors and IT teachers. The selected students attended a June orientation program at Northeastern University and completed their first college course during the 2012 summer semester. Students in this STEM ECHS will participate in online courses and attend mandatory meetings with their cohort group at Northeastern or another pre-determined site. Students may also take courses in-person at Northeastern.

Randolph. Randolph is preparing to launch a STEM ECHS in the fall of 2012. A partnership has been established between Randolph High School and Massasoit Community College, Canton campus. A total of 39 students have registered to participate in STEM ECHS activities during the 2012-13 school year. Each of these students is participating in an Early College Experience and is scheduled to complete the math Accuplacer. Twelve of these students are currently registered for a Heavy Truck Systems course, in addition to their other courses. All courses are scheduled to be taught at Randolph High School. Participating students are eligible to receive high school and transferrable college credit. The Early College program is currently available to all high school students in grades 9-12.

Worcester. Worcester is preparing to launch a Health Sciences Academy at North High School in the fall of 2012 with 50 sophomores and juniors. The program intends to target low-income, underrepresented, and potential first-generation college students. Twenty-five rising juniors were identified for participation in STEM ECHS activities during the spring of 2012 and attended a college orientation course offered by Quinsigamond Community College (Worcester’s IHE partner). Worcester is in the process of finalizing a Memorandum of Understanding with QCC.

Dearborn. Dearborn was awarded $1.2 million in School Improvement Grant funds to implement an aggressive redesign plan to accelerate student achievement in the middle grades. Much of their focus is on completing the activities related to that goal. The STEM ECHS project is part of that plan, but few program specifics were available at the time of our evaluation activities. Developments at this site will be monitored during Year 3.

Planning

During interviews and site visits, STEM ECHS program administrators described a wide range of successes and challenges related to their STEM ECHS planning processes, as reported below.

Planning successes.

Assembling a planning team. Interviewees from five districts described assembling a planning team as a successful aspect of their planning process. They said that a key to success was the intentionally including representatives from diverse stakeholder groups. All districts included building administrators and representatives from STEM ECHS faculty and partnering IHEs, and some districts also included guidance counselors, union representatives, department heads, district-level administrators, representatives from partnering middle schools, and other educational leaders. Interviewees said that the inclusion of diverse stakeholders fostered support and buy-in for the project.


55


Several interviewees said that it was important to recruit volunteers for service on the planning committee whenever possible. Several also said that they included people who understood the importance of college readiness and who had experience in relevant content areas. Two interviewees stressed the importance of including practitioners on the planning team, with one saying “In my view, the [people in the] trenches every single day…the people who live it are the people that can tell you the most.”

The reported structure of planning team meetings was relatively uniform across sites. Teams and subcommittees held regular weekly, biweekly, or monthly meetings that were mostly scheduled before or after school hours to minimize time that teachers and administrators were removed from their regular duties. Several interviewees said that supports for planning that were provided by JFF had been useful in identifying important planning topics. Several interviewees said that consistent collaboration among team members was a key to their success. Several also said that holding planning meetings on a regular basis was difficult, because most team members had many other substantial responsibilities and commitments. Interviewees agreed that they were pleased with the level of commitment, skill, and ability that members of their planning teams had demonstrated.

Strong support from school and district leaders. Interviewees from five districts cited support from district leadership as a success, and several said that this support was essential to the planning process. They reported that their district and school leaders were generally responsive, positive, supportive, and helpful. Some also said that it was important that district leadership had expressed a strong commitment to the project and a willingness to participate in the planning process and assist with challenges. One interviewee said, “We have the support right from the top, which is the superintendent. That does not always happen. That is not the case in every district involved in this initiative, and it should be. The district leadership has to be aware of the decisions that are being made and has to support the work that is being done. That is the biggest hurdle.”

Establishing and formalizing relationships with IHE partners. Interviewees from five districts said that they had established strong working relationships with their IHE partner. Four said that IHE representatives regularly attended planning team meetings and were viewed as important team members. Interviewees frequently noted that their IHE partners were willing to share expertise, contribute ideas, and assist students in their transition into a college experience.

Supports for teachers. One district highlighted supports for teachers as a success in the planning process. Supports for teachers at this site included the development of structured small learning communities, common planning time, reduction of teacher duties, and professional development relevant to English language learners and students with disabilities. Each of these supports was intended to help teachers increase the quality of instruction provided to all students.

Other areas of planning success included selecting and securing a school site (two districts),excitement in the school and/or community for the project (two districts), strong sense of entrepreneurship and innovation (two districts), engaging the community (one district), being ahead of schedule (one district), curriculum development (one district), and student recruitment (one district).

Planning challenges.

Budgeting. Three districts identified budget management as a challenge. One noted a lack of clarity regarding the amount of funds available and how those funds could be used. Another wanted greater clarity in the budgeting process, because funds received from the state for middle school and high


56


school activities had traditionally been lumped together and then reallocated at the district level. A third interviewee reported not having the autonomy to make funding decisions for the program, resulting in program delays while waiting for district administrators to make decisions.

Instructional design, curriculum development and assessment. Three sites said that they had not made as much progress with instructional design and curriculum development as they had hoped, despite having thought that they had established a clear process and appropriate supports. The first site explained that their district had asked them to delay this work until a larger plan for the project was in place. The second site said that delays had resulted from the time needed for staff members to receive training to implement the target curriculum. The third site simply realized that their plans and supports had not been as well-developed as they had believed.

Staffing. Three districts reported staff-related challenges. One interviewee from a relatively small district said that they were constantly being asked to “do more with less” and assume multiple roles that exceeded their capacity. Lack of capacity also limited the number and type of courses that this school could offer to STEM ECHS students. Another interviewee said that staff turnover was a challenge, because several members of their STEM ECHS planning team were replaced during the fall. A third interviewee was concerned that staff might lack the skills to teach some of the courses that had been envisioned for the STEM ECHS.

Engaging the community. Three districts said that engaging the community was a challenge. Two said that parent/guardian and community interest and involvement in school-sponsored activities was typically weak in their districts. One of these interviewees reported that their project had repeatedly made efforts to contact parents and community members, with little success. The other site reported that they had not made any effort to explain or promote their STEM ECHS project to the broader community. One interviewee said, “We have not had the money or time to devote to telling our story, of how our STEM ECHS is configured and [has] evolved.” One interviewee described the challenge and goal of engaging the community by stating, “How do we engage our community in a fashion to show them that, through the development of this new school, we were going to be doing the best for all kids in the city?”

A third site said that efforts to engage the community prompted negative feedback from local parent groups and the media. This site reported that some parents were upset that the program had planned to prioritize admitting underrepresented students:

We had an article written about us. It was an editorial article in one of the local newspapers that talked about reverse discrimination, and the author of that editorial basically said that by…giving preference to underrepresented kids, we were showing reverse discrimination towards those kids who were high achievers, and whose parents were involved, and whose parents happen to be either highly educated or educated…But we continued to present this as an opportunity for all kids as we had established it. We really sort of just held our ground. We knew that a lot of the clarity for the community would not occur until they actually saw this in action, saw it working for students who hadn’t shown high levels of achievement in the same classrooms as students who had shown high levels of achievement. And, more importantly, that even those students who had exhibited high levels of achievement would still be challenged by the approach that we were developing.


57


This district held three public forums to address the concerns of parents and guardians. Although the district had originally intended to provide preferential admission to underrepresented students, in the end the district decided to accept all students who applied.

Communication. Several districts reported challenges related to the quality, frequency, or clarity of communication. Two said that it was difficult to communicate with parents/guardians, and another reported difficulty communicating with their IHE partner. A third said that it was sometimes difficult to communicate with district leadership. One interviewee said, “When you are recruiting students, informing parents, talking to teachers…It is a tall order to make sure that people have a consistent conceptual understanding of what is supposed to happen, how it is supposed to happen, when it is supposed to happen, and what we hope to accomplish.”

Making connections between middle schools and high schools. Three sites are attempting to integrate middle and high school learning experiences into a comprehensive STEM ECHS experience, but none of these sites has clearly articulated a plan for bridging the two levels. One district reported that there has been no significant contact between the middle and high schools related to the STEM ECHS project.

Continuity. Two districts reported difficulty maintaining the continuity of their planning team. One district reported that a change in school leadership had caused significant change to the planning team, which disrupted work that was in progress. A second district had concerns about continuity, because members were only assigned to the project for one year at a time. This interviewee said, “My big question is whether people who are on the planning committee this year continue to be on the committee next year. The degree to which that continuity is maintained or lost will determine, to a large degree, the success of this project next year.”

District support. One interviewee reported challenges with securing district support, saying “We need the collaboration and support of the whole district, and that has been difficult…The district asked the school to put some structures in place before moving forward, and we are waiting for a final decision to be made.”

Union concerns. Two districts reported challenges related to their teachers’ unions. The union in one district said that the smaller class sizes in the STEM ECHS were not equitable, and the union in the other district expressed concern that the agreement being negotiated with the IHE partner would reduce employment opportunities for union members.

Student preparation. One interviewee said that students in their district had received little exposure to science in elementary and middle school, which was raising concerns among school and district personnel that students lacked ability and interest in STEM-related activities.

IHE partner. One district reported challenges in establishing an effective working relationship with their IHE partner. The interviewee described a series of “stop and start” interactions, perceived confusion by both parties, and a generally poor record of communication that resulted in what the school perceives as a delay in establishing an articulation agreement in a timely fashion.

Suggestions for planning. Interviewees offered the following suggestions for others who are planning to launch a STEM ECHS.


58


Schedule adequate planning time. Having a minimum of a 12-month planning period is a realistic timeframe for planning the launch of a STEM ECHS. Schools and districts should be aware that launching a STEM ECHS is difficult and that they should set aside sufficient time and resources to address challenges as they arise.

Visit other STEM ECHS sites. Visit other sites and get information from people who have already launched their programs. It is difficult to anticipate challenges if you do not talk to people who have been through the process. These individuals can offer valuable insights.

Build interest and awareness of the program at the elementary and middle schools. Go to elementary and middle schools to build interest in the STEM ECHS within your district. This will help foster support for the program in the district and community.

Identify a point person for the project. There should be a “point person” for the STEM ECHS project. Ideally, this person’s sole responsibility would be to coordinate the planning and implementation of the program.

Build the program from the bottom up. Building a strong foundation of student success prior to offering college courses is important. Start with a group of 9th graders, for example. Give them the support they need. Have a variety of courses available, as well as a planned course sequence.

Work closely with your IHE partner. Your IHE partners may have a lot to offer and play a critical role in program development and implementation. IHEs may see the STEM ECHS as a money-making proposition, and it is important that schools and IHE partners see the collaboration as a “win–win” situation that is not confined to finances. The broader interests of the school and of the IHE partner should be well-aligned. It may also be valuable to explore your IHE partner’s capacity and interest in conducting research at your site related to the development and success of the STEM ECHS project. Work with your IHE partner to develop a clear articulation agreement. Plan a transitional experience for the students who will be taking college courses with your IHE partner.

Identify concepts central to the work. Two concepts that everyone involved in STEM ECHS should agree upon are that (a) the STEM ECHS should be available to and in the best interest of all learners, not just high achievers; and (b) STEM comes first – the focus needs to be on sustaining and strengthening the STEM pipeline.

Link the STEM ECHS to higher education and industry partners. Making meaningful connections among school systems, higher education, and industry is key.

Combat a lack of support with clarity and honesty. Be transparent with your partners, and be willing to answer questions and address challenges as they become apparent.

Implementation

For analytic purposes, only Marlborough is classified as an “implementing” site in this report. UMDI’s understanding had been that two STEM ECHS sites (Marlborough and Quaboag) were operational and conducted site visits to both of these sites. During our site visit, Quaboag representatives said that they considered their site to be engaged in an “extended planning year” and did not consider themselves to be an implementing site. One interviewee said, “We are still involved in the planning process. We had a soft launch, a kind of pilot class running this fall, the electronics course…As we continue to develop the program for a full launch this fall, we are still in the middle of program design and curriculum design.”


59


Therefore, the findings presented below are specifically about Marlborough, and are drawn from a site visit to the school, interviews with program staff, a focus group with two middle school and two high school teachers, and an interview with Marlborough’s IHE partner.

Implementation successes.

Continued support for students. One interviewee said that a primary success in launching their STEM ECHS has been the staff’s ability to encourage and support all students in their efforts to complete a curriculum of five honors-level courses. Because many students were included whose skill levels would not typically lead to participating in an honors-level curriculum, teachers have had to structure their curriculum to address gaps in students’ skills, content knowledge, and habits of mind. The interviewee said that a key to the program’s success was the staff’s shared commitment to making every student successful, including an intentional effort to address both academic and cultural components of the student’s educational experience that can contribute to achievement.

All teachers said that the majority of STEM ECHS students were having a positive experience and were “moving in the right direction.” Teachers also said that the school’s one-to-one laptop initiative had been a success. One middle school teacher said, “Our 6th-grade students’ skills on computers far surpass 8th-grade students’ skills on the computers. Word processing, AutoSketch, research, the whole bit…I think it has been wonderful.”

Curriculum development. An interviewee from the implementing site said that curriculum development had been a significant strength, and that most students were enjoying their classes, engaging with a challenging curriculum, and increasing their levels of achievement. The interviewee explained that much of the STEM ECHS curriculum was based on the Massachusetts curriculum frameworks, saying

Massachusetts is well known for having a respected curriculum. Many people have done the legwork to identify the concepts that need to be introduced to students at different points in time, and we continue to improve upon that with things like the Common Core. So the conceptual work – identifying the concepts – has been done for us in this state.

The interviewee said that, in order to build on the state curriculum frameworks, teachers continue to research how they can integrate a “STEM philosophy” and “21st-century skills” into the students’ learning experiences in every class. One successful strategy has been infusing activities that foster collaboration, critical thinking, and creativity. These concepts have been integrated into the curriculum through project-based learning experiences, such as a first-term project related to transportation safety.

Collaboration with IHE partner. One interviewee described FSU's partnership with Marlborough as collaborative and said, “I really enjoy working with this group of people. It has been a really good experience.” He noted that there have been regular communications and meetings throughout the year, and that both FSU and Marlborough have hosted partnership meetings. At one such meeting, an FSU faculty member met with representatives from Marlborough to discuss key aspects of college readiness. The interviewee reported that this meeting enhanced FSU's understanding of the scope and structure of the STEM ECHS and of the students being served. In addition, the FSU faculty member shared information intended to help the Marlborough school personnel to prepare the STEM ECHS students for a successful college experience.


60


FSU faculty members also worked with Marlborough personnel to develop online surveys that were distributed to all STEM ECHS students and teachers in April. Students were asked why they applied to the STEM ECHS, how the STEM ECHS was making a difference for them, what their expectations were, and how their experiences differed from previous years of schooling. Teachers were asked how their STEM ECHS experiences differed from previous years of teaching, and what impacts they felt the program was having on student learning. These surveys were followed by focus groups led by FSU faculty, and results were shared with Marlborough representatives in May. The interviewee said that students most frequently cited the desire to earn college credit while in high school as a reason for applying to the STEM ECHS. Both students and teachers reported that connectedness, collaboration, project-based and hands-on learning, and high expectations were contributing to the success of the program. FSU and Marlborough personnel are discussing how they could conduct longitudinal research to track the progress of the STEM ECHS students for several years.

Factors facilitating success. Teachers said that one key to program success has been the district’s commitment to providing teachers with time for professional collaboration. This time has been used to clarify working relationships, develop curriculum, and discuss plans for supporting individual students. One teacher said,

Any success we have had has been the direct result of us being able to collaborate as a team. At the high school we have a period every single day that we meet for 48 minutes. Being able to talk to the whole group each day, the six of us, has really helped so we all know what is going on. Just the communication. We use email a lot, but just seeing each other in person has really helped out.

All high school teachers, not just those participating in the STEM ECHS, have one period per day to collaborate. STEM ECHS teachers meet as a team each day during that time, rather than participating in departmental meetings. In contrast, at the middle school only STEM ECHS teachers were given a collaboration period, as a duty period, for a portion of the year. That period was later taken away after the teachers union filed a grievance citing unfair distribution of labor, arguing that some teachers had been assigned more traditional duties than others. Despite this challenge, a middle school teacher said, “At the middle school, I think that coming together as a team has been a success. And I do think that we have come together as a team…When we had that dedicated 45 minutes of collaboration, that was key in helping us to continue to move towards the shared projects that we were working on.” Teachers at the middle school now meet after school one day per week.

Interviewees reported that strong support from school and district administrators had been essential to the success of the STEM ECHS. Teachers reported that administrators had made strong efforts to support the work being done in their classrooms.

Implementation challenges.

Staff selection. The implementing site said that staff members at the middle school were assigned to the STEM ECHS by their building administration and were not given the opportunity to opt out of the program. One interviewee said that this had caused some tension with middle school teachers and that the situation is now being addressed through clear communication with these teachers, who will be allowed to opt out in the future. In addition, extensive efforts have been made to increase the enthusiasm of these discontented middle school STEM ECHS teachers about the curriculum they are teaching.


61


Tensions between STEM ECHS staff and other staff. One interviewee said that some teachers at the high school are resentful of teachers working with the STEM ECHS project, because the STEM ECHS teachers benefit from increased resources (e.g., increased planning and collaboration time, professional development) and decreased class sizes. The interviewee said that there is some sense in the middle and high schools that STEM ECHS teachers are viewed as privileged. Although no formal grievance has been filed by the local teachers union, the union is monitoring the situation. The interviewee reported that, in response to this challenge, the next STEM ECHS cohort would increase to 100 students from its current level of 72 students.

Teachers said that they have been challenged by colleagues who doubt their ability to deliver an honors-level curriculum to students with a broad range of academic abilities. One teacher said, “People don’t understand what we do and who our kids are. We teach all honors classes to students who would otherwise not be exposed to an honors curriculum.” Some teachers reported having to defend their work when confronted by colleagues who perceive inequity in the STEM ECHS program's access to common planning time and resources such as computers.

Technology. Teachers reported that they had faced many challenges with technology, primarily related to the one-to-one laptop initiative. The initiative, which provided all STEM ECHS students with a laptop computer for educational use, was not fully implemented until January. Teachers said that it was difficult and time-consuming to integrate the use of laptops into the curriculum at that point in the year. It took time for both teachers and students to become accustomed to having full access to the technology. One teacher said, “It has been just as much of learning experience for the teachers as for the students.” Teachers also said that there have been a significant number of reported issues with damaged computers (e.g., cracked screens), and that the computer hardware was less durable than they had anticipated. Despite these challenges, teachers agreed that they now view laptops as an important component of the students’ classroom experience, and an essential tool for learning.

Teachers from both the middle school and high school reported that the availability of Wi-Fi internet access had been somewhat sporadic, and that it was difficult to conduct technology-dependent lessons with an unreliable internet connection. They said that retrofitting their older school buildings with Wi-Fi technology proved to be challenging.

Teachers also said that some students lacked the maturity and sense of responsibility required to work in a digital learning environment. Some students used their laptops for non-class-related activities (e.g., using Skype during class), and these activities distracted other students and reduced the quality of their learning experiences. Teachers also said that there were always one or two students who forgot to bring their laptops to class, and that this also disrupted classroom activities.

Project-based learning. Teachers said that balancing the STEM program’s project-based learning activities with more traditional approaches has been challenging. They said that they spent more time on project-based learning activities at the beginning of the year, and less time on these activities toward the end of the year. Teachers said that they enjoyed the project-based learning activities and that they were still learning how best to leverage this approach to maximize student learning. Some teachers said that they reduced the amount of classroom time devoted to project-based learning during the second half of the year, so that they could cover more material during “crunch time.” Teachers said that it was a challenge to cover all key content standards during the year, and one teacher said that it was “pretty much impossible” to cover all of the relevant standards using a project-based curriculum, because there are “no perfect projects.” Middle school teachers also noted that a lot of “core teaching and learning” still needed


62


to take place at the 6th-grade level, and that in some cases they preferred to use more traditional pedagogical approaches.

Varied levels of student ability. Teachers said that it was challenging to implement the STEM program’s honors-level curricula with students of a wide range of academic ability and skill levels. High school teachers reported making continual efforts to support struggling students. Many students were encouraged to, and often did, stay after school to receive extra support. English language learners, in particular, had struggled with the STEM ECHS curriculum, and the system for supporting these students was still being modified. A representative from FSU who expressed that preparing a heterogeneous cohort of high school juniors and seniors for successful participation in college-level courses was going to be a significant challenge said,

I think the challenge is in the philosophy of wanting as many kids as possible to already have college credit when they enter college. To me, that is very different than being college ready. That is going to be a challenge for some of these kids. We have lots of FSU students who are freshmen and sophomores…and even they can’t succeed. With the STEM ECHS, we are talking about an even younger group – juniors and seniors in high school. We have to make sure that they are ready for this experience. Intellectually, psychologically, and socially ready. That is a huge challenge. I don’t know if enough has gone into this kind of thing...I understand that the objective is to get them all college credits by the time they finish high school, but there is a reality, too. I think about that.

This FSU representative said that next year will be pivotal for sophomore STEM ECHS students, because they will be preparing for participation in college courses as juniors. The representative said that these students need to have a clear understanding of what it means to be prepared for college so that they can have a positive experience at FSU.

Changes in teaching practices.

Teachers reported that they had made significant changes in their teaching practices, spending less time lecturing and more time with project-based, lab-based, and hands-on activities. All students are required to complete carefully planned multi-disciplinary group projects at the end of each quarter, and one teacher said that team members were constantly thinking about ways to foster such cross-curricular connections. All teachers agreed that teaching in the STEM ECHS felt like their first year of teaching in the sense that there was a “huge growth curve.” Other instructional and support practices mentioned by teachers included:

Collaborating with colleagues to schedule “project days” when students can work together across multiple class periods to complete a group project.

Learning about best practices for facilitating, supporting, and grading group work (e.g., students working in a group may not receive the same grade).

Covering similar topics at the same time across multiple content areas so that students can make connections across the curriculum.

Conducting regular “intervention team meetings” to discuss individual student progress and strategies for supporting student achievement.

Suggestions for implementation.


63


Teachers made the following recommendations for others trying to implement a STEM ECHS:

Common planning time is essential for success. Structure the time with a leader, goals, and an agenda, but be flexible.

Carefully consider the content and structure of public and parent communications. Invite community members and parents to view student accomplishments.

Be willing to change (e.g., they revised grading rubrics to reflect individual student effort).

Solicit student feedback about the STEM ECHS.

Publicly recognize student achievements (e.g., annual STEMy awards, student speakers).

Allow adequate planning time for all components of the program (e.g., content, pedagogy, administrative structures, collaboration).

Technical Assistance and Supports

All interviewees reported that ESE had consistently provided effective, professional, and timely assistance. District representatives said that ESE staff were accessible, helpful, and accommodating. Five of six districts said that ESE was viewed as a valued partner in their work. The one district that did not make this assertion said that their interactions with ESE related to this project had been limited, but of an appropriate level for their situation. Districts noted that ESE staff were aware of, and sensitive to, the contexts in which the districts were working.

Jobs For the Future has provided technical assistance to the six Goal 4E sites, working in consultation with ESE. The technical assistance has included network meetings, individual technical assistance, and “just-in-time” problem solving. Network meetings have included professional development and technical assistance activities that were intended to facilitate the development of the STEM Early College High Schools. While technical assistance efforts have been tailored to meet each site’s needs, JFF has also provided a planning framework with benchmarks and timelines intended to assist sites in determining appropriate timelines and criteria for making key development and implementation decisions. JFF has also provided each site with just-in-time phone and email access to technical support. JFF has provided ESE with multiple written updates for each site, based on site visits and technical assistance conversations. JFF also reports monitoring progress at each site to ascertain if key milestones are met, and has hired an outside consultant who has overseen much of the technical assistance.

District representatives were asked to provide feedback on their interactions with JFF. Interviewees from five of six districts said that JFF was viewed as a valued partner that was helping to move their project forward. Districts said that periodic check-ins, combined with specific suggestions for next steps, have been most helpful. The following comment was typical across these five sites:

Phone calls and conference calls with JFF and our planning committee have always been very helpful. Our interactions with JFF at the recent technical assistance meeting helped us move forward. They provided helpful suggestions for us, such as related to our communication challenges. JFF has helped to pull the group together, including our IHE partners. I would like to continue to work with JFF. They have really listened to where we are at and provided suggestions for helping us move forward. Just working with someone who has gone through this process before, and can anticipate our challenges before we can, is very helpful.


64


Two of these five sites said that additional support from JFF would be helpful but did not know what additional services or supports JFF could offer. One interviewee suggested that it would be helpful if JFF could identify, describe, and provide training on a suggested instructional approach or set of instructional best practices for STEM ECHS sites, particularly those known to increase the success of at-risk students.

One of the six interviewees did not consider JFF to be a supportive partner, but sharing specific comments here risks the interviewee’s confidentiality. After the interview, UMDI followed up to request permission to share specific concerns with ESE, in accordance with the confidentiality agreement. The interviewee responded that, because the topic and context were sensitive, he preferred and intended to address concerns with ESE directly. He later confirmed that he had indeed spoken with ESE about it.

The teachers who participated in the focus group were not aware of technical assistance that had been provided by JFF or ESE. They said that their only exposure to JFF had been when the vendor conducted classroom observations. One teacher said, “I am sure there is stuff that we haven’t seen…but it would be hard to say what benefit we got from these visits.” Teachers reported that they had made no requests for assistance from JFF or ESE.

Similarly, a JFF representative expressed a desire to increase the scope of their work with the sites:

I just wish [the resources were available to] do more intensive work. This is our first venture of this kind in Massachusetts along with the intensive work we have been doing [with other clients]…if you don’t get into classrooms and really work with teachers it’s very hard to feel like you are going to have a major impact and change the students’ outcomes…Given our resources, we could really only do very limited professional development.

When asked how they would like to expand their work, a JFF representative stated that they would increase the amount of coaching at each site and increase connections between the sites and the STEM business community. She said,

I think that with these schools, if we were going to really fast track their progress, we would really need much more coaching on a much more regular basis. With additional support, we would do much more intensive professional development than we have been able to do in the past. If we were doing this project in an intensive way, we would be meeting with small and large employers consistently. We’d bring them into the schools and try to create some teacher externships, because the first thing that needs to happen is that teachers need to understand the contemporary STEM workplace, and most of them actually have no way of knowing much about it. There are many things I would do to create urgency, visibility, and a sense of the importance of the project.

The JFF representative also acknowledged current budget limitations and realities, and noted that these limitations were well understood. She said,

Unfortunately it comes down to how much funding we have and how much time, and I think everybody understood at the outset [what the constraints were]…I really do think that with some exemplary professional development in some consistent way, we could get some classrooms to really be clicking the way we would like them too.


65


Interviewees across all six sites made few specific requests for technical support. Two specific requests included the following:

Members of this team have never done this sort of thing before. We are looking for expertise in establishing a program so that we don’t spend our time going in wrong directions. There are documents that provide timelines as to what tasks are supposed to be accomplished, but there are limits to how helpful those documents are. Having a person to help guide or coach us through discussions, not to take over, would be helpful.

Your reputation [as a program and as a school] is often defined by the local media, and they don’t always put the message out there that you would prefer. [Media relations] is an issue that we would like to have some assistance around, and hear what the other schools are doing to market themselves and get their message out.

Several sites also expressed interest in additional structured time to meet with personnel from the other STEM ECHS programs to collaborate on specific topics such as scheduling and sustainability. One interviewee said,

Presentations and meeting times with other projects are very helpful. It is helpful to know what others are doing, and to share ideas and best practices. It is important to keep those conversations going. Perhaps not formal presentations, but sharing out about what we are doing and where we see our programs going. That is important because this is new territory. Meeting quarterly is good. That is how we learn.

One interviewee said that he would like JFF to provide additional technical assistance on how to better prepare students, who have a broad range of skills and abilities, to participate in college courses, as well as address the issues that high school students and administrators face when the students take courses on college campuses. He said,

Kids learn at different rates, and this hasn’t really been discussed. I have no problem with focusing on the administration of the program [at technical assistance meetings]. They have presented some best practices, but I would really like to see some more best practices from around the country in how these [STEM ECHS] partnerships are dealt with…I think one of the things that JFF can do is talk about the different issues that students and administrators face when these types of programs are at a college campus. I think it is really easy to do this at a high school level. I think it is much harder to do this at a college campus.

Sustainability

All interviewees (administrators, teachers, IHE partners, technical assistance vendors, and ESE) indicated that finding the financial resources to sustain the STEM ECHS programs is a significant challenge. Districts acknowledged that, without additional funding, their STEM ECHS programs would be at risk. Several districts said that the work of developing and launching a STEM ECHS was cost-intensive, and that staff development, curriculum development, planning time, and supplies were all major expenses related to the planning process. These districts also said that funding to support student participation in college courses was tenuous, and that cost structures and articulation agreements with IHE partners were still being negotiated. One interviewee said,


66


Sustainability is the huge, giant, 800-pound gorilla in the room that I think about all of the time.” Another said, “Funding college courses will come down to the money that is available for this. Ultimately, at the end of this there is going to be a need for a sustainability plan…The cost pieces include transportation, supplies and books, and the actual [college] credits. That is going to be the critical piece. Beyond the initial years of the grant, how do we make this sustainable?” A third interviewee seemed pessimistic about long-term prospects, saying “After a few years the money is going to go away. I think peoples’ ideas of how this program can be maintained over time will be deflated.

Marlborough’s IHE partner predicted sustainability challenges, noting that the program's viability was secure during, but not beyond, the RTTT grant period. He said,

Our tuition is basically $1,000 per course right now, and the grant funds that Marlborough has put aside for college courses will not cover that entire cost ...We will provide the discount that is necessary for this to happen. This is an important part of our mission in public education. I think it is critical that there is cost sharing on this type of thing. But I don’t know how it is going to play itself out after the Race to the Top grant is over. We are going to have to look for funding from other sources for sure.

Although all districts identified sustainability as a significant concern, none have established a clear plan for addressing this issue. Some said that they were beginning to discuss strategies for securing future funding, and several said that they anticipated committing increased levels of attention to the issue of sustainability over time. Ideas offered by district representatives to promote sustainability included: altering program recruitment practices to attract students from other districts to participate; hosting fund-raising events; exploring cost reduction strategies (e.g., grants, rate reductions, purchasing courses) with IHE partners; securing support from business and industry partners, and other donors; and applying for additional grants from public and private funding agencies.

A representative of Marlborough’s IHE partner reported that they plan to explore several funding options, including recruiting community members to sponsor one or more students; contracting with the Marlborough Public Schools to provide a fixed number of courses for a fixed number of students at a pre-determined rate; and having students and their families pay part of the FSU tuition, with reduced rates offered to students from low-income households. The interviewee reported that the STEM ECHS partnership has been designated as a continuing education activity by FSU administrators, which provided some financial flexibility because of unique aspects of the continuing education department's financial structures.

An interviewee from Jobs For the Future said that although sustainability challenges were substantial, securing and maintaining a strong commitment from stakeholders at each site was critical to their long-term viability. She said,

These schools have so little funding [from the state] to begin with, that if the small dollars go away it’s not going to mean they are going to lose a lot of staff…What will be a problem is if there are cuts across the board that impact the schools in general, since these STEM schools are making additional demands on people without providing with much by way of additional resources.


67


This interviewee also said that, over time, the state could consider adopting policies that would benefit or expedite the development of STEM ECHSs and other programs providing high school students with early college experiences. The ESE program manager for the STEM ECHS program provided similar thoughts on sustainability, saying that sustainability “comes down to the commitment of the people at the individual sites…We don’t have the needed state policies…there are inadequate connections between K12 and [the Department of Higher Education].”


The following three pages each contain a table paired with a figure that present the following baseline graduation and dropout rate data for the schools in which the STEM ECHS sites are hosted5:




The companion document, Technical Supplement – RTTT C&CR 2012 Evaluation Annual Report, presents tables and figures that provide a more complete summary of these same baseline graduation and dropout data. The Technical Supplement contains tables and figures that present both average and individual graduation and dropout rates for districts and schools implementing a RTTT-funded STEM ECHS. Future evaluation reports will use these baseline data to describe trends in graduation and dropout rates in STEM ECHSs and districts. The data necessary to disaggregate baseline graduation and dropout rates by subgroup (e.g., gender, race/ethnicity) were not available in time for this report, but will be included in future analyses. Future reports will also describe trends in indicators of student engagement and academic achievement in STEM ECHS schools and districts.

5 One middle school site is not currently associated with a high school. For MAVA, Southeastern Regional Vocational-Technical High School was included, due to its role as the applicant district.


68


STEM ECHS: 4- & 5-Year Graduation Rates

4- & 5-Year Graduation Rates for STEM ECHS Schools, 2007-08 to 2010-11

Year Number of Students



Difference Between Rates

2007-08 1,243 77.6% 81.9% 4.3%

2008-09 1,236 75.3% 78.6% 3.3%

2009-10 1,253 76.1% 79.5% 3.4%

2010-11 1,179 75.4% NA NA

4- & 5-Year Graduation Rates for STEM ECHS Schools, 2007-08 to 2010-11

2007-08 2008-09 2009-10 2010-1150%

55%

60%

65%

70%

75%

80%

85%

90%

95%

100%

4 Year Rates

5 Year Rates

School Year

Gra

duat

ion

Rat

e


69


STEM ECHS: Annual Dropout Rates

Annual Dropout Rates for STEM ECHS Schools, 2007-08 to 2010-11

Year Total HS Enrollment Number of Dropouts Annual Dropout Rate

2007-08 4,806 167 2.4%

2008-09 4,675 183 3.9%

2009-10 4,659 168 3.6%

2010-11 4,621 182 3.9%

Annual Dropout Rates for STEM ECHS Schools, 2007-08 to 2010-11

2007-08 2008-09 2009-10 2010-110%

1%

2%

3%

4%

5%

6%

7%

8%

9%

10%

2.4%

3.9% 3.6%3.9%

School Year

Ann

ual D

ropo

ut R

ate


70


STEM ECHS: 4- & 5-Year Cohort Dropout Rates

4- & 5-Year Cohort Dropout Rates for STEM ECHS Schools, 2007-08 to 2010-11


4-Year Cohort Dropout Rate



2007-08 1,243 10.9% 11.0% 0.1%

2008-09 1,236 11.9% 12.6% 0.7%

2009-10 1,253 10.5% 10.9% 0.4%

2010-11 1,179 10.0% N/A N/A

4- & 5-Year Cohort Dropout Rates for STEM ECHS Schools, 2007-08 to 2010-11

2007-08 2008-09 2009-10 2010-110%

5%

10%

15%

20%

25%

30%

4-Year Rate5-Year Rate

School Year

Coh

ort D

ropo

ut R

ate


71


Student Participation

The table below summarizes student participation at STEM ECHS sites during the 2011-12 school year. These figures reflect the activities of each school as described in this report, with full implementation by Marlborough and initial activities by Quaboag, MAVA, and Worcester.

STEM ECHS – Summary of Student Participation – 2011-12District Number of Students ImpactedMarlborough Public Schools 189Quaboag Innovation STEM early college high school (Quaboag Public Schools) 14

Southeastern Regional Vocational Technical (Massachusetts Association of Vocational Administrators - MAVA)

28

Randolph High School (Randolph Public Schools) 0

North High School (Worcester Public Schools) 25

Dearborn Middle School (Boston Public Schools) 0

Total 256

Conclusions

The six Goal 4E STEM ECHS sites are in various phases of project development and implementation. Two sites enrolled students in STEM ECHS activities during the 2011-12 school year, three sites plan to start during the 2012-13 school year, and the last site’s STEM ECHS work has been incorporated into a broader school redesign plan.

Interviewees from five districts described assembling a planning team as a successful aspect of their planning process. Additional factors that contributed to the success of the planning teams were including representatives from diverse stakeholder groups and consistent collaboration across members. Respondents shared that holding planning meetings on a regular basis was difficult, because most team members had many other substantial responsibilities and commitments. Interviewees from all districts cited support from district leadership as essential to the planning process and establishing strong working relationships with their IHE partner.

Informants reported many challenges that arose during the planning phase, including budget management, curriculum development, staffing, engaging the community, and working with teachers’ unions. Several districts also reported challenges related to the quality, frequency, or clarity of communication across stakeholders.

Marlborough is the only site that began full implementation during Year 2, with 189 students in grades 6 and 9. One Marlborough interviewee said that a primary success has been the staff’s ability to encourage and support all students in their efforts to complete a curriculum of five honors-level courses. All teachers said that the majority of STEM ECHS students were having a positive experience and “moving in the right direction.” The interviewees expressed that the following factors have contributed to their successful


72


implementation to date: strong curriculum development, a collaborative partnership with the IHE, and the district’s commitment to providing teachers with time for professional collaboration.

Many challenges were reported by Marlborough teachers. One high school teacher reported that some colleagues working within the main high school resent the increased resources (e.g., structured planning and collaboration time, professional development) and decreased class sizes enjoyed by the STEM ECHS teachers. A middle school teacher reported that building administrators assigned personnel to the STEM ECHS without giving them the opportunity to opt out of the program. STEM ECHS teachers also reported that they had faced many challenges with technology, primarily related to the one-to-one laptop initiative, which was not fully implemented until five months into the school year. Teachers said that it was difficult and time-consuming to integrate the use of laptops into the curriculum at that point in the year.

All districts reported that ESE had consistently provided effective, professional, and timely assistance. Districts noted that ESE personnel were aware of, and sensitive to, the contexts in which the districts were working. Jobs For the Future, working in consultation with ESE, provided technical assistance to the Goal 4E STEM ECHS sites. The support provided to the sites has individual technical assistance, meetings with all six schools, and “just-in-time” problem solving. The majority of districts said that JFF was viewed as a valued partner that was helping to move their programs forward.

All interviewees (administrators, teachers, IHE partners, technical assistance vendors, and ESE) reported that finding the financial resources to sustain these STEM ECHS sites will be a significant challenge. No districts have established a clear plan for addressing this issue, although several said that they anticipated committing increased levels of attention to sustainability issues over time. One interviewee also suggested that, over time, the state could consider adopting policies that would benefit or expedite the development of STEM ECHSs and other programs providing high school students with early college experiences.


Initial lessons from the early-implementing STEM ECHS schools could benefit the other STEM ECHS schools and inform technical assistance from JFF and ESE. These lessons include the importance of budgeting adequate time for planning and collaboration, integrating needed technology, communicating with key constituencies, and preparing high school students for the challenges of college-level courses.

Continuity of leadership and succession planning appear important to success of the STEM sites. Multiple sites reported that lack of continuity in their planning team membership, as well as turnover in school and district leadership positions, had disrupted their planning and implementation processes. JFF also identified lack of continuity as a significant threat to success and sustainability. Continuity should be encouraged whenever possible, and when personnel transitions are anticipated, succession planning should be undertaken to forestall problems.

Securing the financial resources to sustain STEM ECHS partnerships is a primary concern of all STEM ECHS sites. Unsurprisingly, given the more pressing concerns of launching a new school, none of the six schools have developed a plan for addressing this challenge. JFF and ESE may therefore wish to make sustainability a focus for technical assistance.

Changes to state and local policies may facilitate STEM ECHS development. Some informants indicated that state or local policies exist that delay or impinge upon the development of STEM ECHSs. While this was not formally evaluated in Year 2, it will be specifically addressed in Year 3 evaluation activities.


73


Increased connections with the broader STEM community could benefit ECHS programs. The technical assistance vendor noted that limited program funds have precluded some of the STEM connections outside the school that are often beneficial to STEM ECHS activities. To the extent possible with available resources, schools should increase efforts to connect with businesses, universities, and other organizations with STEM expertise.


74


MassCore Policy and Implementation

Background

The Massachusetts High School Program of Studies (MassCore) recommends a set of courses and other learning opportunities which Massachusetts students should complete before graduating from high school, in order to arrive at college or the workplace well-prepared and without need for remedial coursework. The 155 districts that selected the RTTT college and career readiness goal committed to implementing strategies to increase the percentage of their students who complete the MassCore curriculum.

The state’s RTTT goal is to increase the statewide MassCore completion rate from its baseline of 70% of the Class of 2010 graduates to 85% of the Class of 2014 graduates. The state has created a goal for each district, using a formula based on the district’s reported 2010 MassCore completion rate (calculated from the MassCore element of the state’s SIMS database), the district’s number of 2010 graduates, and the total number of graduates statewide needed to bridge the gap between the 70% baseline and the 85% goal. Each district was also expected to determine areas in which courses or supports needed to be expanded in order to meet the 2014 targets, and to create and implement a plan to improve the accuracy of their reporting of MassCore completion levels. The evaluation questions specific to the MassCore evaluation are listed below.


P1. What steps have districts taken to facilitate MassCore completion by students? What are the major challenges to and facilitators of district implementation and student completion of a rigorous course of study in general, and the MassCore curriculum in particular? What midcourse corrections and attempts to overcome challenges have been undertaken? What additional steps are necessary?

P2. To what extent has ESE taken the steps to increase MassCore implementation described in their RTTT grant application? What are the major challenges to and facilitators of district support encountered by ESE? How have challenges been overcome and midcourse corrections undertaken? What additional steps are necessary?

P3. How do district stakeholders explain the relevance of MassCore implementation and completion to their efforts at promoting student college and career readiness, as well as the effectiveness of their efforts toward fostering greater MassCore implementation and completion?

P4. What infrastructure, systems, and processes were put in place to aid sustainability of improvements in MassCore implementation and completion during and beyond the grant period? What supports could ESE or other partners provide to districts to increase levels of MassCore implementation and completion? What are the greatest challenges and barriers to creating sustainability?


75



O1. To what extent are districts increasing MassCore implementation, and to what extent are students achieving increased MassCore completion? What are the gaps in student completion of coursework required for MassCore completion?

O2. At the school and district levels, do observed changes in MassCore completion differ across student characteristics such as gender, race/ethnicity, free/reduced lunch status, ELL status, and special education status? Is there evidence that gaps are narrowing? Are district resources related to increased MassCore program completion reaching students who are at the greatest risk?

O3. To what extent are changes in student MassCore completion attributable to state and district efforts in service of this goal versus other RTTT program activities and/or measurable contextual variables?

O4. What are the major differences in MassCore implementation, completion, and contextual variables across schools and districts whose levels of improvement on student outcomes differ substantially?

O5. What is the relationship between level of MassCore implementation and student achievement of MassCore completion and other targeted student outcomes such as increased graduation rates and reduced dropout rates?

Methods

The Year 2 evaluation of the MassCore initiative including the following activities and data sources:

Interview with district administrators. Five district administrators were interviewed by phone about their district's actions and plans for implementing the MassCore program of studies (see interview protocol in Appendix C1). The interview’s purpose, in addition to providing an in-depth understanding of MassCore implementation in a small number of districts, was to help generate questions for an online survey that was subsequently sent to all MassCore districts. The sample was drawn from a list of 20 potential interviewees from ESE's list of Race to the Top contacts, provided by ESE’s MassCore program manager, who emphasized urban districts because he said they have the greatest need for increased MassCore completion rates. Nine districts were contacted, in some cases multiple times, and five responded and were interviewed. Contacts were told that UMDI was seeking "a district administrator who has in-depth knowledge of the district's positions, actions, and plans related to MassCore," and they were invited to suggest an alternative interviewee as appropriate, which they did in two cases.

Survey of district administrators. A survey was conducted of RTTT administrators from the 155 districts that selected RTTT Goal 4A (MassCore) (see protocol in Appendix C2). The survey questions were informed by responses to the MassCore district administrator survey. The survey sample was 142 of the 155 administrators, because 4 of the districts did not serve high school students, 4 of the administrators had opted out of receiving surveys from the SurveyMonkey website, and 5 of the administrators served two districts each.

Interviews with ESE program manager and project director. Three in-person interviews were conducted, as well as frequent, informal email and phone communications throughout the year.


76


ESE documents and databases. The MassCore quantitative data presented below and in the Technical Supplement to RTTT C&CR 2012 Evaluation Annual Report were developed using ESE’s 4-year and 5-year graduation cohort lists as well as SIMS data from October and June of 2008-2011. ESE documents including the RTTT Goals Workbook, spreadsheet of RTTT participating districts, and graduation and annual dropout rate information reported by ESE, were also used.

Findings

Findings are presented in relation to the process and outcome evaluation questions. MassCore questions P1 and P3 are addressed in the sections on commitment to MassCore, challenges to implementing MassCore, and actions taken to implement MassCore. Questions P2 and P4 are primarily addressed in the sections on support and sustainability. The outcome questions are then addressed with the presentation of baseline graduation and dropout rates for MassCore schools, baseline MassCore completion rates, and a discussion of how outcome data for subsequent project years will be utilized.

The discussion below draws primarily from the MassCore district administrator interviews and surveys, interviews with ESE staff, and review and analysis of state documents and datasets. For the district administrator interviews, the five respondents' districts ranged in enrollment from 5,000 to 14,000 students, with graduation rates ranging from 50% to 74% (mean = 64%) and with rates of free and reduced lunch ranging from 55% to 83% (mean = 71%). Respondent roles within their districts included assistant superintendent (2), director of grant-funded programs, director of student services, and RTTT program specialist. For the district administrator survey, the response rate was 74% (N=106), and the primary district roles of respondents were assistant superintendent (30%), superintendent (23%), curriculum coordinator (11%), RTTT coordinator (10%), and "other" (27%). The "other" category consisted of program, services, curriculum, and charter school directors (N=10); principals (N=2); and grants coordinators (N=2).

Commitment to MassCore

In order to understand MassCore in relation to RTTT, it is essential to recognize that ESE's intention, as expressed in their application for RTTT funding, was that "Massachusetts will make MassCore – currently a recommended program of high school studies – the de facto curriculum for the Commonwealth." As explained by ESE's MassCore program manager, this is still the state's intention, but it has not yet been realized. As a result, although many districts believe that MassCore completion is a worthy goal, and RTTT Goal 4 districts are spending funds on many college and career readiness initiatives, districts have no clear accountability to the state for achieving progress toward greater MassCore completion.

All five administrators who participated in the MassCore interview acknowledged that their district recognized the advantages of MassCore in terms of college and career readiness, and all supported MassCore to differing extents, mostly depending on the extent and types of challenges that they anticipated encountering in attempting to implement it. These concerns are explored in depth in the sections below.

Two districts portrayed themselves as strongly committed to and actively pursuing MassCore, although possibly in modified versions that acknowledged the limitations of their school infrastructure and capacity. One district said that the district administration strongly favors vigorous movement toward


77


increasing MassCore completion rates, but that the school committee is opposed and “doesn’t even want to hear about it” because it would mean that far too many students would fail to graduate. Two additional districts saw themselves as pursuing MassCore partially but anticipating major and potentially intractable challenges. The strong statement of commitment was:

We see major benefits. Making a significant commitment of our Race to the Top dollars in moving ourselves towards a state of readiness for MassCore and promoting that. We see it as essential for college readiness, issues of rigor and readiness amongst students who begin college, and we also see it as a standard that will increase our gradation rate.

Perhaps the least committed statement was “MassCore is kind of on the side burner right now because of the other things that are going on in the system…I can’t even get [school] to take MassCore seriously at this point.”

All districts reported that steps being taken toward higher rates of MassCore completion are being overseen by high-level district administrators and school administrators (e.g., principals, guidance directors, department heads, curriculum coordinators). Some also reported participation by high school and middle school faculty as well as working groups that focus on college and career readiness.

Respondents to the administrator survey were asked if MassCore completion is a graduation requirement in their district, and out of 84 respondents, 50% said yes, 46% said no, and 4% didn't know. As shown in the table below, more than 80% of survey respondents agreed that increasing the rate of student MassCore completion was a high priority and a realistic goal for almost all of their students. Respondents also indicated that they were aware of RTTT Goals Workbook targets, felt a strong level of accountability for achieving them, and regarded them as a realistic goal for their district. In addition, 72% agreed that their district is using RTTT funds to take actions that they believe will lead to increased rates of MassCore completion, and 80% agreed that their district is using non-RTTT funds for this purpose.


78


MassCore Goals, Knowledge, Attitudes, and Use of Funds

Please indicate your district's level of agreement with the following statements.

NStrongly

Agree(%)

Agree(%)

Neutral(%)

Disagree(%)

Strongly Disagree

(%)

Don’t Know

(N)

Increasing the rate of student MassCore completion is a high priority for my district.

86 55 36 6 2 1 0

My district is using RTTT funds to take actions that we believe will lead to increased rates of MassCore completion.

85 38 34 10 10 8 0

My district is devoting non-RTTT resources to taking actions that we believe will lead to increased rates of MassCore completion.

85 43 37 11 7 2 0

My district is aware of the MassCore targets for our district in the RTTT Goals Workbook.

85 46 43 6 4 1 1

My district feels a strong level of accountability for achieving the MassCore targets for our district in the RTTT Goals Workbook.

85 38 45 13 2 2 1

Achieving the MassCore targets in the RTTT Goals Workbook is a realistic goal for my district.

85 40 43 12 5 0 1

MassCore completion is a realistic goal for almost all students in my district.

85 44 40 8 8 0 0

Note: Response options included “Don’t Know.” The number of respondents who selected this choice is listed, but those respondents were not used to calculate the reported percentages.

One survey respondent said that MassCore has not been a district priority due to competing demands, but that their commitment would increase if it became a state requirement. In addition, one respondent was in favor of MassCore, but with exemptions as needed:


79


Maybe the state needs to define graduation requirements, rather than leaving it up to each district. Not to say that EVERY student can achieve MassCore, but maybe there needs to be a process to exempt specific kids for good reasons. MassCore completion should be the expectation for most.

Another respondent felt that MassCore needed to be a funded mandate:

Make MassCore requirements mandatory only IF you can provide the support that districts will need to make it happen. Perhaps you could designate a specific grant toward this goal, if there is not yet one already in place.

Finally, one respondent reported that their district had not received much information on MassCore, so it has not been a priority for them.

Challenges to Implementing MassCore

Districts have taken many actions toward implementing MassCore, and those actions will be described in the next section, as they are easier to understand in light of the challenges districts report. Those challenges are described next.

Four years of mathematics. The most commonly mentioned challenge was related to MassCore’s requirement of four years of mathematics, although the specific math-related challenges varied. One district reported that the math department had recently split Algebra 1 into a two-year course, against some strong opposition in the district, and that many students had been steered toward and selected this course. The math department strongly favors keeping this course due to previously high rates of failure in 9th grade mathematics. Because Algebra 1 only counts for one year of MassCore credit, these students would not complete MassCore even if they did complete four years of math courses. This district also offers a “technical math” course for the fourth year of math, but the interviewee didn’t feel that this course would meet MassCore or Common Core standards and would therefore also not count toward MassCore completion. The interviewee also said that many 9th graders are still failing even the slower-paced algebra course, so reducing rigor is clearly not the solution to the failure problem.

One respondent summarized the rigor issue by saying

MassCore completion is simplistic in terms of really addressing the bigger picture issues…It is a recommended course sequence, but it doesn’t speak to the quality of those courses. We all know that there is quite a range within what is considered to be aligned to standards and what is meeting the expectations of the standard. And so I think we could very easily get a higher percentage of kids completing MassCore if we were willing to have courses that allowed for fourth year math that [only] moves through algebra two, but the quality of the content of those courses still isn’t meeting the needs of our students and getting them college or career ready.

One district reported an issue of low expectations of students, that struggling students were guided away from a fourth year of math, and argued that that aspect of the district’s culture needed to change. Similarly, another respondent argued:

Some school personnel don’t want to eliminate the below-college-prep level. They offer the refrain of ‘You don’t understand; there are some kids who can’t do it,’ but I say ‘all kids can do


80


it.’ We have just got to figure out a way to get there…But at the high school there is still a sentiment of ‘These are the kids who can do it, and these are the kids who can’t. The kids who can are college bound. The kids who can’t are career bound.’ But we are now talking college and career ready, not college or career ready. That is a big paradigm shift for traditional high schools. So having that level of rigor, rather than the sorting of kids, is going to be something we are going to struggle with. It is probably going to take us three years to get there with any level of fidelity, but we will get there.

Another district administrator mentioned a similar situation, but reported a different current status: “The math teachers are fighting us tooth and nail on this, but we are doing away with that standard level.”

Other curricular areas. Districts also reported challenges with implementing other aspects of MassCore. Inadequate building infrastructure was cited by two districts for their inability to fulfill the physical education requirement, and by one district in relation to inadequate science laboratory space. Districts reported a shortage of teachers in MassCore-required areas including fine arts, lab science, and world languages, as well as a shortage of ELL teachers in disciplines other than English Language Arts. While some districts reported that their response to these shortfalls will be to work toward resolving them, others expressed a preference for modifying MassCore so it focuses on core curriculum areas but not on areas such as fine arts or physical education.

Some high-performing students may also encounter challenges with fulfilling MassCore. One district reported having a strong arts program in which many students take Advanced Placement art courses and devote substantial time to creating an arts portfolio. These students often lack openings in their schedule for sufficient physical education classes. Another school reported similar conflicts with students having to choose between a physical education course and an Advanced Placement lab science.

Other MassCore implementation challenges.

Two respondents said that 100% MassCore completion is a worthy goal, but that there will always be a small number of students who, due to severe learning disabilities, won’t be able to meet the MassCore requirements.

Three respondents noted constraints due to school schedules that lacked the flexibility to schedule students for courses required by MassCore.

Two respondents said that they lacked adequate funds to implement changes required in order to meet MassCore requirements. One cited future staff reductions, due to the pending termination of federal ARRA funds, as a serious obstacle to reform.

One respondent said that rather than the 4th year of math, their major challenge was that 15-20% of their 9th graders not being promoted to 10th grade, and concluded that preparedness for high school is the district’s biggest challenge to meeting MassCore requirements. Multiple respondents also cited student preparedness issues.

Survey respondents were asked the extent to which a series of challenges affected MassCore completion in their district. Eight areas were considered major or moderate challenges to MassCore completion by more than a third of respondents, including the need to build capacity to offer needed math courses (38%), science courses (40%), world language courses (51%), special education services (45%), and ELL services (50%); needing additional science laboratory space (41%); course scheduling (38%); and the gap between MassCore expectations and student skill levels (40%). The remaining areas were also seen as


81


major or moderate challenges by a substantial share of respondents, including the need for additional physical education facilities (23%) and art classrooms (20%), and that the MassCore curriculum requires more time for academic curriculum than is available for vocational students (26%).

Challenges to Increasing MassCore Completion Rates

To what extent do the following areas present a challenge to increasing MassCore completion rates in your district?

NMajor

Challenge (%)

ModerateChallenge

(%)

Minor Challenge

(%)

Not a Challenge

(%)

Don’t Know

(N)

We need to build capacity to offer needed math courses.

83 13 25 21 41 0

We need to build capacity to offer needed science courses.

83 11 29 20 40 0

We need to build capacity to offer needed world language courses.

81 26 25 15 34 2

We need to build capacity to offer needed special education services.

82 21 24 21 34 1

We need to build capacity to offer needed ELL services.

79 23 27 20 30 3

We need additional science laboratory space.

80 23 18 13 46 3

We need additional physical education facilities.

81 12 11 14 63 2

We need additional art classrooms.

78 6 14 17 63 3

Our master schedule makes it difficult for students to take the courses they need.

81 17 21 25 37 2

The gap between MassCore expectations and student skill levels.

80 13 28 26 33 3

For vocational students, MassCore completion requires more time for academic curriculum than is available.

53 17 9 19 55 22

Note: Response options included “Don’t Know.” The number of respondents who selected this choice is listed in the table, but those respondents were not used to calculate the reported percentages.

Survey respondents were also asked about the influence of an additional set of challenges on MassCore completion in their district, as shown in the table below. Those considered major or moderate challenges


82


by the most respondents included an inadequate budget to take needed actions (49%), that MassCore completion is not a requirement in their district (36%), and that MassCore completion is not required by ESE (23%). Only a small minority of respondents rated the remaining issues as major or moderate challenges, including lacking a district needs assessment (11%) or action plan (11%), disagreement between the district administration and the school board as to whether MassCore completion should be a graduation requirement (3%), or that the district leadership (1%) or school board (1%) do not believe that MassCore should be a graduation requirement.

Challenges to Increasing MassCore Completion Rates

To what extent do the following areas present a challenge to increasing MassCore completion rates in your district?

NMajor

Challenge (%)

ModerateChallenge

(%)

Minor Challenge

(%)

Not a Challenge

(%)

Don’t Know

(N)

MassCore completion is not a graduation requirement within our district.

81 9 27 10 54 2

MassCore completion is not required by ESE.

82 7 16 15 62 1

The district leadership does not believe MassCore should be a graduation requirement.

79 1 4 6 89 4

The school board does not believe MassCore should be a graduation requirement.

73 1 0 9 90 10

The district administration and the school board disagree as to whether MassCore should be a graduation requirement.

71 0 3 1 96 12

Our budget is inadequate to take the needed actions.

82 32 17 19 32 1

We lack a district needs assessment to identify needed actions.

80 3 8 11 78 2

We lack a district action plan for increasing MassCore completion.

82 2 9 17 72 1

Other 9 33 11 0 56 4


Five administrators who selected the "Other" response provided further explanation. Three reported budget challenges, with one saying, “Our budget was reduced by the town by more than $2 million from our needs-based assessment. This results in major deficits to acquire faculty, equipment, and technology


83


necessary to complete MassCore for our students.” One respondent said their district struggled with facilities and the allocation of time toward arts and physical education or academic courses. One reported that their major challenge is that the district’s mathematics graduation requirement is only three years.

Actions Taken to Implement MassCore

Districts proposed many approaches to moving toward greater implementation of the MassCore program of studies, including structural changes, ways of providing required coursework, needs assessment, and action planning.

Structural changes. Two respondents reported their district’s reform efforts as proceeding in two stages, starting with immediate work (such as online credit recovery) to target students at high current risk of not graduating, followed by or simultaneously with structural changes. One respondent cautioned against overemphasizing remediation, despite its importance, at the expense of larger structural changes, because “there comes a point where we need to look at the existing standard structures of the middle and high school experience and change those, and MassCore gives us the perfect opportunity and mandate how to do that.”

Structural changes include revising the school’s course schedule to accommodate courses required for MassCore completion. One school said that such changes will permit current 9th and 10th graders to take needed courses, but that the changes will be too late for 11th and 12th graders to complete MassCore requirements. Another school said that their recent conversion to a block schedule, although carried out before the current emphasis on MassCore, has had two major benefits related to MassCore. First, the block schedule creates more scheduling flexibility for electives. Second, and of substantial interest in terms of MassCore’s four-year math requirement, the block schedule enables many 10th graders to receive two years of math credit in one year by taking a math course both semesters. This means that many students enter 11th grade having already completed three years of math credit, thereby making the four-year MassCore graduation requirement more easily attainable.

Two districts also reported trying to move toward MassCore implementation at a practical level before trying to do so at a policy level. One district reported that, despite the school committee’s opposition to making MassCore a graduation requirement, “The principals and I and the guidance people have all been told that they are now to schedule kids as though MassCore is the law of the school district, and that is what they are doing.” They believe that establishing MassCore at a practical level will make it easier to convince the school committee to embrace MassCore at the policy level. Another district reported a similar strategy and said, “It is much better to have that internal debate and make these changes … so that it happens more organically. So by the time we are going to a public forum, we can point to where all our kids are, so changing our graduation requirements becomes more feasible.”

One district spoke about changes at the middle school level, explaining that younger students who have a history of promotion despite course failure don’t understand that ways in which that changes in high school and jeopardizes their ability to graduate in four years. As part of their implementation of the Mass Model, this district intends to do some transition planning interventions with middle school students that target this issue, and they believe that this could increase MassCore completion over the long term.

Survey respondents were asked about a series of actions that their district might have taken in service of increasing MassCore completion rates, and their responses are in the table below. The most common actions already taken were conducting a needs assessment to identify areas in which existing courses or


84


supports need to be expanded or new ones added (68%), adding new courses or expanding existing ones (68%), proposing changes to district policies (63%), successfully changing district policies (66%), and starting or expanding online credit recovery opportunities (58%). Only 47% reported that their district had developed a written action plan to increase MassCore completion.

Actions Taken in Service of Increasing MassCore Completion Rates

My district has planned and/or taken the following actions in service of increasing MassCore completion rates:

N

Action Already Taken

(%)

Action Planned ButNot Taken

Yet(%)

No Plan To Take This

Action(%)

Don’t Know

(N)

Conducted a needs assessment to identify areas in which existing courses or supports need to be expanded or new ones added.

81 68 22 10 2

Developed a written action plan to increase MassCore completion. 80 47 37 16 2

Proposed changes to one or more district policies. 78 63 22 15 5

Successfully changed one or more district policies. 77 66 16 18 4

Added new courses or expanded existing courses. 80 68 21 11 1

Hired additional personnel. 79 40 16 44 4

Added facilities (e.g., science labs, art classrooms, physical education). 77 16 20 64 5

Started or expanded online credit recovery opportunities. 77 58 26 16 5

Started or expanded teacher-led credit recovery opportunities. 73 48 19 33 7

Other 15 53 7 40 2


Administrators who selected the "Other" response option said that their students were already meeting MassCore requirements, so their district’s focus was on making sure that this continued (N=3). Respondents also reported partnering with colleges (N=2), providing summer credit recovery programs (N=1), developing middle school programs to increase readiness for 7th and 8th graders (N=1), starting an early college program for first-generation low-income students (N=1), providing dual enrollment (N=1),


85


developing a tiered system for academic and non-academic learning supports (N=1), and changing the graduation requirements for 2016 (N=1).

Ways of providing required coursework. Districts described many strategies that directly address student shortfalls in course completion and high school readiness, including the following:

Purchasing science lab equipment and creating new science lab space.

Hiring personnel to fill gaps in staffing for courses required for MassCore completion.

Hiring personnel to work directly with the highest-risk students, as identified by early warning indicator data, in order to promote credit recovery and prevent students from falling further behind.

Working with the APEX online credit recovery program.

Working with the Massachusetts Math and Science Initiative, both on their Advanced Placement program and on their Pre-AP program, to establish vertical teams between middle school and high school to improve vertical alignment and high school readiness.

Working with Agile Mind to provide a remedial summer algebra program for incoming 9 th graders, so that students can develop algebra skills and build relationships with teachers to help them feel more engaged and welcomed when they arrive for the beginning of the school year.

Piloting a strategy, in collaboration with Middlebury College, to use Plato online courses to fill gaps in world language offerings, not as credit recovery, but as an approach for students who can’t complete world language requirements due to schedule conflicts or personnel shortfalls.

Developing a 4th-year math course that allows and challenges students to apply the algebra and geometry that they learned in the first three years of high school in more exciting and practical ways.

Allowing school-sponsored sports team participation to count toward PE requirements.

Using RTTT funds to finance these activities, and to pay stipends to school and district personnel so that they can carry out MassCore-related administrative activities, curriculum planning, and professional development.

Needs assessment and action planning. All interview districts reported having conducted a review of course offerings and student performance data to identify gaps and problem areas, and all districts were able to articulate strategies that they currently employ and/or intend to employ in order to move toward greater MassCore completion. In addition, as noted in the tables above, a majority of survey districts reported that they have either conducted a needs assessment already (68%) or have plans to conduct one (22%), and nearly half reported that they have either written an action plan already (47%) or have plans to write one (37%). In a discussion of these findings, ESE’s MassCore program manager clarified that all Goal 4A districts had officially fulfilled ESE’s requirements for MassCore needs assessment and action planning by writing relevant paragraphs in their RTTT applications.

Advice to others on increasing MassCore completion rates. Administrators were also asked to provide advice to other districts about how to increase MassCore completion rates, based on their own district’s successes and challenges. The most common advice was to align graduation policy to MassCore


86


requirements (N=8). Two of these respondents also said that districts should include MassCore as part of their strategic planning process.

Six respondents commented on stakeholder buy-in. Two said that a high level of commitment from the administration, school committee, and staff was vital to success. Four suggested that administrators actively reach out, communicate, and campaign the public for support. One reported that their district was planning to begin using the news media for this purpose. Another recommended that administrators begin the campaign early and gather data to support the cause.

Four administrators recommended that districts incorporate intervention strategies and rigor at the elementary and middle school levels so that students enter high school prepared to succeed under the MassCore curriculum. Two respondents said that districts should be prepared to provide extra support and guidance for students in order to help them meet requirements. One suggested that student schedules be flexible enough to accommodate MassCore requirements, while another recommended a 4x4 AB schedule to increase course offerings. One respondent advised districts to work closely with each student and to place students on a trajectory for college acceptance.

Supports and Sustainability

For the most part, interview respondents did not feel that their districts had needed, requested, or received MassCore-related technical assistance. Most mentioned having spoken with ESE’s MassCore program manager when they had questions. One said “MassCore is common sense. I think this is just getting people together on the same page and making sure that they do what they need to do.” Another said,

It was really a matter of diving into our own data and having some meaningful conversations about looking at which students aren’t completing which requirements. What courses have they taken? What does their day look like? What did their day look like as a freshman, as a sophomore, as a junior, as a senior? Where did it fall apart for them? Was it right from the get go in freshman year or was it something happened after MCAS and when they got into the upper high school grades? All of that work we have been able to do ourselves, really.

When asked what their future needs or desires for technical assistance might be, interviewees offered several responses:

One respondent said that maybe they in fact did need technical assistance but just didn’t know what their needs were or what questions to ask.

Two respondents expressed interest in hearing about other districts’ successful practices, including one who said, “I think we would love to all know who is doing it right, particularly for districts that are similar to ours. If others have figured out creative solutions, we should be learning from each other.”

One said that they could use technical assistance with some of the larger structural changes needed related to MassCore completion.

One requested “tools or templates or exemplars that could help a district assess course-taking at the middle and high school level…quick data tools for comparison of course-taking failure rates and course sequencing…if there are tools that allow a district to do that or just some expertise in doing that, that would be most helpful.”


87


One respondent expressed feeling that ESE had provided inconsistent answers to inquiries about MassCore, and that there needs to be a clear repository of information that says either “Our clear policy is as follows: ____,” or “We haven’t totally figured that out yet.” The respondent felt that different people at ESE had provided different answers at different times, and that the answers had at times been confusing.

The administrator survey included three open-ended questions related to supports and technical assistance, starting with “What useful supports could ESE or other partners provide to increase MassCore completion in your district?” The most common responses were related to funding (N=14). One administrator said, “If you truly want districts to implement the MassCore...require it and attached funding to support it.” Respondents said they wanted additional funds for math, science, and technology equipment and resources (N=1); flexible funding for partnerships with higher education (N=1); funding for math and reading tutors to provide intensive intervention (N=1); and more funding for certified science teachers (N=1). One respondent asked ESE to support funding allocations on a per pupil expenditure basis. Another recommended that ESE “provide free Advanced Placement Summer Institute training for districts in all major high school content areas. These courses are extremely expensive for districts to find funds for staff to attend!” One respondent reported having budget and staffing challenges for science courses because of their school’s small size.

Four respondents said they could use support for students with special needs who struggle to meet MassCore requirements. Three of these were related to MassCore’s foreign language requirement, with respondents asking if there were any alternative options or exceptions for students with disabilities. One respondent asked ESE to “give practical suggestions for students with learning difficulties to be successful in completing world languages and advanced math (Algebra II and more), as well as chemistry and those kinds of programs.” One additional respondent asked for support regarding students with interrupted formal education.

Four respondents asked for clarification of MassCore requirements, with two saying that the Board of Higher Education and MassCore standards regarding Engineering and Technology are inconsistent. One asked,

Why is engineering not considered a lab-based science course? Please consider vocational-technical courses taught by Chapter 74 instructors who, while not considered to be highly qualified under NCLB, nevertheless teach rigorous courses with embedded science. Will an advanced course in Electronics Engineering be able to be counted as a lab-based science to meet college readiness requirements?

One additional respondent asked, “PARCC assessments – do we prepare for them? Science frameworks – how do we change when the old standards are now determining AYP status?” One administrator recommended that ESE provide criteria for determining course standards that meet MassCore requirements.

Respondents also asked for more support with scheduling (N=4). Three said that vocational schools need support with master schedule issues and special circumstances that can prevent students from taking required MassCore courses. One of these respondents asked for guidance on how vocational students can meet requirements for both MassCore and their vocational certificates in a seven-period day. One respondent asked that ESE or their partners “hold some workshops where districts could discuss and brainstorm solutions around master scheduling issues.”


88


Three respondents asked for more workshops and training. One expressed interest in attending workshops on conducting needs assessment and strategic planning. Another asked for more professional development for foreign language teachers who are delivering tiered instructions. Respondents also asked that ESE and other partners make available more tools and information (N=3). Administrators requested to have credit recovery tools at low cost, detailed information on how to develop an action plan, and an electronic resource of tools and approaches. One additional respondent said that the Massachusetts Teacher’s Association would be a useful resource to help with understanding teachers’ roles and responsibilities in meeting the needs of all students. Two respondents expressed their challenge with inadequate facilities, with one saying,

The best thing that you could do is eliminate the Physical Education requirement from MassCore. We have neither the space nor the personnel to fulfill the “spirit” of the PE requirement, and we will never be successful as long as it is a MassCore requirement – unless jumping jacks before lunch counts for the “letter” of the law. I spend too much time on this one point, and it has nothing to do with CCR (CPR maybe, but not CCR). As a graduation requirement, I would settle for some version of MassCore, especially concerning the academics and arts.

One respondent asked ESE to “increase pressure on districts to achieve targets, or at least to show focused, serious, coordinated effort. Link funding to achieving your outcomes, or at least to being able to document that you're doing everything in your power to achieve them.” Finally, one respondent said their district needed more time due to budget cuts and staff downsizing (N=1), and one requested support in educating the community about MassCore to gain public buy-in (N=1).

Specific questions about sustainability will be a greater focus of data collection efforts in the next two years, but the above comments clearly indicate that there are many supports that could aid districts’ implementation of a curriculum that fulfills the MassCore requirements. These include technical assistance focused on implementing MassCore and related structural changes; sharing of best practices from other districts; needs assessment tools; and an online repository of answers to MassCore policy questions, as well as funding and policy support, including a funded state mandate related to higher rates of MassCore completion and offering of the needed courses. Additional needs relate to staffing, facilities, scheduling (particularly for vocational schools), and alternatives or exemptions for students with disabilities. No technical assistance vendor has been enlisted to provide supports for and enhance sustainability of the MassCore initiative, unlike the STEM ECHS and Pre-AP initiatives that are also part of this evaluation of the RTTT college and career readiness initiatives.


The following three pages each contain a table paired with a figure that present the following baseline graduation and dropout rate data for the 155 schools that selected RTTT Goal 4A:




The companion document, Technical Supplement – RTTT C&CR 2012 Evaluation Annual Report, presents tables and figures that provide a more complete summary of these same baseline graduation and


89


dropout data. The Technical Supplement contains tables and figures that present both average and individual graduation and dropout rates for RTTT Goal 4A districts and individual schools. Future evaluation reports will use these baseline data to describe trends in graduation and dropout rates in MassCore schools and districts. The data necessary to disaggregate baseline graduation and dropout rates by subgroup (e.g., gender, race/ethnicity) were not available in time for this report, but will be included in future analyses. Future reports will also describe trends in indicators of student engagement and academic achievement in MassCore schools and districts.

MassCore: 4- & 5-Year Graduation Rates

4- & 5-Year Graduation Rates for MassCore Schools, 2007-08 to 2010-11





2007-08 44,196 78.3% 81.7% 3.4%

2008-09 44,646 77.9% 81.0% 3.1%

2009-10 44,376 78.5% 81.4% 2.9%

2010-11 42,710 79.8% NA NA

4- & 5-Year Graduation Rates for MassCore Schools, 2007-08 to 2010-11

2007-08 2008-09 2009-10 2010-1150%

55%

60%

65%

70%

75%

80%

85%

90%

95%

100%

4 Year Rates

5 Year Rates

School Year

Gra

duat

ion

Rat

e


90


MassCore: Annual Dropout Rates

Annual Dropout Rates for MassCore Schools, 2007-08 to 2010-11

Year Total HS Enrollment Number of Dropouts Annual Dropout Rate

2007-08 170,164 6,999 4.1%

2008-09 169,525 6,245 3.7%

2009-10 167,975 5,892 3.5%

2010-11 166,543 5,730 3.4%

Annual Dropout Rates for MassCore Schools, 2007-08 to 2010-11

2007-08 2008-09 2009-10 2010-110%

1%

2%

3%

4%

5%

6%

7%

8%

9%

10%

4.1%3.7% 3.5% 3.4%

School Year

Ann

ual D

ropo

ut R

ate


91

MassCore: 4- & 5-Year Cohort Dropout Rates

4- & 5-Year Cohort Dropout Rates for MassCore Schools, 2007-08 to 2010-11





2007-08 44,196 11.9% 11.7% -0.2%

2008-09 44,646 11.5% 11.7% 0.2%

2009-10 44,376 10.6% 10.9% 0.3%

2010-11 42,710 9.2% NA NA

4- & 5-Year Cohort Dropout Rates for MassCore Schools, 2007-08 to 2010-11

2007-08 2008-09 2009-10 2010-110%

5%

10%

15%

20%

25%

30%

4-Year Rate5-Year Rate

School Year

Coh

ort D

ropo

ut R

ate


92


Baseline Rates of MassCore Completion

The table below shows the distribution of MassCore completion rates for the 2009-10 school year for Massachusetts districts that include high school students. The data were provided by ESE and are based on the MassCore completion element in the SIMS database, which is provided by districts based on their internal determination of each graduate’s completion of MassCore requirements. The intervals in the table include 0% and 100% as separate categories, which is unconventional but deliberate in this case, as will be clear from the discussion below. The 2010 MassCore completion average was 70.3% (44,311 out of 63,072 graduates).

MassCore Completion by District for SY2009-10 Graduates

% Completion # of Districts % of Districts

0 18 6

1-20 14 5

21-40 20 7

41-60 19 7

61-80 27 10

81-99 41 14

100 146 51

Avg: 70.3% Total: 285 Total: 100

As shown in the table, 18 districts (6%) indicated that none of their graduates completed MassCore, while 146 districts (51%) indicated that all of their graduates completed MassCore. Conversations with ESE’s MassCore program manager, and with other ESE personnel, have made it clear that these numbers are inaccurate. Specifically, ESE personnel have pointed out that some of the districts that report 0% are high-performing districts by other academic measures, and some of the districts that report 100% are low-performing districts by other academic measures. These are very improbable outcomes.

It is, of course, possible that some districts do not have the capacity to fulfill certain MassCore requirements, such as if they lack adequate physical education facilities, which could explain a zero percent completion rate. Also, more than 40 survey respondents to the MassCore survey indicated that MassCore completion is a graduation requirement in their district, and additional districts could provide a rigorous curriculum to the vast majority of their students, leading to a MassCore completion rate near 100%. However, with a statewide reported baseline (2010) rate of 70%, it is unlikely that more than half of the districts have a 100% completion rate.

In short, many districts are reporting the SIMS MassCore element inaccurately. This means that baseline estimates of MassCore completion for the state, and particularly for individual schools, are likely to be very inaccurate. Moreover, estimates of post-baseline changes that are based on the SIMS MassCore element are likely to reflect factors other than actual changes in MassCore completion rates. Such factors could include improved reporting accuracy, which might actually lower the baseline level, since many districts are apparently over-estimating their completion rates at present. In addition, the RTTT targets that the state calculated for districts are, in many cases, different from the targets that would have been calculated if accurate completion rates had been available.


93


A final element that challenges the validity of MassCore completion reporting is that districts determine locally whether a given course meets MassCore requirements. There is no clear standard for districts to follow in making this determination, and there is likely to be wide variability in the content and rigor of courses that districts designate as meeting a particular requirement.

ESE reported that they currently do not know how most districts determine students’ MassCore completion status, although ESE reports that they are taking steps to understand this process more clearly. UMDI plans to interview some district data managers in Year 3 for the same purpose. However, given the current data available, it will not be possible to evaluate changes in MassCore completion rates accurately at the school, district, or state levels.


Districts appear to see substantial value in raising MassCore completion rates, but limited accountability for reaching state MassCore targets may subordinate MassCore among district priorities. Districts have very limited accountability regarding implementation of strategies designed to increase MassCore completion, or for raising student MassCore completion rates. MassCore remains a recommended, rather than a default curriculum, which reduces district accountability and the leverage of some MassCore advocates in their own districts. Until greater accountability structures are in place, achieving targeted rates of MassCore completion may be difficult.

Meaningful assessment of changes in MassCore completion is challenged by serious validity and reliability limitations of the MassCore element of the SIMS database. This SIMS element is the state’s primary way of assessing MassCore completion rates, yet analysis of the SIMS element shows extensive inaccuracies in district reporting. Improvements to implementation of the SIMS MassCore element, combined with increased district accountability for accurate MassCore reporting, would contribute to the ability to develop meaningful conclusions related to changes in MassCore completion. Implementation of such changes would complicate interpretation of changes from the state’s RTTT baseline MassCore completion rate of 70%, however, because the baseline rate was derived from the flawed SIMS indicator. When the new Student Course Schedule database has been in use for several years, and if implemented with high accuracy, it could also serve as a valuable source of information about MassCore completion.

A MassCore exemption process, or the promotion of alternative ways to fulfill certain requirements, may reduce some districts’ concerns about conflicting mandates and priorities. Districts said that some high-performing students, who excel on typical measures of academic achievement, may fall short of MassCore requirements due to taking Advanced Placement courses, or developing advanced art portfolios that prevent them from taking some required MassCore coursework. In addition, some students with severe learning disabilities may be unable to complete MassCore requirements, and being required to do so may be inconsistent with their IEPs. Some districts also lack costly infrastructure, such as adequate physical education facilities, that is needed for students to complete MassCore. The state may wish to consider whether there are cases in which conflicting priorities warrant exemption from specific MassCore requirements. Permitting certain extended and out-of-school learning opportunities to meet MassCore requirements may also address some of these concerns.

MassCore specifies the quantity of courses that must be completed, but questions remain regarding expectations of course quality. Guidance from the state on issues of expected course requirements, while perhaps unenforceable due to local control, may be


94


useful to and/or appreciated by districts who seek to understand what the state considers adequate for MassCore completion.

The most common challenges to increasing MassCore completion rates are: building capacity to provide required courses, services, and facilities; inadequate funding; the gap between student skill levels and MassCore expectations; course scheduling; and MassCore completion not being required for graduation.

Additional supports and policy changes may increase district progress toward MassCore completion goals. Supports could include technical assistance with implementing structural changes related to MassCore, sharing of best practices from other districts, needs assessment tools, and an online repository of answers to MassCore policy questions. Additional support may also include funding and policy changes, including a funded state mandate related to higher rates of MassCore completion and the offering of required courses. ESE may wish to consider enlisting a technical assistance vendor for the MassCore initiative as they have done for the Pre-AP and STEM ECHS programs.

More than a quarter of district administrator survey respondents did not agree that their district is using RTTT funds to take actions that they believe will lead to increased rates of MassCore completion. While in some cases this could mean that districts are taking actions but anticipate failure, but a more plausible interpretation is that they are using their RTTT Goal 4 funds for purposes unrelated to MassCore. For these districts – some of whom said that increasing MassCore completion is a high priority for their district – MassCore completion may be a lower priority than other strategies for increasing college and career readiness.


95


Appendices


96

Documents

Race to the Top College and Career Readiness Initiatives ... Web viewRTTT C&CR Evaluation Annual Report, September 2012. ... gender and race/ethnicity) ... Respondents’ administrative