Evaluating a Multi-Year, Federally-Funded Educational Initiative: Lessons from a Successful School District-Evaluator Partnership Kelly Feighan, Research for Better Schools Rorie Harris, Memphis City Schools Review characteristics of successful partnerships Summarize five-year partnership forged between K- 12 school district and evaluator Present how we addressed challenges that are inevitable to conducting school-based evaluations Provide lessons learned from the partnership to stimulate thought about improving evaluation work Objectives Great Programs Provide direct services that address needs Have a life cycle that ends Attract resources for expansion Benefit from constant monitoring Lieberman (2009) Great Evaluations Begin when program is first conceptualized Measure identifiable outcomes Demonstrate program effectiveness and value to funders Inform the field about effective programs Program immediate indicators of program decline Data collection interferes with program activities Data do not show positive outcomes Partners do not understand others role/expertise Program staff cannot articulate evaluation purpose Evaluators cannot describe the program Potential Sources of Conflict (Lieberman, 2009) Five-year Striving Readers Grant supported literacy integration in eight middle schools MCS: large, urban district with economically disadvantaged students, low state test outcomes, and high turnover in leadership and staff positions Evaluation research design: RCT for targeted intervention and quasi-experimental design for whole-school intervention Background MCS provided implementation leadership, READ 180 classes, literacy coaches, PD instructors, R & E staff and numerous inputs University of Memphis developed and delivered the four-semester teacher course and mentored coaches RBS collected data through surveys, daily coaching logs, individual and group interviews, classroom observations, program record review, and secondary analysis of testing data Partnership Roles Evaluator Buy-in among school personnel & respondents time Entre into the schools Responsive R & E district staff District Unobtrusive, sensitive evaluators who understand context and minimize respondent burden Timely, actionable formative information What both Parties Need from the Other Maintaining random selection compliance Persuading actors to participate in evaluation Bridging cultural divides Logistical issues related to data collection (e.g., arranging observations) Obtaining parental consent for Iowa Test of Basic Skills (ITBS) Evaluators Anticipated Challenges Translating evaluators needs to school personnel Administering and scheduling time for ITBS Ensuring continued school buy-in despite changes in leadership and district-wide objectives Balancing the needs of schools, IHE actors, and evaluators in context of competing demands Communicating up the chain of command how operations will look at the school level Districts Anticipated Challenges Consulted PI on all data collection scheduling Carefully-nurtured connections Three cups of tea Do no harm PI characteristics: 25 years teaching in district Included R & E district staff on monthly calls Possible Reasons for Strong Partnership Respected the skills/knowledge of district actors Relied on local evaluators where possible Exercised common courtesy Convened sessions focused on mapping program Ensured that evaluators and program staff had shared language Possible Reasons for Strong Partnership Model good etiquette for field staff Reinforce importance of sensitivity to actors on- the-ground Schedule monthly calls with any actor charged with collecting/sending data Invite stakeholders from all authority levels to working sessions that result in better articulating goals and objectives Recommendations for Evaluators Kelly Feighan at Research for Better Schools Rorie Harris at Memphis City Schools, Office of Research, Evaluation, and Assessment Contact: