Evaluation Team M.A. Henry Consulting, LLC Co-Lead Marty Henry, EdD, CEP EvaluatorsKeith S. Murray, CEP Quantitative Mark Hogrebe, PhD Analyst Qualitative AnalystKatherine A. Phillips, PhD ContentBob Coulter, PhD AdvisorsCharles R. Granger, PhD Improving Teacher Quality Program Theory of Change The Cycle 12 Projects ITQG Cycle 12 Projects Early Elementary Environmental Education: Field Based Approach Missouri State University, West Plains (4E) (Year 3 of 3) Alice (Jill) Black, Project Director Transforming Mathematics Instruction Using Inquiry and One-to- One Environments (TRIM I+121) Missouri State University, Springfield (Year 3 of 3) Lynda Plymate, Project Director Jan Van-Gilder, Project Director Diana Piccolo, Key Project Leader ITQG Cycle 12 Projects Science Education and Quantitative Literacy: An Inquiry-based Approach Missouri University of Science & Technology, Rolla (SEQL) (Year 1 of 3) V.A. Samaranayake, Co-Director Oran Allan Pringle, Co-Director David Westenberg, Co-Director Ronald Bieniek, Co-Director Evalee Lasater, Co-Director Janna Neiss, Co-Director Southeast Missouri State University: Making Mathematicians: Learning to Think and Apply, Cape Girardeau (MM) (Year 2 of 3) Rita Fisher, Project Director Linda Null, Co-Director Julie Antill, Co-Director ITQG Cycle 12 Projects University of Missouri: Preparing Elementary Mathematics Specialists to Serve Missouri Schools (PrepEMS) (Year 1 of 1) Barbara Reys, Co-Director Ann McCoy, Co-Director Location of Projects in ITQG Cycle 12 Location of Project Schools in ITQG Cycle 12 Location of Teachers Schools in ITQG Cycle 12 Location of Students in ITQG Cycle 12 All Full-Time Participants in Cycle 12 were LEA Educators Teachers Educational Attainment in ITQG Cycle 12 (reported n=159) Percent of Teachers by Grade Levels in ITQG Cycle 12 (reported n=327) Grade-Level Distribution by Project in ITQG Cycle 12 Federal Funding for Missouris ITQG Program Across Cycles Mean Cost per Teacher by Cycle Selected Characteristics of High-Quality PD Cycle 12 Areas of Strength Directly linked to improved standards-based student learning Developed with teacher Provides time/resources for learning, practice, follow-up Grade-level collaboration and work Content-area collaboration and work Specialization-area collaboration and work Action research and sharing of findings Ongoing participation by school and district leadership Strategies for students of limited English proficiency Methods of teaching children with special needs Provides teachers with the opportunity to give the district feedback on the projects Selected Characteristics of High-Quality PD Cycle 12 Areas of Challenge What the ITQG Program Aims to Accomplish through Its Projects Objectives for Teachers Increase teachers knowledge in targeted mathematics and/or science content. Improve teachers pedagogical knowledge and practices in inquiry-based instruction. Improve teachers knowledge and skills in designing, implementing and using of assessment instruments and data. Sample Content Test Item A study reported that in a random sampling of 100 women under the age of 35 showed that 8 of the women were married 2 or more times. Based on the study results, how many women in a group of 5,000 women under the age of 35 would likely be married 2 or more times? Show your work or explain your thinking. Sample Content Test Item Based on the data here, what can you conclude about women over 45 years old? Sample Content Test Item Sample Environmental Education Content Test Item Jose lives in Silver, Monica lives in Lincoln, and Blake lives in Nobel. They want to meet in a city on the map. No one wants to travel for more the 15 minutes. 1. Where could they meet? 2. Explain your answer. Cycle 12 Impact on Teachers: Content Teacher Content Test Results (N=137) P = 0.06 No statistically significant increase pre to post Cycle 12 Impact on Teachers: Practice Classroom Observations Focus Teachers for Case Study 2+ Teachers per Project Lessons Observed Two Times Each Use of Inside the Classroom Observation and Analytic Protocol, Horizon Research, Inc., 2002) Ratings: 1=ineffective instruction, 2=elements of effective instruction, 3=beginning stages of effective instruction, 4=accomplished, effective instruction, 5=exemplary instruction. ITQG Cycle 12 Impact on Teachers Practice Ratings of 1 st & 2 nd Classroom Observations: Design & Implementation (N=22 observations among 11 teachers) ITQG Cycle 12 Impact on Teachers Practice Ratings of 1 st & 2 nd Classroom Observations: Content & Classroom Culture (N=22 observations among 11 teachers) Ratings: 1=ineffective instruction, 2=elements of effective instruction, 3=beginning stages of effective instruction, 4=accomplished, effective instruction, 5=exemplary instruction. ITQG Cycle 12 Classroom Observation Capsule Ratings Using a 7-Point Scale (N=22 observations among 11 teachers) Self-Reported Obstacles to Cycle 12 Teachers Teaching Performance (n=98, 107 items reported) Self-Reported Benefits & Improvements to Cycle 12 Teachers Teaching Performance (n=98, 121 items reported) Knowledge and Skills in Designing and Using Assessment Instruments and Data Many variations of implementation One day focus, integrated throughout programming, use of local district data to inform instruction Designed to meet needs of local teachers in each project What the ITQG Program Aims to Accomplish through Its Projects Objectives for Students Improve student achievement in targeted mathematics and/or science content areas. Cycle 12 Impact on Students: Results Change in analysis due to lack of DESE student state data Examination of schools with treatment teachers vs. comparison schools a. Propensity Score Matching b. Regression Models All grades Science and Math Content c. Compared % Achieving Proficient and Advanced Cycle 12 Impact on Students: Results Regression Analysis Cycle 12 Impact on Students: Results Impact on ITQG Schools Examination of ITQG-only schools for a. Effects of # of Students with ITQG Teachers b. Effects of # of Teachers with Masters Degrees c. Effects of multiple years of ITQG Participation Cycle 12 Impact on Students: Results # of Students with ITQG Teachers Schools with greater % of students who had ITQG Teachers had a higher % of students scoring Proficient/Advanced on MAP tests. r = 0.24, p < 0.05 for Math r = 0.36, p < 0.01 for Science Cycle 12 Impact on Students: Results Percent of Teachers in ITQG School with Masters Degrees Schools with greater % of students who had Teachers with Masters Degrees had a higher % of students scoring Proficient/Advanced on MAP tests r = 0.34, p < 0.01 for Math r = 0.26, p < 0.05 for Science Cycle 12 Impact on Students: Results Number of Years Teachers Have Been in ITQG Schools with greater % of teachers with multiple years in an ITQG program had a higher % of students scoring Proficient/Advanced on science MAP tests. r = 0.32, p < 0.05 for Science No relationship for Math What the ITQG Program Aims to Accomplish through Its Projects Objectives for Educational Institutions Improve the preparation of pre-service teachers through improvements in mathematics/science content and/or pedagogy courses. AND Sustain the successful efforts of the ITQG program. Cycle 12 Impact on Educational Institutions Effects on university pre-service programs Change in preservice ed courses Change in some content courses Preservice effects vary across institutions from none to moderate Administration supportive mostly disengaged with direct project activities Effects on schools and school districts is teacher- dependent and will diminish if/when teachers leave Cycle 12 Case Study Aims: To Investigate Effects or effectiveness of ITQG participation by multiple teachers in isolated rural schools Degree to which ITQG participation affected collaborative learning, school curriculum, teacher performance Role of administrative support in ITQG impact Motivations and risk factors involved for participation Cycle 12 Case Study Methods Mixed-method, intensive study of two rural ITQG schools with multiple teachers in multiple projects through more than one cycle Seven teachers Three projects represented Data from surveys, content tests, observation results, interviews with school faculty and staff, extant school and community sources Cycle 12 Case Study Does participation in an ITQG project by multiple teachers in small, rural schools serve to intensify positive impact? Yes With multiple ITQG teachers, schools experienced Enhanced teacher collaboration and group planning Greater cross-grade integration of content Discussion of STEM content and sharing of activities School curriculum effects related to adoption of ITQG components (where teachers responsible for school curriculum planning) Does participation in an ITQG project by multiple teachers in small, rural schools serve to intensify positive impact? But Effects wholly teacher-determined and activated Decision to participate affected by PD distance, stipends Lack of administrator engagement in ITGQ challenged sustained incorporation of improvements Lack of administrator engagement in ITQG had directly negative consequences for teachers, in their view Institutionalization impossible without institutional intent Cycle 12 Case Study Cycle 12 Institutional Impact Sustainability promising results for local district sustainability through teacher networks Projects teams are strong Will sustain effects if they remain engaged with teachers and schools Much depends on integration with school and district planning and administrative connectedness Sample Pre/Post SNA Potential for Sustaining Support Network at Start of First Cycle of Funding (Project 1) Support Network At End of Three Cycles of Funding (Project 2) Circles = Project Teachers, Orange Circles = former or other project teachers, Squares = Administrators; Triangles = PIs; Down Triangles = Other teachers; Diamonds = Relatives of Participants; + = Others, Yellow Figure = Internet Question: To what extent does school/district administration participation/partnership in ITQG control or mediate achievement of program and project objectives? Feedback to MDHE You have completed at least one year of your project. What feedback or questions do you have for DHE or evaluators that would help you do your work better? Feedback from Projects Consider administering external evaluation posttest at next to last follow up session; follow up with absentees; projects will assist in follow-up Clarify external eval needs regarding student internal test data (one copy of each test instrument with grade level, teacher, total test is worth, students coded with pre/post scores) Have teachers send student data to projects as soon as they give the posttest Contact Marty Henry Keith Murray St. Louis Office Phone: Fairfax Office Phone: