Evaluation Systems: Context, Issues, and Perspectives

  • Upload
    maren

  • View
    40

  • Download
    0

Embed Size (px)

DESCRIPTION

Evaluation Systems: Context, Issues, and Perspectives. Charles Igel Senior Researcher, McREL Brian Ewert Superintendent, Englewood Public Schools Barbara Lunsford Associate Superintendent, Georgia DOE - PowerPoint PPT Presentation

Citation preview

  • Evaluation Systems: Context, Issues, and Perspectives

    Charles Igel Senior Researcher, McRELBrian Ewert Superintendent, Englewood Public SchoolsBarbara Lunsford Associate Superintendent, Georgia DOEKerry Englert President, Seneca ConsultingJessica Allen Researcher, McRELTony Davis Senior Director, McREL

  • Framework for Evaluation SystemsTeacher EffectivenessQuality StandardsLeadershipEnvironmentContentLearningPracticeGrowth50%50%Performance Ratings DevelopingProficientAccomplishedDistinguishedProfessional Practice

  • Examples of large-scale evaluation initiatives

  • Considerations for developing and rolling-out educator evaluation systems

  • Colorado SCEE Example Human Judgment: Data should inform decisions, but human judgment will always be an essential component of evaluations

    Continuous Improvement: The implementation and assessment of the evaluation system must embody continuous improvement

    Feedback: The purpose of the system is to provide meaningful and credible feedback that improves performance

    Alignment: Educator evaluation must take place within a larger system that is aligned and supportive

    State Council for Educator Effectiveness. (2011, May). State Council for Educator Effectiveness report and recommendations. Denver, CO. Available at www.cde.state.co.us/EducatorEffectiveness

  • Additional considerations: Educational Leadership Policy Standards (ISLLC 2008)

    S1: Facilitating the development and implementation of a shared vision for learningS2: Nurturing a culture conducive to student learning and professional growthS3: Managing for a safe, efficient, and effective learning environmentS4: Collaborating with and mobilizing faculty and community resourcesS5: Acting in a fair and ethical mannerS6: Responding to and influencing the political, legal, and cultural environment

  • Additional considerations: InTASC Model Core Teaching Standards

    S1: Uses of developmentally appropriate instructionS2: Establishes inclusive learning environmentsS3: Promotes positive peer interactions and engagement with materialS4: Understands content and makes it meaningful and accessible to all learnersS5: Connects content in ways that promote creativity and critical thinkingS6: Uses multiple methods of assessmentS7: Uses all professional knowledge in lesson planningS8: Uses multiple instructional strategies S9: Active in own professional growthS10: Take appropriate leadership positions and actively engages in the profession

  • Additional considerations: Educational Data Exchange Network (EDEN) Reporting

    What is reported?1) Teacher counts categorized by performance level 2) Total number of teacher evaluated3) Principal counts categorized by performance level4) Total number of principals evaluated

    Guidance on 1) teachers and/or principals assigned to multiple schools in the same LEA2) teachers and/or principals assigned to multiple schools in different LEAs

  • Charles Igel, PhDMid-continent Research for Education & Learning (McREL)[email protected]

  • THE Sinking of the Titanic

  • Purpose of EvaluationServes as an accountability and compliance systemServes as a tool to help in employment decisionsProvides a set of rigorous research-based standardsServes as a measurement of performance and effectivenessPromotes leadershipServes as a personal, reflective tool to become more effectiveServes as the basis of instructional improvement (and school improvement)Enhances the implementation of the approved curriculumGuides/drives continuous improvement through professional developmentServes as as a tool in a coaching/mentoring modelServes as a tool to determine performance pay (in some places)Informs higher education & research

    Brian Ewert -

    Michael Porter - and research

  • Englewood Schools: ICEBERG DEAD AHEAD!Lack of Expectations & NonnegotiablesLack of AccountabilityLack of Transparency Lack of QualityLack of Fidelity Lack of ConsistencyLack of Reliability & ValidityLack of Integrity

    Michael Porter - years in between eval. Michael Porter - no follow up on a systems levelMichael Porter - Englewood's Iceberg...

    15 seconds of Englewood context (turnaround, leadership changes, declining enrollement)

  • ICEBERG DEAD AHEAD!Lack of Reflective PracticeDeficit/Punitive Model vs. Continuous Improvement ModelLack of Performance Pathways for GrowthLack of Policies, Procedures, and Agreed Upon PracticesLack of a Framework to Drive ConversationLack of Goal Setting and Professional Growth

  • The LifeboatTeacher/Principal Effectiveness BillCreates a four-tiered system to evaluate the effectiveness of licensed personnel as a means of improving the quality of education in Colorado. The basic purposes of the statewide system are:To ensure that all licensed personnel are evaluated using multiple, fair, transparent, timely, rigorous, and valid methods, fifty percent of which evaluation is determined by the academic growth of their studentsTo ensure that all licensed personnel receive adequate feedback and professional development support to provide them a meaningful opportunity to improve their effectivenessTo ensure that all licensed personnel are provided the means to share effective practices with other educators throughout the state

  • The LifeboatSolidarity & Alignment: Board, Administrators, Principals, & UnionNon-negotiables and ExpectationsAccountability: 100% ParticipationCoaching/Collaborative/Reflective ModelClearly Defined Framework with Common LanguageLow Stakes Implementation with: Quality, Fidelity, Intensity, Consistency

  • The LIfeboat21st Century Instructional ModelState and Common Core StandardsData TeamsTied to School & District Improvement PlansPerformance Pathways for GrowthAligned, Collaborative Goal SettingProfessional Development Based on Data

    Michael Porter - No "got you's"Michael Porter - based on needs data, both observed and self-reported.

  • RMS Carpathia &Pier 54-NYCMuch Work To Do!Robust, Reliable, Intuitive, Flexible, & Easy-to-Use Technology & Web ToolsSeamless Integration with Other Initiatives8:1 RatioPeer Coaching and EvaluationQuality-Fidelity-Intensity-Consistency-Reliability-Validity

  • RMS Carpathia &Pier 54-NYCFormative and Summative Observation & Evaluation DataDeep, Meaningful Conversations around Teacher Standards, Elements, Rubrics, & RatingsReal Time, Needs-Based Professional DevelopmentValue-Added, Student Growth MeasuresValid, Reliable,Vetted Metrics & AlgorithmsGroup/Collective Performance Pay Employment Decisions

    Michael Porter - valid repeated twiceMichael Porter - End with an anecdote about the quality of the conversations, teachers making independent connections between indicator language and SIP's, bring in back to a snap shot of what it sounds like when it;s working well.

    Dr. John D. Barge, State School SuperintendentMaking Education Work for All Georgianswww.gadoe.org

    Georgia Department of Education6/19/2011*CLASS KeysClassroom Analysis of State Standards A Teacher Evaluation Process Standards

    Barbara LunsfordAssociate Superintendent School Improvement

    Dr. John D. Barge, State School SuperintendentMaking Education Work for All Georgianswww.gadoe.org

    CLASS KeysSM Strands and Elements6/19/2011*

    Dr. John D. Barge, State School SuperintendentMaking Education Work for All Georgianswww.gadoe.org

    ASSESSMENT - The collecting and analyzing of student performance data to identify patterns of achievement and underachievement in order to design and implement appropriate instructional interventions.STRANDSTANDARDELEMENTCONTINUUM OF IMPROVEMENT RUBRIC6/19/2011*

    Dr. John D. Barge, State School SuperintendentMaking Education Work for All Georgianswww.gadoe.org

    6/19/2011*

    Not Evident Emerging Proficient Exemplary

    Dr. John D. Barge, State School SuperintendentMaking Education Work for All Georgianswww.gadoe.org

    6/19/2011*

    Dr. John D. Barge, State School SuperintendentMaking Education Work for All Georgianswww.gadoe.org

    Performance on the elements of the CLASS KeysSM is identified on a four-level continuum.This continuum is not utilized to label teachers as Not Evident, Emerging, Proficient, or Exemplary.

    The continuum is used to describe a teachers PERFORMANCE on specific elements.6/19/2011*

    Not EvidentEmergingProficientExemplary

    Dr. John D. Barge, State School SuperintendentMaking Education Work for All Georgianswww.gadoe.org

    GTDRPerformance An array of evidence is collected from multiple sources during the year.AnnualEvaluationAnnounced,Longer ObservationsUnannounced, Short ObservationsOther artifacts and evidence from conferences, meetings, review of teacher and student productsStudent Achievement Data6/19/2011*

    Dr. John D. Barge, State School SuperintendentMaking Education Work for All Georgianswww.gadoe.org

    Module 1: Content and Structure (The What?)Module 2: Overview of the Evaluation Process (The How?)Module 3: Self-Assessment and ReflectionModule 4: Professional Growth PlanModule 5: Pre-Evaluation ConferenceModule 6: Informal ObservationsModule 7: Formal ObservationsModule 8: Georgia Teacher Duties and ResponsibilitiesModule 9: Annual EvaluationModule 10: Professional Development PlanCLASS KeysSM Modules6/19/2011*

    Dr. John D. Barge, State School SuperintendentMaking Education Work for All Georgianswww.gadoe.org

    Evaluator Training AssessmentThree Sections: Scoring the Elements in a Classroom Video Annual Evaluation Performance Task Technical Questions Regarding Process6/19/2011*

    Dr. John D. Barge, State School SuperintendentMaking Education Work for All Georgianswww.gadoe.org

    6/19/2011*

  • KERRY ENGLERT, PH.D.SENECA CONSULTING

    JESSICA ALLEN, MAMcREL

    COUNCIL OF CHIEF STATE SCHOOL OFFICERSNATIONAL CONFERENCE ON STUDENT ASSESSMENTJUNE 19TH, 2011Validity in a Standards-Based Teacher Evaluation System

  • Presentation PurposesOverview of our experiences conducting an initial validity study of a states standards-based teacher evaluation system.Propose a framework that can be used when designing these validation studies.Offer suggestions and considerations for validation studies of educator evaluation systems.

  • National Issues in Teacher EffectivenessEducation Research Connection between teacher effectiveness and student learning that is well documented (ex. Goe (2007) ).Education Policy Focusing on teacher evaluation and the relationship between teacher effectiveness and student achievement.Race to the Top White Houses Blue Print for Reform

  • Standards-Based Evaluation SystemThree core components:A set of standards.Procedures for collecting performance data on the standards.Rubrics to evaluate performance.Optional fourth component:Methodology for incorporating evaluation results into personnel decisions.

    * from (Odden, 2004):

  • Why do a validity study?Race to the TopRequires a valid and reliable teacher evaluation instruments.Validity evidence Builds credibility to the instrument and evaluation system.Provides information on data use and implementationProvides information on continual development.Can focus training efforts.

  • CLASS KeysSM Teacher Evaluation SystemPurposesThat serves a two-fold purpose improvement and accountability

    Core ComponentsFive major strandsTraining materials28 performance rubrics.

  • Validity Study of CLASS KeysSM SystemStudy ContextQuick turn around timeExtant data from pilot studyReadily available and accessible information Broad examination of the literature Inform, focus, and guide the study Better capitalize on the available data

  • Relevant Literature Validity Theory and StandardsA unified theory and system uses (Standards for Educational and Psychological Testing, 1999; Messick, 1989)A strong theory of action (Kane, 2006) Personnel EvaluationApplying the concepts of fairness, transparency, efficiency and practicality to the personnel evaluation context (Gullickson, 2008)Quality Rubrics Characteristics of quality rubrics (Arter & McTighe, 2001) Implementation FidelityThe degree to which the system is being implemented according to the specification (Ruiz-Primo, 2006)

  • Potential framework for validity in Educator EvaluationValidity in Educator Evaluation

  • Methods related to the frameworkValidity Theory and Rubric Quality ContentValidity Theory ConstructPersonnel Evaluation Standards and Validity TheoryFairness and ReliabilityImplementation Fidelity and Validity TheoryUse and Interpretation

  • Future ConsiderationsValidation of educator evaluation systems might benefit from multiple perspectives. Pilot and field tests are opportunities not only to test the instrument but apply validity methodologies and instruments.Validation is an ongoing process and provides a framework to understand how the instrument is being used.

  • ReferencesAmerican Psychological Association, American Educational Research Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological tests and manuals. Washington, D.C.: American Psychological Association. Arter, J., & McTighe, J. (2001). Scoring rubrics in the classroom: Using performance criteria for assessing and improving student performance. Thousand Oaks, CA: Corwin. Goe, L. (2007). The link between teacher quality and student outcomes: A research synthesis. National Comprehensive Center for Teacher Quality. Retrieved from http://www.tqsource.org/publications/LinkBetweenTQandStudentOutcomes.pdfGullickson, A. R. (2008). The Personnel Evaluation Standards: How to Assess Systems for Evaluating Educators, 2nd edition. Joint Committee on the Standards for Educational Evaluation. Corwin Press, Thousand Oakes, CA.Kane, M. T. (2006). Validity. In R. L. Brennen (Ed.), Educational Measurement (Vol. 4th edition). Westport, CT: Praeger Publishers.Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational Measurement (3rd ed., pp. 13-103). New York: American Council on Education & McMillian. Odden, A. (2004). Lessons learned about standards-based teacher evaluation systems. Peabody Journal of Education, 79(4), 126-137. Ruiz-Primo, M. A. (2006). A multi-method and multi-source approach for studying fidelity of implementation. National Center for Research on Evaluation, Standards, and Student Testing (CRESST).

    *Professional practice input standardsGrowth outcome standard**Structural way to think about develping a theory of action

    Premises: beliefs, assumptions, values, intended outcomes, what you are trying to achieve, foundation for a set of beliefs about educator evaluation, should drive all other decisions, establish key priorities, serve as a touchstone (evaluation is a process rather than an event, evaluation should drive professional growth, evaluation as a collaborative activity, student performance is a necessary but insufficient indicator of effective instruction)

    Purposes: premises operationalized, what are you looking to achieve

    Principles: operating principles, exemplify the premises and purposes of the system (e.g., transparency, collaborative, results oriented?)

    Policy: explicit expectations, timelines, roles & responsibilities state, district, school all need to be aligned (definitions of teacher effectiveness, teaching standards, inclusion of student performance measures, licensure, number of observations, peer observations?) (see SCEE p. 158)

    Process: explicit and implicit structures and operational systems (collection use and reporting of evaluation data, how to incorporate student performance measures, methods for revising system, how observational data will be collected)

    Practice: day-to-day way they systems are carried-out, impact on the ground (effect on other responsibilities, relationship between participants, mental habits, impact on leadership, impact on instruction)*Each standard begins with, An education leader promotes the success of every student by*New version release every month or so

    No requirement for performance level categories***.**************