District LeaDer’s GuiDe

ensurinG Data inteGrity anD transparency

Data infrastructure is an essential component of any

educator evaluation system.

© Copyright 2012 Colorado Legacy Foundation. All rights reserved.

1

2

3

impLementinG evaLuation systems: LearninG from pioneerinG Districts

In an effort to help districts implement local educator evaluation systems in line with the requirements of SB10-191, the Colorado Legacy Foundation is providing a suite of resources to district and school leaders. CLF worked with three Colorado districts who recently revised their evaluation process. Although these districts implemented their new systems prior to passage of sB191, the process that each district went through is informative; Highlighting these districts and the lessons they learned along the way will help other districts leverage their success and avoid re-living their most difficult challenges.

three case studies provide the foundation for this work. District and school leaders can read the case studies to learn how different districts have approached similar goals. A series of District Leader’s Guides builds on the case studies and provides more direct and specific guidance to district leaders as they move forward with implementation.

Brighton Case Study - Brighton educators and administration agreed that their evaluation system was a “dog and pony show.” In 2009 they revised their system, with terrific buy-in from the union, to more meaningfully support teachers.

Eagle Case Study - Eagle has spent nearly a decade developing their evaluation system and aligning it to instruction, assessment, and professional development.

Harrison Case Study - In 2007, Harrison hired a new superintendent who instituted a new evaluation system along with rigorous instructional supports, interim assessments, and a pay-for-performance system.

District LeaDer’s GuiDe

Ensuring Data Integrity and Transparency

© Copyright 2012 Colorado Legacy Foundation. All rights reserved.

Who should use this Guide?

This guide is developed for district leaders who:

• Havereadpreparing your District to implement a new educator evaluation system, communicating effectively with stakeholders, selecting and using multiple measures to evaluate educators, Determining Levels of performance, and training and selecting evaluators;

• Haveestablishedadistrictevaluationimplementationteamwhichislookingforexamples,lessonslearnedandimplementationtipsfromColoradoschooldistricts;

• Arepreparedtoexplore,withthedistrictimplementationteamandtechnologyprofessionalsinthedistrict,thedatainfrastructureneededtoeffectivelyevaluateeducatorsandprovidethemwithtargetedsupport. Ensuring Data Integrity and Transparency

© Copyright 2012 Colorado Legacy Foundation. All rights reserved.

introduction

Datainfrastructureisanessentialcomponentofanyeducatorevaluationsystem.Districtswillneedtohaveasysteminplacethatallowsthecomponentsofaneffectivenessrating-studentgrowthmeasuresandeducators’professionalpracticescores-tobestored,analyzed,andcombined.Beforeteacherperformancedatacanbeanalyzedandtheresultssharedwithstakeholders,districtsshouldthoughtfullydecideuponproceduresfordatacollection,validation,andwarehousing;thesedecisionsarethereforenecessarytoinformstakeholders,guideprofessionallearning,andassessthemeasuresusedandtheevaluationsystemasawhole.Furthermore,thedataelementsselectedandused,andthedecisionsaboutdatainfrastructure,shouldbeguidedbytheevaluationgoals.i

in thinkinG aBout Data manaGement, there are severaL important consiDerations:

1. First,measuresthatassessstudentperformancemustbereliable and valid. Reliabilityreferstotheconsistencyofthemeasure/instrument;inotherwords,itprovidesanestimateofhowmuchorhowlittleresultswouldvaryacrossmultipleadministrationsofaninstrument.Validityreferstotheextenttowhichaninstrumentismeasuringwhatitissupposedtomeasure.

2. Verifyingand“scrubbing”existingdataisimportanttoensurethatdataisasaccurateaspossible.Teachersneedtohaveconfidencethatstudentdatabeingattributedtothemisanaccuraterepresentationofwhotheytaught.Thiswillfacilitateunderstanding in teachers and principals on what data is beingusedandhow.

3. Measuresthatproviderealtimefeedback,areaccessibleandeasilyunderstood,andinformteacherpracticearemorelikelytohaveanimmediateimpactonteachingandlearning.ii

4. Datashouldalsobetimelyandeasytouse.Forexample,districtsusingbenchmarkorcommonassessmentstoguidestudent interventions need to share results with teachers in a timelymannersoteacherscandifferentiateinstruction,makedecisionsaboutstudentgrouping,orre-teachkeyconcepts.Theeasiestandmosteffectivewaytoensurethatteachersmeaningfullyengagewithachievementdataistoprovideindividualteacherswithaccesstotheirstudents’datausingapasswordprotectedsystem.

© Copyright 2012 Colorado Legacy Foundation. All rights reserved.

state Board of education rules and cde Guidance

UnderSB10-191,CDEisresponsibleformonitoringtheimplementationoflocalevaluationsystemstoensuredistrictsareusingmultiple,fair,transparent,timely,rigorousandvalidmethods.Itisequallyimportantthateducatorsreceiveadequatefeedbackandprofessionaldevelopmentsupporttoprovidethemameaningfulopportunitytoimprovetheireffectiveness.WhiletherulesthatgovernSB10-191arenotexplicitaboutdistrictrequirementsfordatainfrastructures,thereportingrequirementsfordistrictsshedlightonwhatmaybeneeded(see section 6.04).

Therulesproposehoweachofthefollowingobjectivesmaybeevaluatedandthedatathatmaybeused.Therules for sB10-191statethat,ataminimum,monitoringeffortsshallfocuson:

1. Increasingtheeffectivenessofalleducators.

2. Analyzingthecorrelationbetweenstudentperformanceoutcomesandtheassignmentofeducatorstoperformanceevaluationratings.

3. Analyzingtheequitabledistributionofeffectiveandhighlyeffectiveeducators.

4. Analyzingtheextenttowhichprincipalsandteachersunderstandhowtheyarebeingevaluated,whattheyneedtodotoimprove,andhowtoaccessresourcestheyneedtosupporttheirprofessionaldevelopment.

Therulesalsoproposehoweachoftheaboveobjectivesmaybeevaluatedandthedatathatmaybeused.MoreinformationfromCDEonhowdistrictswilldemonstratecompliancewiththelawwillbeavailablepriortoJuly2013,thedatebywhichdistrictswillneedtodecideiftheyareusingtheirownsystemoradoptingthestatemodelsystem.

It is equally important that educators

receive adequate feedback and

professional development support to

provide them a meaningful opportunity to

improve their effectiveness.

© Copyright 2012 Colorado Legacy Foundation. All rights reserved.

GuidinG Questions

Questions for the District impLementation team are:

• WhatinformationdoesCDEprovidethroughSchoolViewandotherdatabasesthatcanbeusedininformingthedesignoftheevaluationsystemandreportingresults? a. Howdoesthisdataanswerthecriticalquestionsandservethepurposesoftheevaluationsystem? b. Whatadditionaldatamightbeneededtomeettheneedsofthedistrict?

• Inwhatwaysmightthedistrict’sinformationtechnologypersonnelbebroughtintothediscussion?

• Whatarethedistrict’sstrengthsandweaknesseswithregardtotechnologyandhumancapacityneededtocollectandreportdataaccurately? a. Whataresomepotentialwaysthedistrictmightbuildcapacityregardingdatacollection, analysis,andreporting? b. Whattrainingwilldistrictpersonnelneedtoensureaccuratedatacollectionandreporting? c. Whatarebarrierstobuildingcapacitywithinthedistrict?

• Doesthedistricthaveadatainfrastructurethatiseasilyaccessibleandused?

• Canthedatainfrastructurecombinedifferentdatabasessuchasschool-,teacher-,andstudent-leveldemographicinformation;studentachievementmeasures;humanresources/personnelrecords;andfinancialexpenditures?

• Howwillthedistrictlinkeducatorevaluationdataandprofessionallearningneeds?

• Howwilleducatoreffectivenessresultsbesharedwithkeystakeholders?

• Howwillresultsbemadeavailableforindividuallearningplansaswellasschoolimprovementplans?

learninG from PioneerinG districts

AlthoughBrighton,EagleandHarrisonundertookreformoftheirevaluationsystemspriortothepassageofSB191,theprocessbywhichtheydevelopedandimplementedtheirsystemsprovidesmanylessonsandsuggestsseveralimplementationtips.

Thefollowingsectionsprovideexamplesandlessonslearnedfromthedistrictcasestudies.

Formoredetailedinformationabouttheselessons,districtevaluationteamsareencouragedtoreadthecase studies and use the case study highlightstofurtherinformtheirdiscussions.

© Copyright 2012 Colorado Legacy Foundation. All rights reserved.

District leaders, principals, and teachers have easy access to student achievement data reported by any number of

subgroups and grade levels. For example, student-, classroom-, and school-level reports are available, and can be

disaggregated by student demographics (like ELL status, gender, and IEP status), year-to-year cohorts, or specific

academic standards.

The commercial data warehouse used by ECS is designed to report Colorado CSAP data and other state reporting

requirements. Summative CSAP reports provide growth information grouped by a teacher-student link. The district’s

NWEA data is also available. Educators have access to student reports on specific standards, test administrations,

longitudinal analyses, and other queries.

Student reports can be generated that provide data to develop response to intervention (RTI) plans, individualized

learning plans, and literacy plans. This process is streamlined and very accessible to teachers. As many as ten

templates are available to report data aligned to specific learning plans and teachers only need to enter a student ID

number and data are automatically pulled into the student report for each teacher’s use.

In the 2011-12 school year, the district has begun using a new data infrastructure that includes a data dashboard

function. Instead of logging into multiple databases to access information, the new system will integrate data from

multiple sources on a single data dashboard.

Also during school year 2011-12, rigorous

curriculum design (curriculum units that align

to the new state standards, instruction, and

assessment) is being implemented. Common

formative assessments are available for each

unit of study which allows educators to assess

student progress on priority standards during

the school year. Formative assessment results

are quickly available, encouraging teachers to

modify or supplement instruction every four

to eight weeks based on student assessment

results.

To the left is an example of the data dashboard

that teachers can access at any time

throughout the school year.

creatinG anD usinG a Data infrastructure for improvement—eaGLe county schooLs (ecs)

© Copyright 2012 Colorado Legacy Foundation. All rights reserved.

Pioneeringdistrictssuggestedthefollowingimplementationtips:

imPlementation tiPs

• Educators must be trained in the use of the data system. Provide one district training tailored to people’srolesandthedatatheywouldneedtouse,ratherthantrainingeveryoneonthewholedatasystem.Trainingisneededinorderforeducatorstounderstand how to access data, understand what it means,andhowtousedatatoimproveinstruction.

• Districts can consider developing standard data displays, also known as “dashboards.” Decisions aboutwhattoincludeinthesedatadisplaysshouldbeinformedbythedatamostfrequentlyrequestedbyprincipals,teachers,anddistrictleaders.

• Data systems are most useful when designed to be queried.Flexibledatasystemsallowuserstoruntheirownanalyses,andpersonalizetheirinquiryinwaysthataremostusefultothedecisionstheymustmakeintheirpracticeandprofessionaldevelopment.

• Teachers should have 24/7 access to their students’ data and their own evaluation data.Timelyaccesssolvestwoissues:(a)teachers’abilitytomakefastandinformeddecisionsabouttheirstudents’instructionand(b)teachers’adjustmentstotheirpracticeandinstructionalstrategiesbasedonfeedback.

• Specific software programs may need to be developed to weight data and calculate scores.Programscanalsobedevelopedtomonitorinter-raterreliabilityofevaluators.

• Data must be easily retrievable, especially by teachers and principals.

• Educators who know how to use data for instructional improvement will make better use of the data system and have better results.

• District leaders should analyze school data and make decisions on how to better support schools, their leaders, and their teachers.Datathatareanalyzedquicklyallowfortargetedlearningforteacherswithspecificneeds.

Datainfrastructuresarecriticaltostoringdataandprovidingtheflexibilitytogeneratereportsthatanswerthelargenumberofquestionsfacingdistrictsregardingaccountability,policy,andpractice.Datamustbeaccurate,transparent,andeasilyunderstood.Trainingeducatorstousethedatasystemisessentialandshouldleadtodata-informeddecision-making,includingtargetedlearningforindividualstudentsandprofessionallearningforschoolstaff.

conclusion

© Copyright 2012 Colorado Legacy Foundation. All rights reserved.

Author:JaneArmstrong,JMArmstrongandAssociates

Research Support and Editing:HeatherChikoore,ColoradoLegacyFoundation

Research Support for the Case Studies:UlccaHansen,ColoradoLegacyFoundationandKristenDavidson,UniversityofColoradoatBoulder

Centralofficestaff,schoolboardmembers,principals,teachers,parentsandcommunitymembersparticipatedininterviewsthatinformedthedevelopmentofthisguide.Brighton,EagleandHarrisonSchoolDistrictsopenedtheirdoorstoresearcherstodescribewhattheyweredoing,howtheyweredoingit,thechallengestheyfacedandlessonstheylearned.Intervieweeswerebothgraciousandcandidintheirinterviews.Withoutthem,thisknowledgecouldnotbecapturedandshared.

i LauraGoe,LynnHoldheldeandTriciaMiller.A Practical Guide to Designing Comprehensive Teacher Evaluation Systems. (Washington,DC:NationalComprehensiveCenterforTeacherQuality).May2011.

ii LauraGoe,LynnHoldheldeandTriciaMiller.A Practical Guide to Designing Comprehensive Teacher Evaluation Systems.(Washington,DC:NationalComprehensiveCenterforTeacherQuality).May2011.

acknoWledGements

© Copyright 2012 Colorado Legacy Foundation. All rights reserved.