R-Key Elements for Educational Accountability Models

8/7/2019 R-Key Elements for Educational Accountability Models

1/117

Counc il of Chief Sta te School Offic ers

Washing ton, DC

Key Elements for Educ a tiona l

Ac c ountab ility Models

Prepared by

Marianne Perie, Center for Assessment

withJud y Park, Utah

Kenneth Klau, Ma ssac husetts

A paper c ommissioned by the

Counc il of Chief Sta te Sc hool Offic ers

Ac c ountab ility Systems and Reporting Sta te

Collaborative

Dec ember 2007


2/117

COUNCIL OF CHIEF STATE SCHOOL OFFICERSTe Council o Chie State School Ocers (CCSSO) is a nonpartisan, nationwide, nonprotorganization o public ocials who head departments o elementary and secondary educa-tion in the states, the District o Columbia, the Department o Deense Education Activity,and ve U.S. extra-state jurisdictions. CCSSO provides leadership, advocacy, and techni-

cal assistance on major educational issues. Te Council seeks member consensus on majoreducational issues and expresses their views to civic and proessional organizations, ederalagencies, Congress, and the public.

STATE COLLABORATIVE ON ASSESSMENT AND STUDENT STANDARDSTe State Collaborative on Assessment and Student Standards (SCASS) Project was initi-ated in 1991 to encourage and assist States in working collaboratively on assessment designand development o high quality assessments in relation to standards or student learning.State education agencies voluntarily support collaborative work across a variety o topics andsubject areas. Te Accountability Systems and Reporting (ASR) state collaborative project has

been assisting States to develop and improve statewide systems since 2000. Currently 28 stateeducation agencies are participating members, together with researchers, consultants, andmeasurement specialists. Priority activities o the ASR Collaborative are: state planning anddevelopment to meet requirements or state systems under NCLB, sharing strategies and re-search-based evidence to improve the validity o accountability systems, developing reportingormats that improve inormation access or constituents, and establishing standards or theessential components o accountability systems. Funding support or this paper was entirelyrom member States o the Accountability Systems and Reporting State Collaborative (ASR-SCASS). For urther inormation about ASR SCASS and resources on accountability acrossthe 50 States, see the CCSSO web site: http://www.ccsso.org/projects/Accountabilty_Sys-tems/.

2007Council of Chief State School Officers

Elizabeth Burmaster (Wisconsin), President

Rick Melmer (South Dakota), President-Elect

Valerie Woodruff (Delaware), Past President

Gene Wilhoit, Executive Director

Rolf K. Blank, Director of Education Indicators

Council of Chief State School Officers

Attn: Publications

One Massachusetts Ave., NW, Suite 700

Washington, DC 20001202-336-7016

Fax: 202-408-8072

www.ccsso.org

ISBN 1-884037-24-0

Copyright 2007 by the Council of Chief State School Officers, Washington, DC.

All rights reserved.


3/117

Key Elements for Educational Accountability Models iii

Key Elements for EducationalAccountability Models

Prepared by

Marianne Perie, Center for Assessment

withJudy Park, Utah

Kenneth Klau, Massachusetts

A paper commissioned by the

Council of Chief State School Officers

Accountability Systems and Reporting State Collaborative

December 2007


4/117

iv Key Elements for Educational Accountability Models

Te authors wish to thank all members o the Accountability Systems and Reporting StateCollaborative or their assistance in conceiving and executing this paper, especially the ol-lowing:

Jacqueline Soychak, MaineRachelle ome, MainePat McCabe, Caliornia

Pat Roschewski, NebraskaKenna Seal, West Virginia

Mitchell Chester, OhioRobin aylor, Delaware

Margie Vandeven, MissouriJennier Stegman, Oklahoma

Kevin Hill, KentuckyCathy Wagner, Minnesota

Bob Bernstein, CaliorniaRol Blank, CCSSO

Andra Williams, CCSSO

We also wish to thank our expert reviewers or their advice and eedback, Ellen Forte, BillErpenbach, Arie van der Ploeg, Bob Linn, and Sue Rigney.


5/117

Key Elements for Educational Accountability Models v

Table of ContentsExecutive Summary ............................................................................................................ 1Chapter 1: Introduction and Background........................................................................... 3

Dening Accountability .......................................................................................... 3A Brie History o Accountability ............................................................................ 4

Overview o the Report ........................................................................................... 5Literature Review .................................................................................................... 6Synthesizing the Literature ...................................................................................... 7Dierences Between Federal and State Accountability Systems.............................. 12Summary .............................................................................................................. 13Reerences and Suggested Readings or this Chapter ............................................. 15

Chapter 2: Goals o Accountability Systems ..................................................................... 17Teory o Action ................................................................................................... 19Reerences and Suggested Readings or this Chapter ............................................. 24

Chapter 3: Perormance Indicators ................................................................................... 25Validity o the Indicators ....................................................................................... 28

Reerences and Suggested Readings or this Chapter ............................................. 29Chapter 4: Design Decisions ............................................................................................ 31

Combining Indicators to Make Accountability Decisions ..................................... 32Incorporating Subgroup Inormation .................................................................... 35Determining Issues o iming ............................................................................... 36Setting argets ...................................................................................................... 37Examining echnical Issues ................................................................................... 37Reerences and Suggested Readings or this Chapter ............................................. 39

Chapter 5: Consequences ................................................................................................. 41Rewards ................................................................................................................ 42

Sanctions .............................................................................................................. 43Reerences and Suggested Readings or this Chapter ............................................. 45Chapter 6: Communication ............................................................................................. 47

Communicating Results or Dual Accountability Systems ..................................... 54Reerences and Suggested Readings or this Chapter ............................................. 58

Chapter 7: Supports ......................................................................................................... 59Reerences and Suggested Readings or this Chapter ............................................. 67

Chapter 8: System Evaluation, Monitoring, and Improvement ......................................... 69Conclusion ........................................................................................................... 72Reerences and Suggested Readings or this Chapter ............................................. 73

AppendicesAppendix A: Annotated Bibliography ............................................................................... 75Appendix B: Matrix Synthesizing the Literature Review ................................................... 89Appendix C: Checklist ..................................................................................................... 95Appendix D: Specic State Examples ............................................................................. 103

List of Figures

Figure 1: A Simplied Teory o Action ........................................................................... 20


6/117

vi Key Elements for Educational Accountability Models


7/117

Key Elements for Educational Accountability Models 1

Executive Summary

Te decision to work toward this report was stimulated by leaders o state departments oeducation asking the question where can I go to nd key inormation about developingstate accountability systems? Currently, there is no one document that provides a set o

recommendations based on evidence or experience that covers all o the decisions involvedwith developing state accountability systems, and at the same time there is an extensive listo papers, articles, and other resources that should be read and considered. Te educationalmeasurement and evaluation eld has spent years developing its latest version oStandardsor Educational and Psychological esting(APA, AERA & NCME, 1999) and plans to spendseveral years developing the next version. Tis resource has become a common, much-usedresource to many in the testing industry to support inerences made rom tests or to critiquetesting practices. However, there is no parallel document or educational accountability sys-tems. Although several organizations and individual researchers have recommended practicesor accountability systems, we have not yet collaborated as a eld to produce a universallyaccepted set o standards.

Te purpose o this report is to summarize the work that has been done to date on develop-ing a set o standards or accountability and inorm those not amiliar or well experienced inaccountability about essential elements o a good/valid accountability system. In addition, wewanted to create a tool that states could use in developing a new accountability system or inevaluating a current one. Te audience is intended to be state or district policymakers whoare designing, redesigning, or reviewing their accountability systems. Tis report expandsaccountability beyond the ederal denition under NCLB. Tus the overarching goal o thisreport is to answer the question As states consider developing accountability plans separaterom NCLB or as an enhancement to NCLB, what are the elements, standards, etc. o agood accountability system?

o accomplish this goal, the Accountability Systems and Reporting State Collaborative (ASRSCASS), managed by the Council o Chie State School Ocers, developed a process orworking with state leaders on identiying the key elements or an accountability system, andASR selected the National Center or the Improvement o Educational Assessment to serve asconsultants and writers or the project, including work on three tasks:

Synthesize the literature to date on critical elements in an accountability systemDevelop a ramework incorporating core concepts, guiding questions, key ele-ments, and various options listed in previous literatureProvide descriptive examples o more ne-grained elements or options that in-

crease eectiveness and success o state-level accountability systemsAs the report was developed, three complementary documents were created. Ater reviewingthe immense amount o literature already published on accountability systems, both romeducation and business contexts, an annotated bibliography was developed summarizing over70 reports, articles, and chapters. A second product is a matrix developed to synthesize all othe reviewed literature. It was the rst step to developing our ramework o core concepts,guiding questions, key components, and options discussed in this paper. It is available in ullcolor with each color reerencing a specic article, chapter, or report. Te guiding questions

1.2.

3.


8/117

2 Key Elements for Educational Accountability Models

or key components are written in the authors own words to allow the reader to trace anyeature back to its original source.

Finally, the third product is a checklist or designing or evaluating an accountability system.Te checklist is organized by core concept, and each relevant section is presented as in intro-duction to a chapter, also organized by core concept. It was created to help state- and district-level policymakers evaluate the design o their accountability system. A checklist approachseemed most user-riendly as it highlights the considerations states should undertaking dur-ing either the development o a new accountability system or an evaluation o a current one.

Te checklist is organized under seven core concepts based on the various organizationalstrategies o leading researchers:

GoalsPerormance indicatorsDesign decisionsConsequences

CommunicationSupportSystem evaluation, monitoring, and improvement

Within each o these core concepts are a list o guiding questions and a set o key elementsthat provide guidance on developing a strong educational accountability system. Tis reportis organized around those seven questions, with the key elements described and explainedin each chapter. Examples rom state accountability systems also serve to provide rich detailregarding the various components.

1.2.3.4.

5.6.7.


9/117


Chapter 1: Introduction and Background

Educational accountability has been a much-used phrase since the 1970s. Te 1980s saw amovement to standards-based accountability. Te Improving Americas Schools Act (IASA)o 1994 moved the discussion to state-level educational accountability systems. Te onus

was initially put on state governments to dene their accountability systems. Te No ChildLet Behind Act o 2001 (NCLB) laid out a much more prescriptive accountability system,providing a specic ramework within which states must develop their accountability system.Because o this legislation, the eld as a whole has learned a great deal about education ac-countability. Much has been written about what works and what does not. Yet there still doesnot exist one comprehensive set o standards or key elements or developing eective educa-tional accountability systems.

Te educational measurement and evaluation eld has spent years developing its Standardsor Educational and Psychological esting(AERA, APA & NCME, 1999) and plans tospend several years developing the next version. Te esting Standards document has become

invaluable as the common reerence source or many decisions and practices in the testingindustry, and it is typically cited to support inerences made rom tests or to critique testingpractices. Likewise, resources exist that provide standards or student and program evaluation(Joint Committee on Standards or Education Evaluation, 2002 & 2004). In addition, theAmerican Evaluation Association has produced documents guiding evaluators and those us-ing high-stakes tests or evaluative purposes (AEA, 2002). However, there is no parallel docu-ment or educational accountability systems. Although several organizations and individualresearchers have created standards or accountability system, we have not yet collaboratedas a eld to produce a universally accepted set o standards.

Tis report was prepared by the ASR State Collaborative in response to requests rom state

department o education leaders ocused on the question where can I go to nd key in-ormation about developing state accountability systems? Te answer, it turned out, was alist o resources that were all must reads. Te present report is intended to synthesize andorganize the available inormation on accountability research and practices and assemble it allinto one document. Te key elements identied in the report are intended to clariy anddocument what must be present in an eective accountability system.

Defining Accountability

Te rst step or any book or article on education accountability is to dene accountability.However, dening accountability has become more complex as our understanding o ac-countability has grown. In the past, denitions ocused primarily on the interaction o goals,indicators, decision rules, and consequences. Although those components are still central toany accountability system, the latest movement has been to ocus more on capacity buildingand providing appropriate supports. Tat is, the purpose o accountability is not simply toidentiy and punish ineective schools, but to provide appropriate supports to ensure that allschools are eective.


10/117


o illustrate, consider the ollowing two denitions proposed in the 1970s and 1990s:

An accountability system is a closed loop refecting a chain o responses to perceivedneeds or demands; an activity or set o activities that emerges to ll those demands;outcomes that result rom those activities; and eedback on outcomes to the source othe demands. Te eedback may generate new demands or a regeneration o the oldones; in either case, the previous set o activities may be modied or remain intact; anew or altered set o activities may be modied or remain intact; a new or altered seto outcomes may be produced; and the loop is completed again with eedback to thesource o the demands (Levin, 1974, p. 375).

Accountability describes a relationship between two parties in which our conditionsapply: rst, one party expects the other to perorm a service or accomplish a goal;second, the party perorming the activity accepts the legitimacy o the others expec-tation; third, the party perorming the activity derives some benets rom the rela-tionship; and ourth, the party or whom the activity is perormed has some capacityto aect the others benets (Stecher and Hanser, 1992).

Both denitions imply a reciprocal relationship between results and positive or negative con-sequences. Accountability systems start with a set o goals and a theory o action that statesthat a specic act will produce a desired outcome. Tose actions are rewarded when success-ul; other actions that do not produce the desired outcome are sanctioned. Te system mustundergo constant monitoring to ensure that the action will produce the desired outcome,that the rewards and sanctions are eective, and that the eedback given to the various partiesprovides useul inormation on how to adapt their actions to produce the desired outcomes.

However, neither denition ocuses on the responsibility o the party who sets the goals toprovide the supports or meeting those goals. Accountability provides a mechanism or think-

ing globally about schooling, how we measure the eectiveness o our schools in promotinglearning, and how we provide the supports necessary to ensure our schools become evenmore eective. o that end, there has also been a greater recognition o the reciprocity o therelationship between state governance, district governance, and school governance in thateach has both expectations o and responsibilities to the others. Tere has been more workexploring the essential conditions (not just structures) that must be in place to eect changein low-perorming schools, such as time or teachers to collaborate, aligned curriculum, datasystems that provides useul data, and coaches or reading and math. Ultimately, account-ability has become more than a system to evaluate schools and allocate sanctions and rewards:it is a system that allows the public to understand how well their schools are working andto provide inormation to policymakers on the changes that are needed to make the schoolsmore eective and to continually improve all students educational opportunities.

Brief History of Accountability

Looking back over the history o accountability, the approach to designing accountabilitysystems has changed over time. Te eld o education has moved rom nancial account-ability and accountability based on inputs to standards-based accountability systems basedon outputs. Fuhrman (1999) ocuses on how the goals o accountability models have moved


11/117

Key Elements for Educational Accountability Models

rom compliance with regulations to improved student perormance with a resulting ocuson outcome data such as test scores and graduation results. Goertz (2001) likewise discussessome o the changes in educational accountability over the past 30 years, including whetherthe ederal education requirements supplement or supplant state and district requirements.She also discusses changes rom procedural accountability to educational accountability.

In addition to changes in the type o accountability measures, other changes are occurringin terms o who is accountable. Accountability systems are now ocusing more on schools asthe unit o change. As a result, continuous improvement strategies are being adopted at theschool level. Another area o change is how results are reported. For example, more categorieso perormance are being developed to report student scores. Students are no longer testedas passing or ailing but oten receive inormation that places them in one o our to veperormance levels (e.g., Below Basic, Basic, Procient, and Advanced). Schools are otenlabeled as either eective or needing improvement. Finally, more consequences are attachedto student perormance, including public reporting and tangible orms o recognition. Poorperormance is now likely to result in sanctions such as state intervention, visits by technical

advisory teams, and reconstitution o schools.Student assessment or educational accountability has moved rom matrix sampling to wholesystem. Tat is, in the past, schools, teachers, or students were matrix sampled so that only asample was given tests or other instruments to measure the desired outcomes. Ten these re-sults were generalized to the ull population. Now, however, the movement has been towardsassessing every student and evaluating every teacher and school.

Te recent changes in educational accountability have resulted in some state systems thathave become overloaded in trying to serve too many purposes simultaneously, ailing to serveany o them well. In addition, expectations have increased. Stakeholders expect to see imme-diate improvements and evidence that student learning has increased, leading to quick xesor short-term strategies. Te changes have discouraged a thoughtul approach to the entireeducational accountability system ocused on the goals.

Overview of the Report

Te report ocuses on a whole-system standards-based approach to educational accountabilityand the report is designed to provide inormation about essential elements o a good/valid ac-countability system. A central eature o this report is a tool that states can use in developinga new accountability system or in evaluating a current one. It is intended or use by state ordistrict policymakers who are designing, redesigning, or evaluating their accountability sys-tems. Te audience may include legislators, state and district superintendents, state or local

boards o education, directors o accountability and assessment, those providing supports andtraining to schools, school principals and education leaders, as well as researchers. Tis reportis not intended to discuss accountability systems only under the NCLB system, but beyondthe ederal denition. Tus the overarching goal o this report is to answer the question Asstates consider developing accountability plans separate rom NCLB or as an enhancement toNCLB, what are the elements, standards, etc. o a good accountability system?


12/117


o accomplish this goal, the Accountability Systems and Reporting SCASS contracted withthe National Center or the Improvement o Educational Assessment to undertake three taskstoward completing the paper:

Synthesize the literature to date on critical elements in an eective accountabilitysystemCrat a ramework incorporating core concepts, guiding questions, key elements,and various options listed in previous literatureProvide descriptive examples o more ne-grained elements or options that in-crease eectiveness and success o state-level accountability systems

Te results o the rst two steps are summarized in the next section. We developed a rame-work to use as a supporting organizational structure, a matrix and checklist created to ex-amine accountability systems using a series o questions and key elements within severalmajor topics. Te ramework was developed through a synthesis o the current literatureand resulted in seven core concepts undamental to any educational accountability system.Each o these core concepts is discussed in its own chapter. Each chapter provides additionaldetail rom the literature review, ocuses on the guiding questions, key elements, and optionswithin that core concept, and provides descriptive examples rom various state departmentso education. Tese examples, as described in step 3 above, are illustrations o how dierentcomponents have been implemented in one or more states.

Tere are also three appendices to this report. Te rst appendix includes a ull bibliography,annotated with 2-3 sentence summaries each article, report, chapter, or book. Tis bibli-ography matches the list o recommended reading at the end o each chapter. Te secondappendix includes the ull matrix developed to synthesize all o the literature. It was the rststep to developing our ramework o core concepts, guiding questions, key elements, and op-tions. It is presented in color, with each color reerencing a specic article, chapter, or report.Te guiding questions or key components are written in the authors own words to allow thereader to trace any eature back to its original source. Finally, the third appendix contains theull checklist or designing or evaluating an accountability system. Te checklist is organizedby core concept, and each relevant section is presented as in introduction to a chapter, alsoorganized by core concept.

Literature Review

Te literature review includes more than 70 articles, reports, chapters, and ull books. Teull annotated bibliography can be ound in Appendix A. Our initial review o the literatureshows one important ndinga surprising lack o conficting inormation across the stud-

ies. Authors dier primarily in the level o detail provided or in the component o ocus.Some authors ocused on guiding questions, while others ocused on developing a set o ac-countability standards or core elements. Some authors concentrated on the technical aspectso various indicators while others ocused on the goals and consequences o a system. Tissection summarizes some o the key ndings.

Kirst (1990) provides an historical overview o accountability research. He reviewed six broadapproaches to accountability, including (1) accountability through perormance reporting;

1.

2.

3.


13/117


(2) accountability through monitoring and compliance with standards/regulations, (3) ac-countability through incentive systems; (4) accountability through reliance on the market;(5) accountability through changing the locus o authority or control o schools; and (6)accountability through changing proessional roles. Most educational accountability systemsinclude some combination o these approaches. In act, Raudenbush (2004) argues that

accountability systems must include measures o processes, such as inormation on organi-zational and instructional practice, in addition to measures o outcomes i the goal is to helpimprove schools.

Kirst and others argue that a limiting actor in the eectiveness o current accountabilitysystems is erroneous assumptions by policymakers about the nature o schools and theirinternal capacities. Tese systems assume that schools have the capacity to improve and lackonly the standards, assessment tools, and incentives to do so. However, policymakers mustconsider what schools will do with the data provided. Tey then need to develop policies thatoster the leadership, collaboration, and skills that are essential to school improvement and toaccountability. o strengthen these, schools need both time and resources. Accountability sys-

tems can then accommodate and support specic practices and conditions as well as providethe structure o standards, assessment, and consequences.

Much o the literature echoes the same argument and encourages us to look at the system as awhole. An accountability system should be driven by its goals. Te indicators o perormanceselected, the design decisions, and the consequences applied all directly relate to the goals.Te goals must be well communicated. Te degree o support provided toward meeting thegoals and the communication o these goals are strong indicators o the ability o a systemto meet its goals. Tereore an overall theory o action should be both explicated and moni-tored.

Beyond a ocus on the goals, authors diered on how they communicated strategies or de-veloping an accountability system. Some created a set o standards or rules to ollow, oten-times grouped into major categories. Other authors highlighted the questions policymakersshould ask as they develop an accountability system. Others ocused on just one part o thesystem, such as evaluating the technical quality o the perormance indicators or determiningeective rewards and sanctions to employ.

Small components o an accountability system have been widely explored. For example,there is a air amount o research at the school level on the eect o consequences on studentachievement. Rather than try to synthesize every study, we chose to ocus on the systems ona macro level and examine how the various components work together to build a cohesivesystem. Many o these specic studies are included in the annotated bibliography or readerswho are interested in a specic topic within accountability.

Ater reviewing this research and various organizational strategies, we created our own orga-nizational strategy as a rst step in creating a tool that state or district policymakers could usein developing or evaluating their own system. Te next section will describe how we syn-thesized the current literature and created a matrix to rame our thinking about the variouscomponents that comprise all accountability systems.


14/117


Synthesizing the Literature

As a rst step in synthesizing the inormation into a useul ramework, we created the matrixshown in Appendix B. Tis matrix allowed us to group common ideas and sort the elementsby both major category o an accountability system and the ormat o the advice given.

Te matrix includes core concepts, guiding questions, and key elements taken directly romthe literature. Each cell is color coded to indicate the original source. In some cases, an ideacame rom multiple sources or was developed in discussions with the ASR SCASS, in whichcase the color was let as black on white.

Core Concepts

We developed a list o seven core concepts based on the various organizational strategies oother authors. Previous researchers and authors diered in how they modeled the essentialcomponents. Stecher and Hanser (1992) began with Levins (1974) denition and elaboratedan educational accountability model with our major components, arguing that any eective

accountability system must include each o these components:Goals (e.g., 90 percent o high school students will graduate);Measuresmeans or assessing progress toward the goals (e.g., cost accounting,comprehensive record keeping);A eedback loopto provide assessment inormation and constituent input backto the system; andA systemic change mechanismor reacting to eedback by changing the system,as appropriate.

CRESS (Baker, Linn, Herman, & Koretz, 2002) created a set o 22 standards or educa-tional accountability that they separated into ve key categories:

System componentsestingStakesPublic reportingEvaluation

Hanusheck & Raymond (2002) stated that the basic skeleton o accountability systemsincludes

Goals

Content standardsMeasurementConsequencesReporting

According to Carlson (2002), there are ve key elements o accountability systems:

Te goals o the systemTe selection o key indicators o success and ways to measure them (multiple

1.2.

3.

4.

1.2.3.4.5.

1.

2.3.4.5.

1.2.


15/117

Key Elements for Educational Accountability Models

measures), rather than merely using inormation that is availableDecisions about how the selected indicators will be scaled, weighted, combined,and reportedTe types o actions that will be taken based on the resulting perormance data(rewards and sanctions)

Steps that will be taken to determine and improve the eectiveness o the ac-countability system itsel

And, in their ramework or evaluating the validity o an educational accountability systemunder NCLB, Forte Fast and Hebbler (2004) highlighted our primary components:

Perormance indicators (Percent procient, participation rates, graduation/atten-dance rates, other)Decision rules (how scores are combined and interpreted)Consequences (Sanctions and rewards or changes in PD, pedagogy, or resourceallocation)Goals (e.g., 100% prociency in reading and math by 2014)

Almost all o these organizational schemes include goals and indicators or measures. Severalalso include consequences or ollow-up actions. Reporting, evaluation, and internal monitor-ing were also mentioned.

o develop our list o core concepts, we started with the our core concepts presented in thevalidity discussion by Forte Fast & Hebbler (2004). Te original category decision rulesappeared too specic to test scores. Tus, to better meet our purposes we renamed it designdecisions to ocus it on the design o the system and how to ensure the various indicatorsworked well together. We then added communication and support to emphasize the im-portance o the reciprocal relationship between state policymakers and local administrators.

Finally, work by Carlson, Linn, Baker, and others emphasized the importance o continuallyevaluating the system itsel in order to improve it (Baker, Linn, Herman, & Koretz, 2002;Carlson, 2002). Some authors wrote about building in a eedback loop to provide inorma-tion and input back into the system and a systemic change mechanism or reacting to theeedback by adapting the system as necessary (e.g., Stecher & Hanser, 1992). We thereoreadded a nal category or evaluating, monitoring and improving the system.

Guiding Questions

Te next column in the matrix is called guiding questions as it lists questions that aremeant to be asked by state (or district) policymakers as they develop or evaluate their ac-

countability system. Tese questions help to rame the issues or consideration. Tese ques-tions come rom multiple sources but primarily rom Ananda & Rabinowitz (2001), Erpen-bach (2002), and Gong (2002). Te questions generated in these three reports are more alikethan dierent. Tey all ollow the process o designing an accountability system rom theestablishment o goals, through the design and determination o indicators to the eects andevaluation.

Ananda & Rabinowitz (2001) identied a sequence o key questions that must be addressedin planning an accountability system. Tey ocused on the big-picture issues in designing

3.

4.

5.

1.

2.3.

4.


16/117


an accountability system asking question starting rom the rst issue in accountability sys-tem that is What are the primary goals you are trying to accomplish with an accountabilitysystem? through the question o what to do with the results What will you do about theproblems uncovered through the accountability system? Tey laid out a simple sequence oquestions and issues rom the goals to the indicators to the consequences to the eects.

Erpenbach (2002) and colleagues ocused primarily on the use o multiple measures in devel-oping accountability systems. Tey developed ten questions moving rom the purposes o thesystem to the selection o multiple indicators, to decisions on how to combine the indicators,to reporting, to nally evaluating the eects and the system itsel.

Gong (2004) developed an earlier report with the ASR SCASS creating a linear sequence oten questions that state policymakers could use when designing an accountability system. Tequestions were developed to include criteria and comments intended to provide a structureor helping states moved through the process o designing a school accountability system. (p.2). Te questions start with dening the purposes, move to dening how the data will be col-lected, combined, and reported and conclude with how the system itsel will be evaluated.

Again, all three have common elements refected in the guiding questions section o thereport. Tey all include the same sequence o events. Tey all highlight the importance ostating clearly the goals and purposes o the systems. Tey all discuss the perormance indi-cators and design decisions. Tey all include a consideration o an internal evaluation andmonitoring system.

Key Elements

Te third column, Key Elements, include both standards that researchers and practitionersrecommend or inclusion in any accountability system and considerations that are more

dependent on the goal.Te most commonly cited standards include those developed by the National Center orResearch on Evaluation, Standards, and Student esting (CRESS). Baker, Linn, Herman,& Koretz (2002) developed a set o 22 standards or accountability systems in ve categories:system components, testing, stakes, public reporting, and evaluation. Tese standards repre-sent models o practice rom three perspectives: research knowledge, practical experience, andethical considerations.

Hill and DePascale (2003) provide a ramework or evaluating technical considerations o theaccountability system. Te standards developed in their work involve tradeos between valid-ity and reliability and inorm such issues as minimum group size and condence intervals.

Other key elements come rom Elmore (2003) who developed a set o principles ater study-ing two low-perorming schools and the reorms they implemented. His standards ocusmost on supporting schools as they work to improve student outcomes. Elmore (2004)continues to ocus on the reciprocity between accountability and capacity, emphasizing theimportance o setting realistic goals. Forte Fast and Hebbler (2004) ocus on consistencywithin the entire system. Tat is, once an explicit theory o action has been developed, careshould be taken to match indicators and decision rules to the goals. Te consequences should


17/117


18/117


Progress monitoring (conducting district audits and onsite reviews)

Elmore (2003) ocuses on supports teachers need to help improve student learning. Tesesupports are in the orm o nancial, including loan orgiveness and extra compensation;proessional development, such as access to courses and advanced degree program); and im-proved working conditions, including reduced teaching loads, improved physical inrastruc-ture, and better materials or the classroom.

Ohio Department o Education has developed a strategic plan that includes supporting theaccountability goals by communicating clear, high expectations to students; aligning theteachers proessional development to the districts mission; providing teachers sucient timeor planning; providing students with appropriate opportunities to learn; and implementingappropriate instructional interventions.

Further and more detailed examples o state strategies or meeting their goals within eachcore concept are provided in Chapter 3. Te above examples are intended to be brie men-tions o the types o options that may be used or dierent components o an accountability

system. Specic examples o options states have used or each o the core concepts are pro-vided in subsequent chapters.

Differences between Federal and State Accountability Systems

One important theme in the literature since the passage o NCLB is the challenge o in-corporating existing state accountability systems into the ederal accountability require-ments. Tough the Improving Americas Schools Act o 1994(IASA), a reauthorization othe Elementary and Secondary Education Act, helped to kick o the standards movement,implementation o statewide standards was inconsistent. For example, although IASA re-quired states to develop a single statewide accountability system, many states were not yet

in compliance with this act and had varying degrees o state level accountability in place in2001. NCLB brought more pressure on the states to develop strong accountability systemsthat met very specic criteria; however, it did not require one single accountability system. Asurvey by CCSSO ound that most states adjusted their own accountability systems to meetthe ederal requirements and created one system to serve both purposes. Other states did nothave a well-developed state accountability system at the time NCLB was passed and devel-oped one to meet the new ederal requirements. A handul o states, however, have attemptedto maintain their own state accountability system and adopted a separate system to meet theederal requirements.

For purposes o simplication, we will call the ederal accountability system theAdequate

Yearly Progresssystem or AYP. Consider a state such as Florida that has maintained its A+accountability system or Caliornia that still uses the API to rate schools. Tese states mayproduce dierent classications or their schools when using their state system and AYP.

Linn (2005b) has written about the mixed messages schools receive rom such biurcatedsystems. Tat is, a school might meet the goal under the state accountability system but notmeet it under the ederal system, or vice versa. For example, Kentucky uses an index basedon tests in seven content areas, attendance, and graduation rates. argets are set biennially by

5.


19/117


school. In 2004, 74% o Kentucky schools made ederal AYP targets, while 95.6% met theirstate targets. Not all schools who met AYP met state targets. Approximately 25% o schoolsmet state targets while missing AYP, providing mixed messages to teachers, administrators,and the public.

In Florida, 56% o schools that received an A on the state accountability system in 2004ailed to make AYP. In Colorado in 2003, 21.9% o schools rated as unsatisactory and47.5% o schools rated as low made AYP, while 13.7% o schools rated as high ailed tomake AYP.

Linn (ibid.) discusses several eatures o NCLB that make it likely to have results confictingwith state accountability system results:

Te use o absolute targetsTe need or conjunctive targetsTe requirement o meeting targets by subgroup

Every school has between 5 and 37 targets (hurdles) under NCLB, meaning there are manyways to miss AYP and only one way to make it. Tat is, a school must meet its target orevery single subgroup to make AYP, but i it misses just one no matter which one or howmany over one it is placed into an improvement category. Novak and Fuller (2003) oundthat schools serving more diverse students were less likely to meet AYP requirements thanschools serving less diverse student bodies.

Other dierences in ways in which school achievement benchmarks are set or NCLB com-pared to state accountability systems include:

NCLB ocuses on status measures, while state systems typically incorporate im-provement or growth measures

NCLB sets an absolute level o perormance regardless o schools starting point,whereas state systems typically set goals based on the schools baseline peror-mance.NCLB and states dier on the long-range goals and the timeline or meetingthem.

Goertz (2001) also identies how state accountability systems varied prior to the passage oNCLB. Her analyses correspond well with those by Linn. She discusses how state systemstended to use a progress goal rather than an absolute goal. Tus, the expected level o studentperormance was dierent then than under NCLB. Other variations between state systemsand NCLB included the percentage o students that schools were required to have meet this

expected level and the length o time schools were given to meet their goal. All o these ac-tors are now regulated under NCLB.

States that maintained their previous accountability system or who later supplemented theAYP model with their own state model now ace a situation in which school may be shownto be high quality under one system and poor perorming under another. As will be discussedin the next chapter, policymakers need to consider how these mixed messages may aectschool reorm eorts and parent decision making.


20/117


Summary

Te literature on this topic is extensive. In act, there is much more literature than what hasbeen summarized in this chapter. Te next chapter continues the literature review in greaterdetail, but in the context o the core concepts listed in the matrix. Overall, it is worth notingthe extent to which dierent researchers and educators agree on the necessary components oan accountability system.

Te matrix that synthesized the literature review was reormatted into a checklist tool, as-sembling key elements in a manner that can be used by states. Tere are seven core conceptsas described in the previous chapter. Each o these concepts is used as an organizer. Withineach o the core concepts, we have listed guiding questions and key elements. Te guid-ing questions are intended to help policymakers rame their system as they explore theiranswer to each question. Te key elements include both standards o what should be doneand considerations o what can be done. For instance, develop an explicit theory o actionis a standard that should be ollowed, while consider whether to use all grades or selectedgrades is not a directive, merely a reminder that this is an issue that should be deliberated.Tere are areas where (1) we do not know the most eective model, such as the best type oincentive to use, or (2) the most eective model is completely dependent on the goal o thesystem. Since we do not presume to know every states goal(s), we cannot provide exact guid-ance on the best option to select or each element. Te purpose o this tool is not to endorseone model over another but to ensure that those using it consider all possible options and usebest practices when they are known and appropriate.

Te organizing structure ocuses on seven core components that must be considered in anyeducational accountability system:

Goals

Perormance indicatorsDesign decisionsConsequencesCommunicationSupportSystem evaluation, monitoring, and improvement

Each o these components is addressed in a subsequent chapter.

1.

2.3.4.5.6.7.


21/117


References and Suggested Readings for This Chapter

American Evaluation Association (2002). Public statement: High stakes testing in preK-12education. Fairhaven: MA: American Evaluation Association. Downloaded October 8, 2007rom http://www.eval.org/hst3.htm.

American Institutes or Research. (2006). Summary o state strategies or districts identiedor improvement under NCLB. Sacramento, CA: Caliornia Comprehensive Center.

American Psychological Association, American Educational Research Association, & the Na-tional Council on Measurement in Education. (1999). Standards or educational and psycho-logical testing. Washington, DC: American Psychological Association.

Ananda, S. & Rabinowitz, S. (2001). Building a workable accountability system. San Fran-cisco: WestEd. Downloaded February 20, 2007 rom www.WestEd.org/online_pubs/kn-01-03.pd.

Bailey, A., Winter, P., & CAS SCASS. (2002). Incorporating ederal requirements into stateaccountability plans. Washington, DC: Council o Chie State School Ocers.

Baker, E., Linn, R., Herman, J., & Koretz, D. (2002). Standards or educational accountabil-ity systems[Policy Brie 5]. Los Angeles, CA: National Center or Research on Evaluation,Standards, and Student esting

Carlson, D. (2002). Te ocus o state educational accountability systems: Four methods ojudging school quality and progress. In W. J. Erpenbach et al., Incorporating multiple mea-sures o student perormance into state accountability systemsA compendium o resources(pp. 285-297). Washington, DC: Council o Chie State School Ocers.

Darling-Hammond, L. (2006). Standards, assessments, and educational policy: In pursuit ogenuine accountability. Princeton, NJ: ES

Elmore, R. (2003). Knowing the Right Ting to Do: School Improvement and PerormanceBased Accountability, Washington, DC: NGA Center or Best Practices

Elmore, R. (2004). Conclusion: Te Problem o Stakes in Perormance-Based Accountabil-ity Systems. In Fuhrman, S. & Elmore, R., (2004). Redesigning Accountability Systems orEducation. New York, NY: eachers College Press.

Erpenbach, W. (2002). Incorporating multiple measures o student perormance into stateaccountability systemsA compendium o resources. Washington, DC: Council o Chie

State School Ocers.

Forte Fast, E. & Hebbler, S. (2004).A ramework or examining validity in state accountabil-ity systems. Washington, DC: Council o Chie State School Ocers

Fuhrman, S. (1999, January). Te new accountability. CPRE Policy Bries, RB-27. Philadel-phia, PA: Te Consortium or Policy Research in Education.


22/117


Fuhrman, S. & Elmore, R., (2004). Redesigning Accountability Systems or Education. NewYork, NY: eachers College Press.

Goertz, Margaret. (2001). Te Federal Role in an Era o Standards-Based Reorm. in TeFuture o the Federal Role in Elementary and Secondary Education: A Collection o Papers.Washington, DC: Center or Education Policy.

Gong, B. & ASR SCASS. (2002). Designing school accountability systems: oward a rame-work and process. Washington, DC: Council o Chie State School Ocers.

Hanushek, E. & Raymond, M. (2002). Sorting out accountability systems. In W. Evers &H. Walberg (Eds.) School accountability(pp. 75-104). Palo Alto, CA: Stanord University,Hoover Press.

Hill, R. & DePascale, C. (2003). Reliability o No Child Let Behind accountability designs.Educational Measurement: Issues and Practice, 22(3), 12-20.

Joint Committee on Standards or Education Evaluation. (2004). Te Program Evaluation

Standards. How o Assess Evaluations o Educational Programs. (2nd Ed.). Tousand Oaks,CA: Sage Publications.

Joint Committee on Standards or Education Evaluation. (2002). Te Student EvaluationStandards. How o Improve Evaluations o Students. Tousand Oaks, CA: Corwin Press.

Kirst, Michael W. (1990).Accountability: Implications or State and Local Policymakers.Washington, DC: Inormation Services, Oce o Educational Research and Improvement,U.S. Department o Education.

Levin, H. (1974). A Conceptual Framework or Accountability in Education. School Review82(3), pp. 363-391.

Linn, R. (2005a). est-based educational accountability in the era o No Child Let Behind.[echnical Report 651]. Los Angeles, CA: National Center or Research on Evaluation, Stan-dards, and Student esting.

Linn, R. (2005b). Conficting demands o No Child Let Behind and state systems: Mixedmessages about school perormance. Educational Policy Analysis Archives, 13(33). RetrievedJanuary 15, 2007 rom http://epaa.asu.edu/epaa/v13n33.

Novak, J. R. & Fuller, B. (2003, December). Penalizing diverse schools? Similar test scores,but dierent students, bring ederal sanctions. Policy Brie034, Berkeley, CA, University o

Caliornia Berkeley: Policy Analysis or Caliornia Education, available online at http://pace.berkeley.edu.

Raudenbush, R. (2004). Schooling, statistics, and poverty: Can we measure school improve-ment? ES Ango Lecture, 2004.

Stecher, B. & Hanser, L. (1992). Local accountability in vocational education: A theoreticalmodel and its limitations in practice. MDS-291. Santa Monica, CA: RAND.


23/117


Chapter 2: Goals of Accountability Systems

Checklist for State Accountability Systems: Goals

What are the purposes of the accountability system? Why are you implementing the

system?

Explicate the local, state, and federal requirements that this system is trying to

fulfill.

What are the primary goals you are trying to accomplish with an accountability sys-

tem? What are you hoping to accomplish?

Establish and prioritize goals.

What accountability decisions will be made and with what consequences?

Develop an explicit theory of action that relates the goals and design of the

system to anticipated outcomes by the state, schools, and others.

In what systems are you working? What are the main legal and policy constraints orspecifications?

Explore the legislative, policy, and legal influences and understand how those

considerations will determine the design and implementation of the account-

ability system.

In broad terms, what are the schools/students (or others) to be held accountable

for?

Match the indicators and decision rules to the goals.

Begin to consider how to combine multiple indicators to match the goals

by determining whether the goals best fit a status, improvement, or growth

model.Te goals portion o the ramework emphasizes the purposes, uses, and contexts or theaccountability system. We distinguish between the purposes, which provide an overarch-ing reason or using an accountability system, and the goals, which speciy the intendedoutcomes. Tird, we introduce the idea o decisions and consequences here, although theyeach are considered a core concept and have a separate section later. It is important to startthinking about decisions and consequences when developing a theory o action. Developinga theory o action is one o the key elements stressed here as it links the intended outcomes tothe various indicators and supports provided. Te next key element ocuses on understandingthe context in which the system must operate and the external pressures that may constrain

certain measures or decisions. For example, legal restrictions on individually identiabledata may require a state to set a minimum number o students that must be in any categorybeore data can be reported in that category. Finally, we must begin to think about the systemin broad terms under this rst core concept, ocusing on the relationship between the goals,intended outcomes, measures, decisions rules, supports, and consequences.

Linn (2001) encourages state policymakers to be clear about the intended purpose(s) or theireducational accountability system. For example, while most states or districts would agree


24/117


that the purpose o accountability is to improve student learning, Linn argues that statesneed to be more specic in stating their priorities or achieving such a goal. For instance,beyond improving student learning, state policymakers may specically desire to:

Reinorce content standards in priority subjectsSupport deep understanding and problem-solving abilityAssure a given level o achievement or students beore promotion

He also recommends that policymakers clariy the uses o their system, asking specic ques-tions such as:

What results will be given to parents?What will be done with school-level results?How much emphasis should be placed on status versus improvement?

Te report by Gong and the ASR SCASS (2002) ocused on the importance o understand-ing the legal and policy constraints surrounding the system. Policy contexts, in particular, not

only drive the implementation o an accountability system, but may change as an adminis-tration changes requiring constant updating o a system. Erpenbach (2002) also considerslegislative and policy infuences a crucial consideration in developing goals and designing theaccountability system. On one level, NCLB has driven the policy agenda to ocus on a basicskills model with its requirement o 100% prociency. By requiring 100% o students toreach the bar, the legislation has eective negated the importance o achievement beyond thatbar. And, as shits are made in the law allowing a growth model, or example accountabil-ity systems shit to adjust or these changes.

Other legal considerations may ocus more on the specics within a system. Legal require-ments are particularly important when considering issues o access and equity. For example,

Baker, et al. (2002) highlighted one legal constraint in their comments on Standard 4 insaying Legal requirements as well as ethical considerations demand that all students beincluded in the accountability system. Whether these types o policy and legal infuences arepositive or negative can be debated, but it is important to acknowledge their existence andtheir ongoing eect on the accountability system.

Ananda and Rabinowitz (2001) discuss the problem many states have with trying to set toomany goals. For instance, they discuss the pressures policymakers eel to embrace multiplesweeping education improvement goals, such as improving student learning, motivatingteachers and students, reducing achievement disparity between majority and minority stu-dents, monitoring education costs, improving access to education, building public con-

dence in education, and improving the states competitive economic status as compared toother states. (p. 2). Tey recommend state policymakers consider selecting short-term andlong-term goals or phasing in the dierent goals. Tere is a widely held belie that systemswith too many goals will ail to achieve any. It is important to ocus the goals to make themattainable.


25/117


26/117


Figure 1. A Simplified Theory Of Action: How Accountability Expectations And ActionsWill Lead To Higher Achievement

Taken from Marion, S., White, C., Carlson, D., Erpenbach, W, Rabinowitz, S., & Sheinker, J. (2002). Making

valid and reliable decisions in determining adequate yearly progress. Washington, DC: Council of Chief State

School Officers. p. 36.

Several state policymakers have taken this idea and developed an explicit theory o actionor their state that depicts how its actions and resources will be used to achieve its goals bothwithin and beyond NCLB. For example, Nebraskas operational theory o action has threemain goals: improving student achievement, maintaining eorts in school improvement, and

building the capacity o educators to implement needed changes. As described by the com-missioner o education, Doug Christensen:

Nebraskas approach to standards, assessments, and accountabilitySchool-basedeacher-led Assessment and Reporting System (SARS)is rmly grounded in thebelie that decisions about student learning should be standards-based and should beocused on classroom knowledge o the student. Tis process relies upon the proes-sional expertise o Nebraska educators and has been built upon a statewide initiativeto develop educator capacity in assessment design and the use o assessment data orimproved instruction. Te requirements o ederal NCLB have been integrated intothe accountability requirements.

Nebraskas theory o action and thus its system diers rom that o most other states in thatit is locally based. eacher-designed classroom assessments are the oundation o the systemrather than statewide assessments. Te state role is to support the development o classroomand district assessments through specialized training and proessional development activi-ties. Te values embodied by this approach include providing real-time inormation back toteachers and embedding improvement activities into everyday practice o school and districtsta.


27/117


28/117


OhioDepartmentofEducation(ODE)T

heoryofAction


29/117


describe them as elements that must be in place to achieve their desired result. Each indicatoralso has a series o measures associated with it to monitor the degree to which these eectsare occurring. For instance, the success indicator Educators receive proessional developmentaligned with academic content standards and efective teaching and leadership strategieshasthree measures associated with it:


30/117


Percentage o teachers participating in high-quality proessional developmentPercentage o ODE-sponsored proessional development activities that meet thestandards o high-quality proessional developmentNumber o teachers and administrators completing Entry Year Programs

Finally, ODE lists 12 measures or the results o higher achievement or all students. Inaddition to the ederally mandated measures o percentage o students, schools, and districtsmeeting AYP, this system also looks at measures o improvement (number o districts andbuildings that improve rating rom previous year), growth (number o districts and buildingsshowing Perormance Index gain), and other measures such as graduation rates, participationin post-secondary institutions, percentage o students passing advanced courses (AP exams).Te system also considers perormance on non-state tests, such as NAEP, AC, and SA.

ODE states their belie that this aligned education system will promote higher achievementor all students. By constantly monitoring success at all stages o the system, they can evaluateand adjust their strategies to ensure they are implemented eectively. Furthermore, this typeo documentation addresses at least three o the our aspects o a theory o action. Te onepiece that is not clearly articulated is their theory or the consequences. One sub-strategy in-dicates that developing a air accountability system includes rewarding success, and we knowthat specic sanctions are required or schools and districts that miss AYP two years in a row,but this theory o action does not explain how the rewards or sanctions will lead to improvededucation system or student eects and thus work to meet the desired result.

1.2.

3.


31/117



Ananda, S. & Rabinowitz, S. (2001). Building a workable accountability system. San Fran-cisco: WestEd. Downloaded February 20, 2007 rom www.WestEd.org/online_pubs/kn-01-03.pd.

Baker, E., Linn, R., Herman, J., & Koretz, D. (2002). Standards or educational accountabil-ity systems[Policy Brie 5]. Los Angeles, CA: National Center or Research on Evaluation,Standards, and Student esting

Erpenbach, W. (2002). Incorporating multiple measures o student perormance into stateaccountability systemsA compendium o resources. Washington, DC: Council o ChieState School Ocers.


Gong, B. & ASR SCASS. (2002). Designing school accountability systems: oward a rame-work and process. Washington, DC: Council o Chie State School Ocers.

Hamilton, L., Stecher, B., & Klein, S. (Editors.) (2002). Making sense o test-based account-ability in education. Santa Monica, CA: RAND

Linn, R. (2001). Te design and evaluation o educational assessment and accountabilitysystems. [echnical Report 539]. Los Angeles, CA: National Center or Research on Evalua-tion, Standards, and Student esting.

Marion, S., White, C., Carlson, D., Erpenbach, W, Rabinowitz, S., & Sheinker, J. (2002).Making valid and reliable decisions in determining adequate yearly progress. Washington,

DC: Council o Chie State School Ocers.


32/117



33/117


34/117


A major issue in any accountability system is the question o what to measure. o answer thatquestion, we need to examine the data that are available, the targets o the data collection,and the timing o the data collection. Once those pieces have been decided, considerationalso needs to be given to ensuring the reliability and validity o the data used to inorm theaccountability system.

Researchers strongly suggest that systems include multiple measures. No student, school,or district should be sanctioned on the basis o one indicator, such as a test. According toRaudenbush (2004), Accountability systems must include measures o processes, such as in-ormation on organizational and instructional practice, in addition to measures o outcomesi the goal is to help improve schools. Such measures may include teacher qualications,curriculum, learning environments, attendance rates, graduation rates, school size, class size,responsiveness to parents, and teacher preparatory time. For research on education indicators,also see Shavelson, et al, RAND, ; Blank, 1993; and the CCSSO annual reports on stateeducation indicators,, http://www.ccsso.org/projects/State_Education_Indicators.

Porter (1991) recommends a list o indicators or measuring school quality including:

Instructional leadershipExistence o a ocused missionHigh expectations or all studentsask-oriented climateSystem or monitoring outcomes within the schoolSchool and class sizeGrade level organizationeacher planning timeAvailability, quantity, and credentials o teacher aidesNumber o teacher preparations requiredShared decision making

Porter also recommends gathering data at the state or district level on the ollowing elements:

Subject matter rameworkStudent assessmentsControl o textbooksSpeciying graduation requirementsGuidelines or allocating time to core subjectsSchool accreditation programsManagement inormation systems

Even in ocusing on student perormance as measured by assessments, policymakers need toconsider which subjects will be included in the accountability model. NCLB bases decisionssolely on reading and mathematics perormance, but several state systems also include sci-ence, social studies, and writing.

For example, Caliornias accountability system includes both their ederal AYP goals andindicators, but they use their state indexthe Academic Perormance Index (API)as theiradditional indicator. Te API is a weighted index that includes results rom both their crite-


35/117


rion-reerenced and norm-reerenced assessments in ELA, math, science, and history/socialsciencesdierent indicators are used at dierent grade levels. More inormation is providedon the API in chapter 4, but it is important to note that Caliornia has made the policy deci-sion to include science and history/social sciences in their accountability system, even thoughthose subjects are not required to be used to determine a schools classication under the

ederal system.Kentucky also uses results rom their science and social studies tests in their state account-ability system. In addition, they include a writing portolio at grades 4, 7, and 12, test arts& humanities at grades 5, 8, and 11, and practical living & vocational studies at grades 5, 8,and 10.

In addition, while NCLB uses all grades 38, policymakers should deliberately considerwhether assessing every grade makes sense, or i they want to measure progress at key grades,

perhaps one at each school level: elemen-tary, middle, and high. Prior to NCLB,many states had assessments at keygrades, such as grades 4, 7, and 10. Ithe requirement to test at every grade38 did not exist, what testing schedulewould make the most sense in achiev-ing the goals o the system? Likewise, ipolicymakers decide to include additionalsubjects beyond reading and math intheir accountability system, how otenshould achievement in the other subjectsbe assessed?

NCLB requires one additional indica-tor to supplement the participation andprociency indicators. Most states havechosen to use attendance rates at theelementary and middle school levels andgraduation rates at the high school levels.

However, some states have gone a step urther by including at least one other additional in-dicator to those required by NCLB. For example, Delaware uses a vertical scale across grades3 through 8 to report their assessment results. An additional indicator in their accountabilitysystem, then, is whether students are maintaining or showing progress rom one grade to the

next. Delaware, Kentucky, and Caliorniaas well as other statesalso consider studentperormance in science and social studies as part o an additional indicator. Oklahoma andOhio use additional indicators in their high schools that measure the percentage o studentsreceiving advanced placement (AP) credit and taking the AC or SA.

Another consideration besides simply adding more indicators is to dierentiate the indicatorsbased on school or district characteristics. Tat is, consider using dierent indicators o excel-lence in rural versus urban schools or in schools with high diversity. Some states already use

Additional Indicators

Missouri uses the following indicators in their

state accountability system in addition to the

traditional achievement and participation

measures:

ACT (percent of graduates scoring at or

above the national mean)

Advanced Courses (percent of credits

earned)

Career Education Courses (percent of

credits earned)

College Placement (percent of graduates

entering college)Career Education Placement (percent of

career education completers placed in

occupation)

Graduation & Attendance Rates


36/117


dierentiated indicators across dierent school levels (e.g., elementary versus high school).As stated in the example above, Oklahoma includes inormation on academic excellence atthe high school level that does not exist at the elementary or middle school level, includingperormance on the AC and AP tests and the need or college remediation. Tese indicatorsare part o their academic index at the high school level, but there are no equivalent indica-

tors at the elementary or middle school level.

Validity of the Indicators

Te validity or each measure should be documented. Because a major indicator in any ac-countability system is perormance on an assessment, the use o the assessments should bevalid. Tere are many standards that address the validity o test interpretation in the AERA/APA/NCME Standards or Educational and Psychological estingthat should be ollowed.Some areas that help build the validity argument include demonstrating the alignment othe test items with the content standards, using a representative sample o judges to set cutscores, providing the rationale or creating subscores or composite scores, and including

interpretation aids in score reports.Validity or assessment is generally considered to have two aspects: Te assessment measureswhat it intended to measure, and the interpretations and uses (consequences) are justied.Similarly, validity or accountability decisions should refect accurate evaluation o what wasintended to be measured and justication o the interpretations and uses (especially conse-quences ormally specied as part o the accountability system). By extension, some theoristswould hold that an accountability system is ully valid only i its consequences are also ullyvalid.

Although the validity o an accountability system builds on the validity o assessment sys-tems, having a valid assessment system is not sucient. A state could have valid assessments

and not have a valid school accountability system. For example, the states assessments inmathematics may be highly valid and reliable, but the states use o the assessment data in itsaccountability system may not be valid or reliable. In addition, most accountability systemsincorporate additional indicators beyond assessment outcomes. It is important that a qual-ity assurance plan be developed to validate the use o each measure. As an example, the ASRSCASS recently released a report detailing a quality assurance plan or calculating graduationrates (aylor, Beaudoin, & Goldschmidt, 2007). Te report encourages those working onaccountability systems to implement quality checks both at the time o collecting data andwhen the results are reported. Tese quality checks begin with training those who are collect-ing the data, periodic monitoring o the data collection, internal and external audits o the

calculations, and statistical verication o the results.Tus it is important not only to ensure the validity o each indicator in an accountabilitysystem, but to ensure the indicators are combined, used, and reported in a meaningul way.Tese concepts will be discussed in the next two chapters.


37/117



Blank, R. (1993). Developing a System o Education Indicators: Selecting, Implementing,and Reporting Indicators. Educational Evaluation and Policy Analysis, 15(1). pp. 65-80.

Council o Chie State School Ocers. State Education Indicators with a Focus on itle I,

(Annual Reports 1997- present). Washington, DC: U.S. Department o Education.


Jones, K. (2004). A balanced school accountability model: An alternative to high-stakes test-ing. Phi Delta Kappan, 85(8). pp. 584-590.

Porter, Andrew C. (1991). Creating a System o School Process Indicators. EducationalEvaluation and Policy Analysis, 13, pp. 13-29.

Raudenbush, R. (2004). Schooling, statistics, and poverty: Can we measure school improve-

ment? ES Ango Lecture, 2004.

Shavelson, R., McDonanell, L, Oakes, J., Carey, N. (1987). Indicator Systems or Monitor-ing Mathematics and Science Education, Santa Monica, CA: Rand

aylor, R., Beaudoin, J. & Goldschmidt, P. (2007). Quality assurance practices associatedwith producing cohort graduation rates. Washington, DC: Council o Chie State SchoolOcers.


38/117


39/117


Chapter 4: Design Decisions

Checklist for State Accountability Systems: Design Decisions

Who is held responsible?

Develop a system of shared responsibility.

Consider providing information both on students (achievement) and educa-

tors (instruction).

What accountability model best serves your purpose?

Deliberately consider status versus improvement versus growth and align the

model with the goals.

If the goals imply treating low performing but improving schools differently

from low performing schools, ensure accountability system can distinguish

among schools that are improving and those that are not.

How will data from multiple measures and indicators be combined to categorize

schools and make an accountability judgment?Make explicit the weighting of elements in the system, including different types

of test content and different information sources.

Decide whether your system will use compensatory or conjunctive rules

(both at the individual level and in the aggregate) by considering the goals

and theory of action.

Consider the use of an index to weight elements appropriately.

Consider the complexity and trade-offs with fairness (complex formulas to

create indices can include all relevant factors but be difficult to communicate

to the public).

What is satisfactory progress?Proceed at a pace that is technically defensible and politically reasonable.

Decide whether to hold schools accountable for how they increase student

learning or to the absolute level of learning (which includes effects of prior

instruction).

Consider holding school systems (e.g., districts) accountable for the cumula-

tive learning of students over their career in the system.

Decide between having the starting point for a school determine the goal or

setting the same goal for all schools.

Decide between determining the end point as a fixed point in time, or allow-

ing the annual goals determine the end point for the final goal.

Decide whether each subgroup should meet a separate bar based on their

starting point or the same bar.

How do we balance reliability and validity concerns in making decisions about

schools?

Strive to make the overall design of the system valid and implement it in a way

that leads to valid results. Consider all potential threats to validity and eliminate

those as a first step.


40/117


Carefully consider the real costs of Type I and Type II errors and attempt to

reach a reasonable balance between the two. Do not assume that the costs

associated with either type of error are insignificant.

Consider the goal of inclusion of all subgroups when making determinations

of a minimum N for subgroups.

Explore the uncertainty involved in measuring school improvement over oneyear in terms of making reliable judgments for most schools.

Develop rules for determining satisfactory progress of schools and individu-

als to avoid erroneous judgments attributable to fluctuations of the student

population or errors in measurement.

How do we move from an old accountability system to a new one?

Consider previous policies when making new. Decide whether the new sys-

tem should replace or supplement the old by considering all implications.

Determine how to transition any school categorizations from the old system

to the new.

Include a reasonable timeline for the transition.

Once policymakers have decided on a set o indicators, the next question is how to use themto make decisions about teacher, school, or district eectiveness. Tis issue gets at one o themain sources o discussion o the NCLB regulationswhether school eectiveness is bestmeasured using a status, improvement, or growth model, or some combination o these.Although this issue should have been addressed under goals and used to select indicators, itis discussed here because the model selected relates to how accountability decisions are made.Next, policymakers should decide how to combine the indicators to make decisions aboutstudents, teachers, schools, and districts. Will the indicators be combined in a compensa-tory ashion, where low perormance on one measure can be oset by high perormance on

another or will there be a minimum level o perormance set or each measure? Again, the an-swer to this question will be determined by the goals o the system. In addition, this sectionaddresses decisions regarding school classication, such as how high to set a bar, how oten toraise a bar, and how to balance reliability and validity concerns.

Combining Indicators to Make Accountability Decisions

Te rst design decision involves determining how to combine the dierent perormanceindicators to determine i the school has met the goal(s). In a coherent system, many o thesedecisions will be based on the goals. For instance, weighting the various elements in a systemrelates directly to the values placed on each element. A dierent signal is sent i perormance

indicators are given equal weight as process indicators than i perormance indicators aregiven twice the weight as process indicators.

Another issue is how ne grained the decision measure should be. NCLB breaks down alldecisions into meets annual measurable objectives or does not meet annual measurable ob-jectives. However, Hanushek and Raymond (2002) argue that binary pass/ail decisions leadto a set o complication that can be avoided by providing more detailed inormation aboutthe distribution o scores.


41/117


Example: Oklahoma

Oklahoma uses an Academic Performance Index (API). The API is a numeric score that measures school

and district performance based on a variety of educational indicators. It allows schools and districts to

gauge their progress toward improving student achievement. Seven indicators were mandated to be

included in this index, including the Oklahoma State Testing Program (OSTP), ACT, advance placement

courses, college remediation, dropout and graduation rates, and attendance. Components of the API are

used to meet NCLB reporting requirements. The scale for API score ranges from 0 to 100; the state aver-

age for general education students was 1180 in 2006.

Oklahomas API: Point Contributions for Elementary and Middle Schools and K-6/K-8 Districts

Oklahoma SchoolTesting Program (OSTP)

0-1350 points

Attendance Rate0-150 points

Baseline API

0-1500 points

Oklahomas API: Point Contributions for High Schools and K-12 Districts

Example: Kentucky

The Commonwealth Accountability Testing System (CATS) goal for every school in the state is Proficiency

by 2014 as defined by the Kentucky Board of Education. The goal of Proficiency translates into a school

accountability index value of 100. Intermediate targets that will eventually take a school to the goal of 100

are set biennially starting in 2002. As such, there are seven biennia or accountability cycles between 2002

and 2014, as well as recognition points.

Both academic content-based and non-academic measures are used in CATS. These measures include

custom, criterion-referenced Kentucky Core Content Tests (KCCT) in reading, mathematics, science, social

studies, arts and humanities, practical living/vocational studies and writing, as well as a nationally norm-refer-

enced test in mathematics and reading.

Non-academic measures include attendance rate, retention rate, dropout rate and transition-to-adult-life.

Transition-to-adult-life data is collected in the fall of each year via a short survey completed by school

personnel. This includes the number of graduates planning to enter the work force, the military, a college or

a vocational/ technical school.

These multiple measures provide a snapshot of schools and communicate to schools strengths and

weaknesses in order to allocate resources and analyze instructional programs. Each of the measures is

combined into a composite to obtain a schools Accountability Index. The CATS goal for every school inthe state is Proficiency as defined by the Kentucky Board of Education. The goal of Proficiency translates

into a school Accountability Index value of 100. The State goal is for each school to achieve an account-

ability index of 100 by 2014.

The CATS Accountability Index calculation begins with simple number-correct raw scores and ends with

an accountability index that summarizes a schools progress toward the states goal of Proficiency. Raw

scores give rise to scale scores; scale scores have been related to Novice, Apprentice, Proficient and Dis-

tinguished (NAPD) performance levels (via standard setting and cut-scores); NAPDs get weighted numeri-

cally and combined within each content area; the content areas are weighted and combined to form a

schools academic index; and finally, the academic index is combined with the norm-referenced test (NRT)

and non-academic factors to generate the accountability index.


42/117


Oklahoma School

Testing Program (OSTP)

0-1200 points

Dropout Rate

0-50 points

Graduation Rate

0-50 points

Attendance Rate

0-50 points

School Completion

0-150 points

College Remediation Rates

0-15 points

Advanced Placement (AP)

0-45 points

ACT

0-90 points

Academic Excellence

0-150 points

Baseline API

0-1500

OK state law requires that a minimum 60 percent weight be assigned to the OSTP. After due consideration

of the merits of the various combinations, weights were assigned based on site grade span (e.g., elementa-

ry) and district type. Table 1 contains the weighting for each component by grade span and district type.

For example, in a K-12 district, the OSTP has a weight of 80 per

Documents

R-Key Elements for Educational Accountability Models