Embed Size (px)
Citation preview
1
The Evolution of Accountability Reporting:Lessons Learned Over Nine Years By a State Education Agency
Mindy Crain-Dorough, PhDSoutheastern Louisiana University
Robyn Madison-Harris, EdDLouisiana Department of Education
2
Overview of presentation
Background on report development and what Louisiana does
History of Accountability Changes Problems/Solutions Recommendations for future directions
3
Purpose of this study
To compile a list of lessons learned as a reference for other states
To document the history of accountability reporting in Louisiana
To explain the complexities of the report development process
To provide recommendations for improvements accountability reporting in Louisiana and other states
4
Report development – Not an easy thing to do
Present complicated information and data in a format that is aesthetically pleasing and understandable
Negotiate many conflicting opinions about content Turn complicated and detailed policy into computer
programs that calculate scores Coordinating many large data files Extensive checking of all aspects of calculations and
report generation has to be conducted. All of these must be done in a very rigid timeframe.
5
Why do we go to all this trouble? Not just to meet federal and state mandates Parents make decisions about which schools
to which they will send their children and even where to live.
Parents are also driven to take action at their children’s schools.
Teachers, principals, and district staff use the results to make decisions about school improvement efforts and funding priorities
6
Louisiana’s choices
Reports produced at the state level for districts and schools
School Report Card for Parents in paper format and sent home with every child
Electronic PDFs of reports are available on LDE website
Assessment and accountability reporting are separate
LDE works with a contractor to produce reports
7
Back to the beginning
Louisiana had report cards as far back as 1996-1997 as part of a legislative mandate
Transformed report cards into accountability reports in 1998-1999
8
School Report Card for Parents
9
Inside
10
Back
11
Changes to reports over the years
Year Change Incurred Impact on Reports
SY 1998-1999 First year of school accountability Complete overhaul2 new reports
SY 1999-2000 Interim year – no new labels reportedScience and Social Studies data reported
Addition of new tables and text
SY 2000-2001 Grades 9-12 entered accountabilityFirst year growth was evaluated
Complete overhaul
SY 2001-2002 District Accountability data first reported (1 year lag)
New district reports
SY 2002-2003 Incorporated NCLB mandates Complete overhaul1 new report
SY 2003-2004 Graphic design overhaul of all reports Complete overhaul
SY 2004-2005 No major changes
SY 2005-2006 Adjustments for hurricanesIncorporation of iLEAP
Complete overhaul
SY 2006-2007 Incorporation of graduation rate and indexRevised school sanction classifications
Addition of new tables and text
12
Along comes NCLB
First reports containing NCLB mandates in 2002-2003
Louisiana had to incorporate NCLB into policy and thus the existing reports
A dual system created – SPS Component and Subgroup Component
Conducted an evaluation of existing reports for what was and was not working
Prioritized contents Put highlights in report cards and details in a new
report
13
Incorporation of NCLB
2001-2002 Reports
Added in
2002-2003
Intended Audience
NCLB Requirements
Included
School Report Card for Parents
Parents General information on dual system, table with Subgroup AYP information
School Report Card for Principals
Principals, faculty, and district staff
More details than Parent report card on dual system and Subgroup AYP
School Accountability Report (“Press Report”)
Media, researchers, and the general public
Major changes to format
Subgroup Component Report
Principals, faculty, and district staff
All other NCLB requirements
14
School Report Card for Parents
15
Inside
16
Back
17
Methodology
Qualitative – authoethnographic, case study, and phenomenological approaches
Data Collection Authoethnographic recount by authors Interviews of current and former accountability
staff Document analysis
Data Analysis – cross-case examination and content analysis strategies
18
Problems/Solutions
Accountability system changing every year Be organized Planning meetings with key parties Preparation for planning meetings
“Regrouping” at the end of each year
19
Problems/Solutions Errors in the reports
Improve the checking process every year using a log of problems found in previous year(s)
Include cross-report consistency checking in the process Create thorough processing rules, “data maps,” and record
layouts that work together Communication between report developers, data calculation
staff, and the contractor on these issues is key Allow districts to review data in aggregated form before
inclusion in reports, especially for new indicators Have emergency plans in place should errors occur Use everyone’s time effectively – hire temps if necessary to
do more tedious checking
20
Problems/Solutions
Too many conflicting opinions about content Use focus groups to obtain evidence of what
works best for consumers Develop a protocol for making decisions
regarding conflicting viewpoints
21
Problems/Solutions
Miscommunication within a state agency Is the agency structure conducive to functional report
development? Develop an issues database to track issues and their
resolutions as well as a process for maintaining the database
Develop a thorough timeline with a process for ensuring everyone is on task and a protocol for revising the timeline should unexpected events arise
Find a method of meeting regularly to discuss issues and timelines that is most effective for all involved – we are currently having weekly conference calls with the contractor and impromptu meetings in-between
22
Problems/Solutions
Vendor relations Good vendor qualities include flexibility, availability to
devote extensive time to the project, advanced technical skills, and good quality assurance practices
Balance of working for us and working with us Establish protocol for staff turnovers and management
changes Set standards and roles for quality assurance
procedures and processes Regular communication and established
communication procedures
23
Problems/Solutions
New mandates (internal and external to state) Evaluate the current reports for what works
and what doesn’t Prioritize contents of the report (current and
anticipated) Take advantage of the opportunity to make
enhancements to the reports that are undergoing major revision anyway
Look at what other states are doing to get ideas
24
Problems/Solutions
Disasters occur Meet with those most impacted by the events
to obtain all information before making decisions
Consider all aspects of the accountability system and brainstorm possible impacts
25
Problems/Solutions
Giving one individual too much responsibility Have a team of report developers Rotate lead responsibility for individual reports
every two or three years
26
Problems/Solutions
Promoting and supporting report release Use Press Conferences, written press
releases, and radio announcements Create a “Help Desk” to answer questions
about accountability and other agency programs
Provide accountability trainings on policy changes, the importance of data quality, and description of report contents to school and district staff
27
Problems/Solutions
Making reports useful Brainstorm areas in the agency that can be
highlighted in the report cards Provide a message to parents in the report
card from the superintendent to inspire parents to become more involved in their child’s education
28
Validity and reliability
Validity Confirm valid use of reports through focus
groups and training Reliability
Thorough and detailed checking process
29
The current “Parent Report Card”
30
Recommendations
Transition to interactive accountability reporting and tie to school improvement efforts
Combine assessment and accountability data Reduce the number of releases and release
the Parent Report Cards earlier Re-evaluate the content and purposes of the
current reports Have an “Accountability Czar”
