ASSESSMENT AND ACCOUNTABILITY LAB PRESENTATION

An Analysis Approach to Pinpoint Underperformance

in School Districts

Dr. Xin Ma, Professor, University of Kentucky

Spencer Fellow of (U.S.) National Academy of Education

(Former) Canada Research Chair

Accountability Is Not to Rank School Performance but to Monitor School Performance

• Recent work for the Appalachian Regional Education LAB (Appalachian REL) provides the foundation for an analysis approach to pinpoint underperformance in school districts.

• Current work for the P20 Assessment and Accountability LAB aims to develop an analytical framework for data-informed decision-making.

• Ultimate goal is to help school districts develop a monitoring system for school performance based on a data-informed decision-making framework.

• First things first: Descriptive statistics based on KCCT data generates a package of profiles for each school district in Kentucky (Appalachian REL).

Organisation for Economic Co-operation and Development (OECD) Pushes for Academic Performance of Students in Membership Countries

OECD (2010) relates academic performance to economic growth (p. 6).• This relationship indicates that relatively small improvements in the skills of a

nation’s labour force can have very large impacts on future well-being.• A modest goal of having all OECD countries boost their average PISA scores

by 25 points over the next 20 years—which is less than the most rapidly improving education system in the OECD, Poland, achieved between 2000 and 2006 alone—implies an aggregate gain of OECD GDP of USD 115 trillion over the lifetime of the generation born in 2010.

• Bringing all countries up to the average performance of Finland, OECD’s best performing education system in PISA, would result in gains in the order of USD 260 trillion.

• Other aggressive goals, such as bringing all students to a level of minimal proficiency for the OECD (i.e. reaching a PISA score of 400), would imply aggregate GDP increases of close to USD 200 trillion according to historical growth relationships.

Blue Ribbon Panel Pushes for Academic Performance of Students in Kentucky

• The Kentucky Board of Education has expressed serious concern with the large number of middle and high schools that do not appear to be on track to reach proficiency by 2014.

• Kentucky is in a desperate search for ANY strategy (no matter how drastically different from conventional ones) as long as it can “improve student performance as quickly as possible” (Blue Ribbon Panel, 2008, p. 7).

• Middle and high school performance is ranked as one of the top issues by the Kentucky Department of Education.

• One of the most effective strategies is to pinpoint the “origin” or “root” of underperformance in a school district.

Accountability and Assessment LAB Helps School Districts Improve Academic Performance of Students

The LAB takes a data-informed decision-making approach to help school districts• Monitor the relational patterns of academic performance among elementary,

middle, and high schools in a school district.• Identify where underperformance occurs in a school district during the two

transitions in learning:– from elementary school to middle school– from middle school to high school.

Appalachian Regional Education LAB Takes an Analytical Approach to Profile Reading, Mathematics, and Science Performance in a School District

• Using data on academic performance in elementary, middle, and high schools in a school district, we can create profiles of academic performance of students across elementary, middle, and high schools.

• These profiles can help identify where gaps in performance occur in a school district during the two transitions in learning.

• For example, a middle school “feeds” its students into a high school. If the performance is high in the middle school but low in the high school, then a district leader would want to look closer at what is happening at this transition point.

(Not verified for meeting IES standards; Not for distribution.)

2008 Kentucky Department of Education (KDE) Kentucky Core Content Test (KCCT) Provide Data for Profiles

• Students’ Kentucky State Percentile (SKSP) score is aggregated to the school level as the chief measure of school performance.

• Using the school average SKSP, indices are developed to measure the direction and strength of academic progression from elementary school to middle school as well as from middle school to high school.

• KDE classifies Performance Level (PL) of students into: Novice Low, Novice Medium, Novice High, Apprentice Low, Apprentice Medium, Apprentice High, Proficient, and Distinguished.

• Proportion of Proficient and Distinguished Students (PPDS) is calculated for each school and used as another chief measure of school performance.

(Not verified for meeting IES standards; Not for distribution.)

Accountability and Assessment LAB Addresses Advantages and Disadvantages of School Profiles

• Main advantage is that profiles help quantify transition gaps in academic performance in a school district.

• Accountability and Assessment LAB seeks to develop a longitudinal approach to examine the stability of the relational pattern of school performance across elementary, middle, and high schools in a school district.

• Main disadvantage is that profiles are descriptive (without any test of substance).

• Accountability and Assessment LAB seeks to bring in significance tests to focus on substantial patterns for more relevant policy implications.

Follow-Up Efforts to Help School Districts in Kentucky to Develop A Monitoring System for School Performance

• Work with individual school districts to refine school district profiles base on more accurate feeding patterns from elementary to middle to high schools.

• Train school districts personnel to develop a monitoring system for school performance.

• Provide technical assistance to school districts to maintain a monitoring system for school performance.

Final Remark:

NO LONG-TERM EDUCATIONAL PROGRESS

CAN BE POSSIBLY SUSTAINED

WITHOUT A CREDIBLE MECHANISM

TO MONITOR SCHOOL PERFORMANCE