Inferring Adequate Yearly Progress of Schools from Student Achievement in Highly Mobile Communities

This article was downloaded by: [Northeastern University]On: 06 October 2014, At: 04:57Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH,UK

Journal of Education forStudents Placed at Risk(JESPAR)Publication details, including instructions forauthors and subscription information:http://www.tandfonline.com/loi/hjsp20

Inferring Adequate YearlyProgress of Schools fromStudent Achievement in HighlyMobile CommunitiesRobert M. OffenbergPublished online: 16 Nov 2009.

To cite this article: Robert M. Offenberg (2004) Inferring Adequate Yearly Progressof Schools from Student Achievement in Highly Mobile Communities, Journalof Education for Students Placed at Risk (JESPAR), 9:4, 337-355, DOI: 10.1207/s15327671espr0904_2

To link to this article: http://dx.doi.org/10.1207/s15327671espr0904_2

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all theinformation (the “Content”) contained in the publications on our platform.However, Taylor & Francis, our agents, and our licensors make norepresentations or warranties whatsoever as to the accuracy, completeness,or suitability for any purpose of the Content. Any opinions and viewsexpressed in this publication are the opinions and views of the authors, andare not the views of or endorsed by Taylor & Francis. The accuracy of theContent should not be relied upon and should be independently verified withprimary sources of information. Taylor and Francis shall not be liable for anylosses, actions, claims, proceedings, demands, costs, expenses, damages,and other liabilities whatsoever or howsoever caused arising directly orindirectly in connection with, in relation to or arising out of the use of theContent.

http://www.tandfonline.com/loi/hjsp20

http://www.tandfonline.com/action/showCitFormats?doi=10.1207/s15327671espr0904_2

http://www.tandfonline.com/action/showCitFormats?doi=10.1207/s15327671espr0904_2

http://dx.doi.org/10.1207/s15327671espr0904_2

This article may be used for research, teaching, and private study purposes.Any substantial or systematic reproduction, redistribution, reselling, loan,sub-licensing, systematic supply, or distribution in any form to anyone isexpressly forbidden. Terms & Conditions of access and use can be found athttp://www.tandfonline.com/page/terms-and-conditions

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

04:

57 0

6 O

ctob

er 2

014

http://www.tandfonline.com/page/terms-and-conditions

Inferring Adequate Yearly Progress ofSchools from Student Achievement in

Highly Mobile Communities1

Robert M. OffenbergAccountability, Assessment, and Evaluation: Research and Evaluation

School District of Philadelphia

Many attempts at educational reform, among them No Child Left Behind and a re-cent Philadelphia effort, assume that the quality of the educational programs beingoffered by schools can be inferred from the achievements of the children who attendthem. This article explores the reasonableness of this assumption for Philadelphiapublic schools by following a cohort of students for 3 years after the completion of1st grade, when most students would conclude 4th grade. It examines the rates ofschool-to-school mobility and exit from schools. Using a family of hierarchical mod-els, it explores how 1st-grade report card marks predict the odds and character ofwithin-district school-to-school transfers, and the odds of student exits from the pub-lic schools. The analyses then examine the effects of two 1st-grade school–commu-nity variables, a poverty index and a performance index, on the mobility of students.The study found high mobility prevalent in most schools. Moreover, combinations ofstudent and school–community variables typically associated with the need forschool improvement were present at schools with the highest mobility rates. The arti-cle concludes that a policy of inferring the success of school-based educational en-deavors from school-level statistics can often be invalid in urban school districts dueto student mobility, with the risk of error likely to be the greatest at the schools wherereform is most needed and No Child Left Behind sanctions are most likely.

The strategy for improving the educational attainment of students in the UnitedStates is often expressed as improving schools; if schools were better at teachingstudents, the population would be better educated. Currently, this attitude is being

JOURNAL OF EDUCATION FOR STUDENTS PLACED AT RISK, 9(4), 337–355Copyright © 2004, Lawrence Erlbaum Associates, Inc.

Requests for reprints should be sent to Robert M. Offenberg, Research and Evaluation, School Dis-trict of Philadelphia, 2120 Winter Street, Philadelphia, PA 19103. E-mail: [email protected]

1Aversionof thisarticlewaspresentedat the2003meetingof theEasternEvaluationResearchSociety.

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

04:

57 0

6 O

ctob

er 2

014

reflected by the 2002 reauthorization of the Elementary and Secondary EducationAct, No Child Left Behind. The act requires each state to develop a 12-year plan ofAdequate Yearly Progress (AYP) steps for every school that culminates in all stu-dents attending it for most of the school year scoring as proficient in reading andmathematics on standardized tests. No Child Left Behind then requires states to as-sess AYP of each school each year and to identify those not meeting its criteria 2years in a row. A school failing to meet the state standards is allowed to appeal itsbeing identified, and if the appeal is unsuccessful, it is required to develop an im-provement plan. A school failing to meet state standards several years in a row issubject to sanctions, among them restructuring the school and giving parents theoption of moving their children to another school at district expense (Council ofGreat City Schools, 2002; House of Representatives, 2001).

This approach to improving student achievement is clearly school-centered. No-where does it acknowledge that student achievement is influenced by factors out ofschool, despite research showing that nutrition, family functioning, exposure to tox-ins, and a host of community factors also influence student behavior and achieve-ment (see Del Gaudio Weiss & Fantuzzo, 2001). So when action must be taken, NoChild Left Behind remedies focus on the organization, personnel, and instructionalstrategiesof schools. In the spirit ofNoChildLeftBehind,Pennsylvaniawarned884schools statewide that they were at risk because they failed to meet federal standardson its 2002 standardized tests (Langland, 2003). The Chancellor of New York CityPublic Schools gave 50 school principals “report cards” with Fs because students attheir schools had unsatisfactory test scores, absenteeism, and suspension rates (Me-dina, 2003). And, when a joint state–city School Reform Commission in Philadel-phia replaced the locally appointed Board of Education during the summer of 2002,it addressed low achievement at 86 or 39% of the public elementary and middleschools by changing them in one of four ways: privatization, staff reconstitution, re-organizationascharters,orprovisionofadditionalper-pupil funds—with thegoalofcreating a variety of instructional approaches each presumed to be capable of yield-ing greater student achievement than the programs they replaced.

The spirit of No Child Left Behind legislation and the policies underlying theseefforts are based on the belief that equilibrium exists between the services a schoolprovides and the attainment of its students. A corollary to this belief has becomethe basis of the performance criterion built into No Child Left Behind—if efforts toimprove the quality of a school have not produced improvements in scores of stu-dents on the criterion measures specified by the act, then they must have failed, andremediation is required. Using historical data, this study explores a common char-acteristic of urban schools, substantial levels of student mobility arising when stu-dents transfer among schools and exit from the system. This study will make thecase that the link between the programs offered by a school and its attendees’ per-formance on criterion measures is, at best, tenuous. It will show that ascribing stu-

338 OFFENBERG

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

04:

57 0

6 O

ctob

er 2

014

dent achievement to the operation of schools is often inappropriate, and implemen-tation of reforms that ignore student mobility is poor educational policy.

The idea that student school transfer and system exit rates could affect the ap-parent performance of schools is not new, but it is still not treated appropriately.Ten years ago, a discussion of ways to conceptualize student mobility indexeswas motivated by concerns that the effects of freedom of choice, site-based man-agement, and discontinuity of learning could be mistaken for the effects of pro-grams on student achievement (Ligon & Paredes, 1992). More recently, a Gen-eral Accounting Office (GAO) House of Representatives report on theeffectiveness of private education companies running schools in the District ofColumbia rejected three of the five reviewed studies because, among other rea-sons, there were high attrition rates due to student mobility without correspond-ing adjustments (Shaul, 2002).

Many studies of urban districts begin by assuming that a student’s act of movingfrom school to school or leaving a school district has a damaging effect on chil-dren’s achievement, but then find disconfirming evidence. After reviewing severalof these studies and carrying out their own five-year study in Baltimore, Alexan-der, Entwisle, and Dauber (1996) concluded that students who moved had lowertest scores, poorer marks, and elevated risks of retention and special education;however, there was only weak support for the belief that mobility caused these out-comes. Once background and early achievement variables were introduced, thevalue of mobility as a predictor of the later scores was reduced and fell short of sig-nificance in most of the analyses where school-to-school mobility and exit rates atfirst seemed to predict later test scores.

A study contemporaneous with the Alexander et al. (1996) article suggestedthat the apparent lack of effect of mobility on achievement in these studies wasthe result of assuming that mobility causes low achievement, when in fact lowachievement is part of the constellation of factors that predict mobility. Kerbow(1996) was concerned that mobility created educational disjuncture that reducedthe effectiveness of local school reform in Chicago. He found that mobility wasso prevalent that the core of many elementary schools’ first-grade student bodiesdisappeared over time. The mobility rates were the highest in the schools thathad a high number of at-risk students enrolled. Kerbow discussed the instabili-ties created by these processes, and how despite being widely known, these in-stabilities tend to be ignored when discussion turns to school reform. He thenshowed how student mobility could lead to mistaken beliefs about the effective-ness of school reforms.

This study explores whether student mobility could jeopardize the attributionof educational outcomes to schools. It focuses on discovering how prevalentschool-to-school transfers and system exits were among Philadelphia publicschool students, whether the incidences of these events were influenced by stu-

AYP, STUDENT ACHIEVEMENT, AND MOBILITY 339

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

04:

57 0

6 O

ctob

er 2

014

dents’ early achievement, and whether they were moderated by the social classand the academic standing of students’ first-grade schools. The implications offindings for local and national policies—assuming that school effectiveness canbe inferred from student achievement—are then discussed.

METHOD

This study was based on students who were in first grade at the inception ofChildren Achieving, the reform strategy of the previous administration of theSchool District of Philadelphia. It followed a cohort of students for a period of 3years. It took a parsimonious approach that did not attempt to fully explain the ori-gins of either student mobility or achievement. Rather, it explored whether a hand-ful of student and institutional variables available for nearly every urban publicschool but weakly related, if at all, to the educational programs provided byschools, could distort the performance standard of No Child Left Behind.

The entire population of schools offering regular, elementary-grade educationin the School District of Philadelphia was in the study, which followed a cohortcomprising virtually all students who were in Grade 1 in the district for the firsttime in 1994–1995. The goal throughout the study was to come as close to study-ing all students in this cohort as possible, recognizing the vagaries of a system notdesigned for research purposes. As a result, the students for each study componentwere chosen so that findings would be based on the largest student-pool possible,with cases excluded only because of missing data.

Students and Student Variables

The study followed students from June 1995, when members of the cohort com-pleted their first year in first grade, through June 1998, when most students com-pleted fourth grade. The students could not be followed any further without someof these students moving to middle schools. The cohort comprised 18,225 studentsattending all 172 of the district’s regular-education elementary schools. This wasvirtually the entire population of first-grade students in the Philadelphia schooldistrict who attended one school for 180 of 185 days of first grade.

Grade Point Averages. Most of the study used end-of-first-grade, ma-jor-subject Grade Point Averages (GPAs) computed from marks on report cards inuse in 1994–1995—the only academic achievement measures that were generallyavailable for students this early in their careers. Six schools and some students atother schools who did not have a complete set of marks in Reading, English, Mathe-matics, Science, and Social Studies had to be excluded from analyses where GPAwasused; thus, thestudycomponentsusingmarkswerebasedonfirst-gradestudents

340 OFFENBERG

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

04:

57 0

6 O

ctob

er 2

014

in 166 schools instead of the original 172. The excluded schools did not use tradi-tional first-grade report cards. The largest groups of students who did not have com-plete sets of marks despite attending schools using traditional report cards are be-lieved to be English Language Learners and mainstreamed special educationstudents. In total, there were 14,725 students in the analyses based on GPA. Theirmean GPA was 2.6, with a standard deviation of .97. The mean was just above a C+lettergrade; thestandarddeviationwasequivalent toa little less thana lettergrade.

Mobility. Two binary student mobility outcome variables were explored: 1)Whether the student’s first move, if any, during the 3 years following the comple-tion of first grade was a transfer to another school within the district, and 2)whether the student’s first move during the 3 years following completion of firstgrade was an exit from the school district.

There were 13,958 students in the school-to-school transfer analysis, 5,342(38%) of whom changed schools as their first school-mobility act. There were9,401 students in the school-district exit analysis, 767 (8%) of whom left theschool district as their first school-mobility act. In both analyses, there was a groupof 8,634 students who remained in one school continuously, neither transferringnor exiting. They served as the control group to whom both the transferring and ex-iting students were compared.

For students who transferred school-to-school within the district, a third mobilityoutcomevariablewasexplored: thedifferencebetween the1995–1996PerformanceIndex of the school the student attended in first grade and the school to which the stu-dent moved. The Performance-Index-difference variable had a positive value whenstudents moved within the school district to a school that in 1995–1996 had out-per-formed the one being left, a negative value when the move was to a school that un-der-performedtheonebeing left.Forstudents remaining in thesameschool through-out the study, the difference was equal to zero. As the distribution ofPerformance-Index-difference values is a result of the student mobility processesbeing studied, the character of this variable will be discussed in the Results.

School Level Variables

The relationship between first-grade GPAs of students and their mobility was ex-pected to be influenced by the character of the school they attended during firstgrade. The two school–character variables, 1995–1996 Performance Index and in-cidence of poverty, were chosen because they were the best, most readily availablemeasures of the school–community environments, in which the first-grade stu-dents found themselves at the beginning of their educations. The two school-levelvariables were the following.


Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

04:

57 0

6 O

ctob

er 2

014

The 1995–1996 Performance Index. The Performance Index is a measureof the school environment in which the students spent first grade. Because this wasthebaselineyearof the index, itsvaluepredates theeffectsof theChildrenAchievingschool improvement effort and its innovations. Stanford 9 Achievement Tests(SAT-9) in Reading, Mathematics, and Science administered to students in Grades 4and 8 accounted for most of the variance of the index of elementary schools. Studentattendance, staffattendance,andpromotionrateswere its remainingcomponents.

All of the components comprising the 1995–1996 Performance Index were ex-pressed as an extended scale of Performance Levels conceptually akin to the perfor-mance levelsof theSAT-9 thatwerederivedfromstudent scoresby the testpublisher.The original SAT-9 Performance Levels range from Below Basic to Advanced infour categories. To better measure the achievement of lower performing students inPhiladelphia, theBelowBasiccategorywasdivided into threeequal segments tocre-ate an expanded scale with a maximum value of six. Expanded performance-levelscales paralleling those of the SAT-9 but based on local criteria were then created forstudent attendance, staff attendance, and promotion rates. Each index componentvalue of a school was based on the percentage of its students or staff at each of its per-formance categories, multiplied by the values assigned to the category.

The overall Performance Index score of a school was the sum of its three testand three non-test performance values, weighted to make the student attendanceand staff attendance components half as important as the others. The mean Perfor-mance Index of the 172 schools being studied was 67.5 points, with a standard de-viation of 7.2 points.

Test-score components were computed from fourth and eighth grade scores, notscores of children in this study; therefore, the students in this study had no influ-ence on test-based components of the 1995–1996 Performance Index value. Thestudy participants comprised only a minority of the students whose attendance andpromotion rates were considered and had little or no influence on its teacher com-ponent. Thus, the index was a measure of the school community that could serve asan independent or causal variable.

Incidence of poverty. The incidence of poverty is a measure of the socialclass of the student body of each school. It was computed from the 1994–1995 per-centages of low-income students at the schools where the members of the cohortattended first grade. The values were derived from the percentage of students whohad resided at addresses where someone received public assistance and the per-centage of students who had been eligible for federally supported free or re-duced-price lunches the previous year. As with the Performance Index, incidenceof poverty was a measure of the school community that could serve as an inde-pendent variable for exploring the behavior of the students being studied. Themean poverty incidence of schools was 57.2%, with a standard deviation of 20.9%.

342 OFFENBERG

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

04:

57 0

6 O

ctob

er 2

014

Data Analyses

Summary statistics were computed, then Hierarchical Linear Modeling (HLM) wasused to understand how student achievement, school character, student transfer, andstudent exit were related (Raudenbush, Bryk, Cheong, & Congdon, 2000). HLM isused when data are organized into levels. In these analyses, students-within-schoolscomprised the first level. Within each school, it was assumed that students’1st-gradeGPAs reflected an array of unstudied prognosticators of student mobility. Amongstudents who transferred school-to-school within the district, 1st-grade GPA wasalso hypothesized to be an important determinant of whether the student moved to ahigher or lower performing school than the one being left.

The second level of the HLM analyses explored whether the relationships be-tween GPAs and student moves were influenced by the character of the schoolswhere the student attended first grade, measured by Performance Index and inci-dence of poverty values. The results of the two levels of each analysis were joinedcomputationally and illustrated on graphs showing trends for schools whose1995–1996 Performance Index values were at the district mean, one standard devi-ation above it, and one standard deviation below it. Together, outcome predictionsbased on the two analysis levels and displayed on the graphs reflect the behavior ofstudents in the study as a function of their first grade GPA and the character of theschools in which they attended first grade.2

The HLM analyses used population-centered forms of all independent vari-ables. For GPA, the population-centered value of each student was the differencebetween his or her value and the average GPA of all students being studied. ForPerformance Index and incidence of poverty, the population-centered value ofeach school was the difference between the school and the average of all schools.As a result of centering, zero always refers to the mean of the school district. In theResults tables, positive numbers are above the district mean whereas negativenumbers are below the district mean. Population-centered scales were also used inthe figures graphically displaying the findings.

Two of the outcome variables, the incidence of student transfers to anotherschool and the incidence of students leaving the district, were ratios of odds. Tocompute appropriate analyses, the Bernoulli form of the odds-ratios—the naturallogarithm of their values—was used. HLM-analysis tables of these variables dis-play results based on the Bernoulli form of their dependent variables. However, thelogarithmic transformations were removed from odds-ratios when they were dis-cussed in the text and displayed in graphs to make their meaning clearer.


2The HLM analyses were computed twice. In the first set, full explanations of student behavior bythe Performance Index and the index of poverty were examined. In the second set, the only relationshipsincluded were ones where statistically significant findings were likely. The second set of analyses ispresented in this report because it makes meaningful trends in the data more evident.

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

04:

57 0

6 O

ctob

er 2

014

RESULTS

Transfer and Exit Rates of Schools

The incidences of students transferring and exiting the school district were high.At an average school, 40.2% of the students completing first grade did not attendthe same school continuously for the next 3 years. The range was from 6.1% of thestudents in the most stable school to 69.6% in the least stable school.

The first action of a student was much more likely to be a transfer to anotherschool than an exit from the school district.3 The average student-transfer rate was36.4% of the students who completed first grade at a school; however, it ranged be-tween 2.6% and 65.5%. In contrast, the average exit rate of all schools was 3.7% ofthe students who completed first grade at them. The school exit rates ranged from0.0% to 16.5%.

All combinations of transferring and exiting were found, including students re-turning to their 1st-grade school after transferring or exiting. Many students movedseveral times; therefore, these findings are under-estimations of the overall preva-lence of student mobility.

Although this study was of events occurring from 4 to 7 years before school re-form decisions were made in Philadelphia, the mobility rates it found were clearlyrelated to the later decisions. The schools chosen to be privatized, reconstituted,granted charters, and given additional funding grants in 2002 had histories ofhigher within-district transfer rates than other schools between 1995 and 1998, butlower rates of school-district exit. On average, 43.8% of the students in theto-be-reformed group of schools transferred, compared to 33.6% in the otherschools (F = 18.3, df = 1/170, p < .001). The difference between the two groupswas partly due to very low rates of transfer among the students attending schoolsthat were not to be reformed. Although, the lowest uninvolved-school transfer ratewas 2.6% of the cohort being studied, the lowest transfer rate among the schools tobecome a part of the reform effort was 26.7%. Clearly, the social forces resulting inhigher rates of student transfers preceded the identification of schools judged to bein need of reform; instead, they may reflect community forces that contributed tothe selection of the schools.

The schools to be reformed had much lower rates of district exit than otherschools. Even though the average rate of exit from schools to be reformed was1.8%, the average of schools that were not going to be reformed was 4.5% (F =14.3, df = 1/170, p < .001). In the case of exit rates, the difference between the twogroups of schools was partially due to high exit rates among students attendingsome of the schools not chosen for reform. Whereas the highest rate found at

344 OFFENBERG

3When a student both transferred to another school and exited the district in the same year, theschool transfer was assumed to have preceded the district exit. The entire cohort of students was used toderive overall school-to-school transfer and exit statistics, as report card marks were not required.

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

04:

57 0

6 O

ctob

er 2

014

not-chosen schools was 16.5%, the highest rate among the to-be-reformed groupwas only 6.3%. As these district exit rates, like the transfer rates, precede later re-form decisions, they too may reflect community factors contributing to school-re-form selection decisions.

Predicting Within District Transfers

The high incidence of school-to-school transfers and system exits observed insome schools suggests that holding them responsible for the low achievement oftheir students may be inappropriate, especially if mobility is associated with stu-dent and school–community characteristics. The following hierarchical analysisshows that the odds of a student transferring to another school were, in fact, influ-enced by the school–community variables. (A similar analysis of the odds of stu-dents exiting from the district will be presented later on.)

The first level of the school-transfer hierarchical analysis predicted, within eachschool, the Bernoulli or natural logarithm of the odds of students transferring fromfirst-grade, major-subject GPAs. There was, of course, a first-level transfer predic-tion equation for each school. The next level examined whether the relationshipsbetween student transfer and GPA captured by the equations were the same for allschools, or whether they were influenced by the two school–community variables,Performance Index and incidence of poverty. Table 1 shows the second-level re-sults. Figure 1 illustrates the trends the findings imply—the odds of a student trans-ferring to another school as a function of the student’s own GPA and the Perfor-mance Index and the incidence of poverty of the student’s school. In the figure andthe discussion of the meaning of the findings shown in the table, the Bernoullitransformation used in the computation was removed so the odds of a studenttransferring (not the logarithm of the odds) would be evident. Both the analysis andthe figure derived from it show those low-achieving students who were in low-per-forming, high-poverty schools were more likely to move to another school thanwere other students.

The Student with Average GPA section of Table 1 describes the behavior of thestudents in the school district with average marks. Because the independent vari-ables had been centered, the Intercept coefficient of this section means that, whenthese Average-GPA students attended an average Performance Index school in anaverage index-of-poverty community, they were two times as likely to remain inthe same school as transfer. The Performance Index and Poverty Index weightsshown in this part of the table indicate that being a first-grade student at schoolswith lower poverty or a higher performance decreased the odds that an average stu-dent would transfer, whereas attending a higher poverty or lower performanceschool increased them.

The Effect of Student GPA section of the table shows that the behavior of theaverage student cannot be generalized to students whose first-grade report card


Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

04:

57 0

6 O

ctob

er 2

014

marks were not average. The Intercept coefficient of this section of the table showsthat having an above-average GPA decreased the odds of a student transferring.The School Performance Index coefficient shows that if this above-average studentattended a high-Performance-Index school, the odds of transferring were de-creased even more. Conversely, being a low-GPA student in a low-Perfor-mance-Index school increased the odds of students transferring to another school.

346 OFFENBERG

TABLE 1Odds of Student Transferring to Another School Within the District as a

Function of Student GPA, School Performance Index, and School Poverty Index

Effect: Natural Logarithm of theOdds of Students Transferring

Approximate

Coefficient t df p <

Students with average GPAIntercept –0.591007 –18.945 163 .000School performance index –0.017761 –2.833 163 .005School poverty index 0.016869 8.273 163 .000

Effect of student GPAIntercept –0.362515 –16.626 164 .000School performance index –0.019211 –5.433 164 .000

Note. Grade Point Average (GPA) from letter grades: A = 4, B = 3, C = 2, D = 1, F = 0.

FIGURE 1 Odds of a student transferring to another school in district.

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

04:

57 0

6 O

ctob

er 2

014

Figure 1 displays the joint effect of students’ GPA and the two school variableson the odds that a student would transfer. To accommodate the four dimensions onthe graph, three Performance Index levels are shown as separate planes: one stan-dard deviation above the district average, the district average, and one standard de-viation below it. The three planes all show similar trends. As the table suggests,students with below-district-average GPAs in higher poverty schools were themost likely to transfer, although students with higher GPAs attending lower pov-erty schools were the least likely to transfer. As the Performance Index of a schoolincreased, the trend got stronger; however, in schools at all three graphed Perfor-mance Index levels, there were combinations of low student GPA and high schoolpoverty, which made students more likely to transfer than to stay.

The effects described by the analysis and the graphs are educationally meaning-ful. As shown in the left-rear corner of Figure 1, odds-of-transferring greater than1.4 were predicted for students with GPAs that were about 1.5 standard deviationsbelow the district average who also attended schools where the Poverty Index wasabout 1.5 standard deviations above the average, regardless of whether they at-tended low-, average-, or high-performing schools. This means that, for every Per-formance Index level graphed, there were combinations of poverty levels andfirst-grade GPAs predicting that 14 students had transferred for every 10 who didnot. In the front-right corner of the graph, there were odds-ratios of 0.3 on all threegraphed surfaces—for students with GPAs about 1.5 standard deviations above theschool-district average who were attending schools where the poverty level wasabout 1.5 standard deviations below the district average. This means that, for allschool performance levels, there were combinations of first-grade GPAs andschool poverty levels predicting that only three students had transferred for every10 who remained. Earlier, this report discussed how district-wide school-to-schooltransfer rates might be too high to justify holding some schools responsible for theachievement of students who were tested there. These findings suggest that thelow-performing students in high-poverty schools, the students whose needs schoolreforms are supposed to address, will be over-represented among the transferees.

Prediction of School Performance Index Differences WhenStudents Transfer

Typically, when students transferred from one school to another they moved toschools with Performance Index values slightly better than the schools they left.The magnitude of the difference can be viewed in two ways. The 1995–1996 Per-formance Indexes of the new schools of the 5,324 students who transferred withinthe district averaged 1.2 points above the Index values of the schools they left. Thestandard deviation associated with this average difference was 7.48 points. Themean school Performance Index difference among the 13,958 students comprisingthe population that was able to transfer because they remained in the district, in-


Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

04:

57 0

6 O

ctob

er 2

014

cluding the students who remained in one school throughout the study, was .47with an associated standard deviation of 4.66. The discrepancy between the twosets of values is the presence of students who did not transfer, and therefore had a1995–1996 Performance-Index difference of zero in the second set.

The goal of this HLM analysis was to determine whether, among the nearly14,000 students who could have transferred, the pattern of school PerformanceIndex differences was systematically related to the school and student variablesbeing studied. Table 2 shows the analysis, whereas Figure 2 illustrates the rela-tionship to students’ first-grade GPAs and the character of the schools the stu-dents had attended.

The first section of Table 2 describes the transfer rates of students whosefirst-grade report card marks were at the school district average. The statisticallysignificant positive intercept shows that if these average students attended an aver-age Performance Index, average poverty-level school, then they were more likelyto transfer to a school that was a little better than the one they left; however, theschool-to-school transfer process of these average students depended in part on thecharacter of the school where they attended first grade. Average GPA students whocompleted first grade at schools with higher Performance Indexes tended to trans-fer to schools with lower Indexes than the school they left, even though those start-ing at low Performance Index schools tended to transfer to higher Index schools.This pattern was also found for poverty values, with average GPA students whowere first graders at high-poverty and low-poverty schools tending to move toschools where the poverty levels of students were less extreme. Regression to themean, a statistical artifact predicting that students who began at schools with ex-treme Performance Index and poverty values would tend to end up at schools withless extreme values, probably contributed to the average-GPA students’ transferpatterns. This was not the sole explanation, however, because regression effectswould not have been influenced by the GPA of other, non-average students in theway shown in the next section of the table.

The second section of the table, Effect of Student GPAs, shows that having hadabove- or below-average first-grade report card marks altered the previous pattern.Together, the significant intercept, School Performance Index coefficient, and theschool poverty coefficient show that the trends for average students discussedabove were weakened when students had high marks in first grade; however, thetrends were strengthened when students had low marks.4

348 OFFENBERG

4Prior school-district evaluations of the School Performance Index showed that, like most goodmeasures, it was reasonably reliable but imperfect. Difference scores based on an imperfect measuretend to be less reliable than the measure itself, so results of an HLM model based on them might be mis-leading. To address this issue, the School Performance Index differences for each student were com-pared to its predicted value from this model. The correlation obtained was highly significant (r = .391,df = 13,958, p < .001). It showed that 16% of the Performance Index difference variance was ex-plained—a very strong outcome given the many factors involved in school transfer decisions and thelarge number of students who did not move.

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

04:

57 0

6 O

ctob

er 2

014

The joint effect of these trends on the transfer rates of students to higher orlower performing schools than the ones they left is difficult to envision from the re-sults displayed in Table 2. Figure 2 addresses this issue by showing the pattern ofSchool Performance-Index differences resulting from the school-to-school trans-fers of students that the HLM analysis implied. The three surfaces again distin-guish among students who were in low-, average-, and high-performing schools at


TABLE 21996 School Performance Index Differences in Schools-to-School Transfers

as a Function of Student GPA, School Performance Index, and School Poverty

Effect: Difference between Index ofSchools (New–Previous)

Approximate


Students with Average GPAIntercept 0.248791 3.509 163 .001School peformance index –0.343041 –16.554 163 .000School poverty –0.049630 –8.994 163 .000

Effect of Student GPAIntercept 0.403119 8.721 163 .000School performance index 0.099400 8.006 163 .000School poverty 0.014647 3.908 163 .000


FIGURE 2 Difference in Performance Index of the school the student left and the school towhich the student has transferred.

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

04:

57 0

6 O

ctob

er 2

014

the end of first grade. When a student remained at the first-grade school throughoutthe study or transferred to a school with a Performance Index that equaled the in-dex of the first-grade school, the Performance Index difference was zero. The re-gion above the zero contains the students who went to schools with higher Perfor-mance Indexes than the schools they left; the region below the zero containsstudents who transferred to schools with lower indexes than the ones they left.

Figure 2 shows that students who moved from low Performance Index schoolstended to go to schools with higher Index values than the one they left; however,the trend was not uniform. The difference between the two schools was greater forstudents who left lower poverty schools. The greatest difference was for studentsin the group who had been low achieving early in their school careers. This patternis consistent with the hypothesis that in middle-class communities, parents whowere dissatisfied with their children’s achievement or school conditions tended tomove them to other, higher Performance Index schools, especially when the chil-dren were not succeeding academically.

In contrast, students who were first graders in high Performance Index schoolstended to go to schools that had lower Performance Index values than the schoolsthey left; again, this trend was not uniform. Students who were from low-povertycommunities, especially students who also had better first-grade marks, tended totransfer between schools with similar Performance Indexes; on the other hand, stu-dents who began at higher performing schools in higher poverty communitiestended to move to lower performing schools than the ones they left. This effect wasthe strongest for students whose first-grade achievement was also low. Thesetrends are consistent with the hypothesis that when high Performance Index valuesare found in schools serving high-poverty communities, the high Performance In-dex may be a by-product of students with poor marks leaving them.

As expected, the pattern for students who were first graders in schools at thedistrict-average Performance Index was a blend of the high-performing andlow-performing school trends.

The analysis in the previous section showed that the character of students whoare likely to transfer away from a school can affect school-based outcomes. Thisanalysis suggests that the nature of the school to which a student transferred wassomewhat predictable from his or her previous achievement and the character ofthe school from which he or she came. It implies that Adequate Yearly Progressand other similar school-effectiveness measures are likely to be influenced by theachievement levels and histories of the students who tend to transfer to a school.

Predicting System Exit

The final HLM analysis examines the odds of students exiting from the district ver-sus staying at the same school throughout the study. Preliminary analyses showed

350 OFFENBERG

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

04:

57 0

6 O

ctob

er 2

014

the odds were strongly related to first-grade GPAs and school–community income,but the Performance Indexes of first-grade schools had virtually no predictivevalue. Moreover, the effect of a student’s GPA on the odds he or she would exitfrom the district was not influenced by the poverty of students attending his or herschool. As a result, the HLM-based prediction of student exit from the school sys-tem shown in Table 3 and Figure 3 was simpler than were the school-transfer mod-els discussed in the preceding sections.

The table and figure show that the odds of an average student in an averageschool exiting from the district were low (about nine students exiting for every 100that remained in the same school throughout the study). Attending a low-povertyschool and having a low first-grade GPA each increased the odds of students leav-ing the district. Unlike findings for school-to-school transfers, the HLM model inTable 3 suggests that these two motives for exiting the district were independent. Ifa student was both low achieving and in a low-poverty school, the effects were ad-ditive but they did not strengthen each other (i.e., there was no interaction in whichschool index of poverty value changed the relationship between GPA and odds ofexiting from the school district).

The trend of students exiting low-poverty schools was consistent with a so-cial-class hypothesis: Families in middle-class neighborhoods are more likely tohave the resources to move to suburbs or to choose non-public education. Thetrend of students with low GPAs exiting is consistent with a hypothesis that somefamilies attribute the low GPAs of their children to urban, public school education,regardless of the social class of the community in which they live. They respond bymoving from the city or enrolling their children in other types of education (e.g.,home schooling, private schools).

IMPLICATIONS OF THE FINDINGS

Despite the prevalence of contradictory evidence, No Child Left Behind and manyother educational reform policies view trends in the achievement of successive co-horts of students passing through a school as if they were unbiased measures of in-stitutional change—assuming that successive sets of cross-sectional student datainevitably form valid, longitudinal trends. The predictable, high rates of studentmobility comprising the major findings of this article suggest that in Philadelphia,and probably in all urban school districts, this view is not justified. The achieve-ment of students who happen to be attending a school cannot always be used to as-sess the quality of the educational experience it provides to them. Thus, the defini-tion of AYP specified by No Child Left Behind can result in values and multiyeartrends that are unrelated to the quality of the program being delivered by schools.


Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

04:

57 0

6 O

ctob

er 2

014

Educational improvement plans generally assume that if the achievement of thestudents at a school changes in accordance with a new strategy, then the new strategycaused the change. This study suggests conditions where this may not be the case.Anything that modifies the relationship between one of the study predictor variablesand either student transfer or exit rates can change apparent school performancewithout changing its effectiveness. Linn and Haug (2002) found that traditionalschool accountability index scores, which compare the achievement of successive

352 OFFENBERG

TABLE 3Odds of Leaving the District as a Function of Student GPA and School Poverty

Effect: Natural Logarithm of Oddsof Students Leaving District

Approximate


Students with Average GPAIntercept –2.442399 –54.912 163 .000School poverty –0.013012 –6.386 163 .000

Effect of student GPAIntercept –.103424 –2.855 164 .005


FIGURE 3 Odds of a student leaving the school district: leaving versus staying.Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

04:

57 0

6 O

ctob

er 2

014

student cohorts, were unstable. Changing relationships among the variables in thisstudy can contribute to the instability that they observed. For example, an innovationthat convinced parents to have their lower achieving students continue at a schoolrather than transfer would make school-level statistics decline even if the innovationhad no effect. A second example comes from one of the provisions of No Child LeftBehind. As noted at the beginning of this article, when a school fails to meet its AYPimprovement goals, parents acquire the right to move their children to anotherschool; the probability that this right will be exercised is likely to be related to thevariables in this study, hence likely to affect annual AYP values and trends.

There are at least four approaches to evaluating efforts designed to enhance stu-dent achievement that reduce or eliminate the effects of school-to-school mobility.None of them addresses school-system exit satisfactorily, and each has characteris-tics that require rethinking the goals of No Child Left Behind. They are multiplemeasures of program effectiveness, aggregation of scores to multi-school groups,student mobility weighting, and value-added approaches:

• Multiple Program Effectiveness Measures. The finding that student mobilitycan invalidate inferences of school effectiveness from student achievement sug-gests that No Child Left Behind should have required direct assessment of schools’delivery of educational reforms and additional measures of the program outcomes.Specification of standards for instructional strategies to be used, evidence that theywere implemented, and outcomes, in addition to AYP, should have been required.Then, should a school fail to meet AYP-based performance standards, because ofmobility or other factors not explored in this study, more valid, less draconian ac-tions than those in No Child Left Behind could be considered.

• Aggregation of Scores. The risk that high rates of student mobility will resultin AYP values not reflecting the educational program being provided is greaterwhen the population of a district is divided among many small schools and stu-dents can move easily from school to school. Thus, the findings of this study are, inpart, the result of there being 172 elementary schools in Philadelphia with mostserving compact neighborhoods. If AYP were redefined as a regional performancestandard rather than a school performance standard, the effect of mobility wouldbe mitigated to the extent that transfers occurred within regions. In Chicago,Kerbow (1996) found students tended to transfer among schools forming natural,regional clusters. In Philadelphia, schools are divided into within-district aca-demic regions serving fairly homogenous parts of the city. If No Child Left Behindallowed regional goals instead of school goals, it would encourage educationalstrategies and remediation alternatives recognizing urban mobility and reducingthe effects of mobility on AYP.

• Student Mobility Weighting. When high rates of student mobility at someschools were first considered in Philadelphia, one proposal was to reduce the re-


Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

04:

57 0

6 O

ctob

er 2

014

sponsibility of schools for students who did not attend them continuously byweighting scores by the amount of time a student was on the roster at a school.This strategy seems fair to schools and permits school-level results to be com-puted. However, the findings of this study suggest that this policy might also ob-scure the effects of schools on students who should be the principal beneficiariesof school reform.

• Value-Added Approaches. These approaches focus on student-by-studentgains, and whether they exceed expectations. They project the achievement of eachstudent over a short period, usually one school year, from his or her growth, history,and background. The difference between the projection and actual achievement isthe “value added” (Archer, 1999; Sanders & Horn, 1998). Individual student datacan be used to estimate the typical, short-term, value-added effect of a school. Be-cause it takes social factors and early achievement into account, value-added ap-proaches may be the fairest to mobile students, but because expectations are relatedto individual studenthistories, it isnotagoodstrategyforevaluatingprogress towardthe broad, “all students will be proficient” societal goal of No Child Left Behind.

In a politically charged, reform-oriented school district environment, the attri-bution of the achievement of mobile students to schools is likely to become an is-sue. Should the evaluation of a school with an innovative program be limited to thestudents who participated in it continuously? Should a school be held accountablefor all students present for assessment, even though many of their competencieswere acquired elsewhere? The way these questions are addressed can lead to over-estimating the effectiveness of schools by seriously undercounting students need-ing the most help, or by underestimating it and holding schools responsible for stu-dents whose achievement they had little opportunity to affect; neither is anappropriate policy.

Our society grants parents high levels of independence and rights to act in wayseducational planners, politicians, and government policymakers often do not antici-pate; the findings of this article suggest that local reform efforts and the creators ofNo Child Left Behind did not give adequate attention to this fact. As a result, they de-veloped school-based performance standards that are likely to yield misinformationabout many schools, impose unjustified penalties, and grant unearned rewards.

ACKNOWLEDGMENTS

Dr. Robert Sebastian, Barbara Holden, Coleen Kavanaugh, Catherine Conroy,Katherine Walker, Dmitry Pavolotsky, and Nicholas Angers contributed to thepreparation of text, data files, and graphics.

354 OFFENBERG

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

04:

57 0

6 O

ctob

er 2

014

REFERENCES

Alexander, K. L., Entwisle, D. R., & Dauber, S. L. (1996). Children in motion: School transfers and ele-mentary school performance. Journal of Education Research, 90, 3–12.

Archer, J. (1999). Sanders 101. Education Week, 18(34), 26–28.Council of Great City Schools. (2002). Initial analysis of Elementary and Secondary Education Act

Reauthorization. Washington, DC: Author.Del Gaudio Weiss, A., & Fantuzzo, J. W. (2001). Multivariate impact of health and caretaking risk fac-

tors on the school adjustment of first graders. Journal of Community Psychology, 29(2), 141–160.House of Representatives. (2001). No Child Left Behind Act of 2001, Conference Report to Accompany

HR 1, Report 107–334. Washington, DC: U.S. Government Printing Office.Kerbow, D. (1996). Patterns of urban student mobility and local school reform [Report No. 5]. Balti-

more, MD: Center for Research on the Education of Students Placed at Risk, Johns Hopkins Univer-sity.

Langland, C. (2003, January 7). Schools are warned. Philadelphia Inquirer. Retrieved on January 29,2004 from http://www.philly.com/images/philly/inquirer/4884/23858221784.gif.

Ligon, G., & Paredes, V. (1992, April). Student mobility rate: A moving target. Paper presented at theannual meeting of the American Educational Research Association, San Francisco, CA. (ERIC Doc-ument Reproduction Service No. ED 349 335)

Linn, R., & Haug, C. (2002) Stability of school-building accountability gains. Educational Evaluationand Policy Analysis, 24(1), 29–36.

Medina, J. (2003, January 4). Chancellor gives out report cards, failing 50 principals. The New YorkTimes, B2.

Raudenbush, S., Bryk, A., Cheong, Y. F., & Congdon, R. (2000). Hierarchical linear and non-linearmodeling. Lincolnwood, IL: Scientific Software International.

Sanders, W. L., & Horn, S. P. (1998). Research findings from the Tennessee Value-Added AssessmentSystem (TVASS) Database: Implications for educational evaluation and research. Journal of Person-nel Evaluation in Education, 12, 247–253.

Shaul, M. S. (2002). Public schools: Insufficient research to determine the effectiveness of selected pri-vate education companies [Appendix 2, Document GAO-03-11]. Washington, DC: U.S. General Ac-counting Office.


Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

04:

57 0

6 O

ctob

er 2

014

Documents

Inferring Adequate Yearly Progress of Schools from Student Achievement in Highly Mobile Communities