Developing an Early Warning System: Identifying Factors that may Predict High School Completion by 6th Grade Students

8/14/2019 Developing an Early Warning System: Identifying Factors that may Predict High School Completion by 6th Grade S

1/32

Developing an Early Warning System: Identifying

Factors that may Predict High School Completion

by 6th Grade Students

(Revised 3/25/2009 1:46 PM)

Prepared by

Kenneth T. Wilburn, Ph.D.

University of North Florida

Marcia Lamkin, Ed.D.

University of North Florida

Dax M. Weaver, MPH

President, Health-Tech Consultants, Inc.

Prepared For

The Community Foundation of Jacksonville

For additional information, contact

Ms. Pam D. Paul, Vice President, Community Initiatives

The Community Foundation in Jacksonville

121 West Forsyth St., Suite 900, Jacksonville, FL 32202

(904) 356-4483, www.jaxcf.org
http://www.jaxcf.org/http://www.jaxcf.org/http://www.jaxcf.org/


2/32

EarlyWarningSystem 2Executive Summary

Purpose

The primary purpose of this study was to identify specific academic and social factors

that may be used as flags to identify Duval County Public Schools (DCPS) 6th grade students

who are at high risk of not completing high school. A secondary purpose was to understand

how these factors may or may not be aligned with those identified by Neal, Balfanz, and other

researchers (Neal and Balfanz, 2006).

Four questions guided our 6th grade analysis: (1) Are significant numbers of students

showing the signs of disengagement previously identified in our examination of 9th grade

students who fail to complete high school? (2) Can we identify a set of indicators that flag sixth

graders who have high odds of failing to complete high school? (3) From this set of indicators,

can we identify a practical set of indicators routinely collected and reported at the individual level

that flag sixth graders who have odds of failing to continue in school until graduation? (4) Can

we then identify a set of those indicators that may be influenced by school and community

programs?

Data

The data used in this analysis were provided by the DCPS Department of Instructional

Research and Accountability. The data were gathered from the districts data warehouse and

provided to the research team in the form of several Excel spreadsheets. The data included

selected demographic, academic, attendance, and disciplinary information from each student

enrolled in grades 612 during the 20012002 through the 20072008 school years.

Approximately 60,00065,000 students were enrolled in grades 6-12 during each school year

with each grades enrollment averaging between 10,00012,000 students.

Since the focus of this analysis was 6th grade students, we began by identifying all 6th

grade students for the 20012002 school year and then tracking that cohort of students through

each school year through the 12th grade (2007-2008). Ten thousand ninety-three (10,093)

students remained for inclusion in the analysis after the elimination of withdrawals, transfers,

and incomplete data sets.


3/32

EarlyWarningSystem 3

Methods and Procedures

Once the data set had been cleaned of cases with missing data or incorrectly coded

data, an initial factor analysis was conducted to determine whether variables fell into coherent

categories (i.e., factors) as predictors of completion. No one piece of data, such as attendance

or mathematics standardized test scores, has enough weight to serve as a valid predictor.

Therefore, this process of combining various data sets was repeated until a consistent set of

categories was identified. Once a coherent set of predictors was identified, linear regression

(stepwise) and logistic regression analyses were conducted on each identified factor in order to

estimate the strength of each factor.

Results

A significant number of 6

th

grade students displayed the signs of disengagement, suchas poor academic achievement and excessive absences, previously identified in our study of 9 th

grade students and in studies conducted by other researchers. Five factors proved to be of the

greatest value in identifying members of the 2001-2002 6th grade cohort who did not graduate

on-time from high school in 2007-2008. Each flag was a statistically significant flag (p< .05)

even after controlling for the other flags. The five flags listed below, individually and in

combination, identified 81.52% of the non-graduates.

Standardized Math Test Scores: 69% of the 6th grade students who failed to graduate

on-time scored below Level 3 on the Florida Comprehensive Assessment Test. Standardized Reading Test Scores: 56% of the 6th grade students who failed to graduate

on-time scored below Level 3 on the Florida Comprehensive Assessment Test.

Failing Course Grades in Mathematics: 30.2% of the 6th grade students who failed to

graduate on-time failed a math course.

Failing Course Grades in English/Language Arts: 28.1% of the 6th grade students who

failed to graduate on-time failed an English/Language Arts course.

Excessive Absences: 12.2% of the 6th grade students who failed to graduate on-time

missed more the 20% of the scheduled school days.

The number of Class 3 and 4 Code of Conduct violations occurring at the 6th grade among

the study population was not very large (3%). As a result, this flag only identified 6.7% of the 6th

grader who failed to graduate on-time. However, our tracking of students indicated that the

number of Class 3 and 4 violations increase significantly as students get older and as more


4/32

EarlyWarningSystem 4students become over-age for their assigned grade. Consequently, the conduct violations from

subsequent years have more predictive power than those from Grade 6. While the yield for this

factor was relatively small as a single factor and no greater when considered in combination

with test scores or grades, we nevertheless consider this to be an important flag to monitor as it

becomes a key indicator in later grades.

Results from our study indicated

1. Significant numbers of 6th grade students show signs of disengagement previously

identified in our examination of 9th grade students who fail to complete high school.

2. We can, with a reasonable degree of validity, identify a set of indicators that flag sixth

graders who have high odds of failing to complete high school.

3. From this set of flags, we can identify a practical set of indicators routinely collected and

reported at the individual level that flag sixth graders who have odds of failing to

continue in school until graduation.

4. These indicators are factors that may be impacted by school and community programs.

In conclusion, as we have learned from our previous studies of first-time 9th grade

students and focus groups with students who had previously dropped out of school, students do

not graduate from high school for a multitude of reasons. Many of those reasons are not linked

directly to academic achievement, attendance, and behavior at school. However, our research

supports the work by Balfanz (2006) and others (Balfanz, Hertzog, Neild, and MacIver, (2007))

that there is ample, readily available data for early identification of students with the greatest risk

of failing to graduate high school.


5/32


Table of Contents

Page

INTRODUCTION ... 7

BACKGROUND.....

A Descriptive Study of First-time 9th Grade Students

Focus Groups with Dropouts, Parents and Service Providers

8

METHODS AND PROCEDURES

Guiding Questions

Outcome Variable

Data Source

Sample and Measures

Data Analysis Procedures

12

FINDINGS...

Identifying Warning Flags

Attendance

Standardized Test Scores

Course Grades

Discipline

Status Variables

Final Selection of Identification Flags

Validity of the Flags

Limitations of the Study

17

SUMMARY.. 26References.. 30

Appendix: Demographic Characteristics of the Study Population. 31


6/32

EarlyWarningSystem 6List of Tables Page

Table 1. School Related Variables of the Study Population 14Table 2. Graduate and Non-graduate Percentages Identified by Preliminary Flags 19Table 3. Double Warning Flag Combinations and Counts for all Duval 6thGrade

Students in 20012002 Who Failed to Graduate in 2007-2008 19


7/32


Introduction

This study is the third in a series sponsored by The Community Foundation of

Jacksonville Florida (CF) designed to examine issues related to Duval County Public School

(DCPS) students who fail to graduate from high school. The first study, Where Have All the 9th

Graders Gone?(Wilburn, Weaver, and Wilburn, 2008),was a descriptive study of a cohort of

first-time 9th grade students. The study was designed to provide the local community with a

more detailed understanding of the non-completion problem in Jacksonville. The second study,

Kids Say the Darndest Things (Roush and Wilburn, 2008), was a qualitative analysis of focus

group data which had been collected by The Community Foundation and their community

partners in the Learning To Finish Project. These focus groups were conducted with 1624

year old students who had been identified by the school district as being of high risk of dropping

out, currently enrolled in a school or community sponsored dropout prevention program, or had

dropped out of school and were incarcerated in the Duval County jail for a criminal offense. As

with the descriptive study, the primary goal of the effort was to gain a richer understanding of

students who have dropped out of the public school system. This study used the information

learned in the first two studies to develop an early warning system that could be used by the

community to identify the students, at the end of the 6th grade, who are the most at risk for

failing to complete high school.

The report will be organized into the following four sections:

Background. This section provides a review and summary of the 9th grade descriptive

and focus group studies and how the outcomes from those studies were used to inform

the 6th grade predictive study.


8/32

EarlyWarningSystem 8 Methods and Procedures. This section describes the methods and procedures used to

conduct the 6th grade predictive study.

Findings. This section reports the results from the predictive analysis and their

application to the development of an early warning system as well as the limitations of

the study.

Summary. The final section reviews the main points of the study.

Background

In this section we provide a summary of the two studies we conducted as precursors to

beginning the process of developing flags for identifying 6th grade students with a high

probability of not completing high school.

A Descriptive Study of First-time 9thGrade Students

The first study was designed with the overall goal of developing a better understanding

of issues surrounding the graduation crisis and the dropout problem found in DCPS. Simply

stated, the design of the study was to follow three cohorts of students from the time they first

entered the 9th grade through their 4th or 5th years of high school. Using data provided by the

DCPS Department of Research and Accountability, we were able to select a study population

that met our criteria for inclusion in the analysis. As recommended by Balfanz (2007), we sought

to study cohorts of 9th grade students who had been enrolled in DCPS during the previous

grades, were enrolled in the 9th grade for the first time, and were engaged in an academic

program for which a standard high school diploma was a reasonable expectation. Additionally,

we targeted a study population that was reflective of the overall 9th grade population in gender,

race, and socioeconomic status.


9/32

EarlyWarningSystem 9Graduation and Dropout Rates. The analysis indicated that by the end of the 3rd year of

high school, approximately 488 students (2.5%) of the original first-time 9th grade cohort had

received their diploma. By the end of the 4th year, the number of students who had graduated

increased to 11,842 students (60.74%), and by the end of the 5th year, the students who had

received a standard high school diploma was 11,901 (61.04%). While approximately 61% of the

first-time 9th grade students eventually graduated, 95% of those did so in the customary 4th year

of high school. Only a small percentage (4.1%) of the students graduated early, and an even

smaller percentage (0.005%) graduated after a 5th year of high school. In addition to tracking

graduates and non-graduates, we also identified the number and characteristics of students

labeled dropped out (i.e., did not withdraw or transfer but just failed to show up the next school

year). Our calculations indicate that the number of students who dropped out increased with

each succeeding cohort. The 2002 cohort had an estimated dropout rate of 25.5%, followed by

29.5% for the 2003 cohort, and rising to a high of 31.3% with the 2004 cohort. When examined

as a whole, the estimated dropout rate for the combined three student cohorts was

approximately 28.2%. What this means in terms of students lost is that over the 4 years of high

school almost 2,000 students left each cohort without receiving diplomas.

Focus Groups with Dropouts, Parents and Service Providers

In 2007-2008, under the direction of Dr. Shannon Perry, chair of the projects qualitative

focus group, seven focus groups met to explore the dropout situation in Jacksonville.

Information from these sessions were transcribed and provided for analysis to Dr. Connie

Roush, a qualitative research specialist with the Brooks College of Health at the University of

North Florida. In this report, qualitative themes were identified with highlighted responses from

each focus group and quotes provided to illustrate some of the major points. The major themes

identified were participants attitudes and beliefs about the dropout situation; the influence of the

family, school, and community environments; specific issues related to peer pressure; the


10/32

EarlyWarningSystem 10culture of violence; the need to know and care for students; and accountability for school

attendance. Each group of participants approached the focus groups from a different point of

view or attitude shaped partially by their own beliefs about what spurs school disengagement

and dropout.

Major Themes. Each group reinforced the belief that getting to know a student takes

continuous contact over time and a lot of listening. This lays the groundwork for a trusting and

caring relationship necessary to (as one student said) attract us to school and keep us

there. Listening was a topic most often addressed by the middle school and incarcerated

youth while all groups offered wisdom about the caring relationship and the time that it takes to

evolve. Another theme our analysis identified was the power of the need to fit in to school

culture and/or the wrong crowd. Whether it is with other students in school or with groups in

the neighborhood, peer pressure is extremely strong. A third theme identified was the culture

of violence that is so much a part of students lives. Many students described their

disengagement from school as being sucked in to using and selling drugs and the violent

lifestyle that goes with it. While the family emerged as a critical theme, it was seen by many

teens as equally as positive and negative as a factor. For many, the discussion centered around

the barriers put up by family to be supportive. In a smaller but equally powerful part of the

discussions, students spoke about the people who motivated and inspired them to stay in

school. In the course of reports on all previous themes, the influence of the school environment

on disengagement and dropout is evident. There was a great deal of concern by all

participating groups regarding the familys communication with the school, conditions that

prompt children and teachers to stay at a school, and the need for a variety of educational

options. While all emphasized that the lines of communication must be open for parents,

students, and the school to work together, few could provide any positive examples. Most of the

examples were negative in nature describing the problems and consequences of poor


11/32

EarlyWarningSystem 11communication. In regard to perceptions of the community, little time in the focus groups

examined this in particular. However, most comments conveyed the message that the

community was not aware and did not understand why students are leaving school and was not

doing anything to address the problem. During the focus groups, participants explored the issue

of school attendance from multiple perspectives. Students described skipping school and

being kicked out of class; parents were concerned about timely notification of absences; and

all participants discussed the need for accountability. One other topic of concern was the

enforcement of attendance policies and the potentially negative consequences for students with

special circumstances.

Taking Action. Students, parents, and teachers discussed the different programs that

can lead to graduation, the need to graduate on time, and the pros and cons of graduation

options (the usual high school diploma, the G.E.D. pathway, and the certificate of graduation).

They also discussed the experience of transferring to an alternative program (specifically the

Pathways program). All three groups agreed that the high school diploma was the most

desirable of the three. According to one teacher, the high school certificate has a major impact

on the community as well as the students because it limits their employment opportunities.

Another teacher stated that it basically indicates, I went to school for 12 years but I did not earn

a high school diploma. When comparing the G.E.D. to the diploma, one parent told her child,

Some places wont take the G.E.D., so it is better for you to get your high school diploma. It will

open more doors. From the perspective of a student, programs to help students stay in school

were described this way:

When we get funding we try to make the kid fit into the grant so we arent actually

working on what needs to be worked on. The people who are giving the money or

disbursing the money do not understand the population to which they are giving. Schools


12/32

EarlyWarningSystem 12try to make the money fit in the programs but it doesnt work (Roush and Wilburn, 2008,

p. 17).


Guiding Questions

Four questions guided our 6th grade analysis:

1. Are significant numbers of students showing the signs of disengagement previously

identified in our examination of 9th grade students who fail to complete high school?

2. Can we identify a set of indicators that flag sixth graders who have high odds of failing to

complete high school?

3. From this set of indicators, can we identify a practical set of indicators routinely collected

and reported at the individual level that flag sixth graders who have odds of failing to

continue in school until graduation?

4. Can we identify indicators within that set that may be influenced by school and

community programs?

Outcome Variable

The outcome variable used was whether or not the sixth graders in the 20012002

cohort graduated from high school on time or after one extra year. We chose this outcome, in

part, because of our desire to replicate the study reported by Balfanz, Herzog, Neild, and

MacIver (2007) which previously identified a set of sixth grade predictor variables. In addition,

we selected this variable because in the previous analysis of 9th grade students, the vast

majority of students who earn a diploma did so on time or within one additional year. Therefore,

extending the time frame would be of little significance. It is also important to note that we


13/32

EarlyWarningSystem 13selected to use graduating from high school as opposed to dropped out of school since each

student who receives a standard high school diploma is clearly identified with a high degree of

accuracy in the districts data system; however, students who fail to graduate can do so for a

multitude of reasons (e.g., transfers, withdrawal, failing to report to school) that are not captured

with a high degree of accuracy by the districts data system.

Data Source

As with previous studies, the data used in this analysis were provided by the DCPS

Department of Instructional Research and Accountability. The data were gathered from the

districts data warehouse by a district staff member and provided to the research team in the

form of several Excel spreadsheets. The data included selected demographic, academic,

attendance, and disciplinary information from each student enrolled in grades 612 during the

20012002 through the 20072008 school years.

Sample and Measures

Since the focus of this analysis was 6th grade students, we began by identifying all 6th

grade students for the 20012002 school year and then tracking that cohort of students through

each school year including 12th grade (20072008 school year). We created a longitudinal

dataset designed to follow the performance of students enrolled in Grade 6 during the 2001-

2002 school year. The dataset included attendance, demographic components, math and

reading course grades, and math and reading standardized test scores per year. These

students were traced through the 2007-2008 school year, the year during which students

normally complete Grade 12 and graduate or finish a program. Those still enrolled as DCPS

students at the end of the 2008 school year were categorized as either Graduated with

diploma, Received certification of completion, or Transferred/withdrew. These categories

were added to the dataset.


14/32

EarlyWarningSystem 14In our data file 10,516 students were identified as being in Grade 6 for the 20012002

school year. From this initial population, 423 students were deleted due to incomplete data sets

which resulted in a data set of 10,093 students for our study population. This sample of

students in Grade 6 consisted of White (47.7%), African-American (44.3%), Hispanic (3.7%),

Asian-American (2.8%), and Multiracial (1.3 %) students. Slightly more students were male

(51.5%) than female (48.5%) and slightly more than eight percent (8.3%) of these sixth graders

required services as English Language Learners. For the purpose of this study, students in the

two most severe classifications of special education (those who were not expected to graduate

or to earn a certificate of completion) were eliminated from the sample. The remaining ESE

students (those who spent at least part of the school day in regular education classes) were

handled as regular education students.

No data about free or reduced lunch eligibility were available for about half the students

in the Grade 6 population. Data for 5,015 of the students indicated that 94% qualified for free or

reduced lunch (about half of the total Grade 6 sample). Of the complete study population

(10,093), only about 1% was younger than expected in the Florida public schools, while 41.5%

were of normal age for the grade level and 58.4% were over the expected age indicating that

they had been retained at some earlier grade level. At the end of the 2008 school year, 66.7%

(6,733) of the students graduated by 2008 while 33.3% (3,360) did not.

Table 1 provides an overview of the study population in regard to the school related

variables such as program participation, disciplinary infractions, course grades and

standardized test scores.

Table 1. School Related Variables of the Study Population

Group Variable Completers Non-completers

Drop-OutProgram % Enrolled in Dropout PreventionProgram 12.4 28.2


15/32

EarlyWarningSystem 15% Not Enrolled in a Drop-OutPrevention Program

87.6 71.8

% Serious Violation (Code 3/4) 6.4 11.3Discipline

% No Serious Violation (Code 3/4) 93.6 88.7

Reading 2002 Mean Score 1721.67 1698.17

Math 2002 Mean Score 1724.63 1741.99


Math 2003 Mean Score 1819.72 1826.28


Math 2004 Mean Score 1907.80 1772.70


StandardizedTest Scores(FCATDevelopmentScale Score)

Math 2005 Mean Score 1941.38 1793.39

Language Arts 2002 Mean Grade 2.57 1.98

Math 2002 Mean Grade 2.47 1.83






End ofCourseGrade GPA


As presented in Table 1, there are significant differences in academic related

characteristics between those 6th grade students who graduated from high school and those

who did not. For example, for those students who were enrolled in a dropout prevention

program, only 12.4% graduated high school while 87.6% of those who were not in the program

earned their diploma. Among students who committed a serious conduct violation, a much

larger percentage did not complete high school. There were also significant differences

between completers and non-completers on the Developmental Scale Scores of the Florida


16/32

EarlyWarningSystem 16Comprehensive Assessment Test (FCAT). When compared over the four years students were

tracked, those who completed high school had higher reading scores each of the four years and

higher math scores two of the years. While the differences were not always great enough to be

significant, the final course grade point averages (GPA) in language arts and mathematics were

also higher for high school completers for each of the four years.

Data Analysis Procedures


data, an initial factor analysis was conducted on the 23 different variables available in the Duval

dataset to determine whether variables fell into coherent categories (i.e., factors) as predictors

of completion. One of the reasons for beginning with this data reduction procedure was to

determine if one piece of data, such as attendance or mathematics standardized test scores,

had enough weight to serve as a valid predictor or would combinations of variables serve as

stronger predictors. The process of combining various data sets was repeated until a consistent

set of categories was identified. Once a coherent set of predictors was identified, linear

regression (stepwise) analyses were conducted on each identified factor in order to estimate the

strength of each factor.

The factor analysis yielded eight variables strong enough to be included in the regression

analysis. These are as follows:

1. End-of-sixth-grade standardized test scoresin reading and mathematics were used.

Developmental scores on the Florida Comprehensive Achievement Test (FCAT) were

converted to one of five individual levels in each student record, and the five levels were

employed in statistical testing.


17/32

EarlyWarningSystem 172. Although the nature of the courses varied from student to student, end-of-sixth-grade

course grades in English and mathwere included. Letter grades were converted to

numeric equivalents for the purpose of statistical testing.

3. Students who had incurred serious infractions of the conduct code (major violations or

violations of zero tolerance policies) were identified and labeled dichotomously from

students who had not incurred such infractions.

4. Absencesduring Grade 6 were calculated and included. No distinctions were made

among reasons for absence: a simple total of days missed was initially used. Based on

indications from previous studies, in subsequent analyses, absences were divided into

two categories: less than 20% of the school days in 2001-2002 and more than 20% of

the school days that year.

5. Graduation statusfor each yearup to and including the routine six years of high school

were aggregated and used as a single variable. Students were designated as

Graduated with diploma, Received certification of completion, or

Transferred/withdrew by the end of the 2007-2008 academic year.

6. Participation in dropout prevention programsduring Grade 6 only was included. Duval

County maintains seven different prevention programs: educational alternatives, dropout

retrieval, disciplinary, alternatives to expulsion, teen pregnancy, Department of Juvenile

Justice, and neglected or delinquent. Students were labeled by the specific program in

which they participated or as non-participants in any of these programs.

7. Demographic variablessuch as race, gender, and date of birth were included. Dates of

birth were subsequently converted to age in months and then divided into three

categories: under expected age, normal expected age, and over expected age.

8. The need for services as English Language Learnerswasidentified and coded

dichotomously (services/no services) into the dataset.


18/32

EarlyWarningSystem 18Findings

Identifying Warning Flags

As we worked through the identification procedure, it became apparent that the most

valuable warning flags from an educational standpoint would be those that could be impacted by

the school and/ or community based educational programs. These are (1) measures of

academic learning, (2) school attendance, (3) student behavior, and (4) program placement.

Consequently, some demographic variables (e.g., race, gender, primary language) were

dropped from further analysis.

Table 2 shows the yield of each of the eight preliminary predictors. Seven of these

independent flags met our test of being within the influence of the educational program and

accounting for at least 10% of the students who failed to graduate, but one did not.

(1) Over-age for 6th grade

(2) Earned a low FCAT score (Level 1 or 2) in math

(3) Earned a low FCAT score (Level 1 or 2) in reading

(4) Failed a 6th grade math course

(5) Failed a 6th grade English course

(6) Was enrolled in a Dropout Prevention Program

(7) Absence from school more than 20% of the time

(8) Receiving a serious (Code 3 or 4) disciplinary referral (This flag only identified 6.7% of

the non-graduates. It is possible that this flag has some value but failed to meet the two

pronged test because of the low number of students in the overall population who met


19/32


Table 2. Graduate and Non-graduate Percentages Identified by Preliminary Flags

Absence>20%

VCode LowFCATMath

LowFCATReading

Over-age for6

thGrade

DPP Failed 6thgrademathcourse

Failed 6thgradeEnglishcourse

Graduatedon time

158 96 4099 3157 3470 341 1097 860

Did notgraduate

409 207 2313 1878 2423 517 1016 946

Yield: %of non-graduatesflagged

12.2 6.7 68.8 55.9 72.1 15.4 30.2 28.1

N = 10,093 6th

grade students/3,360 Non-graduates

One of the lessons learnedfrom our qualitative analysis of interviews with students who

had dropped out of school was that often there was no single factor that prompted students to

leave school. As Balfanz has reported, examining the occurrence of multiple flags and their

impact on students graduation chances provides additional insight into the process and impact

of student disengagement at the start of the middle grades (Balfanz, 2007, p. 229). With this in

mind, in our next step we set out to determine how our predictors might be used in combinations

to better flag potential 6th grade non-graduates. Using the three strongest flags (i.e., being over

age, FCAT scores, and course grades) we reanalyzed the data to examine the power in

identifying non-graduating students when using two factors. These results are presented in

Table 3.

Table 3. Double Warning Flag Combinations and Counts for all Duval 6thGrade Students in20012002 Who Failed to Graduate in 2007-2008

Risk Category Did not graduate n Valid %

FCAT Scores


20/32

EarlyWarningSystem 20Low FCAT Math + Overage 1753 52.17

Low FCAT Math + Low FCAT Reading 1676 49.88

Low FCAT Reading + Overage 1548 46.07

Course Grades

Failed Math Course + Low FCAT Math 1638 48.75

Failed Math Course + Overage 1618 48.15

Failed English Course + Overage 1349 40.15

Failed English Course + Low FCAT Math 1340 39.88

Failed Math Course + Low FCAT Reading 1313 39.08

Failed English Course + Low FCAT Reading 1115 33.18

Failed Math Course + DPP 415 12.35

Failed English Course + DPP 378 11.25

Absence

Absence>20%+Older 381 11.34

Absence>20% + Failed English Course 358 10.65

Note: Yield < 10% for all other combinations

Attendance

Our analysis confirmed the information previously reported by Balfanz (2007) that

attending school less than 80% of the time increases the chance that students will not complete

high school. In our study population, approximately 11% of the 6th grade students who failed to

graduate were absent from school for more than 36 days. However, as reported in our previous

study of 9th grade students (Wilburn and Weaver, 2007), attendance was not as strongly

associated with failing to graduate as academic factors. While useful, in combination with other

factors, independently attendance does not provide a meaningful yield as a predictor variable.


21/32

EarlyWarningSystem 21Standardized Test Scores

Consistent with findings from other studies, academic achievement identified the highest

number of 6th grade students who failed to graduate on time (Balfanz & Boccanfuso, 2007;

Balfanz 2007). However, unlike other studies, in this case standardized test scores were the

most reliable predictor of failure to graduate. When used independently, having a low (Level 1

and Level 2) Florida Comprehensive Assessment Test score in Reading or Math accounted for

approximately 62% percent of the 6th grade students who failed to graduate. While being over-

age identified the highest number of future non-graduates among the 6th grade study population,

in most cases the reason that a student is overage is because that have been retained for failing

to make an acceptable score on the FCAT. Consequently, being over-age may simply be a

proxy for poor FCAT performance over time. In addition, it takes more than one year to be

overage while FCAT score are available on a year to yearly basis.

Course Grades

In our work with the data provided by the school district for our descriptive study of 9th

grade graduation rates and this study, we found many inconsistencies in the course grade data

sets. In our efforts to identify early warning flags for 6th grade non-completers, we found the

data to be so inconsistent that we initially removed the course grade variable form our analysis.

However, because it is information that is readily available, we have included it in this analysis.

When taken independently, failing one 6th grade math or English course identified approximately

36.2% to 28.1%, respectively, of the students who eventually failed to complete high school.

Because of inconsistency in course grades from year to year, we believe that standardized test

scores provide the most valid and reliable predictor of high school completion for sixth grade

students, and we strongly recommend caution in using of course grades as an independent

warning flag for identifying students. This point may be illustrated by the fact that course grade


22/32

EarlyWarningSystem 22is the only academic factor that is stronger when combined with other factors. For example

when failing a math course is combined with the low FCAT math score factor, the number of

students identified increases from a math course alone yield of 30.2% to combined yield of

48.7%. A similar gain is also realized when the two factors of a failed English course and a low

FCAT reading score are combined. For this reason we do not recommend using course grades

as an independent predictor.

Discipline

The discipline codes used in this study were those that represented the most serious

(i.e., Class 3 and 4) offences that a student could receive. For example, students behaviors that

involve physical assault, weapons and the use and/or possession of illegal drugs fall into these

categories. In the vast majority of cases, when a student receives this type of disciplinary

referral, he/she is automatically suspended from school and/or removed from their home school

and assigned to one of the Districts alternative schools for a period of 45 to 90 days.

Consequently, the number of Class 3 and 4 Code of Conduct violations occurring at the 6th

grade was not very large. As a result, this flag did not identify a large number of the 6

th

graders

who eventually failed to complete high school. From our work in tracking the 6th grade student

as well as our previous analysis of 9th grade students who failed to graduate, we know that the


students become over-age for their assigned grade. Consequently, the conduct violations from


factor was relatively small (6.7%) as a single factor and no greater when considered in

combination with test scores or grades, we nevertheless consider this to be an important flag to

monitor.

Status Variables (Dropout Prevention Program/Over-age)


23/32

EarlyWarningSystem 23Being either in the Districts Dropout Prevention Program (DPP) or over-age for the 6th

grade significantly reduces the probability that a student will successfully complete high school.

Our analysis indicates that of the 858 6th grade students enrolled in one or more of the Districts

Drop Out Prevention Programs, 341 (40%) graduated from high school on time while 60% failed

to complete high school. Obviously, for a student to be placed in one of these programs as early

as the 6th grade, he/she exhibits a number of our warning flags such as poor attendance, low

FCAT scores, and poor course grades. Consequently, the probability for completing high school

is low (i.e., 4 out of 10) for this student group. While the value of this factor as an early warning

flag for identifying large numbers of 6th grade students is limited, 15% of the 6th graders who

failed to complete high school were in the DPP. We believe it would be useful to examine the

program as to a better understanding of what initiatives may be successful in working with high

risk students.

On the other hand, being over-age for sixth grade appears to account for a large number

on non-high school completers. In our study population, 72% of the 6th grade students who

were over-age failed to complete high school. While this initially appears to be a strong flag for

identifying those who will not graduate from high school, as indicated in Balfanzs 2007 study,

this is primarily because a high percentage of over-age students scored at Level 1 or 2 on the

FCAT, failed a 6th grade math or English course, and attended less than 80% of the time. The

few over-age students who did not exhibit any of the other flags tend to graduate at the same

rate as other 6th grade students. Consequently, being over-age is not the issue.

Final Selection of Identification Flags

Our final action in the identification of variables (i.e., predictive flags) was to narrow the

number of flags to those that independently or in combination with other flags provided the

strongest practical tool for early identification of 6 th grade students who had the highest


24/32

EarlyWarningSystem 24probability of not graduating from high school. As discussed above, some of the variables, such

as being over-age and/or enrolled in a dropout prevention program, were actually products of

other variables such as poor academic achievement or attendance. While a case could be

made for including the discipline code flag in the final selection as it seems to be such a reliable

predictor in later grades, it also was very duplicative in that over 90% of the 6th grade students

who had a Class 3 or 4 code of conduct violation also had one of the other flags such as poor

attendance or failing course grades. Therefore, we limited our final selection of flags to five

school related factors:

1. FCAT math scores

2. FCAT reading scores

3. Failing a 6th grade math class

4. Failing a 6th grade English class

5. Greater than 20% days absent from school

By using these five factors and eliminating duplicate records, we were able to identify

approximately 82% of the students who failed to graduate.

Validity of the Flags

In order to estimate the validity of our final five flags while controlling for the other flags,

we conducted a multivariate logistic regression to estimate the predictive power of each flag.

The analysis showed that, all else being equal, 6th grade students who missed more than 20%

of school attendance days were 6.07 times more likely not to graduate than students who

missed less than 20% of school attendance days (confidence interval (CI) = 5.03 to 7.32).

Students who failed English/Language Arts in the 6th grade were 2.86 times more likely not to


25/32

EarlyWarningSystem 25graduate than students who passed Language Arts (CI = 2.15 to 3.17). Sixth grade students

who failed a math course were 2.38 times more likely not to graduate than those who passed

their math courses (CI = 2.15 to 2.62). Those 6th grade students who scored below 3 on the

FCAT reading test were 1.62 times more likely not to graduate than students who scored 3 or

above (CI 1.49 to 1.77). Finally, those 6th grade students who scored below 3 on the FCAT

math test were 1.71 times more likely not to graduate than those who scored 3 or above (CI =

1.56 1.88). Each flag was a statistically significant flag (p< .05) even after controlling for the

other flags. Our five flags, individually and in combination, identified 81.52% of the non-

graduates. Among the 10,093 total study population, 49% (4,975) graduated with none of the

five flags while 17% (1,706) had no flags but still did not graduate.

Limitations of the Study

It is important to recognize that, as with all investigations of this type, these numbers

provide a description of the 2001-2002 6th graders who had not graduated by the end of 2008.

In the course of our data analysis, we conducted both linear and logistic regressions with the

common wisdom being that a logistic regression is the better test to measure predictors on a

dichotomous variable, resulting in the graduation variable being yes/no on graduation. Though

none of the 4,975 students in our 10,093 member 6th grade cohort had any of the five flags and

82% of the non-graduates had one or more of the flags, it is not valid to state that the

combination of factors has a predictive power. The regression calculates mathematical

possibilities, not actual counts. This is evidenced by the fact that 1,706 of the 6th graders had

no flag but had not graduated by the end of the 2008 school year.

In generalizing our results to other populations, it is also important to note that we did not

control for race in the regressions that we ran. Since our 6th grade population was divided so

evenly between Black and White, there seemed no need to do so. It is common practice to only

control for such a factor if there are large imbalances inside the group. Consequently, caution


26/32

EarlyWarningSystem 26should be exercised when applying our results to populations that have racial imbalances within

the study cohort.

Summary

Purpose

The primary purpose of this study was to identify specific academic and social factors

that may be used as flags to identify DCPS 6th grade students who are at high risk of not

completing high school. A secondary purpose was to understand how these factors may or may

not be aligned with those identified by Neal, Balfanz, and other researchers (Neal and Balfanz,

2006).

Data

The data used in this analysis were provided by the DCPS Department of Instructional

Research and Accountability. The data were gathered from the districts data warehouse and

provided to the research team in the form of several Excel spreadsheets. The data included

selected demographic, academic, attendance, and disciplinary information from each student

enrolled in grades 612 during the 20012002 through the 20072008 school years.

Approximately 60,00065,000 students were enrolled in grades 6-12 during each school year

with each grades enrollment averaging between 10,00012,000 students.

Since the focus of this analysis was 6 th grade students, we began by identifying all 6th grade

students for the 20012002 school year and then tracking that cohort of students through each

school year through the 12th grade (2007-2008). Ten thousand ninety-three (10,093) students


27/32

EarlyWarningSystem 27remained for inclusion in the analysis after the elimination of withdrawals, transfers, and

incomplete data sets.


28/32




data, an initial factor analysis was conducted to determine whether variables fell into coherent

categories (i.e., factors) as predictors of completion. No one piece of data, such as attendance

or mathematics standardized test scores, has enough weight to serve as a valid predictor.

Therefore, this process of combining various data sets was repeated until a consistent set of

categories was identified. Once a coherent set of predictors was identified, linear regression

(stepwise) and logistic regression analyses were conducted on each identified factor in order to

estimate the strength of each factor.

Four questions guided our 6th grade analysis:

1. Are significant numbers of students showing the signs of disengagement previously

identified in our examination of 9th grade students who fail to complete high school?

2. Can we identify a set of indicators that flag sixth graders who have high odds of failing

to complete high school?

3. From this set of indicators, can we identify a practical set of indicators routinely collected

and reported at the individual level that flag sixth graders who have odds of failing to

continue in school until graduation?

4. Can we identify indicators within that set that may be influenced by school and

community programs?

Results

A significant number of 6th grade students displayed the signs of disengagement, such

as poor academic achievement and excessive absences, previously identified in our study of 9 th


29/32

EarlyWarningSystem 29grade students and in studies conducted by other researchers. Five factors proved to be of

greatest value in identifying members of the 2001-2002 6th grade cohort who did not graduate

on-time from high school in 2007-2008. These factors were:

Standardized Math Test Scores: 69% of the 6th grade students who failed to graduate


Standardized Reading Test Scores: 56% of the 6th grade students who failed to graduate


Failing Course Grades in Mathematics: 30.2% of the 6th grade students who failed to

graduate on-time failed a math course.

Failing Course Grades in English/Language Arts: 28.1% of the 6th grade students who

failed to graduate on-time failed an English/Language Arts course.

Excessive Absences: 12.2% of the 6th grade students who failed to graduate on-time

missed more the 20% of the scheduled school days.

The number of Class 3 and 4 Code of Conduct violations occurring at the 6th grade among

the study population was not very large (3%). As a result, this flag only identified 6.7% of the 6th

grader who failed to graduate on-time. However, our tracking of students indicated that the


students become over-age for their assigned grade. Consequently, the conduct violations from


factor was relatively small, as a single factor and no greater when considered in combination

with test scores or grades, we nevertheless consider this to be an important flag to monitor.

Results from our study indicate:

1. Significant numbers of 6th grade students show signs of disengagement previously

identified in our examination of 9th grade students who fail to complete high school.


30/32

EarlyWarningSystem 302. We can, with a reasonable degree of validity, identify a set of indicators that flag sixth

graders who have high odds of failing to complete high school

3. From this set of flags, we can identify a practical set of indicators routinely collected and

reported at the individual level that flag sixth graders who have odds of failing to

continue in school until graduation

4. These indicators are factors that may be impacted by school and community programs.

In conclusion, as we have learned from our previous studies of first-time 9th grade

students and focus groups with students who had previously dropped out of school, students do

not graduate from high school for a multitude of reasons; and many of those reasons are not

linked directly to academic achievement, attendance, and behavior at school. Even though the

issues is complex and no one single factor or group of factors can account for every childs

decision to leave school, we believe that it would be unconscionable to ignore the early warning

flags readily available to even the most casual observer.


31/32

EarlyWarningSystem 31References

Balfanz, R., Hertzog, L., Neild, R. C. and Mac Iver, D. J. (2007). Preventing student

disengagement and keeping students on the graduation path in urban middle-grades

schools: Early identification and effective interventions. Educational Psychologist. 42(4),

223-235.

Balfanz, R. (2006). Unfulfilled promise: The dimensions and characteristics of Philadelphias

dropout crisis, 2000 2005. Philadelphia, PA: Philadelphia Youth Transitions

Collaborative.

Roush, C., Wilburn, K. T. and Weaver, D.M. (January, 2009). Learning to finish focus group

data: qualitative analysis report. Jacksonville, FL: The Community Foundation.

StatSoft, Inc. (2007). Electronic Statistics Textbook. Tulsa, OK: StatSoft. WEB:

http://www.statsoft.com/textbook/stathome.html.

Wilburn, K. T. and Weaver, D. M. (November, 2008) Where Have All the 9th Graders Gone? A

Descriptive Study of Three First-Time 9th Grade Student Cohorts. Jacksonville, Florida:

The Community Foundation.
http://www.statsoft.com/textbook/stathome.htmlhttp://www.statsoft.com/textbook/stathome.html


32/32

EarlyWarningSystem 32Appendix

Demographic Characteristics of the Study Population

Group Variable Completers (%) Non-completers (%)

Female 51.8 46.8Gender

Male 48.2 53.2

Asian American 3.6 2.6

Black 45.2 43.4

Hispanic 3.9 4.0

American Indian 0.1 0.2

Multiracial 1.0 1.1

Race/Ethic Group

White 46.2 48.8

Non-English Speaking 9.2 7.0Primary Language

English Speaking 90.8 93.0

Free Lunch 23.3 20.5

Public Assistance 5.1 6.7

Reduced Lunch 9.3 6.7

Not Eligible 2.7 1.5

Socioeconomic Status(School LunchProgram)

No Information 59.6 64.5

Documents

Developing an Early Warning System: Identifying Factors that may Predict High School Completion by 6th Grade Students