Using High-Quality Data to Monitor

Student Performance

Mark Baird, Ph.D.Division of Career and Adult

Education

AECP Leadership InstituteDecember 5, 2012

The Impact of Fee and Residency Policy Changes on 2011-12 AGE Enrollment

Compared to Previous Year

School DistrictsHeadcount: -33.9%FTE: -31.6%

Florida College SystemHeadcount: -31.0%FTE: -29.4%

School District Enrollment (Headcount) Changes

Program 2010-11 2011-12 DifferenceAcademic Skills for Adult ESOL Learners 2,545 1,849 -27.3%Adult Basic Education (ABE) 78,914 51,802 -34.4%Adult English for Speakers of Other Languages (ESOL) 94,402 59,215 -37.3%Adult General Education for Adults with Disablitites Educational Plan 809 994 22.9%Adult High School 13,074 10,545 -19.3%Adult High School Co-Enrolled 57,070 36,134 -36.7%Applied Academics for Adult Education 11,973 8,814 -26.4%Citizenship 6,874 6,855 -0.3%English Literacy for Career and Technical Education (ELCATE) 4,386 6,326 44.2%General Educational Development (GED) 31,722 21,051 -33.6%Literacy Skills for Adult ESOL Learners 4,882 2,411 -50.6%Pre-Applied Academics for Adult Education 2,386 1,781 -25.4%Pre-General Education Development (GED) 5,310 3,806 -28.3%Workplace Readiness Skills for Adult ESOL Learners 562 577 2.7%

Florida College System Enrollment (Headcount) Changes

Course 2010-11 2011-12 DifferenceAdult Basic 31,396 22,123 -29.5%Adult Secondary 6,436 5,034 -21.8%EAP Literacy 14,916 10,070 -32.5%EAP Vocational Prep. 3,883 3,333 -14.2%GED Prep. 10,646 6,944 -34.8%Vocational Prep. 5,018 2,398 -52.2%

Enrollment Decline More Dramatic When Summer Excluded (District Data)

All Terms Winter/Spring Terms

2010-11 268,663 229,5072011-12 177,559 132,840Decline 91,104 96,667Percent Decline 33.9% 42.1%

Early Indication That Completion Rate Increased

District AGE Program2010-11, No Summer 2011-12, No Summer

Completion to Headcount Ratio

Completion to Headcount Ratio

Adult Basic Education (ABE) 0.52 0.65Adult High School 2.25 2.49General Educational Development (GED) 1.23 1.27Adult English for Speakers of Other Languages (ESOL) 0.41 0.45English Literacy for Career and Technical Education (ELCATE) 0.44 0.29Academic Skills for Adult ESOL Learners 0.23 0.29Workplace Readiness Skills for Adult ESOL Learners 0.53 0.47Citizenship 0.31 0.41Adult General Education for Adults with Disablitites Educational Plan 0.00 0.26Pre-General Education Development (GED) 0.49 0.48Literacy Skills for Adult ESOL Learners 0.40 0.50Pre-Applied Academics for Adult Education 0.32 0.35Applied Academics for Adult Education 0.55 0.66TOTAL 0.63 0.72

The Central Questions

How do you ensure the quality of your data?

How do you use this high-quality data to improve your programs?

Questions of Program Quality Can you answer these questions

without using anecdotal evidence? How good are your programs? How good are your teachers? How much are your students learning? Are your programs a good investment for

Florida taxpayers? Are your programs as effective and efficient

as they should be?

Four Key Areas of Accountability Measurement

EnrollmentProgressCompletionOutcomes

The Necessity of Locally Generated Reports Performance reports from the state will

ALWAYS be significantly lagged Not all useful data are reported to the

state Timely local response to student

performance data feedback is critical The timeliest and most granular

performance data will always come from local data systems

Local Reports Local data needs to be updated on a

regular basis (daily if possible) Store data in a back-end database Use front-end tool to slice and dice data

(e.g. Access, Excel, SPSS) Create reports that can be distributed

to stakeholders Liberate your data!!!

The Key Feedback Loop

Classrooms & Testing Centers

Local Program Office

If this is not flowing, you have a

problem

The Data Roach Motel Problem

Data go in, but it doesn’t get out.

The Citrus Solution

Classrooms & Testing Centers

Local Program Office

Program Database

NRS Accountability Reports

Your check engine Light

What You Can Get Out of NRS Data

ABE & ESOL: Percentage of students who complete a functioning level

ASE low: Percentage of students completing courses or GED® subtests.

ASE high: Percentage of students earning diploma

Average instructional hours Comparisons of above between post-

tested and non-post-tested students.

What do WE look at? Percentage of students reported as

post-tested Percentage of students earning LCP Average LCPs per student Average instructional hours per LCP Percentage of ASE students earning

diplomas Transition of diploma earners to

postsecondary education and employment

Transition of ABE High completers to ASE or diploma

Transition of ESOL students to ABE, ASE, diploma or postsecondary

Percentage of GED® Prep students taking all five subtests and of those, percentage earning diploma

DOE Reports Provided to District and College AGE Directors

LEA-Level NRS Report Student Progression (NRS) Student Instructional Hours (NRS) Student Demographics Student Performance Student Transition

New State Data Tools

Florida is blessed with copious data Complex systems of data collection and

management are in place Next step: Getting data into the hands

of administrators and teachers Reports and tools need to be intuitive

and easy to use; if not, formal training required

Statewide Longitudinal Data System (SLDS) Projects

Upgrade state data systems and consolidate silos into a comprehensive Education Data Warehouse v 2.0.

Customizable report-building tools and dashboards for stakeholders

Access to state data via single sign-on gateway (“one-stop shopping”).

Data mining tool for FLDOE researchers Minimum standards for local data systems, a forum for

information exchange, and financial support to small and rural districts.

Statewide unique identifier to all students and staff. State research agenda and automated delivery of

data to researchers.

Local Student Data Management Requirements in AEFLA Grant AEFLA grantees must

Collect data from instructional and testing sites on a monthly basis at a minimum and store the data in a relational format

Produce monthly reports and make them available to instructional and testing sites for review and correction

Produce monthly reports, as requested locally, for program improvement and monitoring

Use Your Local Data: Examples of Local Report Content Instructional site administrators should have

access to timely classroom-level data on: Attendance (e.g. average monthly) Completions (e.g. completion rate) Contact hours between test administrations Ratio of headcount and contact hours to

completions Disaggregation by race, gender, and other

relevant variables, e.g. day/evening classes

Test Data

Beginning in 2010-11, the state required school districts and colleges to report student-level literacy test data for adult education students

Use the data for your own analysis by comparing pre- and post-test scale scores

Limitation of Measurement by Functioning Level

Time

Tes

t S

core

s

Level 2 Floor

Level 1 FloorPre-test

No learning gain recorded despite growth

Learning Growth

Curve

Post-testIs this a better teacher?

LOOK OUT! Comprehensive Courses: AKA The BLOB!

Impossible to determine the success rate of students in comprehensive ABE and GED courses by skill/subject area and functioning level

NRS Changes for 2012-13 Adult student goal will no longer be used

to identify follow-up cohorts (Table 5) Enter employment and job retention based

on employment status at entry Enter postsecondary cohort will be all

diploma earners Obtain secondary credential

Adult high school: Number of ASE high students who earn diploma

GED: Number of students taking all subtests who earn GED (pass rate)

NRS Changes for 2012-13

Participant Status (Table 6) Highest degree or level of schooling

completed U.S.-based schooling Non-U.S.-based schooling

Personnel (Table 7) Teachers’ years of experience Teacher certification

2011-12 Data Reporting Approximately half of all districts had extensive

problems with their data on the first load attempt Two districts were required to submit supplemental

files to correct instructional hours because of pervasive problems

Districts with over 1% of AGE course records invalid Supplemental files accepted to validate records

and include instructional hours in funding calculations

Work on 2011-12 WDIS data ended on October 15, 52 work days after the close date.

Numerous reporting issues on the college side

Data Quality Issues

Problems we saw in 2011-12 WDIS reporting Invalidated course records Over-reporting of instructional hours Under-reporting of instructional hours Large changes in enrollment or average

instructional hours compared to previous year

Some district reporting staff are Not reviewing data before it is submitted Not reviewing feedback reports on

submitted data to investigate and fix errors Files of student records flagged with validation

errors are available for download and analysis

State MIS staff do not have the time to fix local data problems or perform analysis that local staff should handle

WDIS Submission Periods, 2012-13

Survey Period Opens Required Load Period Closes

F/G Summer & Prelim Fall

September 4, 2012

September 13, 2012

October 18, 2012

W/X Fall & Prelim Winter

January 7, 2013 February 7, 2013 March 7, 2013

S June 3, 2013 July 3, 2013 July 11, 2013

S Update Window

July 15, 2013 N/A August 1, 2013

College Data Submission Periods, 2012-13

Submission Period

Period Opens Required Load Period Closes

Summer End/Fall Beginning

August 27, 2012September 24, 2012

October 8, 2012

Fall End/Winter-Spring Beginning

January 7, 2013 February 4, 2013 March 4, 2013

Winter-Spring End

April 22, 2013 May 13, 2013 June 17, 2013

The time between the required load date and the close of the submission period allows for corrections to be made and data to be resubmitted

Some are missing the load deadline and loading data at the close of the submission period, allowing no time for corrections.

Developing a Local Data Quality Assurance System Identify your Reports Coordinator Build reporting timelines into your planning

calendars Establish with Reports Coordinator a data review

committee that includes program, data, and budget staff Determine what pre-load reports the committee should

review Ensure that the committee has access to validation and

mid-survey reports provided by FDOE showing data you have submitted

For each survey, set up times for the committee to meet and review data twice: Before the data are first loaded After data are loaded but before the close of the survey

The Submission Review Process

1. Review data2. Fix problems3. Load data4. Review edit and validation reports5. Review mid-survey reports6. Fix problems7. Certify data before deadline

Pre-Load

Post-Load

Pre-Load – Review Local Reports You should have a set of local reports to

review, by program, before data are loaded into WDIS

If your system is incapable of producing pre-load data reports, this is an unacceptable situation

Ideally your system should run all WDIS validation edits locally BEFORE the data are loaded CCTCMIS can provide COBOL code that runs the

data through the edits

Pre-Load – What to Look for In Local Reports

If your system runs edits, number and percentage of valid records

Average instructional hours by program; school and program

LCPs reported by program; school and program

Another Fine Mess

Not reviewing your data before you load it is like not scraping out the lasagna dish before putting it in the dishwasher.

Post-Load – Review Reports from Mainframe and Clean Data

When your data are loaded, the system generates reports upon request How many records were accepted, rejected

with critical errors, and flagged with non-critical errors

Lists of student records with error codes This information should be used to

clean up submitted data

The mainframe verification reports are accessible online, but are secured requiring login and password information

Your Reports Coordinator must have access to these reports– it is possible that is not the case

Post-Load - Review Mid-Survey Reports to Detect Anomalies

CCTCMIS produces a set of mid-survey WDIS reports that are useful for detecting anomalies in your data These reports are available on the CCTCMIS

restricted access website: http://www.fldoehub.org/CCTCMIS/Pages/default.aspx

College reports on enrollments, FTE, and completions available on mainframe

Your Reports Coordinator must have a username and password from CCTCMIS for authorized login

Post-Load – Key WDIS Mid-Survey Reports

Enrollment and Instructional Hour Comparison with Previous Year 01 – By District 03 – By Program 05 – By School 07 – By School and Program 07A – By School, Program, and Course

11 - Valid Course Records

Completer Counts 14 – By District 15 – By Program 16 – By School 17 – By School and Program

State MIS Advisory Committees The Workforce Education District Data Advisory

Committee (WEDDAC) is an group comprising representatives from school district workforce education data reporting units

The Management Information Advisory Task Force (MISATFOR) is the corresponding college organization

The groups meet jointly with FDOE staff approximately three times a year to discuss data issues, including changes to the state’s databases

Make sure your district/college is represented!

Consequences of Poor Data QualityFundingAccountabilityCredibilityAudit Risk

ContactsAccountability & Reporting

Craig.Winger@fldoe.org , 850-245-9059Mark.Baird@fldoe.org , 850-245-9060

State BudgetDistricts – Tara.Goodman@fldoe.org ,

850-245-9002State Colleges – John.Holdnak@fldoe.org ,

850-245-9764Federal Grants

Your regional program manager