View
214
Download
1
Category
Tags:
Preview:
Citation preview
Initial Research Findings from Arkansas’ Statewide Longitudinal Data System:
Using Standards-Based Research to Solve Real Education Problems
Jake Walker, PhD, Project Director Arkansas Research CenterGreg Nadeau, Director, State Core Model, Public Consulting Group
2012 MIS Conference 1
The CEDS Data Model Process
2
A foundation for standards convergence
CEDS Domain Entity Schema (DES)
Domain EntityEL EL ChildEL EL EnrollmentEL EL AssessmentEL EL ProgramEL EL GroupEL EL StaffEL EL Parent
Domain EntityK12 K12 StudentK12 K12 IncidentK12 K12 ParentK12 K12 StaffK12 K12 Staff AssignmentK12 K12 SectionK12 K-12 CourseK12 K12 Migrant FacilityK12 K12 ProgramK12 K12 SchoolK12 K12 DistrictK12 State Education AgencyK12 AssessmentK12 Learning Standard
Domain EntityPS PS StudentPS PS AssessmentPS PS StaffPS PS SectionPS PS ProgramPS PS Defined ProgramPS PS Teacher ProgramPS PS CampusPS PS SchoolPS PS Institution
CEDS NDS Conceptual Model
Organizations People
Assessment Standards[Content]
Roles
Time
CEDS Logical Model (NDS): High Level
6
7
Parsing NCES Forum Questions with a Standard Syntax
Using CEDS for Analysis# Logic Statement1 The purpose of this artifact is to provide a multi-criterion decision framework to improve Secondary Data Analysis
from P-20 Longitudinal Data Systems. It assumes a set of education and workforce data that is common across the 50 states.
2 In this artifact we will lay out a P-20 Standard Analytic Model (P-20 SAM) built off of the new CEDS 2.0 Data Model.
3 The P-20 SAM is a framework for quantitative research and performance management utilizing valid metrics based on standard data definitions and relationships. It can be used as a logical framework for dimensional models.
4 Valid metrics are aggregate measures (counts, averages, ratios) that have been demonstrated to correlate with educational indicators. For example Student Growth Percentile (SGP) is a measure that correlates with K12 quality learning and Degree Production Rate (DPR) is a measure that correlates with postsecondary effectiveness. A list of P-20 Metrics is provided with a standard definition with transformation rules tied directly to standard CEDS DES element definitions and NDS tables and fields.
5 A list of P-20 Metrics is provided with a standard definition with transformation rules tied directly to standard CEDS DES element definitions and NDS tables and fields.
6 The P-20 SAM provides Standard Syntax for expressing research questions and performance reports. This syntax must specify the metric, entity/ies, and daterange/s to be considered.
7 Metrics are often expressed as rates. Rates express the ratio between a sub-group count (little n or numerator) and a general population count (big N or denominator).
8 Application of a metric to a given data set must consider the quality and quantity (n-size) of the data as well as the sensitivity of the metric to determine the overall reliability of the measure.
A Framework for Using CEDS for Analysis and Performance Management
Using the Framework• Step 1: Framing the Issue
• The Question • The Audience• The Audience’s Focuses• The Approach
• Step 2: Selecting the Right Metrics• Data Sources • Data Matching• Variables Used
• Step 3: Constraining the Metrics• Output
• Step 4: Data Use• Wages by Degree Level by Major• UAMS Study 10
The Question
• What is the value of public higher education in the state of Arkansas?
11
The Audience
• State Level Policy Makers• Administrators of Public Institutions of
Higher Education• Students and Families
12
The Audience Focuses
• Policy and Funding Issues• Program Planning and Staffing Issues• College Outcomes for Life
13
The Approach
Original Report• What percentage of
Arkansas residents and non-residents who graduate from public colleges and universities remain in Arkansas?
• How does level of education impact salary?
• What is the average salary by field of study and level of degree of Arkansas graduates?
New Version• What are the employment
outcomes of graduates from 2002 to 2009 as of 2010 in the state?
• What percentage of Arkansas residents and non-residents who graduate from public colleges and universities remain in Arkansas either in the workforce or enrolled in the higher education system?
• How does level of education impact salary?
• What is the median salary by field of study and level of degree of Arkansas graduates?
• In which industry areas are Arkansas graduates employed after they finish college?
14
Variables UsedOriginal Report
• SSN • Graduation Year• Degree Level• Degree Area• Residency Status• Employment Status• Annual Wages at Year 1• Annual Wages at Year 5 • Residency Status
New Version• SSN• DOB• First Name• Last Name• Graduation Date (MM/YYYY)• Degree Level• Degree Area• Residency Status• Enrollment Status• Employment Status• Average Quarters Employed• Annual Wages at Year 1 • Annual Wages at Year 2• Annual Wages at Year 5• NAICS Code
15
ADH
E
ADH
E
ADW
S
ADW
S
Data SourcesOriginal Report
• Arkansas Department of Higher Education
• Arkansas Department of Workforce Services
New Version
• Arkansas Department of Higher Education
• Arkansas Department of Workforce Services
• National Student ClearingHouse
16
Data MatchingOriginal Report
• SSN from ADHE matched to SSN from ADWS
New Version
• TrustEd Broker System• SSN• DOB• First Name• Last Name
• Uses multiple pieces of information about a person to determine if they match.
• Personally identifiable information is removed and replaced with a non-traceable id.
17
• ~55,000,000 records for 10 years, 2,938,718 unique SSNs, no DOBs, inconsistent naming standards.
• 7,865 SSNs used by two or more people, for a total of 18,278 different individuals. Those would be combined incomes and treated as the same person if SSN was the primary key.
• The same person has two or more SSNs (because of a typo/transposition) 13,373 times. There would be 13,373 additional (non-existent) people with separate incomes if SSN was the primary key.
Identity Resolution (Wage Data)
A trusted broker maintains a cross reference table, encoding the identifiers for various agencies and for various representations of the entities.
Trusted Broker
Bob Smith P0236 Robert Smith K4297
Agency 1 Agency 2
Internal
Identifier Identity Information
Identifier Agency1
Identifier Agency2
KB5765 Bob Smith, Barton P0236 K4297
KB5765 Robert Smith, Barton P0236 K4297
KB5765 Bob Smith, Wilson P0236 K4297
Dual Database: Regulatory Compliance and Privacy
Identity Information: Agency ID Generated
Information of Interest: Stored With Agency ID
Knowledge Base:Identity Management Edge Servers:
Shareable Data
De-identified Data
Output Original Report
• Percentage of Graduates Remaining in Arkansas
• Average Salary by Degree Level
• Average Salary by Degree Level by Degree Area/Field of Study
New Version• Percentage of Graduates
Remaining in Arkansas • Median Wages and
Average Quarters Employed by Degree Level
• Median Wages and Average Quarters Employed by Degree Level by Degree Area/Field of Study
• Industry Area Employed in by Degree Level
22
Population of Interest and Entry Time Into the Workforce
• Old Report• Graduates from an academic year would only be counted as
employed if they were present in wage data during the 4th QUARTER AFTER THE END OF THE GRADUATION YEAR.
• New Report• Graduates were classified if they were employed at any point
during the FIRST FOUR FULL QUARTERS AFTER THEIR GRADUATION DATE.
23
Population of Interest and Entry Time Into the Workforce
• Example• Graduates from the academic year 2005-2006• Anyone who graduated from 07/01/2005 to 06/30/2006
• Old Report• 4th Quarter after the graduation year=2nd quarter of 2007
(April-June 2007)• Graduates during July-September of 2005 would not be
classified as employed unless they had wages for the 2nd quarter of 2007 which is actually 7 quarters after they graduated not 4.
24
Population of Interest and Entry Time Into the Workforce
• Example• Graduates from the academic year 2005-2006• Anyone who graduated from 07/01/2005 to 06/30/2006
• New Version• First Full Four Quarters after the graduate date
25
1st Four Full Quarters After Graduation
Grad Date Q1 Q2 Q3 Q4
07/2005 to 09/2005 20054 20061 20062 20063
10/2005 to 12/2005 20061 20062 20063 20064
01/2006 to 03/2006 20062 20063 20064 20071
04/2006 to 06/2006 20063 20064 20071 20072
Remained vs. Employed
26
OLDPercentage Remained
= # Employed/# of Graduates
NEWPercentage Remained
= # Employed Only + # Enrolled Only + # Enrolled and Employed/# Graduates
Original Report
Degree LevelPercentage Employed
Year 5Percentage Employed
Year 12002 2003 2004 2006 2007 2008
Certificate 63% 64% 61% 75% 76% 76%Associate 73% 73% 72% 78% 78% 76%Bachelor 62% 63% 64% 74% 72% 71%Master 62% 60% 58% 67% 68% 66%Doctoral 44% 32% 44% 49% 54% 43%1st Professional 55% 61% 57% 51% 63% 62%
2002 2003 2004 2005 2006 2007 2008 2009
% in AR Year 1 77% 85% 84% 82% 82% 83% 85% 87%
% in AR Year 2 66% 70% 64% 64% 67% 70% 73%
% in AR Year 5 55% 55% 48% 51%
% in AR Year 1 87% 90% 90% 90% 91% 92% 91% 90%
% in AR Year 2 81% 85% 86% 85% 85% 87% 88%
% in AR Year 5 73% 76% 77% 76%
% in AR Year 1 89% 90% 90% 90% 90% 90% 89% 90%
% in AR Year 2 84% 86% 86% 86% 86% 85% 85%
% in AR Year 5 75% 75% 74% 75%
% in AR Year 1 80% 80% 81% 82% 83% 81% 81% 80%
% in AR Year 2 73% 74% 74% 76% 77% 75% 75%
% in AR Year 5 67% 64% 64% 66%
% in AR Year 1 75% 73% 73% 73% 73% 74% 72% 75%
% in AR Year 2 70% 68% 67% 68% 68% 69% 68%
% in AR Year 5 63% 61% 59% 60%
% in AR Year 1 48% 51% 49% 53% 49% 64% 63% 63%
% in AR Year 2 45% 50% 49% 51% 62% 62% 62%
% in AR Year 5 55% 59% 57% 61%
% in AR Year 1 53% 48% 60% 52% 55% 62% 50% 53%
% in AR Year 2 47% 40% 53% 44% 48% 54% 43%
% in AR Year 5 45% 33% 44% 42%
New Version
Technical Certificate
Associate’s
Bachelor's
Master's
Doctoral-1st Professional
Doctoral-Research/
Scholarship
Past Graduates Recent Graduates
Graduation Year
Certificate of Proficiency
Remained vs. Employed
27
OLDPercentage Employed
= # Employed/# of Graduates
NEWPercentage Employed
= # Employed + #Employed and Enrolled/# Graduates
Original Report
Degree LevelPercentage Employed
Year 5Percentage Employed
Year 12002 2003 2004 2006 2007 2008
Certificate 63% 64% 61% 75% 76% 76%Associate 73% 73% 72% 78% 78% 76%Bachelor 62% 63% 64% 74% 72% 71%Master 62% 60% 58% 67% 68% 66%Doctoral 44% 32% 44% 49% 54% 43%1st Professional 55% 61% 57% 51% 63% 62%
2002 2003 2004 2005 2006 2007 2008 2009
% in AR Year 1 72% 81% 81% 78% 77% 78% 76% 76%
% in AR Year 2 63% 67% 62% 62% 65% 67% 69% 75%
% in AR Year 5 53% 54% 47% 49%
% in AR Year 1 81% 84% 87% 86% 86% 87% 84% 81%
% in AR Year 2 78% 83% 84% 83% 83% 84% 83% 80%
% in AR Year 5 71% 74% 74% 73%
% in AR Year 1 80% 79% 80% 79% 81% 80% 78% 77%
% in AR Year 2 76% 78% 78% 78% 77% 77% 76% 76%
% in AR Year 5 72% 73% 71% 72%
% in AR Year 1 75% 77% 77% 78% 79% 77% 76% 76%
% in AR Year 2 69% 71% 71% 73% 73% 71% 70% 71%
% in AR Year 5 62% 63% 63% 64%
% in AR Year 1 72% 70% 70% 71% 72% 72% 70% 73%
% in AR Year 2 67% 66% 65% 66% 66% 67% 65% 69%
% in AR Year 5 61% 60% 57% 59%
% in AR Year 1 48% 51% 49% 52% 48% 63% 62% 62%
% in AR Year 2 44% 50% 48% 51% 62% 62% 61% 62%
% in AR Year 5 55% 59% 56% 61%
% in AR Year 1 53% 47% 59% 52% 55% 62% 49% 52%
% in AR Year 2 46% 40% 53% 43% 48% 54% 42% 45%
% in AR Year 5 45% 32% 44% 42%
New Version
Technical Certificate
Associate’s
Bachelor's
Master's
Doctoral-1st Professional
Doctoral-Research/
Scholarship
Past Graduates Recent Graduates
Graduation Year
Certificate of Proficiency
Average Salary vs. Median Wages
28
OLDAverage Wages
= Sum(4th Quarter Wages * 4)/# Employed
NEWMedian Wages Y1
= Median of Sum(Q1+Q2+Q3+Q4)
Original Report
Degree LevelAverage Salary Year 1
2006 2007 2008Certificate $ 19,396 $ 19,876 $ 18,639 Associate $ 18,868 $ 19,261 $ 19,244 Bachelor $ 26,860 $ 27,332 $ 26,116 Master $ 40,767 $ 41,274 $ 42,305 Doctoral $ 54,966 $ 54,707 $ 54,153 1st Professional $ 44,102 $ 40,551 $ 46,888
Median Wages Year 1Grad. Year 2006 2007 2008 2009Wage Year 2007 2008 2009 2010
CP $ 12,693 $ 13,370 $ 12,657 $ 11,203 TC $ 20,656 $ 20,396 $ 20,261 $ 20,140 AA $ 16,128 $ 16,468 $ 17,350 $ 16,519 BS $ 21,232 $ 21,574 $ 20,831 $ 19,708 MS $ 37,735 $ 38,593 $ 39,780 $ 39,583
1st Pro $ 35,998 $ 28,004 $ 44,250 $ 45,356 PhD $ 50,059 $ 48,981 $ 47,728 $ 44,511
Median Wages Year 2Grad. Year 2006 2007 2008 2009Wage Year 2008 2009 2010 2011
CP $ 16,459 $ 17,207 $ 15,272 $ 14,306 TC $ 23,943 $ 23,650 $ 23,206 $ 23,240 AA $ 19,747 $ 19,380 $ 19,949 $ 19,449 BS $ 28,162 $ 28,749 $ 27,595 $ 27,217 MS $ 41,104 $ 41,842 $ 42,928 $ 42,721
1st Pro $ 42,495 $ 48,347 $ 48,424 $ 49,392 PhD $ 53,900 $ 55,079 $ 54,430 $ 53,225
Median Wages Y2= Median of Sum(Q5+Q6+Q7+Q8)
Average Quarters Employed
29
OLDNot Used
NEWAverage Quarters Employed
=Sum(Quarters Employed)/# Employed
Average Quarters Employed Year 1 Average Quarters Employed Year 2Grad. Year 2006 2007 2008 2009 2006 2007 2008 2009Wage Year 2007 2008 2009 2010 2008 2009 2010 2011
CP 3.21 3.23 3.25 3.29 3.35 3.41 3.38 3.38TC 3.61 3.55 3.57 3.55 3.66 3.63 3.65 3.60AA 3.51 3.51 3.52 3.50 3.58 3.53 3.57 3.55BS 3.47 3.45 3.46 3.45 3.59 3.61 3.58 3.58MS 3.64 3.68 3.69 3.71 3.77 3.79 3.80 3.78
1st Pro 3.44 2.90 3.62 3.55 3.05 3.78 3.83 3.79PhD 3.51 3.65 3.47 3.52 3.76 3.65 3.68 3.76
Average Salary vs. Median Wages
30
OLDAverage Wages
= Sum(20th Quarter Wages * 4)/# Employed
NEWMedian Wages Y5
= Median of Sum(Q17+Q18+Q19+Q20)
Original Report
Degree LevelAverage Salary Year 5
2002 2003 2004Certificate $ 25,921 $ 26,008 $ 24,317 Associate $ 28,902 $ 29,837 $ 29,835 Bachelor $ 35,866 $ 36,766 $ 37,227 Master $ 45,277 $ 46,836 $ 47,888 Doctoral $ 63,028 $ 61,145 $ 66,059 1st Professional $ 78,095 $ 72,426 $ 79,324
Median Wages Year 1Grad. Year 2002 2003 2004 2005Wage Year 2003 2004 2005 2006
CP $ 14,995 $ 12,864 $ 11,909 $ 11,865 TC $ 16,397 $ 17,650 $ 18,392 $ 20,285 AA $ 14,509 $ 14,483 $ 15,120 $ 15,139 BS $ 17,186 $ 17,500 $ 19,325 $ 20,175 MS $ 32,417 $ 32,987 $ 34,398 $ 35,694
1st Pro $ 29,996 $ 35,647 $ 37,477 $ 31,500 PhD $ 44,673 $ 41,398 $ 45,897 $ 47,013
Median Wages Year 2Grad. Year 2002 2003 2004 2005Wage Year 2004 2005 2006 2007
CP $ 18,382 $ 17,044 $ 15,662 $ 15,790 TC $ 20,161 $ 20,786 $ 21,836 $ 23,023 AA $ 17,874 $ 17,122 $ 17,798 $ 18,089 BS $ 23,908 $ 24,923 $ 26,998 $ 27,404 MS $ 35,785 $ 37,496 $ 39,732 $ 39,228
1st Pro $ 43,731 $ 46,101 $ 48,638 $ 47,242 PhD $ 47,059 $ 48,477 $ 54,713 $ 55,730
Median Wages Year 5Grad. Year 2002 2003 2004 2005Wage Year 2007 2008 2009 2010
CP $ 26,504 $ 26,445 $ 21,794 $ 22,176 TC $ 26,920 $ 26,312 $ 27,865 $ 29,282 AA $ 29,899 $ 30,017 $ 30,617 $ 29,851 BS $ 36,580 $ 37,644 $ 37,762 $ 37,797 MS $ 46,555 $ 47,114 $ 48,648 $ 47,574
1st Pro $ 82,954 $ 83,162 $ 81,453 $ 68,626 PhD $ 59,657 $ 67,444 $ 67,665 $ 67,169
Average Quarters Employed
31
OLDNot Used
NEWAverage Quarters Employed
=Sum(Quarters Employed)/# Employed
Average Quarters Employed Year 1 Average Quarters Employed Year 2 Average Quarters Employed Year 5Grad. Year 2002 2003 2004 2005 2002 2003 2004 2005 2002 2003 2004 2005Wage Year 2003 2004 2005 2006 2004 2005 2006 2007 2007 2008 2009 2010
CP 3.33 3.20 3.09 3.09 3.50 3.36 3.35 3.37 3.60 3.62 3.52 3.50TC 3.56 3.60 3.60 3.59 3.67 3.62 3.67 3.64 3.71 3.68 3.71 3.68AA 3.52 3.50 3.51 3.52 3.58 3.56 3.56 3.53 3.72 3.71 3.73 3.71BS 3.42 3.40 3.45 3.47 3.55 3.56 3.60 3.59 3.73 3.75 3.75 3.74MS 3.67 3.66 3.65 3.66 3.76 3.77 3.75 3.73 3.81 3.82 3.84 3.82
1st Pro 3.25 3.44 3.50 3.32 3.68 3.61 3.69 3.69 3.69 3.76 3.76 3.65PhD 3.49 3.36 3.58 3.56 3.72 3.64 3.70 3.79 3.64 3.97 3.86 3.73
Wages by Degree Level by Major
32
OLDAverage Wages
= Sum(4th Quarter Wages * 4)/# Employed
NEWMedian Wages 2010
= Median of Sum2010(Q1+Q2+Q3+Q4)Original Report-HEALTH SCIENCES MAJORS
Degree LevelAverage Salary Year 5 Average Salary Year 1
2002 2003 2004 2006 2007 2008Bachelor $ 43,713 $ 44,882 $ 47,581 $ 37,448 $ 38,034 $ 38,317
Years Since Graduation 1 Year 2 Years 3 Years 4 Years 5 Years 6 Years 7 Years 8 Years
Graduation Year 2009 2008 2007 2006 2005 2004 2003 2002
HEALTH PROFESSIONS AND RELATED PROGRAMS
CP%Employed 81% 79% 79% 75% 75% 77% 71% 72%
Median Salary 11,662 15,007 16,612 22,374 21,432 19,449 22,608 28,253
TC%Employed 86% 88% 85% 82% 79% 79% 76% 70%
Median Salary 25,676 25,593 28,131 29,402 29,274 30,220 31,247 31,587
AA%Employed 83% 75% 76% 75% 73% 71% 69% 67%
Median Salary 38,144 39,159 40,314 42,691 41,982 43,568 44,713 45,641
BS%Employed 76% 70% 68% 68% 68% 66% 67% 63%
Median Salary 37,067 39,156 43,681 44,751 44,711 48,977 48,200 46,942
MS%Employed 66% 63% 60% 53% 55% 61% 58% 58%
Median Salary 55,070 52,754 58,686 58,532 55,550 55,503 58,551 56,490
1STPRO%Employed 62% 67% 64% 68% 64% 57% 58% 61%
Median Salary 56,939 52,351 80,197 105,787 111,052 118,980 138,125 126,844
PHD%Employed 69% 79% 75% 79% 65% 54% 38% 46%
Median Salary 56,000 57,927 62,482 60,479 66,671 68,803 75,749 78,659
Wages by Degree Level by Major
33
OLDAverage Wages
= Sum(4th Quarter Wages * 4)/# Employed
NEWMedian Wages 2010
= Median of Sum2010(Q1+Q2+Q3+Q4)
1 Year 2 Years 3 Years 4 Years 5 Years 6 Years 7 Years 8 Years
2009 2008 2007 2006 2005 2004 2003 2002
%Employed 79% 78% 90% 73% 80% 80%
Median Salary 10,446 13,761 14,222 17,634 17,522 18,577
%Employed 88% 91% 78% 76% 73% 80%
Median Salary 12,949 14,419 12,680 15,816 20,437 18,846
%Employed 78% 84% 81% 86% 79% 85% 79% 95%
Median Salary 10,908 14,737 20,737 21,795 23,027 23,456 29,082 24,039
%Employed 88% 85% 82% 83% 79% 79% 77% 70%
Median Salary 30,666 33,468 35,166 35,973 36,526 37,906 38,947 39,744
%Employed 87% 85% 80% 80% 78% 77% 75% 74%
Median Salary 41,374 43,624 44,555 45,295 46,750 49,226 48,834 49,690
%Employed 86% 83% 81% 78% 76% 83% 77% 68%
Median Salary 63,659 57,067 62,445 61,426 64,973 63,253 69,595 66,009
%Employed 53% 55% 63% 66% 56% 69% 44% 57%
Median Salary 58,331 72,495 68,985 72,351 71,961 79,259 74,227 73,275
BS
MS
1STPRO
EDUCATION
Years Since Graduation
Graduation Year
CP
TC
AA
PHD
Original Report-EDUCATION
Degree LevelAverage Salary Year 5 Average Salary Year 1
2002 2003 2004 2006 2007 2008Bachelor $34,791 $36,121 $36,218 $30,233 $30,806 $30,534
UAMS Study
Arkansas Research Center
Contact Information
arc.arkansas.gov
36
Recommended