Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Observations on Developing a
Postsecondary Institution
Rating System (PIRS)
Dr. Braden J. Hosch
Asst. Vice President for Institutional Research, Planning & Effectiveness
Stony Brook University
January 22, 2014
2
Current role:
Asst. VP for Institutional Research, Planning &
Effectiveness at Stony Brook University (NY)
IPEDS National Trainer, Advisory Board Member
Former role:
Director of Policy & Research,
Connecticut Board of Regents for Higher Ed
CONTEXT (1)
3See Higher Education Coordinating Council. (2012). Public Policy Framework for Higher
Education. http://www.ct.edu/files/opr/A_CTFramework.pdf
• Connecticut changed its approach to higher education
accountability in 2011
• Broad stakeholder involvement
• 5 goals with 23 indicators, disaggregated
• Used data publicly available through IPEDS
• Institutional performance measured over time
compared to similar institutions
• Accountability more than consumer information
• Work in progress
CONTEXT (2)
4
Comparison Groups
Graduation Rates
Learning Outcomes
Affordability
Employment Outcomes
CHALLENGES
5
• Exclusively mathematical approach to identify institutions
for comparison groups was elusive
• A balance of data, statistics, and judgment was needed
• Data from IPEDS based on about 10 institutional characteristics
generated lists of 20-50 institutions in most cases
• Outliers remained a problem, likely due to data quality / unique
state practices
• Campuses asked to add or remove institutions from the
comparison group based on judgment to arrive at 10-15
comparison institutions
COMPARISON GROUPS
6
• Controls for student inputs are essential to the
interpretation of graduation rates; individual factors
matter much more than institutional factors
• Data from HERI illustrate this relationship, independent of
institutional effects
GRADUATION RATES
7
INSTITUTION-BASED 6-YEAR GRAD RATES BY
SAT COMPOSITE SCORE
Source: IPEDS Data Center Public Institutions Private Not-for-Profit Institutions
8Source: Higher Education Research Institute (UCLA)
INDIVIDUAL-BASED GRADUATION RATES
BY HIGH SCHOOL GPA
6.3 9.815.5
25.2
35.9
47.8
58.3
21.227.7
36.6
48.7
59.8
70.679.3
0
10
20
30
40
50
60
70
80
90
100
C or less C+ B- B B+ A- A+, A
Gra
du
atio
n R
ate
(per
cen
t)
HS GPA as Reported on the CIRP Freshman Survey
4-Years 6-Years
9Source: SBU Office of Institutional Research, Planning & Effectiveness
STONY BROOK UNIV. GRADUATION RATES
BY HIGH SCHOOL GPA
33.940.2
46.150.3
60.658.962.6
67.371.9
79.1
0
10
20
30
40
50
60
70
80
90
100
Below 87 87-89.9 90-92.9 93-95.9 96+
Gra
du
atio
n R
ate
(per
cen
t)
HS GPA
4-Years 6-Years
10Source: SBU Office of Institutional Research, Planning & Effectiveness
STONY BROOK UNIV. GRADUATION RATES
BY HIGH SCHOOL GPA – AFRICAN AMERICAN ONLY
39.7
51.7 52.3 49.156.9
66.3
75.2 74.5 71.980.4
0
10
20
30
40
50
60
70
80
90
100
Below 87 87-89.9 90-92.9 93-95.9 96+
Gra
du
atio
n R
ate
(per
cen
t)
HS GPA
4-Years 6-Years
11
Connecticut adopted a completions per FTE metric
intended to supplant graduation rate
• Statistical adjustments were necessary
– Lag enrollment by 2 years
– Weight certificates by 1/3
– Degree-seeking UG enrollment only
• Data adjustments were necessary
• Results not comparable across sectors
• Cannot be interpreted without reference groups
COMPLETIONS PER 100 FTE ENROLLMENT
12
Deemed very important for quality assurance but
measurement deferred pending further research
Research had demonstrated• Testing is sensitive to recruitment practices and testing conditions
• Student motivation affected institutional results
State pursuing external validation of learning using
“authentic assessment” through a nine state
multi-state collaborative
LEARNING OUTCOMES
13
• Net price involves reasonable components, but
wide variation exists in how institutions determine
costs associated with room & board and other
expenses for students living off-campus.
• Regulatory or legislative action needed to define
calculation method
AFFORDABILITY AND NET PRICE
14Source: IPEDS Data Center, components are for 2011-12
EXAMPLE NET PRICE ANOMALIES
$6,840 $5,823
$13,117
$4,900
$7,410
$2,146
$8,528
$3,450
$0
$5,000
$10,000
$15,000
$20,000
$25,000
Assoc(public under 4-yr)
Assoc(public rural)
Private, for-profitnon-degree granting
Private, not-for-profit4-yr Bacc
Expenses Living Off-Campus, Not with Family:4 Institutions within 1.6 miles
room and board other expenses
0 0.4 1.5 1.6Distance (miles)
15
Available employment data have severe limitations
• Alumni survey response rates for public institutions typically range
20-30%, with likely high levels of non-respondent bias
• State unemployment insurance (UI) data do not cross state lines or
identify full-time/part-time employment or partial/full quarter earnings
• IRS data are problematic for joint returns
• Age and prior employment confound central tendencies – more
research is needed
EMPLOYMENT OUTCOMES
16Source: CT Employment & Training Commission Legislative Report Card (2013)
EXAMPLE WAGE DATA ANOMALIES
$540
$1,076
$473 $507
$254 $270
$451 $472
$0
$200
$400
$600
$800
$1,000
$1,200
CT Community Colleges Charter Oak State College(Public online college)
CT State Universities(Public Regional)
Univ. of Connecticut(Public Flagship)
Avg. Weekly Wages & Change in Weekly Wages for Undergraduate Completers
Weekly Wages Upon Entering Employment Change in Wages from Start of Program
Data gathered from unemployment insurance records through CT Dept. of Labor
17
• Consider institutional input for comparison
groups
• Adopt individualized graduation rate calculator
• Define cost of attendance methodology for off-
campus arrangements
• Defer inclusion of and support research on:
• Learning outcomes
• Employment outcomes
RECOMMENDATIONS
18
Connecticut Dept. of Labor and Connecticut Dept. of Higher Education (2010). Building Connecticut’s
Workforce: Report on 2007-08 Graduates. Retrieved January 13, 2014 from
http://www.ctdhe.org/info/pdfs/2010/HigherEdReport-2008grads.pdf
Connecticut Higher Education Coordinating Council. (2012). Public Policy Framework for Higher
Education. Retrieved January 10, 2014 from http://www.ct.edu/files/opr/A_CTFramework.pdf.
DeAngelo, L., Franke, R., Hurtado, S., Pryor, J. H., & Tran, S. (2011). Completing college: Assessing
graduation rates at four-year institutions. Los Angeles: Higher Education Research Institute, UCLA.
Hosch, B. (2008). Institutional and student characteristics that predict retention and graduation rates.
Annual Conference of the North East Assoc. for Institutional Research, Providence, RI.
Hosch, B. (2012). Time on test, student motivation, and performance on the Collegiate Learning
Assessment: Implications for institutional accountability. Journal for Assessment and Institutional
Effectiveness 2(1): 55-76.
National Governors Association. (2010). Complete to Compete: From Information to Action: Revamping
Higher Education Accountability Systems
REFERENCES