Upload
diana-bond
View
222
Download
1
Tags:
Embed Size (px)
Citation preview
“DEFINING PROMISE: OPTIONAL STANDARDIZED TESTING POLICIES IN
AMERICAN COLLEGE AND UNIVERSITY ADMISSIONS”
William C. Hiss, Principal Investigator Valerie W. Franks, Co-Author & Lead Researcher
OACAC Articulation ConferenceDenison University
Granville, OHSeptember 9, 2014
http://www.nacacnet.org/research/research-data/nacac-research/Documents/DefiningPromise.pdf
2
“Take Back the Conversation…” *
Ethical questions… • Can optional testing lower access barriers and expand college bound populations?• Who is exploring the breadth of human intellect and promise in imaginative ways?• Who is doing the heavy lifting, serving broad constituencies?
Practical questions… • What happens when institutions admit students without considering their
standardized test scores?• Who are the non-submitters?• How are these “non-submitter” students doing, compared to submitters?• Are college admissions decisions still reliable without testing?• Does a non-submitter policy come with any advantages or disadvantages to the
institution? (e.g., institutional geographic reach, diversity, academic achievement, tuition income and financial aid)
*The NACAC Commission on Standardized Tests included a recommendation to “take back the conversation” about testing from the various groups for whom testing was either a profession or a cause. This study is a contribution to that conversation.
3
Policy Variations
Policy Variations Not Considered in this Study: Test Flexible – Policy whereby students have the option to submit scores from other standardized tests in place of the SAT or ACT
Very Limited Options – Only specific categories, e.g., non-traditional students, specific degree programs, …
• Institutions on the Fairtest list were reviewed and categorized as follows:
1. Academic Threshold – Policy whereby students who meet certain academic criteria (e.g., rank, HSGPA) can choose whether or not to submit standardized testing scores as part of the admissions decision, or their score is still required to be submitted but waived in the admissions decision if criteria is met (typically referred to as Assured Admission or Guaranteed Admission).
2. Recommended for Placement – Policy whereby standardized testing is recommended to be submitted for placement purposes, and if submitted, is typically not used as a data point in the admissions decision.
3. Optional for All – Policy whereby almost all prospective students can choose whether or not to submit standardized testing scores (SAT or ACT) as part of their admissions application.
4. Optional Plus – Policy whereby students are required to submit something else in lieu of testing (e.g., schedule an interview or supply extra writing samples).
5. Recommended, Not Required – Policy whereby testing options are qualified for prospective students in some fashion (e.g., recommended, highly recommended, preferred, etc.).
43%
13%
23%
5%
17%
4
Policy Breakdown of our Institutions
• The institutions in our study use four of the policy category options. 60% of students in our sample entered higher education under the “Academic Threshold” type of policy, whereby meeting a HSGPA or rank threshold gained them guaranteed admission.
Optional Plus
Recommended for Placement
Optional for All
Academic Threshold
0 10000 20000 30000 40000 50000 60000 70000 80000
Number of Students Enrolled by Policy Type
Number of Students in the Study
5
Research Sample Overview
20 Private
Colleges and Universities(37,611 records)
6 Public
Universities(71,831 records)
5 Minority Serving
Institutions(12,691 records)
2Arts
Institutions(783 records)
• Total of 33 Institutions with 122,916 Student Records (Approximate 5% sample of optional testing institutions in each institutional category)
• From 23 states and US territories
• Normally submitted 4 cohort years, with data from alumni and currently enrolled students
• Pool: roughly 850 institutions listed by Fairtest, reduced to about 450 to only include 4-year, non-profit, IPEDS-submitting, and with national visibility. 120 institutions and state systems examined and contacted to choose 33.
6
Publically Announced Institutions in the Study
7
Summary by Institution Type
Private Colleges and Universities• Primarily “Optional for All” policy, the
best-known form of optional testing.• Average non-submitter population was
35% in 2010 (up from 26% in 2003).
Public Institutions• Primarily “Academic Threshold” policies based
on HSGPA or HS Rank. Often state mandated, & called “guaranteed” or “assured” admission.
• 62% average non-submitter population in 2010 (down from 66% in 2003).
• This analysis focuses on non-submitters with testing below that institution’s standardized testing threshold or average. This allowed us to better understand the students who might not have been admitted without the policy.
Minority Serving Institutions• Primarily “Recommended for Placement”
or open admission.• Average non-submitter population was
27.5% in 2010.• Data gaps limit the analysis in this dataset.
Arts Institutions• Primarily “Optional for All” • Average non-submitter population was
64% in 2010.• Portfolio ratings were included in the data
request.• Small N’s, so limited conclusions.
8
Principal Findings
1. There are no significant differences in either Cumulative GPA or graduation rates between submitters and non-submitters. Across the study, non-submitters (not including the public university non-submitters with above-average testing, to focus on the students with below-average testing who are beneficiaries of an optional testing policy) earned Cumulative GPAs that were only .05 lower than submitters, 2.83 versus 2.88. The difference in their graduation rates was .6%. By any standard, these are trivial differences.
2. College and university Cumulative GPAs closely track high school GPAs, despite wide variations in testing. Students with strong HSGPAs generally perform well in college, despite modest or low testing. In contrast, students with weak HSGPAs earn lower college Cum GPAs and graduate at lower rates, even with markedly stronger testing. A clear message: hard work and good grades in high school matter, and they matter a lot.
3. Non-submitters are more likely to be first-generation-to-college enrollees, all categories of minority students, Pell Grant recipients, and women. But across institutional types, white students also use optional testing policies at rates within low single digits of the averages, so the policies have broad appeal across ethnic groups.
4. College admissions decisions made without testing are apparently just as reliable as those made with testing. Testing may serve to artificially truncate the applicant pools of students who would succeed if they could be convinced to apply.
9
Additional Findings
5. In a surprise finding, non-submitters display a distinct two-tail or bimodal curve of family financial capacity. First-generation, minority and Pell-recipient students will often need financial aid support, but large pools of students not qualifying for or not requesting financial aid help balance institutional budgets.
6. LD students, from a modest sample of 1050 students at 8 institutions, are much more likely to apply as non-submitters, and much more likely to apply ED. They perform at levels close to the rest of their classmates. The evidence from a long-term study at Bates found that given the modest accommodations to which these students are legally entitled, their GPAs and graduation rates come up to class averages, helping to increase the institution’s overall graduation rates.
7. Non-submitters may commonly be missed in consideration for no-need merit financial awards, despite better Cum GPAs and markedly higher graduation rates than the submitters who receive merit awards. Institutions may want to examine their criteria for merit awards, especially the use of standardized testing to qualify students for no-need merit funding.
8. Non-submitters often expand applicant pools, apply Early Decision at higher rates, increase minority enrollments, expand geographic appeal, and allow for success by Learning Difference students.
10
College Cumulative GPA versus High School GPA(Aggregate Data – All students entering 2003, 2004, 2006, 2007, 2008, 2009, 2010)
1.6 2.0 2.4 2.8 3.2 3.6 4.01.6
2.0
2.4
2.8
3.2
3.6
4.0Submitter (N:41608)
Non-Submitters -- W/O Public Above-Average-Testing (N:24610)
High School GPA
Colle
ge C
umul
ative
GPA
11
College Cumulative GPA versus SAT(Aggregate Data – All students entering 2003, 2004, 2006, 2007, 2008, 2009, 2010)
600 800 1,000 1,200 1,400 1,6001.6
2.0
2.4
2.8
3.2
3.6
4.0Submitter (N:55673)
Non-Submitters -- W/O Public Above-Average-Testing (N:24824)
SAT(No Writing Score; All ACT scores converted to SAT Scores)
Colle
ge C
umul
ative
GPA
12
Without Above-Average-Testing Students
Aggregate Graduation Rate
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
15488
15488
10409
18193 students
Non-Submitter Submitter
The aggregate non-submitter graduation rate is: 65.8%.
Non-submitter rate is 1.3% higher.
The aggregate non-submitter graduation rate is: 63.9%
Non-submitter rate is 0.6% lower.
Graduation Rate
Graduation Rate Comparison(Aggregate Data – Graduated Cohorts entering 2003,2004, 2005, 2006, 2007)
13
Summary of Key Statistics - All 33 Institutions(students entering 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010)
< 0.1 = trivial difference0.1 - 0.3 = small difference0.3 - 0.5 = moderate difference> 0.5 = large difference
COLOR KEYCohen’s d
Note: Details on Cohen’s d can be found in the accompanying excel
spreadsheet.
No Significant Difference
Statistically Significant Difference p < .001
COLOR KEYChi-Square Tests
Note: Details on chi-square tests can be found in the accompanying
excel spreadsheet.
Non-Submitter
s
Submitters
n 62067 60743
High School GPA 3.45 3.28 Cohen’s d
SAT (See caveat below) 1129 1154 Cohen’s d
Cumulative GPA 2.92 2.88 Cohen’s d
Graduation Rate 65.8% 64.5% Chi-Square
Without Above-Average-Testing Students
n 36648 60743
High School GPA 3.35 3.28 Cohen’s d
SAT (See caveat below) 1041 1154 Cohen’s d
Cumulative GPA 2.83 2.88 Cohen’s d
Graduation Rate 63.9% 64.5% Chi-Square
SAT Caveat: 82.0% of Non-Submitters still submitted scores . This data only represents that 82.0%. For the second chart, the results were calculated with those students at the six public universities removed who had testing above the average of their institution. In this way, the data reflects only those students in public institutions with testing below their institutional averages who were beneficiaries of an automatic admission program based on HSGPA or HS rank,, or who chose to apply as a non-submitter in an institution that had a pure optional testing policy.
14
Academic Rating: All institutions submitted their respective scales, but for comparison purposes we converted all of them to a 10 point scale, where 10 is the highest rating.SAT Caveat: Only 41% of Non-Submitters still submitted scores . This data only represents that 41%.
Non-Submitter
s
Submitters
n 12004 24855
High School GPA 3.47 3.51 Cohen’s d
Academic Rating 6.53 6.76 Cohen’s d
SAT (See caveat below) 1096 1245 Cohen’s d
First Year GPA 2.98 3.13 Cohen’s d
Cum GPA (enrolled cohorts) 3.04 3.17 Cohen’s d
Cum GPA (graduated cohorts) 3.08 3.18 Cohen’s d
Graduation Rate** 77.7% 76.6% Chi-Square
Completion Rate** 101.4% 102.2% Cohen’s d
Underrepresented Minority 16% 9% Chi-Square
First Generation 16% 10% Chi-Square
Gender (Female) 65% 59% Chi-Square
Pell 23% 17% Chi-Square
EFC $21,790 $26,303 Cohen’s d
EFC – Adjusted for Inflation $10,570 $12,817 Cohen’s d
STEM Major 24% 32% Chi-Square
Summary of Key Statistics – Private Institutions(students entering 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010)
< 0.1 = trivial difference0.1 - 0.3 = small difference0.3 - 0.5 = moderate difference> 0.5 = large difference
COLOR KEYCohen’s d
Note: Details on Cohen’s d can be found in the accompanying excel
spreadsheet.
No Significant Difference
Statistically Significant Difference p < .001
COLOR KEYChi-Square Tests
Note: Details on chi-square tests can be found in the accompanying
excel spreadsheet.
15
SAT Caveat: Only 98.9% of Non-Submitters still submitted scores . This data only represents that 98.9%.
Non-Submitter
s
Submitters
n 19976 26330
High School GPA 3.40 3.12 Cohen’s d
SAT (See caveat below) 1037 1130 Cohen’s d
First Year GPA 2.76 2.68 Cohen’s d
Cum GPA (enrolled cohorts) 2.74 2.74 Cohen’s d
Cum GPA (graduated cohorts) 2.78 2.62 Cohen’s d
Graduation Rate** 67% 63% Chi-Square
Completion Rate** 112.2% 113.8% Cohen’s d
Underrepresented Minority 24% 15% Chi-Square
First Generation 32% 22% Chi-Square
Gender (Female) 60% 45% Chi-Square
Pell 27% 15% Chi-Square
EFC $14,825 $17,271 Cohen’s d
EFC – Adjusted for Inflation $7,409 $8,627 Cohen’s d
STEM Major 51% 53% Chi-Square
Summary of Key Statistics – Public Institutions(students without above-average-testing entering 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010)
< 0.1 = trivial difference0.1 - 0.3 = small difference0.3 - 0.5 = moderate difference> 0.5 = large difference
COLOR KEYCohen’s d
Note: Details on Cohen’s d can be found in the accompanying excel
spreadsheet.
No Significant Difference
Statistically Significant Difference p < .001
COLOR KEYChi-Square Tests
Note: Details on chi-square tests can be found in the accompanying
excel spreadsheet.
16
SAT Caveat: The average for non-submitters represents only one institution that had scores for non-submitters, so it is not an accurate comparison with submitters across the institutions. ** Graduated Cohorts Only
Non-Submitter
s
Submitters
n 3494 9197
High School GPA 2.61 3.00 Cohen’s d
SAT (See caveat below) 791 974 Cohen’s d
First Year GPA 2.52 2.76 Cohen’s d
Cum GPA (enrolled cohorts) 2.43 2.69 Cohen’s d
Cum GPA (graduated cohorts) 2.31 2.66 Cohen’s d
Graduation Rate** 24% 37% Chi-Square
Completion Rate** 114% 117% Cohen’s d
Underrepresented Minority 51% 41% Chi-Square
First Generation 42% 40% Chi-Square
Gender (Female) 55% 59% Chi-Square
Pell 49% 43% Chi-Square
EFC $8,966 $13,634 Cohen’s d
EFC – Adjusted for Inflation $4,586 $6,666 Cohen’s d
STEM Major 5% 11% Chi-Square
Summary of Key Statistics – Minority Serving Institutions(students entering 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010)
< 0.1 = trivial difference0.1 - 0.3 = small difference0.3 - 0.5 = moderate difference> 0.5 = large difference
COLOR KEYCohen’s d
Note: Details on Cohen’s d can be found in the accompanying excel
spreadsheet.
No Significant Difference
Statistically Significant Difference p < .001
COLOR KEYChi-Square Tests
Note: Details on chi-square tests can be found in the accompanying
excel spreadsheet.
Application Comparisons – Merit Award Recipients(students entering 2003, 2004, 2006, 2007, 2008, 2009, 2010)
17SAT Caveat for All Merit Recipient Chart: 90.7% of Non-Submitters still submitted scores . This data only represents that 90.7%.SAT Caveat for Merit Recipients Without Above-Average-Testers: Only 74.9% of Non-Submitters still submitted scores . This data only represents that 74.9%.** Graduated Cohorts Only
Non-Submitters
Submitters
n 13,708 13,603
High School GPA 3.69 3.44 Cohen’s d
SAT (See caveat below) 1205 1197 Cohen’s d
First Year GPA 3.21 3.07 Cohen’s d
Cum GPA (enrolled cohorts) 3.19 3.13 Cohen’s d
Cum GPA (graduated cohorts) 3.24 3.02 Cohen’s d
Graduation Rate** 81% 70% Chi-Square
Completion Rate** 109% 107% Cohen’s d
Underrepresented Minority 16% 14% Chi-Square
First Generation 20% 18% Chi-Square
Gender (Female) 57% 54% Chi-Square
Pell 16% 15% Chi-Square
EFC $21,607 $24,364 Cohen’s d
EFC – Adjusted for Inflation $10,734 $12,045 Cohen’s d
STEM Major 51% 44% Chi-Square
Non-Submitters
Submitters
5064 13,603
3.54 3.44
1054 1197
2.99 3.07
2.97 3.13
3.05 3.02
76% 70%
107.2% 106.9%
26% 14%
24% 18%
64% 54%
18% 15%
$22,702 $24,364
$11,267 $12,045
49% 44%
All Merit Recipients Merit Recipients Without Above-Average-Testers
18
Not a victory lap but a legacy lap… The next steps?• Examine the twin issues of “false negatives” of low testing with potential “false
positives” of high testing created by coaching. ₋ An ethicist’s question: if we had a medical test for a serious condition with a
30% rate of false negatives, would that be OK?₋ If 30% is the non-submitter share of enrolling students, what is the true share of
false negatives created by testing, including those who are refused, attending community colleges and for-profit colleges, or not attending at all?
• Examine college success using 4-year Cum GPAs and graduation rates rather than first-year GPAs as the principal yardstick. Add alumni and grad school outcomes in future studies. (See the Bates 25-year look-back study for some longer-term data.)
• Evaluate a broader band of research tools. Add Cohen’s d, Chi Square, bar charts and scatterplots to regression (R-square) analysis.
• Share published research on optional testing. Good models are available from Bates and Ithaca, and in Joseph Soares, ed: SAT Wars: The Case for Test-Optional Admissions. See also Bowen, Chingos & McPherson, Crossing the Finish Line: Completing College at America’s Public Universities.
• A one-day conference on optional testing?
19
Institutions Recently Adopting Test-Optional Policies
20
Questions and Discussion
William C. HissPrincipal Investigator
[email protected] Hadfield RoadMinot, ME 04258
Valerie W. FranksCo-Author and Lead Researcher
[email protected] Lindenwood Road
Cape Elizabeth, ME 04107