Embed Size (px)
Citation preview
ESL and
Developmental
Placement:
What’s the
Connection?
2
Who I AmEnglish (etc.) teacher since 1988
Director of First Year Experience at Central Piedmont
President of TYFY (Two Year First Year) (Sept. 26)
Community college researcher and structuralist
Chaired the NCCCS placement testing committee
Author of “Placement of International English Language Learners: How Different Is It?” (JIMPHE)
Not an ESL expert
3
Topics for Today
A comparison of placement systems
ESL versus Developmental systems
Traditional versus Innovative systems
Including what’s at stake
Assessing your own placement system using a rubric
4
My Suggestion
Do as I say, not as I do
5
The Need
T or F: all learners come to community college ready to
succeed in college level classes on their own
T or F: all English language learners come to community
college with sufficient language skills to succeed in
college level classes taught in English, on their own
Q: which courses could a student pass if they lacked the
English language? Reading and writing skills? Math?
6
On The Other Hand
The Growth Mindset
“In the case of international students, language challenges are 90%
symptom and 10% problem.”
“I could take a student with low TOEFL score and turn her into a
dynamo (even if we consider just ONE factor—instructor’s
pedagogical approach) or I could take the highest scorer and fail
him miserably.”
I start my first-year writing class by saying: “Yes, you can write terrific
papers in terrible English.”
7
The Treatment
NCCCS colleges can offer free ESL classes to
members of our communities
NCCCS colleges can also offer credit EFL classes
5 credits each, 4 levels, 4 subjects plus 2 additional
final classes
4 of 58 colleges offer EFL classes
8
Placement Theory
Traditional: we should place students based on
what they know
Better: we should place students based on
what they can learn
9
Current Practice
Developmental Placement
Which learners do you place into
developmental English and math classes?
How do you determine their placements?
How do you validate those placements?
10
Current Practice
ESL Placement
Which English language learners do you place
into ESL classes?
How do you determine their placements?
How do you validate those placements?
11
ESL Challenges
Identification of English Language Learners
In 2016, 22% or 12+ million kids spoke a language other than
English at home
4% or 2.4 million kids had difficulty speaking English
3% or 2.5 million kids were foreign born (Kids Count Data
Center)
12
ESL Challenges
How do we accurately assess which ELLs need help?
Prescription: what kind of help do they need?
For how long?
If it takes 4 to 7 years for an ELL to learn their way out of
that designation, do we know where each student is on
that continuum?
13
Identification
For research, I created a cohort definition of ELLs as having any of these:
Citizenship from a non-English speaking country
Primary language other than English
Taking a test of English language skills
Taking a credit Academic ESL course (EFL)
This yielded 418 (6.8%) ELLs out of 6133 new fall 2016 curriculum students
14
ESL Challenges
What kind of help do they need?
For how long?
15
71.5%
79.9%
44.9%
63.5%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
New Fall 2016 Native Speakers New Fall 2016 Non-Native Speakers
English Language Learners: At Risk?
Course Success Fall-to-Fall Retention
16
69.2%73.4% 72.6% 71.2%
76.3%80.3%
86.3%
78.8%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
DRE ENG DMA Non-Dev, Non-EFL
English Language Learners: Course Success
New Fall 2016 Native Speakers New Fall 2016 Non-Native Speakers
17
Identification
According to the literature, ELLs are over-represented in
developmental classes, presumably because they don’t
do as well on standard placement tests
The challenge is not simply to identify non-native speakers,
but to quickly identify the ones likely to struggle in classes
18
Predictive Validity: Method 1
Correlations between placement test score and first
term grade, ideally with no intervening treatment
0% to 10% for developmental placement tests
0% to 12% for ESL placement tests
19
0.8% 0.9%1.8% 2.1%
9.1%
2.8%
6.7%8.4%
10.7%
14.7%
0%
5%
10%
15%
20%
25%
NCDAP DREWritePlacer
NCDAP DREReading
NCDAP DREComposite
NCDAP DRERevising & Editing
HSGPA
Fall 2016 New Student Success By Placement Factor
English 111 Grade Mean Grade Online Success Rate Online GPA
Predictive Validity: Method 2
Accuracy Rates
21
Predicted to Pass College Level
Predicted to Fail College Level
Developmental Placement
Underplaced Accurate
College Level Placement Accurate Overplaced
Predictive Validity: Method 3
Customer satisfaction (students and faculty)
But satisfaction data doesn’t always mirror reality
22
Predictive Validity
We often don’t choose predictors based on strong
correlations
Students in research don’t mirror our new students
populations: they are often traditional age university
students
We haven’t always considered what the placement
system does to the student—the consequence
23
Consequence Validity
What goals do we have for students?
Under what system are students most likely to reach
those goals?
24
25
16%20%
24%
63%
33%39%
47%
71%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
20
10
/20
11
20
13
/20
14
20
14
/20
15
GA
Co
req
uis
ite
20
10
/20
11
20
13
/20
14
20
14
/20
15
GA
Co
req
uis
ite
Math English
Completed Gateway Course: NC Versus Georgia
Multiple measures Multiple measures
Consequence Validity
Measures the benefits and harm that result from a system
We can compare similar students who do or do not undergo a
certain placement or treatment
We can use various statistical approaches such as random
assignment experiments, regression discontinuity, propensity score
matching, difference in differences, simple means comparisons,
and other quasi-experimental designs
26
The Ideal Placement Validity
Is supported by theory
Facilitates predictions
Generates positive outcomes
Positive outcomes are the most important, or even the only
important factor, and until recently, the least considered
27
Single Measures28
Arkoudis (2011): "The sector's blind faith in
language testing inhibits the development of
more robust ways of addressing English
language outcomes for graduates (p. 230)"
Single Measure = Test
Tests are fast and cheap
Tests are validated . . . using predictive validity
only, and not with your students
Tests are standardized . . . but students aren’t
29
Tests Involve Mind-Reading
7 I get up _______ 7 o’clock in the morning.
A for B at C in
8 I like apples, but I _______ bananas.
A don’t like B like C do like
9 Excuse me, _______ speak French?
A do you B you do C you
To a native English speaker, all 9 of these 9 constructions could work correctly,
depending on the purpose and the resulting inflection
30
Multiple Measures31
ETS (1997, 2011): colleges should base ESL
placements on “all available relevant
information, not solely on TOEFL scores” (1997)
Multiple Measures32
ETS (2011): use “grade point average, scores
on other admissions exams, teacher
recommendations, and interviews with
individuals,” plus total score not specific sub-
scores, and avoid rigid cut scores (2011, 1997)
Multiple Measures and
Accuracy Rate
James and Templeman (2009) found that a
multiple measures approach improved
placement accuracy over ACCUPLACER ESL
test scores alone from 66.5% to 84.1% in ESL
reading, and from 46.2% to 81.3% in writing
33
Some Data Issues
Age and experience matter
The newest data is the most predictive
The oldest data yields the greatest success
34
Should Placements Expire?35
Is old data still valid?
Judging by NCCCS data (used in Belfield & Crosta, 2011) it takes years before the predictive power of HSGPA drops for math. For tests, the drop is rapid.
0.26
0
0.41 0.39 0.43
0 0
0.14 0.18 0.19
0 0
0.24 0.24
0.43
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
2006 (N = 21/-/-) 2007 (N = -/-/-) 2008 (N =1279/424/476)
2009 (N =1453/563/509)
2010 (N =132/68/44)
Correlation to NCCCS MAT-161 College Algebra Grade
HSGPA ACCUPLACER Elementary Algebra SAT Math
Study of New Fall 2015 Students (196 With 8+ Year Old High School
Transcripts)
Is our 5 year HSGPA expiration date warranted?
88.0%
77.3% 78.3% 80.3% 79.1%73.3% 74.8%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
16+ Years2.6+
8-16 Years2.6+
0-5 Years2.6+
Other 16+years
Other 8+Years
Other 0-5Years
All New2015
All Delivery Methods Success Rates By Transcript Age
Meets 2.6+ HSGPA Does Not Meet 2.6+ HSGPA
Study of 108 New Fall 2016 Students by Age of College Transcripts
(Using the student’s most recent prior transcript)
Should college GPAs and credits be used for placement? Is an expiration data warranted for them?
College GPA (CGPA) beats HSGPA. Experience matters
0.01 0.04 0.05 0.080.17
0.330.41
0.000.100.200.300.400.500.600.700.800.901.00
By HighSchool (N =
21)
By HighSchool (N =
39)
By College(N = 31)
By College(N = 71)
If CollegeTranscriptAge > 10
Years
By All HighSchool (N =
60)
By Age ofCollege
Transcript(N = 102)
If CollegeTranscriptAge <= 10
years
If CollegeTranscriptAge > 10
Years
By AllCollege (N =
102)
If CollegeTranscriptAge <= 10
years
Correlation to 1st Term GPA for Students Transferring In Credits
Students with older transcripts had lower high school and prior college GPAs
2.31
2.58
2.692.75
2
2.1
2.2
2.3
2.4
2.5
2.6
2.7
2.8
High School (N = 21) College (N = 31) High School (N = 39) College (N = 71)
College Transcript Age> 10 Years
College Transcript Age> 10 Years
College Transcript Age<= 10 years
College Transcript Age<= 10 years
Transcript GPA Newer Transcripts
Older Transcripts
But the students with older transcripts also had higher 1st term CPCC GPAs
2.75 2.823.02 3.06
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
College (N = 71) High School (N = 39) College (N = 31) High School (N = 21)
College Transcript Age<= 10 years
College Transcript Age<= 10 years
College Transcript Age> 10 Years
College Transcript Age> 10 Years
CPCC 1st Term GPA
Older TranscriptsNewer Transcripts
Should Placements Expire?43
In terms of consequences, using older
transcripts and test scores makes sense
Suppose you go to buy this car
44
But the dealer wants cash
45
46But you don’t have $72,000. The dealer is
skeptical about your ability to afford the
vehicle, and says you can finance the
Navigator only if
you first buy a
Fiesta for
$14,000 and
finance that.
And no early
payoffs!
How would you feel?
47
The StakesOut of State Tuition, Fees, and Books
$22,352 and 2.5 years to take EFL and
developmental English and math, doubling the cost
and more than doubling the time to degree
$44,612 and 4.5 years to complete a 2-year degree
48
Optional Treatments49
If the treatment is helpful but expensive, why
not let students choose?
After all, students are adults with some
understanding of their own needs
Required Treatments50
But if the treatment is important, why leave it to
chance?
After all, students don’t do optional
And we’ve known since the 1970s that laissez-
faire policies don’t work
Treatment
According to Fass-Holmes and Vaughan
(2015), we should target “the specific students
with demonstrable academic struggles rather
than all incoming international undergraduates
who are not native English speakers”
51
More High Stakes
Should Students Prepare?
Is the test more accurate if
students “test cold?”
52
Sometimes even a wizard needs to review
How much time and money would be saved if all students reviewed before testing?
Test Time MoneyArithmetic $546,980.0
Algebra $500,321.5
Reading Comprehension $219,316.8
Sentence Skills $273,446.5
Total 20 Weeks $1,540,064.79
20 Weeks
8 Weeks
Expiration Dates Cost $
Moving to a 10-year expiration date would save
students $181,440 per fall term at my college
Or $3,000,000 per year across all 58 NCCCS colleges
An unlimited policy could save $4,250,000 per year
across the 58 colleges
And boost, not reduce, completion rates
55
But Whose Money Is It?
A dean once asked me that question
I said it was the students’ money
He said “Oh”
56
Predictive Identification57
Is weaker
Traditionally locks us and students in to a course of action that may not be needed or helpful
Done right, takes time and a team
But then, whose time is it? The hour of staff time or the months or years of a student’s time?
Diagnostic Identification58
Could come too late
Could mislead us about causes
But is up to 100% accurate in identifying a
condition
Why corequisite?
People LBD
In both ESL and developmental curriculums, there is a
tendency toward skills based instruction
For example, we might teach someone how to define
something, then at a higher level to recognize it, then later
to perform it, then perform it in context
59
Prerequisite Versus
Corequisite
Corequisites are better pedagogically as they are
more likely to recognize that people LBD
Corequisites work better structurally because more
students complete when we open the open door
60
Required Treatments
MALDEF settlement (Romero-Frias et al. v. Mertes et al.,
1988) stipulated that we should require prerequisites only
when a student without the prerequisite is “highly unlikely
to receive a satisfactory grade”
61
Required Treatments
In Kanno and Grosik (2011), non-F-1 ESL students at one university
saw the mandatory placement and treatment policy they faced as
unfair because the courses carried no credit, were expensive, and
represented a burden that other student populations did not face
In contrast, students interviewed at a different university expressed
appreciation for their free, non-remedial ESL instruction, which
involved special ESL sections of first-year writing courses
62
Putting It Together
Lower the stakes by making the treatment cheap and convenient
Do as your colleagues do:
Improve your placement system (Ozarks Tech p. 52)
Open the open door using corequisites (Pellissippi State, p. 58)
Teach the student, not the subject (Reading Area CC, p. 41)
Monitor success using analytics and KPIs (Piedmont Tech, p. 45)
63
Placement System Rubric64
Validated by . . . Predictive validity Consequence validity
Validated using . . . Students elsewhere Student
Consists of . . . A single measure Multiple measures
Applies to . . . A population group Individuals choosing placement
Resulting treatments are . . . Required Optional
Identifies students . . . Predictively before entry Diagnostically after entry
Resulting treatments are given as . . . Prerequisites Corequisites
Resulting treatments are . . . Time-consuming or inconvenient Relatively brief and convenient
Resulting treatments are . . . Expensive Inexpensive
