View
29
Download
0
Category
Tags:
Preview:
DESCRIPTION
Strengths & Weaknesses of IALS as an Assessment Framework: A Longitudinal Perspective. Stephen Reder Portland State University. I. The Centre for Literacy Fall Institute. Banff, Alberta - October 24, 2011. IALS, Its Meaning and Impact for Policy and Practice. Where I’m Coming From. - PowerPoint PPT Presentation
Citation preview
Strengths & Weaknesses of IALS Strengths & Weaknesses of IALS as an Assessment Framework:as an Assessment Framework:
A Longitudinal PerspectiveA Longitudinal Perspective
Stephen RederStephen RederPortland State UniversityPortland State University
I
IALS, Its Meaning and Impact for Policy and Practice
The Centre for Literacy Fall Institute
Banff, Alberta - October 24, 2011
Where I’m Coming FromWhere I’m Coming From
Literacy researcher interested in how the IALS Literacy researcher interested in how the IALS Framework affects policies and programs in Framework affects policies and programs in adult basic educationadult basic education
Some of my researcher and practitioner Some of my researcher and practitioner colleagues believe that IALS is not relevant to colleagues believe that IALS is not relevant to their programs and studentstheir programs and students
Although I believe there Although I believe there isis a disconnect between a disconnect between the IALS Framework and the needs of adult the IALS Framework and the needs of adult education, we can fix iteducation, we can fix it! !
Strengths of IALS FrameworkStrengths of IALS Framework (for adult education)(for adult education)
Contextualizes literacy in terms of important Contextualizes literacy in terms of important social & economic indicators social & economic indicators Provides compelling Provides compelling snapshotssnapshots of adult literacy of adult literacy at given points in timeat given points in timeBuilds technically sound assessment Builds technically sound assessment instruments and proceduresinstruments and proceduresPays careful attention to maintaining Pays careful attention to maintaining comparability of large-scale assessments over comparability of large-scale assessments over time and spacetime and space
Weaknesses of IALS FrameworkWeaknesses of IALS Framework(for adult education)(for adult education)
Supports increased literacy proficiency as a Supports increased literacy proficiency as a policy goal but…policy goal but…
……does not provide useful tools for assessing does not provide useful tools for assessing program impact on literacyprogram impact on literacy
……is of limited use for short-term accountability is of limited use for short-term accountability regimes & program improvement effortsregimes & program improvement efforts
The Longitudinal Study of Adult Learning The Longitudinal Study of Adult Learning (LSAL)(LSAL)
funded by
U.S. Department of Education
and
National Institute for Literacy
Portland State University
Longitudinal Study of Adult LearningLongitudinal Study of Adult Learning
Decade-long panel study of Portland high school dropouts, Decade-long panel study of Portland high school dropouts, age 18-44 at the beginning of the studyage 18-44 at the beginning of the studyRepresentative sample of ~1,000 drawn from local rather than Representative sample of ~1,000 drawn from local rather than national population of dropoutsnational population of dropoutsPeriodic in-home interviews and literacy assessments and Periodic in-home interviews and literacy assessments and SSN-linked administrative data (with individuals’ permission)SSN-linked administrative data (with individuals’ permission)Examines program participation and other learning activities, Examines program participation and other learning activities, social and economic changes, and changes in literacy skills, social and economic changes, and changes in literacy skills, literacy practices & technology use over timeliteracy practices & technology use over timeSmaller-scale qualitative componentsSmaller-scale qualitative components
LSAL TimelineLSAL Timeline wavewave 1 1 1998 – 19991998 – 1999
wave 2wave 2 1999 – 20001999 – 2000
wave 3wave 3 2000 – 20012000 – 2001
wave 4wave 4 2002 – 20032002 – 2003
wave 5wave 5 2004 – 20052004 – 2005
wavewave 66 2006 – 20072006 – 2007
Literacy Measures in LSALLiteracy Measures in LSAL
Measures of ProficiencyMeasures of ProficiencyRepeated measures of TALS Document LiteracyRepeated measures of TALS Document LiteracySSN-matched GED test scoresSSN-matched GED test scores
Measures of PracticesMeasures of PracticesRepeated measures of literacy practicesRepeated measures of literacy practicesRepeated self-reported changes from wave-to-wave in Repeated self-reported changes from wave-to-wave in reading, writing and mathreading, writing and math
Measures of Component SkillsMeasures of Component SkillsOral vocabularyOral vocabulary Word recognition Word recognition Fluency Fluency Holistic writingHolistic writing
Some LSAL DemographicsSome LSAL Demographics
Average age is 28 (at Wave 1)Average age is 28 (at Wave 1)
50 % female and male50 % female and male
35 % minority35 % minority
9 % foreign-born9 % foreign-born
34 % live in poverty34 % live in poverty
29 % report a learning disability29 % report a learning disability
34 % took special education34 % took special education
Broad range of assessed basic skillsBroad range of assessed basic skills
Up Close and PersonalUp Close and Personal
Looking at Things Seen in IALS Looking at Things Seen in IALS in the LSAL Panel Studyin the LSAL Panel Study
• AgeingAgeing
• Proficiency & economic indicatorsProficiency & economic indicators
NALS Document Literacy by AgeNALS Document Literacy by Age(1992 national data corrected for education & disabilities)(1992 national data corrected for education & disabilities)
200
225
250
275
300
16-24 25-34 35-44 45-54 55-64 65+
Age
Pro
ficie
ncy
UK Document Literacy by AgeUK Document Literacy by Age(IALS data, corrected for education)(IALS data, corrected for education)
200
225
250
275
300
16-25 26-35 36-45 46-55 56-65
Age
Pro
ficie
ncy
Employment and Earnings DataEmployment and Earnings Data
Using individuals’ quarterly administrative Using individuals’ quarterly administrative employment & wage data reported by employment & wage data reported by employers for unemployment insuranceemployers for unemployment insurance
Covers more than 90% of labor activityCovers more than 90% of labor activity
Examining 1997-2003 (7 years / 28 qrtrs)Examining 1997-2003 (7 years / 28 qrtrs)
Major recession in Portland 2001-2003Major recession in Portland 2001-2003
Wages by Proficiency, 1997-2003Wages by Proficiency, 1997-2003
Earnings by Proficiency GrowthEarnings by Proficiency Growth
So What’s The Problem?So What’s The Problem?
Longitudinal data seem consistent with IALS snapshots Longitudinal data seem consistent with IALS snapshots and allow us to look more closely at the individual level of and allow us to look more closely at the individual level of the large-scale phenomenathe large-scale phenomena
Controlled comparisons of adults who participate in basic Controlled comparisons of adults who participate in basic skills programs with similar adults who do not participate skills programs with similar adults who do not participate show there is no significant impact of programs on show there is no significant impact of programs on proficiency one or two years following participationproficiency one or two years following participation
Does this mean programs are not effective? NO! But it Does this mean programs are not effective? NO! But it illustrates the practical disconnect of IALS from programsillustrates the practical disconnect of IALS from programs
0% 20% 40% 60% 80% 100%
read instructions
do financial math
use ATM
read street maps
read entertain,tv guides
read news section
read fiction
read non-fiction
read mags, comic bks
write diary,story,poem
write note, letter, email
read manuals, ref mats
do math
use comp at home
never rarely < weekly weekly > weekly daily
Repeated Measures of Engagement in Literacy Practices
Program ImpactProgram Impact
Literacy proficiency growth over relatively short periods of time is Literacy proficiency growth over relatively short periods of time is not not affected by program participationaffected by program participation
Accountability data show proficiency gains but do not contrast Accountability data show proficiency gains but do not contrast participants’ gains with those of comparable non-participants; LSAL participants’ gains with those of comparable non-participants; LSAL indicates their gains are equivalentindicates their gains are equivalent
Growth in literacy practices over short time periods Growth in literacy practices over short time periods is, is, on the other on the other handhand,, directly impacted by programs directly impacted by programs
These findings are consistent with cross-sectional research (e.g., These findings are consistent with cross-sectional research (e.g., Smith & Sheehan-Holt) and classroom studies (e.g., Purcell-Gates, Smith & Sheehan-Holt) and classroom studies (e.g., Purcell-Gates, Jacobson & Degener)Jacobson & Degener)
DisconnectDisconnect: Proficiency measures do not reflect the impact that : Proficiency measures do not reflect the impact that programs have in the short-term nor can support evidence-based programs have in the short-term nor can support evidence-based program improvement processesprogram improvement processes
Repairing the Disconnect:Repairing the Disconnect:Practice-Engagement TheoryPractice-Engagement Theory
Predicts that engagement in literacy practices over time leads Predicts that engagement in literacy practices over time leads to proficiency growthto proficiency growth
Proposed by Reder from ethnographic work and by Smith & Proposed by Reder from ethnographic work and by Smith & colleagues in analyses of NALScolleagues in analyses of NALS
LSAL confirms through longitudinal modeling:LSAL confirms through longitudinal modeling:
Adults with the same proficiency level and a higher level of Adults with the same proficiency level and a higher level of engagement in literacy practices at one point in time have engagement in literacy practices at one point in time have significantly higher levels of literacy proficiency 6 years latersignificantly higher levels of literacy proficiency 6 years later
What Should We Do?What Should We Do?
Stop relying on only IALS (or other) proficiency Stop relying on only IALS (or other) proficiency measures for measures for short-termshort-term accountability & accountability & program improvementprogram improvementUse literacy practices measures along with Use literacy practices measures along with proficiency measures in large-scale proficiency measures in large-scale assessments and for assessments and for short-termshort-term program program accountability and improvementaccountability and improvementUse proficiency measures as Use proficiency measures as long-termlong-term (5-6 (5-6 year) follow-up outcome measuresyear) follow-up outcome measures
Contact informationContact information
Steve RederSteve RederProfessor, Chair – Department of Applied LinguisticsProfessor, Chair – Department of Applied LinguisticsPortland State UniversityPortland State University(503) 725-3999(503) 725-3999reders@pdx.edureders@pdx.edu
For more information and references to the research covered in the Powerpoint For more information and references to the research covered in the Powerpoint presentation, please see:presentation, please see:
Longitudinal Study of Adult Learning Longitudinal Study of Adult Learning www.lsal.pdx.eduwww.lsal.pdx.edu
Some Thoughts on IALS Measurement Validity, Program Impact, and Logic Models for Some Thoughts on IALS Measurement Validity, Program Impact, and Logic Models for Policy Development Policy Development www.centreforliteracy.qc.ca/sites/default/files/Rederthinkpiece.pdfwww.centreforliteracy.qc.ca/sites/default/files/Rederthinkpiece.pdf
Recommended