Longitudinal Approaches to Longitudinal Approaches to Research and EvaluationResearch and Evaluation

in Adult Education in Adult Education

Professor Stephen RederProfessor Stephen Reder

Portland State UniversityPortland State University

Charles Darwin UniversityCharles Darwin UniversityOctober 9, 2014October 9, 2014

Outline of SessionOutline of Session

Flexible & negotiable depending on your Flexible & negotiable depending on your needs and interestsneeds and interests

Longitudinal Research MethodsLongitudinal Research Methods Applications to Research and Program Applications to Research and Program

Evaluation in Adult EducationEvaluation in Adult Education Example applications in my own longitudinal Example applications in my own longitudinal

study of adult learningstudy of adult learning

Longitudinal Methods TextsLongitudinal Methods Texts Singer, J. D. & Willett, J. B. (2003). Singer, J. D. & Willett, J. B. (2003). Applied Applied

longitudinal data analysis: Modeling change longitudinal data analysis: Modeling change and event occurrenceand event occurrence. Oxford & New York: . Oxford & New York: Oxford University Press.Oxford University Press.

Magnusson, D., Bergman, L. R., Rudinger, G., Magnusson, D., Bergman, L. R., Rudinger, G., et al. (Eds.) (1994). et al. (Eds.) (1994). Problems and methods in Problems and methods in longitudinal research: Stability and changelongitudinal research: Stability and change . . New York: Cambridge University Press.New York: Cambridge University Press.

Saldaña, J. (2003). Saldaña, J. (2003). Longitudinal qualitative Longitudinal qualitative research: Analyzing change through timeresearch: Analyzing change through time . . Lanham, MD: AltaMira Press.Lanham, MD: AltaMira Press.

Longitudinal Adult LLN Longitudinal Adult LLN Research TextResearch Text

Reder, S. & Bynner, J. (Eds.) (2009). Reder, S. & Bynner, J. (Eds.) (2009). Tracking adult literacy and numeracy skills: Tracking adult literacy and numeracy skills: Findings from longitudinal researchFindings from longitudinal research . New . New York & London: Routledge.York & London: Routledge.

Many Fundamental Research Questions in Many Fundamental Research Questions in Adult Education Involve Change Over TimeAdult Education Involve Change Over Time

How adults learnHow adults learn How participation in programs influences How participation in programs influences

adults’ acquisition of knowledge and skillsadults’ acquisition of knowledge and skills How adults’ educational development interacts How adults’ educational development interacts

with their social and economic performancewith their social and economic performance How adults’ private and public situation in the How adults’ private and public situation in the

family and community may enhance or family and community may enhance or obstruct their progress in learningobstruct their progress in learning

Methodological Concepts and Issues Methodological Concepts and Issues in Longitudinal Researchin Longitudinal Research

Sample attrition and missing dataSample attrition and missing data Longitudinal stability of measures and scalesLongitudinal stability of measures and scales Methods for analyzing change in longitudinal Methods for analyzing change in longitudinal

datadata

Sample Attrition and Missing DataSample Attrition and Missing Data

Missing item and unit dataMissing item and unit data Ignorable and Non-Ignorable MissingIgnorable and Non-Ignorable Missing Missing Completely at Random (MCAR)Missing Completely at Random (MCAR) Missing at Random (MAR)Missing at Random (MAR)

Constructing Measures and Scales Constructing Measures and Scales That Are Longitudinally StableThat Are Longitudinally Stable

Essential to know that the same construct is being Essential to know that the same construct is being measured at different time pointsmeasured at different time points

Scales having good psychometric properties for Scales having good psychometric properties for measurement on a single occasion do measurement on a single occasion do notnot necessarily have good psychometric properties necessarily have good psychometric properties for measuring change across multiple occasionsfor measuring change across multiple occasions

Cannot determine longitudinal stability in Cannot determine longitudinal stability in advance, must use longitudinal data to examine advance, must use longitudinal data to examine measurement stabilitymeasurement stability

Some Methods For Analyzing Some Methods For Analyzing Change in Longitudinal DataChange in Longitudinal Data

Growth Curve ModelingGrowth Curve Modeling Event Occurrence ModelingEvent Occurrence Modeling Structural Equation ModelingStructural Equation Modeling Treatment Effects ModelsTreatment Effects Models Dynamic Panel ModelsDynamic Panel Models

The Longitudinal Study of Adult Learning The Longitudinal Study of Adult Learning (LSAL)(LSAL)

funded by U.S. Department of Educationfunded by U.S. Department of Education

funded by

U.S. Department of Education

National Institute for Literacy

Portland State University

LSAL PerspectiveLSAL Perspective

Look at how programs fit into the lifelong and Look at how programs fit into the lifelong and life-wide landscape of adults’ learning, rather life-wide landscape of adults’ learning, rather than at how adults fit into LLN programs as than at how adults fit into LLN programs as studentsstudents

We’ll see that things look considerably We’ll see that things look considerably different from this vantage pointdifferent from this vantage point

LSAL DesignLSAL Design

Decade-long Decade-long panel studypanel study of Portland (Oregon) early school leavers, of Portland (Oregon) early school leavers, age 18-44 at the beginning of the studyage 18-44 at the beginning of the study

Representative sample of ~1,000 drawn from local rather than Representative sample of ~1,000 drawn from local rather than national population of dropoutsnational population of dropouts

Includes both program participants and nonparticipantsIncludes both program participants and nonparticipants

Examines program participation and other learning activities, social Examines program participation and other learning activities, social and economic changes, and changes in literacy skills, literacy and economic changes, and changes in literacy skills, literacy practices & technology use over timepractices & technology use over time

Periodic in-home interviews and literacy assessments and SSN-Periodic in-home interviews and literacy assessments and SSN-linked administrative data (with individuals’ permission)linked administrative data (with individuals’ permission)

Smaller-scale qualitative componentsSmaller-scale qualitative components

LSAL Realized SampleLSAL Realized Sample

N = 940N = 940 496 from RDD Frame496 from RDD Frame 444 from Student Frame444 from Student Frame

High level of diversity in sampleHigh level of diversity in sample 90% sample retention over 8 years90% sample retention over 8 years 39 additional pilots for instrument development, 39 additional pilots for instrument development,

training & qualitative studiestraining & qualitative studies

Some LSAL DemographicsSome LSAL Demographics

Average age is 28 (at Wave 1)Average age is 28 (at Wave 1) 50 % female and male50 % female and male 35 % minority35 % minority 9 % foreign-born9 % foreign-born 34 % live in poverty34 % live in poverty 29 % report a learning disability29 % report a learning disability 34 % took special education34 % took special education Broad range of assessed basic skillsBroad range of assessed basic skills

LSAL TimelineLSAL Timeline

wave 1wave 1 1998 – 19991998 – 1999

wave 2wave 2 1999 – 20001999 – 2000

wave 3wave 3 2000 – 20012000 – 2001

wave 4wave 4 2002 – 20032002 – 2003

wave 5wave 5 2004 – 20052004 – 2005

wave 6wave 6 2006 – 20072006 – 2007

0% 20% 40% 60% 80% 100%

read instructions

do financial math

use ATM

read street maps

read entertain,tv guides

read news section

read fiction

read non-fiction

read mags, comic bks

write diary,story,poem

write note, letter, email

read manuals, ref mats

do math

use comp at home

never rarely < weekly weekly > weekly daily

Repeated Measures of Engagement in Literacy PracticesRepeated Measures of Engagement in Literacy Practices

Scaling of Literacy PracticesScaling of Literacy Practices•Started with 14 parallel questions, such as:Started with 14 parallel questions, such as: How often do you read the news section of the newspaper?How often do you read the news section of the newspaper? never / rarely / less than once a week / once a weeknever / rarely / less than once a week / once a week a few times a week / every daya few times a week / every day

• Confirmatory factor analyses of wave 1 data were used to Confirmatory factor analyses of wave 1 data were used to examine internal reliability & factor structuresexamine internal reliability & factor structures

• Two factors emerged, one assessing frequency of Two factors emerged, one assessing frequency of engagement in engagement in literacy practicesliteracy practices and one in and one in numeracy numeracy practicespractices

• Structural equation models indicate these factors have Structural equation models indicate these factors have stationary measurement properties over timestationary measurement properties over time

Literacy Growth CurvesLiteracy Growth Curves With multiple time points and repeated measures, we can With multiple time points and repeated measures, we can

develop and test models of develop and test models of individualindividual change in literacy change in literacy We can examine how individual characteristics (e.g., years We can examine how individual characteristics (e.g., years

of schooling, nativity), key life events (e.g., birth of a of schooling, nativity), key life events (e.g., birth of a child) and time-varying experiences (e.g., program child) and time-varying experiences (e.g., program participation) affect the growth processparticipation) affect the growth process

Linear growth curve models closely fit LSAL data on Linear growth curve models closely fit LSAL data on literacy proficiency and literacy practicesliteracy proficiency and literacy practices

Individuals’ literacy proficiency and practices Individuals’ literacy proficiency and practices dodo change change substantially after leaving schoolsubstantially after leaving school

HeterogeneityHeterogeneity of individual change is crucial of individual change is crucial

Person-specific intercept (“starting level”)

Person-specific slope (“rate of change”)

Change in Proficiency (Wave 1-4) by Change in Proficiency (Wave 1-4) by Age at Wave 1Age at Wave 1

-4

-2

0

2

4

6

8

10

Change in Document Literacy

18-24 25-31 32-38 39-44

Wave 1 Age

NALS Document Literacy by AgeNALS Document Literacy by Age(1992 national data corrected for education & disabilities)(1992 national data corrected for education & disabilities)

200

225

250

275

300

16-24 25-34 35-44 45-54 55-64 65+

Age

Pro

ficie

ncy

Similar results in ALLS & PIAAC Similar results in ALLS & PIAAC data from Australiadata from Australia

Key FindingsKey Findings

• Measures of literacy proficiency and literacy practices show systematic change over time across the adult lifespan

• Growth curve models of these literacy measures show impressive heterogeneity of change: some adults show literacy growth, others little change, and others literacy loss over time

• Age, birthplace, parental education, intergenerational Age, birthplace, parental education, intergenerational reading practices, K-12 schooling experiences, and reading practices, K-12 schooling experiences, and health systematically influence adult literacy health systematically influence adult literacy developmentdevelopment

Key Findings (con’t)Key Findings (con’t)

• Key Key life history eventslife history events – such as changes in family – such as changes in family composition and employment changes -- influence adult composition and employment changes -- influence adult literacy developmentliteracy development

• The dynamics of change are quite different for literacy The dynamics of change are quite different for literacy proficiency and literacy practices: e.g., program proficiency and literacy practices: e.g., program participation directly affects literacy practices measures participation directly affects literacy practices measures but not literacy proficiencybut not literacy proficiency

Outcomes & Program ImpactOutcomes & Program Impact

Literacy proficiency growth over relatively short periods of time Literacy proficiency growth over relatively short periods of time is is not not affected by program participationaffected by program participation

Pre-post test accountability data, that apparently show Pre-post test accountability data, that apparently show systematic gains in participants’ proficiency, do not contrast systematic gains in participants’ proficiency, do not contrast participants’ gains with those of comparable non-participants; participants’ gains with those of comparable non-participants; LSAL indicates those gains are equivalentLSAL indicates those gains are equivalent

Literacy practices growth over relatively short periods of time Literacy practices growth over relatively short periods of time is, is, on the other handon the other hand,, directly affected by program participation directly affected by program participation

These findings are reinforced by cross-sectional research (e.g., These findings are reinforced by cross-sectional research (e.g., Smith & Sheehan-Holt) and by classroom studies (e.g., Purcell-Smith & Sheehan-Holt) and by classroom studies (e.g., Purcell-Gates, Jacobson & Degener)Gates, Jacobson & Degener)

Proficiency measures thus do not reflect the impact that Proficiency measures thus do not reflect the impact that programs have or support evidence-based program and policy programs have or support evidence-based program and policy improvement processesimprovement processes

Identifying Causal RelationshipsIdentifying Causal Relationships

Temporal orderingTemporal ordering Including all relevant variablesIncluding all relevant variables Analytical methods for experimental designsAnalytical methods for experimental designs Analytical methods for non-experimental designsAnalytical methods for non-experimental designs

Dealing with selection biasDealing with selection bias Standard regression methodsStandard regression methods Propensity score matching methodsPropensity score matching methods Instrumental variables methodsInstrumental variables methods

Dealing with omitted variables biasDealing with omitted variables bias Fixed effects panel regressionFixed effects panel regression

Estimating Participation ImpactEstimating Participation Impact Adults decide whether to participate in basic skills Adults decide whether to participate in basic skills

programs, so participants and nonparticipants are not programs, so participants and nonparticipants are not usually comparable (usually comparable (selection biasselection bias))

Several analytical methods can be used to address Several analytical methods can be used to address selection bias in comparing program participants & selection bias in comparing program participants & nonparticipants:nonparticipants: Treatment effects (propensity score matching)Treatment effects (propensity score matching) Difference-in-differences (propensity score matching)Difference-in-differences (propensity score matching) Fixed effects panel regressionsFixed effects panel regressions

Propensity Score MatchingPropensity Score Matching

Compares participants and nonparticipants Compares participants and nonparticipants matched on their likelihood of participating matched on their likelihood of participating based on based on observedobserved pre-participationpre-participation characteristics:characteristics: AgeAge Gender Race/Ethnicity EducationGender Race/Ethnicity Education Immigration status IncomeImmigration status Income Learning disabilities Parents’ educationLearning disabilities Parents’ education

Income Growth in Matched Participants (100+ hours) & Nonparticipants

Treatment Effects ModelsTreatment Effects Models Estimates Estimates average treatment effect on treated average treatment effect on treated by comparing 2007 incomes by comparing 2007 incomes

of matched participants and nonparticipantsof matched participants and nonparticipants

With participation defined as With participation defined as any any amount of attendance, there is no amount of attendance, there is no significant difference between groupssignificant difference between groups

With participation defined as With participation defined as 25 or more hours 25 or more hours of attendance, there is no of attendance, there is no significant difference between the groups’ 2007 incomessignificant difference between the groups’ 2007 incomes

With participation defined as With participation defined as 75 or more hours 75 or more hours of attendance, there is a of attendance, there is a nearly significant (p=0.053) difference between the groups’ 2007 incomesnearly significant (p=0.053) difference between the groups’ 2007 incomes

With participation defined as With participation defined as 100 or more hours 100 or more hours of attendance, there a of attendance, there a statistically significant difference: participants average $9,621 statistically significant difference: participants average $9,621 moremore in in annual income over what they would have received if they had not annual income over what they would have received if they had not participated (in 2013 USD) participated (in 2013 USD)

Treatment Effects of Participation on 2007 Earnings

Model

Minimum Hours of

Attendance to Be

Considered “Treated”

Number of

Propensity-Score-

Matched “Treated”

Number of Propensity

-Score-Matched

“Controls”

Average Treatment Effect on

the “Treated” (ATET)

t pEffect Size

A 1 396 105 1.111 1.411 0.159 n.s.

B 25 216 82 0.789 0.953 0.342 n.s.

C 75 216 80 2.014 1.942 0.053 n.s.

D 100 197 68 2.142 2.469 0.014 0.45

E 150 154 62 2.272 2.085 0.039 0.48

Difference-in-Differences (DID) ModelDifference-in-Differences (DID) Model Compares income changes over a decade (1997-2007) Compares income changes over a decade (1997-2007)

between matched participants and nonparticipants between matched participants and nonparticipants

There is no statistically significant DID between groups if There is no statistically significant DID between groups if participation is defined as participation is defined as anyany amount of attendance amount of attendance

If participation is defined as If participation is defined as 100 or more hours100 or more hours of attendance, of attendance, there is a statistically significant DIDthere is a statistically significant DID

Despite different statistical assumptions, estimates 2007 Despite different statistical assumptions, estimates 2007 incomes to average $10,179 more because of participation, incomes to average $10,179 more because of participation, comparable to the treatment effects estimate of $9,621 (in comparable to the treatment effects estimate of $9,621 (in 2013 USD)2013 USD)

Difference-in-DifferencesDifference-in-Differences

0

2000

4000

6000

8000

10000

12000

14000

16000

Before (1997) After (2007)

An

nu

al I

nco

me

(199

7 $)

Nonparticipants Participants

DIFFERENCE-IN-DIFFERENCES ESTIMATION100+ hours

--------------------- ------------ BASELINE --------- ----------- FOLLOW-UP ---------- --------- Outcome Variable | Control | Treated | Diff(BL) | Control | Treated | Diff(FU) | DID---------------------+---------+-----------+----------+---------+-----------+----------+---------Log Income | 7.651 | 6.126 | -1.525 | 5.386 | 6.935 | 1.549 | 3.074Std. Error | 0.513 | 0.468 | 0.695 | 0.617 | 0.513 | 0.802 | 1.061t | 14.91 | 4.39 | -2.19 | 3.98 | 9.85 | 2.31 | 2.90P>|t| | 0.000 | 0.000 | 0.029** | 0.000 | 0.000 | 0.054* | 0.004***-------------------------------------------------------------------------------------------------* Means and Standard Errors are estimated by linear regression.**Inference: *** p<0.01; ** p<0.05; * p<0.1

Fixed Effects Panel Regression ModelFixed Effects Panel Regression Model

Within-subject models of year-to-year variations in Within-subject models of year-to-year variations in income in relation to year-to-year program income in relation to year-to-year program participation and other life eventsparticipation and other life events

Eliminates selection bias due to observed Eliminates selection bias due to observed andand unobserved time-invariant individual characteristicsunobserved time-invariant individual characteristics

Reveals how temporal details of participation -- Reveals how temporal details of participation -- intensity, duration and elapsed time – are reflected in intensity, duration and elapsed time – are reflected in observed changes in economic statusobserved changes in economic status

Fixed Effects Panel Regression (con’t)Fixed Effects Panel Regression (con’t) Results consistent with other models Results consistent with other models

Only when participation involves about 100 or more Only when participation involves about 100 or more hours of attendance does it have a significant & hours of attendance does it have a significant & substantial impact on future earningssubstantial impact on future earnings

Concentrated hours have a larger impact on earnings Concentrated hours have a larger impact on earnings than hours distributed over yearsthan hours distributed over years

The impact of participation on earnings takes several The impact of participation on earnings takes several years to develop after program exityears to develop after program exit

Fixed Effects Panel Regressions of Log Annual Earnings

ModelParticipation

VariablesCoeff. in

Panel Regr.Robust Std.

Err.t

A CUMHOURS .0032 .0014 2.39*

B YEARS .1221 .1756 0.70

C YRSCUM100 .4005 .1510 2.65**

DYRSCUM100CUMHOURS

.3106

.0020.1411.0013

2.20*1.55

Pulse, Step, Growth:Pulse, Step, Growth:The Shape of Program ImpactThe Shape of Program Impact

<=

<=

PULSEPULSE

• High burst, short-lived impactHigh burst, short-lived impact

• Example in LSAL data: effects of receiving Example in LSAL data: effects of receiving GED credential shows a short-lived “brushing GED credential shows a short-lived “brushing up” of proficiencyup” of proficiency

P

rofic

ienc

y

Time

<=

<=

STEPSTEP

• Abrupt, qualitative & lasting impactAbrupt, qualitative & lasting impact

• Some changes in literacy practices seem to Some changes in literacy practices seem to have this temporal shapehave this temporal shape

P

rofic

ienc

y

Time

<=

<=

GROWTHGROWTH

• Slow, steady & progressively accumulating impactSlow, steady & progressively accumulating impact

• This is the shape of program impact on proficiencyThis is the shape of program impact on proficiency

• This is the shape of program impact on earningsThis is the shape of program impact on earnings

• Life history events -- such as the birth of children, taking on or losing a Life history events -- such as the birth of children, taking on or losing a partner, or a significant change in employment -- have similarly shaped partner, or a significant change in employment -- have similarly shaped impacts on the course of literacy developmentimpacts on the course of literacy development

P

rofic

ienc

y

Time

Summary: Impact of Participation on EarningsSummary: Impact of Participation on Earnings

Multiple methods of controlling for selection bias all indicate that Multiple methods of controlling for selection bias all indicate that participation in LLN programs has a significant positive impact on participation in LLN programs has a significant positive impact on adults’ future earningsadults’ future earnings

The significance of the impact requires a minimum amount of The significance of the impact requires a minimum amount of program attendance, about 100 hours in the LSAL dataprogram attendance, about 100 hours in the LSAL data

The earnings premium grows over time and becomes substantial 5-6 The earnings premium grows over time and becomes substantial 5-6 years after program exit: the annual premium was nearly half (0.45) a years after program exit: the annual premium was nearly half (0.45) a standard deviation of 2007 incomesstandard deviation of 2007 incomes

The impact of participation is not at all evident in short-term follow-The impact of participation is not at all evident in short-term follow-ups to program participationups to program participation

Post-program learning, proficiency growth, and postsecondary Post-program learning, proficiency growth, and postsecondary education and training may all play a role mediating the continuing education and training may all play a role mediating the continuing impact of participation on labor market outcomesimpact of participation on labor market outcomes

Fixed Effects Panel Regressions of Log Annual Income

ModelParticipation

Variables

Coeff. in Panel Regr.

Robust Std. Err.

t

ECUMHOURSHAVEGED

.0030

.4793.0013.4648

2.25** 1.03

FCUMHOURSYEARSGED

.0028

.4160.0013.1394

2.17* 2.98**

GYRSCUM100HAVEGED

.3848

.5360.1478.4542

2.60**1.18

HYRSCUM100YEARSGED

.3025

.3821.1437

.1364` 2.10*

2.80**