A Review of 50-Years of Literature in Postsecondary Education and Disability

A Review of 50-Years of Literature in Postsecondary Education and Disability

Joseph W. Madaus, Adam R. Lalor, Jennifer S. Kowitt & Allison LombardiUniversity of Connecticut

Lyman Dukes IIIUniversity of South Florida St. Petersburg

Michael Faggella-LubyTexas Christian University

Presentation at the 27th Annual PTIPhiladelphia, PA

June, 2014

Session Objectives

• Explain the rationale for conducting a comprehensive literature review

• Explain the background and methods used• Present specific findings• Highlight any effective practices for students

with disabilities in higher education

Project Background

• The Vocational Rehabilitation Act of 1973 required that all institutions of higher education receiving Federal funding ensure access to qualified students with disabilities (SWD).

• Forty years after the passage of the Act, ~11% of college freshmen report having a disability (U.S. G.A.O., 2009).

Project Background

• To date, a comprehensive analysis of the disability and higher education literature has yet to be conducted

• This literature is broad in scope and dispersed across a variety of disciplines (e.g., special education, higher education, psychology, sociology)

• Given the 40-year anniversary of the passage of Section 504 in 2013 and the 25-year anniversary of the ADA in 2015, it is a suitable occasion to review the field’s literature:– What topics have been studied?– What methodologies have been employed?– What portion of the literature can be defined as data-based?– What practices have substantial evidence and support?– What topical areas within the field may receive greater attention in coming

years?– Why is this relevant to practitioners, researchers, policy makers?

Project Background

• Genesis was a request from NSTTAC to present information about evidence-based practices regarding:– Successful transition to postsecondary education– Success in postsecondary education

• Our plan:– Initially, to follow the NSTTAC meta-analysis procedures– But, postsecondary education lacks a taxonomy for the literature– Postsecondary education does not use the evidence based

practice standards required in secondary education– No prior sorting of the literature, either by topical or research

categories– Required a regrouping and new direction

Our Method• Each article of the Journal of Postsecondary Education and Disability from 2000-

2010 was reviewed and common themes and topics were identified.• Discussion of the JPED data revealed overlaps, determination of key terms

– e.g., “policies and procedures”, “experiences”– Where does eligibility “belong”?– Difference between institutional and program legal compliance?– What about studies of instruments and proposed constructs?

• Initial domains collapsed and updated:– Student level – Program level– Faculty/staff level– Construct level

• JPED articles from an additional 5 issues reviewed by four coders• Reliability determined at 75%-85%• Debriefing led to 100% agreement; refinement of terms, inclusion and exclusion

criteria

Our Method

• Sorted into domains; reliability measured– 88% - 96% for sorting

• Articles provided a broader perspective and led to further refinement of the subdomains

• Validity check by 8 former editors or co-editors of JPED– Measured the clarity of domain definitions

• all were strongly agree or agree that the definition is clear

– Requested suggestions for missing domains– Fit of the subdomains– Suggestions for missing subdomains and clarification of subdomains

• (e.g., legal compliance at the program or institutional level)

Inclusion and Exclusion Criteria• Inclusion criteria:

1. The article is about Postsecondary Education for Students with Disabilities (broadly considered to include faculty, disability services, etc.)2. The article is about one of the following topics/populations:a. Programs for accepted students into degree granting programs at a 2 or 4 year college or universityb. Programs, services, or experiences of matriculated studentsc. Articles about the experiences of students with disabilities who have dropped out of degree granting programs at a 2 or 4 year college or universityd. Articles about the experiences of students with disabilities who are graduates of degree granting programs at a 2 or 4 year college or university

• Exclusion criteria1. Articles that are primarily about secondary students in transition or transition aged programs.

Domain DescriptionsDomain Name Domain Description

Student-Level Studies

Experiences and perceptions of students with disabilities in and after higher education.

Program or Institution-Level

Studies

Service provision by the disability services office in a higher education institution. Can also relate to institutional policies and procedures pertaining to students with disabilities.

Faculty/Non-Disability Support Staff-Level Studies

Knowledge, attitudes, beliefs of faculty and non-disability services personnel to enhance access to higher education for students with disabilities. Also education or support for faculty and staff in this practice.

Construct Development-Level

Studies

Development, evaluation, or validation of a variable, including development/validation of assessment instruments, evaluation metrics, theoretical models of service delivery, standards of practice, or ethics. The variable must be under proposal, in development, or being used in practice to gather empirical evidence.

No Fit Studies that do not relate to any of the above domains.

Sub DomainsStudent Level Studies

• Access (physical, cognitive, attitudinal)• Assistive technology use• Career development• Experiences, perceptions, knowledge, attitudes, or beliefs of students with

disabilities• Learning/using study skills, learning strategies • Mainstream technology use• Meeting institutional requirements (e.g., degree requirements, foreign language

requirements, math requirements)• Post-undergraduate program experiences and/or outcomes (e.g., graduate school,

employment)• Profiles of students (e.g., diagnostic profiles, profiles of successful and/or

unsuccessful students)• Requesting or using accommodations (e.g., assistive technologies, separate testing

location, course substitutions)• Self-determination skills (e.g., self-advocacy, student goal attainment, self-

disclosure, self-management, legal rights and responsibilities)• Statistics on students with disabilities (e.g., rate of access to postsecondary

education, student retention, graduation rate, statistics on accommodation use)

Sub DomainsProgram/Institutional Level Studies

• Collaboration with faculty or academic departments• Collaboration with other campus services • Experiences, perceptions, knowledge, attitudes, or beliefs of disability service

providers• General or specific descriptions of disability programs and resources and/or

recommended program components• Institutional Policies/Procedures• Legal compliance (institutional specific)• Legal compliance (program specific)• Program development• Programs for incoming students (e.g., freshmen, transfer students)• Programs for students transitioning to graduate school or employment• Programs for specific cohorts of students (e.g., LD, Aspergers, etc)• Policies and procedures (e.g., determining student eligibility for services,

determining reasonable accommodations, determining access to assistive technology)

• Professional development/training for disability services staff • Program evaluation (e.g., student retention, student use of program related

services, student graduation rates)• Program fit within the institution (e.g., student affairs v. academic affairs)

Sub DomainsFaculty/Non-Disability Support Level Studies

• Campus staff development and training• Campus staff knowledge, attitudes and beliefs (e.g., about

students with disabilities)• Campus staff practices• Faculty development and training• Faculty knowledge, attitudes and beliefs (e.g., about students

with disabilities; about providing accommodations)• Faculty teaching practices

Sub DomainsConstruct Development Studies

• Assessment instruments (development, validation, use to develop diagnostic profiles)

• Conceptual models or discussion of issues in disability services (e.g., eligibility for services)

• Conceptual models of service delivery (e.g., Universal Design, other models)

• Conceptual models of instruction/assessment of learning• Evaluation metrics or methods• Instructional practices• Standards of practice, performance or ethics.• Other (including disability studies)

Method

• 1,346 articles identified by searches of multiple data bases (e.g., Academic Search Premier, EBSCO)

• Published between 1955 and 2012• Articles grouped into domains, reliability measured

– Coding resulted in some articles shifting domains

Instrument

• An electronic coding instrument was designed and refined with two pilots using multiple coders.

• The instrument allowed for the researchers to code:– Did the article meet inclusion criteria?– Did the article present original data?– If not data-based, what type? (e.g., lit review, legal analysis)– If data-based, what type? (with multiple layers)– What was the setting for the article? (US, Canada, international,

2- or 4-year)– Who was in the sample? (numbers, gender, disability, race, etc.)– Domain and sub-domain

• Across coding sheet, 148 choices were possible• To achieve agreement, coders selections must be exact

Inter-Rater Reliability

• For today’s presentation, four subsets were analyzed.

• Each article coded twice to check for inter-rater reliability.

• Discrepancies discussed and reconciled

• A third coder was used as needed to reconcile disagreements

Frequency and Reliability by DomainDomain n Reliability

Domain 1: Student-Level Studies 431 92%

Domain 2: Program or Institution-Level Studies

292 88%

Domain 3: Faculty/Non-Disability Support Staff-Level Studies

132 94%

Domain 4:Construct Development-Level Studies

138 91%

No Fit 14 100%

OVERALL RELIABILITY 91%

Frequency of Articles by Domain Over Time

1951-1955

1951-1955

1956-1960

1961-1965

1966-1970

1971-1975

1976-1980

1981-1985

1986-1990

1991-1995

1996-2000

2001-2005

2006-2010

2011-20120

50

100

150

200

250

300

Domain 1Domain 2Domain 3Domain 4Total

Journals with the Highest Frequency of Articles About Higher Education and Disability Across Domains

J of P

ostseco

ndary Educa

tion & D

isabilit

y

J of L

earning D

isabiliti

es

J of C

ollege St

udent Deve

lopment

College St

udent Journ

al

Disabilit

y & So

ciety

J of V

ocational R

ehabilitation

Learn

ing Disa

bilities R

&P

New Dire

ctions f

or Student S

ervice

s

J of C

ollege St

udent Psyc

hotherapy

Career D

ev & Tra

ns for E

xcept In

d

Dyslexia

0

50

100

150

200

250

300

# Articles

Unique Journals: 249

Journals with the Highest Frequency of Student-Level Articles

J of P

ostseco

ndary Ed & Disa

bility

J of L

earning Disa

bilities

J Colle

ge Student Development

Disabilit

y & Socie

ty

College Student Jo

urnal

Learn

ig Disabiliti

es R&P

Career Dev &

Trans for E

xcept In

d

J of V

isual Im

pairment a

nd Blindness

Learn

ing Disabilit

y Quarte

rly

Studies in Higher E

ducation

0

20

40

60

80

100

# Articles


Journals with the Highest Frequency of Program/Institutional-Level Articles

J of P

ostseco

ndary Ed & Disa

bility

J of L

earning Disa

bilities

J of C

ollege Student D

evelopment

J of V

ocational R

ehabilitation

New Directi

ons for S

tudent Service

s

J of D

istance

Ed

Learn

ing Disabiliti

es R &

P

New Directi

ons for H

igher Ed

College Student Jo

urnal

Internet &

Higher Ed

0

20

40

60

80

100

120

# Articles


Journals with the Highest Frequency of Faculty/Non-Disability Staff-Level Articles

J of P

ostseco

ndary Ed & Disa

bility

College Student Jo

urnal

J of C

ollege Counselin

g

New Directi

ons for S

tudent Service

s

College Teach

ing

J of C

ollege Student D

evelopment

J of C

ollege Student P

sychotherapy

J of L

earning Disa

bilities

05

1015202530354045

# Articles

Unique Journals: 66

Journals with the Highest Frequency of Construct Development-Level Articles

J of P

ostseco

ndary Ed & Disa

bility

J of L

earning Disa

bilities

Equity &

Excelle

nce in

Ed

Dyslexia

J of V

ocational R

ehabilitation

J of C

ollege Student D

evelopment

New Directi

ons for H

igher Ed

0

10

20

30

40

50

# Articles

Unique Journals: 59

Frequency of Data-Based vs. Non-Data-Based Student-Level Studies Over Time

1976-1980

1981-1985

1986-1990

1991-1995

1996-2000

2001-2005

2006-2010

2011-2012

0

20

40

60

80

100

120

Data-BasedNon-Data-Based

Proportion of Student-Level Studies by Research Methodology

56%33%

10%

Quantitative (n = 202)Qualitative (n = 120)Mixed Methods (n = 37)

Proportion of Student-Level Studies With and Without Control/Comparison Groups

29%

71%

With Control/Comparison Group (n = 15)Without Control/Compar-ison Group (n = 36)

Proportion of Data-Based Student-Level Studies Including Disability-Related Demographic Information

69%

18%

13%

Study Provided Frequen-cies of Disability Type (n = 201)Study Did Not Provide Frequencies of Disability Type (n = 52)Coder Disagreement (n = 39)

Twelve Subdomains of Student-Level Studies and Their Frequencies

(Articles could be coded as multiple subdomains)

• Experience, perception, knowledge, attitude of SWD (n = 272)

• Profiles of SWD (n = 123)

• Requesting/using accommodations (n = 57)

• Learning/using study skills or learning strategies (n = 56)

≥ 50

• Access (physical/cognitive/attitudinal) (n = 47)

• Statistics on SWD (n = 41)

• Self-determination (n = 38)

• Assistive technology use (n = 23)

• Career development (n = 21)

20 - 49

• Mainstream technology use (n = 15)

• Meeting institutional requirements (n = 11)

• Post-undergraduate experiences or outcomes (n = 9)

≤ 19

Frequency of Data-Based vs. Non-Data-Based Program/Institution-Level Studies Over Time

1951-1955

1956-1960

1961-1965

1966-1970

1971-1975

1976-1980

1981-1985

1986-1990

1991-1995

1996-2000

2001-2005

2006-2010

2011-20120

5

10

15

20

25

30

35

40


Proportion of Program/Institution-Level Studies by Research Methodology

65%1%6%

25%

3%

Descriptive Quantitative (n = 72)Group Design (interven-tion study; n = 1)Mixed Methods (n = 7)Qualitative (n = 27)Single Subject (n = 3)

Proportion of Program/Institution-Level Studies With and Without Control/Comparison Groups

100%

With Control/Comparison Group (n = 0)

Without Control/Compar-ison Group (n = 99)

Sixteen Subdomains of Program/Institution-Level Studies and Their Frequencies


• Descriptions/recommendations of disability programs/resources ( n = 97)

• Policy and procedure (eligibility, accommodations, etc.)(n = 68)

• Programs for specific cohorts of SWD (n = 59)

• Institutional policies/procedures (n = 58)

• Legal compliance (Institution specific) (n = 42)

≥ 22

• Experience, knowledge, attitudes, beliefs of DSPs (n = 21)

• Program development (n = 21)

• Collaboration with faculty or academic departments (n = 17)

• Other (n = 16)

• Programs for incoming students (n = 13)

• Collaboration with other campus services (n =13)

• Programs for students transitioning to grad school/employment (n = 12)

• PD and training (n = 11)

11 – 21

• Legal compliance (program specific)(n = 10)

• Program evaluation (n = 10)

• Program fit within institution (n = 2)

≤ 10

Frequency of Data-Based vs. Non-Data-Based Faculty/Non-Disability Support Staff-Level

Studies Over Time

1981-1985

1986-1990

1991-1995

1996-2000

2001-2005

2006-2010

2011-2012

0

5

10

15

20

25


Proportion of Faculty/Non-Disability Support Staff-Level Studies by Research Methodology

61%23%

16%

Quantitative (n = 46)Qualitative (n = 17)Mixed methods (n = 12)

Proportion of Faculty/Non-Disability Staff-Level Studies With and Without Control/Comparison Groups

4%

96%

With Control/Comparison Group (n = 3)


Six Subdomains of Faculty/Non-Disability Staff-Level Studies and Their Frequencies


• Faculty knowledge, attitudes and beliefs (n = 59)

• Faculty teaching practices (n = 36)

≥ 30

• Faculty development and training (n = 24)

• Campus staff practices (n = 20)

20 – 29

• Campus staff knowledge, attitudes, and beliefs (n = 19)

• Campus staff development and training (n = 9)

≤ 19

Frequency of Data-Based vs. Non-Data-Based Construct Development-Level Studies Over Time

1951-1955

1956-1960

1961-1965

1966-1970

1971-1975

1976-1980

1981-1985

1986-1990

1991-1995

1996-2000

2001-2005

2006-2010

2011-20120

5

10

15

20

25

30


Proportion of Construct Development-Level Studies by Research Methodology

66%3%

9%

22%

Descriptive Quantitative (n = 38)Group Design (interven-tion study; n = 2)Mixed Methods (n = 5)Qualitative (n = 13)Single Subject (n = 0)

Proportion of Construct Development-Level Studies With and Without Control/Comparison Groups

67%

33% With Control/Comparison Group (n = 2)


NOTE: There were two design studies (one with control group) and a third descriptive study with a group for comparison.

Seven Subdomains of Construct Development-Level Studies and Their Frequencies(Articles could be coded as multiple subdomains)

• Conceptual models of service delivery (e.g., Universal Design, other models)(n = 52)

• Assessment instruments (development, validation, use to develop diagnostic profiles)(n = 42)

≥ 22

• Conceptual models of instruction/assessment of learning (n = 20)

• Conceptual models or discussion of issues in disability services (e.g., eligibility for services) (n = 14)

• Standards of practice, performance or ethics (n = 13)

11 – 21

• Evaluation metrics or methods(n = 9)

• Other (including disability studies) (n = 3)

≤ 10

No Fit Article Descriptions

• 14 articles were coded as meeting the criteria to be included in the study, but did not fit a domain.

• Examples of article topics:– Disability and higher education testing agencies– Interviews with researchers studying disability and

higher education– Disability training and programming for students

without disabilities

Discussion – Is the glass half empty or half full?

• Articles on higher education and disability have been published in 249 unique journals

• These journals have a range of purposes, styles, level of rigor, etc.

• The overall number of published articles in the field has increased considerably from the late 1970s.

• A limited number of studies have clear control/comparison groups (n = 20)


Domain 1• More data-based studies exist than non data-based • The number of data-based studies has been

increasing • 56% of studies use quantitative methodology, 34%

qualitative, 10% mixed methods• Only 4% data-based articles have a

comparison/control group• 69% of data-based studies provided demographic

data


Domain 1 (cont)• Most popular subdomains:

– Experience, perception, knowledge, attitude of SWD (n = 272)

– Profiles of SWD (n = 123)

• Least popular subdomains:– Mainstream technology use (n = 15)– Meeting institutional requirements (n = 11)– Post-undergraduate experiences or outcomes (n = 9)


Domain 2• Data-based articles constitute 37.2% of total

– First 20 years: <1% of the data-based articles– Last 12 years: 57.3% of the data-based articles

• Comparison group design n=0• Research Method

– 65.5% Descriptive Quantitative– 24.5% Descriptive Qualitative– 10% group design, mixed methods or ss (n=11)


Domain 2 (cont)• Most popular subdomains

– Descriptions/recommendations of disability programs/resources ( n = 97)

– Policy and procedure (eligibility, accommodations, etc.)(n = 68)– Programs for specific cohorts of SWD (n = 59)– Institutional policies/procedures (n = 58)– Legal compliance (Institution specific) (n = 42)

• Least popular subdomains– Legal compliance (program specific)(n = 10)– Program evaluation (n = 10)– Program fit within institution (n = 2)


Domain 3• The domain with the fewest articles (n = 132)• Data-based articles constitute 37.2% of total

– Prior to 2001: 29.7% of the data-based articles– Since 2001: 70.3% of the data-based articles

• Comparison group design 4% (3/71 articles)• Research Method

– 61% Quantitative (n = 46)– 23% Qualitative(n = 17)– 16% Mixed methods (n=12)



– Faculty knowledge, attitudes and beliefs (n = 59)– Faculty teaching practices (n = 36)

• Least popular subdomains– Campus staff practices (n = 20)– Campus staff development and training (n = 9)


Domain 4• Data-based articles constitute 42% of total

– First 35 years: 1.7% of the data-based articles– Last 12 years: 86.2% of the data-based articles

• Comparison group design n=2• Research Method

– 65.5% Descriptive Quantitative– 22.4% Descriptive Qualitative– 12.1% group design, mixed methods or ss (n=11)



– Conceptual models of service delivery (e.g., Universal Design, other models)(n = 52)

– Assessment instruments (development, validation, use to develop diagnostic profiles)(n = 42)

• Least popular subdomains– Evaluation metrics or methods(n = 9)– Other (including disability studies) (n = 3)

Discussion – Limitations

• Not possible to gather every published article– Search terms as broad as possible (28 keywords)– Use of a range of data-bases

• Domains and codes for data-collection determined by the research team– Iterative process– Examined multiple journals– Feedback from outside experts

• Coding errors– Each article double coded– Reconciliations

Discussion – Effective Practices: What we know so far

• 3 studies on self-advocacy had effective experimental effects

• Interventions with some empirical support:- Strategy instruction- Strategic content learning instruction- Guided Notes Instruction

• A deep dive is currently underway to identify further effective practices• Caveat: There is not a mandate in higher education that requires the

use of evidence-based practices

ReferencesCameto, R., Mazzotti, V. L., & Test, D.W. (2011, April). High-quality research in secondary transition: Current status and future need. DCDT Showcase presented at the Council for Exceptional Children Annual Convention, Nashville, TN.

Helsel, F. K. I, Hitchcock, J. H., Miller, G., Malinow, A., & Murray, E. (2006). Identifying evidence-based, promising and emerging practices that use screen-based technology to teach mathematics in grades K-8: A research synthesis. Presented at AERA 2006 Meeting, San Francisco, CA.

Rehabilitation Act of 1973, 29 U.S.C. § 701 et seq.

Test, D. W., Fowler, C., Kohler, P., & Kortering, L. (2010, August). Evidence-based practices and predictors in secondary transition: What we know and what we need to know, Executive Summary.(Revised). Charlotte, NC: National Secondary Transition Technical Assistance Center. Available athttp://www.nsttac.org/sites/default/files/assets/pdf/pdf/ebps/ExecsummaryPPs.pdf

Test, D. W., Fowler, C. H., Richter, S. M., White, J., Mazzotti, V., Walker, A. R., & Kortering, L. (2009). Evidence-based practices in secondary transition. Career Development for Exceptional Individuals, 32(2), 115-128.

U.S. Department of Education, National Center for Education Statistics. (2012). Digest of Education Statistics, 2011 (2012-001)

http://www.nsttac.org/sites/default/files/assets/pdf/pdf/ebps/ExecsummaryPPs.pdf

Q & AThank you!!

Documents

A Review of 50-Years of Literature in Postsecondary Education and Disability