INDIANA IUPUI ASSESSMENT CONFERENCE

N A N C Y L . R U T H E R

J U D I T H D . H A C K M A N

C Y N T H I A L A N G I N

YALE UNIVERSITY

O C T O B E R 2 6 , 2 0 1 5

Assessing the International: Linking Resources to

Student Outcomes at Yale

NA NCY L. RUTHER – YA LE FELLOW/ RETIRED A SSOCIA TE DIRECTOR, YA LE MA CMILLA N CENTER /

PRINCIPA L, G A ZELLE INTERNATIONALnancy .rut h er@yale. edu

JUDITH HA CKMA N – RETIRED YA LE COLLEG E A SSOCIA TE DEA N FOR A SSESSMENT & PA ST

DIRECTOR, OFFICE OF INSTITUTIONAL RESEA RCH / SENIOR PA RTNER, IDEA S FOR A CTION

Presentation Participants

Key Lessons We Want to Share

We designed a comprehensive assessment of international outcomes for Yale students: graduate, professional, and undergraduate.

The original design was tied to DOE policy goals and its Title VI Programs and also to Yale’s emphasis on international education.

We believe this design can be helpful to other schools and students –We want to hear from you at the end of this session!

We were challenged by a variety of organizational/political forces –both at Yale and nationally – Have you had similar challenges? And how have you (and we) worked with these challenges?

At the end of this presentation, we ask you to complete a survey about your international assessment experience – see last page of handouts.

Highlights of Our Results

International specializing students clearly stand-out from their peers In their plans for the future and actual lives, they are: More likely to plan to and actually use their foreign languages in work More likely to pursue careers in government or education

In their utilization of and demand for IAFL resources during studies: Start with more and higher levels of FL proficiency, gain more and lose less In their pursuit of critical and least commonly taught languages Greater demand for international travel resources to support their studies

Yale’s IAFL curricular resources were incredibly stable over 12 years Variation by region and in high-content courses as “yellow flag”

* IAFL = International, Area & Foreign Language Study

Challenges to International Assessment

Interdisciplinary is a challenge. International even more so. Traditional organization by discipline and profession vs. interdisciplinary training Span of fields… Internationalization means resources in ALL fields Geographic coverage… all world regions and all languages (or some subset….)

Questions to assess international, interdisciplinary, university-wide? What student outcomes to measure? How to link them to resources? International course strength? Non-specialist Fields? Study Abroad? Extracurricular resources available? Travel Support? Nationality Houses? Clubs? Student use of IAFL resources? Not just the obvious specialist degrees or majors?

Federal Policy as a Frame for Assessment

Higher Education Act Title VI and the Fulbright-Hays* programs administered by US Department of Education

Develop and sustain training [and assessment] capacity for IAFL** across the higher education system.

Produce expertise, both in faculty and graduates. Graduates with advanced knowledge of world areas and field or issue expertise Graduates with advanced ability in less and least commonly taught language Careers using those skills in government and education at top priority

Contribute to developing globally capable citizens

* Fulbright-Hays is the institutional parallel to State Department Fulbright Program* * IAFL = International, Area & Foreign Language Study

Title VI Grants Fund the Assessment -- 2011-14

• Grant priority => outcomes linked to resources

• Common plan across 5 regional studies units• Africa, East Asia, Europe, Latin America, MidEast

• Total funding $9.1 M

• Long-term focus of assessment aimed at aligning Title VI priorities and Yale’s international direction

INTERNATIONAL, AREA AND FOREIGN LANGUAGE[IAFL] ASSESSMENT

Concept and Design

Pathways to International Expertise

Alumni: IAFL Specialists, Global

Citizens

Graduates: PhD, Grad-

Professional, Undergrad

IAFL RESOURCES:

Faculty, Courses, Travel

• IAFL use in career and life• Employment sector (government, university, education)• Residence country/significant travel region

• Plans for future (IAFL use; career sector)• Travel (research, study, work-related)• Student focus (specialist; generalist, none)

• Degree focus (specialist, other grad/prof’l)• Travel (research, study, internship)• Enrollments (language, area courses)

• Baseline and trends (intensity, breadth)• Course and degrees, extracurricular resources• Faculty and advising staff

Pathways to International Expertise

The Linkage Question: How well do Yale’s IAFL resources support students’ development of IAFL skills and expertise?

The Engagement Question: How do students engage Yale’s IAFL resources?

The Outcome Question: What proportion of Yale students graduate with international, area, and/or foreign language (IAFL) skills and expertise?

The Utilization/Impact Question: How do students and graduates plan to and actually use their IAFL skills over time?

Planned Mix of Three Approaches

LONGITUDINAL – to see relationships over time Courses, enrollment, outcomes at graduation, impact 5 & 10 years out

CROSS-SECTION – to compare across Participating schools and degrees World regions

Title VI Funded: Africa, East Asia, Europe, Latin America, Middle East Non-Title VI: South Asia, Southeast Asia, International/Global Affairs

BENCHMARK WITH PEERS By institution, e.g. COFHE * By school associations, e.g. Graduate, Law, International Affairs Schools

*COFHE – Consortium on Funding Higher Education – 35 Private Universities and Colleges

Data Sources

IAFL Resources (University Data, Yearly)Core: (Faculty, Degrees, Courses)

Extracurricular (Field Travel, Non-credit Language Training)

Alumni (Und, Grad, Prof) Database, Surveys of IAFL Usage in

Life, Careers

Undergraduates Transcript Analysis, Surveys of IAFL Plans at Graduation

Grad & Prof Students Survey at Graduation

IAFL Usage and Plans

Implementation: A Tale of Two Environments

PHASE I. Original Design and Early ImplementationFull university support, major international initiatives at

Presidential levelMajor federal grant funding to the MacMillan CenterUniversity support with Institutional ResearchExternal consultants and university partners

PHASE II. Re-Design and Final Implementation End of spring 2011 into AY 2011-12, 2012-13, 2013-14, 2014-15

Presenter

Presentation Notes

But then we were turned upside down….

Implementation: A Tale of Two Environments

PHASE I. Original Design and Early Implementation AY 2009-10, 2010-11, hand-off in spring summer 2011 for AY 2011-12

PHASE II. Re-Design and Final Implementation Federal funding cut 47% Economic downturn takes full effect, cutbacks at university Change in Yale leadership, Faculty of Arts & Sciences reorganization Priorities re-shaping Loss of OIR as partner, no consultants, DIY within MacMillan Center

GRADUATE & PROFESSIONAL STUDENTS

SURVEY AT TIME OF GRADUATION

(2012, 2013,2014, 2015)

Results

Highlights

4 years of solid outcome results using a simple, effective survey instrument at time of graduation

Can differentiate the way the student groups engaged Yale resources and plan to use their IAFL skills and networks in the future

Able to separate populations of IAFL-focused students by degree of specialization

Serendipity – insight into ways non-native English speakers enhanced their language skills, crucial to their future success

Population and Instrument

Population over four years (handout)4 years – MacMillan MA’s, PhD’s, Law3 years – Forestry & Environmental Studies, Public Health1 year – Management, Development Economics

International Focus Survey (handout)Ran annually, late April early May with graduating students

Clear Separation of Internationally-focused Students

The International degrees automatically went to the “specialist” groupMacMillan MA’s in Africa, East Asia, European & Russian Studies

or Global Affairs (as well as International Development Economics)

The separating question for the other degrees was: Did you focus on IAFL in your coursework? NB. Extracurricular focus tracked coursework directly, not extra

explanatory power.

IAFL Focus in Coursework: Issues

IAFL Focus in Coursework: World Regions, Cross-Regional

Proportion of IAFL-focused Students

050

100150200250300350400

PhD (n=368) Law (n=363) FES (n=258) SPH (n=192) SOM (n=69)no [625] yes [638] total [1,263]

Employment Plans: Government and Education

The Title VI-priority fields of employment Government and international organizations Teaching in higher education or K-12

Clear separation across groups PhD’s (IAFL focus) more likely to go into higher education (p =<.001) Law grads (IAFL focus) more likely to go into government (p = <.002) Other schools IAFL-focused also tend to enter government more than non-IAFL but

not statistically significant

Employment Plans: % of All vs. IAFL-focused, non-MCMC

IAFL-focused, non-MCMCAll Respondents

Careers X Area of Study and IAFL Focus

Expected Use of IAFL Skills, Networks in Future Work

Program

International Area/Knowledge

Skills Foreign Language SkillsInternational

Networks

MCMC 86% 70% 65%PHD 63% 55% 69%YLS 75% 47% 51%FES 86% 48% 64%SPH 77% 46% 77%SOM 86% 45% 77%IDE 85% 39% 62%

Language Use in Future Work: IAFL Focus PLUS Language Use/Study in Program

Two Key Variables Separating Specialist vs Non-Specialist

Foreign Language use/study is the most concrete measure of seriousness and intensity of international focus Q. #1. What FL they studied/used (not their primary/home tongue)? Q. #2. How their FL proficiency changed on scale of 1-5 (with 1 low, 5 highest)? NB. Critical languages are highest priority for federal grant funding NB. We did not indicate “non-English” so “English” could have been a legitimate response for

a non-native speaker of English

Travel for program-related purposes is the as 2nd key measure Q. Did you travel for MORE THAN 4 weeks, related to your programs of study? NB. Application for competitive travel support = indicator of demand (as cross-check)

Languages: Critical, Other, None Studied by All Respondents

FL Proficiency Changes: Average Gain/Loss by School

Language: Proficiency Gain/Loss Across All Respondents

None BeginnerInter-

mediate AdvancedHighly

Advanced Fluent

None, n=112 2.7% 37.5% 42.0% 10.7% 3.6% 3.6%

Beginner, n=45 2.2% 17.8% 48.9% 22.2% 6.7% 2.2%

Intermediate, n=89 0.0% 0.0% 22.5% 52.8% 21.3% 3.4%

Advanced, n=85 0.0% 0.0% 2.4% 32.9% 43.5% 21.2%Highly Advanced,

n=67 0.0% 0.0% 1.5% 3.0% 65.7% 29.9%

Fluent, n=71 0.0% 0.0% 0.0% 0.0% 5.6% 94.4%

None Beginner Inter-

mediate Advanced Highly

Advanced Fluent

None, n=28 3.6% 14.3% 57.1% 14.3% 7.1% 3.6%

Beginner, n=8 0.0% 0.0% 37.5% 62.5% 0.0% 0.0%

Intermediate, n=26 0.0% 0.0% 7.7% 73.1% 11.5% 7.7%

Advanced, n=17 0.0% 0.0% 0.0% 52.9% 47.1% 0.0%

Highly Advanced, n=18 0.0% 0.0% 0.0% 0.0% 77.8% 22.2%

Fluent, n=19 0.0% 0.0% 0.0% 0.0% 0.0% 100.0%

None Beginner Inter-

mediate Advanced Highly

Advanced Fluent

None, n=23 0.0% 60.9% 34.8% 4.3% 0.0% 0.0%

Beginner, n=7 14.3% 42.9% 42.9% 0.0% 0.0% 0.0%

Intermediate, n=14 0.0% 0.0% 21.4% 57.1% 21.4% 0.0%

Advanced, n=11 0.0% 0.0% 0.0% 45.5% 45.5% 9.1%

Highly Advanced, n=7 0.0% 0.0% 0.0% 0.0% 71.4% 28.6%

Fluent, n=5 0.0% 0.0% 0.0% 0.0% 0.0% 100.0%

MCMC

FES

Program-focused Travel: Student Demand Indicator

Serious travel for research, study, language study or work/internships is a useful and concrete measure of students’ seriousness of pursuing IAFL skills and know-how

Tracking travel rates helps us measure student demand for IAFL engagement beyond coursework Application as indicator of intent or demand; award as an indicator of supply (not necessarily

merit of the candidates though clearly linked)

Track travel ≤4 weeks as an indicator of seriousness of purpose Undergrad and graduate/professional data available from a university-wide system

for 2009 to 2013

IAFL-Focused Student Travel Application Rate Higher than Non-IAFL

0% 20% 40% 60% 80% 100%

Area Studies, n=91

Forestry, n=90Forestry, IFL, n=167

PhD, n=176PhD, IFL, n=109

Yes No

ALUMNI SURVEY 2012AND

BENCHMARK DATA WITHCOFHE FOR YALE COLLEGE

Results

% Alumni Employment: IAFL-specialized BA, MA

17% 15%8%

13%

3%

22%

7% 9%13% 13%

42%

8% 5%11%

3%

37%

10%7%

12%4%

0%

25%

50%

Higher ed Gov't (US) Int't org NGO/Res. Own biz, prof.practice

% of Alumni in key work sectors by international focus in their degree (BA, MA, PhD/Prof) [2012 survey]

BA grads specialist [n=93] BA grads 10+ Eur, IR [n=46]Grad & Prof Specialist MA [n=38] Int'l PhD or Prof Degree [n=259]

Alumni Language Use: IAFL vs non-IAFL Majors

21%

12%

27%

47%

28%

56%

0%

10%

20%

30%

40%

50%

60%

In their work In their studies In other activities

% of Yale College alumni who report using foreign language in each context 5 years after graduating [classes of 2007, 2009]

All other majors [n=1384] Int'l or For. Lang. major *[n=70]

Benchmark Opportunity: FL Studied/Used

DIRECT LINK OF COURSES TO OUTCOMES:

TRANSCRIPT ANALYSISOF IAFL-FOCUSED BA’S, MA’S:

(2000-2009)

Results

IAFL-focused Alumni: Transcript Analysis 2000-09

Ten year baseline of IAFL-focused graduates [BA,MA]

IAFL-specific degrees MacMillan BA, MA’s in Area and International Studies Yale College BA’s, e.g., Spanish, other Lang/Lit

IAFL-shadow degrees [BA’s] by transcript 10+ Courses in a world area in a major, e.g. History IAFL courses contained 25-100% IAFL content

0%

10%

20%

30%

40%

50%

60%

Class Year

% o

f g

rad

uat

ing

sen

iors

ERS 54% 52% 45% 52% 48% 42% 46% 45% 38%

IR 40% 40% 23% 39% 38% 25% 37% 33% 26%

All Others 2% 3% 3% 3% 3% 4% 6% 9% 8%

2000 2001 2002 2003 2004 2005 2006 2007 2008

0%

3%

6%

9%

12%

15%

Class Year

% o

f gr

adu

atin

g se

nio

rs

AS 0% 1% 0% 1% 0% 0% 1% 2% 1%

EAS 1% 2% 1% 2% 3% 3% 4%

LAS 1% 2% 1% 1% 1% 0% 1% 2% 2%

MES 0% 0% 1% 1% 1% 3% 1%

2000 2001 2002 2003 2004 2005 2006 2007 2008

10+ Eur, IR, other areas 10+ smaller other areas

Engagement Trends of IAFL “Shadow Majors”

CURRICULAR RESOURCE AND ENROLLMENT

STABILITY AND VARIATION2000-2012

Results

IAFL Course Resource: Strong and High Density

IAFL Course Resource: Europe and “ROW”

IAFL Enrollments: Undergrad, Grad, Professional

IN THE END……

If you build it, they will come MacMillan built the pathway to excellence and many others walk the path

The value proposition is strong In the “budget battles”, IAFL costs clearly support more than small group of MCMC

specialist degree students Cost-effective “shadow” options, e.g. certificates, targeted language programs

Title VI got its money’s worth The design works -- All the pieces reveal different parts of the overall IAFL outcomes picture deep thanks to Cyndi Langin

Feedback from Audience

Collect information from each school participating through survey: We will summarize your responses and send the report to your email address if you request. The survey is the last page of the handout – please tear it off and respond in the next 5 or so minutes.

If your school measures international education outcomes: What is measured?

Groups of students assessed: International programs assessed Outcomes while in school (please list): Outcomes after graduation as alumni (please list):

How much university/college support is there for international assessments?

Is this assessment funded by other sources? Government? Other?

What have been primary findings from your assessments?

Have you completed a report that you can share?