Fostering Global Citizenship in Higher Education

Assessment and Evaluation

Session Agenda

Topics• Setting the context• Defining

competencies• Assessing growth Contexts

• Graduate• Undergraduate

Objectives

• identify ideas on how to implement assessment & evaluation techniques at the program and degree levels

• review how developing competencies are linked to program design and measurement

• understand the merits and challenges of different methods and timing of data collection; and

• implement how to use what has been found in future program design

…a number of inter-related trends at the macro and mezzo level currently

are affecting program design throughout higher education in the US..

Among those trends are:

An increase in short-term, faculty-led programs to accommodate new majority students, professional

school students, and other student groups new to education abroadAn increase in required pre-departure, re-entry, and embedded programsAn increase in home campus or sponsoring agency intervention in the experience to support/direct

students’ learningAn increase in student/parent expectations for student support services that mirror those found in the US,

whether or not such services are typically provided to students in the host countryAn increase in education abroad being part of a campus-wide plan for internationalizationAn increasing emphasis on program evaluation, particularly outcomes assessmentAn increase in the emphasis on economic rationales for education abroad, consumerism, and education

abroad in its job preparation contextRodman & Merrill, 2008

Developments in “the academy”

Classroom-basedTour-based

Experience-based

embedded

grand tour

hybrid

direct enroll

service learning

service

branch campus

study center

work internships

short-term group excursion

independent study

language training

volunteerism

adventure

field-based

mission work

Situating the Changing Study Abroad Program Models

Classroom-basedTour-based

Experience-based

embedded

grand tour

hybrid

direct enroll

service learning

service

branch campus

study center

work internships

short-term group excursion

independent study

language training

volunteerism

adventure

field-based

1950s

2000s

1960s

1970s SIETAR

Intercultural Communications

Literature Maturation

mission work

Study Abroad Program Model Development & ICC Maturation

SIT/PIM Definition of Competency

• Describe outcomes of the learning process. – action-oriented statements – tell students what they will be able to do, to

know, or be aware of by graduation.– related to their professional and personal

lives.

Competencies are found in the following domains:

• Knowledge (Cognitive)– Theories, concepts, and models the student can

discuss

• Skills (Behavioral)– What the student can do

• Self-awareness (Affective)– The awareness of personal values, attitudes, and

attributes, and how they connect the individual to their larger environments.

Competency delivery

flow through phases of the program:

• On campus format:– Prior experience + on-campus program core courses and

educational activities + degree courses + reflective practice phase+ capstone + language and cultural requirement

• Low Residency format:– Prior experience + summer residency + online coursework with

reflective practice + summer residency + online coursework with reflective practice + capstone + language and cultural requirement

Program Competency Areas

1. Theory and Practice of Learning2. Professional Development 3. Identity, Culture, and Communication 4. Reflection and Inquiry 5. Leadership 6. Social Change Processes 7. Teams and Organizations 8. Community and Global Awareness 9. Degree specific competencies (e.g., for

International Education program design)

Community and Global Awareness

Global and community awareness is based on our respect for and appreciation of the interconnections of individuals and systems in our environment and on the planet. This awareness includes one’s capacity to:

• Identify the issues and their impact on one’s own and others’ communities

• Make informed choices regarding personal community involvement, social justice issues and leadership roles

• Recognize the value of multiple perspectives in community and global life

• Understand macro level political and social theories on community and global awareness

• Identify the role of conflict transformation, power, and ethics• Articulate the way in which technology affects interconnections of

local and global communities

Name of competency

Current status of delivery Review core course objectives, campus workshops & activities, etc.

Value to mixed degree student group(1,2,3) *below

When delivered Other potential places to deliver

Measurement

1-LearningTheory and Practice

Learning styles assessment ,all program phases as articulated through students’ learning plans

Early in Fall termOrientation

workshop

2- Professional Development

Workshops career development, On-going classroom presentations, course and capstone research, Professional contacts with faculty, alumni, staff, Courses-OB1, practitioner inquiry, Training Design RPP

8-Community & Global Awareness

Courses- ICC, Social Change, On-campus extra-curricular groupsRPP

2 Fall 2 course, Spring 1 courseYear longpracticum

Orientation workshop, speakers across degrees during the academic year

1=necessary2= highly desirable3-nice but not required

Evaluation is “any effort to use assessment evidence to improve individual, departmental, divisional or institutional effectiveness.”

Upcraft, 2003

What do we mean by “evaluation”?

Assessment is “any effort to gather, analyze and interpret evidence, which describes institutional, divisional, or agency effectiveness.”

Upcraft, 2003

What do we mean by “assessment”?

Assessment and Evaluation are tied in with Design

From Trochim

Gardner’s 5 evaluation frameworks (1994)

• Professional judgment

• Measurement

• Decision-oriented evaluation

• Goal-free/responsive evaluation

• Assessment of congruence between performance and objectives

Assessment of congruence between performance and objectives

A Formative and Summative Process of:

• Identifying goals, objectives or standards of performance

• Identifying or developing tools to measure performance;

• Comparing measurement data collected with previously identified objectives or standards.

(Gardner, 1994)

Assessment of congruence between performance and objectives

Individual Program

What competencies (knowledge, attitudes, skills, awareness) will the individual learner develop?

What does the program (institution) hope to accomplish by this program?

Examples? From whose perspective?

Examples? From whose perspective?

Methods

Choice depends on:

• Scope of assessment

• Values or paradigm (you, the institution)

Scope of the Assessment

Institution

Program

Course

Student

Institutional Context

• Who are the decision makers inside and outside of your program or institution?

• What do they value?

Challenges of assessment and evaluation

• Conflicting needs and values• Range of methodological approaches • Criteria ambiguity or disagreement • Difficulty in attributing change• Overzealous desire to measure• Complexity• Controversy • Clarity

Best Practices of Assessment and Evaluation

• Involve all stakeholders at all stages.• Determine individual meaning of assessment

elements• Clarify purpose, timing, uses and appropriate

approaches • Identify possible “intervening factors” • Triangulate methods and participants. • Work ethically • Sort out “measurability”

Group Discussion

•Application

•Questions