Embed Size (px)
Citation preview
HolisticApproachestoAssessingInterculturalAgility
❏ Dr. Nick Gozik, Boston College, [email protected]❏ Emelye Neff, Middlebury Institute of
International Studies at Monterey, [email protected]
❏ Dr. David Wick, Middlebury Institute of International Studies at Monterey, [email protected]
❏ Heather Young, IFSA, [email protected]
Resource Folder https://tinyurl.com/y9kp6mje
InterculturalAgilityInAction
HolisticApproachestoAssessingInterculturalAgility
Agenda: I. Our Assessment KnowledgeII. Designing and Refining
AssessmentsIII.Your Assessment Plan
Resource Folder https://tinyurl.com/y9kp6mje
OurAssessmentKnowledge:GroupIdeas
What single words describe the value or benefit of studying abroad?
ReflectionandDiscussion
• Individual Reflection: What responses stand out to you? Why?
• Turn and talk: With your partner, select one to two values. • What would these values look like in
action? • How could you verify growth in learning
about these values?
● Whatevidenceofstudentlearningdowealreadyhave?
● Whatdataarewealreadycollecting?
SectionII:DesigningandRefiningAssessments
Resource Folder https://tinyurl.com/y9kp6mje
AssessmentCycle
● Assessment begins and ends with a mission
● Should be holistic and cyclical, with continual revisions
Mission Statement
Tools Learning Outcomes
GoalsRecipients & Actions
LevelsofAssessment
While there are multiple levels of assessment, this workshop will focus on assessing learning outcomes
University International Initiatives
Student Learning Outcomes
Programs
Office
Assessment Plan
SampleSMARTLearningOutcomesforIATable of KSA’s for Intercultural Agility that are SMART and reflect Bloom’s taxonomy.1. Knowledge Cultural Worldview Frameworks
By the conclusion of a program, students demonstrate an increasing understanding of “culture” as a concept as well as specific elements important to members of another culture in relation to its history, values, politics, communication styles, economy, or beliefs and practices.
1. Skills Verbal and Nonverbal CommunicationBy the midway point of the program, students recognize and participate in culturally appropriate verbal and nonverbal communication, beginning to negotiate a shared understanding based on cultural differences.
1. Perspectives Openness Before returning to their home institution, students begin to initiate and develop interactions with culturally different others, sometimes suspending judgment in valuing her/his interactions with culturally different others.
Howcanthedirectandindirectevidencewehavehelpustoshapestudentexperiencesandassessstudentlearning?
Direct: Evidence of student learning
Indirect: Student perception of their competencies
Form
ativ
e: E
vide
nce
gath
ered
du
ring
lear
ning
to c
heck
pro
gres
s an
d ad
just
Summ
ative: Evidence of
achievement of learning objectives
AddressingtheGoalofIncreasingInterculturalAgility
Choose One Activity:A. Making Learning
Objectives SMART -Emelye
B. Application Essays to Post-Program Prompts - David
C. Rubric Norming for Assignment Review -Heather
D. Satisfaction Survey Refinement - Nick
Mission Statement
Tools(B → C → D)
Learning Outcomes (A → B)
Goals(A → B)
Recipients & Actions
SectionIII:YourSMARTAssessmentGoals
Resource Folder https://tinyurl.com/y9kp6mje
15
ComposethreeSMARTgoalsrelatedtodevelopingaholisticassessmentsystem
Short Term: By end of Summer
Mid Term: By end 2018
Long Term: Before IFSA Conf 2019
HolisticApproachestoAssessingInterculturalAgility
❏ Dr. Nick Gozik, Boston College, [email protected]❏ Emelye Neff, Middlebury Institute of
International Studies at Monterey, [email protected]
❏ Dr. David Wick, Middlebury Institute of International Studies at Monterey, [email protected]
❏ Heather Young, IFSA, [email protected]
Resource Folder https://tinyurl.com/y9kp6mje
Holistic Approaches to Assessing Intercultural Agility
Friday, July 27, 2018, 8:45 - 10 IFSA Butler Annual Conference 2018, Indianapolis, IN
Folder Link: https://tinyurl.com/y9kp6mje
Activity: Creating and Testing SMART Learning Outcomes
Adapted from Dr. Darla K. Deardorff’s book Demystifying Outcomes Assessment for International Educators Outcome vs. Goal: Goals are generally too broad to assess. Goals ask “where are we going?” Outcomes answer the question “what change will occur?” They define the desired impact of a learning intervention. What is a SMART Learning Outcome?
Language You Can Use to Make SMART Outcomes: Bloom’s Taxonomy Verbs and Relevant Activities
Image from Regis University, College for Professional Studies
Holistic Approaches to Assessing Intercultural Agility
Friday, July 27, 2018, 8:45 - 10 IFSA Butler Annual Conference 2018, Indianapolis, IN
Folder Link: https://tinyurl.com/y9kp6mje
PRACTICE I: SMART-test and revise the following learning outcomes
1. Learners understand multiple cultural perspectives on climate change.
________________________________________________________________________________________
________________________________________________________________________________________
2. Learners know that there are many worldview orientations.
________________________________________________________________________________________
________________________________________________________________________________________
3. Learners can communicate effectively about cultural differences.
________________________________________________________________________________________
________________________________________________________________________________________
4. Learners show empathy for the perspectives of others.
________________________________________________________________________________________
________________________________________________________________________________________
PRACTICE II: Develop a learning outcome for competencies related to Intercultural Agility.
INTRAPERSONAL:
Cultural self-awareness
COGNITIVE:
Knowledge and Meaning-making
INTERPERSONAL:
Communication & empathy
___________________________ ___________________________
___________________________
___________________________
___________________________
___________________________
___________________________ ___________________________
___________________________
___________________________
___________________________
___________________________
___________________________ ___________________________
___________________________
___________________________
___________________________
___________________________
Image from 123RF.com Image from GameGrin.com Image from Psychology for Tasmania
Holistic Approaches to Assessing Intercultural Agility Friday, July 27, 2018, 8:45 - 10
IFSA Butler Annual Conference 2018, Indianapolis, IN Folder Link: https://tinyurl.com/y9kp6mje
Activity: Revising Application Questions for Assessment
Education abroad applications serve many functions. This group will consider how to align application essay questions in order to gather qualitative pre-program evidence of student knowledge and skill. This approach has many potential benefits. It can,
● yield stronger written work to assess student readiness and facilitate selection, ● provide a baseline related to a student’s intrapersonal, interpersonal, and intercultural
strengths and opportunities for growth, ● and initiate reflection on the purpose and meaning of the international experience to
initiate learning related to program goals. Using similar questions during pre-departure, while abroad, and after students return and scoring all against a rubric can provide evidence of student learning. Refining application prompts begins at the Student Learning Outcomes stage in the assessment cycle. For this exercise we have selected some outcomes from the AAC&U Values Rubric on Intercultural Knowledge and Competence (https://www.aacu.org/value/rubrics/intercultural-knowledge), because it is closely aligned with IFSA’s Intercultural Agility construct. Examples for Future Reference: Each of these has strengths and opportunities for development. Together they may give you ideas for building a strong set of questions and evaluation rubrics for your program.
Beloit College Essay Questions and Rubric: https://www.beloit.edu/oie/assets/Study_Abroad_Essay_Prompts_and_Rubrics_2016_November.pdf Santa Clara University Prompts and Evaluation Rubric: https://www.scu.edu/globalengagement/study-abroad/apply/ Follow links for “Preview the Application Questions” and “Read the Application Rubric” University of Delaware Interview Prompts for Program Leaders: http://www1.udel.edu/global/studyabroad/information/interviewing_students.html
Tools/Resources for this Activity:
AAC&U Intercultural Knowledge and Competence Values Rubric: https://www.aacu.org/value/rubrics/intercultural-knowledge Bloom’s Question Stems: https://www.centergrove.k12.in.us/cms/lib/IN01000850/Centricity/Domain/1217/blooms_qstems.pdf
Student Learning Outcomes Definitions and Outcomes from: American Association of Colleges & Universities
Intercultural Knowledge and Competence Values Rubric*
Definition: Intercultural Knowledge and Competence is "a set of cognitive, affective, and behavioral skills and characteristics that support effective and appropriate interaction in a variety of cultural contexts.” (Bennett, J. M. 2008. Transformative training: Designing programs for culture learning. In Contemporary leadership and intercultural competence: Understanding and utilizing cultural diversity to build successful organizations, ed. M. A. Moodian, 95-110. Thousand Oaks, CA: Sage.)*
Knowledge/Cognitive Attitude/Affective Skill/Behavioral
Student Learning Outcome: Articulates insights into own cultural rules and biases (e.g. seeking complexity; aware of how her/ his experiences have shaped these rules, and how to recognize and respond to cultural biases, resulting in a shift in self-description.)*
Student Learning Outcome: Initiates and develops interactions with culturally different others. Suspends judgment in valuing her/ his interactions with culturally different others.*
Student Learning Outcome: Interprets intercultural experience from the perspectives of own and more than one worldview and demonstrates ability to act in a supportive manner that recognizes the feelings of another cultural group.*
Current Questions: What experience(s) do you have with other countries and cultures? What academic, professional, and personal knowledge do you hope to gain?
Current Questions: Why do you want to study abroad? Where do you want to study abroad?
Current Questions: What do you hope to gain from this experience? How do you contribute to a group?
What question could you ask to yield richer data that allows you to assess against the outcome above? Consider prompts such as “what do you know about?” or “how would you describe?”
What question could you ask to yield richer data that allows you to assess against the outcome above? Use “provide examples,” “list and explain,” or “
What question could you ask to yield richer data that allows you to assess against the outcome above? Start with “describe a time when,” “explain how you,” “what will you do to,” or “give an example of.”
Holistic Approaches to Assessing Intercultural Agility Friday, July 27, 2018, 8:45 - 10
IFSA Butler Annual Conference 2018, Indianapolis, IN Folder Link: https://tinyurl.com/y9kp6mje
Activity: Rubric Norming for Assessing Learning
Develop your institutionally specific rubric
1. Use a rubric, like an AAC&U Value Rubric, or one that your institution already uses, to start a frame of reference for your thoughts and ideas.
The Rubric Norming Cycle: (using Intercultural Agility as an example)
Developing a Rubric: The following provides a procedure for developing a rubric, from the Yale Center for Teaching and Learning, adapted from Brown's Sheridan Center for Teaching and Learning
● Define the goal and purpose of the task that is being evaluated ○ What are the organizational goals? What is the Mission and Vision of your
organization and what outcomes are you hoping to achieve with this assessment?
● Decide what kind of rubric to use ○ The kind of rubric used may depend on the nature assessment, the intended
learning outcomes. For instance, does the task require the demonstration of several different skills? What is the amount and kind of feedback students will receive? Is the task a formative or a summative assessment? Is this rubric student facing? Will peer or self assessment occur?
● Define the criteria ○ Review student learning outcomes and assessment parameters to determine
specific criteria for the rubric to cover. What competencies and skills are prioritized by your organization? Criteria should be distinct and clearly described, and ideally, not surpass seven in number.
● Frame your language ○ Instructors should consider what knowledge and skills are required for successful
completion of the competencies and skills defines in the previous step, and create a list of criteria that assess outcomes across different vectors (comprehensiveness, competency, attitudes, career readiness).
● Define the rating scale to measure levels of performance
Holistic Approaches to Assessing Intercultural Agility Friday, July 27, 2018, 8:45 - 10
IFSA Butler Annual Conference 2018, Indianapolis, IN Folder Link: https://tinyurl.com/y9kp6mje
○ Choose the type of scale that best suits your institutional needs. Examples (below) from the Harriet W. Sheridan Center for Teaching and Learning. Rubrics typically include 3-5 levels in their rating scales. The AAC&U uses 4 levels in the Value rubrics, but breaks down levels 2 and 3 under the same scale.
■ Three Levels ● Beginning, Intermediate, High ● Developing, Competent, Exemplary
■ Four Levels ● Beginning, Developing, Accomplished, Exemplary ● Emerging, Progressing, Partial Mastery, Mastery
■ Five Levels ● Poor, Minimal, Sufficient, Above Average, Excellent ● Novice, Intermediate, Proficient, Distinguished, Master
■ Six Levels ● Unacceptable, Emerging, Minimally Acceptable, Acceptable,
Accomplished, Exemplary ● Write descriptions for each performance level of the rating scale
○ Each level should be accompanied by a descriptive paragraph that outlines ideals for each level, lists or names all performance expectations within the level, and if possible, provides a detail or example of ideal performance within each level. Across the rubric, descriptions should be parallel, observable, and measurable.
● Test and revise the rubric ○ The rubric can and should be tested before implementation. Utilize previous
examples of student work to test your rubric and clarify questions and concerns. For example, using the IFSA “Unpacked” student examples on the IFSA website.
Practice norming a rubric for your institution: 1. Define the goal and purpose
a. 2. Define the criteria
a. b.
3. Frame your language. a. b.
4. Define the rating scale a.
5. Write descriptions for each performance level based on student learning outcomes a.
For reference and planning:
● “Parts of the rubric”: https://www.aacu.org/parts-value-rubric ● Bloom’s Taxonomy Cheat Sheet (from Canterbury College)
Holistic Approaches to Assessing Intercultural Agility
Friday, July 27, 2018, 8:45 - 10 IFSA Butler Annual Conference 2018, Indianapolis, IN
Folder Link: https://tinyurl.com/y9kp6mje
Activity: Study Abroad Program Evaluation Sample for Revision
Office of Study Abroad Services
1. Indicate your level of satisfaction with the resources, information and support provided (circle below).
Extremely Dissatisfied Dissatisfied Satisfied Extremely Satisfied
2. List resources that you found most helpful.
On-site Support
3. Indicate your level of satisfaction with the resources, information and support provided by the on-site staff.
Extremely Dissatisfied Dissatisfied Satisfied Extremely Satisfied
4. Indicate your level of satisfaction with the program leader(s) that accompanied you while abroad. Extremely Dissatisfied Dissatisfied Satisfied Extremely Satisfied
Academics Abroad
5. How many hours of credit will you receive for your program? ______
6. Indicate your level of satisfaction with the number of credits awarded for your study abroad program.
Extremely Dissatisfied Dissatisfied Satisfied Extremely Satisfied
7. Indicate your level of satisfaction with your classroom instructors. Extremely Dissatisfied Dissatisfied Satisfied Extremely Satisfied
8. Indicate your level of satisfaction with your host institution.
Housing
9. What housing was provided?
University housing Host family accommodation Apartments Hotels Other: ______
Holistic Approaches to Assessing Intercultural Agility
Friday, July 27, 2018, 8:45 - 10 IFSA Butler Annual Conference 2018, Indianapolis, IN
Folder Link: https://tinyurl.com/y9kp6mje
10. Please indicate your level of satisfaction with the housing arrangements.
Extremely Dissatisfied Dissatisfied Satisfied Extremely Satisfied
11. Describe your living situation and how it impacted your study abroad experience.
Social & Cultural Experiences
12. What social and cultural adjustments did you make in your host country?
13. In what setting(s) did you interact with locals? Circle all that apply.
Class Housing Service-learning or Volunteering Errands Excursions
Internship Other
14. How did the social and cultural experiences impact your study abroad experience?
Overview
15. How has studying abroad impacted your academic and/or career goals?
16. What benefits did you receive from studying abroad?
17. What was the most satisfactory aspect of the program?
18. What was the least satisfactory aspect of the program?
19. What suggestion(s) do you have to improve the program?
Holistic Approaches to Assessing Intercultural Agility Friday, July 27, 2018, 8:45 - 10
IFSA Butler Annual Conference 2018, Indianapolis, IN Folder Link: https://tinyurl.com/y9kp6mje
Activity: Assessment vs. Customer Satisfaction
Consider whether or not you are asking questions that are helping to assess student learning vs. gauging customer satisfaction; these are not the same.
Examples:
● “I enjoyed my homestay” … “To what extent did your homestay help you improve your language skills?”
● “My coursework was rigorous” … “In what ways did your coursework abroad give you an appreciation for a different educational system?
Using the sample evaluation, within your table:
● Determine which questions help assess learning outcomes vs. those that gauge satisfaction (Mark “A” or “S” next to questions).
● Adapt five (5) questions that you marked as “S”, so that they become assessment measures.
Extra Credit:
● Create one additional open-ended question to gauge an important student learning outcome on your respective campus/in your organization.
