Learning Outcomes: Critical to Assessment

Learning Outcomes: Critical to Assessment

Prepared for Office of Dean of Students

Assessment Training

January, 2011

Gail S. Rooney, Ph.D.

Director, The Career Center

University of Illinois at Urbana-Champaign

2

Session Purposes

Introduce a learning framework for student affairs

Gain experience writing learning outcomes tailored to your unique offices

Share sample ideas for engaging in the process of defining outcomes

Share sample evaluation project using learning outcomes

Engage in hands-on learning that you can take back to your offices

3

Session Outline

1. Learning as a framework for student development programs and services

2. Share a process for writing effective learning outcomes

• Defining context, brainstorming outcomes, writing outcomes statements, and considering assessment processes

3. Considerations of a learning outcomes approach for your office

4

Warm Up Activity

Write a learning outcome for a program or service that you might offer.

Rotate, discuss, and rate outcomes

Scale of 1 (not good) to 8 (excellent)

Add the rating scores for a total score

5

Part 1:

A Learning Framework for Student Affairs

6

Learning Framework

To be a part of the core functions of our institution, we need to think of ourselves as learning partners In an educational environment, learning is the

name of the game Student development programming is education As student affairs practitioners, we are facilitating

the development of lifelong and decision-making skills

Learning can be a paradigm for describing the value of our work

7

What is Learning? *

Learning is a complex, holistic, multicentric activity that occurs throughout the lifespan

Learning is more than information transfer – it is identity development (transformation), giving meaning, reflection and application in the context of the individual’s life

* Learning Reconsidered, 2004

8

What are Learning Outcomes? * Learning outcomes focus on intentional student

experiences (activities, courses, programs, services, resources)

What clients should be able to know, do, demonstrate or feel as a result of engaging in a learning experience

Outcomes help us define how clients have changed; they tell us how we are making a difference

* Keeling & Associates, 2007

9

Why Learning Outcomes? *

Outcomes define impact – how the client has changed

Focusing on learning moves beyond “bodies in the door” and “smiles on the way out”

The achievement of learning outcomes (individual success) measures institutional or intervention effectiveness


10

How do Learning Outcomes Shift Our Focus? *

Process

Planning activities and things to “do”

“What are we going to do?”

Counting and asking

Outcome

How the client will change as a result of what we do

How will clients be different; how will they learn, change, grow?

Documenting change; showing results


11

TCC’s Office-Wide Student Learning Outcomes

Posed two questions:

What do we want undergraduate students to know and be able to do upon graduation regarding personal and professional career development?

When a student graduates from the University of Illinois, what difference should The Career Center have made in their lives?

Gathered input from:

All office staff – counselors, support staff, graduate assistants, undergraduate outreach presenters, undergraduate office assistants

TCC Advisory Committee members from around campus

Questions

back in

Spring 2005

12

Application of Theories, Models, & Standards – TCC’s Office-Wide Student Learning Outcomes Brainstormed responses,

which you see in your handouts

13

TCC’s Office-Wide Student Learning Outcomes

There are no “right” answers… but you may be wondering: What did TCC come up with?

Employed a counseling model

Affective, Behavioral, and Cognitive outcomes

Followed learning domains from professional standards –

Explore self and options, Manage education, Use resources, Gain experience, Communicate accomplishments, Conduct a search

Recognized career development theory

Self, options, decision-making, life-long process

Affective components of encouragement, hopefulness, etc

(See “Samples” Packet, page 2)

14

TCC’s Office-Wide Outcomes: Practice

(See “Samples” Packet, pages 3 - 4)

15

Part 2:

A Process for Writing Effective Learning Outcomes

16

Learning Outcomes Cycle

Defining Context

Brainstorming Outcomes

Connecting Theories / Models /

Standards

Writing Outcome Statements

Prioritizing Focus Areas

Evaluating Interventions

Reconsidering Outcomes,

Evaluations, Interventions

17

Demonstrating a Process

Steps addressed in learning outcomes activities:

Defining context

Brainstorming outcomes

Writing outcomes statements

Connecting theories, models, & professional standards

Additional steps:

Prioritizing focus areas

Evaluating interventions

Re-considering outcomes, evaluations, and interventions

18

Defining Context

What is your program / service? This could be…

Small – a single workshop or activity

Medium – multi-session interactions

Large – all interactions between students and your office

Who are your intended learners? This could be…

Specific – first-year students in a particular minority group

Broad – all community members who may seek services

19

Defining Context – Example 1 Resume Review Services

Intervention: Daily resume review services provided on a drop-in basis to clients who are conducting job and internship searches, as well as by students who need to prepare a resume as a part of an academic class.

Intended Learners: Current students and alumni at the University of Illinois

(See handouts for more detail)

20

Defining Context – Example 2 Exploring Pathways in Career Success

Intervention: EPICS is an interactive, online, modular program that engages students in thinking about their career decision-making early in their college experience.

Intended Learners: Current undergraduate students at the University of Illinois at Urbana-Champaign


21

Defining Context – Example 3 Health and Graduate School Fair

Intervention: Health and Grad School Fair is held each fall in a large ballroom. It is a time of information networking and sharing regarding educational opportunities for college graduates.

Intended Learners: Undergraduate students who may or may not have decided to attend grad school.


22

Defining Context – Your Turn! Review the tips, examples, and prompts on pages

2 and 3 of your worksheet packet

Share with your partner: the program or service that you will work with

today the intended learners for that program or service

Record your responses on page 4 in your packet

23


What should (intended learners) know, be able to do, be able to demonstrate, value, or feel when they have completed (program / service)?

Or, stated another way…

What difference should (program / service) have made for (intended learners) who participate?

24

Brainstorming Outcomes – Resume Review Services Example What should undergraduate students know, be

able to do, be able to demonstrate, or feel when they have completed a drop-in resume review? Know the characteristics of a well formatted

resume (Know) Demonstrate accomplishments in their descriptions

of past work experiences (Do / Demonstrate) Feel confident in their ability to create a resume

that highlights their strengths and abilities (Feel)


25

Brainstorming Outcomes – EPICS Example What should undergraduate students know, be able to do,

be able to demonstrate, or feel when they have completed online modules in the EPICS program?

Understand how exploring their own personal interests is helpful for narrowing down potential major choices (Know)

Identify skills that they would like to use in their major (Do / Demonstrate)

Feel encouraged – that they can choose a major that will be a good fit for them (Feel)


26

Brainstorming Outcomes – Health and Grad Fair Example What should undergraduate students know, be able to do,

be able to demonstrate, or feel when they have attended the Health and Graduate School fair?

Understand the requirements for health and graduate school programs (Know)

Identify and implement next steps to prepare for graduate school (Do / Demonstrate)

Feel confident talking with health and graduate school representatives and exploring options (Feel)


27

Brainstorming Outcomes – Your Turn! Return to working with your partner

Review the tips, examples, and prompts on page 5 of your worksheet packet

Try brainstorming potential learning outcomes for the intervention and intended learners that you selected


28

Writing Outcomes Statements – First Round (Intended learners) who

(intervention) will be able to

(intended outcome)?

Engage in

Participate in

Complete

Identify Summarize

List Discuss

Describe Explain

Demonstrate{ }

{ }


29

Writing Outcomes Statements – Resume Review Services Example Undergraduate students who engage in drop-in

resume review services will be able to articulate a clear target or purpose for their resume.

30

Writing Outcomes Statements – EPICS Example Undergraduate students who complete EPICS

Competency Builder will be able to use resources to strengthen communication skills needed in their potential work environments.

31

Writing Outcomes Statements – Health and Grad Fair Example Juniors and Seniors who attend the Health and

Graduate School fair will be able to identify at least 3 educational options beyond their undergraduate degree.

32

SMART Approach to Writing Outcomes S Specific

M Measurable

A Achievable

R Relevant

T Time-based

33

Writing Outcome Statements –

Your Turn! Return to working with your partner

Review the tips, examples, and prompts on pages 7 and 8 of your worksheet packet

Try writing some outcome statements that connect intervention, intended learners, and intended outcomes that you have brainstormed


34

Writing Outcomes Statements – Second Round

Evaluation

Knowledge

Comprehension

Application

Analysis

SynthesisHigher Cognitive Levels

Lower Cognitive Levels

Bloom’s Taxonomy*Different Levels of Student Understanding & Performance

* Bloom, 1956

35

Writing Outcomes Statements with Bloom – Example Undergraduate students who engage in drop-in

resume review services will be able to:

list appropriate content for the education section of a resume (Knowledge)

use strong, varied action verbs to describe their experiences (Application)

integrate accomplishments / results into descriptions of their experiences (Synthesis)

36

Writing Outcome Statements with Bloom – Your Turn! Return to working with your partner

Review the tips, examples, and prompts on pages 10 and 11 of your worksheet packet

Try writing some outcome statements that connect activity, intended learners, and intended outcomes that you have brainstormed

Record your responses on pages 12 and 13 in your packet

37

Connecting Theories, Models, & Standards Theories / Models

Cognitive Information Processing

Krumboltz’s Learning Theory

Person-Environment correspondence (e.g., Holland’s typology)

Social Cognitive Career Theory

Super’s life-span, life-space Tiedeman’s decision

making model and many more…

Professional Standards National Career

Development Guidelines (NCDG; 1996, 2004)

National Association of Colleges and Employers (NACE) Principles (2006)

Council for the Advancement of Standards in Higher Education (CAS; 2003)

American School Counselor Association’s National Standards for Students (ASCA; 2003)

and many more…

38

Theories & Models – Benefits & DrawbacksBenefits

Helps us understand our assumptions

Provides recognized support for program and service decisions

Provides insights into:

important program and service components

target learners

potential learning outcomes

evaluation strategies

Drawbacks

Can create “blind spots”

May not adequately translate to our unique populations and contexts

At times, challenging to figure out how to translate theory into practice

39

Professional Standards – Benefits & DrawbacksBenefits

Provides recognized practices for program and service decisions

Provides a framework, each through a different lens CAS → system / program

level NCDG & ASCA →

individuals NACE → partnerships

between career practitioners and employers

Drawbacks

Standards speak broadly Need to tailor outcomes to

our unique learners / environments

Can be overwhelming Must prioritize to make it

manageable

40

Connecting Theories, Models, & Standards– Homework!

Review the tips, examples, and prompts on page 14 of your worksheet packet

Brainstorm the theories, models, and standards that influence your practice.


41

Connecting Theories, Models, & Standards – Homework A “take-home” exercise!

How do your learning outcomes currently speak to your theory/model/standards?

Would you change or re-word any of your learning outcomes based on your theory/etc.?

What learning outcomes might you add based on your theory/etc.?

How might your preferred theory/etc. help justify or support your choice of learning outcomes?

We’ll still share examples

42

Part 3:

A Sample Learning Outcomes

Evaluation

43

Learning Outcomes Cycle

Defining Context


Connecting Theories / Models /

Standards

Writing Outcome Statements

Prioritizing Focus Areas

Evaluating Interventions

Reconsidering Outcomes,

Evaluations, Interventions

44

Setting Evaluation Priorities Learning Reconsidered Framework, with activities

to engage staff in thoughtful discussion

Write learning outcomes for specific programs and services

Discuss program evaluation possibilities for the next year

(See “Samples” Packet, pages 5 - 9)

45

Setting Evaluation Priorities, Con’t

Selected three programs for “critical incident” evaluations

Exploring Pathways in Career Success (EPICS)

Resume Review Services

Mock Interview Program

Imagine building a “house of evidence” and each program evaluation is a brick

46

Evaluation Example: Resume Review Services Investigates:

Student learning outcome of effectively communicating accomplishments, as demonstrated through resume writing abilities

Three experimental groups

Participated ONLY in resume review services

Participated in resume review services AND other programs/services

Have NOT used programs/services (control group)

47

Evaluation Example: Resume Review Services Sample consisted of Juniors and Seniors in the

College of Liberal Arts & Sciences who had posted resumes on our online job posting system

Blind review of “self-selected, final product” resumes with pre-determined rubric (See handout)

Compared with ANOVA analysis and Dunnett’s Test of Multiple Comparisons Total rubric score Eight individual components of effective resume

writing

48

Evaluation Example: Resume Review Sample Findings Overall resume quality

In comparison to “No TCC” group: “Resume services and more” group found to have

significantly higher quality resumes (α<.05) “Resume services only” group found to have significantly

higher quality resumes (α<.10)

25.70**

24.60*

22.90

22.00

23.00

24.00

25.00

26.00

No TCC Resume Only Resume Plus

Mea

n S

core

49

Evaluation Example: Resume Review Sample Findings Resume components

Significant differences found between control and experimental groups on three components: education, additional resume sections, clear purpose/target.

3.50**

3.35**

2.80

3.15**

2.852.65

2.80*

2.252.25

1.00

2.00

3.00

4.00

No TCC Resume Only Resume Plus

Me

an

Sc

ore

s

Education

Additional Sections

Clear Purpose /Target

50

Evaluation Example: Resume Review Discussion Items Overall highest ratings

found on resume sections with “fairly standard” content Name / contact

information (3.82/4.00)

Spelling / grammar (3.47/4.00)

Education (3.22/4.00)

Modest overall means on areas that have general tips and guidelines, yet leave ultimate decision-making up to the resume writer. General format (2.90/4.00)

Additional resume sections (2.88/4.00)

Positive / professional tone (2.87/4.00)

Primary experience (2.82/4.00)

51

Evaluation Example: Resume Review Discussion Items Lowest overall mean on

component that requires themes to be threaded through various experiences, requiring high-level analysis and synthesis skills Clear purpose / target

(2.43/4.00)

52

Evaluation Example: Resume Review Discussion Items Overall significantly higher quality for both experimental

groups, in comparison to our control

Significantly higher quality for both experimental groups on the education section of the resume Content was more complete, organized, clear, and well-defined

than control group

53

Evaluation Example: Resume Review Discussion Items Significantly higher quality for “resume services and

more” group on the additional resume sections and clear purpose/target section

These sections demonstrate a “global sense” of career development and focus

Possible that participation in a wider array of career development programs and services is beneficial to assist students in developing skills to effectively communicate a cohesive set of career aspirations through connections among the wide varieties of experiences that make up the content of their resumes

54

Evaluation Example: Resume Review Limitations Small sample (20 resumes in each groups = 60 total)

Possible pre-existing differences in student populations

Who uses the online job posting service that resumes were drawn from? How might they differ from other students?

Students may be receiving resume and career assistance from other sources

Private counseling

Family members

Professional networks

Etc.

55

Evaluation Example: Resume Review Next Steps Recognize the good work that the resume review and counseling

teams have done in working with students (Celebrate successes!)

Consider areas for improvement in staff training, resume workshops, resume reviews, and (where appropriate) counseling appointments

Focus on room for growth in several resume component areas, such as: general formatting, experience sections, tone, clear purpose/target.

Training staff to make referrals to counseling appointments, particularly for students who are having difficulty:

Defining a clear purpose or target

Developing a cohesive, logical flow to the variety of experiences on their resumes

YOUR Evaluation

How will you evaluate your learning outcomes?

Examples?

Brainstorm assessment approaches

Surveys

Documents

Experimental – Pre & Post

Qualitative

Others?

57

Value of Learning Outcomes

What are learning outcomes?

Learning outcomes define impact; they measure institutional or intervention effectiveness

Learning outcomes tell us how the client has changed and how we are making a differenceLearning Reconsidered, 2004

58

Connections Between Learning Outcomes Evaluation and Advocacy Bringing learning outcomes and advocacy

together

Learning outcomes tell us much more about the impact of our programs and services than traditional strategies of participant counts and satisfaction surveys

They show the difference that we make, giving us powerful evidence of the value of our work

59

Learning Outcomes Evaluations: “The Challenges” Pioneering work that may not be fully understood

Requires sustained effort, energy, resources, and leadership over time

Crosses many evaluation fields leading to confusion regarding terms and strategies

Requires developing new creative thinking and technical evaluation skills

Other ideas?

60

Thank You for Participating! For more information, contact me at:

Gail S. [email protected]