Learning Outcomes Made EasyUsing the Best Tools

Jeffrey D. Keith, Ph.D.

J. Kelly Flanagan, Ph.D.

Russell T. Osguthorpe, Ph.D.

Danny R. Olsen, Ph.D.

Tom Mallory

Jeff Fox, Ph.D.

Brigham Young University, Provo UT

2

11 Academic Colleges 56 Academic Departments 400 Degree Programs 1,600+ Faculty 8,000+ Courses

33,000+ Students

3

Carnegie Classification: Research University (High Research Activity)

Northwest Commission on Colleges and Universities

30 Specialized Accrediting Organizations

BRIGHAM YOUNG UNIVERSITY

4

U.S. Regional Accreditors

5

Assessment Charge

Publish expected learning outcomes for all of their courses and programs

Provide evidence that the expected learning outcomes are realized by students

Demonstrate how such data collection and analyses leads to continuous improvement of student learning, the curriculum and the university

Colleges and universities across the country are being asked to:

6

Why “Student Learning Outcomes”?

1. Ethics: Assessment should ultimately (first and always) be about student learning Student major selection Aligning institutional, department goals Student “take-aways”

2. Pressure: Stakeholders Accrediting entities Government agencies Employers

7

What are Student Learning Outcomes?1. Definition: Statements defining what students

should know, understand and/or be able to do as a result of their learning experiences in a degree program Cognitive (knowledge) Behavioral (skills) Affective (attitudes)

2. Assessment: What counts as evidence that a given level of competence has been achieved.

3. Clear and assessable statements of the “essential and enduring knowledge, abilities, and attitudes or dispositions” that enable a learner to apply what he or she has learned to situations encountered in the real world.

Direct & Indirect EvidenceDIRECT MEASURES Imbedded performance

assessment mapped to learning objectives

Course and homework assignments

Exams, quizzes Standardized tests Term papers, reports Research projects Capstone projects, senior

theses, exhibits or performances

Pass rates, scores on licensure, certification, area tests

Rubric scores

INDIRECT MEASURES Course evaluations Alumni questionnaire data Senior survey data National Survey of Student

Engagement data Employer surveys Graduate school placement

rates Department exit interviews Focus group interviews with

students, faculty, etc. Job placement Registration or course

enrollment information8

9

Evidences of Learning on Pathways to Graduation

10

University Goal AwarenessTo Publish Course Level Learning Outcomes by August 31, 2011

Law

Relig

ious

Ed

Man

agem

ent

Educa

tion

Engin

eerin

g

Fine

Arts

Socia

l Scie

nces

Physic

al S

ci

Life

Scie

nces

Human

ities

Nursin

g

Unive

rsity

0%

20%

40%

60%

80%

100%

22%

40% 44%60% 61% 62% 68% 71% 72% 73%

80%

63%

By College as of June 1

11

Published, Mapped Course Outcomes

October 2010

April 18 June 1 July 21 Aug 31 GOAL

0%

20%

40%

60%

80%

100%

9%22%

29% 32%

100%

0.0830000000000001

% Courses with Published Outcomes% Course Outcomes Mapped to Program Outcomes

2011

12

Obstacles & Challenges Culture of assessment disparity across campus. Quality of stated “Expected Learning Outcomes”. Obtaining faculty consensus within departments. Faculty engagement in the process as part of the

status quo without any extra compensation. Moving from a faculty-centric to a student centric

language of stated outcomes. Appropriate channels of evidence—direct, indirect. Demonstrating closing the assessment loop Applied Technology—cost, time, scalability, usability

Design Goals• Ease of use

Editing, viewing, and linking of program and course outcomes must be easy

• Leverage existing IT web services Security, identity, course information

• Data accessibility Existing university applications

Course catalog, registration, syllabus builder

14

Achieving Design Goals

• Ease of use Usability

• Leverage existing OIT web services University IT department adopted SOA SOA registry

• Data accessibility Adopted a service model

Common interfaces for interoperability Service registered in SOA registry

Learning Outcomes

Service

Learning Outcomes Web site

Exams

Syllabus Builder

Discussions

CourseContent

Program Assessment

Tool

CourseRegistration

CourseCatalog

Course Assessment Tool

CourseEvaluations

Challenges

• Course Learning Outcome Outliers Certificates Emphases GE Service Course

• Building a Learning Outcomes Assessment tool that meets disparate needs.

Faculty Involvementhttp://www.youtube.com/byuctl#p/u/0/-oy6ztc1kq0

Course Catalog

Course Evaluations

Program Assessment Tool

Just Beginning Design…

Course Content

Discussions

Exams

Syllabus Builder

26

LO Website 1

29