Strengthening Academic Assessment for Programs

A Systemic Approach

Timothy S. Brophy, Ph.D., Director of Institutional Assessment

University of Florida Office of the Provost

Today’s GoalsDescribe and explain assessment systems

Review UF’s program assessment components

Discuss program assessment systems and steps to strengthen them

Review examples

Assessment SystemsA system is a complex whole in which there are multiple parts or phases that can stand alone but are stronger when linked logically into a cohesive combinationThe system can use linear and nonlinear sequences of events or processes, or a combination of both

Assessment systems consist of curricular, educational, and procedural components

The UF Assessment System is one example

UF Assessment System

Establish Mission,

Goals, and Outcomes

Assessment Planning

Implement the Plan

and Gather Data

Interpret and

Evaluate the Data

Modify and Improve

May –

Asse

ssmen

t P

lan

s an

d

Eff

ective

ness

Docu

men

tatio

n R

ep

orts

sub

mitte

d fo

r th

e n

ext A

Y

October - Assessment

Data, results, and use of results for

previous AY entered into Compliance

Assist!

Assessment and Institutional Effectiveness

Data Reporting

Elements of program assessment systems

Catalog• ALCs• Graduation requirements

Assessment Plans• SLOs• Measurement tools

• Curriculum Maps and Assessment matrices

Compliance Assist!• Goals • Results• Use of results• Student work samples

The Academic Learning Compact

2012-13 UF Undergraduate Catalog

Academic Learning Compacts identify the skills students should acquire if they follow their major’s prescribed course of study. These skills, known collectively as Student Learning Outcomes, describe the core learning expectations that UF is required to assess for each baccalaureate degree program. Core learning expectations identify communication, critical thinking, and content knowledge skills, as well as additional learning outcomes specific to the major.

Student Learning Outcomes

What students are expected to learn by the completion of their baccalaureate degree

Individual Student AssessmentsThe different ways in which UF will measure whether students have successfully completed the learning outcomes for a major. These assessments can include a passing score on a particular test or final project, term paper, portfolio, etc.- 2012-13 UF Undergraduate Catalog

How are these components related?

Academic Learning

Compact

What we sometimes have

SLO

SLO

SLO

Assessm

en

t Assessment

Assessment

Course

Cou

rse

Course

What we should have in all programs

Academic Learning Compact

SLO

Assessment

Course

Assessment

Course Course

SLO

Assessment

Course

Assessment

Course

SLO

Assessment

Course

Assessment

Course

How do we get from

ALC

SLO

Assessment

Course

Assessment

Course Course

SLO

Assessment

Course

Assessment

Course

SLO

Assessment

Course

Assessment

Course

SLO

Ass

ess

ment

Co

urse

Course

Academic

Learning

Compact

SLO

SL

O

AssessmentCourse

THIS TO THIS?

And avoid this when dealing with assessment?

SLOs!

!

Plans!

!! Rubric

s!!!

Start with the ALC program description

Components of a strong ALC program description:

Establishes learning goals for the degree program – the “core learning expectations”

Addresses the relationship of the degree to the job or career

Characteristics of Program Learning Goals found in program descriptionsThese are broad statements that present multiple

measurable components so the statement itself may be “unmeasurable”

These often use verbs like “understand” and “learn” and “gain the ability to…” etc. in the program description

Why? These are the broad LEARNING GOALS of the program –

not the LEARNING OUTCOMESSLOs represent the measurable components of the SLOs

– they are “S.M.A.R.T.” SLOs are stated using active verbsHowever, the SLOs and goals share common language

and intent

This model allows you to develop assessments that measure the outcomes, and that then connect directly to the program goals

Specific Learning Outcome – the course-level SLO

General Learning Outcome – the program-level SLO – these describe what students will do to demonstrate the learning

goals

Learning Goal Level – programs establish learning goals for the degree – at UF these are described in the Academic

Learning Compact

Developing an Assessment System: A Three-level Model (Carriveau, 2010)

A sample description:Materials Science EngineeringThe major enables you to develop an understanding of materials systems and their role in engineering. Emphasis is placed on the ability to apply knowledge of mathematics, science and engineering principles to materials science and engineering; to design and conduct experiments, as well as to analyze and interpret data; and to design a system, component or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability and sustainability.

Learning Goals

Connection to the job or career

Students who complete the MSE degree will:

Understand materials systems and their role in engineering

Gain the ability to apply knowledge of mathematics, science and engineering principles to materials science and engineering

Level 1: Learning Goals based on the MSE ALC

Content Knowledge• Apply knowledge of mathematics, science and engineering

principles to materials science and engineering. • Design and conduct materials science and engineering

experiments and analyze and interpret the data.

Critical Thinking• Design a materials science and engineering system, component

or process to meet desired needs within realistic economic, environmental, social, political, ethical, health and safety, manufacturability and sustainability constraints.

Communication• Communicate technical data and design information effectively

in speech and in writing to other materials engineers.

Level 2 – Program Student Learning Outcomes for MSE

Learning Goals: Understand materials systems and their role in engineeringAbility to apply knowledge of mathematics, science and engineering principles to materials science and engineering.

Student Learning Outcomes: Design a materials science and engineering system, component or process to meet desired needs within realistic economic, environmental, social, political, ethical, health and safety, manufacturability and sustainability constraints.Communicate technical data and design information effectively in speech and in writing to other materials engineers

MSE: Connecting Goals to Outcomes

Goal

SLO

Learning Goals: Understand materials systems and their role in engineeringAbility to apply knowledge of mathematics, science and engineering principles to materials science and engineering.

Student Learning Outcomes: Apply knowledge of mathematics, science and engineering principles to materials science and engineering. Design and conduct materials science and engineering experiments and analyze and interpret the data

MSE: Connecting Goals to Outcomes

Level 1

Level 2

SLOs            Additional Assess-ments

Content Knowledge

EMA3050

EMA3066 EMA4714EMA3080

CEMA3513

CEMA4714  

#1 I R A      Senior exit

survey

#2       I R ASenior exit

surveyCritical

ThinkingEMA306

6EMA4223 EMA4714        

#3 I R A      Senior exit

surveyCommunicati

onEMA308

0CEMA3013

CEMA3513

C       

#4 I R A      Senior exit

survey

Connecting Program SLOs to Courses MSE Curriculum Map

Assessments in the boxes marked A are conducted using specific homework, exam, or assignment questions aligned with that SLO.Source: 2011-12 MSE Academic Assessment Plan

Completing the System

Align course objectives with program SLOs

Develop assessments of the course objectives

Select course assessments as assessments of the program SLOs – vet these with the program faculty

Completing the System

Collect data

Analyze and interpret

Modify

Report to Compliance Assist!

A synopsis…

Program Description

Learning Goals

Program SLOs

Course objectives

and assessment

s

Data collection, analysis,

and interpretati

on

Annual Reporting

and Planning

All of this leads to

SUCCES

S!

KEEP UP THE GOOD WORK