Writing Effective Course Learning Outcomes€¦Writing Effective Course Learning Outcomes Jeffrey E. Froyd Research Professor Texas A&M University [email protected]

Writing Effective Course Learning Outcomes

Jeffrey E. Froyd Research Professor

Texas A&M University [email protected]

mailto:[email protected]

Workshop Ground Rules

• Purpose: Participant Learning

– The purpose is not content coverage

• Questions

– Please ask whenever you have a question

• PowerPoint Presentation

– A copy of the presentation will be made available to Dr. Kamel at the conclusion of the workshop

– Please contact Dr. Kamel for a copy

Acknowledgements

• This workshop has been given and revised many times and many people have contributed to its development.

–Prudence Merton, Dartmouth College

–Debra Fowler, Texas A&M University

–Nancy Simpson, Texas A&M University

– Edamana Prasad, Indian Institute of Technology Madras

PowerPoint vs. Blackboard

• Which is a better way to teach?

– Writing class notes on a blackboard

– Showing class notes via PowerPoint

Fork vs. Spoon

• Which is a better way to eat?

– With a fork

– With a spoon

Course Delivery Cycle

What will the students be able to do and

how will the students think when they

complete my course?

What evidence will the students

and instructors have of success?

How will I conduct class

to assist students

in their learning/success?

Are the students meeting

the learning outcomes/succeeding?

What is working/not working

in my course?

What changes will be

incorporated in the

next course offering?

• Homework • Exams • Portfolios • Presentations • Written reports • Course survey data

Reflection/ Documentation Course Portfolio

Use Think/Pair/Share Do Demonstration Write Reflections Conduct Lectures Model thinking

Course Learning Outcomes

Prior Knowledge?

Your Expectations

Center for Teaching Excellence at Texas A&M University

Workshop Series

Workshop No. 1: Writing Effective Course Learning Outcomes

Workshop No. 2: Preparing a Formative Course Assessment Plan

Workshop No. 3: Applying Research-based Instructional Strategies

Workshop No. 4: Course Design

Exercise: Write a learning outcome

• Think of a course you will be teaching next semester

• Write a learning outcome for this course. –Please write out your learning

outcome on paper

Overview

• Part 1: Writing effective course learning outcomes

– Purpose of learning outcomes

– Challenges in learning outcomes

• Reliance on wanting students to understand

• Lack of technology/terminology

• Part 2: Effectively using course learning outcomes

– Purpose of learning outcomes

– Engaging students

– Higher level course learning outcomes

Purpose of Learning Outcomes

• Learning outcomes should answer the following questions:

– “What should students know and be able to do as a consequence of instruction?”

– “How will students be able to think?” or “At what cognitive level will the students perform?”

Learning Outcome Guidelines

• Outcome must:

– Contain a verb describing an observable action

– Focus on the student as the performer

• What is a student expected to be able to do?

• How is a student expected to be able to think?



• Write a learning outcome for this course. –Please write your learning outcome

on paper

Learning Outcome Guidelines

• Outcome must:

– Contain a verb describing an observable action

– Focus on the student as the performer

• What is a student expected to be able to do?

• How is a student expected to be able to think?

Learning Outcomes vs. Syllabus

Syllabus Learning Outcomes

Describes topics/content to be delivered/covered

Describes expected learning in observable terms

Noun-oriented Verb and noun-oriented

Instruction-focused Learning-focused

Lists content Describes what students will be expected to do with the content

Teacher-focused Student-focused

Learning Outcomes: Number One Challenge

• Faculty members when they initially write learning outcome, want to use verbs such as

– Understand or “demonstrate understanding”

– Know or “demonstrate knowledge”

– Appreciate or “demonstrate appreciation”

– …


• Guideline: Illegal Verbs – Do Not Use When Writing Learning Outcomes

– Understand

– Demonstrate understanding

– Know

– Demonstrate knowledge

– Appreciate

– Demonstrate appreciation

– …


• Write out a question about this guideline

• Share your question with a colleague and craft one improved question by combining questions

• Questions

Exercise: Learning Outcomes

• What does it mean to “understand”?

– Think about a room with 40 people. Half of the people understand what you want them to understand, half do not.

–What activities will you ask them to do to distinguish the two groups of people?

– You can then use these activities to write your learning outcomes.


What do we want our students to be able to do with the [BLANK] knowledge that they acquire? We certainly expect them to be able to predict the responses of a [BLANK] system if it is disturbed. We expect students to be able to explain the responses that occur in systems that have been disturbed. Sometimes we want them to solve quantitative problems (calculate something). And, we expect them to be able to do this with systems and disturbances that they have not encountered in lecture or the textbook. That is, we expect them to be able to apply what they know about [BLANK] to novel situations. When they can do this, we say they “understand” [BLANK]. Or, in the terminology I have been using, we can say that meaningful learning has occurred.

Michael, J. (2001). In Pursuit of Meaningful Learning. Advances in Physiology Education, 25, 145–158

• What does a learner do to improve “understanding”?

– If you were learning a new subject, what would you do to understand the subject better?

– You can then use these activities to improve the learning outcome.


• What does a teacher do to improve “understanding”?

– If you were teaching a new subject, what would you do to help you students understand the subject better?

– You can then use these activities to improve the learning outcome.


• What is the number 2 challenge?

–Technology (or terminology) is needed to describe and address the number 2 challenge.

–The technology is called taxonomies of learning outcomes.

Learning Outcomes: Number Two Challenge

Bloom’s Taxonomy

• History

– 1950s- developed by Benjamin Bloom and others

– 2001 revised by another group

• Language to express qualitatively different kinds of thinking

• Adapted for classroom use as a planning tool

• Continues to be one of the most universally applied models

(Pohl, 2000, Learning to Think, Thinking to Learn, pp. 7-8)

Levels in Bloom’s Revised Taxonomy

• Creating

• Evaluating

• Analyzing

• Applying

• Understanding

• Remembering

Verbs from Levels in Bloom’s Taxonomy

• Creating – Construct, design, generate, plan, produce

• Evaluating – Critique, justify, conclude, judge, hypothesize

• Analyzing – Correlate, diagram, distinguish, deconstruct

• Applying – Determine, develop, compute, utilize, implement

• Understanding – Classify, explain, discuss, give example, summarize

• Remembering – Define, describe, list, reproduce, identify

BLOOM’S REVISED TAXONOMY Creating

Generating new ideas, products, or ways of viewing things Designing, constructing, planning, producing, inventing.

Evaluating

Justifying a decision or course of action Checking, hypothesising, critiquing, experimenting, judging

Analysing

Breaking information into parts to explore understandings and relationships Comparing, organising, deconstructing, interrogating, finding

Applying

Using information in another familiar situation Implementing, carrying out, using, executing

Understanding

Explaining ideas or concepts Interpreting, summarising, paraphrasing, classifying, explaining

Remembering

Recalling information Recognising, listing, describing, retrieving, naming, finding

Levels in Bloom’s Revised Taxonomy

• Creating

• Evaluating

• Analyzing

• Applying

• Understanding

• Remembering

Notes • Levels in Bloom’s

Revised Taxonomy are not a developmental model

Exercise: Bloom’s Revised Taxonomy

• Review the learning outcome you wrote earlier

• At which level in Bloom’s Revised Taxonomy would you place your learning outcome?

• Is that the level at which you want your students to be performing?

Revised Bloom’s Taxonomy - Cognitive Domain

R

e

m

e

m

b

e

r

U

n

d

e

r

s

t

a

n

d

A

p

p

l

y

A

n

a

l

y

z

e

E

v

a

l

u

a

t

e

C

r

e

a

t

e

Factual

Conceptual

Procedural

Metacognitive

http://www.learningsolutionsmag.com/assets/images/learningsolutions/2013/130214/model.png

http://www.learningsolutionsmag.com/assets/images/learningsolutions/2013/130214/model.png

Exercise: Bloom’s Revised Taxonomy

• Think of a course you have taught.

• Think of an exam problem you have given in this course

–Which cells in the Revised Bloom’s taxonomy did the exam problem address?

• What is the number 2 challenge?

–Most people who initially try to write learning outcomes write lower level learning outcomes and are unhappy with the level of learning described by their learning outcomes.

– They do not have or apply the terminology for expressing higher order learning

Learning Outcomes: Number Two Challenge



• Write several learning outcomes for this course.

Minute Paper

• Write brief answers to the following questions: – What is most valuable or

helpful about writing a learning outcomes?

– What is the “muddiest or most confusing point” about writing a learning outcomes?

How well does using minute papers in courses work?

• Findings: “This result suggested, as we hypothesized, that the use of the one-minute paper improves student performance. Its coefficient implied that the use of the one-minute paper increased student performance by approximately .5 of a point on the postTUCE [Test of Understanding of College Economics] exam, ceteris paribus.”

• Findings: “This evidence suggests that the benefit to students from using the one-minute paper does not depend on the instructor who implements it.”

• Findings: “This evidence supported our initial hypothesis that the benefit to students from using the one-minute paper does not depend on their ability level.”

• Assertion: “When asked by college teachers to identify the single pedagogical innovation that would most improve their teaching, Light (1990, 35) always responds with the one-minute paper, an idea that ‘swamped all others.’”

Chizmar, J. F., and Ostrosky, A. L. (1998). The One-Minute Paper: Some Empirical Findings. The Journal of Economic Education, 29(1), 3–10

How does using minute papers in courses work?

• Findings: Overall results indicate that performance on subsequent essay quizzes was significantly higher by students who wrote one-minute papers than performance by students who did not write the papers.

• Findings: Of particular interest to instructors was that the increase in quiz scores when one-minute papers were not graded was significantly higher than when the one-minute papers were graded.

Almer, E. D., Jones, K., and Moeckel, C. L. (1998). The impact of one-minute papers on learning in an introductory accounting course. Issues in Accounting Education, 13(3), 485–495

Minute Paper: Resources

• Chizmar, J. F., and Ostrosky, A. L. (1998). The One-Minute Paper: Some Empirical Findings. The Journal of Economic Education, 29(1), 3–10.

• Almer, E. D., Jones, K., and Moeckel, C. L. (1998). The impact of one-minute papers on learning in an introductory accounting course. Issues in Accounting Education, 13(3), 485–495.

• Stead, D. R. (2005). A review of the one-minute paper. Active Learning in Higher Education, 6(2), 118–131. doi:10.1177/1469787405054237.

Purpose

• What are purposes for learning outcomes?

– For yourself

Take two minutes to

write down your

thoughts and then

share with a colleague


• Know where we are and what we have achieved • Chose teaching approach likely to achieve outcome • Find strengths and opportunities for improvement in

learning approach • Guides selection of teaching approach and instructional

materials • Course design, tailor course design to learning outcomes • Ensure students learn skills intended • Verify achievement and identify opportunities for

improvement • Guide what to teach and what to assess • Satisfy accreditation requirements

Purpose

• What is the purpose of learning outcomes for faculty?

– Design assignments

– Help design course structure

– Provide framework for course and exams

– Judge whether my intent was fulfilled

– Provide consistency across multiple sections and years

– Self organization

– Create a measurement system

– Help pick a video

– Set a goal

– Schedule presentations

Purpose of Learning Outcomes: Faculty

• Outcomes help instructors

–Guide preparation of classroom material

–Keep the focus on specific end results

–Create homework

assignments

–Aid in test design

Purpose

• What are purposes for learning outcomes?

– For your students

Take two minutes to

write down your

thoughts and then

share with a colleague


• Know more about the course and the major

• Follow progression of the course

• Self-assessment

• Select study strategies

• Apply what they have learned

• Predict their achievement

• Study to acquire each outcome

Purpose

• What is the purpose of learning outcomes for students?

– Clear expectations

– Understand course intent

– Can prioritize

– Help students decide whether to take the course

– Better prepared and motivated

– How to study

– Plan study

– Help students prepare for lecture

Purpose

• What is the purpose of learning outcomes for students?

– Provide clear goals for studying

– Provides a big picture of the course

– May increase intrinsic motivation

– Focus, optimize time

– Students fit this course within other courses

– Identify opportunities to improve

– Know about what will be tested and how to prepare

– Focus on outcomes, not grades

Purpose of Learning Outcomes: Students

• Outcomes help students

–Clarify their personal course goals

–Provide framework for measuring their success

–Reduce their anxiety

– Improve their studying effectiveness

Source - http://ublib.buffalo.edu/libraries/projects/tlr/importance.html

Inputs for composing course learning outcomes

• Examinations

• University learning outcomes

• Program learning outcomes

• Peers teaching courses in sequence (pre or post to your course)

• Accreditation Criteria

• Bloom’s Taxonomy

(Original and Revised)

Minute Paper

• Write brief answers to the following questions: – What is most valuable or

helpful about writing a learning outcomes?

– What is the “muddiest or most confusing point” about writing a learning outcomes?

BREAK

Using Course Learning Outcomes Effectively

Jeffrey E. Froyd Research Professor

Texas A&M University [email protected]

mailto:[email protected]

CHRIS SCHWARTZ, MECHANICAL ENGINEERING, IOWA STATE UNIVERSITY

An example

Learning Outcome Level 1

Calculate/Identify

Level 2

Apply/Analyze

Level 3

Evaluate/Design

1. Functional Decomposition

2. Material Transitions

Transition modes: (i) yielding, (ii) fracture, (iii) deformation, (iv)

buckling

•Concepts of failure

•Factor of safety

•Strength

3. Stress

Normal stress: engineering vs. true

•Shear stress

•Stress concentration

4. Strain

Normal strain: engineering vs. true

Shear strain

5. Stress vs. strain behavior

•Elasticity

•Plasticity

•Viscoelasticity

•Thermoelastic behavior

6. Multiaxial loading behavior

•Principal stress

•Mohr’s Circle

•Principal strain

7. Specific geometry behavior

Beams

bending

shear

torsion

Thin wall objects

Connecting Learning Outcomes to Course Grades • Dr. Christian Schwartz (Iowa State University) developed the

following approach to evaluating student learning that connects course learning outcomes to grades.

• You will be assigned a letter grade based upon the number and level of LO’s that you attain in the LO matrix (next slide). Letter grades will be assigned as follows:

– A: At least 4 level-3 outcomes attained

– B: At least all level-2 outcomes attained

– C: At least 4 level-2 outcomes attained

– D: At least all level-1 outcomes attained

– F: Less than 4 level-1 outcomes attained

Exercise: Using learning outcome effectively

• Assume you have developed an excellent set of learning outcomes for a course you will be teaching

• Generate ideas for using this set of learning outcomes effectively

–Think – Pair - Share

Exercise: Using learning outcome effectively

• ??

List of Ideas for Using Learning Outcomes Effectively

• Share learning outcomes with your students

• Use learning outcomes in preparing lectures

• What is an appropriate number of learning outcomes?

• Use learning outcomes in composing exams

• Student development with respect to applying, analyzing, evaluating, creating

• Throw away topic list

Student Resistance

“If you have tried anything innovative in class, you know that students hate

not knowing the rules.”

Felder, R. M. (1987). On creating creative engineers. Engineering Education, 77(4), 222-227.

Share learning outcomes with your students

• Share Bloom’s Taxonomy with your students

• Share specific learning outcomes for each lecture

– Help them see how to use learning outcomes to prepare for lecture

• Share specific learning outcomes for each upcoming exam

– Help them see how to use learning outcomes to prepare for lecture

• Provide students feedback on progress with respect to learning outcomes

Exercise: Engaging your students with learning outcomes

• Sketch a plan for engaging your students with the learning outcomes that you prepare for a course

–Think – Pair – Share

Use learning outcomes in preparing lectures

• Build a schedule for when learning outcomes will be

addressed in the academic term.

• Start each lecture with the course learning outcomes

to be address in the lecture

• Select one or more learning outcomes for a lecture

– What do students need to know for achieving the learning

outcome?

– What can students easily learn on their own?

– Where do students most need your input and feedback?

One Organization Scheme

Lecture 1 Lecture 2 … Exam 1 Exam 2

LO #1

LO #2

LO #3

LO #4

…

Student development with respect to applying, analyzing, evaluating, creating

• Think of a subject you will teach next semester

• Write a problem at one of the following levels: applying, analyzing, evaluating, creating

• …


• Assign in-class problems and homework problems

• Give students opportunities to practice

• Peer feedback

• Rubrics


• Example: Pick a set of homework problems in a textbook. Ask students to develop a script/flowchart/… for solving the set of homework problems.


• Example: Design and build two bridges. Grade on the design that performs worse.

Cowan, J. (2006). On Becoming an Innovative University Teacher: Reflection in Action (2nd ed.). Maidenhead, UK: Open University Press.

Use learning outcomes in preparing exams

• Write down the learning outcomes that will be evaluated with the exam

• Share the list of learning outcomes with the students a week or so in advance

• Generate multiple potential problems for each learning outcome

• Limit the number of higher-order learning outcomes per exam

• Select from the potential problems the final problems

Use learning outcomes in preparing exams

• Should you use problems that evaluate

multiple learning outcomes?

– Physics chapter example

Minute Paper

• If I were doing another session on learning outcomes, what should I:

–Continue to do

–Start doing

–Stop doing

References

• Anderson, L. W., Krathwohl, D. R., & Bloom, B. S. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives. Allyn & Bacon.

• Barr, R. B., & Tagg, J. (1995). From teaching to learning: A new paradigm for undergraduate education. Change, 27(6), 12-25. doi: 10.1080/00091383.1995.10544672

• Felder, R. M., & Brent, R. (2003). Designing and teaching courses to satisfy the ABET engineering criteria. Journal of Engineering Education, 92(1), 7-25. doi: 10.1002/j.2168-9830.2003.tb00734.x

• Felder, R. M., & Brent, R. (1997). Objectively speaking. Chemical Engineering Education, 31(3), 178-179.

• Felder, R. M., Woods, D. R., Stice, J. E., & Rugarcia, A. (2000). The future of engineering education II. Teaching methods that work. Chemical Engineering Education, 34(1), 26-39.

Documents

Writing Effective Course Learning Outcomes€¦Writing Effective Course Learning Outcomes Jeffrey E. Froyd Research Professor Texas A&M University [email protected]