Two Levels of Course Assessment to Identify Student Learning

Carol Lerch, Ph.Dlerch@dwc.edu

S 52 Friday 11:45 – 12:35

AMATYC 2007 2

Background

• Assess new format – College Algebra– Introduced fall 2000– Previous course failure rate +40%

• Assess new format – College Algebra– Introduced fall 2000– Previous course failure rate +40%

• Looking at students in one specific major

• Assess new format – College Algebra– Introduced fall 2000– Previous course failure rate +40%

• Looking at students in one specific major• Course description– Purpose of this course is to provide students with

a proficiency in algebraic skills necessary for success in future coursework …

AMATYC 2007 3

Collect Data

• Specifics Fall 2001 – Summer 2007– Pre-2000 60% pass

(anecdotal)

• Fall average 85%• Spring average 64%• Summer 80%• Includes Withdraws

f01s0

2 f02s0

3 f03s0

4

sum04 f04

s05

sum05

f05 * s06

f06*s0

7

sum07

0%10%20%30%40%50%60%70%80%90%

100%

MA 122 Pass Rates

AMATYC 2007 4

Analysis

• Improvement over previous College Algebra course

• Improvement over previous College Algebra course

• Fall pass rates acceptable– Continuing to improve

• Improvement over previous College Algebra course

• Fall pass rates acceptable– Continuing to improve

• Spring pass rates much lower– Needs analysis & new plan

• Improvement over previous College Algebra course

• Fall pass rates acceptable– Continuing to improve

• Spring pass rates much lower– Needs analysis & new plan

• Summer needs more data– Online first time summer 07

AMATYC 2007 5

Changes to Course

• Added text related course management system– online homework– Interactive textbook– Instant help– Video, animation

• Added text related course management system– online homework– Interactive textbook– Instant help– Video, animation

• Added section JITTs– Just-in-time-tests

AMATYC 2007 6

Purpose of AssessmentPlan

• Pass rates– Pass rates for particular course

• Pass rates– Pass rates for particular course

• Course sequence– Semesters to complete math sequence• Thanks to Jack Rotman

• Pass rates– Pass rates for particular course

• Course sequence– Semesters to complete math sequence• Thanks to Jack Rotman

• Concept assessment– Learning goals/objectives

AMATYC 2007 7

Semesters to Complete

• Semesters to pass through the math sequence, passing MA152 for each semester since fall 03:– 82% of those who started in MA098 passed

MA152 in 4 semesters (4 courses)– 93% of those who started in MA122 passed

MA152 in 3 semesters (3 courses)

AMATYC 2007 8

Levels of Assessment• Gathering information about students’

achievement (Marzano, 2001, p. 115)• Level I Assessment– General Course information• Pass rates• Success in subsequent math courses• Semesters to complete sequence

AMATYC 2007 9

Levels of Assessment• Level II Assessment– Content specific– Concepts

• Learning Mapo Summative assessment

Overall course content Final exam

• Item Analysiso Formative assessmento Individual concepts tested

AMATYC 2007 10

Cycle of Assessment

Plan

Collect DataAnalyze

AMATYC 2007 11

• Online homework required as of spring 2006– Introduced as option for fall 2005

• Daily quiz– 2-3 questions on concepts in each section• Read text• Come to class prepared• Carrot – replace test grade with quiz average

Plan

AMATYC 2007 12

Collect Data

• Fall pass rate improved overall– 90%

• Spring stayed same– 63%

f05 * s06 f06* s07 sum070%

10%20%30%40%50%60%70%80%90%

100%

MA122 Pass Rates - new text

Series1

AMATYC 2007 13

Analysis

• Fall pass rates acceptable– Online homework helped• Anecdotal evidence

– Students like all options available with online homework– Have more opportunities to learn

• Spring rates need improvement

AMATYC 2007 14

Plan

• Assesses student learning– Concept assessment• Learning goals/objectives

• Develop “Learning Map”– Identify major concepts• Identify teaching objectives• Identify student learning objectives• Means of assessment• Set success standard

Plan

AMATYC 2007 15

Learning Map

AMATYC 2007 16

Learning Map

• 1st step– Major topics of course• Linear equations, quadratic equations, inequalities,

absolute value, exponential & log equations• Modified as further developed

• 1st step– Major topics of course• Linear equations, quadratic equations, inequalities,

absolute value, exponential & log equations• Modified as further developed

• 2nd step– What do I want them to learn• Teaching objectives

AMATYC 2007 17

Learning Map

• Knowledge level• Cognitive

Process– Action verbs

AMATYC 2007 18

Levels of Knowledge

• Factual– Basic elements of the discipline– Facts specific to topic• Parts of linear or quadratic equation• Methods to solve quadratic equations

• Factual– Basic elements of the discipline– Facts specific to topic• Parts of linear or quadratic equation• Methods to solve quadratic equations

• Procedural– Processes of mathematics– Algorithms, techniques, methods• Complete the square• Quadratic formula

AMATYC 2007 19

Levels of Knowledge• Conceptual– Interrelationships– Classifications– Principles, generalizations• Pythagorean Theorem• Properties

• Conceptual– Interrelationships– Classifications– Principles, generalizations• Pythagorean Theorem• Properties

• Metacognitive– Self knowledge– Students need to reflect to learn

AMATYC 2007 20

Teaching vs. Learning Objectives• Teaching objective– What I will do– What I want the

students to know

• Student learning experience• What they will do

• Student learning objective– “Students will be able to

…”

AMATYC 2007 21

Assessment on Map

• Each objective has generic question

• Indicates how student is assessed

AMATYC 2007 22

Learning Map

• Gave to students as part of syllabus– Type of questions included– ‘Truth in advertizing’

• Matched final exam to map– Redesigned map• Concepts tested but not part of map

– Redesigned final exam• Some concepts not tested but important• Some concepts tested multiple times

AMATYC 2007 23

Level II Assessment

• Assess concepts learned• Allow overall picture of student progress• Shows strengths• Identifies weaknesses– Areas for further development

AMATYC 2007 24

Collect Data

• Collected chapter test data– Right/wrong only

• Collected final exam item results• 4th semester of data collection

• Stats: Percent correct

AMATYC 2007 25

Collect Data• Identified concept tested by semester– Shows what is not tested (over-tested)

Results of chapter tests for Fall 06, Spring 07, Summer 07

AMATYC 2007 26

Analysis - Tests

• More Student Learning Objectives– Needed to add very specific objectives– Dropped some that I thought I wanted but don’t

test for

• More Student Learning Objectives– Needed to add very specific objectives– Dropped some that I thought I wanted but don’t

test for• Success measure – 65% or better correct

• More Student Learning Objectives– Needed to add very specific objectives– Dropped some that I thought I wanted but don’t

test for• Success measure – 65% or better correct• Results below need attention

• More Student Learning Objectives– Needed to add very specific objectives– Dropped some that I thought I wanted but don’t

test for• Success measure – 65% or better correct• Results below need attention• 0 - 10% indicates problem with question as

well as topic

AMATYC 2007 27

Analysis – Final exam• Final used by all sections

– Data just my sections

• Shows problems with online course on specific concepts

AMATYC 2007 28

Asides

• Only list what is being assessed– If can’t assess, don’t put as objective

• Have a thick skin• Data show changes made in course– Provides longitudinal look at course

• Allows comparison of student results by semester or year

• Data show where more emphasis needed in teaching

AMATYC 2007 29

Before Beginning Assessment

• Purpose of assessment– What do you want to know?– Why do assessment?

• Action leading from assessment– What do you do with results?

AMATYC 2007 30

5 Basic Questions1. Which objectives to assess?– Student learning– What should they know?

2. How to assess?– Multiple layers

3. What are results?– Collect/collate data

4. What is ‘success’?5. What to do about results?– Next steps

AMATYC 2007 31

References• Angelo, T. A. (1991). Introduction and overview: From classroom assessment to classroom

research. Classroom research: Early lessons from success. T. A. Angelo and K. P. Cross. San Francisco, Jossey-Bass: 7-15.

• Brockbank, A. and I. McGill (1998). Facilitating reflective learning in higher education. Philadelphia, Society for Research into Higher Education & Open University Press.

• Carson, T. (1990). "What kind of knowing is critical action research?" Theory Into Practice XXIX(3): 167-173.

• Cochran-Smith, M. and S. L. Lytle (1993). Inside outside: Teacher research and knowledge. New York, Teachers College.

• Elliott, J. (1991). Action research for educational change. Philadelphia, PA, Open University Press.

• Leibowitz, M. (1997). Instruction for process learning. Supporting the spirit of learning: When process is content. A. Costa and R. M. Liebmann. Thousand Oaks, CA, Corwin Press, Inc.: 47-54.

• Marzano, R. J. (2001). Designing a new taxonomy of educational objectives. Thousand Oaks, CA, Corwin Press.

• Mezirow, J. (1990). Fostering critical reflection in adulthood: A guide to transformative and emancipatory learning. San Francisco, Jossey-Bass.

• Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA, Harvard University.

AMATYC 2007 32

Participation

• Gather into groups• Pick one course you

teach• List the major concepts– Very few topics

• Pick one or two to use• Identify Teaching

Objective

• Identify Student Objective

• Identify knowledge levels– Can be more then one

• Identify possible means of assessment

• Identify success measures

AMATYC 2007 33

SHARE

Two Levels of Course Assessment to Identify Student Learning

Carol Lerch, Ph.Dlerch@dwc.edu

S 52 Friday 11:45 – 12:35