Presented at North Country Community College29 September 2011

Michael A. HeelCoordinator of Academic Assessment and Program Review

Monroe Community College

SUNY Initiative was a mixed blessing◦ New York out in front of many other states◦ SUNY schools better positioned than independents◦ Emphasized numbers (science) over evaluation (art)

Middle States (MSCHE) “upping the ante” The Feds are coming (parents, too)! As budgets tighten, “assessment for

accountability” becomes an even bigger deal…

2

◦ A) SUNY, Middle States, and other specialized accrediting bodies (ABET, AACSB, etc.) require it;

◦ B) other external stakeholders (like donors, trustees, grant applications) want us to;

◦ C) the federal government wants institutions of higher education to be more mindful of how college students are educated;

◦ D) but most importantly, we want to avoid…..THIS:

3

4

5

Legitimate case to be made for a mechanism in education that promotes some kind of quality standards◦ Think of all the goods and services that you would not

buy without a system of quality control in place Most college instructors are untrained as

educators (we are specialists and experts in our respective content fields, but few of us hold dedicated education degrees)◦ Assessment processes provide a vehicle for focusing

attention on teaching and learning Better we “police ourselves” than have to

submit to the alternative…

6

Ideally, assessment processes are NOT all about the numbers and statistics, but about the content and context of the educational product (learning)

Assessment can help department faculties organize systematically some of the business they are already conducting, and provide a sensible framework for those endeavors

Even accrediting bodies are now focusing on the “back end” of assessment, rather than the “front end…”

7

Assessment processes are not about the faculty (assessment results cannot reasonably be used for hiring, promotion or tenure decisions)

Assessment is not about the numbers (accrediting bodies generally don’t care about the statistics; they want to know how we are using that information to make strategic choices)

Assessment should not be the top priority of a faculty (“What are faculty members not doing while they are engaged in excessive assessment activities?” – Suskie)

MSCHE Principles: Assessment should be…UsefulCost-effectiveReasonably accurate and truthfulPlanned Organized, systematized, and sustained

The process and product of academic assessment will differ depending upon how the results will be usedHigh Stakes vs. Low StakesInternal vs. External

8

9

Connect what we do in the classroom with what’s going on the real world

Demonstrate that students are learning what we intend for them to learn

Understand whether or not the courses we are teaching substantively support the curriculum we have designed

Pause and consider whether the education we are providing supports the needs of our constituencies

Busy program and department faculties run the risk of operating their courses and programs on “auto-pilot.”◦ “It ain’t broke!”◦ “The assessment results show we’re doing

okay…”◦ “I’m doing assessment, I’m doing program review,

what more do you want from me?” Even the most invested faculty members

may seem reluctant to add (what are perceived to be) extra steps in the assessment and program review process

10

11

12

No one likes doing extra work BUT, what if that work, once undertaken,

winds up being valuable, and offers up results that are immediate and worth the time spent?

The first-time investment requires the greatest effort

The value of doing academic assessment is not always apparent – curriculum alignment often clarifies that value (remember – “useful” and “truthful”)

13

The language of assessment is still not uniform Differentiating “goals” from “objectives” and

“outcomes” can be difficult and irritating What about “mission” versus “vision,” “values,”

or “principles?” Find a sensible institutional language standard,

and develop your own internally consistent approach

Explanation provided in accompanying packet of information

14

Some assumptions prior to embarking upon curriculum alignment exercises –◦ Program Mission may or may not be present (but

it’s better to have one)◦ Program Goals are present (or “vision” or

“values”)◦ Program Objectives/Outcomes have to be present

and be well-developed◦ Course Learning Objectives/Outcomes are present◦ A system to measure CLOs is in place

If the above items are not all evident, then (the good news is that) by engaging in a process of curriculum alignment, all will be present and in place at the end

15

Is the Mission of the program supported by Program Goals?

Assuming that some Program Objectives are oriented toward learning in the classroom, and others are not…Are the Program Learning Objectives/Outcomes

supported by Course Learning Outcomes?(Are the Program’s Non-Learning Objectives supported

by other activities?) Are Course Learning Outcomes supported by

student activities such that student learning can be measured?

16

Review your department/program mission, goals, and objectives/outcomes

If a mission or goals set is missing, consider creating it

The “review” will work best if:◦ Faculty sees value in the process◦ Faculty is ready to do things differently◦ Faculty is willing to “reinvent” what they do

One successful approach – “Pretend that you are creating the degree program from scratch, and you are not limited by current practices…”

Finalize this work before moving on…

17

Again, your faculty may be skeptical that this process is even necessary

Let them know that the first time is the most challenging

While undertaking the process is meant to help show gaps, weaknesses, and redundancy, it can also serve to confirm successes in curriculum design as well!

18

Distribute a numbered/lettered list of Program Learning Objectives among the relevant faculty members

Challenge each faculty member:◦ For each class he/she teaches in the program,

determine which of the Program Learning Objectives is covered in the class as a MAJOR part of the course content; indicate also MINOR coverage (separately)

◦ This exercise benefits from input from multiple faculty members who teach different sections of the same course, but who respond independently

Collect and display the information in grid format

19

20

21

A graduate of the A.A.S. degree program in Apicultural Science will be able to:a. Construct and maintain beehives of appropriate design relative to the beekeeper’s purpose

(breeding, honey production, supporting agriculture, etc.) and the breed of bee;b. Create appropriate breeding environments and apply professionally-recommended

techniques in promoting healthy fertility and growth of bee populations;c. Apply and interpret basic genetic tests of bee samples;d. Conduct appropriate hybridization techniques;e. Identify, recognize, and respond to insect behaviors, including those of bee, competing

insect, and predator insect species;f. Promote health and wellness among bee populations;g. Cultivate plant species supportive of the healthy maintenance of bee populations;h. Recognize and manage responses to threats to bee populations, such as changes in the

seasons, extreme weather conditions, and the presence of pesticides and other toxins in the environment;

i. Apply basic business management principles to the management of the costs and expenses of beekeeping;

j. Demonstrate an understanding of the ethical principles underpinning beekeeping as both a hobby and a profession;

k. Demonstrate an understanding of the role of bee species in the maintenance of a healthy and sustainable agricultural business;

l. Teach and train apprentice beekeepers in the rudimentary aspects of the practice of beekeeping;

m. Capably communicate, orally and in writing, the basic tenets of good beekeeping practices to individuals unfamiliar with bees and the practice of beekeeping.

22

PLOAPS 101

ENV 111

APS 112

ENV 172

APS 122

ENV 231

APS 201

BIO 205

APS 244

APS 260

APS 252

a         M       m M M

b         m   M m m M M

c               m     m

d         M     m m   m

e     M     m        

f m   m     m M m m m M

g           m M   M m  

h m m m M m m M     M  

i m       M   m   m M M

j m   M   m            

k m         M m     m  

l                     m

m                      

23

Program Learning Outcome “m” is not covered◦ Faculty discussion revealed that virtually every

faculty member thought others were covering the outcome in their respective courses

◦ Most faculty members did not feel qualified to teach or grade students on their communication skills

ENV 111 and APS 112 seemed of little value to the curriculum

PLO’s “c,” “e,” and “l” are under-covered

24

The curriculum mapping exercise ALSO picked up some areas where there were redundancies, and where courses seemed over-packed

Outcomes “f” and “h” were covered in almost every course, and Outcomes “b” and “g” seemed also to be overly emphasized given their relative importance to the whole curriculum

Four courses (APS 122, APS 201, APS 260 and APS 252) seemed too densely packed with redundant content

25

The program faculty is now both empowered and challenged to “fix” these curriculum issues

Benefit of this simple procedure is that the pictorial representation gives participants a quick and easy understanding of an otherwise complex system

26

27

Outcome “m” now covered in an APS course each semester of a student’s plan of study

Outcomes “c,” “d,” and “e” were dependent upon BIO 205, so courses scheduled in the second year had to address these outcomes

Significant course redesigns for APS 112, APS 244, and APS 252 to accommodate and address both gaps and redundancies in curriculum

ENV 112 retained, to fulfill Natural Science requirement, and because it is a pre-req for ENV 172

Moving forward, the faculty must now document student learning

28

Some courses are stronger in some areas of interest than others

Although multiple courses may cover the same outcome, not every course provides good assessment opportunities

Some courses offer breadth over depth Some PLO’s can only be covered (and

measured) in specific courses

29

PLOAPS 101

ENV 111

APS 112

ENV 172

APS 122

ENV 231

APS 201

BIO 205

APS 244

APS 260

APS 252

a     m   M       M m

b         m   M m m M M

c               m M   M

d         M   M m M   m

e     M     m   M    

f m   m     m M m m M

g           m M   M m  

h m m M M m m M      

i m       M   m   m M M

j m   M   m           M

k m   m     M m     m  

l                 M M m

m M   M       M       M30

The faculty has to determine which Course Learning Objectives/Outcomes best suit the matching PLO

The assumption is that individual courses are taught according to their respective CLOs

If CLOs are supported by graded (and thus, embedded) student assignments, then that documentation will directly demonstrate support for the PLO (MSCHE wants direct evidence where possible)

Common sense rules for assessment still apply (as do the broad MSCHE assessment principles)

31

PLOAPS 101

ENV 111

APS 112

ENV 172

APS 122

ENV 231

APS 201

BIO 205

APS 244

APS 260

APS 252

a       clo #2       clo #1

b         clo #6   clo #1

c               clo #5   clo #4

d         clo #4   clo #3 clo #4  

e           clo #1    

f       clo #1 clo #2

g           clo #8   clo #2  

h clo #3 clo #2      

i       clo #5     clo #3

j   clo #1             clo #7

k clo #3   clo #4         clo #2  

l                 clo #6 clo #4

m clo #7   clo #9             clo #1132

Note that all APS courses are participating in the assessment of at least one Course Learning Outcome

All PLO’s are covered by at least one assessed course

Remember that MSCHE cares only that the Department faculty demonstrate that PLO’s have been assessed; they aren’t concerned about the minutiae

This process allows the faculty to show that, by fulfilling linked CLOs, they are likewise showing that PLOs have been measured and analyzed

33

As with many assessment activities, once the faculty moves beyond the “worker bee” focus (pardon the pun), the opportunity to consider the bigger picture is usually gratifying and engaging

Relationship and roles of individual courses within a curriculum are greatly clarified

The benefit extends also to other stakeholders, who can now be reassured that a program curriculum “delivers” on what is advertized

This particular exercise can help build faculty consensus, teamwork, and identity

34

Consider the following application of curriculum mapping (based on an actual experience; the discipline has been changed, and the institution remains anonymous):

Department of Religion at a 4-year institution offers a major, but serves far more students who take courses to satisfy the institution’s humanities requirement◦ What should that faculty’s approach be to

assessment?◦ How can the process of curriculum mapping help

the faculty to figure this out? 35

The Mission of the Religious Studies department is to provide students with a complete educational experience. Students majoring in Religious Studies will grow intellectually, civically, emotionally and spiritually. Students graduating with a degree in Religious Studies will be well-rounded individuals capable of pursing graduate studies, will have a tolerant world view, and will have knowledge of a variety of cultures and belief systems, allowing them to capably interact with people of wide and varying backgrounds.

36

To offer students an understanding of the basis for spiritual thought;

To develop in students an understanding and respect for religious thinking, both western and non-western

To provide students with an understanding of the historical developments of world religions

To expose students to a variety of ethics models

To help students become better citizens To promote strong emotional and spiritual

health among students

37

Graduating students with a Bachelor of Arts degree in Religious Studies will be able to:

1) Interpret, analyze and evaluate literature, both religiously-based and secular-based, in terms of the spiritual and ethical standards from within the society in which that literature is based.

2) Interpret, analyze and evaluate public discourse, in all media forms, in terms of the spiritual and ethical standards from within the society in which that discourse occurs.

38

3) Discuss the main historical developments of at least four of the following world religious paradigms: Judaism, Christianity, Hinduism, Islam, Buddhism.

4) Identify cultural differences based on unshared historical religious experiences

5) Differentiate between philosophical and religious thinking.

6) Demonstrate an understanding of how human spiritual pursuits have an impact on cultural, economic and political developments of societies.

39

PLO

REL 101

REL 112

REL 201

REL 221

REL 216

REL 228

REL 311

REL 312

REL 343

REL 410

REL 422

1 m M M M M M M M m M

2 m m M M m M M M m M M

3 M M M M M M M m M M

4 m m m M M M M M M

5 M M M M m M m M M m M

6 M M M M m M m M M M40

Mission and Goals are part of identity, and so, cannot easily be criticized

Department seemed to ignore non-majors in their set of objectives

Program Learning Outcomes not well-developed◦ Some are too complex & overlapping; others too

vague◦ Set of six outcomes probably doesn’t do their

program justice

41

Faculty recognized that their PLOs needed attention

More importantly, the faculty recognized that, even with more well-developed PLOs, their courses were overlapping significantly◦ One faculty member noted a fear that they were

unwittingly teaching variations of the same course 12 times

◦ Another faculty member indicated a desire to create departmental “tracks” to help differentiate content from one course to another

Curriculum mapping triggered a wholesale revision of the department’s curriculum and focus

42

General Education Learning Outcomes can be similarly “mapped” to individual CLO’s

Curriculum planning in General Education is difficult; a curriculum alignment exercise enhances and enables current activities to “pay off”

Curriculum alignment pertaining to General Education can also give a faculty greater understanding of the possibilities of assessment for educational purposes, while still fulfilling accountability expectations.

43

44

Questions/Comments/Criticisms?

Thanks and acknowledgement to Gary Larsen, whose Far Side cartoons lightened the mood of this presentation.

45