California Community Colleges Registrars and Admissions Officers (CACCRAO) Development of Meaningful Learning Outcomes Dotti Cordell BS, MPH, BSN Director

California Community Colleges Registrars and Admissions Officers

(CACCRAO) Development of Meaningful Learning Outcomes

Dotti Cordell BS, MPH, BSNDirector of Student Health Services

San Diego City CollegeMay 4, 2009

rekindle

reorient

REMEMBER relearn rechargerecharge REEXAMINEREEXAMINE

reviewreview A Challenge to ...A Challenge to ...

RECONNECTRECONNECT

revisitrevisit restore recommitrecommit REDISCOVERREDISCOVER

And Restart?...And Restart?... redirectredirect

replenishreplenish rechallenge recall rejuvenate

recapture

Presentation Objectives Review the process of

Outcomes Assessment Clarify misconceptions

relating to outcomes in Student/Administrative Services

Explain the Six Column Model utilized at SD City College Equip you with an

organized systematic knowledge base in this sometimes unfamiliar or uncomfortable territory

Outline potential trouble spots

Choice and Attitude

I hope that you can turn what might be a negative external mandate into a positive internal challenge.

I challenge you to view assessment as an opportunity for admission program improvement and a means to seek the best possible way to serve students.

If you do what you've always done, you'll get what you've

always gotten

Do You Feel A Time-Crunch?

What do I need to know?What does my department

need to know?What do I do?

Don’t Panic

You and your department are probably already doing it. You just may not know it. Or you may not be documenting the process.

Or you think there is a perfect way to do it and you are afraid to jump in.

Or it does not seem “natural” and seems to be a large additional work load.

What is YOUR Idea of a Quality Admissions Department? Of quality student service?

Quality: What Is It?

Quality of service: the concept does not possess substance or mass. Cannot place quality on the table to

analyze or study. Any definition of quality almost always

seems deficient Quality is in the eye of the beholder. Quality definitions reflect the tastes

and biases of those doing the defining.

Quality

A quality admissions program may be one in which services for students faculty and administration increases the likelihood of desired outcomes and is consistent with current professional knowledge

11

Performance

+ ServiceWrong Things Done RightInefficient admission process or system but people do an excellent job of entering the data

Right Things Done Right

Wrong ThingsDone WrongInefficient system to enroll students and staff using system make many errors entering the data

Right ThingsDone WrongState-of-the-art equipment used incorrectly by staff

(-)

P

R

O

C

E

S

S

+(-)

QualityDoing the right

things and doing them well.

QualityQualityNaked assertions of high quality

admissions, records, evaluations, counseling or any college department or program is no longer sufficient. Consumers, society, accrediting agencies and campus administrators hold us accountable to provide objective, quantifiable standards for quality student and service outcomes.

Admissions Assessment and Improvement

You know, WE know, that admission departments help maintain the institution and are ESSENTIAL to operations.

Assessment will help demonstrate that each staff member in your admissions/ registration departments are endeavoring to do their jobs better every day.

Assessment Principles

Assessment is NOT a one-time activity; it is evolutionary, ongoing, incremental.

Over time, assessment efforts should become more comprehensive, systematic and integrated.

Assessment efforts must be compatible with the institution's mission

Diane Ramirez of College of the Desert and Merrill Deming at Crafton Hills College

How Can You/We Do It All?

FERPABudget Crises Increasing

Demand for Services

Day-to-Day Operations OUTCOMES

MEASUREMENT?

What is Assessment?

Assessment is an ongoing process aimed at understanding/improving student learning and campus services and processes It serves to clarify our values Sets criteria and standards for learning and services We gather data to see how well performance

matches standards Assessment is deciding what YOU as THE subject

matter experts value and wish to measure and then using the resulting information to document and improve performance

Key Questions

What are we trying to do? What is it that admissions should accomplish? How is success defined? How successful are our students? How well have they learned?

How do we know? How well are we performing our functions?

How do we know? How do we use that information to continually

improve? Does the improvements we make work and do we

maintain the gain?Adapted From M. Bresciani, Director of Assessment SDSU and from CUPR Guidelines

The answers to these questions form the essence of what is called assessment.

Student Learning OutcomesDefinition

The knowledge, skills, abilities, attitudes, habits of mind or competencies that students have at the completion of a

Course Program Activity or

interaction The presence and quality of

which can be measured and evaluated.

Categorization of Student Learning Outcomes

Complex Cognitive Skills Knowledge Acquisition Interpersonal

Development Social Development Academic Achievement Persistence Practical Competence Civic Responsibility

National Assoc. of Student Personnel Administrators, Upcraft 1999

Assessment to Improve Admissions

Processes/Services Redesign services based upon best

practices Use information technology to improve

access to information and to support decision-making

Improve workforce knowledge and skills Develop effective teams within and

outside the department(s) Coordinate referrals Measure performance and outcomes

Why Do We Use AnAssessment Framework?

Model provides common language and a consistent framework

Allows celebration of small wins Demonstrates accomplishments

Is readily explained, one page Model clearly shows linkages with larger

institutional competencies and college mission

San Diego City College’s Six Column Model

Credit to Jim Nichols – We have adapted his work with over 380 colleges and are using an expansion of his model.

Columnar model--helps individuals see linkages between levels of organization.

City College Six Column Assessment Model

Column 1: Statement of Institutional Purpose—for City this is our Mission and our Institutional Priorities

Column 2: Institutional Competencies Communication/Interpersonal Skills Critical thinking Analyses/Computation Cultural Sensitivity/Global Awareness Information Management/ Literacy Personal Responsibility Civic and Environmental Responsibility

Column 3: Programmatic Student Learning Outcomes Column 4 : Means of Assessment and the Criteria for Success Column 5: Summarizes data collected (Results) Column 6: Designated to demonstrate how you will use

results to improve the program

Note: Linkages between the 6 columns is essential

Mission

Your departmental mission is your reason for being, the motivation for doing this work, the soul of the department

Column One

To guide City faculty and staff in formulating outcomes, column one is comprised of the City College mission statement and our college priorities

Admissions/Registration 2009 Outcome

COLUMN ONE

College Mission and Priorities

Column Institutional Competencies

Communication/Interpersonal Skills

Critical thinking

Analyses/Computation

Cultural Sensitivity/Global Awareness

Information Management/ Literacy

Personal Responsibility

Civic and Environmental Responsibility



COLUMN 2:

Institutional CompetenciesCommunication/Interpersonal SkillsCritical thinkingAnalyses/ComputationCultural Sensitivity/Global AwarenessInformation Management/ LiteracyPersonal Responsibility Civic and Environmental Responsibility

Column :Development of Departmental/Programmatic SLO’s

Departmental staff members convene to discuss and reach consensus on the focus of assessment for the coming semester/year.

Developing Programmatic Student Learning OR Administrative Unit

Outcomes

Collect and review documents that describe the program (catalog, your program’s mission)

Examine outcome statements from similar programs at other institutions

Review assessment criteria of WASC accrediting body

Review criteria from your professional organization--CACCRAO

Collect and review departmental materials

Brainstorm with your staff and utilize the short list/long list concept

Long and Short List Concept

Now choose 1-2 to study this semester/year. Have your department collaboratively discuss and reach consensus on what is important THIS year or semester:

1.2.

Make a LOOONG list of all the learning or administrative outcomes of importance to your department:1. 2. 3.…..42.

Writing Outcomes Uses clear, observable and measurable verbs

CompileCompile Arrange Arrange Classify Classify AnalyzeAnalyze Identify Identify Operate Operate DesignDesign Solve Write Solve Write ApplyApply Differentiate Calculate Differentiate Calculate DemonstrateDemonstrate Formulate Formulate Compose Compose ExplainExplain Predict Predict Assess Assess CompareCompare Estimate Estimate Critique Critique

The following verbs are relatively vague states of mind. They are The following verbs are relatively vague states of mind. They are very difficult/ impossible to measure:very difficult/ impossible to measure: Know Understand Appreciate Learn

Credit to Gavilan College for this content



Institutional Competencies


Critical thinking






COLUMN 3

Programmatic Learning or Administrative Outcome

After an encounter in the Admissions office or through the website students will _____________________________________________

Or the Admissions Dept. will_______________________________________________

Column 4—Means of Assessment and Criteria for Success

“Rubber hits the road” in this column Here you are defining the specifics of

measurement and precisely what you meant in column 3. You are describing the context for the outcome. What do you mean that you want the student to be better at availing him/herself of campus resources? Able to navigate the catalog?

It is in this column you must be very clear on the specifics of measurement, and at what point you will consider yourself successful. 78% of the students will…..etc..



Institutional Competencies


Critical thinking






COLUMN 3



Or the Admissions Dept. will decrease turnaround time for ____ filing.

COLUMN 4:

Means of Assessment and Criteria for Success

Using the 2009 Admissions satisfaction survey 72% of students will demonstrate 3.5 or above on the satisfaction scale of the admissions dept.’s ability to assist them with ________.

Using computer data from_____the Admissions dept. will demo a 12.6% decrease in turn-around time for ______ in the 2009-2010 school year.

Column 5: Log results obtained and compare to the criteria for success that the department or program set for itself

Column 6: Compare how actual results stacked up against the criteria for success. Will the department rotate the outcome off the list as “accomplished” and not problematic, or does it see the issue as an opportunity for programmatic improvement?

What are the departmental actions to improve? NOW you have “closed the loop” and the entire goal

of this assessment effort comes full cycle: you now have data and actions to support program improvement efforts.



Institutional CompetenciesCommunication/Interpersonal Skills

Critical thinking






COLUMN 3



Or the Admissions Dept. will decrease turnaround time for ____ filing.

COLUMN 4:

Means of Assessment and Criteria for Success

Using the 2009 Admissions satisfaction survey 72% of students will demonstrate 3.5 or above on the satisfaction scale of the admissions dept.’s ability to assist them with ________.

Using computer data from_____the Admissions dept. will demo a 12.6% decrease in turn-around time for ______ in the 2009-2010 school year.

COLUMN 5: Results

Using the 2009 Admissions satisfaction survey 64% of students demonstrated 3.5 or above on the satisfaction scale of the admissions dept.’s ability to assist them with ________.

Using computer data from_____the Admissions dept. demo’ed a 4% decrease in turn-around time for ______ in the 2009-2010 school year.

COLUMN 6: Use of Results for Program Improvement

With these results, we have decided to:

Do nothing because our findings are so good. We will move on to something else.

OR

We will do ________ to improve our findings and will re-measure again in _____ 2009/2010.

Column 6: Closing the LoopKey Questions

How well are we achieving the admissions/registrar program or department goals?

How do we know? How can we use

data/information to continually improve the processes and systems of the department?

Student Learning Outcomes And Administrative Unit Outcomes:

Clarification: FAQ’s

What is a student learning outcome (SLO)?Knowledge, skills, abilities, attitudes, habits of mind that students have at the completion of a series of activities or interactions--the presence and quality of which can be measured and evaluated

What is an AUO (Administrative Unit Outcome)?An AUO describes what an “Administrative Unit” such as your Admissions Depts. will accomplish or do in the provision of their services. The focus of AUOs is services. How you know you are accomplishing the purpose of Admissions, Records, Registration? THE FOCUS IS ON IMPROVEMENT IN SERVICES PROVIDED TO SUPPORT STUDENT LEARNING.

Why is it important to include student support services in this work?Key to understanding the importance of this initiative is the premise that the entire campus supports and is crucial to student learning. As such it requires a commitment of the entire campus as part of a comprehensive Institutional Effectiveness program

Why should I engage in outcome work if I’m not in an instructional area of the campus?Student learning experiences most definitely do not just occur in the classroom. Some of the most important learning on campus occurs outside of the classroom. It is important that the campus participate to demonstrate learning outside the classroom and the efficiency of campus operations.

Similarities between SLOs and AUOs

Both focus on improvement (in either student learning or administrative functioning)

Both contribute to and are part of the quality picture of the entire campus

Both use assessment data to effect improvement

Examples Of Programmatic Outcomes— Admissions

Admissions will shorten the response time for _____from 3 days to 2.5 days .

Over the counter requests for __________will be delivered in ____minutes/hours versus ____minues/hours.

94% of students will articulate a high level of satisfaction with the overall registration process as evidenced by an increase from 2.7 to 3.5 on a 5 point satisfaction scale.

Or 85% of students will demonstrate a high( remember to define what you mean by “high”) level of satisfaction on exit interviews from the admissions dept.

Admissions office complaints or dissatisfaction reports will decrease from ___ per semester to <2 per semester.

Admissions professionals will contribute to the overall campus commitment to a diverse student body by increasing_____________.

The Admissions department will institute the following interventions this semester to increase the % students having a valid ID card prior to “x” date of the semester from ___% to ___%:

Admissions will demonstrate an increase in students successfully( how is successfully defined?) navigating the college catalogue from ___% to ___% by______________________________.

To improve the learning outcome related to services referral and campus networking---As reported on an exit questionnaire from the Admissions department, students will accurately report 3 of 5 departments necessary to meet their needs and educational goals.

Admissions will provide accurate class enrollment data_____________.

Key Foundational Assessment Factors

at City College Not all outcomes are student learning outcomes, some may be

administrative outcomes which are essential to, and form the foundation upon which, student learning and success are built

This whole process is directed towards incremental program improvement, not individual evaluation

The “program” is as the department/area/service defines it. Key themes emerged at City on learning outcomes and Student

Services: Students learn about other campus resources from those of us in Student Services Students learn from Student Services professionals how to better advocate for

themselves Self-advocacy Self-assertiveness Networking

To have buy-in, sustainability, and assessment of value-- it must be an inclusive staff/faculty driven process: a grassroots effort

Values driven You and your department are the subject matter experts —what do

YOU feel is important to study in your department? Part of daily/weekly/monthly/semester work--not a set-aside project

Student Services

The campus is our classroom. All of us are teachers and each of us is

essential to the college mission and the work and learning that occurs daily with students.

Assessment is good---it demonstrates for us the vital role that each of us play in the educational process.

How To Start What do you and your dept. value or feel is important

to do well in your area? Seek consensus in your area on choosing this issue to

study. Pick one for now. Keep it simple

Write it down Decide on a measure

Observe students? Survey?

Collect data Act on the results. Write it down.

Progressive improvements beat postponed perfection every time

Direct and Indirect Measures

Satisfaction surveys Provide feedback on student opinions Indirect measure of assessment

Measure what the student feels or thinks about the program

Move away from satisfaction surveys Optimal to use satisfaction measures in conjunction with other measures

The assessment effort seeks to find what the students know, have learned or can demonstrate Direct measures of learning

Tips in Creating a Survey Limit number of questions

Include explicit directions

Check reading level

Eliminate having two questions built into one

Pilot the survey

Consider alternative methods of distribution/collection

Consider whether anonymity is an issue

Lessons from the Field A thousand mile march really does begin with your first

step.

Change is rarely convenient. The only who likes it is a wet baby.

Who are the assessment champions? Persons willing to be accountable to keep the process going forward? Set meetings? Keep notes? HOPE IS NOT A PLAN.

Focus, Focus, Focus. Start with just one question, one form, one anything!

KNOW how you will know that what you have done is an improvement. Design for incremental improvement, not publication or perfection.

Don’t get hung up on names: outcomes, objectives, competencies, goals, SLO’s, AUO’s, program definition. You know what is important to your dept.

Sometimes you just have to jump in and DO IT!

You are doing assessment much more than you think…you just are not taking credit for it.

Contact Information

Dotti CordellSan Diego City College1313 Park Blvd. Rm. A116San Diego, CA 92101(619) 388-3903Email: [email protected]

Documents

California Community Colleges Registrars and Admissions Officers (CACCRAO) Development of Meaningful Learning Outcomes Dotti Cordell BS, MPH, BSN Director