From Compliance to Improvement: Accountability and Assessment

for California’s Community Colleges

Norena Norton Badway, Ph.D.University of the Pacific

Higher Education Evaluation and Research Group

Introduction:Choosing

Improvement over

Compliance

Compliance…ImprovementIm

pro

vem

en

t C

om

plia

nce

Improvement…Compliance C

om

plia

nce

Imp

rovem

en

t

Saying YES to Assessment and Accountability

• acknowledges community colleges’ appropriate roles in equity, upgrade training, lifelong learning, and other unconventional missions

• gives faculty an appropriate voice in running their institutions

• promotes a form of research in teaching and the creation of improvements in teaching

• provides a foundation for widespread institutional improvement

• become more effective learning environments

California Community Colleges operate under at least four accountability

systems:1) PFE uses system level goals2) State Report Card assesses

performance of all publicly funded workforce preparation programs

3) Federal Vocational and Technical Education Act

4) Workforce Investment Act

Accountability is NOT new in CA

• WASC has been the last regional accreditation commission to require colleges to develop mechanisms of assessment and use of student learning outcomes

•••WASC lets LOCALITIES choose which aspects of SLOs to measure and how to measure them

… and now WASC

Well developed system of internal accountability

Ability to respond to external

accountability requirements

Taking StockTaking StockOf ExistingOf ExistingResourcesResources

Cycle ofInquiry ofStudentLearning

CHOOSEIMPROVEMENT

OVERCOMPLIANCE

TakeStock

NormStudentLearningOutcomes

NormAssessments

Identify,Implement,

EvaluateImprovements

BuildInstitutional

Capacity

Implementand

AnalyzeAssessments

It is unlikely that any campus does not have in place

— either informally or formally — some aspects of a Cycle of Inquiry

about Student Learning.

TAKING STOCK Acknowledges Existing Practices

Presenting student learning outcomes

assessment cycles as if it is a totally new process

ignores existing practices.

TAKING STOCKFosters Faculty Dialogue

Dialogue about a department’s

philosophy about teaching and learning guides the choice of student learning outcomes.

TAKING STOCKFacilitates Norming

• Expectations for for what students know and can demonstrate upon completion of the course are collaboratively authored and collectively accepted.1

• Full time and adjunct faculty who teach a course come to consensus.

1Maki, P.L. (2004). Assessing for Learning. American Association for Higher Education, Sterling, VA: Stylus.

The Benefit of AlignmentWhen these five elements

• student expectations, • faculty expectations, • curriculum content, • institutional support, • governance for assessment)

are in alignment, or in equilibrium, classrooms and student learning are likely to run smoothly.

The Risks of Non-alignment

• If professors disagree with the curriculum, they may undermine or embellish it, for good or bad.

• If institutions fail to support faculty, they may undermine the possibility of being a “teaching institution”.

• If professors and students disagree about the content or teaching methods, classes may be hostile, with little learning going on.

Instructors

StudentsContent/Curriculum

THE CRUCIBLE OF THE CLASSROOM

• What do you know about your students’ attitudes and beliefs

about learning?

TAKING STOCK: STUDENTS

Students ?

Taking Stock

• We have a systematic way of gathering information about student beliefs and values about learning.

Faculty usually know a great deal about students and their lives, and

we try to be sympathetic to the “busied-up” conditions caused (often) by the need to work and maintain family responsibilities.

We may know much less about how students think about the purpose of college and the nature of learning.

In the conventional professorial modelof college, invisible disjunctions betweenstudents’ and professors’ understandingabout teaching and learning become the

students’ responsibility.

In a more collaborative model of teaching, part of the professor’s

responsibility involves understanding how students perceive college, the

curriculum, and the nature of learning.

ACTIVITY: STUDENTS

• Into what key groups do you subcategorize your students?

ACTIVITY: STUDENTS

• How do you identify the changing needs of your student groups?

ACTIVITY: STUDENTS

• What are your students’ beliefs and values about learning?

ACTIVITY: STUDENTS

• How do you know? • What is the mechanism for

gathering this information• What is the forum in which you

discuss categories, changing needs, and attitudes of students?

ACTIVITY: STUDENTS

• Are your students credentialist, wanting credit/ credentials but not necessarily the learning the credential signifies (grades matter more than content)?

ACTIVITY: STUDENTS

• Are your students vocationalist, using college as a route to employment (relevancy matters more than intellectualism; students continuously make cost-benefit calculations)?

ACTIVITY: STUDENTS

• Do your students undermine their own learning outcomes by being fearful, afraid of being caught unprepared, isolated, intimidated by professors?

ACTIVITY: STUDENTS

• Do they manage their fear in unproductive ways by keeping quiet in class, by avoiding hard classes, by scaling down their ambitions, by failing to submit work even when it’s completed, or by dropping or stopping out?

ACTIVITY: STUDENTS

• Do your students define learning as the accumulation of facts?

Discussion: What are the results of mis-alignment between

faculty and student goals?

• Students rebel against professor efforts to expand students’ knowledge.

• The counter-productive behavior of students can generate counter-productive reactions from faculty.

Changing the attitudes of studentsmay be difficult,

because we have to fight against larger social trends and pressures,

but it is likely to be better accomplished

through the collective actions of all faculty,

than by the efforts of professors one-by-one.

• What do you know about instructors’ attitudes, beliefs and knowledge about

teaching and learning?• Is teaching “community property”?

TAKING STOCK: INSTRUCTORS

Instructors ?

Higher education is unique in thatit generally denies employment to those schooled in its craft:

teaching and learning.

Faculty autonomy, academic freedom, and professional discretion weigh

against shared (normed) expectations, assessments and criteria.

The issue of inter-rater reliability rarely

is raised in higher education.

Taking Stock• Everyone in our department agrees

that we have common learning objectives for each course.

• Everyone in our department agrees on a process for determining how we will assess student learning.

• We regularly share what works and what doesn’t work with subgroups of students.

• We have a formal process for using what we learn to improve what we do related to student learning.

ACTIVITY: FACULTY

• What philosophies, principles, models of teaching, research on learning, or shared assumptions about teaching and learning underlie your curricular or co-curricular design, instructional design, or use of educational tools?

ACTIVITY: FACULTY

• What pedagogies or educational experiences develop the knowledge, understanding, habits of the mind, ways of knowing, and problem solving that this discipline, profession, program or institution values?

ACTIVITY: FACULTY

• How do faculty and student support services build on each others’ courses and educational experiences to achieve departmental/ programmatic/ institutional learning priorities?

ACTIVITY: FACULTY

• What are faculty attitudes and knowledge about learning, teaching, assessment, “teaching as community property”, and continuous improvement?

ACTIVITY: FACULTY

• Is there a forum for discussing examples and reasons for student success or lack of success, teaching ideas and methods? How often are these the topics of discussion among faculty?

ACTIVITY: FACULTY

• How have faculty previously developed, shared and implemented student learning outcomes and assessments?

• What are external influences on curriculum?• What is the consistency of curriculum?

TAKING STOCK: CURRICULUM

Curriculum ?

Taking Stock

• Everyone in our department shares the same expectations of what students should know and be able to do at the end of each of our courses.

• At the beginning of the semester, we share with students what is expected of them, the criteria by which they will be measured, and what standards are required for grades and course completion.

ACTIVITY: CURRICULUM

• Which of your critical curriculum is set by external agencies?

ACTIVITY: CURRICULUM

• What is the consistency in expectations across sections of a course?

ACTIVITY: CURRICULUM

• When/ where do instructors norm content and assessment?

ACTIVITY: CURRICULUM

• Do instructors collaborate on and/or do peer review of learning outcomes for critical courses?

• What regulations impact student learning outcomes?

• How do local practices and policies impact student learning outcomes and

assessment?

TAKING STOCK: INSTITUTIONAL

Institutional Support ?

Taking Stock

• Our organizational practices and processes are designed to strengthen student learning outcomes.

• Faculty time, professional development, as well as hiring, promotion, and tenure policies place highest priority on student learning outcomes.

ACTIVITY: INSTITUTION

• What federal, state, and/or professional-industry regulations impact student learning outcomes, their assessment, and improvement on your campus?

ACTIVITY: INSTITUTION

• How much faculty time is devoted to meetings unrelated (or only peripherally related) to student learning outcomes?

• Could that time be released for focusing on student learning outcomes?

ACTIVITY: INSTITUTION

• Does professional development at your institution focus on norming student learning outcomes and assessments?

ACTIVITY: INSTITUTION

• Where in your institution do faculty collectively set evidence-driven decisions related to student learning outcomes?

• What assessments are in place now (placement tests, capstone projects, portfolios, paper-pencil tests, etc.?)

• How do those assessments contribute to improving student learning outcomes?

TAKING STOCK: EXISTING ASSESSMENTS

Existing Assessments ?

Taking Stock

• Our entering assessments (placement tests) accurately and reliably assign students to the appropriate courses.

• Our exit assessments gather evidence that is used for the review of program content and pedagogy.

ACTIVITY: ASSESSMENTS

Name of assessment

Purpose of assessment

Effectiveness of assessment

• Who coordinates student learning outcomes, assessments, improvement

strategies and continuous improvement?

TAKING STOCK: GOVERNANCE of ASSESSMENT

Governance ?

If assessment is to be continuous, on-going and stable, then it must be overseen by a group that takes responsibility for all aspects of assessments.

In a self-reforming institution focused on instruction, the Assessment Committee would be the central committee in a college, so that

concern over the nature and effectiveness of instruction drives all

other aspects of a college.

In this way, the Assessment Committee should have responsibility

not only for creating a series of assessments but also for overseeing

the subsequent stages in the assessment system.

ACTIVITY: GOVERNANCE Student Assessment Leadership Team

Membership Now Add? Rationale

Reporting Out“When we take stock, we believe

we have strength from _________ and we want to build __________.”

• student attitudes & beliefs; • instructor attitudes & knowledge;• consistency of pedagogy & curriculum;• local practices & policies; • existing assessments; • local governance of SLOACs

A Primer

Setting and Assessing Student Learning

Outcomes

Definition: SLO“Robust” student learning outcomes

incorporate • behavioral objective- what a student

should know, value and be able to demonstrate/ perform

• conditions under which performance will be assessed (simulation, lab, portfolio, writing task)

• Criteria/ performance standards/ primary traits for assessing student performance

• Rubric for scoring student performance

1Adapted from Scroggins, B. (2003, 2004). Targeting Student Learning. Modesto Junior College. <http:cai.cc.ca.us/workshops/SLOFocusOn Results.doc>

Confusion: Terminology??

Course objectives versus student learning outcomes:

Generally, course objective states what student will demonstrate, represent or produce at end of course.

SLO also incorporates the conditions under which assessment will occur (test, portfolio, demonstration, etc) as well as evidence/criteria

slo — SLOA — SLOACSLOAC

• Departmental/ institutional practitioners NORM- objectives for student performance (what should students know, be able to do, value)- conditions under which performance will be assessed (simulation, portfolio, lab experiment, writing assignment)- traits/ criteria for assessing student performance and rubrics for scoring

• Implement assessment plan • Compile and collectively analyze results from

scoring student performance • COLLECTIVELY SET IMPROVEMENT PLAN• IMPLEMENT IMPROVEMENT PLAN• CONTINUE CYCLE

Norming1

• “Nested” discussions, decisions and actions

• Collaboratively authored and collectively accepted expectations for student learning and assessment

• Norming does NOT mean identical learning activities, emphases, pedagogy — it means C&C

1Maki, P.L. (2004). Assessing for Learning. American Association for Higher Education, Sterling, VA: Stylus.

Norming — Higher Ed Culture

• SLOs, criteria/ primary traits, rubrics are set collaboratively with full and adjunct professors

• Outcomes/ examples of student work are shared and peer-reviewed

• Improvement alternatives are agreed upon

• Autonomy/ Academic freedom/ Professional discretion and expertise

••••••••

••

•••

Reliability:Inter-rater or test-retest

Validity:Instrument/

procedure measures what it is intended

to measure

THE

BASICS

Types of Data

•Quantitative•Qualitative

SYSTEMATIC

References: Norm & Criterion

• Norm-referenced assessments measure individual outcomes relative to the sample of people taking the test- grading on curve

• Criterion-referenced assessments measure individual outcomes compared to certain norms or criteria -mastery, licensure

** Criterion-referenced assessments are appropriate for measuring improvement in SLOs.

Direct vs. Indirect Measures

• Direct measures are reasonable replications of real world tasks; authentic assessment- DO IT

• Indirect measures are proxies for demonstrated performance: grades, persistence, transfer (legislated measures are often proxies)

External vs. Internal Accountability

• External accountability is used to meet requirements of funding/ regulatory agencies

• Internal accountability is used to improve student learning within courses, programs or degrees.

Well developed cycle of internal accountability

Ability to respond to external

accountability requirements

FORMS

OF

ASSESSMENT

Assessments• Capstone projects• Demonstration• Simulations• Portfolios } AUTHENTIC

ASSESSMENTS

•Criterion-referenced tests (licensure exams)•Norm-referenced tests (curve)

Embedding Assessments

• Align with certificate, departmental, degree, institutional goals

• Assessment is woven into existing courses

• Identify SLO demonstration points• Retain and analyze results

EXAMPLE: Embedding Assessments in a Program

Course Assessment Criteria Rubric

Eng 101 Write for audience

• Writes research paper appropriate to a specific audience

A= Uses language & concepts appropriate for a professional, technical or literary audience

Social Issues

Identifies potential conflicts among diverse groups

• Designs Valuing Diversity workshop for a specific setting

A= Incorporates ethnic, life style, age and gender diversity; incorporates activities appropriate for one specific setting

Levels of Outcomes

• Targeted population of students• Lesson/ unit of study• Program

– Occupational certificate– Major– Department/ Division

• Associate degree (A.A./A.S./A.A.S)• Institutional

Targeted Populations

• For VTEA and some grants, a college may wish to focus on retention, persistence and/ or achievement for special populations of students– CalWorks– First generation– Limited English proficient

Lesson/ Unit-Level Assessment

Classroom assessment techniques (Cross & Angelo)– Systematic but informal, frequent gathering of

information about content and pedagogy:• What was hard to understand

today?• How did this teaching method work for you?

Course Level SLOs

• Most campuses are emphasizing course level assessments as part of program level SLOACs.1

1 Friedlander, J. & Serban, A. (2004). Meeting the Challenges of Assessing Student Learning Outcomes, in Friedlander, J. & Serban, A. [Ed.]Developing and Implementing Assessment of Student Learning Outcomes. New Directions for Community Colleges No. 126. San Francisco, Jossey Bass.

Norena Norton Badway, Ph.D.Principal

Phone 209-951-7477 home office 209-946-2168 University office 209-601-7121

Email badway@aol.comnbadway@pacific.edu