Making It Meaningful: Authentic Assessment for Intentional EducationDavid W. Marshall, PhDJoanna M. Oxendine, MEd

Overview Two Cultures: A Contrast in Emphasis Defining Outcomes Principles of Meaningful Assessment Making Meaningful Change

Two Cultures: A Contrast in Emphasis

CCSF’s Assessment Cycle

The Assessment Cycle

Write Outcomes

Identify Assessment Measures

Gather Evidence

Analyze Results

Strategize Program

Improvement

Perception of the Assessment Cycle

Write Outcomes

Identify Assessment Measures

Gather Evidence

PACKAGE RESULTS

SUBMIT REPORTS

ACCREDITATION

The Culture of Compliance

Sees accreditation as an end in itself Seeks information on what accreditors want to

see Worries about whether what they have matches

accreditors’ expectations

Students become unimportant elements of the assessment process

Another View of the Assessment Cycle

Articulate Intended

Outcomes

Work Toward Intended

Outcomes

Gather Evidence

Determine Extent to

Which Outcomes Were Met

Plan for Intentional

Improvements

STUDENTS

The Culture of Intentionality

Is student-centered Seeks information about how well students are

learning and/or how well various areas of the college are supporting the student experience

Reflects on what we teach or do and how we teach or do it

Accepts (some) responsibility for student learning and the student experience Experiments with new strategies for student

success

Students become the primary focus of the assessment process

Evidenced-Based Change in the Context of a Culture of Intentionality

What is Evidence-Based Change?Evidence-based change is:

An approach to organizational activities and decision-making that uses information about the effectiveness of those activities and decisions to implement strategic and targeted revisions towards increased impact of the organization’s key goals.

What is Evidence-Based Change?Evidence-based change is:

An approach to organizational activities and decision-making that uses information about the effectiveness of those activities and decisions to implement strategic and targeted revisions towards increased impact of the organization’s key goals.

What is Evidence-Based Change?Evidence-based change is:

An approach to organizational activities and decision-making that uses information about the effectiveness of those activities and decisions to implement strategic and targeted revisions towards increased impact of the organization’s key goals.

What is Evidence-Based Change?Evidence-based change is:

An approach to organizational activities and decision-making that uses information about the effectiveness of those activities and decisions to implement strategic and targeted revisions towards increased impact of the organization’s key goals.

The Organization’s Key Goals??

Articulate Intended

Outcomes

Work Toward Intended

Outcomes

Gather Evidence

Determine Extent to

Which Outcomes Were Met

Plan for Intentional

Improvements

STUDENTS

Defining Shared PurposeAn approach to organizational activities and decision-making that uses information about the effectiveness of those activities and decisions to implement strategic and targeted revisions towards increased impact of the organization’s key goals.

An Analogy Those who: Pray/Teach Fight/Defend Farm/Provide Food

Mutuality

Taking the Analogy Too Far

Defining Outcomes

Two Types of Outcomes

Student Learning Outcomes Statements that describe the knowledge, skills,

attitudes, and habits of mind that students take with them from a learning experience (adapted from Suskie, 2009)

Service Area Outcomes Statements that describe the intended results of

specific programmatic, operational, or administrative activities

Two types at CCSF Student Service Outcomes: statements that identify

what a student will be able to know and/or do after receiving a service and/or participating in student

support programs Administrative Unit Outcomes: statements directly

related to the college’s operations and non-academicservices that support the learning environment

Outcomes indicate means by which success in achieving goals might be defined.

Two Types of Outcomes

Intentionality and Outcomes

From the student perspective, outcome articulation: Makes explicit for students what is expected of them so they know what to expect. Removes the need for students to intuit what learning is expected.

From the college’s perspective, outcome articulation: Directs efforts to evaluate how successful activities have been. Helps to identify what kind of information to collect.

Outcomes indicate means by which success in achieving goals might be defined.

Outcomes and the Two CulturesCULTURE OF COMPLIANCE

Rarely communicates outcomes to students

Files outcomes with the appropriate office or administrative oversight entity

Sticks with what has always been done

Works on outcomes assessment for an accreditation cycle

CULTURE OF INTENTIONALITY

Makes outcomes visible to students

Incorporates outcomes into practice

Assesses outcomes appropriately

Uses outcomes for ongoing conversations about effectiveness and continual

improvement

Outcomes and the Two CulturesCULTURE OF COMPLIANCE

Rarely communicates outcomes to students

Files outcomes with the appropriate office or administrative oversight entity

Sticks with what has always been done

Works on outcomes assessment for an accreditation cycle

CULTURE OF INTENTIONALITY

Makes outcomes visible to students

Incorporates outcomes into practice

Assesses outcomes appropriately

Uses outcomes for ongoing conversations about effectiveness and continual

improvement

Principles of Meaningful Assessment

A Process of Questions

1. What do we want students to know, understand, and be able to do? What are the intended results of our programmatic, operational, or administrative activities?

2. Where do students learn what we expect them to learn? How do we accomplish what we set out to do?

3. How well did students learn what we expected them to learn? How well did we do what we intended to do?

4. How do we know how well they learned what we expectedthem to learn?

How do we know how well we did what we expected to do?

5. What does our evidence tell us? How can we use what our evidence tells us to make appropriate decisions?

The Culture of Intentionality opens a clear approach to assessment.

A Process of Questions

1. What do we want students to know, understand, and be able to do? What are the intended results of our programmatic, operational, or administrative activities?

2. Where do students learn what we expect them to learn? How do we accomplish what we set out to do?

3. How well did students learn what we expected them to learn? How well did we do what we intended to do?

4. How do we know how well they learned what we expectedthem to learn?

How do we know how well we did what we expected to do?

5. What does our evidence tell us? How can we use what our evidence tells us to make appropriate decisions?

The Culture of Intentionality opens a clear approach to assessment.

Outcomes

A Process of Questions

1. What do we want students to know, understand, and be able to do? What are the intended results of our programmatic, operational, or administrative activities?

2. Where do students learn what we expect them to learn? How do we accomplish what we set out to do?

3. How well did students learn what we expected them to learn? How well did we do what we intended to do?

4. How do we know how well they learned what we expectedthem to learn?

How do we know how well we did what we expected to do?

5. What does our evidence tell us? How can we use what our evidence tells us to make appropriate decisions?

The Culture of Intentionality opens a clear approach to assessment.

Outcomes

Mapping

A Process of Questions

1. What do we want students to know, understand, and be able to do? What are the intended results of our programmatic, operational, or administrative activities?

2. Where do students learn what we expect them to learn? How do we accomplish what we set out to do?

3. How well did students learn what we expected them to learn? How well did we do what we intended to do?

4. How do we know how well they learned what we expectedthem to learn?

How do we know how well we did what we expected to do?

5. What does our evidence tell us? How can we use what our evidence tells us to make appropriate decisions?

The Culture of Intentionality opens a clear approach to assessment.

Outcomes

Mapping

Alignment of

Measures

A Process of Questions

1. What do we want students to know, understand, and be able to do? What are the intended results of our programmatic, operational, or administrative activities?

2. Where do students learn what we expect them to learn? How do we accomplish what we set out to do?

3. How well did students learn what we expected them to learn? How well did we do what we intended to do?

4. How do we know how well they learned what we expectedthem to learn?

How do we know how well we did what we expected to do?

5. What does our evidence tell us? How can we use what our evidence tells us to make appropriate decisions?

The Culture of Intentionality opens a clear approach to assessment.

Outcomes

Mapping

Alignment of

Measures

Evidence

A Process of Questions

1. What do we want students to know, understand, and be able to do? What are the intended results of our programmatic, operational, or administrative activities?

2. Where do students learn what we expect them to learn? How do we accomplish what we set out to do?

3. How well did students learn what we expected them to learn? How well did we do what we intended to do?

4. How do we know how well they learned what we expectedthem to learn?

How do we know how well we did what we expected to do?

5. What does our evidence tell us? How can we use what our evidence tells us to make appropriate decisions?

The Culture of Intentionality opens a clear approach to assessment.

Outcomes

Mapping

Alignment of

Measures

Evidence

Closing the Loop

Making Meaningful Change

Responding to the Results

If an outcome meets expectations at the standard:

Take no action

Identify this area as a program strength

Consider increasing expectations outlined in the outcome

Raise the standard of attainment

Consider surveying students about their experience of the program or service area

Assessment may find that student learning or a program meets expectations at the determined standard for some outcomes.

Responding to the Results

If results for an outcome do not meet expectations:

ASK QUESTIONS!! Are our outcomes aligned with our expectations? What aspect of our program or activities might be in need of

revision? Are our teaching and learning methods in need of revision? With whom might we need to work to strengthen our

outcomes? What crazy idea might we come up with to tackle the problem?

Assessment may find that student learning or a program does not meet expectations at the determined standard for some outcomes.

TOTAL ACROSS ALL SECTIONSMeets the outcome: 287 (74%)Developing the outcome: 74 (20%)No evidence: 21 (6%)

CRN: 31251Daphne: MetJerome: DevelopingJanet: Not assessedBen: MetSu-Lin: DevelopingAhmed: Not assessed

CRN: 31252Daphne: MetJerome: DevelopingJanet: Not assessedBen: MetSu-Lin: DevelopingAhmed: Not assessed

CRN: 31253Daphne: MetJerome: DevelopingJanet: Not assessedBen: MetSu-Lin: DevelopingAhmed: Not assessed

CRN: 31254Daphne: MetJerome: DevelopingJanet: Not assessedBen: MetSu-Lin: DevelopingAhmed: Not assessed

CRN: 31255Daphne: MetJerome: DevelopingJanet: Not assessedBen: MetSu-Lin: DevelopingAhmed: Not assessed

CRN: 31256Daphne: MetJerome: DevelopingJanet: Not assessedBen: MetSu-Lin: DevelopingAhmed: Not assessed MEETS THE SLO:

Online: 45 (85%)Face-to-Face: 242 (74%)

MEETS THE SLO:BOG Waiver: 192 (78%)No BOG Waiver: 65 (72%)

MEETS THE SLO:Morning: 146 (78%)Afternoon: 96 (68%)

PROGRAM IMPROVEMENTS!

CCSF Progress on NEW ACCJC STANDARD for Disaggregated SLO Data

MEETS THE SLO:Male: 161 (77%)Female: 123 (73%)

BIO 9 CurricUNET Data Processed by Office of Research (ARGOS)

SLO: Critique scientific information in the media for accuracy and reliability.

287 Meets SLO (74%)

Male: 161 (77%)Female: 123 (73%)

Morning: 146 (78%)Afternoon: 96 (68%)

BOG Waiver: 192 (78%)No BOG Waiver: 65 (72%)

Online: 45 (85%)Face-to-face: 242 (74%)

74 Developing SLO (20%)

Male: 37 (22%)Female: 36 (17%)

Morning: 34 (18%)Afternoon: 53 (23%)

BOG Waiver: 57 (21%)No BOG Waiver: 15 (18%)

Online: 5 (9%)Face-to-face: 67 (20%)

21 No evidence of SLO (6%)

Male: 8 (5%)Female: 13 (6%)

Morning: 10 (5%)Afternoon: 12 (8%)

BOG Waiver: 18(5%)No BOG Waiver: 3 (4%)

Online: 3 (6%)Face-to-face: 22 (7%)

For meaningful change to occur…

Faculty and staff must be the driving force and they must work collectively!

Faculty and staff: know the curriculum or systems in which they work. are responsible for seeing that outcomes are met. will be the ones who innovate practice.

The administration: creates space for the work to occur. fosters the culture of intentionality.

Responding to the ResultsStudents benefit from an institution’s thoughtful response to an honestly undertaken attempt to determine a program’s strengths and weaknesses.

Articulate Intended

Outcomes

Work Toward Intended

Outcomes

Gather Evidence

Determine Extent to

Which Outcomes Were Met

Plan for Intentional

Improvements

STUDENTS

To Conclude With an approach that strives for student-centered intentionality, compliance becomes a matter of simply completing forms and writing narratives, thereby accreditation takes care of itself.

Articulate Intended

Outcomes

Work Toward Intended

Outcomes

Gather Evidence

Determine Extent to

Which Outcomes Were Met

Plan for Intentional

Improvements