Educational

Effectiveness

Fall Faculty Retreat 2006

Leanne Neilson Halyna Kornuta

Key Learning Points

Common understandings

“Educational Effectiveness”

Evidence

Use of Evidence

Educational Effectiveness:What is it?

What we want our students to know

• Competencies, knowledge, skills, abilities,

applications, and dispositions

How we know they know it

• Evidence

Why Bother? Our need for Evidence

Student Recruitment

Accreditation

Fundraising / Grants

Institutional Improvement, Self-

study and Reflection

Indirect EvidencePerceptions and Input

Methods

Student Satisfaction Surveys

Student Exit Surveys Alumni Surveys Employer Surveys Focus groups Exit Interviews with

Graduates

Data Mining

Faculty/Student ratios Percentage of students who

study abroad Enrollment trends Percentage of students who

graduate within six years Diversity of students, faculty,

staff Retention and Transfer Studies Percentage of students who go

on to graduate school Job Placement Statistics

Direct Evidence Products of Student Learning

Student work samples

Portfolios Capstone projects Assessment of

student performance

Case Analysis

Pre-and post-tests

Blind scored assignments

Locally developed tests

Standardized tests

National licensure examinations

What Constitutes Good Evidence

1. Purposeful

2. Representative

3. Varied

4. Actionable

CLU Evidence

Indirect• NSSE / BCSS• Noel-Levitz • Library

Direct• WIC• SOE

Think About. . .

CLU Indirect Evidence

N S S ENational

Survey of

Student

Engagement

B C S SBeginning

College

Student

Survey

NSSE 2005 / 2001

Table 6 Diversity: CLU First-years/Seniors compared with Selected Peers and NSSE Nationwide First-year/Seniors

11: Educational and Personal Growth: To what extent has your experience at this institution contributed to your knowledge, skills, and personal development in the following areas? (Scale: 1=Very little 2=Some 3=Quite a bit 4=Very much)

Means Comparison

CLU NSSE Nationwide

N= 174 first-year (FY) students (2005) N= 159 senior year (SR) students (2005)

2005 2001 2005 2001

l. Understanding people of other racial and ethnic backgrounds

NSSE 2005 / 2001

Table 6 Diversity: CLU First-years/Seniors compared with Selected Peers and NSSE Nationwide First-year/Seniors

11: Educational and Personal Growth: To what extent has your experience at this institution contributed to your knowledge, skills, and personal development in the following areas? (Scale: 1=Very little 2=Some 3=Quite a bit 4=Very much)

Means Comparison

CLU NSSE Nationwide

N= 174 first-year (FY) students (2005) N= 159 senior year (SR) students (2005)

2005 2001 2005 2001

FY 2.70 2.45 2.59 2.58 l. Understanding people of other racial and ethnic backgrounds

SR 2.66 2.49 2.60 2.64

NSSE 2005 / 2001

Table 6 Diversity: CLU First-years/Seniors compared with Selected Peers and NSSE Nationwide First-year/Seniors

1: Academic, Intellectual, and Social Experiences: In your experience at your institution during the current school year, about how often have you done each of the following? (Scale: 1=Never 2=Sometimes 3=Often 4=Very often)

Means Comparison

CLU NSSE Nationwide

N= 174 first-year (FY) students (2005) N= 159 senior year (SR) students (2005)

2005 2001 2005 2001

FY 2.73 2.67 2.60 2.60

SR 2.75 2.58 2.65 2.59

u. Had serious conversations with students of a different race or ethnicity than your own

NSSE 2005 / 2001

Table 6 Diversity: CLU First-years/Seniors compared with Selected Peers and NSSE Nationwide First-year/Seniors

1: Academic, Intellectual, and Social Experiences: In your experience at your institution during the current school year, about how often have you done each of the following? (Scale: 1=Never 2=Sometimes 3=Often 4=Very often)

Means Comparison

CLU NSSE Nationwide

N= 174 first-year (FY) students (2005) N= 159 senior year (SR) students (2005)

2005 2001 2005 2001

FY 2.73 2.67 2.60 2.60

FY *

1 – 15% 2 – 27%

1 – 16% 2 – 29%

1 – 15% 2 – 34%

1 – 15% 2 – 34%

SR 2.75 2.58 2.65 2.59

u. Had serious conversations with students of a different race or ethnicity than your own * Note: % for respondents who selected 1 (never) and 2 (sometimes)

SR *

1 – 10% 2 – 37%

1 – 13% 2 – 39%

1 – 12% 2 – 35%

1 – 13% 2 – 37%

BCSS

17j. Participate in a

school-sponsored

community service

project

NSSE

1k. Participated in a

community-based

project (e.g. service

learning) as part of a

regular course

BCSS / NSSE

High

Low

Low High

Eng

agem

ent

Expectation

BCSS / NSSE

High 1% 8%

Low 50% 41%

Low High

Eng

agem

ent

Expectation

Service to the

Community

Noel-Levitz 2006

Noel Levitz Adult Student Priorities Survey CLU

N=310 National Adult

Students

Item (Scale: 1= not at all satisfied to 7= very satisfied) Satis / SD Satis / SD Mean

Difference

3. Faculty care about me as an individual. 5.96 / 1.09 5.65 / 1.32 0.31

19. My academic advisor is knowledgeable about requirements in my major.

6.06 / 1.29 5.66 / 1.49 0.40

26. Faculty provide timely feedback about student progress in a course.

5.77 / 1.21 5.41 / 1.44 0.36

32. My classes provide opportunities to improve my technology skills.

5.61 / 1.25 5.29 / 1.46 0.32

40. Faculty are usually available for adult students outside the classroom by phone, by e-mail or in-person.

6.11 / 1.03 5.90 / 1.25 0.21

50. My advisor helps me apply my academic major to specific career goals.

5.44 / 1.49 4.97 / 1.74 0.47

CLU Indirect Evidence

Library Expenditures and Usage

Total FTE

Total Library Expenditures

Per FTE

Circulation Transaction (Including Reserves) Per FTE

Total Interlibrary

Loans Received

Comparison Group Median

State Median (CA)

National Median

CLU 2,500 $276 8 1,680

Whitworth College, WA

Pacific University, OR

Univ of Portland, OR

Pacific Lutheran Univ, WA

FTE: Full Time Equivalent Student 12 month Enrollment

Source: Education Dept. Biannual Survey of Academic Libraries 2003 - 04

CLU Indirect Evidence

Library Expenditures and Usage

Total FTE

Total Library Expenditures

Per FTE

Circulation Transaction (Including Reserves) Per FTE

Total Interlibrary

Loans Received

Comparison Group Median 3,362 $446 12 4,223

State Median (CA) 1,464 $179 7 103

National Median 1,431 $277 9 365

CLU 2,500 $276 8 1,680

Whitworth College, WA 2,045 $387 12 2,435

Pacific University, OR 2,222 $660 21 10,054

Univ of Portland, OR 2,927 $560 21 5,929

Pacific Lutheran Univ, WA 3,167 $654 15 6,758

FTE: Full Time Equivalent Student 12 month Enrollment

Source: Education Dept. Biannual Survey of Academic Libraries 2003 - 04

CLU Direct EvidenceStudent Learning OutcomesStudent Learning Outcomes

Written Communication

Information Literacy

Critical Thinking

W I C

WIC 2005

Table 2 Spring 2005 Summary of Criteria and Mean Scores

Course Type Mean Scores Summary Criteria Freshmen

English N =88

Writing Intensive N = 152

Capstone

N = 99

N = 339 Written Communication (5 point analytic rubric)

1. Clear thesis/purpose/topic 3.0 3.2 3.3 3.2 2. Paper organized/unified 2.9 3.3 3.2 3.1 3. Free of errors—grammar, punctuation, spelling 3.3 3.3 2.9 3.2 4. Purpose/thesis addressed throughout 3.0 3.4 3.4 3.3 5. Supporting argument, evidence, examples 2.9 3.4 3.3 3.2

6. Language and Vocabulary 3.5 3.5 3.4 3.5 7. Conclusion (synthesis & addresses thesis/purpose) 2.7 2.9 3.1 2.9

8. Maintains level of excellence throughout 2.9 3.2 3.1 3.1 Written Communication Mean 3.0 3.3 3.3 3.2

Critical Thinking Score and Mean (4 point holistic rubric)

2.5 2.8 2.7

WIC 2006

Table 3 Spring 2006 Summary of Criteria and Mean Scores

Course Type Mean Scores Summary Criteria

Freshmen English N =56

Writing Intensive N = 88

Capstone

N = 37

N = 181

Written Communication, Critical Thinking, and Information Literacy (4 point analytic rubric)

1. Articulates focus 3.1 2.8 3.6 3.0 2. Finds sources 2.1 2.9 3.2 2.7 3. Analyzes and evaluates sources 2.7 2.7 3.3 2.8 4. Synthesizes information to support arguments 2.7 2.6 3.1 2.7 5. Presents arguments 2.6 2.5 3.2 2.7 6. Uses conventions of standard U.S. English 3.0 3.1 2.9 3.0 7. Creates conclusion 2.6 2.5 3.4 2.7

Mean 2.7 2.7 3.3 2.8

CLU Direct EvidenceSOE

Field Supervisor Evaluation

Course Work (Portfolio)

Comprehensive Exam / Thesis / Webfolio Defense

Entrance / Exit Interviews

Alumni Survey

Employer / Administrator Survey

Additional Evidence

Assignment / Evidence C E D SLO e.g. Complete an analysis of a journal article requiring students to identify, interpret, assess, and evaluate relevant information. This assignment occurs again later in the semester. A rubric is used to score both assignments providing evidence of student achievement.

Critical Thinking

Key Learning Points

Common understandings

“Educational Effectiveness”

Evidence

Use of Evidence

Educational

Effectiveness

Fall Faculty Retreat 2006

Written and Produced byLeanne Neilson Halyna

Kornuta