Assessing General Education: Options, Choices and Lessons Learned

Jo-Ellen Asbury, Ph.D.Rebecca Kruse

Office of Institutional Research and Assessment Stevenson University

Assessing General Education

We don’t have all the answers

We invite audience input and insights

We are not cheerleaders for national tests, it was the decision that we made at that time

No, we get no kick-back from ETS!

* We are not here to advocate use of the MAPP ETS Proficiency Profile or any specific test or assessment. We want to share our experience and generate a conversation.

2

Min. 16 academic courses in liberal arts and sciences and 1 course in phys ed. All students must complete the following:

Skills Courses: Three writing courses One communication course One physical education course Computer literacy requirement

Distribution Courses: One fine arts course Two social science courses Three math and science courses (at least one lab) Four humanities courses humanities

Core Electives (2 courses, 6 credits) Foreign Language (Bachelor of Arts only) 2 courses

3

SU Core Curriculum Requirements (Bachelor’s Degree)(General “Cafeteria” Style)

The problem:How to assess the General Education program

Unlike a major (psychology, math, etc.) does not have: A firm fairly prescribed list of requirements. A faculty member (or group of faculty members) who take sole responsibility for

oversight. A capstone project/paper/experience that could be used to assess student

learning outcomes.

Student learning outcomes for gen ed were evolving. Currently, no centralized oversight.

4

5

Individual Course-Based Approach Information collected about learning in individual courses. Faculty

demonstrate that students acquiring knowledge, skills, values associated with one or more gen ed goals. Assignments, exams, portfolios, etc.

Multicourse (Theme-Based) Approach Focus on faculty from number of disciplines rather than individual courses.

Review of syllabi, focus groups.

Noncourse-Based Approach Campuswide focusing on individual or groups of students rather than

courses. Gen ed assessment given to all or a sample of students. Standardized testing, student and alumni surveys, transcript analysis.

Possible General Education Assessment Approaches

Source: Palomba, C. A., & Banta, T. W. (1999). Assessing general education. In Assessment essentials: Planning, implementing, improving (pp. 239-268). San Francisco: Jossey-Bass.

6

Method(s) used needs to match learning goals Because gen ed programs include a broad range of

learning goals and objectives, critical thinking, communication, values, attitudes…. Need to be careful that the methods used will address all of these objectives

May need more than one method

Settled on some type of nationally-normed instrument.

Selecting a Gen Ed Assessment Method

Use of Published Tests / Assessments

Developed by testing professionals (test design, quality of questions better)

Can provide comparison data Provide detailed, diagnostic feedback Variety of published tests to reflect diversity among schools

and programs Longitudinal data confidence

7

Pros

~ from the paper, “The Role of Published Tests and Assessments in Higher Education”, March 2006, by Linda Suskie, MSCHE Vice President

Use of Published Tests / Assessments

8

Consider the distinct set of knowledge, skills and competencies your institutions seeks to instill and should be used in combination with other evidence of student learning.

Examples of Tested Writing Skills

Examples of Tested Critical Thinking Skills

ETS Measure of Academic Proficiency & Progress (MAPP)

Discriminate between appropriate and inappropriate use of parallelism.

Recognize redundancy

Evaluate competing casual explanations.

Determine the relevance of information for evaluating an argument or conclusion.

ACT Collegiate Assessment of Academic Proficiency (CAAP)

Formulate an assertion about a given issue.

Organize and connect major ideas.

Generalize and apply information beyond the immediate context.

Make appropriate comparisons.

Council for Aid to Education Collegiate Learning Assessment (CLA)

Support ideas with relevant reasons and examples.

Sustain a coherent discussion.

Deal with inadequate, ambiguous, and/or conflicting information.

Spot deceptions and holes in the arguments made by others.

“The Role of Published Tests and Assessments in Higher Education”Linda Suskie, Middles States Commission on Higher EducationMarch 25, 2006

Use of Published Tests / Assessments

If no compelling incentive, students may not give best effort. Challenge to get students to take and to give best effort.

Published tests for higher ed have less evidence of quality than K-12 tests. Smaller # of students, may not be representative, less funds, etc.

Certain published tests may not yield enough useful feedback .

9

Cons

from “The Role of Published Tests and Assessments in Higher Education”, Linda Suskie, Middles States Commission on Higher Education, March 25, 2006

Use of Published Tests / Assessments

Match goals for student learning set by the institution Specific content must correspond with institution’s concepts

(how does institution define critical thinking for example) Provide rich, detailed feedback that can be used to identify

areas for improvement Have evidence of validity and reliability Provide some incentive for students to do their best

10

Chosen Assessment Should:

from “The Role of Published Tests and Assessments in Higher Education”, Linda Suskie, Middles States Commission on Higher Education, March 25, 2006

ETS Proficiency Profile / MAPP TestSelected MAPP by ETS: Measure of Academic Proficiency and Progress. (now called ETS Proficiency Profile)

Corresponds well with university core and measures what we want to measure

Several different formats to choose from (online, standard, abbreviated)

Can add up to 50 of our own supplemental questions Rich reporting features including comparative data and diagnostic

feedback, norm-referenced scores and criterion-referenced scores SU has changed so rapidly and is still changing – important for us

to be able to do comparisons, benchmarking, see differences between cohorts, etc.

11

More on ETS Proficiency ProfileMeasure of Academic Proficiency and Progress

(now called ETS Proficiency Profile…)

Assesses four core skill areas – critical thinking, reading, writing and mathematics at three levels

Measures academic skills developed, as opposed to subject knowledge taught, in general education courses

12

Results and Reporting Multitude of reporting options available

Comparison between cohorts/subgroups (separate out specific groups - majors, schools w/in University, commuters vs. noncommuters, etc. Can ask different cohorts different suppl. questions.)

Identify specific proficiency level (1-3) of core skill deficiencies (ETS has specific definitions at each level)

External and internal benchmarking

Value-Added – compare against other metrics such as GPA, SAT, etc.

Identify patterns (e.g. do students do better in certain areas if certain courses are taken in a certain order? Etc.)

13

Next Issue: Schedule for Administration

14

Pre-Post Test

Test students when then enter, then test again at a later point in their Stevenson career. WHEN should the second testing take place? Internal validity threats

History Maturation Mortality Selection Testing

15

Cohort-Sequential Design

Compensates for (most of) the internal validity threats

Provides both between subject and within subject data.

16

How the cohort-sequential design is being used SU

COHORT #1 (entered F ‘08) COHORT #2 (enter F ‘10)

AY 2008-2009 Fall, 2008

AY 2009-2010 Spring, 2010

AY 2010-2011 Fall, 2010

AY 2011-2012 Spring, 2012

17

Next Issue: Planning for Administration

18

Our Plan

19

Administer to incoming freshman

Test same students again in the end of sophomore year

Freshmen

20

How do we get a large number of freshmen to take the test? Commitment from Director of First Year

Experience to administer in First Year Seminars (all incoming freshman take a FYS)

Goes on the syllabus Peer leaders (not us) to administer

Freshmen – Issues/Challenges

21

Test version? (long, abbreviated, online) 2007 – used long version (2 hrs) switched to abbreviated (40

mins) Cost (tests, materials) Student leader instructions for administering

Very specific instructions / script Customize instruction book

Materials to and from student leaders Tests, pencils, instructions, ID Cards, calculators

Retesting as Sophomores

22

384 freshmen took in fall 2008

Where and how can we test that amount of students now as sophomores?

Do we test all 384 at same time on same day in same location? Do we have the room on campus?

Do we have enough supplies to test all at one time? What’s the best time during the semester? Who would proctor the tests? How do we get sophomores to volunteer to take

test? No way to capture – no one class that all take.

Recruiting Sophomores

23

Used to use scholarship hoursPizza lunchGift card drawingsOffered choice of two different days Marketed through emails, plasma screens in

student union, faculty

Recruiting Sophomores

24

A week before, response still not greatAdded more gift cardsOpened up to ALL sophomores, not just ones

who took it as freshmen46 students out of 384 signed up27 showed up split between both days

Other Recruiting Ideas: Gift certificates or pay for all students who take the

test Change test format – use online format Reward those with high scores so test is taken

seriously and they do their best ETS reports that most effective is combination of extrinsic and

academic reward – something to get them there and something to get them to take it seriously

Put high scores on an honor roll Make it a requirement for registration for junior year Withhold grades until test is taken

25

Latest Plan

26

Try online non-proctored version.

Recruit 100 random students from the 384 tested as freshman in 2008 who didn’t retake it in spring.

Give each one $10 gift card to take online

Data ReceivedClosing the Loop/Going Forward

27

28

Cohort 1: Summary of Stevenson University Proficiency Classifications (natl. comparison in parenthesis)

  Proficient - Freshmen (FA08)

Proficient - Sophomore

(SP10)

Marginal - Freshmen (FA08)

Marginal - Sophomore

(SP10)

Not Proficient - Freshmen (FA08)

Not Proficient - Sophomore

(SP10)

Reading Level 1 56% (57%) 67% (59%) 24% (22%) 26% (23%) 20% (20%) 7% (18%)

Reading Level 2 26% (28%) 44% (27%) 19% (19%) 26% (20%) 56% (53%) 30% (53%)

Reading Level 3 (Critical Thinking) 3% (4%) 7% (3%) 9% (12%) 7% (10%) 88% (84%) 85% (86%)

Writing Level 1 62% (61%) 70% (59%) 26% (25%) 26% (28%) 12% (14%) 4% (13%)

Writing Level 2 15% (16%) 19% (14%) 38% (35%) 37% (34%) 48% (49%) 44% (52%)

Writing Level 3 6% (7%) 4% (5%) 24% (25%) 30% (24%) 70% (68%) 67% (71%)

Math Level 1 52% (50%) 59% (45%) 26% (28%) 22% (29%) 22% (22%) 19% (26%)

Math Level 2 26% (25%) 22% (19%) 26% (26%) 37% (26%) 48% (49%) 41% (54%)

Math Level 3 7% (6%) 7% (4%) 14% (15%) 11% (12%) 78% (79%) 81% (84%)

29

Cohort 1: Distribution of Individual Student Scores and Subscores

 Possible Range

SU Mean Score

Freshmen (FA08) n=380

Natl. Comparison (Freshmen)

SU Mean Score

Sophomore (SP10) n=27

Natl. Comparison (Sophomore)

SU Score Increase/ Decrease

(pts)

SU Score Increase/

Decrease (%)

Total Score 400-500 439.81 441.1 443.41 439.6 3.60 0.82%

Skills Subscores:              

Critical Thinking 100-130 110.23 110.3 110.26 110.0 0.03 0.03%

Reading 100-130 116.34 117.1 118.89 117.1 2.55 2.19%

Writing 100-130 113.75 113.8 114.67 113.5 0.92 0.81%

Mathematics 100-130 112.80 113.0 112.81 112.0 0.01 0.01%

Context-Based Subscores:              

Humanities 100-130 113.59 113.9 114.30 113.8 0.71 0.63%

Social Sciences 100-130 112.15 112.6 112.59 112.5 0.44 0.39%

Natural Sciences 100-130 114.17 114.0 115.93 113.9 1.76 1.54%

Closing the Loop/Going Forward

Determine the mechanism for internal decision-making and the process used for identifying deficiencies and implementing change

Share results

Other measures of same core skills

Content mapping

30

Other Ideas for….

- assessing general education?- recruiting students?

- using data and closing the loop?- other?

31

Suskie, L. (2006, March 25). The role of published tests and assessments in Higher Education. In Middle States Commission on Higher Education [Report]. Retrieved from http://www.msche.org/publications/ published-instruments-in-higher-education.pdf

ETS® Proficiency Profile Case Studies. (2008). Educational Testing Services. Retrieved from http://www.ets.org/proficiencyprofile/case_studies/

ETS® Proficiency Profile Content. (n.d.). Educational Testing Service. Retrieved from http://www.ets.org/proficiencyprofile/about/content/

Walvoord, B. E. (2004). For general education. In Assessment clear and simple: A practical guide for institutions, departments and general education (pp. 67-79). San Francisco: Jossey-Bass.

Palomba, C. A., & Banta, T. W. (1999). Assessing general education. In Assessment essentials: Planning, implementing, improving (pp. 239-268). San Francisco: Jossey-Bass.

32

References