Assessing Quality and Sharing Best Practices for the Undergraduate Senior Thesis

Collaboration Across Three Liberal Arts Colleges

Student Success and the Quality Agenda, AAC&U 2013

Teagle Senior Thesis Evaluation

• Capstone as most significant student artifact

• No systematic evaluation• Developed Senior Thesis Rubric• Performed cross-institution thesis

evaluation• Administered Thesis Experiences

Survey• Will perform second thesis

evaluation on individual campuses using rubric

Collaborators New College of Florida Bard College Bennington CollegeColorado CollegeHampshire CollegeSmith CollegeWellesley College

Key Questions for Thesis Evaluation

• Are there common evaluative criteria across analytic disciplines? Can a common rubric be developed?

• Can a rubric improve on the validity, reliability and systematicity of grades for evaluating analytic work?

• Can using a rubric help institutions improve pedagogy?

• To what extent do student input characteristics predict high-quality senior theses?

• Can we identify student experiences that predict high-quality theses?

• Do the same experiences lead to high-quality work no matter whether theses are required, optional, or honors-only?

Timeline at Bard College

• June 2009: first reading/evaluation of senior theses using the rubric. Three Bard faculty participated.

• June 2010: full consortium meeting at Bard. Faculty discussed results from the June 2009 reading and planned for improvement.

• February 2012: meeting at Bard for Bard faculty to further disseminate the work of the grant. Led by Steve Weisler and Carol Trosset in conjunction with Bard Center for Faculty and Curricular Development. Steve and Carol also held a focus group with students.

• April 2012: meeting at Bard with faculty in the performing and practicing arts from Bard, Bennington, and New College. Explored possibilities for rubrics in the arts.

• May 2012: first survey of graduating seniors at Bard about their Senior Project experiences. The response from the class of 2012 was encouraging. ~160 students responded.

Results at Bard College• The results from the June 2009 meeting showed that Bard senior theses

were not theses that stand alone well with respect to the rubric, in comparison with the other colleges in consortium.

• The June 2010 meeting produced enough momentum to focus the discussion of the Senior Project on campus for first time in many years. Indeed, the discussion has shifted away from anecdotal stories about individual students to a discussion focused on talking about how we prepare and evaluate all students.

• A number of programs have instituted practices that help students prepare for the Senior Project and provide a scaffolding for students as they progress through the Senior Project. In addition, several programs are experimenting with rubrics to advise and evaluate students.

• Take home lesson for Bard: make the Senior Project more like a course.

New College of Florida Senior Thesis Practices

• Public honors college, narratives instead of grades, contracts not credit hours

• Senior thesis/project required of each student. • 6th semester Thesis Prospectus form• Thesis advisor and 2 additional faculty serve as

Baccalaureate Committee for oral exam. • Thesis is a major focus of final semester.

Results:NCF Assessment and Priorities for Discussion

2009 Thesis Assessment Results

• In 2009 Thesis Assessment, New College compared favorably with others

• New College theses weakest areas Scholarly Context and Position

2012 Faculty and Student Priorities for Discussion

• Students & Faculty: Preparing students for thesis success before thesis year

• Faculty: Managing thesis workload

• Students: regular meetings with advisor, time to revise drafts, support for interdisciplinary and non-standard projects

Evolving NCF Thesis Discussions

• Initial skepticism/resistance to rubric assessments

• Later, rubric seen as possible tool: set expectations, guide theses, replace SAPA

• Positive and energizing conversations with Bard and Hampshire Faculty

• Hampshire’s workload depiction and assigning advisors

• Bard’s midway and moderation practices• Reinvest in “projects” as well as “theses”

NCF Enduring Insights

• Learning process can be highly successful even when the senior thesis itself is disappointing.

• No matter how well we prepare students, there will always be a distribution from low to high quality.

• There is tension in expecting students to be flexible, independent, risk-takers and also meet honors standards for senior thesis.

Thesis Rubric

Introduction: Rational

Most theses get A’s.

Grade Number of theses Percent of theses

A 185 66%

A- 54 20%

B+ 28 10%

B 7 2%

B- 2 1%

C+ 1 0.3%

C 1 0.3%

D 1 0.3%

Using a Rubric Diversifies and Focuses the Evaluation Process

Level 1 Level 2 Level 3 Level 4 MeanRationale 8% 36% 41% 13% 2.6Complexity 13% 37% 38% 12% 2.5Approach 7% 46% 29% 8% 2.3Context 18% 43% 33% 6% 2.3Position 14% 51% 25% 10% 2.3Argument 13% 44% 37% 6% 2.4Evidence 6% 41% 42% 10% 2.6Insight 11% 47% 33% 8% 2.4Usage 1% 10% 57% 31% 3.2Organization 6% 37% 45% 11% 2.6Style 6% 37% 47% 10% 2.6

Scores Can Show Institutions How to Improve

A B C D E F G

Rationale 2.1 2.5 2.4 2.6 2.9 2.8 3.0

Complexity 1.9 2.5 2.1 2.6 2.7 2.7 2.8

Approach 1.7 2.1 2.0 2.3 2.6 2.5 2.7

Context 1.7 2.2 2.1 2.4 2.5 2.4 2.6

Position 1.7 2.3 2.0 2.2 2.5 2.5 2.8

Argument 2.0 2.2 2.1 2.4 2.6 2.6 2.7

Evidence 2.2 2.4 2.3 2.6 2.7 2.7 2.9

Insight 1.9 2.5 2.1 2.4 2.6 2.5 2.7

Usage 3.0 3.0 3.1 3.2 3.4 3.3 3.4

Organization 2.3 2.5 2.3 2.8 2.8 2.7 2.9

Style 2.3 2.4 2.4 2.7 2.8 2.9 2.9

Grade 2.51 2.99 2.76 3.10 3.48 3.47 3.51

Grades and Self-Rated Quality

External readers are more discriminating.

Student judgments correlate with external readers more than with grades.

Do inputs predict quality?

Are theses better at more selective institutions?

Different experiences are more common at different institutions.

Smallest Value

Largest Value

Average # courses taken while writing thesis 2.9 5.8

Worked on thesis 6 or 7 days in a typical week 10% 32%

Worked on thesis 1 to 3 days in a typical week 22% 51%

Worked on thesis more than 6 hours per day 13% 34%

Worked on thesis less than 4 hours per day 26% 54%

Met with adviser weekly 52% 100%

Topic was closely related to adviser’s research 7% 53%

Student usually follows adviser’s advice 19% 48%

Thesis topic was settled during junior year 24% 63%

Student plans to seek an external audience 19% 50%

What makes for a good thesis?

Predicted Grade

• Student felt prepared to find and evaluate relevant scholarship

• Student felt prepared to fairly consider conflicting points of view

Predicted Self-Rated Quality

• Student felt prepared to form their own position or argument on a subject

• Student felt prepared to manage their own time

• Hours per day spent on the thesis

• Student usually took the adviser’s advice

The Best Predictors of Quality

Only two survey items predicted* both the adviser-assigned grade and self-rated quality:– Student views the thesis as the most

important work done in college– Student feels challenged by the adviser to do

his/her best work*in a stepwise regression equation, for more than one institution

Next Steps

• Survey will be administered again this spring.• Each institution will conduct a thesis-reading

applying the rubric to theses written by students who completed the survey.

• A new analysis will discover whether particular experiences predict higher rubric scores for the same theses.

Discussion