Erin Baumgartner & Ava Howard Biology Department

Western Oregon University

Data Driven Decision-Making

How do you get a group of passionate faculty with diverse backgrounds to agree on instructional practice?

DATA!

For Example…• What’s better for students- many small or few large exams?

• What’s better for students to have the same lecture and lab instructor or different instructors?

• What’s better for students – multiple choice or open-ended assessments?

• How do we improve the laboratory experience?

• How consistent are exam practices between instructors? Does it matter?

• Is there a difference between 4-item or 5-item response choices on multiple choice questions?

• How are we serving non-traditional, underrepresented, and other student groups?

Hypothesis:

• More frequent, smaller exams will improve student study habits and performance and decrease anxiety.

Exam Size & Frequency

• Control-comparison– Two midterms, 90 points, 50 minutes– Four miniexams, 45 points, 25 minutes

• Quantitative: – Embedded indicator questions on exams– Student survey – Likert response scale

• Qualitative: – Student comments

Methods:

• Decrease anxiety – No • Improved study habits – No• Improved performance – Minimal

… But students love them

Results:

ConclusionsConclusions:

• Students with same lecture and lab instructor gain more content knowledge and improve attitudes about science.

Lecture and Lab Instructors

Hypothesis:

• Naturally occurring comparison groups– Students with same lecture-lab– Students with different lecture-lab

• Quantitative: – Proportion of correct answers on concept inventory– Student attitude survey – Likert response scale– Two-sample t-test overall comparison

• MANOVA across demographic groups

Methods:

same lecture-lab different lecture-lab0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Average Post-Course Content Scorep = 0.7481; t = 0.3215 w/265 df

Condition

Aver

age

perc

ent s

core

same lecture-lab different lecture-lab0

1

2

3

4

5

Average Post-Course Attitude Scorep = 0.3453; t = 0.9456 w/237 df

Condition

Aver

age

Liker

t Sco

re

Results:

ConclusionsConclusions:

• Students perform better on open-ended assessment items.

Open Ended:What is a sphincter?Identify a place in the body where you might find one.

Multiple Choice: What is the role of the sphincters in the digestive system?

Hypothesis:

Types of Exam Questions

A. Secrete enzymesB. Prevent food from moving too quickly

or moving backward through the system

C. Keep food moving through the systemD. Adjust pH in different parts of the

system

• Within group comparison – Exams consisted of multiple choice and open-

ended items (short-answer, fill-in-the-blank, labeling, essay)

• Quantitative: – Correlation of performance on item type against

overall performance

Methods:

20 30 40 50 60 70 80 90 1000

10

20

30

40

50

60

70

80

90

100

Exam item type score correlated with total exam score

Percent MC Score

Linear (Percent MC Score)

Percent written score

Linear (Percent written score)

Overall exam score

Scor

e on

exa

m it

em ty

pe

Pearson correlation coefficient = 0.855

Pearson correlation coefficient = 0.900

Results:

Question Type Points each # on exam total points % allocation notes

multiple choice (MC)

2 16 32 53% 2 points per item

Matching 1 12 12 20% 1 point per item, combined into appropriate number of questions

Fill in the blank or label diagram

2 6 12 20% can be combined into fewer questions; or split- single word answers can be worth 1 point, while short (phrase or one sentence) answers worth 2

Open Ended (Short answer/ Drawing/ Essay)

4 1 4 7% can be divided into multiple parts or combined into one question with several elements

TOTALS:   35 60 100%  

Conclusions:

5 items?1. A carefully formulated scientific

explanation that is based on a large accumulation of observations and has been tested and supported by multiple scientistsa. Controlb. Theoryc. Hypothesisd. Facte. Postulate

4 items? 1. A carefully formulated scientific

explanation that is based on a large accumulation of observations and has been tested and supported by multiple scientistsa. Controlb. Theoryc. Hypothesisd. Fact

• Morrison, Susan and Free, Kathleen. (2001). Writing multiple-choice test items that promote and measure critical thinking. Journal of Nursing Education 40, 17-24.

• Rodriguez, M.C. (2005). Three Options are Optimal for Multiple-Choice Items: A Meta-Analysis of 80 Years of Research. Educational Measurement: Issues and Practice, 24(2), 3-13.

Dissemination is Important

IRBs are Important

A Consistent Framework is Important

• Lindsay Biga• Karen Bledsoe• Jim Dawson• Bryan Dutton• Emma Dutton• Amy Harwell• Kelly Kissane• Kristin Latham• Liz Martin• Jeffrey Snyder

Julie Grammer

Thanks!