Comparative effectiveness of research ethics teaching methods

Michael Kalichman and Dena Plemmons

UC San Diego

Research on Research Integrity Annual Meeting

Niagara Falls, NY

May 16, 2009

Goal

Assess effectiveness of teaching research ethics

Challenges Different teaching objectives Different institutions and audiences Different instructors

Assess relative effectiveness of different methods for teaching research ethics

Approach: The Course

The course: Scientific Ethics (UC San Diego) 10 sessions, 1-1/2 hrs per week 3 sections each week 2 instructors (DP, MK)

Teaching Methods: Week 1: Lecture Weeks 2-10

Lecture + Small group case discussion Lecture + Role play Case-based Lecture

Approach: The Students

Students Graduate Students, some postdocs Biology, Neurosciences,

Other (e.g., Bioengineering) UC San Diego,

Other (Salk, The Scripps Research Institute) Number

Total = 57 18 – 20 assigned per section

Approach: Randomization

Week Section 1 Section 2 Section 3

1 MK Introductory Lecture

2 DP L C R

3 MK C R L

4 DP R L C

5 MK L C R

6 DP C R L

7 MK R L C

8 DP L C R

9 MK C R L

10 DP R L C

L = Lecture + small group case discussion

R = Lecture + role-playC = Case-based lecture

Approach: Outcomes

1. Pre-Test: 18 multiple choice questions (2 for each of 9 topics)

2. Weekly Quiz: 5 multiple choice questions 3 attitude questions about the topic 2 attitude questions about the survey (enjoyable?

useful?)

3. Post-Test: Same as Pre-Test

4. Final Evaluation: Satisfaction, Perspectives

Overall Impact of Course: PerceptionsHow if at all have your awareness, knowledge, skills,

or attitudes been changed by participating in this course?

“not at all”

“not much”

Overall Impact of Course: PerceptionsHow if at all have your awareness, knowledge, skills,

or attitudes been changed by participating in this course?

“great increase in knowledge” “definitely more aware and now know options available” “the course stressed topics that are easy to ignore, …

useful for recognizing similar situations before it's too late”

Overall Impact of Course: PerceptionsHow if at all have your awareness, knowledge, skills,

or attitudes been changed by participating in this course?

“…I will be able to more effectively deal with many types of situations”

“learned new strategies for dealing with complex ethical issues”

“I am now better prepared to deal with problems like the ones discussed in class”

Overall Impact of Course: PerceptionsHow if at all have your awareness, knowledge, skills,

or attitudes been changed by participating in this course?

“It has helped me to see other perspectives and change my mind on several topics. ”

“it increase[d] my moral outrage”

Sample Question #1

Research data are the property of:A. The institution which employs those who are

collecting the data.

B. The person(s) who collect the data.

C. The head of the research group that collects the data.

Sample Question #2

The primary basis for requirements for review of human subjects research is in regulations created by the

A. State government

B. Federal government

C. University

Overall Impact of Course: Knowledge Knowledge improved (P<0.001) However, improvement was modest

Pre-test median = 12.0 Post-test median = 13.0

Effects of Methods

Knowledge:No statistically significant difference among methods for any of the 9 test weeks.

Attitudes:No statistically significant difference among methods for any of the 9 test weeks.

Perceptions of Methods

Useful or Enjoyable? No statistically significant difference among methods for 5

of the 9 test weeks. In 4 of the 9 test weeks, Lecture+Roleplay judged to be

more enjoyable than Case-Based Lecture and/or Lecture+Case Discussion

In 3 of the 9 test weeks, Lecture+Roleplay was judged to be more effective than Case-Based Lecture and/or Lecture+Case Discussion

However, when different methods compared across all weeks (2-factor ANOVA), no effect of method.

Student Preference - Methods Which of the methods did you find to be most

useful for meeting the goals of this course?

05

1015202530354045

L R C Any Noopinion

None

Student Preference - Methods Which of the methods did you find to be most

enjoyable for meeting the goals of this course?

05

1015202530354045

L R C Any Noopinion

None

Student Preference - Methods In future courses, would you recommend using:

0

10

20

30

40

50

60

70

80

Only one Mixture Some other No opinion

Student Preference - Comments

“I really disliked the role-playing.I didn’t think it was beneficial at all.”

13 of 51 respondents (>25%) specifically commented on not liking the role-play exercises

Summary and Conclusions

Summary Student perceptions: positive impact on knowledge, skills,

and attitudes Knowledge improved, but is it worth the cost? No difference in methods for knowledge or attitudes Lecture+roleplay considered more enjoyable and/or useful

during several weeks of course, but least liked overall at end of course

Mixed methods preferred

Conclusions1. Impact of course on attitudes needs to be assessed2. Teachers may be more important than methods