Upload
theodora-watson
View
217
Download
0
Embed Size (px)
DESCRIPTION
RESULTS OF STUDY 1 FACE-to-FACEONLINE AMOUNT OF TALK GREATERLESS (than f2f) LEVEL OF CRITICAL THINKING LOW LEVELLOWER (than f2f) OTHER (chit chat) GREATERLESS (than f2f)
Citation preview
Can we improve the quality & quantity of online discussion with
rubrics and peer rating?
Leonard ShedletskyUniversity of Southern Maine
Department of Communication & Media StudiesAlice O. Goodwin, Research Assistant
University of Southern MaineDepartment of Communication & Media Studies
John BroidaUniversity of Southern Maine
Department of Psychology
17th Annual Sloan Consortium International Conference on Online Learning (Orlando, 2011)
RESULTS OF STUDY 1FACE-to-FACE ONLINE
AMOUNT OF TALK
GREATER LESS (than f2f)
LEVEL OF CRITICAL THINKING
LOW LEVEL LOWER (than f2f)
OTHER (chit chat)
GREATER LESS (than f2f)
RESULTS STUDY 1TRIGGERING EXPLORATIO
NINTEGRATION RESOLUTION OTHER
TYPE OF REPORT (INDIV. Vs. CONSENSUS)
> TRIG IN CONSENSUS
> EXPLOR. FOR CONSENSUS
NO DIFFERENCE
--------- > OTHER FOR CONSENSUS
TYPE OF ANALYSIS (EX. Vs. ABSTRACT)
> TRIG IN ABSTRACT
NO DIFFERENCE
NO DIFFERENCE
---------- > OTHER FOR ABSTRACT
RESULTS OF STUDY 1
• 1. GREATER AMOUNT OF TALK F-2-F BUT MOSTLY CHIT CHAT (LOW LEVEL OF CRITICAL THINKING);
• 2. ONLINE PRODUCED AN EVEN LOWER LEVEL OF CRITICAL THINKING & LESS TALK THAN F-2-F;
RESULTS OF STUDY 2
• Summaries from online students received higher grades from 2 professors blind to the study than summaries for F2F students.
RESULTS of STUDY 3
• The TA’s involvement in discussion had little to no effect on student-to-student interaction and only affected the level of EXPLORATION responses for critical thinking.
Study 4
• Study 4
Results of Study 4
• Personal Relevance of topics had no effect on student postings or critical thinking. Students’ self reported prediction of how personal relevance would affect their postings was not found in actual behavior observed.
Study 5
• STUDY 5
RESULTS OF STUDY 5WITH RUBRICS AND GRADING POSTS
WITHOUT RUBRICS AND GRADING POSTS
CRITICAL THINKING
INCREASED LOWER
INITIAL POSTS
MORE LESS
EARLIER POSTS
EARLIER LATER
STUDY 6:
• Can we improve the quality & quantity of online discussion with rubrics and peer rating?
• Since study 5 showed us that rubrics and grading posts increased critical thinking, produced earlier posts and more interaction, we wanted to find out if we could have the students rate one another.
STUDY 6: PROCEDURES & RESULTS
SPRING 2008 [NO RUBRICS, NO PEER RATINGS]
FALL 2009 [RUBRICS & PEER RATING]
CRITICAL THINKING
no difference
no difference
DAY OF INITIAL RESPONSE
earlier later
FREQUENCY OF RESPONDING
no difference
no difference
STUDY 7:
• Can we improve the quality & quantity of online discussion with rubrics and peer rating?
• Study 7 was undertaken to pursue further the question of whether or not peer rating of posts with a rubric plus emailing the rating and rationale for rating to the teacher would produce an effect on when posts were made in the week and an effect on the quality of posts (critical thinking).
STUDY 7: PROCEDURES
• We collected data from another section of the same course, (same teacher) with rubrics and peer ratings and rationale emailed to the teacher. In this third section of the course, students were asked to rate 1 to 5 other students’ posts and to email once a week to the teacher a copy of one of these posts, their rating and their rationale for the rating (see Appendix 3 at: http://media.usm.maine.edu/~lenny/appendix_3.docx
PROCEDURE
• We already had collected data on a section of Introduction to Communication– without rubrics or ratings of posts
• and a section of the same course (taught by the same teacher) with rubrics and graded posts
PROCEDURE• We wanted to know if rubrics, peer rating and emailing the
rated post-plus-the rating-plus-the rationale to the teacher would produce an effect on: – (1) when posts were made and on – (2) the quality of posts in the course as a whole;
• In effect, we wanted to know if the additional requirement of considering the rationale would improve scores in the class as a whole for critical thinking;
• In addition, in study 7, we did not grade the student for level of critical thinking or for day of the week for initial posts and responses;
RESULTS
• Three groups: – 1. No Rubrics/Ratings– 2. With Rubrics/Grading– 3. Email/Rationale to Teacher
• Overall, students used the lower end of the critical thinking scale for posts under all 3 conditions, though negligibly higher when graded or peer rated compared to no rubrics or oversight.
RESULTS: DAY OF INITIAL POSTA.Graded Class
B.Non-Graded/rated
Day of Initial Post
Day 5 A signif. earlier than B
Day 8
Day of Initial Post
A.Graded Class A signif.
Earlierthan C
C.Peer ratings+rubrics+rationale emailed
RESULTS: DAY OF INITIAL POST
• WHAT STANDS OUT IS THE MUCH LARGER NUMBER OF NON-POSTS IN THE NO RATING CONDITION THAN IN THE GRADED CONDITION
(287 TO 136 NON-POSTS, RESPECTIVELY).
RESULTS: INITIAL RESPONSESGraded Class
Peer rating + rationale+emailed
Day of Initial Response
EARLIERRESPONSE
sign.diff. LATER RESPONSE
AGAIN, far fewer non-posts for the graded condition than for the peer rating condition (136 vs. 205 non-posts, respectively).
RESULTS: INITIAL RESPONSESB. No Ratings
C. Peer rating + rationale+emailed
Day of Initial Response
LATERRESPONSE
sign.diff. EARLIER RESPONSE
Far fewer non-posts for the peer rating +rationale+emailed to teacher condition than for the no rating condition (205 vs. 287 non-posts, respectively).
Results: Overall
• For all 3 conditions there is a very low number of responses within the assigned week for discussion:– No rating—5.5%– Graded—48%– Peer rating/rationale/email—19.6%
Results: Email with Rationale
• 67 emails were received (105 possible total);• 4.2 mean emails out of possible 7• 14 of 15 students submitted email with rationale
in the 7-week course;• In short, the number of students who sent emails
with rating and rationale was poor;• The rationales offered did not clearly display use
of the rubrics (roughly 50% made reference to the rubrics);
EMAIL WITH RATIONALEStudent 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
# of emails
6 1 5 5 4 0 1 5 8 5 4 6 3 4 6
Discussion & Conclusions
• So—CAN WE IMPROVE THE QUALITY AND QUANTITY OF ONLINE DISCUSSION WITH RUBBRICS AND PEER RATING?
• WE CAN INFLUENCE – WHEN STUDENTS INITIALLY POST – WHEN STUDENTS RESPOND TO OTHER STUDENTS– HOW FREQUENTLY STUDENTS POST– SOME SMALL INCREASE IN CRITICAL THINKING
(Exploration)
Conclusions
• Some form of oversight is necessary, e.g., grading, having students email ratings to the teacher
• We could answer the title question with a ‘yes’ though the amount of critical thinking affected is low
SUGGESTIONS
• CHOOSE SOME FORM OF OVERSIGHT WITH EXPLICIT RUBRICS AND A REWARD PROCEDURE;
PROBLEM
• WE STILL HAVE NOT FOUND A WAY TO SUBSTANTIALLY INCREASE CRITICAL THINKING;
THE NEXT STUDY
• MAPPING—in study 8, which is underway, we will explore the effect of having students map with visual software, their argument (SeeTim van Gelder)—preliminary data suggest that this may have a strong effect but we will see;
MAPPING
APPENDICES
• APPENDIX 1 [STUDY 1, CRITICAL THINKING]• APPENDIX 2 [STUDY 5, RUBRICS]• APPENDIX 3 [STUDY 7, RATING+EMAILING]• APPENDIX 4 [RUBRICS]
REFERENCES
• Cases on Online Discussion and Interaction