Embed Size (px)
Citation preview
The Metacognitive Benefits ofSelf- and Peer Review
Edward F. Gehringer
Department of Computer ScienceNorth Carolina State University
Our work in peer and self-review is funded by Google and NSF
http://tinyurl.com/CHEP-SPR
Signup sheet Name E-mail address Institution, if not in your domain name
Outline What is metacognition? Advantages of peer review? Advantages of self-review? Self-review Peer review Combining peer and self-review Reviewing vs. being reviewed Accuracy of peer grading
Gehringer, Benefits of self- and peer review [email protected]
http://tinyurl.com/CHEP-SPR
What is metacognition?
Gehringer, Benefits of self- and peer review [email protected]
http://tinyurl.com/CHEP-SPR
What is metacognition? “Thinking about one’s thinking” I am engaging in metacognition if …
I notice I am having more trouble learning A than B.
it strikes me that I should double-check C before accepting it as fact.
I realize I don’t understand what the instructor wants me to do
if I sense that I should write down D so I don’t forget it
–John Flavell, 1976
Gehringer, Benefits of self- and peer review [email protected]
http://tinyurl.com/CHEP-SPR
Advantages of peer review?
Gehringer, Benefits of self- and peer review [email protected]
http://tinyurl.com/CHEP-SPR
Some advantages of peer review Feedback is
more extensive quicker scalable
Peers may understandstudents’ comprehension problems better and communicate in language more easily
understood Can’t blame the reader! Forces students to think metacognitively
Gehringer, Benefits of self- and peer review [email protected]
http://tinyurl.com/CHEP-SPR
Advantages of self-review?
Gehringer, Benefits of self- and peer review [email protected]
http://tinyurl.com/CHEP-SPR
Some advantages of self-review Every student gets
immediate feedback Judging correctness
of answers deepensstudents’ understandingof material
Students become awareof own strengths, progress, gaps
Gehringer, Benefits of self- and peer review [email protected]
http://tinyurl.com/CHEP-SPR
Self-review: pitfalls Confounding effects
Early studies asked students to predict how well they would do in course.
Effort is often included as a criterion. What is a good
self-assessment? Is it the same kind of
assessment an instructor would give?
Gehringer, Benefits of self- and peer review [email protected]
http://tinyurl.com/CHEP-SPR
Rubrics Why use a rubric?
Tell students what to look for “Fairness” in assessment
Students can helpcreate the rubric
How detailed?
Gehringer, Benefits of self- and peer review [email protected]
http://tinyurl.com/CHEP-SPR
Rubric advice
http://tinyurl.com/CHEP-SPR
Self-review vs. self-grading Self-grading is more effective on items
requiring lower-order cognitive skills. Freshmen vs. seniors …
Would students cheat??? … Wilkowski et al., 2014
70.3% of students in 1st MOOC awarded selves full credit
9.9% of these submissions were blank or nonsense
8.5% were evid. plagiarized
Gehringer, Benefits of self- and peer review [email protected]
http://tinyurl.com/CHEP-SPR
Combining self-review & peer review
was the first review system to incorporate both peer and self-review.
Step 1: Instructor gives students a writing assignment.
Step 2: Students submit their document to CPR.
Gehringer, Benefits of self- and peer review [email protected]
http://tinyurl.com/CHEP-SPR
Calibration Step 3: CPR gives the student 3
calibration documents for review. Basic idea: Training course for reviewers
How they do how much credence they get Before students review peers, they get 3
works to review 1 exemplary
Their agreement with instructor Reviewer Competency Index
Others have known defects
Gehringer, Benefits of self- and peer review [email protected]
http://tinyurl.com/CHEP-SPR
How Does CPR Work?
Step 4: After calibration, CPR gives student 3 peer documents for review. Step 5: CPR gives the student his or her document for self-review. Step 6: CPR provides a detailed report of the peer review and the self-review.
Gehringer, Benefits of self- and peer review [email protected]
http://tinyurl.com/CHEP-SPR
The Review Process
The review process is the same for calibration, peer review, and self-review. The student answers content questions for each document. The student answers style questions for each document. The student assigns a score to each document on a scale of 1 to 10.
Gehringer, Benefits of self- and peer review [email protected]
http://tinyurl.com/CHEP-SPR
How self-review fits in Students get full credit for self-reviewing if
their grades agree within the range set by the instructor
between their self-assessment and the weighted value of their peer reviews.
Instructor is notified to check reviews/self-assessment when reviewers of a student did not “train well.”
Gehringer, Benefits of self- and peer review [email protected]
http://tinyurl.com/CHEP-SPR
Better to give than to receive? Would students learn more by …
peer-reviewing, or being peer-reviewed?
Gehringer, Benefits of self- and peer review [email protected]
http://tinyurl.com/CHEP-SPR
What did students appreciate?
Kulkarni et al., 2013, ACM TCHIGehringer, Benefits of self- and peer review [email protected]
http://tinyurl.com/CHEP-SPR
Other results
Shah-Nelson, 2014, OLC Annual ConferenceGehringer, Benefits of self- and peer review [email protected]
http://tinyurl.com/CHEP-SPR
“To give is better than to receive”
“Givers”: reviewed, but received no peer feedback “Receivers”: got peer feedback, but didn’t review “Receivers” made more signif. gains in writing,
especially those at lower proficiency levels.
Gehringer, Benefits of self- and peer review [email protected]
http://tinyurl.com/CHEP-SPR
Peer review vs. peer grading Formative—text feedback Summative—Likert scale Should peer review be used summatively?
Gehringer, Benefits of self- and peer review [email protected]
http://tinyurl.com/CHEP-SPR
Peer grading—how reliable? Two studies on Coursera MOOC [2013]
Piech et al.: ≥ 26% of grades ± 5% from “ground truth.”
Kulkarni et al.: 40% of grades off by 1 letter grade!
But … simplistic calibration this was, after all, a MOOC
Gehringer, Benefits of self- and peer review [email protected]
http://tinyurl.com/CHEP-SPR
Factors affecting accuracy Students can only effectively peer-review
what they understand. This suggests
overweighting scores assigned by better-performing students.
Gehringer, Benefits of self- and peer review [email protected]
http://tinyurl.com/CHEP-SPR
Summary Metacognition: thinking about thinking Peer review: rapid, extensive feedback Self-review: deepens understanding Self-grading: don’t use by itself! Can combine peer & self-review Learn more as reviewer than reviewee Peer grading: not ready for prime time
Gehringer, Benefits of self- and peer review [email protected]
http://tinyurl.com/CHEP-SPR
