How honest are the students when assessing each other’s work? Diogo Montalvão

1/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

How honest are the students when assessing each other’s work?

Diogo Montalvã[email protected]

Senior Lecturer, Dipl MSc PhD CEngSchool of Engineering & Technology

2012 Annual L&T Conference, 3rd May


1. Why student’s peer-assessment?2. Description of a case study at IST3. Some of the students’ works4. Peer-assessment results5. Conclusions

Outline


Learn from their peers

Encourages critical analysis of student work

Increased feedback to

students

Helps them perceiving their

own learning

Helps them interpreting assessment

criteria

Why student’s peer-assessment?Student benefits

source: http://www.heacademy.ac.uk/hlst/resources/a-zdirectory/peer_assessment

http://www.heacademy.ac.uk/hlst/resources/a-zdirectory/peer_assessment



Summative grading purposes

Reduced marking loads

To get students more involved and to make

them take more responsibility

Assess individual effort in team work

Engagement

Why student’s peer-assessment?Teacher benefits





Why student’s peer-assessment?

Disadvantages may be encountered when students:

• Lack the ability to evaluate each other

• Do not take it seriously

• Fear discrimination






Outline


Description of a Case Study at IST**Instituto Superior Técnico, Technical University of Lisbon

Modules:• Technical Drawing and Geometrical

Modelling I & II

• 1st year students• 2 groups/module, ~50 students/module

• BEng / MEng in Mechanical, Aerospace and Marine Engineering

• 2010/2011 session


Description of a Case Study at ISTTechnical Drawing and Geometrical Modelling I & II - Assessment

In-class tests30%

Group presenta-

tions10%

Group project40%

CAD* assign-ment10%

“Other” tests10%

*CAD – Computer Aided Design

In-class tests30%

Group presenta-

tions10%

Group project40%

CAD* assign-ment10%


“Other” tests 10%


(“Other” tests – 10%)Every week in-class 15~30 mins assignments marked from 0 to 1

0 1Wrong Right

Goals:• To engage the students in practice and study• Without significant increase of the marking load

How do we define the boundary in a binary marking scheme for drawings?

This slide is outside the main scope of the presentation (peer-assessment), but is shown for helping the author explain a concept.



0 1Wrong Very good


How to differentiate a student that does not deliver the assignment from the others?

.5Acceptable




0 1Zero


.5.25 .75Poor Accept. Good Very good

2 levels of fail 3 levels of pass

Problems:• Poor feedback• “Ethical” question: marking is based on comparison



(“Other” tests – 10%)Examples of “other” tests


1 0.75 0.5


Description of a Case Study at ISTTechnical Drawing and Geometrical Modelling I & II - Assessment

In-class tests30%

Grouppresenta-

tions10%

Group project40%

CAD* assign-ment10%


*CAD – Computer Aided DesignLecturer

5%Peer-assessment

5%

In-class tests30%

Public presenta-

tions10%

Group project40%

CAD* assign-ment10%


Group presentations

10%


Description of a Case Study at IST

• Done in-class, during last week

• 10~15 minutes/group

About the public presentation

• Students had to attend all the presentations

• Marks ranged from 1 to 10 (0 for no shows)

• Peer-assessment marks kept confidential by the lecturer

• Non-differentiating peer-assessment marks were rejected

1-2 9-105-63-4 7-8Very Poor Poor Accept. Good Very good



Outline


Mower

Rodrigo Ferreira

Boring mill

Wilson CarreiraHélder Camacho

Filipe Moreira

Some of the students’ works

Reactive suspension

Leonor NetoSoraia Ribeiro


Some of the students’ works

A Racing bike!!! (Solidworks)

(Author deliberately left anonymous)



Outline


Peer-assessment results

First observation:• Both the teacher’s and students’ marks follow a Gaussian

distribution.





The students did not fail their colleagues



The two students with very poor presentations are given a pass by their

peers...



... and the four students just below pass were given a comfortable pass by their peers



72% of the marks given by the

students are in the 7-8 (Good) range



However, the students seem to

be afraid to overrate too much their colleagues



• The averaged values are not very different• The teacher’s standard deviation is practically twice as much

as the peers’



The marks are concentrated around the same mode, although the teacher’s assessment “leaks” towards the whole marking range.



Outline


Conclusions

Overall conclusions from this study:

• Students have a tendency to overrate their “mates”

• They are able to make a correct relative judgement between their peers’ performances (and thus themselves)

• Most often, they will avoid to:• Fail their “mates”• Give their peers topper marks


Conclusions

How honest are the students when assessing each other’s work?

Students move by their own code of conduct and live in a competition environment.

Qualitatively honest Quantitatively biased


THANK YOU

Diogo Montalvã[email protected]

Senior Lecturer, Dipl MSc PhD CEngSchool of Engineering & Technology

Acknowledgments are due to Prof. António Ribeiro, Assistant Professor at IST, for sharing the data from two of their groups of students.

Documents

How honest are the students when assessing each other’s work? Diogo Montalvão