An experience of peer evaluation

in a b-learning environmentEliana Scheihing

Univ. Austral of Chileescheihi@inf.uach.cl

Julio Daniel GuerraUniv. Austral of Chile

& University of Pittsburghjguerra@inf.uach.cl

Sergio BustamanteUniv Austral of Chile

bustamantemsergio@gmail.com

EDMEDIA 2014, 23 to 26 juin 2014 Tampere-Finland

The Kelluwen b-learning community

Who participate at Kelluwen?

Teachers and students open to innovation processes in their classes

From 2010 we work with a community 3 regions (Los Ríos, Los Lagos y Aysén) 17 cities 57 schools 93 teachers and 4517 students 160 classes from Valdivia to Aysén

Which is the proposal of Kelluwen?

Social Web

Socio-communicative

skills

Collaborative learning

Which is the opportunity that address Kelluwen?

Motivation of the students with the Social Web tools

Large inversion in technologic infrastructure in the schools but underused

Low development of socio-communicative skills in the students

How we contribute to the scholar context?

Motivating the creation of a learning

community

Coordinating scholar networks by means of a didactic proposal (Didactic design) and a supporting Web platform

The peer review module in Kelluwen platform

The Didactic Designs (DDs) consider students working in teams within each classroom

For some activities within the DDs, we promote that teams from different schools and different geographical locations do peer review activities

Coordination between different schools is challenging, and to support this process we implemented a peer review module, called Works Tool

The peer review module in Kelluwen platform: teacher’s view

The peer review module in Kelluwen platform: teacher’s view of reviews

Imagen con la pauta de evaluación

An example of peer review

Qualitative study of the peer review module

Sample of the survey questions for students (top) and teachers (bottom).

Classroom and data

Class School Nº of Students

Nº of teams

Nº of Publications

Nº of received reviews

Nº of sent reviews

1ºA Colegio Adventista

39 10 32 17 20

1ºA Liceo Aysén 42 18 54 15 0

1ºC Liceo Inglés 31 6 13 7 12

1ºB Colegio Adventista

45 9 23 22 28

1ºA Colegio Helvecia

37 12 21 23 24

Totals 194 55 143 84 84

Didactic Design 21: Literary and non-literary in Youtube

Classroom and data

Class School Nº of Students

Nº of teams

Nº of Publications

Nº of received reviews

Nº of sended reviews

1ºA Colegio Darío Salas 38 7 16 14 14

4ºM Liceo Río Bueno 25 5 6 10 10

1ºB Colegio San Francisco

25 5 36 10 10

1ºA Colegio Emprender 42 9 19 18 18

4ºA Liceo Rayen Mapu 36 7 14 14 14

1ºM Alianza Hospitalaria 4 1 1 2 2

4ºM Colegio Austral 20 8 13 16 16

Totals 190 42 105 84 84

Didactic Design 88: Building a slideshow of the Twentieth Century

Quantitative study of the peer review module

DD Classes Different schools

Within class reviews

Between class reviews

Valid comments

within class

Valid comments between classes

21 5 4 32 52 30 52

88 7 7 28 56 28 55

Analysis and results

DD Number of questions

Correlation of scores by question

Number of products Correlation of scores by product

21 138 0.0503 31 -0.1115

88 245 0.1161 45 0.1817

Correlations between two reviewers

Analysis and results

DD Same class Twin class p-value Wilcoxon

testNºrev Min 1st qu. Med. 3th qu. Max Nº rev Min 1st qu Med 3th qu. Max

21 32 1.25 2.47 2.9 3.75 4 52 1.2 2.4 2.75 3.05 4 0.0943

88 30 2.6 3.4 3.6 3.9 4 56 2 3 3.4 3.8 4 0.0818

p-values of the Wilcoxon test to contrast localization parameters of score distributions

H0: Ls = Lt vs H1: Ls > Lt

DD Assessment Feedback Form

21 0.879341 0.759162 0.594582

88 0.896736 0.739674 0.488020

Analyzing quality of commentsTwo independent judges evaluated the comments with respect

to three criteria: assessment, feedback, and form with a four level rubric:

Weighted Kappa statistics to measure agreement between judges

Not made Fairly achieved achieved Fully achieved

1 2 3 4

Assessment criteria Feedback criteria Form criteria

DD Judge 1 Judge 2 Judge 1 Judge 2 Judge 1 Judge 2

21 0.5377376 0.5299191 -0.1761536 -0.0591588 0.3469381 0.2030653

88 0.6024059 0.6734458 -0.4291461 -0.3331847 -0.0070753 -0.0254401

Analyzing quality of comments

Correlations between reviewers’ scores and judges’ labels

DD Criteria Same class Twin class p-valueWilcoxon test

Nº rev. Median Nº rev. Median

21 Assessment 30

3 52

2 0.0479

Feedback 3 3 0.5333

Form 3 3 0.6916

88 Assessment 28

4 55

3 0.0818

Feedback 3 3 0.7791

Form 3 3 0.8482

p-values of the Wilcoxon test to contrast localization parameters of label distributions

Analyzing comments: same class vs twin classes

Conclusions We developed a peer review module in our Web platform that support the pairing processset a non-anonymity environment where • students can review, send free text comments and discuss

among them• teacher can monitor the work

We analyze potential differences between peer reviews when they are done by students: on the same class group or from class groups of different schools.

We analyze several class groups doing review activities in which pairing resulted to be mixed.

Conclusions We found significant differences between reviewers from the same class versus those from different classes, in the review scores and in the quality of feedback

When reviewers are from the same class, they tend to give higher scores to their peers and tend to write more accurate assessment comments.

This seems to confirm the idea that peer reviews gain quality when reviewers know the reviewed students, in the sense that non anonymity make reviewers more aware of what they are reviewing.