PeerWise and Contributing Student Pedagogyj-hamer/Monash-Dec07.pdfLinear regression model for participation Introduction Results ENGGEN 131 Course details Grade distributions for MATLAB

PeerWise Monash, December 2007 – 1

PeerWise and Contributing StudentPedagogy

John Hamerwith Paul Denny and Andrew Luxton-Reilly

Department of Computer ScienceUniversity of Auckland

Talk given at Monash University, 10 December 2007

Introduction

IntroductionContributing

Student Pedagogy

What is PeerWise?

Features

Learning

For instructors

Results

ENGGEN 131

PCA

Gender data

Questions?


Contributing Student Pedagogy


Student Pedagogy

What is PeerWise?

Features

Learning

For instructors

Results

ENGGEN 131

PCA

Gender data

Questions?


■ Students create and share learning resources; e.g.notes, visualisations, instructional videos, quizquestions, reading lists

■ Web-based collaboration tool (e.g. wiki) used tostore work-in-progress and share course material

■ Peer feedback and evaluation

■ Related theories: flexible learning (Collis),constructivism, community of practice (Wenger),ZPD (Vygotsky)

What is PeerWise?


Student Pedagogy

What is PeerWise?

Features

Learning

For instructors

Results

ENGGEN 131

PCA

Gender data

Questions?


■ An online bank of multiple choice questions

■ All content is student generated

Features


Student Pedagogy

What is PeerWise?

Features

Learning

For instructors

Results

ENGGEN 131

PCA

Gender data

Questions?


Not just questions and answers

■ explanations

Features


Student Pedagogy

What is PeerWise?

Features

Learning

For instructors

Results

ENGGEN 131

PCA

Gender data

Questions?



■ explanations

■ responses

Features


Student Pedagogy

What is PeerWise?

Features

Learning

For instructors

Results

ENGGEN 131

PCA

Gender data

Questions?



■ explanations

■ responses

■ discussion threads

Features


Student Pedagogy

What is PeerWise?

Features

Learning

For instructors

Results

ENGGEN 131

PCA

Gender data

Questions?



■ explanations

■ responses


■ difficulty and quality ratings

Features


Student Pedagogy

What is PeerWise?

Features

Learning

For instructors

Results

ENGGEN 131

PCA

Gender data

Questions?



■ explanations

■ responses


■ difficulty and quality ratings

■ leader-boards

Learning


Student Pedagogy

What is PeerWise?

Features

Learning

For instructors

Results

ENGGEN 131

PCA

Gender data

Questions?


■ Designing a question

◆ focuses attention on learning outcomes

◆ encourages reflection on course material

■ Choosing distractors

◆ misconceptions are considered

◆ promotes deep understanding

■ Writing explanations

◆ students express understanding in their ownwords

Learning


Student Pedagogy

What is PeerWise?

Features

Learning

For instructors

Results

ENGGEN 131

PCA

Gender data

Questions?


■ Answering questions

◆ useful practice / revision

◆ reinforces learning

■ Evaluating quality

◆ requires critical analysis

■ Providing feedback

◆ encourages peer dialogue around learning

Learning


Student Pedagogy

What is PeerWise?

Features

Learning

For instructors

Results

ENGGEN 131

PCA

Gender data

Questions?


Probably setting up my multi-choice question. This

was pretty hard given that i had to think of the

possible wrong solutions students would fall for and

required a lot of thinking from me, which in the end

was a lot of help because i was just about able to

answer any question that was on the same topic as

my question.

That was the biggest learning experience for me!

— OE4/205

For instructors


Student Pedagogy

What is PeerWise?

Features

Learning

For instructors

Results

ENGGEN 131

PCA

Gender data

Questions?


■ Large question bank developed at low cost

◆ ENGGEN 131 (introductory programmingcourse)

◆ 570 students

◆ 1,700 questions

◆ 35,000 responses

◆ Sept 9th – Nov 1st

■ Can be used as a basis for exam questions

For instructors


Student Pedagogy

What is PeerWise?

Features

Learning

For instructors

Results

ENGGEN 131

PCA

Gender data

Questions?


■ assists staff inidentifyingweaknesses

■ reveals how wellstudents areengaging withcertain topics

Results

Introduction

Results

Courses

Overall use

ENGGEN 131

PCA

Gender data

Questions?


Courses

Introduction

Results

Courses

Overall use

ENGGEN 131

PCA

Gender data

Questions?


■ Five courses used PeerWise in 2007

■ Varied in: coursework marks awarded for theactivity (Worth), questions to write and answer,use of MCQs in exam and tests, and whether thePeerWise authors taught the course.

Course Worth Write Answer Exam PW taught?

CS101 2% 2 10 Both YesCS111 1% 2 2 None YesCS105 2% 2 10 None NoCS220 1% 2 10 Both NoEG131 7% 2 20 Exam Yes

Overall use

Introduction

Results

Courses

Overall use

ENGGEN 131

PCA

Gender data

Questions?


CourseMean

QuestionsMean

AnswersMean

Comments

CS101 2.9/2 37.5/10 13.5CS111 1.9/2 60.7/2 5.0CS105 2.4/2 47.0/10 9.2CS220 1.6/2 36.0/10 7.2EG131 3.0/2 62.6/20 19.1

ENGGEN 131

Introduction

Results

ENGGEN 131

Course detailsGrade distributions

for MATLAB and C

C vrs. MATLAB

Model

Observations

Gain by quartile

Gain by median

PCA

Gender data

Questions?


Course details

Introduction

Results

ENGGEN 131


for MATLAB and C

C vrs. MATLAB

Model

Observations

Gain by quartile

Gain by median

PCA

Gender data

Questions?


■ Compulsory for all first-year engineering students

■ Two independent 6-week sections: MATLAB, C

■ PeerWise introduced only in the C section

■ MCQs in C section of the exam

■ Collected individual data on: number ofquestions written (Qs); questions answered (As);length of comments (NC); average commentlength (AvgC); MCQ (i.e. C programming) mark;MATLAB mark

Grade distributions for MATLAB and C

Introduction

Results

ENGGEN 131


for MATLAB and C

C vrs. MATLAB

Model

Observations

Gain by quartile

Gain by median

PCA

Gender data

Questions?


0 100 200 300 400 500

2040

6080

Gra

de

Grades distributions for MATLAB (red) and C (blue)

C vrs. MATLAB

Introduction

Results

ENGGEN 131


for MATLAB and C

C vrs. MATLAB

Model

Observations

Gain by quartile

Gain by median

PCA

Gender data

Questions?


0 20 40 60 80 100

020

4060

8010

0

MCQ grade

MA

TLA

B g

rade

Grades in the two sections are similaridentical mean p = 0.11

Linear regression model for participation

Introduction

Results

ENGGEN 131


for MATLAB and C

C vrs. MATLAB

Model

Observations

Gain by quartile

Gain by median

PCA

Gender data

Questions?


0 20 40 60 80 100

020

4060

8010

0

MCQ exam grade

Pre

dict

ed M

CQ

= 0

.12A

s +

12Q

s −

0.1

4NC

+ 0

.33A

vgC

Predicting MCQ grades from As+Qs+NC+AvgC

Observations

Introduction

Results

ENGGEN 131


for MATLAB and C

C vrs. MATLAB

Model

Observations

Gain by quartile

Gain by median

PCA

Gender data

Questions?


■ The linear regression model came up with theprediction

MCQ = 0.125As+12.05Qs−0.14NC+0.33AvgC

■ The largest contribution comes from Qs, thenAvgC, followed by As and (negligibly) NC. Theratios for an average student are: 31 : 16 : 8 : −3

■ Writing questions is the single most significantpredictor of exam performance, followed by theaverage length of comments.

■ These are both activities associated with deeplearning

C-MATLAB gain by quartile

Introduction

Results

ENGGEN 131


for MATLAB and C

C vrs. MATLAB

Model

Observations

Gain by quartile

Gain by median

PCA

Gender data

Questions?


Q1 (low) Q2 Q3 Q4 (high)

−20

020

Diff

eren

ce M

CQ

−M

AT

LAB

Gain in MCQ−MATLAB by participation quartile

High/low participation C-MATLAB gain

Introduction

Results

ENGGEN 131


for MATLAB and C

C vrs. MATLAB

Model

Observations

Gain by quartile

Gain by median

PCA

Gender data

Questions?


Below median Above median

−20

020

Diff

eren

ce M

CQ

−M

AT

LAB

PeerWise participation correlates with gain95% interval = 1.6−5.5, p = 0.0004

PCA

Introduction

Results

ENGGEN 131

PCA

Factor map, all data

Individuals, all data

Individuals,

minimum Qs

Gender data

Questions?



Introduction

Results

ENGGEN 131

PCA



Individuals,

minimum Qs

Gender data

Questions?


−1.0 −0.5 0.0 0.5 1.0

−1.

0−

0.5

0.0

0.5

1.0

Variables factor map (PCA)

Dimension 1 (36.29%)

Dim

ensi

on 2

(24

.56%

)

As

Qs

NC

AvgCMCQExam


Introduction

Results

ENGGEN 131

PCA



Individuals,

minimum Qs

Gender data

Questions?


−5 0 5 10 15

−6

−4

−2

02

4Individuals factor map (PCA)


Dim

ensi

on 2

(24

.56%

)

1

234

56

7

8910

11

12

13

14

151617

1819202122

2324252627

28

29303132333435

36

37

38

3940

41424344

4546474849

50

52

53

54

55

5657

58

59606162

63

64

65

66

6768

697071

727374

75

7677

78

79

80

81828384

85

86 8788

8990

91

9293

94

95

96979899100101 104

105106

107108

109110111112

113114

115116117

118

119120

121

122

123124

125126

127128

129

130

131

132

133134135

136

137

138139140141

142143

144145146147148

149150151152

153

154

155

156

157158159160

161162163164

165166167

168169170

171172173

174

175176177

178

179

180

181182

183184

185

186187188189

190191192

193194195

196197

198199

200

201

202

203

204

205

206

207

208

209210

211

212

213

214

215216217218

219

220221222

223224

225226

227

228

229

230

231

232233

234

235

236

237238

239

240

241

242243244

245

246

247248249250251252253254

256

257

258

259

260261

262

263

264265

266

267268269

270

271272

273274

275

276

277278

279280281282

283

284

285

286

287

288

289

290

291

292

293

294

295296297298299

300301302303304

305

306

307

309

310311

312313314

315

316

317

318

319

320

321322

323324

325326

327

328

329

330

331332333334

335336

337

338

339

340

341342

343344

345

346347348

349

350351352353

354

355

356

357358

359360361

362

363364365

366 367

368369370371

372373374375

376377

378

379

380381382383

384385386

387

388

389

390

391392

393394 395

396

397

398399400401402403404

405

406407

408

409

410411

412413

414415416417

418

419420

421422423

424425426427428

429430431432

433

434

435

436437

438

439

440

441442443444

445

446447

448

449450

451452453

454455456

457

458

459

460461

462

463464

465

466467

468469

470

471472

473474

475

476477

478

479

480

481

482

483484

485486487

488

489

490

491

492

493494495496

497498

499500501

502

503

504

505506507

508509

510511512

513514

515516517518

519

520521

522523

524525

526

527

528529530

531

532

533

534535536537

538539540541542543544

545546547548549

550551552553554

555556557

558559

Individuals, minimum Qs

Introduction

Results

ENGGEN 131

PCA



Individuals,

minimum Qs

Gender data

Questions?


−5 0 5 10

−6

−4

−2

02

4Individuals factor map (PCA)


Dim

ensi

on 2

(25

.2%

)

1

2

4

6

1013

1617

2223

2427

28

2930313334

36

37

39

42434445

4647484950

53

5760

63

64

65

6768

697071

7478

80

828384

85

8688

8990

91

93

94

95

979899100101

106

107108

118

119

123124

126

127129

130

132

133

136

137

139140141

142143145147148

149150

151 155156

161162163

169170

173

174

176177

180185

187

189

190191192

193

194195

196197

199 200

201

202

203 207

208

209210

213

214

219

220221

222224

225227

228

229

230

231

232

234

236

237238

240

241

242243

245

246

247249250251253254

255

257

263

265266

267269271

274276

281

282

283285287

288

289

290

291

292

293295297298299

300301302303304

305

308309

310312313314316

317

318

321322

323

325328

329

330

331332334

335

337

338

339

340

341343344

345347

349

350351352353

354

355

356 359361

362

368

370371376

378

379

381385386

387

388390

391392

393394 395

396

397

399400401402403406

408410

412

415416417

418

419420

421422423

424425426427

429430431432

433

434

435437

438

440

441442443 444

445

447449450

451452453

454455

457

458

461

462

463465

466467

468469

470472

473474

475

476477

478

479

480

481

482

483484

486487

488

489

490491

493495496

497498

499500

501

502

503

504507

508

511

513516518

519

521522523

524525

527528530

531

534535537538540

542543 545546547548

549 550551

552553

554

555556557558

559

Gender data

Introduction

Results

ENGGEN 131

PCA

Gender dataMales slightly

outperform females

No gender difference

in PeerWise

participation

Questions?


Males slightly outperform females

Introduction

Results

ENGGEN 131

PCA


outperform females


in PeerWise

participation

Questions?


Female Male

2040

6080

MC

Q e

xam

mar

k

Men outperformed women in C95% interval = 2.7−9.4, p = 0.0005

Males slightly outperform females

Introduction

Results

ENGGEN 131

PCA


outperform females


in PeerWise

participation

Questions?


Female Male

3040

5060

7080

90

MA

TLA

B e

xam

mar

k

Men do better in MATLAB, but not to the same extent95% interval = 1.3−6.2, p = 0.003

No gender difference in PeerWiseparticipation

Introduction

Results

ENGGEN 131

PCA


outperform females


in PeerWise

participation

Questions?


Female Male

02

46

810

1214

Que

stio

ns p

oste

d

Men and women both contributed Qs equallyp = 0.7


Introduction

Results

ENGGEN 131

PCA


outperform females


in PeerWise

participation

Questions?


Female Male

050

100

150

200

Que

stio

ns a

nsw

ered

Men and women both answered Qs equallyp = 0.6


Introduction

Results

ENGGEN 131

PCA


outperform females


in PeerWise

participation

Questions?


Female Male

020

4060

8010

0

Tot

al c

omm

ents

(ch

ars)

Men and women both posted comments equallyp = 0.27


Introduction

Results

ENGGEN 131

PCA


outperform females


in PeerWise

participation

Questions?


Female Male

050

100

150

200

250

Ave

rage

com

men

t len

gth

Men and women both wrote similar length commentsp = 0.13

Questions?

Introduction

Results

ENGGEN 131

PCA

Gender data

Questions?


Documents

PeerWise and Contributing Student Pedagogyj-hamer/Monash-Dec07.pdfLinear regression model for participation Introduction Results ENGGEN 131 Course details Grade distributions for MATLAB