Practice makes perfect: Testing the testing effect in a naturalistic setting

Dr David Hardman

School of Psychology

London Metropolitan University

d.hardman@londonmet.ac.uk

Twitter: @davidkhardman

Presentation given to the 2009 Teaching and Learning Conference, London Metropolitan University, 7th July.

The forgetting curve

0

One month laterAm

ount

rem

embe

red

Initial learning

20 mins later

1 hour later

Time

2 days later

Schematic based on Ebbinghaus (1885) and many subsequent studies

0

100%

How can we improve long-term retention?

• Elaborative rehearsal (rather than maintenance rehearsal).

• Self-reference effect.

•Distributed practice (not massed practice, or cramming).

• Testing, rather than rereading (after an initial period of reading; Karpicke & Roediger, 2008).

BUT --- rereading is the most common revision strategy (Karpicke et al, 2009) and students predict it will be more successful than recall testing (Karpicke and Roediger, 2008).

Can we generalise from laboratory studies to naturalistic settings?

Gurung & Daniel (2006)…

• Revision tests in unsupervised settings were associated with poorer exam performance.

• Supervised tests were associated with better exam performance.

Overview of the current studies• Revision tests were presented at regular intervals in WebLearn (the London met VLE).

• Students were informed about the benefits of revision testing and warned against looking up the answers.

• On submitting a test, students received performance feedback: a score, their answers, and the correct answers (if different).

• Tests were initially available for a limited period; but later made permanently available.

• Engagement was monitored using the tracking function.

• Students completed a summative test at the end of the semester, and an essay-based final exam about a month later.

Study 1. Autumn 2009.

Certificate level undergraduates taking Cognitive Psychology 1

Study 1. Relationship between number of online revision tests taken and score for the in-class test.

Study 1. Relationship between the summative MCQ and essay exams

Study 1. Two other potential correlates of MCQ test

performance (indirect measures of ability and motivation).

Regression analysis of data (1)

• Reading WebLearn discussions was not associated with MCQ test performance.

•Together, numeracy and revision testing accounted for 38% of the variation in summative MCQ test scores.

The standardized regression coefficients were:

• Numeracy. ß = .49 (p < .001)

• Revision tests. ß = .34 (p < .001)

Regression analysis of data (2)

• Neither WebLearn discussions nor numeracy were predictive of performance on the essay-based exam.

• MCQ revision testing accounted for 9% of the variation in essay exam performance, (p = .002), ß = .29

Study 2

• Spring semester: comparison of undergraduate and graduate students.

• As before, MCQ revision was tracked through the teaching period.

• At the end-of-teaching period, a month prior to the final (essay) exam, paragraph questions were added to WebLearn.

Study 2. The relationship between MCQ revision and the summative MCQ test.

Study 2. The relationship between MCQ revision and performance on the final essay exam

Study 2. The relationship between number of MCQ & paragraph tests and essay exam performance.

Study 2. Statistical analyses.

Comparison of Undergraduate and Graduate students, with revision testing as a covariate.

1. Summative MCQ test.

Both type of student and revision testing were significant factors (ps < .001)

2. Summative essay exam.

Type of student was only marginally significant (p = .05)

Revision testing was significant (p = .001)

Conclusions

• Results are consistent with lab research into the testing effect.

• Practice leads to better test results, even when indirect measures of motivation/ability are taken into account.

• Level of revision practice was the main feature distinguishing undergraduates and graduate students.

Limitation

Unable to determine what students did “offline”

References

Ebbinghaus, H. (1885). Memory: A contribution to experimental psychology. Translated by Henry A. Ruger & Clara E. Bussenius (1913). Originally published in New York by Teachers College, Columbia University. Available at: http://psychclassics.yorku.ca/Ebbinghaus/index.htm

Gurung, R.A.R., and Daniel, D. (2006). Evidence-based pedagogy: Do text based pedagogical features enhance student learning? In D.S. Dunn and S.L. Chew (Eds.), Best practices for teaching introduction to psychology (pp. 41-55). Mahwah, N.J.: Erlbaum.

Karpicke, J.D., & Roediger, H.L. (2008). The critical importance of retrieval for learning. Science, 319, 966-968.

Karpicke, J.D., Butler, A.C., & Roediger, H.L. (2009). Metacognitive strategies in student learning: Do students practice retrieval when they study on their own? Memory, 17, 471-479.