Upload
peter-quinn
View
215
Download
1
Tags:
Embed Size (px)
Citation preview
The Use of Screen Recorders to Achieve More Informative Assessment of Students’ Understanding.
Deborah Abdel Nabi & Paul RogersPsychology School of Health & Social SciencesUniversity of Bolton
Aims of Presentation
• Introduce a new computer-assisted assessment method recently developed in Psychology at the University of Bolton– Project funded by University TQEF sabbatical (2007)
• Highlight potential uses across the psychology curriculum.
Research design and data analysis (RDA) for BSc (Hons)
Psychology courses.
University of Bolton ‘Year 1 - Introduction to Research Methods In Psychology’
Year 2 - Research Methods II and Further Methods
Year 3 – Experimental Report for Honours’ Project
Conventional methods of teaching and assessing RDA
Lecture on Research DesignLecturer explains a particular research design
Lecture on Statistics
Overview of appropriate tests to analyse the data
Statistical Analysis Workshop
Traditionally at UoB Year 1stats analysis conducted using
calculator and formula
3 years ago restructured RDA in Year 1 (re-focus
emphasis on research design rather than calculation of
stats)
Students taught Statistical Package for the Social
Sciences (SPSS) Software from Year 1
Highly structured and detailed handouts for workshops
Module Assignment (s)
• Submit output file from an analysis (e.g. descriptives)• Answer questions on the results (interpreting output)
Problems
• How good are they at navigating around SPSS?
• How much do they really understand in the output ?(Tend to answer questions with ‘text-book’ phraseology)
• Did they get a friend’s copy?
Problem (ctd): Negative skew in grades
30 40 50 60 70 80 90 100
MarkAssignment1 2006-7 (Hard Copy)
0
10
20
30
40
50
60
Fre
qu
ency
Mean = 87.26Std. Dev. = 12.023N = 164
Problems (ctd): Lack of Relationship with other Measures of RDA Knowledge
Grades didn’t correlate with their score on a 30-item MCQ
test on RDA
Correlation 2006/7 (Hard Copy Submission of Assignment 1)
1 .095
. .112
164 164
.095 1
.112 .
164 164
Pearson Correlation
Sig. (1-tailed)
N
Pearson Correlation
Sig. (1-tailed)
N
MarkAssignment1
MCQs
MarkAssignment1 MCQs
Conclusions
New assessments of introductory RDA using SPSS were
better than previous but had room for improvement
(lacked informational value re. the depth of their
understanding)
Solution?
Could have incorporated more (and more specific
questions) on the analysis in written format BUT
• Can’t get people to write answers to everything we wanted to evaluate eg:– competency and fluency at data input,– variable specification – navigation around software
• Suspected there would be a tendency to stick to taught academic linguistic style when writing answers
We wanted to provide a means of promoting answers in
the students own words
i.e. wanted to recruit the truly generative processes
required for oral explanation
• Better comprehension (e.g. Muller, Sharma & Reimann, 2006)
• Better recall (e.g. Bekerian & Dennett, 1988; Goolkasian & Foos, 2002)
• Authenticity of understanding (e.g.Zhou Qi-jia, 2006)
Need means of getting students to demonstrate
and articulate their understanding
One-on-one observation / examination?• Impractical• Intimidating
The Idea
Screen Recorder Software
Record narrated movies of the content of your PC screen
Windows Media Encoder
The Project (Semester 1 and 2 2007/8)
First assessment in ‘Introduction to Research
Methods in Psychology’ students were instructed how to
create a small (approx 5 minute) WME movie of them:
• inputting data into SPSS to a specified format and• explaining why they are taking particular steps in order
to perform a given analysis and;• providing a description of (i.e. interpreting) the output
file
• Upload the file to WebCT for marking
• Observe a ‘model answer movie’
Example of a Movie File Created by A Student
Proposed Potential Impact of the Project
Proposed Impact on Student Learning
Practice
(To produce good movie)
Increased engagement
Deeper encoding
Easier retrieval
Deeper comprehension
Multi-Modal
(Visual, auditory and kinaesthetic)
Encoded on multiple levels
(Brunye, Taylor and Rapp,
2007).
Learning advantages Rich memories
Easier retrieval
Suits all learning styles
Observation of Performance
Watch and hear own knowledge repeatedly in
own words(Muller, Sharman and
Reimann, 2006)
Provides opportunity for reflective learning and possibly / enhanced
metacognitive awareness
Feedback
(Model answer)
Detect differences between their submission
and model answer
Increases metacognitive awareness and
updates concepts
Results
Effect on The Assessment Technique on Mark Achieved for Assignment 1
Frequency Distribution of Marks Across Cohorts
30 40 50 60 70 80 90 100
MarkAssignment1 2006-7 (Hard Copy)
0
10
20
30
40
50
60
Fre
qu
ency
Mean = 87.26Std. Dev. = 12.023N = 164
20 40 60 80 100
MarkAssignment1 2007-8 (Movie)
0
5
10
15
20
25
30
Fre
qu
en
cy
Mean = 72.06Std. Dev. = 16.989N = 145
Correlation 2006/7 (Hard Copy Submission of Assignment 1)
1 .095
. .112
164 164
.095 1
.112 .
164 164
Pearson Correlation
Sig. (1-tailed)
N
Pearson Correlation
Sig. (1-tailed)
N
MarkAssignment1
MCQs
MarkAssignment1 MCQs
Correlations 2007/8 (Movie Submission of Assignment 1)
1 .231**
. .003
145 136
.231** 1
.003 .
136 143
Pearson Correlation
Sig. (1-tailed)
N
Pearson Correlation
Sig. (1-tailed)
N
MarkAssignment1
MCQs
MarkAssignment1 MCQs
Correlation is significant at the 0.01 level (1-tailed).**.
Conclusions
When students are required to:• Demonstrate their knowledge / competency of research
design and SPSS and • Articulate their understanding of the output of stats
tests in their own words
• Mean mark drops and distribution not so negatively skewed.
• Not ‘perceived’ as competent as when just a hard-copy submission of an analysis is required.
Conclusions
Movie is more representative of their actual RDA
knowledge and understanding than traditional hard copy
submissions.
This is more informative for both tutor and student and
allows for better feedback and early intervention.
Feedback from Students
Feedback Questionnaire Results
Did they rehearse more?
1 = 1-2 times
2 = 3-6 times
3 = 7-10 times
4 = 11-15 times
5 = 16-25 times
1 2 3 4 5
No of Times Practiced Prior to Recording
0
2
4
6
8
10
12
14
1
13
9
2 2
Mean = 2.67Std. Dev. = 0.961N = 27
Was the multimedia format useful for learning?
2 3 4 5 6 7
Opportunity for Verbal Rehearsal
0
2
4
6
8
10
12
1 1 1
2
11 11
Mean = 6Std. Dev. = 1.271N = 27
2 3 4 5 6 7
Audio Playback is Useful
0
2
4
6
8
2
3
5
8 8
Mean = 5.58Std. Dev. = 1.447N = 26
4 5 6 7
'Doing' Something was Beneficial
0.0
2.5
5.0
7.5
10.0
12.5
2
3
12
10
Mean = 6.11Std. Dev. = 0.892N = 27
1 2 3 4 5 6 7
Visual / Graphical Feedback Is Beneficial.
0
2
4
6
8
1 1
2
8 8
4
2
Mean = 4.58Std. Dev. = 1.391N = 26
Was Model Answer Movie Useful Form of Feedback?
3 4 5 6 7
How useful was the model answer movie?
0
1
2
3
4
5
6
7
2 2
4
7
4
Mean = 5.47Std. Dev. = 1.264N = 19
1 = Not useful at all
7 = Very useful
Movies watched: 421 times 173 hours
Overall Conclusions and Future Work
• Performance on this assignment is a good (better)
reflection of research design and stats understanding
• It persuades them to engage more with the assignment
• Students generally found it useful and enjoyable.
• The multimedia format is perceived to be beneficial– Dual (triple) representation– Integrative working memory– Generative processes aid comprehension
Future Work
The usefulness of method has been consolidated by
increasing the amount of movie assignments in the
RDA module to two (and more advanced methods
modules)
Real possibility for extending this assessment technique
to e.g.
Honours Dissertation (RDA presentation movies)
and other modules such as Biopsychology
Enhance private study and revision
Own narrated recording of lecturer’s PP or create movies of mind maps with their own narration
Foundation component of a embryonic PLE
The End
Any questions?
Help to improve confidence?
1 = Didn’t improve it at all
7 = Improved it a lot
1 2 3 4 5 6 7
How much has using WME improved your confidence in using SPSS?
0
2
4
6
8
10
12
1 1 1
3
12
5
4Mean = 5.04Std. Dev. = 1.427N = 27
Did they enjoy using WME?
1 2 3 4 5 6 7
How much did you enjoy using WME?
0
1
2
3
4
5
6
7
1
2 2
4
7 7
4
Mean = 4.89Std. Dev. = 1.625N = 27
1 = Not at all
7 = I enjoyed it a lot