Using methodological triangulation to analyse students’ use of recorded lectures

Using methodological triangulation to analyse students’ use

of recorded lectures

July 7, 2012

Pierre Gorissen

Fontys University of Applied Sciences

The Netherlands

• PhD Research:

“Facilitating the use of recorded lectures”

• Pierre Gorissen (Fontys University of Applied Sciences)

• Dr. Jan van Bruggen (Open University of the Netherlands)

• Prof. Dr. Wim Jochems (Eindhoven University of Technology)

Recorded Lectures

Result

Recorded Lectures

• Recorded lectures: • Many• Long (45 min. average)• Expensive to create

• On-campus students

Are students using/watching these recordings?

Multi step approach

#1 Let’s ask them:- Survey

- Interviews

#2 Measure their use:- Mediasite reports

- Log files

#3 Compare the results- Triangulation

#1 Let’s ask them

• 1,122 studenten (203 Fontys / 919 TU/e)• 7 courses (1 Fontys / 6 TU/e)• Response rate 46.1% (517 students)

• Follow up interviews (30 minutes) with 14 students

Main results

• Most students watch recordings at home;• No major technical problems;• They know where to find the recordings;• They would prefer for all courses to be

recorded.

Main results

• Reasons to watch:- Missed one or more lectures;

- Prepare for exams / improving test scores;

- Improve retention of lecture materials;

• Reasons not to watch:- Already attended the live lecture;

- No time;

- Didn’t feel they missed anything.

Further reading on step #1

Gorissen, P., van Bruggen, J.M. and Jochems, W. (in press) ‘Students and recorded lectures: survey on current use and demands for higher education’, Research in Learning Technology.

#2 Measure the use

Logfiles

Mediasite database

Combined dataset

Combine data

sources

Missingdata

Irrelevantdata

Identify users

Identify sessions

Remove Outliers

Filtered total

dataset

Data cleaning

Analyse dataset

Log results

The log data

• 1,500,000 lines of log data• 8,000 recordings• 5,000 hours of video• 263 different courses• 4,927 unique student users• 48,539 learner sessions

Example of result

Further reading on step #2

Gorissen, P., van Bruggen, J. and Jochems, W. (2012) ‘Usage reporting on recorded lectures using educational data mining’, Int. J. Learning Technology, Vol. 7, No. 1, pp.23–40.

#3 Compare the results

Logfiles

Mediasite database

Combined dataset

Combine data

sources

Missingdata

Irrelevantdata

Identify users

Identify sessions

Remove Outliers

Filtered total

dataset

Data cleaning

Compare results

Survey results

Analyse dataset

Log results

!?!?

Reported versus actual use

Reported Actual

n (%) n (%)

Never 13 9.1 6 4.2

< 5 times 22 15.4 35 24.5

5-10 times 51 35.7 43 30.1

> 10 times 57 39.9 59 41.3

Number of times respondents used recorded lectures for the C01 course.

Reported versus actual use

Reported Actual

n (%) n (%)

0% - 10% 2 1.5 27 9.3

10% - 25% 4 3.1 203 69.8

25% - 50% 7 5.4 40 13.7

50% - 75% 26 20.0 13 4.5

75% - 100% 91 70.0 8 2.7

Average percentage of a recording viewed.

Conclusions

• Methodological triangulation is necessary• Using the logged data, some reports by students:

• can be confirmed

• can be refuted

• can only be confirmed/refuted indirectly

• can not be confirmed/refuted

• …so we need both data sources!

Conclusions

• Don’t simply rely on what students tell you. It might be just what they think you want to hear!

• “Views” is a bad measure for use.• Different recordings or different students

mean different use.

P.Gorissen@fontys.nl

http://www.linkedin.com/in/PierreGorissen