Embed Size (px)
Citation preview
Evaluating e-learning
Oxford Centre for Staff and Learning Development
Oxford Brookes University
Oxford Centre for Staff and Learning Development
Critical success factors
• Implementations which tackle real and relevant problems at the course level
• There may be advantages to using a poorly defined term in institutional change
• Institutional rationales which are contextualised and specific• Students’ understanding their own learning and the role of
resources and technology in learning• Course designs or redesigns undertaken as a team,
developed iteratively over a number of years in response to student feedback
Sharpe et al (2006) Review of undergraduate experience of blended e-learning, for the HEA.
Oxford Centre for Staff and Learning Development
Some examples
1. Brookes Virtual
2. Brookes Partnerships in Practice
3. JISC Learner experience programme
4. Brookes Pathfinder
Oxford Centre for Staff and Learning Development
1. Brookes Virtual
Oxford Centre for Staff and Learning Development
Perceptions and satisfaction
University of Wales, Swansea, Blackboard User experience survey 2003
Oxford Centre for Staff and Learning Development
“We’ve never done any surveys, ever, that have given other than the students want more of it, wider and deeper”
(Longside 2)
Oxford Centre for Staff and Learning Development
“We just thought … we’ll just use our ordinary module evaluation. Well it wasn’t going to work was it? It’s not asking the right sort of questions.”
(Deepshire 1)
2. Partnerships in Practice
Oxford Centre for Staff and Learning Development
2. Partnerships in Practice
Collected feedback through
• student perception questionnaire
• student SPOT analysis in groups
Students used VLE to
- Access resources
- support groupwork processes.
Oxford Centre for Staff and Learning Development
“reading through all the feedback data from students and tutors is like standing at the apocryphal spaghetti junction and watching cars going every which way.
Some students call for more group work: others want none at all. ….. Advice fumes the air”.
(Mason & Weller, 2000)
Oxford Centre for Staff and Learning Development
9 funded projects, all using qualitative methods to capture the learner voice
e.g. audio logs, video diaries, interviews.
3. In their own words
Oxford Centre for Staff and Learning Development
“I'm addicted, it's the first thing I turn on in the morning before I even wake up and actually it's very, very bad. I think in the future people can't cope without their laptops.
(Undergraduate Business student,
LEX Final Report)
Oxford Centre for Staff and Learning Development
Oxford Centre for Staff and Learning Development
Coding examples
Document with stripes showing coding at nodes
Context of document showing coding at ‘Motivation’ node
Oxford Centre for Staff and Learning Development
“Besides all the complexity created by marked differences across subject areas and myriad individual differences among both staff and students which prevent simple patterns emerging, there are additional crucial differences between the idealized world described by research and the actual world experienced by the participants.”
Entwistle, N., McCune, V. and Hounsell, J. (2002). 'Approaches to Studying and Perceptions of University Teaching-Learning Environments: Concepts, Measures and Preliminary Findings.' online http://www.ed.ac.uk/etl/docs/ETLreport1.pdf.
Oxford Centre for Staff and Learning Development
4. Brookes Pathfinder
• Combined-methods approach where various types of data have been collected
• Series of seven case studies to evaluate the nature and the impact of innovation on learner experiences
• Questionnaire on learner perceptions of learning technology use
Oxford Centre for Staff and Learning Development
Questionnaire
• Provides descriptive statistics on learner characteristics
• Investigates patterns in learner uses of technology
• Analyses which how study strategies (such as the use of peers or help-seeking) or views of learning (such as the perceived degree of independence) are related to various dimensions of technology use
Oxford Centre for Staff and Learning Development
What we’ve seen
1. Brookes virtual – audit
2. PiP – course improvement
3. Jisc learner voice – discovery
4. Pathfinder - understanding
Oxford Centre for Staff and Learning Development
Possible functions of evaluations
• Is it being used?• How is it being used?• Do the staff and students like it?• How do the students experience it?• Is it effective?• How could we make it better?
Oxford Centre for Staff and Learning Development
Purposes of evaluation
• Formative: To ensure that a product reflects the intentions of its designers AND meets the needs of the users
• Summative: To test a product• Integrative: To integrate a product into a new
environment• Pragmatic: To discover the unexpected benefits• Illuminative: To get a project funded
Goodyear (2001) Effective networked learning in higher education: notes and guidelines’ (p. 37-40)
Oxford Centre for Staff and Learning Development
Thinking about your purposes
Use the lists of purposes and questions to complete the stakeholder template.
Who are your key stakeholders?What are their priorities?What questions are of interest to them?
Oxford Centre for Staff and Learning Development
ReferencesRamanau, R., Sharpe, R. and Benfield, G. (2008). Exploring Patterns of
Student Learning Technology Use in their Relationship to Self-Regulation and Perceptions of Learning Community. Paper to be presented at Networked Learning Conference, May 288, Halkidiki, Greece.
Sharpe, R., Benfield, G., Roberts, G. & Francis, R. (2006) The undergraduate experience of blended e-learning: a review of UK literature and practice undertaken for the Higher Education Academy. At www.heacademy.ac.uk/4884.htm
Sharpe, R. & Pawlyn, J. (2008) The role of the tutor in blended e-learning: experiences from interprofessional education in R. Donnelly (ed) Applied eLearning and eTeaching in Higher Education
