Upload
jasper-shelton
View
214
Download
0
Tags:
Embed Size (px)
Citation preview
11th International CAA Conference, Research into e-Assessment, Burleigh Court International Conference Centre, Loughborough University, 10th July 2007
BENEFITS AND OBSTACLES: FACTORS AFFECTING THE UPTAKE OF CAA IN UNDERGRADUATE COURSES
John Dermo,
Pathfinder Project Team, University of Bradford
2
First, a plea for your help….
“The idea being that authors use the conference to air their work and receive feedback to enable judgements to be taken on progress to date and how things may be taken forward in future.”
(extract from Submissions: 11th CAA Conference)
http://www.caaconference.com/submissions/index.asp
3
Background to the Research Project
University of Bradford
• e-Strategy
• VLE = Blackboard ™
• CAA = QM Perception™
4
Developments in CAA @ UoB so far include:
• Deciding on Questionmark™ Perception™ as our supported enterprise level software for online summative assessment
• Encouraging and supporting its use in formative assessment and feedback
• Centralising the administrative support for all summative assessments in our Examinations Office
• Implementing Questionmark™ Perception™ version 4.3 with a server configuration to ensure security and reliability
5
Pathfinder Project @ UoB
Investment in
1. CAA
2. Enhancing FYE
March 2007-April 2008
6
Pathfinder Project: CAA Aims and Objectives
to embed support processes for e-assessment
to develop the administrative and support systems to ensure reliable and secure large-scale implementation of CAA with first year students so that we can subsequently roll this out to all students
7
Pathfinder Project: CAA Deliverables
A system to support the effective implementation of formative and summative computer-aided assessment across the university, consisting specifically of:
• baseline study and needs analysis report on use of CAA at UoB• research data (quantitative and qualitative) on use of CAA in undergraduate
courses• official guidelines for CAA: admin and support functions to ensure resilient and
manageable online assessment routes• pedagogical guidelines (“best practice”) document for recommended use of
software• training sessions for instructors (Sept/Jan/May)• induction and training sessions for invigilators• in-house training materials and web-based resources for use of software• case studies of institutional implementation of CAA• banks of online assessment materials
8
Aims of my research project
• gather data on attitudes towards computer-assisted assessment
• to identify drivers and obstacles to the uptake of CAA
• test various widely-held assumptions about attitudes towards e-learning
• inform Pathfinder project• anticipate risks
9
Methodology
• descriptive
• survey opinions of comprehensive range of interested stakeholders:– students, academic staff, administrators,
technical and support staff, management and financial departments.
• primarily a qualitative approach
10
Phases of research
• Informal interviews, desktop research to identify key areas of interest
• Online questionnaires
• Follow-up interviews to investigate issues arising
11
Challenges: methodology
• Quantitative vs Qualitative
• Positivist vs Anti-positivist
• Survey vs Case study
How can we draw generalisable conclusions based on individual personal opinions?
12
Challenges: methods
• Mode of delivery (online vs paper)• Item design (open vs closed)• Subject groups and sampling• Piloting• Assuring confidentiality• How to survey opinion on something never
experienced?
13
Important Note:
Spend time at the outset to make sure that the research tools are reliable and valid before moving to data collection.
All too often researchers rush into the data collection stage, only to find that the data they are collecting are useless and any findings based on it equally so.
14
Initial findings and looking ahead
• research still at initial phase• key areas of interest being put to subjects to
inform the main survey questions to come• reviewing the literature• It is hoped that feedback from conference
delegates will be able to feed into this process!!
15
Drivers
savings in human and financial resources
improved reliability in marking
ease of production of results and item analysis data
ease of creation of different versions and randomised assessments
recycling assessments positive backwash effect on teaching and learning
appeal to “digital native” students possible benefits for recruitment and retention
potential of portfolio assessment accessibility issues
encouraging good assessment practise concerning item banking and item analysis
16
Obstacles 1
limited suitable task types inability to assess higher level skills in a valid way
high risk of technical failure initial outlay of time
steep learning curve for instructors high cost of software licenses and support plans
difficulty in convincing examination boards and QAA concerning issues of quality
anonymous submission of assignments
security issues – e.g. passwords / collaboration / collusion / cheating / impersonation
item banking requiring more effort and time
technical expertise required of instructors
lack of immediate technical support
17
Obstacles 2
difficulties for administrators difficulties for invigilators
training implications accessibility issues
health and safety issues difficulty of instructors in moving away from traditional task types
issues of task design threat that CAA will be used to justify increased class sizes or staff reductions
lack of an agreed and enforced institutional policy
discrimination against “non-digital native” students
limited availability of Internet-connected computers at home, in halls of residences, on campus
availability of large computer rooms for examinations
18
Issues to consider
• Does the cumulative effect of the barriers outweigh that of the drivers?
• How to interpret the data in a meaningful way?• To what degree can e-assessment add value to
the learning experience? • Do some factors work as drivers under some
circumstances but as obstacles in others?
19
Closing comment…
The researcher is keen to involve the input of other experienced practitioners and researchers in the field of Computer Assisted Assessment by means of this conference, and hopes to work this short paper up to a full paper submission for the next event in 2008.
20
References
Cohen, L., Manion, L., Morrison, K. (2003) Research Methods in Education. London: RoutledgeFalmer.
Conole, G. (2003) Understanding Enthusiasm and Implementation: e-Learning Research Questions and Methodological Issues in JK. Seale (ed) Learning Technology in Transition: from Individual Enthusiasm to Institutional Implementation, Abingdon: Swets and Zeitlinger, pp. 129-146.
Dillman, DA. (2007) Mail and Internet Surveys: the Tailored Design Method. Hoboken, New Jersey: John Wiley & Sons.
Frankfort-Nachmias, C. and Nachmias, D. (1996) Research Methods in the Social Sciences. London: Edward Arnold.
Oppenheim, AN. (2000) Questionnaire Design: Interviewing and Attitude Measurement. London: Continuum International.
Verma, GK. and Mallick, K. (1999) Researching Education: Perspectives and Techniques. London: Falmer Press.
21
Thank you
Contact: John DermoPathfinder Project,
Learner Support Services,
University of Bradford
Email : [email protected]