Aligning the Ducks: Sustaining the quality of CAA in a Period of Growing Demand Glenis Lambert

Learning and Teaching Enhancement Unit

Aligning the Ducks: Sustaining the quality of CAA in a Period of Growing Demand

Glenis Lambert

LTEU, Canterbury Christ Church University

Glenis Lambert LTEU June 2009




Outline

The history of online summative tests at CCCU using short-answer questions delivered online

A reflection, 2005 – 2009

What do we want from CAA?

How do we match up to this today?

The way forward



Background: Canterbury Christ Church University


Large numbers of students on placement

Faculties of Health and Social Care, Education, Undergraduate

Programmes +++

Diverse student body

Multi-location campusGrowing student numbers

Growing numbers of collaborative programmes

Central Learning and Teaching Enhancement Unit with Learning Technologists for each Faculty


2005

What we didBought QuestionMark/Perception

Mapped with existing University policies and managed conflicts

Using BS ISO/IEC23988:2007 Wrote a policy document

Set up staff procedures and guidance with full documentation for staff

What happened in the pilot phaseOver the academic year 8 end of course exams in 18 sessions were successfully delivered, totalling 660 individual examinations. 5 of these were organised and delivered by the departments, three by the examinations office.


2005

2006

2007

2008

2009


2006


2005

2006

2007

2008

2009

What we didSlight increase in number of tests

Consolidated procedures

What went wrongLack of staff confidence in online environment led to high support demand

Some institutional procedures did not fit the CAA mode of delivery, resulting in complications for the student

Human error!

What we LearnedStakeholder confidence in support and procedures help develop practice

Buy-in at all levels is essential

Clear fall-back strategies are essential

BUT:Exit surveys revealed student satisfaction with online examinationsAll of the fall-back procedures worked wellThe systems/software performed well


2007


2005

2006

2007

2008

2009

What we didDelivered over 5,000 summative tests and a

few other surveys.Support load on LTEU began to be unsustainable.

What went wrongStaff still not engaging with processes and procedures

Infrastructure made process expensive

Some institutional procedures did not fit the CAA mode of delivery, resulting in complications for the student

Human error!

“Last minute” culture led to increased support demand

Infrastructure began to creak!


2008


2005

2006

2007

2008

2009

What we did to try to manage the support loadPut an embargo on all new tests and set out to reinforce our policies by working with stakeholders to devolve some of the activities carried out by LTEU

Bought the Blackboard Bridge to reduce the administrative load

Encouraged more academics to do “open” assessments which reduced the infrastructure and invigilator load, but not the support load, although this changed its emphasis. (Bertolo and Lambert, 2007)

What we learnedOur ambition to control the testing environment was probably never going to happenIt was going to take a long time to get all stakeholders on boardThere needed to be a central agency to co-ordinate all aspects of the processSupport time for CAA was always going to be highInfrastructure/development costs were an ongoing commitment We had yet to muster a persuasive argument to get total institutional buy-in


2008


2005

2006

2007

2008

2009

What went well

We have rarely had to resort to “fall-back” positions, e.g. paperOur quality procedures have been robust and applied throughoutOur planning procedures have raised stakeholder confidenceDocumentation produced for staff has been welcomed resulted in clear definitions of responsibilitiesStaff development has led to improvements in all areas


2009


2005

2006

2007

2008

2009

What we learned

We needed to think carefully about what we were doing and whether we should continue to do it.


What do we require from CAA?

Report on eAssessment Quality (REAQ)

OverviewThere is wide consensus that e-assessment promises many potential benefits

to the JISC community, ranging from lowered costs, higher productivity, and faster feedback, through to assessments which are more accurate, more detailed, and more robust under critical scrutiny and audit. There is equally wide concern that these benefits seem relatively slow to materialise, and may be negated by poorer quality assessment which fails to reliably provide acceptably valid measures of actual student achievement and capability, particularly at the higher levels of Bloom’s taxonomy of the cognitive domain. The issues confronting staff concerned with the quality management of e-assessment suggest a pressing need for evidence, for information, and for guidance.

http://www.jisc.ac.uk/whatwedo/projects/reaq.aspx



What do we require from CAA – the Institution’s view?

Assessed tasks should attract sufficient student time and effort

Tasks should engage the student in productive learning activity

Students' effort should be spread as evenly as possible

Orientation to the task should help students to perceive the demands of an assessment task

Communication through guidance provided should offer clarity on expectations

Sufficient and timely feedback should be provided

Feedback should focus on learning not on the marks or the student

Feedback should be closely linked to assessment criteria

Feedback should be given at a level accessible to the student's sophistication

Students should be required to receive and respond to feedback http://www.canterbury.ac.uk/support/learning-teaching-enhancement-unit/assessment/assessment-handbook/pages/

section1/11-3-strategies.asp



Can CAA ever be “fit for our purposes?”

Develops students’ abilities to evaluate own progress, direct own learning

Emphasises authentic and complex assessment tasks

Uses high-stakes summative assessment rigorously but sparingly

Offers “low-stakes” confidence building opportunities and practice

Is rich in formal comment, (e.g. tutor comment, self-reflective logs)

Is rich in informal feedback (e.g. peer review, collaborative project work)

Above all, it should enhance learning.



Quality Assurance Procedures

‘Openness to challenge is a critical cultural necessity for good risk management and compliance – it is in fact more important than any framework or set of processes’.

Paul Moore, Evidence to Treasury Select Committee (February 2009)


“Over the last twenty years, the higher education sector has been subject to structural and cultural changes associated with a focus on accountability and value for money which have profoundly affected academic life….. In this new reality the term ‘quality’ became synonymous with data collection, performance scrutiny and a massive increase in bureaucracy. ”

George Gordon and Catherine Owen, Cultures of Quality Enhancement: a short overview of the literature for higher education policy makers and practitioners Scottish Higher Education Enhancement Committee (SHEEC) and QAA Scotland (2009)


What are the risks?

Risks involved in the continuing growth of CAA:

High costs of quality-assurance not met by institution

High costs of reliable delivery not met by institution

Concentration on “box ticking” accountability does not lead to enhanced learning

Risks of NOT using or encouraging the growth of CAA:

Opportunity for enhancing learning and feedback lost

Students’ expectations of HE not met (Dermo 2009)



What we want from CAA


Policy

Fit for purpose

Robust

Reliable

Strategy

Embedding Embedding

Strategy

Policy

Robust

Reliable

Fit for Purpose

In addition:That it is used to provide speedy feedback

to studentsThat it is used as a means of enhancing

learning


Where we are at present


Policy

Where people have used CAA it has become embedded in their practice, but summative use isn’t leading to formative use.

There is a growth in the strategic use of CAA, but largely to reduce marking load.

Policy documents are now in existence, but they need reviewing in the light of support demand.

Fit for purpose

Robust

Reliable

We are confident that our policies and procedures do result in adequately assured assessments where they are applied, but the infrastructure and support services are sometimes more fragile

The software and systems have proved to be reliable. Infrastructure and the application of policy and procedures less so.

“Quick Burn” implementation (Warburton 2009) leading to some questionable practiceFeedback not built in to tests, little formative testing

Strategy

Embedding


Conclusions

The support and maintenance costs of quality-assuring CAA are high and ongoing

It is easy to quality-assure CAA if it is only seen as a method of providing accountability for assessment practices and procedures

If the major requirement of CAA is to enhance learning, it can be cost-effective, when “opportunity costs” are factored in (Ridgway et al. 2004).

To achieve the benefits of CAA for learning enhancement staff development has to be the central strategy for assuring quality.

A “student satisfaction” ethos needs to be central to mitigating failures



Forward plans


Embedding

Policy

Strategy

Fit for purpose

Robust

Reliable

In the absence of more resources, designate a number of rooms as testing labs to make infrastructure support more efficientMake sure all stakeholders know about fall-back proceduresMaintain central consultancy role in LTEU for construction of questions and testsDEBUT staff development project

Continue to work with all stakeholders to emphasise the importance of team structuresStaff development for all stakeholders including Examinations Office, User Technology etc.

Raising awareness through Enhancement Theme on Assessment and Feedback

DEBUT staff development projectWork with staff on planning of new courses via FQOsDevelopment of e-learning strategyBenchmarking e-learning, including use of CAA

Inclusion of policy and procedure documents in staff development pack




References

Assessment Handbook, Canterbury Christ Church University

http://www.canterbury.ac.uk/support/learning-teaching-enhancement-unit/assessment/assessment-handbook/pages/section1/11-3-strategies.asp

Bertolo, E & Lambert, G: 2007 Implementing CAA in Chemistry: A Case Study

Procedings of 11th CAA Conference, Loughborough University 2007

Dermo, J. (2008) “Implementing Online Assessment: Finding the Right Path for an HE Institution” in Ladwa, A (ed) E-Learning in HE available online at http://www.rsc-yh.ac.uk/Documents/HEbooklet2_000.pdf (last accessed June 2009)

Gordon, G & Owen, C, 2009,Cultures of Quality Enhancement: a short overview of the literature for higher education policy makers and practitioners

Scottish Higher Education Enhancement Committee (SHEEC) and QAA Scotland

Parsons, R, (2004) Ensuring Quality and Efficiency with Online Assessments JISC

http://www.enhancementthemes.ac.uk/documents/events/20040416/Parsonspaperrevised.doc.

Report on eAssessment Quality (REAQ), JISC 2009

http://www.jisc.ac.uk/whatwedo/projects/reaq.aspx Accessed June 2009-07-02

Warburton 2009, Quick win or slow burn: modelling UK HE CAA uptake

Assessment & Evaluation in Higher Education, 1469-297X, Volume 34, Issue 3, Pages 257 – 272



What we want from questions and tests


Policy

Fit for purpose

Robust

Reliable

Strategy

Embedding

That short, formative online tests are a routine part of assessment strategies

That online assessments are viewed as routine, in the same way as presentations, paper tests etc.

That policies and procedures are routine elements of assessment procedure

That all questions and tests are valid and peer-reviewed

That staff can rely on support and staff development in the preparation of assessments

That all elements are “fit for our purposes”


The question question: a digression.

Stage 1:

This is a duck: True or False?

Stage 2:

This bird is:

a) A coot c) A duck

d) A swan e) A diver

Stage 3:

Identify this common duck and match it with its species

a) Mallard i) Marmaronetta

b) Pochard ii) Netta

c) Teal iii) Aythya



Barriers to improving question quality

Many academic staff operate in a text-based environment and find the black and white nature of computer-marked questions challenging

Learning technologists do not have detailed subject knowledge

Staff development to enable the development of question-setting skills takes time.
