Embedding evaluation

PREEL 15 May 2007Caroline Daly

• What is really useful evaluation in courses designed to embed technologies to support learning?

• What methods are appropriate to support really useful evaluation?

• What is the role of course tutors, administrators, learning technologists in evaluation?

• What are the practical priorities?• So what is embedded evaluation and do we

want it?

The research projectContext Design

Mixed-mode degree (MTeach) Longitudinal – 7 months

Participants were a tutor group of 12 teachers beginning the course

Quantitative data – pre-course questionnaire

Text-based online discussions - compulsory - asynchronous 

Qualitative data – learner narratives - narrative interviews - focus group discussions - online focus groups - online commentaries

Only 2 participants were experienced e-learners

Analytical approach based on systematic reading and open-coding of range of narratives

Main aims: to identify e-learners’ experiences in the mixed-mode context to trial an embedded approach to evaluation to analyse the implications for e-learning course development 

• What is evaluation for?

• Who is evaluation for?

“Professional evaluation is defined as the systematic determination of the quality or value of something” (Scriven, 1991).

It usually serves two main purposes linked to concrete outcomes:

– To find areas for improvement– And/or to generate assessment of the overall

quality or value of something (usually involving reporting/decision-making)

Thoughts?????

Purposes of evaluation – a ‘classic’ view (Chelimsky, 1997)

accountability

knowledgedevelopment

e-VALU-ation

“…involves not only collecting descriptive information …but also using something called “values” to (a) determine what information should be collected and (b) draw explicitly evaluative inferences from the data…

…research can tell us “what’s so” but only evaluation can tell us “so what”

Davidson, E. (2005)

As evaluators, we need “certain enough” answers to evaluation questions…a “close enough approximation to the truth”

(Davidson, E. 2005)

Why narrative evaluation methods?

A way of capturing the learner voice as a prime evaluation tool

A way of better understanding the realities of ‘being an e-learner’ on our courses

To provide ongoing information about the course while it is happening

To impact on the learners’ abilities to adapt to new learning contexts

What narrative evaluation methods did we use?

Narrative ‘postmodern’ interviews (Gubrium, 2003) where participants construct an account of what is happening to them (can be telephone)

Online and face to face focus group discussions, timed at key moments in the module and addressing issues the students raise

Online ‘thinking aloud’ in response to images of e-learning (‘mediating objects’ Eraut, 2000), when they had ‘settled in’ to the module

Issues for online evaluation commentary 1. A sense of community?

Now that you have spent your first month as an online student on the MTeach – what has it been like so far? Some researchers say that it is very difficult to establish a ‘sense of community’ at a distance. Others say that it is certainly possible to form a learning community online, and there are several theories about what this might look like. What are your first thoughts?

2. Writing online/talking in a seminar

Only two of you had prior experience of learning online before joining the MTeach, so this has been a new way of communicating to learn for everybody else in the group. These discussions form a significant alternative to talking about issues at the face to face days or in traditional seminars. What has it been like to ‘discuss’ by using writing to communicate with each other like this?

I’m used to talking to people when I haven’t a clue who they are, so…I find it pretty easy to talk to people online. I was more nervous coming today, having not met anyone than I was chatting online and discussing issues.

Katy

Whereas I felt more comfortable because I wanted to get the non-verbal feedback. Because when you’re talking to somebody you’re getting that feedback all the time, aren’t you? About whether they’re agreeing. You’re nodding at

me now because you understand what I’m saying, whereas when you’re doing that online you don’t know

whether they’re understanding you at all.

Sally

Group narratives

“..intra-group discussion about what a particular story means fuels the learning cycle” (Greenhalgh, 2006)

The process is essentially one of active engagement in the evaluation of experiences, and is driven by the learners’ priorities and requires the sharing of sustained reflective narration.

The course info was clear that it is a mixed-mode course, using online discussion as the core learning mode

They thought it was …ftf…live chat… …video-conferencing …one to one tutoring via email

Reviewed the course literature and webpage to make it even more explicit. Use the entry viva to clarify basics

We thought… They said… So we…

Teachers would learn the principles of socio- constructivist ways of working without too much difficulty

It is hard to trust others at a distance. There is a lot at stake & they had expected to be ‘told’ the ‘answers’ by experts

Spend longer on ftf days discussing the course pedagogy. Devote more time to social bonding

The sample postings suggested helpful ways to contribute online and overcome anxiety about how to join in

The templates have a prescriptive effect and make them feel they have to write in a

particular style

Reviewed the sample postings & used ones from ex-students to encourage more natural & varied writing styles

There was an initial stage of anxiety and adjustment that went as soon as they became established as students

The process takes much longer & they disguise anxieties by ‘secretly’ re-writing draft postings - time-consuming!

Devote more time ftf discussing their online work. They select postings to discuss - the thinking is not finished

We thought… They said… So we…

Photos of group members in an online gallery would help to establish identities and a sense of belonging to a group

They really liked being able to ‘see’ their online peers

Predicted not everyone would post a photo. Took photos on the induction evening to guarantee 100% photo presence

They would be willing to tell us about their experiences as e-learners

They were extremely willing to tell us more candidly than we

anticipated

Built in an ongoing evaluation forum with key prompts from the tutor

ImplicationsThere are benefits for learners of participating in narrative evaluation activities, by which they address explicitly their (new) experiences of learning with technologies.

There are benefits for practitioners in gaining knowledge of unanticipated learner experiences, though these may not always lead to simple modifications of course design or learner tasks.

There is an argument for course design to invest learner time in developing the abilities of participants to evaluate how they are learning. There is growing research evidence that this has a positive impact on what they learn.

There is a need for meaningful evaluation activities which also support learning. Engaging with evaluation becomes a different type of experience for both tutors and students.

Planning for really useful evaluation• What do you see as the purposes of evaluation on

your course?• Who will be involved in evaluation activities?• How will evaluation be conducted:

– Who is responsible for the evaluation activities?– What is their frequency?– What type?– How accessible are the evaluation activities?– When will evaluation activities begin and end?– What will be done with the evaluation data?

• What will be the relationship between evaluation, curriculum and learning experiences?

Bibliography• Chelimsky, E. (1997) ‘Thoughts for a new evaluation

society’ Evaluation 3(1) 97-119. • Davidson, E. (2005) Evaluation Methodology Basics

London: Sage.• Eraut, M. (2000) ‘Non-formal learning, implicit learning

and tacit knowledge in professional work’ in (ed.) Coffield, F. The necessity of informal learning Bristol: Policy Press ESRC Learning Society Programme 12-31

• Greenhalgh, T. (2006) What seems to be the trouble? Stories in illness and healthcare (Oxford: Radcliff Publishing).

• Gubrium, J. (2003) Postmodern Interviewing London: Sage.

• Scriven, M. (1991) Evaluation Thesaurus 4th Edition, London: Sage.