Evaluation in E-learningEvaluation in E-learning(or “How can we know what works?”)(or “How can we know what works?”)

Mark JohnsonMark Johnson

Once upon a time…Once upon a time…

The ‘killer app’ interventionThe ‘killer app’ intervention

And now…?And now…?

Systems everywhere…Systems everywhere…

Questions…Questions…

How can we know anything works? (can How can we know anything works? (can we..?)we..?)

How can the whole sector benefit from How can the whole sector benefit from particular instances of good practice? (can particular instances of good practice? (can it..?)it..?)

What should we be doing technologically? What should we be doing technologically? (hasn’t it all been done..?)(hasn’t it all been done..?)

Is ‘the project’ the only way..?Is ‘the project’ the only way..?

Evaluation in the social worldEvaluation in the social world

Cause and EffectCause and EffectObjectivity and SubjectivityObjectivity and SubjectivityKnowledge and JudgementKnowledge and JudgementParticulars and UniversalsParticulars and Universals

… … what are the features of the ‘successful what are the features of the ‘successful project’project’

A history of causes…A history of causes…

Aristotelian cause: a cause of something Aristotelian cause: a cause of something real which is generated by the nature of real which is generated by the nature of thing itself (thing itself (generative causalitygenerative causality))

The elements of a substance tend to move The elements of a substance tend to move towards one another (i.e. gravity)towards one another (i.e. gravity)

GalileoGalileo

Galileo: the cause of the motion of planets Galileo: the cause of the motion of planets is not inherent in their nature, but due to is not inherent in their nature, but due to external ‘laws of motion’external ‘laws of motion’

Hume and Successionist CausalityHume and Successionist Causality

A cause is not real; it is inferred through A cause is not real; it is inferred through the conduct of scientific investigation into the conduct of scientific investigation into regular successive events (regular successive events (successionist successionist causalitycausality))

A sceptical approach to causality…A sceptical approach to causality…

But are causes ‘real’???But are causes ‘real’???

What is the cause of learning?What is the cause of learning?

Generative?Generative?Successionist?Successionist?Both?Both?

Objectivity?Objectivity? Various philosophical positions:Various philosophical positions:

Idealism: nothing is real (Berkeley) – just ideas in my Idealism: nothing is real (Berkeley) – just ideas in my mindmind

Post-modernism (Baudrillard, Derrida, Barthes)Post-modernism (Baudrillard, Derrida, Barthes) No objectivityNo objectivity

Empirical Realism: Reality can only be known through Empirical Realism: Reality can only be known through scientific experiment scientific experiment

Objectivity through experimentObjectivity through experiment Metaphysical Realism: reality is completely Metaphysical Realism: reality is completely

independent of conceptual schemesindependent of conceptual schemes Objectivity possible through correspondence of ideas with Objectivity possible through correspondence of ideas with

realityreality Critical Realism: Reality comprises generative Critical Realism: Reality comprises generative

mechanisms responsible for producing phenomena: mechanisms responsible for producing phenomena: transitive and intransive mechanisms.transitive and intransive mechanisms.

Objectivity possible through uncovering mechanismsObjectivity possible through uncovering mechanisms

Knowledge and JudgementKnowledge and Judgement

Knowledge: codified, rational, embodied, Knowledge: codified, rational, embodied, habitualhabitual

Judgement: ‘of the moment’, uncodified Judgement: ‘of the moment’, uncodified (uncodifiable??), habitual, instinctive(uncodifiable??), habitual, instinctiveUsually depends to some extent on codified Usually depends to some extent on codified

knowledgeknowledge

Universals and ParticularsUniversals and Particulars

The project and the particularThe project and the particularThe institution, sector (JISC), government, The institution, sector (JISC), government,

etc and the Universaletc and the UniversalThe particular instant and the judgement The particular instant and the judgement

of the individualof the individual ‘‘The project’ is the means by which work The project’ is the means by which work

in e-learning is progressed.in e-learning is progressed.How to go from the project to Universal How to go from the project to Universal

guidance for the sector…guidance for the sector…

Modes of reasoning in evaluationModes of reasoning in evaluation

P

P PP

Preality

Uknowledge

Induction (no theory approach)

Deduction (single theory approach)

P

P PP

P

U

U

UU

U

knowledge

outcomes

reality

retroduction

No shit, Sherlock!No shit, Sherlock!

Differences between induction, deduction Differences between induction, deduction and retroduction are subtleand retroduction are subtle

Retroduction attempts a multi-theory, Retroduction attempts a multi-theory, multi-methodology approach. Attempts not multi-methodology approach. Attempts not to be constrained by existing theory, to be constrained by existing theory, culture, etc..culture, etc..

Much like the work of a detective…Much like the work of a detective…

Context, Mechanism, OutcomeContext, Mechanism, Outcome

What outcome do you observe?What outcome do you observe?What mechanisms do you think might What mechanisms do you think might

produce that outcome?produce that outcome?Within what context do the mechanisms Within what context do the mechanisms

operate?operate?

Mechanisms…Mechanisms…

Cumulating Context, Mechanisms Cumulating Context, Mechanisms and Outcomes…and Outcomes…

Theory

Data

abstraction specification

C M O C M O C M O C M O

C M O C M O C M O

C M O C M O C M O

C M O Programmes

Middle-range theory

O OO Project outcomes

projects

C M O Methodology

Understanding mechanisms in Understanding mechanisms in educationeducation

Educational contexts, learning contextsEducational contexts, learning contextsEducational mechanisms, learning Educational mechanisms, learning

mechanismsmechanismsEducational outcomes, learning outcomesEducational outcomes, learning outcomes

Understanding Projects in e-Understanding Projects in e-LearningLearning

Projects contribute to context, mechanism Projects contribute to context, mechanism and outcomesand outcomes (as do the technologies of projects)(as do the technologies of projects)

Challenge is to describe how the new Challenge is to describe how the new relationships between context, mechanism relationships between context, mechanism and outcome are affected by a projectand outcome are affected by a project

ConclusionConclusion The challenge of evaluation is not necessarily The challenge of evaluation is not necessarily

finding out what works, but examining what does finding out what works, but examining what does and doesn’t work as a way of understanding the and doesn’t work as a way of understanding the mechanisms of education, and planning for mechanisms of education, and planning for future viability.future viability.

The understanding of the institution is crucial: The understanding of the institution is crucial: Realistic Evaluation focuses on understanding Realistic Evaluation focuses on understanding the institution. Positivist evaluation judges the institution. Positivist evaluation judges projects on a pre-given understanding of the projects on a pre-given understanding of the institution.institution.

Through cumulating realist descriptions, Through cumulating realist descriptions, objectivity is possible.objectivity is possible.