Writing course materials for successful flexible learning

Ormond Simpson Visiting Fellow CDE

CDE seminar - 24 June 20141

Variations in OU course module retention

T302

T331

MU120

W300

K224

60

65

70

75

80

85

90

95

100

70

% getting to exam

% p

assi

ng

exa

m

40 50 60 80 90 100

2

3

A successful distance course keeps a student’s learning motivation

switched on

Course Design- possible influences on retention

1. Course Choice - how students choose their course

2. Course workload – how much is in the course

3. Course structure – how the course is organised

4. Course writing - the writing of the course

5. Course assessment strategies - how students are assessed

6. Course evaluations - how courses are assessed

4

1. Course Choice and retention

The second biggest

reason students give

for dropping out (after

time issues) is that

they were on the

wrong course.

5

2. Course workload and retention

No link between students’ reported course workload and dropout - Crooks, 2005

But that’s: Counter intuitive?Counter experience?

Possibly a methodology problem?

Or the variety of concepts and media?

6

3. Course structure and retention

‘Flexibility’

- choice of material to study

- choice of time to study

- choice of assessment

Crooks 2005

7

John Sweller ‘Cognitive

Load Theory’

John Keller ‘ARCS’ theory

John Hattie‘Self –

Reporting’

4. Course writing - theories

Maintaining a student’s motivation to learn

Keller’s ARCS theory

A = Get and keep their Attention

R = Ensure everything is Relevant to their needs

C = Ensure they have Confidence in what they’re

doing

S = And that they are continually Satisfied with their learning experience

9

Getting attention

- Empathy – eg stories sharing personal thoughts, acknowledging difficulties

Incongruity – eg humour

Readability – Flesch tests

Relevance

ConfidencePersonal, Approachable, Using ‘I’

Satisfaction Ensuring students feel progressWorked example

Ensuring relevanceAvoiding redundancy

Conclusions from theories

Keeping attention

Keller – Keeping Attention – Readability

’Flesch Reading Ease’ Score depends on sentence length and number of

syllables per word

Readability score Interpretations

0 - 20 Very difficult

20 - 50 Difficult

50 – 60 Fairly difficult

60 – 70 Plain English

70 – 80 Fairly easy

80 – 90 Easy

90 - 100 Very easy11

Open Poly of New Zealand course on communication

The field of communication studies runs wide. As a discipline, it borders on academic specialities such as linguistics, psychology, media studies, cultural studies, sociology, philosophy, marketing, and business studies. Its diverse components include interpersonal communication, intercultural communication, workplace writing, organisational studies, and mass communication. It can, at times, be difficult to limit the scope of communication studies: it seems to involve pretty much most things human beings to together. This is an indication of the obsession in modern times with communication. As Peters (1999) notes, communication has been viewed as the solution to humanity’s diverse and profound troubles. How many times, for instance, during local or international conflicts have you heard talk of communication breakdowns, or of the need to open channels of communication? In this optimistic view, communication entails connecting with others, expressing our true selves, alleviating loneliness, preventing misunderstandings and conflict. The other side of the attention to communication in the modern world is a focus on the dilemmas and apparently intractable problems of communication. Peters (1999) points to the way that so much twentieth century humour focuses on human miscommunication, where humans are left dazed and lonely – gesticulating wildly, impotent, ridiculous.

12

Open Poly of New Zealand course on communication

The field of communication studies runs wide. As a discipline, it borders on academic specialities such as linguistics, psychology, media studies, cultural studies, sociology, philosophy, marketing, and business studies. Its diverse components include interpersonal communication, intercultural communication, workplace writing, organisational studies, and mass communication. It can, at times, be difficult to limit the scope of communication studies: it seems to involve pretty much most things human beings do together. This is an indication of the obsession in modern times with communication. As Peters (1999) notes, communication has been viewed as the solution to humanity’s diverse and profound troubles. How many times, for instance, during local or international conflicts have you heard talk of communication breakdowns, or of the need to open channels of communication? In this optimistic view, communication entails connecting with others, expressing our true selves, alleviating loneliness, preventing misunderstandings and conflict. The other side of the attention to communication in the modern world is a focus on the dilemmas and apparently intractable problems of communication. Peters (1999) points to the way that so much twentieth century humour focuses on human miscommunication, where humans are left dazed and lonely – gesticulating wildly, impotent, ridiculous.

FRE = 13.5 = Very difficult

13

• The field of communication studies runs wide. As a discipline, it borders on academic specialities such as: – linguistics, – psychology, – media studies, – cultural studies, – sociology, – marketing, – business studies.

• Its diverse components include: – interpersonal communication, – intercultural communication, – workplace writing, – organisational studies, – mass communication.

• It can sometimes be difficult to limit the scope of communication studies. It seems to involve pretty much most things human beings do together. This shows the obsession today with communication.

FRE = 62 = Plain English

14

15

Evidence for Keller’s Theory?

‘Motivational measure of the instruction

compared: Instruction Based on the ARCS

Motivation Theory vs Traditional Instruction

in Blended Courses’

Ozgur M. COLAKOGLU & Omur AKDEMIR

Turkish Online Journal of Distance Education

(TOJDE) 2010

Cognitive Load Theory (Sweller, 1998)

Transfer of learningInitial learning goes into the

working (short) memory and needs to be transferred to long term memory

16

Learning Working memory

Long term

memory

Three types of cognitive load

1. Intrinsic – due to inherent difficulty of subject. Should be managed by e.g. ‘segmentation’ and using worked examples

2. Extraneous – due to way information is presented. Should be minimised by e.g. Ensuring relevance avoiding redundancy

3. Germane – due to way info’ relates to previous info’ Should be maximised by making clear links between new and old information 17

‘Assessing Cognitive Load Theory to improve Student Learning for Mechanical Engineers’ - Impelluso,V. American Journal of Distance Education 23 (4) (2009)

- claimed increased retention and learning

18

5. Course assessment strategies

‘Assessment drives learning’But…

Does assessment also drive dropout?

19

1

100

62 57 52

38

43

48

7 2

5

ASSIGNMENT RIVERGRAM100 students start the course. At each assignment some drop out and enter the ‘exit’ channel. A very few re-enter the ‘progress’ channel having skipped the previous assignment

Assignment 1

Assignment 2

Assignment 3

Progress

Exit

Evidence - what does research say about assessment and retention?

(1) Hattie (2009) most important is ‘self reporting of grades’ – so a student knows how well (s)he is doing

21

‘Self reporting of grades’

Put ‘self-assessment questions’ in text - but how to get students to do them?

1. Tell them

2. Nudging – “most students do the SAQ’s” “research has shown that students who do the SAQ’s do best in the course”

3. Electronic text – make it impossible to move on unless the SAQ is attempted

22

E-teaching- can be provided in two ways

Course texts in print.Teaching / support online

Everything online including course

texts

Reading off a screen can be 30% less effective than reading paper copy http://www.scientificamerican.com/article/reading-paper-screens/

24

E-teaching Advantages of e-teaching – can use many different kinds of media – forums, podcasts, video/audio clips, blogs etc

Disadvantages of e-teaching – can use many different kinds of media….

‘Course Exuberance Syndrome’

‘Learning value’ vs. ‘learning time needed’

Learning value poor

Learning value rich

Learning time needed short

Learning time long

Paper text

Online text

Computer forums

blogs

podcasts

Videoclips

Wikis

Facebook

‘Use line’

Secondlife

email

‘Learning time

needed’

‘Learning value’

pencasts

Audioclips

ebooks

26

6. Course evaluations

1. Asking the students

2. Data analysis

27

Asking the students

OU Courses survey

‘Crash testing’

28

29

Module title Students started

Students passed

Predictedpass

Actual pass

Z-score

‘Creative writing’ 1,995 1,615 77.2% 81.0% + 4.38

‘Introducing religions’

355 254 62.9% 71.5% + 3.71

‘Medicine and society in Europe

1500-1930’357 227 69.6% 63.6% - 2.80

‘Exploring the classical world’

605 414 68.9% 68.4% - 0.29

‘Exploring philosophy’

699 500 65.3% 71.5% + 3.84

‘Inside music’ 413 255 74.0% 61.7% - 6.34

Z-scores for a sample of OU modules

Z-scores outside the range +4 or -4 are considered significant

Variations in course retention

T302

T331

MU120

W300

K224

60

65

70

75

80

85

90

95

100

70

% getting to exam

% p

assi

ng

exa

m

Sifters

Heavy goings

Fair

Knock Backers

40 50 60 80 90 100

30

Barriers to ‘retention-friendly’ text?

31

Material focused

ExclusiveHighly

challenging Serious

Student focusedInclusive

Aptly challengingEnjoyable

Weeding out the unfit –

‘Darwinistas’

Sink or swim - ‘Fatalistas’

Help students be the best they can

be- Retentioneer

Thank you!

More material onwww.ormondsimpson.com

32

33

4. Motivational and ungraded feedback

(i) Motivational feedback (Wigfield et al, 2009):

Learning motivation = (assumed possibility of accomplishing task)

x (perceived value of task)

If either factor is zero then motivation is zero

So feedback tasks should be carefully graded for difficulty

34

Formative assessment – the evidence

• Yorke and Longden (2004)

• Black and Wiliam (1998)

• Kluger (1996)

• Gibbs and Dunbar-Goddet (2007)

35

Without external formative assessment 1

100

62 57 52

38

43

48

7 2

5

ASSIGNMENT RIVERGRAM – FORMATIVE ASSESSMENT100 students start the course. At each assignment some drop out and enter the ‘exit’ channel. A very few re-enter the ‘progress’ channel having skipped the previous assignment

Assignment 1

Assignment 2

Assignment 3

Progress

Exit36

With external formative assessment

100

72 70 65

28

30

35

11 9

5

ASSIGNMENT RIVERGRAM – FORMATIVE ASSESSMENT100 students start the course. At each assignment some drop out and enter the ‘exit’ channel. A very few re-enter the ‘progress’ channel having skipped the previous assignment.

Assignment 1

Assignment 2

Assignment 3

Progress

Exit37

Keller’s A = Attention

1. Getting students attention2. Keeping students attention

38

Keller - Getting Attention

Use:- incongruity – eg humour*- empathy – eg stories, sharing

personal thoughts - authority – displaying expertise

But everything in a course text should be relevant to learning.

*eg see ‘Flipnosis’ – Dutton (2011)

39

Keller - Keeping attention - Reading skills and reading habits

• Researchers used ‘Cloze’ tests (replacing blanks in text) on new students – many would have significant difficulties in understanding their course material.

- 42% new students had lower comprehension than needed for courses - Students’ previous reading was newspapers and magazines

- Datta and Macdonald Ross

(2002)

40

Readability Scales

• Automated Readability Index

• Flesch Reading Ease

• Flesch-Kincaid scores

• Gunning-Fog index

• SMOG index (simplified measure of gobbledygook)

41

Newspapers Flesch Reading Ease score

Sun 62.8 – Plain English

Daily Mail 61.5 – Plain English

Mirror 60.5 – Plain English

Guardian 44.5 – Difficult

Telegraph 48.8 – Difficult

42

OU coursesinitial pages

Flesch Reading Ease score

Arts Foundation 47.1 – difficult

Social science Foundation

55.2 – Fairly difficult

Maths Foundation 39.9 - difficult

Science Foundation

53.7 – Fairly difficult

OU Access Course 58.1 – Fairly difficult

Significant differences in readability levels between tabloid newspapers and UKOU courses - Moore (2004)

43

Fonts

• Times New Roman• Arial • Helvetica• Courier • Comic • Calibri• Cambria• Brush • Kumasi Pioneer

Two groups of students were given exercise advice in Arial and Brush fonts.

Students getting Arial text were more likely to change exercise behaviour than students getting the Brush text

Herbert - Sci. Amer. Mind Feb 2009

44

Type layout - Justifying 1

There appears to have been relatively little work done on course design and retention. Indeed designing a course for better retention rates is a complex concept since course design is intimately bound up with course assessment. In recent years in the UK the annual festival of punditry revolves around the question of whether more students passing the ‘A’ level exams means that ‘standards’ have therefore fallen. Thus the retention debate has tended to avoid questions of the difficulty of content and concepts in the course and concentrated on other possibilities for increasing retention such as course workload, course readability and course design and structure.

45

Justifying 2

There appears to have been relatively little work done on course design and retention. Indeed designing a course for better retention rates is a complex concept since course design is intimately bound up with course assessment. In recent years in the UK an annual festival of punditry revolves around the question of whether more students passing the ‘A’ level exams means that ‘standards’ have therefore fallen. Thus the retention debate has tended to avoid questions of the difficulty of content and concepts in the course and concentrated on other possibilities for increasing retention such as course workload, course readability and course design and structure.

46

Using columns

There appears to have been relatively little work done on course design and retention. Indeed designing a course for better retention rates is a complex concept since course design is intimately bound up with course assessment.

In recent years in the UK an annual festival of punditry revolves around the

question of whether more students passing the ‘A’ level exams means that ‘standards’ have therefore fallen.

Thus the retention debate has tended to avoid questions of the difficulty of content and concepts in the course and concentrated on other possibilities for increasing retention such as course

47

Newspapers

o Different fontso Narrow columnso Short paragraphso Justified – sometimes ragged righto Many graphicso Frequent subheadings and quote

boxeso Variable fontso Colour and shading - all aimed at ‘Attention’

48

R = Relevance

• Avoid redundancy•

Avoid ‘split attention’

• - see ‘Cognitive Load Theory’

49

C = Confidence

• Why are texts largely anonymous?- could we use ‘I’ or ‘we’?

• Why little information about authors?– should we ‘personalise’ texts?

50

S = Satisfaction

Feedback to student - Hattie - most important is ‘self reporting’ – so a student knows how well (s)he is doing’

So use self assessment questions?

• put answers straight after questions

• keep questions short so students can see progress 51

Types of assessment

1. Formative - to enhance learning, not to allocate grades - usually ungraded - feedback for student’s benefit 2. Summative – graded - counts towards final grade

- continuous or final exam

3. Ipsative - iterative feedback

52

‘Meta-cognitive awareness’

(3) Gibbs (2010) – marking exercises - enhanced worked examples

Some indirect evidence for increased retention

53

Gibbs – (1) Marking exercises

Students given specimen student assignment answers and asked to mark them

54

55

Graham Gibbs - Marking exerciseF2F GROUP ACTIVITY 2: What do 'they' want? (25

mins)

Give students copies of a short student assignment.

1. Tell them, “Imagine you are a tutor - read this quickly and mark it. Give it a grade (100 to 0) and helpful comments." (5 mins)

2. In pairs, “Compare your comments and grading together. What have you picked out as important in your comments?“ (10 mins)

3. In plenary, “Right let's now try to draw out some good and bad points from this to help us in our own writing". (10 mins)

Gibbs - (2) Enhanced worked examples

“Last week I found 50 New Zealand dollars in my suitcase.

“So I rushed down to the Post Office where there was one of those illuminated signs in the window. Against NZ$ it said ‘We buy at NZ$2.72 to the £’.

So how much did the man behind the counter give me?”

56

Enhanced worked example - answer

“The statement ‘We buy at NZ$2.72 to the £’ means that for every NZ$2.72 I give the man he’ll give me £1.

I've got NZ$50 so we need to know how many times 2.72 goes into 50 to see how many £’s I'll get.

Now 50/2.72 = 18.51851852 on my calculator. So the man gave me £18.51 as the post office doesn’t deal in less than a penny of course.”

57

Enhanced worked example - answer

“The statement ‘We buy at NZ$2.72 to the £’ means that for every NZ$2.72 I give the man he’ll give me £1. (It helps to write out what the statement ‘We buy at NZ$2.72 to the £’ means in practice).

I've got NZ$50 so we need to know how many times 2.72 goes into 50 to see how many £’s I'll get. (remember to explain each step rather than just write down the calculation like 'NZ$ = £50/2.17' The more you explain what you're doing, the better).

Now 50/2.72 = 18.51851852 on my calculator. So the man gave me £18.51 as the post office doesn’t deal in less than a penny of course.” (Be sure your answer makes sense – it's very easy when using the calculator to write down whatever it tells you!) 58

With external formative assessment

100

72 70 65

28

30

35

11 9

5

ASSIGNMENT RIVERGRAM – FORMATIVE ASSESSMENT100 students start the course. At each assignment some drop out and enter the ‘exit’ channel. A very few re-enter the ‘progress’ channel having skipped the previous assignment.

Assignment 1

Assignment 2

Assignment 3

Progress

Exit59

‘Ipsative assessment’ - a new approach?

• Assessment or feedback which compares a student’s achievement not with an absolute standard, but with their previous performance (Hughes, 2011)

• For example students can repeat an assessment with feedback at each attempt until they reach the required standard

• No retention results as yet

60