The OLDS MOOC Evaluation Plan and an Experiment in Recognising Learning usingBadges
Slides by Simon Cross and Rebecca GalleyInstitute of Educational Technology, The Open University
Presented by Simon Cross as part of:Cross, S. and Mor, Y. (2013) OLDS MOOC: design, evaluation and badges, eLearning Community Event: MOOCs at the Open University: past, present and future, 13 March 2013, OU, Milton Keynes
For more on the event visit: http://cloudworks.ac.uk/cloud/view/8043
1. OLDS MOOC Evaluation Plan(themes, structure, questions)
2. OLDS MOOC Badge Strategy(what are badges, what are they for, and how could we use them)
3. Use and user comments(examples of twitter posts that
comment on role of badges + Bonus slide on expectations)
Presentation overview
OLDSMOOCEvaluationStrategy
Questions onRegistration
Prior understandingand knowledge
Previous Experienceof MOOCsQuestionnaire
n=1400+n=35
MOOC ‘EvaluationPanel’ recruited before start for longitudinal study and to ensure weare likely to interviewsome people at endwho ‘dropped out’
n=15
September 2012
April 2013
Expectations andMotivationsQuestionnaire: oneweek prior to start
n=60+
Weekly manual recording of usestatistics from Cloudworks andBibsonomy
Weekly capture andarchive of all tweetsmade across thechannels
Recording of datarelating to badge approvals
Feedback form submissions
n=25 at present n=80+ and rising
MOOC ExperienceQuesitonnaire - opento all
Opens March
Telephone interviewswith Panel and othersthought to beinteresting cases
Opens March
MOOC STARTS
MOOC ENDS
MOOC Evaluation Themes
Motivations
Expectations
Quality / Quantity
Groups & Support
Badges / Reward
MOOC Makers
Expectations and Motivation in MOOCs A review of pre-course survey results and user posts will provide insight in to what people expected of the MOOC and why they were engaging with it.
What variety of expectation is there?
How does this compare to the reality of the MOOC experience?
Who and when could we use this type of MOOC structure/format?
How is content and structure informed by the aims and intentions of the designers?
Use of Badges and approach to reward
Nine badges have been developed in conjunction with a badging strategy. A method of creating and awarding badges added to Cloudworks.
What role did the badges have in motivating participants and preventing withdrawal?
What can we learn about using badges to support learners and promote learning?
1. What are badges?
2. What are they for?
3. How could we use them?
… September 2012
… 1. What is a badge?
My working definition:
“A badge is visual public symbol that communicates to others a particular quality, achievement or affiliation possessed by the owner. It is one ofa variety of ways that such recognitioncan take a material form.”
… 2. What are they for?
The Role of the badge for receiver and issuer…
As a motivatorAs an engager As a meaning makerAs a signifier of learning objectivesAs a valuerAs a symbol of identityAs a means of associationAs an empowerAs an entrencher
http://www.olds.ac.uk/blog/moocbadgingandthelearningarc
http://www.olds.ac.uk/blog/moocbadgingandthelearningarc
… 3. How can we use them?
Additional Cloudworks technical development undertaken:
•Badge unload functionality•Define approval criteria•Apply for badge•Approval (two processes)•Display on personal profile•Aligned to Mozilla badging
… 3. How can we use them?
Additional Cloudworks technical development
Endeavour and Progress
Achievement
Practice
http://www.olds.ac.uk/blog/olds-moocbadgingstrategy
OLDS MOOC Badges
http://www.olds.ac.uk/blog/moocprofiles
OLDS MOOC Participant Profile
From a sample of registrations
Study planning
[Am] interested in gaining badges, and working with colleagues in a local study group.
[My personal aims]: 1. completing one week of the MOOC and applying for a badge (maybe not getting the badge but at least applying for one week:)
I like the Badges, works as motivation for me. [Will] chose to get [the] Endeavour [ones] due [to] time constraints. Maybe Achievement.
Badges and Participation
122012201310
Week 1 Week 3
340
100
200
3. Week 3 Cloudscape
4. Sharing LD representations Cloud (Activity 6-7)
1. Introduce yourself cloudscape
2. Dream (Project) Bazaar cloudscape
1 2 3 4
FollowingFollowingContributing / DoingUnique viewsBadge
A bit of fun ... even a surprise
@[B]: Earned my Week 1Badge from #oldsmooc ! Unexpectedly pleased to have a reward for efforts! What fun and motivates …
Wooh [I] too have got a badge from #oldsmooc_w1 how #awsome Just applied for my #oldsmooc week 1 badge. Not that it matters but .... if I get it I will print it out (in colour) and pin it on my lapel
First #oldsmooc badge...what fun, feel like a Girl Guide again
Thursday motivation - got my 3 weeks #oldsmooc badge” @[T] Well done and snap! Delighted to get my 3 weeks badge too - likin' those badges more than expected!
Confidence, collecting, something to show
Guiding reflective learning
Hooray for my first @openbadges!
... and badges def. do work. I'm bagsing as many as I can to show that I'm learning loads… I love #openbadges. See my #oldsmooc badges [x]
Impressed #oldsmooc badge linked to evidence of reflective blogging #openbadges > me too, I got a badge :-)
@[R] I love the #oldsmooc badges http://t... - keep up the reporting on your learning, love it
Reason to ask for peer review
Facilitator: OER developer badge is crowdverified, you need to get your peers to approve your application. don't be shy to ask!
Have applied for OER developer badge, pls verify http://t...
Hi Folks Just applied 4 my #oldsmooc collaborator badge http://t.co/[x] Could you do the honours?
Supporting sense of community
Just applied for Resource Gatherer Badge...anyone else? thanks to @[D]'s reminder I have got another #oldsmooc badge http://t.co/[X]...
I got another #oldsmooc badge!! #oldsmooc_wk4 looks puzzling though …See how we go :)
Yay! #cloudworks #badges now link to "Who has been awarded this badge?”
… Just need to figure out my #mozilla backpack now :)
and checking progress
‘Own Path’ participants
I haven't qualified for a badge yet as I'm pursuing a somewhat singular path through #oldsmooc -- I'm OK with that
“Oh - and a final word: don't let badge envy ruin your day :)”
“I’m starting to really enjoy OLDSMOOC, I’ve acceptedthat I’m no longer on the approved route but wanderingabout reading, thinking about stuff and chatting withlike-minded people is great learning anyway. Having said that, I got my Week 1 badge today!”
Concluding remarks
• Promoting deeper and more expansive learning?
• Promoting richer learning experience?
• Alternatives to summative and formative assessment?
• Appeal to broader range of study motivations / practice?
• Help students better value their study?
• Help students better communicate their achievements?
Concluding remarks
Design considerations
• Overjustification• Reflective practice / Affirmation• Status / Reputation• Overlearning• Goal setting / Identity• Community endorsement • Whenever• Fun and enjoyment• Collecting …
Immersion in an online social environmentBrushing up on skills or knowledgeLearning new skills, knowledge, participation skillsInternational working with many people from many different countries Local working with colleaguesStructured format for thinking about and designing learningBeing challengedTaking time out to reflect on practiceNew networks Trying a MOOC / a (c)MOOC / new technologiesAchieving objectivesCompleting the course Completing the subject of the course - a learning projectExploring or gaining exposure to new resources / design tools / teaching tools or technologyExperiencing a MOOC and seeing how it worksEvaluating the potential of the MOOCDocumenting experience to publish, for own reflection or for studentsResearch case study or for research dataTeam work skills required / seeing how team working is made to workPeer discussion of issues around the subjects with peersPeer feedback on own learning projects and ideasCollaboration with others on developing a courseSharing and aggregating toolsComplementing formal study (post-graduate / graduate courses)Gain badgesGroup working with local colleaguesLearning how to exploit technology for teachingOverview of field Extending existing learnt knowledge in area previously studiedCutting edge ideas and research
What do you want to get out of it?
Just 8 of 39 mentioned completion
As a bonus, here is a summaryof just some of the reasons participants gave for wanting todo the MOOC. More to follow inthe full project evaluation
Please use an appropriate reference or acknowledgement when
citing this presentation.
For further information please contact:
[email protected]@[email protected]
This presentation was prepared on the eve of the final weekof the OLDS MOOC. A full evaluation will take place betweenMarch and May 2013 so watch out for more detailedanalysis and review at:
OLDS MOOC Blog: http://www.olds.ac.uk/blog
And personal reflection at:
Simon’s Blog: http://latestendeavour.wordpress.com/