Measuring the impact of social and informal learning
Preview:
Citation preview
- 1. Measuring the Impact ofSocial and Informal LearningJohn
Kleeman / Chairman, Questionmarkjohn@questionmark.com Thursday,
26th January 2012Copyright 1995-2012 Questionmark Corporation
and/or Questionmark Computing Limited, known collectively as
Questionmark. All rights reserved. Questionmarkis a registered
trademark of Questionmark Computing Limited. All other trademarks
are acknowledged.
- 2. Learning is Changing Academic modelworked for 2500 yearsSign
up Internet changingthingsLearnGoogle itChanging work
rolesTestDeeper understanding of learningDo workCopyright 1995-2012
Questionmark Corporation and/or Questionmark Computing Limited,
known collectively as Questionmark. All rights reserved.
Questionmarkis a registered trademark of Questionmark Computing
Limited. All other trademarks are acknowledged.
- 3. Only ~10% of learning Learning from your comes from
studymentors, conversations (reading, researching, with peers and
meetings courses, etc.) (if you pay attention and ask
questions)70+20+10 Formal Learningmodel oflearningLearning From
Others10% 20%Learning By Doing On-The-Job learning by making
mistakes, taking 70% time for retrospectivesand learning from the
experience. Copyright 1995-2012 Questionmark Corporation and/or
Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmark is a registered trademark of
Questionmark Computing Limited. All other trademarks are
acknowledged.
- 4. 70+20+10 model for learning If you invest in~70% from
70+20+10, how do you Experience measure the impact? ~20% from Need
to measure Others Are people learning?Are people performing better?
~10% from StudyCopyright 1995-2012 Questionmark Corporation and/or
Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. Questionmarkis a registered trademark of
Questionmark Computing Limited. All other trademarks are
acknowledged.
- 5. Assessments? Assessments are a critical part of learning
Quizzes Surveys Tests Exams How can assessments Measure informal
learning? Measure improved performance?Copyright 1995-2012
Questionmark Corporation and/or Questionmark Computing Limited,
known collectively as Questionmark. All rights reserved.
Questionmarkis a registered trademark of Questionmark Computing
Limited. All other trademarks are acknowledged.
- 6. Assessing Impact of Social/InformalLearning Initiatives Did
they learn it? Did theyAre theylike it? doing it?ResultsCopyright
1995-2012 Questionmark Corporation and/or Questionmark Computing
Limited, known collectively as Questionmark. All rights reserved.
Questionmarkis a registered trademark of Questionmark Computing
Limited. All other trademarks are acknowledged.
- 7. Did they like it? Use surveys to evaluateattitudes/reactions
Specific content / initiatives Whole 70+20+10 initiative Use
surveys to identifysuccess cases using theBrinkerhoff method
Surveys Side-by-side with learning Delivered on mobile devices More
conventional routesCopyright 1995-2012 Questionmark Corporation
and/or Questionmark Computing Limited, known collectively as
Questionmark. All rights reserved. QuestionmarkSlide 7is a
registered trademark of Questionmark Computing Limited. All other
trademarks are acknowledged.
- 8. Mobile Assessments Questionmark assessments: Can run via
browser Or on mobile devices via iPhone, iPad or AndroidCopyright
1995-2012 Questionmark Corporation and/or Questionmark Computing
Limited, known collectively as Questionmark. All rights reserved.
Questionmarkis a registered trademark of Questionmark Computing
Limited. All other trademarks are acknowledged.
- 9. QR codes can be used to launch surveyseasily BooksPowerPoint
Signage Benefits Ease of access for increased response rates
Demographic data easily captured Better data accuracy Easier
analysis of survey results by demographicCopyright 1995-2012
Questionmark Corporation and/or Questionmark Computing Limited,
known collectively as Questionmark. All rights reserved.
QuestionmarkSlide 9is a registered trademark of Questionmark
Computing Limited. All other trademarks are acknowledged.
- 10. Survey & Course Evaluation Analytics Copyright
1995-2012 Questionmark Corporation and/or Questionmark Computing
Limited, known collectively as Questionmark. All rights reserved.
QuestionmarkSlide 10 is a registered trademark of Questionmark
Computing Limited. All other trademarks are acknowledged.
- 11. Did they learn it? Formative assessments Help direct
learning Strengthen memory Engage learners In 70+20+10
user-generated quizzes provide extra dimension Measure knowledge
and skills Compare before to after Compare to benchmark Compare to
others Copyright 1995-2012 Questionmark Corporation and/or
Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. QuestionmarkSlide 11 is a registered trademark
of Questionmark Computing Limited. All other trademarks are
acknowledged.
- 12. Quiz attached to video in SharePoint Copyright 1995-2012
Questionmark Corporation and/or Questionmark Computing Limited,
known collectively as Questionmark. All rights reserved.
QuestionmarkSlide 12 is a registered trademark of Questionmark
Computing Limited. All other trademarks are acknowledged.
- 13. Embed Youtube and other hosted video inQuestionmark Live
items!Embed YouTube or other hosted videodirectly in
questionstimulusCopyright 1995-2012 Questionmark Corporation and/or
Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. QuestionmarkSlide 13is a registered trademark
of Questionmark Computing Limited. All other trademarks are
acknowledged.
- 14. Questions Through the Learning ProcessReduce Forgetting
CreateIntrigue StrengthenKnowledge/SkillsMemory
QuestionsQuestionsQuestions Questions Questions TimeCopyright
1995-2012 Questionmark Corporation and/or Questionmark Computing
Limited, known collectively as Questionmark. All rights reserved.
Questionmarkis a registered trademark of Questionmark Computing
Limited. All other trademarks are acknowledged.
- 15. Quizzes and tests slow forgetting curve Taking a quiz or
test Knowledge Retained gives retrieval90%80% practice 70%
Reinforces mental60% After 5 pathways to make 50%56% mins40% future
recall more 30%42%After 1 likely 20%week10% 0%Data from
Test-Enhanced Learning,Roediger and Karpicke 2006. Study only
Study, TestPsychological Science Copyright 1995-2012 Questionmark
Corporation and/or Questionmark Computing Limited, known
collectively as Questionmark. All rights reserved. Questionmark is
a registered trademark of Questionmark Computing Limited. All other
trademarks are acknowledged.
- 16. Embed in a Wiki Embedded in a Wiki Copyright 1995-2012
Questionmark Corporation and/or Questionmark Computing Limited,
known collectively as Questionmark. All rights reserved.
QuestionmarkSlide 16 is a registered trademark of Questionmark
Computing Limited. All other trademarks are acknowledged.
- 17. Embedding in Blogs, Portals, Wikis For Learners For
Instructors Focus the learners attention Easily use assessments
Practice memory recallthroughout the learning Identify what was
learned process Provide seamless user Provide feedback to assist
the experience learner to: Centralised management of Correct
misconceptions assessment content and Instill confidence Point to
relevant material results Impact data on social learning Copyright
1995-2012 Questionmark Corporation and/or Questionmark Computing
Limited, known collectively as Questionmark. All rights reserved.
QuestionmarkSlide 17 is a registered trademark of Questionmark
Computing Limited. All other trademarks are acknowledged.
- 18. Compare to Benchmarks
BenchmarkParticipantTopiclevelOverallCopyright 1995-2012
Questionmark Corporation and/or Questionmark Computing Limited,
known collectively as Questionmark. All rights reserved.
QuestionmarkSlide 18is a registered trademark of Questionmark
Computing Limited. All other trademarks are acknowledged.
- 19. Compare to Others Participant Topiclevel Overall Copyright
1995-2012 Questionmark Corporation and/or Questionmark Computing
Limited, known collectively as Questionmark. All rights reserved.
QuestionmarkSlide 19 is a registered trademark of Questionmark
Computing Limited. All other trademarks are acknowledged.
- 20. Are they doing it? In 70+20+10, people are learning In
multiple places At different times In different ways Measuring
performance more meaningful than measuring learning Observational
assessments can do this Copyright 1995-2012 Questionmark
Corporation and/or Questionmark Computing Limited, known
collectively as Questionmark. All rights reserved.
QuestionmarkSlide 20 is a registered trademark of Questionmark
Computing Limited. All other trademarks are acknowledged.
- 21. Observational assessment vs. traditional quiz, test or exam
Traditional assessment Participant is presented with questions for
direct response Observational assessment Observer is presented with
questions to rate participant to measure knowledge, skills and
abilitiesCopyright 1995-2012 Questionmark Corporation and/or
Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. QuestionmarkSlide 21is a registered trademark
of Questionmark Computing Limited. All other trademarks are
acknowledged.
- 22. What are Observational Assessments? Observational
assessments provide a mechanism for an observer to assess a
participant For example, a participant might be observed and rated
on Performance of a specific task or procedure Knowledge
acquisition and application Demonstration of skills/abilities that
traditional assessment may be less reliable in measuringCopyright
1995-2012 Questionmark Corporation and/or Questionmark Computing
Limited, known collectively as Questionmark. All rights reserved.
QuestionmarkSlide 22is a registered trademark of Questionmark
Computing Limited. All other trademarks are acknowledged.
- 23. If assessing performance of a task, observertypically rates
participant in three areas:Performing thePreparationWrap Up task
Copyright 1995-2012 Questionmark Corporation and/or Questionmark
Computing Limited, known collectively as Questionmark. All rights
reserved. Questionmark is a registered trademark of Questionmark
Computing Limited. All other trademarks are acknowledged.
- 24. Typical workflow for observer Monitor/observer logs in
Selects assessmentSelects participant to be observedAnswers/submits
responses to questions Results/feedback reported to
stakeholdersCopyright 1995-2012 Questionmark Corporation and/or
Questionmark Computing Limited, known collectively as Questionmark.
All rights reserved. QuestionmarkSlide 24is a registered trademark
of Questionmark Computing Limited. All other trademarks are
acknowledged.
- 25. Monitor/observer logs in
- 26. Selects assessment
- 27. Selects participant tobe observed
- 28. Answers/submitsresponses toquestions
- 29. Example applications of Observational
AssessmentCertificationObservational assessments can be used as
critical components of the certification process in some cases are
required by regulatory authorities.ProcessPerformanceAn
organization has delivered a training course to its employees and
is now evaluating its effectiveness. By observing and monitoring
behavioral changes inor level 3 the workplace before and after the
training, the organization will be able to assessassessment:the
changes caused by the training and evaluate the return on
investment gained.EquipmentObservational assessment can be used to
record and rate operators skills, abilities, safety practices and
adherence to required proceduresoperation:Customer Workplace
Assessments can also be used to evaluate retail sales and customer
service staff on how well they cope in different situations to
assess whereServiceadditional training is required or where
personnel excel.evaluation:Copyright 1995-2012 Questionmark
Corporation and/or Questionmark Computing Limited, known
collectively as Questionmark. All rights reserved. Questionmarkis a
registered trademark of Questionmark Computing Limited. All other
trademarks are acknowledged.
- 30. Feedback to Participant on PerformanceResults/feedback
reported tostakeholders
- 31. Mobile devices Mobile devices such as tablets and smart
phonesideal for delivering observational assessments Copyright
1995-2012 Questionmark Corporation and/or Questionmark Computing
Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing
Limited. All other trademarks are acknowledged.
- 32. The same assessmentthat is delivered on traditional laptop
can be delivered via ahandheld mobile devicesuch as iPad, iPhone or
Android phone/tablet
- 33. How to measure 70+20+10 impact?Did theylearn it?Did theyAre
they like it? doing it? Results Copyright 1995-2012 Questionmark
Corporation and/or Questionmark Computing Limited, known
collectively as Questionmark. All rights reserved. Questionmark is
a registered trademark of Questionmark Computing Limited. All other
trademarks are acknowledged.
- 34. Visit Questionmarkat Stand 60 Questions? Twitter:
@questionmark and @johnkleemanMore info:
www.questionmark.com/go/observationalCopyright 1995-2012
Questionmark Corporation and/or Questionmark Computing Limited,
known collectively as Questionmark. All rights reserved.
Questionmarkis a registered trademark of Questionmark Computing
Limited. All other trademarks are acknowledged.