• Home

  • Documents

  • “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:)...
30
“Learning engineering”: The Art of Using Learning Science at Scale to Li7 Performance October 2015

“Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

  • Upload
    vandieu

  • View
    224

  • Download
    4

Embed Size (px)

Citation preview

Page 1: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

“Learning  engineering”:  The  Art  of  Using  Learning  Science  at  Scale  to  Li7  Performance  

October  2015  

Page 2: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

2

•  Kaplan University •  Kaplan Legal Education •  Kaplan Professional

Education •  KU Nursing

•  Kaplan Higher Ed – Europe •  Kaplan Professional – Europe

•  Kaplan Higher Ed – Asia •  Kaplan Professional – Asia •  Kaplan Higher Ed – Australia •  Kaplan Professional –

Australia •  In Country Pathways – China •  BEO - HO

Kaplan education spans domains and geography

Kaplan University Group

Kaplan Test Prep Kaplan Performance Solutions

Kaplan Asia Pacific Kaplan United Kingdom •  Kaplan Int’l Colleges

•  Global Pathways •  Dublin Business School

Kaplan International Colleges

•  KTP Grad •  KTP Pre-College •  KTP Med •  KTP Nursing •  KTP Bar Review •  Dev Boot Camp •  KTP International

Page 3: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

3

We know a lot about how expertise works

Page 4: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

4

This allows for a “learning engineering” approach

Cognitive Task Analysis

Page 5: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

5

From expertise to instructional design

Evidence- Based Instructional Design

Cognitive Task Analysis

Page 6: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

6

How  exper)se  gets  acquired  with  prac)ce  

Stage Implications for Instructional Design

Page 7: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

7

How  exper)se  gets  acquired  with  prac)ce  

Stage Implications for Instructional Design

Declarative Clear information displays, e.g., job aids, examples, reference material

Page 8: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

8

How  exper)se  gets  acquired  with  prac)ce  

Stage Implications for Instructional Design

Declarative Clear information displays, e.g., job aids, examples, reference material

Procedural Build varied Practice tasks, and rich feedback/coaching

Page 9: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

9

How  exper)se  gets  acquired  with  prac)ce  

Stage Implications for Instructional Design

Declarative Clear information displays, e.g., job aids, examples, reference material

Procedural Build varied Practice tasks, and rich feedback/coaching

Automated Repeated frequent practice to build speed and accuracy

Page 10: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

10

Knowledge Component

Practice/Assessment S

uppo

rtive

Kno

wle

dge

Research gives us clearer guides to help with hard outcomes

Page 11: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

11

Knowledge Component

Practice/Assessment

Procedure Decide when to use; perform the steps

Sup

porti

ve K

now

ledg

e Research gives us clearer guides to help with hard outcomes

Page 12: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

12

Knowledge Component

Practice/Assessment

Procedure Decide when to use; perform the steps

Sup

porti

ve K

now

ledg

e Fact Recall fact in task context

Research gives us clearer guides to help with hard outcomes

Page 13: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

13

Knowledge Component

Practice/Assessment

Procedure Decide when to use; perform the steps

Sup

porti

ve K

now

ledg

e Fact Recall fact in task context

Concept Classify, identify or generate examples and non-examples

Research gives us clearer guides to help with hard outcomes

Page 14: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

14

Knowledge Component

Practice/Assessment

Procedure Decide when to use; perform the steps

Sup

porti

ve K

now

ledg

e Fact Recall fact in task context

Concept Classify, identify or generate examples and non-examples

Process Identify causes of faults in a process; predict events in a process

Research gives us clearer guides to help with hard outcomes

Page 15: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

15

Knowledge Component

Practice/Assessment

Procedure Decide when to use; perform the steps

Sup

porti

ve K

now

ledg

e Fact Recall fact in task context

Concept Classify, identify or generate examples and non-examples

Process Identify causes of faults in a process; predict events in a process

Principle Decide if principle applies; predict an effect; apply principle to solve a problem, explain a phenomenon or make a decision

Research gives us clearer guides to help with hard outcomes

Page 16: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

16

We  also  know  more  about  mo)va)on.  .  .  Motivation Beliefs

•  Value •  Self-Efficacy •  Attribution •  Mood

Motivation Actions

•  Starting •  Persisting •  Mental Effort

Learning/Results

•  Practice results •  Test results •  Fluency/ease •  Work results

Self-Efficacy

Lear

ning

Effo

rt

High Moderate Low

Page 17: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

17

And about specific ways to make lessons work better

Principle Description Effect size (s.d. units)

Multimedia Use relevant graphics and text to communicate content 1.5

Contiguity Integrate the text nearby the graphics on the screen – avoid covering or separating integrated information

1.1

Coherence Avoid irrelevant graphics, stories, videos, media, and lengthy text 1.3

Modality Include audio narration where possible to explain graphic presentation 1.0

Redundancy Do not present words as both on-screen text and narration when graphics are present

.7

Personalization Script audio in a conversational style using first and second person 1.3

Segmenting Break content down into small topic chunks that can be accessed at the learner’s preferred rate

1.0

Pre-training Teach important concepts and facts prior to procedures or processes 1.3

Etc. Worked examples, self-explanation questions, varied-context examples and comparisons, etc.

??

Source: E-learning and the Science of Instruction, Clark and Mayer, 3rd ed., 2011

Page 18: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

18

The impact is not small!

50% 1 sd

84%!

Page 19: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

19

We  know  there  is  much  poten)al  to  improve*  

*Example: Ratings from Kaplan Way for Learning checklist, applied to 9 Kaplan products

Page 20: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

20

Have to be careful – what we think is “good” may not be

•  Comparison of course teacher view vs. independent teachers’ markings

0"

0.5"

1"

1.5"

2"

2.5"

3"

3.5"

4"

4.5"

course"teacher" Teacher"1" Teacher"2"

Based on 10 randomly selected papers from a writing course

Page 21: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

21

The evidence also shows our intuitions aren’t the best guides LSAT Logical Reasoning example

0"

1"

2"

3"

4"

5"

6"

7"

Study"8"worked"examples"

Study"15"worked"examples"

Use"Exis>ng"Kaplan"On"Demand"Instruc>on"

Test"Only"–"No"instruc>on"

N 153* 148* 107 84 Time (mins) 8.15 12.8 99.32 NA

* Significant difference from “No Instruction”

Page 22: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

22

All this changes how courses should be developed

Read, Write, Discuss •  Outcomes and content not precisely aligned •  Limited demonstrations, worked examples,

and practice •  General assessment rubrics •  High reliance on discussion boards

Existing courses

Prepare, Practice, Perform •  Outcomes and content aligned •  One lesson per objective •  Demonstrations and worked examples •  Practice, feedback before assessment •  Detailed scoring guides •  Less discussion/more practice •  Standard instructor materials •  Monitoring and support for motivation

Redesigned courses

Page 23: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

23

Result: much greater student success

•  11% higher success rate

•  28% increase

•  Students in redesigned courses were 1.6 times more likely to be successful

Wald Chi-Square: 10.42, df=1, n=895, Sig<.001.

39%

50%

20%

30%

40%

50%

60%

70%

Control Pilot

Adj

uste

d st

uden

t suc

cess

rate

Adjusted student success rates with 95% confidence limits

Page 24: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

24

Faculty supports do help students – but need to check

60%

65%

70%

75%

80%

85%

90%

Attendance Number of assignments*

LMS minutes*

Ret

entio

n

Control Treatment

* Improved learning outcomes

§ §

N: 653 498 625

Impact of faculty dashboards on first year college social studies course

Page 25: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

25

At  scale,  we  can  pursue  mul)ple  lines  of  inves)ga)on  

Adaptive

SmartBook

MyLab

Assessment & CLA

CLA Re-Scoring

Assessment

Learning Strategies

Self-Explanation

Self Summary

Self Test

Media Principles

Advanced Organizers

Coherence

Continuity

Modality

Multimedia

Redundancy

Motivation & Self

Efficacy

Judgments of Learning

Motivational Priming

Self Efficacy

Social Norming

Badging

Iconographs

Worked Examples

Early

Later

Others

Section Size

Orientation

Faculty Dashboard

Open Courses

Challenge Exams

Page 26: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

26

Running dozens of trials requires careful tracking

1 Study In Prep

Adaptive Learning (Mcgraw Hill Connect AB219)

3 Studies In Development

Attribution I (SOFI) Standing Out Fitting In (AB/MT140) Indigo Live Seminar Delivery (AB/MT203)

19 Studies In Term

Adaptive Learning (McGraw-Hill Connect CM107) Adaptive Learning (SOOMO PS124) Attribution II (KU160) CLA Anchor Paper (PA106) Competency (CS113) Campus LearnLab (CM220 Blended)

Mind Set Challenge (CM220) PERTS Mind Set Challenge (MM150) Science Center Evaluation Study (SC156) AS Section Size (PS124) Self Regulated Learning (CE100) Indigo Live Seminar Delivery (GB512) StereoType Threat Surveys (CJ100, CJ101, MM150, MM207, MM212, MM255)

8 Studies In Assessment (5 Analysis in Progress; 2 Data Requested; 1 Awaiting Data Availability)

Attribution I (HS100 Pre-Intervention) Harvard Study Supporter (CM107 and MM150) CLA Blind Scoring (CJ101 and GB512) Faculty Dashboard (SS310) Social Norming (HU300, MM150, IT301) Worked Examples (CJ130)

34 Studies Completed Advanced Organizers (HN144) CLA Reliability CLA Assignment Order (CJ100) CLA Training (CS204) Orientation Facilitation Bus Section Size Seminar Delivery (LS) Stereotype Threat Historical Review Library Use Historical Review Judgments of Learning (HS101) V4 Learning Platform (IT133) Math Center Feedback (MM207) Motivational Priming (HS100 and PS124) Self Test (HS200) Adaptive Learning (SmartBook HS120) VoiceThread (PA205 and PA201) Worked Examples (14 instances)

Page 27: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

27

It’s real work to alter how a large number of IDs build. . .

Online Course

Wor

ksho

p

Analysis Micro Design Development

-8 0 1 3 2 4 5 6 7 8 9 10 11 12

Deliverables

Coaching

Production

Week 13 14 15 16

Delivery

2hr 2hr 2hr 2hr

POs Specs Draft Scripts

C C C

Bug List

Moodle

Revise

Rebuild

Team review / preview ...

2hr

Cohort review ...

Final Scripts

Author

2-month online self-study (40 hours)

2-day workshop Team Projects

4 months coaching

Transfer to products

Page 28: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

28

But it matters if you’re after good “learning engineering”

Page 29: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

29

Where  to  find  out  more?  

•  LocaAon  of  course  on  using  (and  downloading)  the  checklist:  

                                                           hHp://goo.gl/f1RCAu  

•  Bror’s  Blog  for  more  on  “learning  engineering”:  

                                       hHp://www.kaplan.com/brorsblog    

•  Contact  me:                                                            [email protected]    

Page 30: “Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:) The)Artof)Using)Learning)Science)at Scale)to)Li7)Performance) October)2015)

30

April 20, 2015 Why We Need Learning Engineers Chronicle of Higher Education


Li7 - archives.nseindia.com

Li7 - archives.nseindia.com

Documents
The artof - New Zealand...employee – and create a desirable workplace culture – organisations need to understand the root causes of employee fulfilment. At Tourism New Zealand,

The artof - New Zealand...employee – and create a desirable workplace culture – organisations need to understand the root causes of employee fulfilment. At Tourism New Zealand,

Documents
Hannah Rudman Artof Digital Londonintro

Hannah Rudman Artof Digital Londonintro

Art & Photos
Celebrating The Artof Life

Celebrating The Artof Life

Art & Photos
EXPERIMENT WITH THE ARTOF CAMPARI · 3cl campari tales • persian orange saffron • ginger ale • rose champagne garnish • edible flowers the diary of davide campari build in

EXPERIMENT WITH THE ARTOF CAMPARI · 3cl campari tales • persian orange saffron • ginger ale • rose champagne garnish • edible flowers the diary of davide campari build in

Documents
THE UNIVERSE and Three Examples Alan Guth, MITweb.stanford.edu/~oas/SI/SRGR/lib/GuthsNotesnontech9.pdf · • The abundances of the light chemical elements— H, H2, He3, He4, Li7

THE UNIVERSE and Three Examples Alan Guth, MITweb.stanford.edu/~oas/SI/SRGR/lib/GuthsNotesnontech9.pdf · • The abundances of the light chemical elements— H, H2, He3, He4, Li7

Documents
Artof stylingsentencespowerpoint

Artof stylingsentencespowerpoint

Documents
Spark artof protest

Spark artof protest

Documents
2009 The Artof Hanukah4 Website

2009 The Artof Hanukah4 Website

Education
Artof pissarro4hc052810 2

Artof pissarro4hc052810 2

Education
Artof Speaking Aug2009 Newsletter

Artof Speaking Aug2009 Newsletter

Documents
Zen$and$the$Artof$Network$ Architecture$llp/zana.pdf · Zen$and$the$Artof$Motorcycle$Maintenance$ by$ Robert Pirsig$ $ • Rejected$by$121$publishers$(World$Record)$ • Classic$v$RomanFc$PerspecFves$

Zen$and$the$Artof$Network$ Architecture$llp/zana.pdf · Zen$and$the$Artof$Motorcycle$Maintenance$ by$ Robert Pirsig$ $ • Rejected$by$121$publishers$(World$Record)$ • Classic$v$RomanFc$PerspecFves$

Documents
Artof Pissarro4 Jae Wbst123009

Artof Pissarro4 Jae Wbst123009

Education
Artof Israel042011

Artof Israel042011

Education
Integration of an interpretable machine learning algorithm to … · 2020. 7. 10. · Tao Huang6, Kelei Li7, Xiaofei Guo7, Huijuan Liu5*, Duo Li7* and Ju-Sheng Zheng1,7* Abstract

Integration of an interpretable machine learning algorithm to … · 2020. 7. 10. · Tao Huang6, Kelei Li7, Xiaofei Guo7, Huijuan Liu5*, Duo Li7* and Ju-Sheng Zheng1,7* Abstract

Documents
The Artof Purim2010

The Artof Purim2010

Documents
ARTOF-2 Electron Spectrometer For Angle-Resolved ...ARTOF-2 Electron Spectrometer The Scienta Omicron ARTOF 10k analyser marked a revolution in the field of angle-resolved photo-electron

ARTOF-2 Electron Spectrometer For Angle-Resolved ...ARTOF-2 Electron Spectrometer The Scienta Omicron ARTOF 10k analyser marked a revolution in the field of angle-resolved photo-electron

Documents
Understanding the alkali – Germanium interaction in the ternary … · 2018. 6. 28. · Ge1 – Li6 2.539 2.514 Ge7 – Li2 3.007 2.980 Ge1 – Li7 2.742 2.709 Ge7 – Li4 2.882

Understanding the alkali – Germanium interaction in the ternary … · 2018. 6. 28. · Ge1 – Li6 2.539 2.514 Ge7 – Li2 3.007 2.980 Ge1 – Li7 2.742 2.709 Ge7 – Li4 2.882

Documents
The artof photography

The artof photography

Documents
Meta-analysis on the effectiveness of team-based learning ... · Chuncheng Lu1, Shouyu Wang1, Shoulin Wang1, Liping Zhao6, Zhong Li7, Huijuan Zhu4, Jianming Wang8, Yankai Xia5 and

Meta-analysis on the effectiveness of team-based learning ... · Chuncheng Lu1, Shouyu Wang1, Shoulin Wang1, Liping Zhao6, Zhong Li7, Huijuan Zhu4, Jianming Wang8, Yankai Xia5 and

Documents
Zen$and$the$Artof$Network$ Architecture$llp/zana.pdf · Zen$and$the$Artof$Network$ Architecture$ Larry$Peterson$ Zen$and$the$Artof$Motorcycle$Maintenance$ by$ ... – Science$vs$Art

Zen$and$the$Artof$Network$ Architecture$llp/zana.pdf · Zen$and$the$Artof$Network$ Architecture$ Larry$Peterson$ Zen$and$the$Artof$Motorcycle$Maintenance$ by$ ... – Science$vs$Art

Documents
Artof pissarro4hc052810

Artof pissarro4hc052810

Education
Unusual Mn coordination and redox chemistry in the high ...people.bath.ac.uk/msi20/publications/pdfs/2015/Roos-PCCP2015-Li7... · his ournal is ' the Oner ocieties 2015 Phys. Chem

Unusual Mn coordination and redox chemistry in the high ...people.bath.ac.uk/msi20/publications/pdfs/2015/Roos-PCCP2015-Li7... · his ournal is ' the Oner ocieties 2015 Phys. Chem

Documents
Artof retirementclientguide

Artof retirementclientguide

Economy & Finance
The artof - New Zealand Business Events and Conferencesbusinessevents.newzealand.com/getattachment/incentiv… ·  · 2017-10-12booklet, you’ll find a curated selection of the

The artof - New Zealand Business Events and Conferencesbusinessevents.newzealand.com/getattachment/incentiv… ·  · 2017-10-12booklet, you’ll find a curated selection of the

Documents
Artof Giving

Artof Giving

Lifestyle
The$Artof$ Zentangle$ - Corona-Norco Unified …Artof$ Zentangle$ The$Zentangle$Method$is$ acreave$way$to$create$ images$by$drawing$ structured$paerns.$It was$developed$by$Maria Thomas$and$Rick$Roberts$

The$Artof$ Zentangle$ - Corona-Norco Unified …Artof$ Zentangle$ The$Zentangle$Method$is$ acreave$way$to$create$ images$by$drawing$ structured$paerns.$It was$developed$by$Maria Thomas$and$Rick$Roberts$

Documents
Artof Pissarro3 Hc071408

Artof Pissarro3 Hc071408

Education
Artof Engagement Ppt

Artof Engagement Ppt

Documents
The light elements Li Be B (Li6, Li7, Be9, B10, B11)...The light elements Li Be B (Li6, Li7, Be9, B10, B11)The most fragile stable isotopes in nature (after D and He3) Always destroyed

The light elements Li Be B (Li6, Li7, Be9, B10, B11)...The light elements Li Be B (Li6, Li7, Be9, B10, B11)The most fragile stable isotopes in nature (after D and He3) Always destroyed

Documents