1

2

3

James Gee Randi Engle

Jim Pellegrino Susan Goldman John Bransford

Paul Pintrich

4

Situative Theorists at the IRL

5

James Greeno Etienne Wenger John Seely Brown

Allan Collins Jean Lave Roy Pea

IRL’s Seven Principles of Learning

1. Learning is fundamentally social. 2. Knowledge is integrated in the life of

communities. 3. Learning is an act of membership. 4. Knowing depends on engagement in practice. 5. Engagement is inseparable from empowerment. 6. “Failure to learn” is the normal result of

exclusion from participation. 7. We already have a society of lifelong learners.

6

9

11

Joyce Moore

Greeno, J. G., Moore, J. L., & Smith, D. R. (1993). Transfer of situated learning. In D. K. Detterman, & R. Sternberg, (1993). Transfer on trial: Intelligence, cognition, and instruction (pp. 99-167) . Norwood, NJ: AblexPublishing.

PLATO

LOGO

2003-2005

2005-2009

Project NML

2008-2011

Quest Atlantis

2005-2009

GenScope1995-2005

Exploring the “Situative Synthesis”• Motivation

– Hickey, D. T. (2003) Engaged participation vs. marginal non-participation: A stridently sociocultural model of achievement motivation. Elementary School Journal, 103 (4), 401-429

– Filsecker, M., Hickey, D. T. (2014). A multilevel analysis of the effects of external rewards on elementary students’ motivation, engagement, & learning. Computers & Education, 75, 136-148

• Program Evaluation– Hickey, D. T, & Zuiker, S. (2003). A new perspective for evaluating innovative science

learning environments. Science Education, 87, (3) 539-563 • Assessment

– Hickey, D. T., & Zuiker, S. J. (2012). Multi-level assessment for discourse, understanding, and achievement in innovative learning contexts. The Journal of the Learning Sciences, 22, (4) 1-65

– Hickey, D. T. (2015). A situative response to the conundrum of formative assessment. Assessment in Education: Principles, Policies, and Practices, 22 (2), 202-223.

• Networked Online Learning– Hickey, D. T., & Rehak, A. (2013). Wikifolios and participatory assessment for

engagement, understanding, and achievement in online courses. Journal of Educational Media and Hypermedia, 22 (4), 229-263.

• Digital Badges and eCredentials– Casilli, C, & Hickey, D. T. (2016). Transcending conventional credentialing and assessment

paradigms with information-rich digital badges. The Information Society, 32 (2), 117-129.17

Online Learning Research Contexts

2009-2012 in Sakai @IU 2012-present in CourseBuilder BOOC

2013-present in Canvas @ IU IUHS 2013-present in Canvas @ IU GSE

Use public contexts give meaning to knowledge tools

Reward productive disciplinary engagement

Grade artifacts through local reflections

Let individuals assess their understanding privately

Measure aggregated achievement discreetly 19

The BOOC TeamThomas Smith Garrett PoortingaTara Kelly

Xinyi ShenRetno Hendryanti21Karthik Bangera

1. Use “public” contexts to give meaning to knowledge tools

• Problematize knowledge from learners’ perspective– Publically open up the discipline

• Learners define (and redefine) a personally-meaningful context– Can be disciplinary problem, project,

investigation, etc. – Embodies their experience, interests,

& aspirations

RogersHall

22

23

24

25

26

28

New Feature #1Personalized Instructor Videos

31

New Feature #2“Drag & Rank”

Interaction

32

New Feature #3Streamlined Participation

Analytics

33

Public Engagement: Average Words/Wikifolo

2013 2014

34

0

500

1000

1500

2000

2500

3000

3500

1 2 3 4 5 6 7 8 9 10 11

Unit

For Credit(12)

OpenCompleters(10)

Open Drop(54)

0

500

1000

1500

2000

2500

3000

3500

1 2 3 4 5 6 7 8 9 10 11

For Credit

OpenCompleters

OpenDroppers

2. Encourage & reward productivelocal disciplinary engagement

• DE involves declarative knowledge and cultural practices

• PDE connects knowledge to practices, makes connections, finds resources, etc.

• Support PDE so it can be rewarded– Comment directly & locally on wikifolios– Work in the open and encourage “lurking” – No grades or mandatory posts

35

Randi Engle

Problematize Content- From learner’s perspective- Open up issues that experts see as closed

Give Students Authority- Avoid “known answer” questions- Position students as stakeholders

Establish Disciplinary Accountability- Defend positions - Be responsive and respectful

Provide Relevant Resources- Include time and information- Provide easy contextualized access

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

Average Comments per Wikifolio

2013 2014

53

0

1

2

3

4

5

6

7

8

9

1 2 3 4 5 6 7 8 9 10 11

Com

men

ts

Unit

For Credit

OpenCompleters

0

1

2

3

4

5

6

7

8

9

1 2 3 4 5 6 7 8 9 10 11

Com

men

ts p

er W

ikifo

lio

Unit

For Credit(12)

OpenCompleters(10)

• Disciplinarity0 = non-disciplinary1 = re: education2 = re: assessment3 = re: chapter

• Contextualization0 = not referenced1 = context referenced

54

0

0.5

1

1.5

2

2.5

3

Enrolled(4, 244)

OpenCompleters

(4, 127)

Open Droppers(4, 71)

Disciplinarity and Contextualization of Representative Sample of Comments

Disciplinarity Contextualization

Disciplinary Engagement in 2013

Disciplinary Engagement in 2014

55

00.5

11.5

22.5

3

Disciplinarity Contextuality

• Disciplinarity0 = non-disciplinary1 = re: education2 = re: assessment3 = re: chapter

• Contextualization0 = not referenced1 = context referenced

Grade artifacts through local reflections

• Evaluate reflections on prior engagement.– Contextual engagement (suitability of context

for learning concepts)– Collaborative engagement (what you learned

from others)– Consequential engagement (consequences of

new knowledge)

• Public context but local interactions56

59

60

2015 Results: Individual Local Engagement with Instructor

61

0

100

200

300

400

500

600

700

1 2 3 4 5 6&7 8 9 10 11 12 13 14 15

Averge Words per Reflection

4. Let individuals assessunderstanding privately

• Offer ungraded open-ended items and performance tasks

• Cover the big ideas in the assignment

• Use to re-engage rather than remediate.– Items themselves have little formative

value beyond the correct answer62

PamelaMoss

Use this self-assessment to check your understanding of the big ideas in this assignment. You should complete these items from memory. If you are unable to answer more than one item you are not prepared for the unit exam. If you did poorly, you should go back and re-engage with your classmates, re-read the text, and discuss these ideas more with your peers. Start with the highly promoted work on the weekly public feedback.

5. Measure achievement discreetly

• Standards-oriented tests have narrow uses– Measure gains using pretests, compare instruction,

document improvement– Primary student function is motivating prior

engagement and catching slackers

• Refine tests (but don’t teach to them)– Use the item analysis routines in your LMS– Aim for about 85%, normal distribution, 1-2 @100%

• Protect test security– Withhold item-level feedback

• Randomly selected timed items (~1 min)– Include difficult “best answer” items– Don’t put searchable terms in item stem

Me!

Disciplinary Achievement 2014 & 2015

65

0

10

20

30

40

50

60

70

80

90

100

Practices Principles Policies Final

For Credit Open Completers

0

10

20

30

40

50

60

70

80

90

100

AssessmentPractices

AssessmentPrinciples

AssessmentPolicies

ComprehensiveFinal

All Students For Credit (8)

Self-Paced Course for 2016New Tools for Finding Peers

66

Self-Paced Course for

2016

New Tools for Archiving Wikifolios

67

69

Connie Yowell

Mimi Itow

DIGITAL BADGES CONTAIN INFOClaims & evidence; links to more

● Contain specific claims of competence

● Contain evidence supporting claims

● Links to more evidence including student work

● This is different from conventional credentials

● Can link to unlimited additional information

● It matters what claims and evidence are included

● Process is transformative and/or disruptive

OPEN BADGES CIRCULATEInfo circulates in value networks● Credibility of claims and

validity of evidence gets “crowdsourced”

● Earners can find opportunities and promote achievements

● Institutions can recognize broader accomplishments

● Employers can communicate needs and find skills

● Amplifies meaning of claims and evidence

● Amplifies deliberation over nature of claims and evidence

• Studied 2012 DML content awardees

• 600 badge content proposals

• 29 content developers supported

• 3 platforms supported

RebeccaItow

CathyTran

KaterinaSchenke

JamesWillis

Nate Otto

ChristineChow

JoshuaQuick

Cindy Cogswell

Accredited Context

Accredited Badges

None

Other

School or University

After School

Teacher PD

Informal & Other

Adult & Career

MuseumExtra-

Curricular

Common Core

Local Standards

Partners for 21CS

None Identified

K-12

Middle School

Secondary

College Students

Educators/ Other Vocational

/ Adult

AccreditationSetting

StandardsEarners

29 Diverse Approaches to Supporting, Assessing, & Recognizing Learning

• Seven “associationist” efforts– Self-paced mastery of specific competencies (CBE &

gamification), individual recognition• Twelve “constructivist” efforts

– Cohorted & self-paced, problem-based, performance & portfolio assessment, individual recognition.

• Five “situative” efforts– Cohorted, networked learning, peer & group

recognition• Five “hybrid” efforts

– All three types of practices

76

Capturing “Practical Wisdom”

• Aristotle’s Phronesis– Contextual and value-

laden

• Bent Flyvbjerg (2001)– Phronetic narratives

• Rich Halverson (2004)– Artifact-based

phronetic narratives

Categories of Badge Functions

Recognizing Learning

Assessing Learning

Motivating Learning

Studying Learning

Emergence of Badge Design Principles

Draft Initial Principles

Formalize General Principles

Bookmark Research

DPD PROJECTFINDINGS ACROSS BADGE EFFORTS

• Documented apparent badge and ecosystem status– Badges system in 2014 (none, different, or

intended)– Ecosystem in 2014 (none, different, or

intended)– Ecosystem in 2015 (none/paused,

existing, or thriving)

Badges appear to work better… In some places than others

Badges appear to work better… with sociocultural and hybrid learning.

How did badges work…where learning was competency-based?• Seven projects were CBE

– Self-paced, rejected “seat time”

– Mostly for course credit– Summative assessment (of

specific competencies)– Extrinsic motivation (i.e.,

credit, points, etc.• Competency badges

– Measurable and measured competencies

How did badges work…where learning was competency-based?

• SA&FS was suspended• All three Gates Project Mastery

awardees were all suspended– Pathways to Global Competency (Asia Society)– Level Up (Colorado) – Youth Film Maker (Philadelphia)

• Manufacturing Institute– SkillsUSA badge suspended– PLTW badge is seldom earned

• YALSA badges are seldom earned• Buzzmath badges are not open but

thriving

• Equating evidence with conventional criteria

• Determining who can authorize credit

• Maintaining a common definition of proficiency

• Building a sustainable model• Equitable education• Barriers to efficiency

– Technical– Financial barriers– Logistical barriers

Challenges that Rand Corp Evaluation Uncovered for Gates Project Mastery Projects

How did open badges work…where learning was participatory?

• Five project emphasized engaged participation– Cohorted course-based contexts– Crowdsourced recognition (e.g,

communal badges)– Transformative assessments– Social motivation (e.g., team

goals)

• Participation Badges – For assuming roles and identities

How did open badges work…where learning was participatory?

• MOUSE network mentors high school tech support– Thriving, many successful grads

• S2R network for high school media producers– Thriving, many successful grads

• PBS Student Reporting Labs for high school media– Thriving, many successful grads

• Cooper-Hewitt/Smithsonian Design Prep helps high schoolers develop design portfolios– Stated out more competency-based badges but thriving with

role-based participatory badges

• Design for America for undergraduate social design and engineering projects– Implemented Digital Loft badges– Program thriving but badges appear paused

How did open badge work…where learning was constructivist?

• Twelve projects emphasized more constructivist practices– Inquiry-oriented and project-based

contexts– Informal recognition– Formative assessment (e.g., open-

ended, rubrics, etc.)– Intrinsic motivation

• Completion badges– Assessable and assessed

• Assessment was a big challenge– More modest assessments and scaled

back assessment in thriving projects

How did open badges work…where practices were hybrid?

• Five projects were hybrids • Generally were broader

learning contexts– Associationist, constructivist,

and sociocultural activities– Mix of recognition and

assessment practices– Various forms of motivation

• Competency, Completion, and/or Participation Badges

• Assessment again a challenge– Thriving projects scaled back or

automated assessment

Six Arguments About Motivating Networked Learning with Open Badges

1. The information in badges is inherently meaningful2. Open information is even more meaningful3. Open credentials are particularly disruptive and

transformative 4. The intrinsic/extrinsic dichotomy is overstated5. The negative consequences of extrinsic rewards is

overstated6. Situative models of engagement are ideal for motivating

learning in digital networks.

90

Hickey, D. T., & Schenke, K. (in preparation). Thinking about motivation when learning is accomplished, assessed, and recognized in digital networks. In K. A. Renninger & S. E Hidi, The Cambridge Handbook of Motivation and Learning. Cambridge, MA: Cambridge University Press

91

92

93

Current Status of ePortfolios & Badges

Integration

Integration w/employer search

LRN in development

Integration

Open Badge Factory default

Tutors issue Credley Badges

Badges in development

“Interested” 94

+

+

+

95

Exploring the “Situative Synthesis”• Motivation

– Hickey, D. T. (2003) Engaged participation vs. marginal non-participation: A stridently sociocultural model of achievement motivation. Elementary School Journal, 103 (4), 401-429

– Filsecker, M., Hickey, D. T. (2014). A multilevel analysis of the effects of external rewards on elementary students’ motivation, engagement, & learning. Computers & Education, 75, 136-148

• Program Evaluation– Hickey, D. T, & Zuiker, S. (2003). A new perspective for evaluating innovative science

learning environments. Science Education, 87, (3) 539-563 • Assessment

– Hickey, D. T., & Zuiker, S. J. (2012). Multi-level assessment for discourse, understanding, and achievement in innovative learning contexts. The Journal of the Learning Sciences, 22, (4) 1-65

– Hickey, D. T. (2015). A situative response to the conundrum of formative assessment. Assessment in Education: Principles, Policies, and Practices, 22 (2), 202-223.

• Networked Online Learning– Hickey, D. T., & Rehak, A. (2013). Wikifolios and participatory assessment for

engagement, understanding, and achievement in online courses. Journal of Educational Media and Hypermedia, 22 (4), 229-263.

• Digital Badges and eCredentials– Casilli, C, & Hickey, D. T. (2016). Transcending conventional credentialing and assessment

paradigms with information-rich digital badges. The Information Society, 32 (2), 117-129.96