Formation and Learning Analytics?

Formation and Learning Analytics? Simon Buckingham Shum Knowledge Media Institute, The Open University Visiting Fellow, Grad. School of Education, University of Bristol From August: Connected Intelligence Centre, University of Technology Sydney http://simon.buckinghamshum.net http://twitter.com/sbskmi

Formation by Design (FxD) Symposium Georgetown University, Washington, June 30 – July 2, 2014 https://futures.georgetown.edu/formation

1

Emergence Learning

It’s out of the labs and into products: every learning tool now has an “analytics dashboard” (a Google image search)

2

http

s://

twit

ter.

com

/Wis

wij

zer2

/sta

tus/

4140

5547

2451

5758

08

3

http

s://

twit

ter.

com

/Wis

wij

zer2

/sta

tus/

4140

5547

2451

5758

08

“Note: check the huge difference between

knowing and measuring…”

4

5 c.f. “Data is the new oil”

Scaleable Precise

Quan?fiable Reprocessable

For Morozov, analytics is where technological solutionism hits education:

6

“This flight from thinking and the urge to replace human judgments with timeless truths produced by algorithms is the underlying driving force of solutionism.”

If something like this is the pedagogical architecture for (trans)formational learning…

7

Structuring Knowledge Performing

Generating Learning Power

Purposing

If something like this is the pedagogical architecture for (trans)formational learning…

8

Structuring Knowledge Performing

Generating Learning Power

Purposing

…what roles can computational learning analytics play?

The worldview clash is what makes this

fascina?ng!

For discussion on how analytics shape educational worldviews

9 http://people.kmi.open.ac.uk/sbs/2014/06/edmedia2014-keynote

thought experiment: could we imagine…

liminal space analy7cs?

10

liminal space

= opportunity for transformation

or withdrawal 11

Liminal space = opportunity for transformation, or withdrawal

12

So while we initially strive to make our students

feel comfortable on campus, we then must help them balance their desire for security with the need to take risks and explore new ideas and possibilities. Rather than attempting to resolve the tension,

a college should help students find their place on this precarious threshold, in the liminal space between the familiar and strange, the old and the new.

Johansson & Felten, 2014

Liminal space = opportunity for transformation, or withdrawal

13

“It is when you have left the tried and true, but have not yet been able to replace it with anything else. [...] It is when you are

between your old comfort zone and any possible new answer. If you are not trained in how to hold anxiety, how to live with ambiguity, how to entrust and wait, you will run... anything to flee this terrible cloud of unknowing.”

Richard Rohr O.F.M. on the spirituality of liminal space

meaningful behavioural proxies?

some clues…

14

interpersonal engagement and social capital

analy7cs 15

Social Network Analysis (SNAPP)

16

What’s going on in these discussion forums?

Social Network Analysis to gain insight into peer-peer and peer-mentor dynamics (SNAPP tool)

http://www.slideshare.net/aneeshabakharia/snapp-20minute-presentation Bakharia A and Dawson S. (2011) SNAPP: a bird's-eye view of temporal participant interaction. Proceedings of the 1st International Conference on Learning Analytics and Knowledge. Banff, Alberta, Canada: ACM, 168-173.

Social Network Analysis (SNAPP)

18 http://www.slideshare.net/aneeshabakharia/snapp-20minute-presentation

2 learners connect otherwise separate clusters

tutor only engaging with active students, ignoring disengaged ones on the edge

Visualizing and filtering online social ties: by topic and type of tie

Schreurs B., Teplovs C., Ferguson R., De Laat M. and Buckingham Shum S. (2013). Visualizing Social Learning Ties by Type and Topic: Rationale and Concept Demonstrator. Proc. 3rd Int. Conf. on Learning Analytics & Knowledge. Leuven, BE: ACM, 33-37. Open Access Eprint: http://oro.open.ac.uk/36891





disposi7onal learning analy7cs

24

Dispositional Learning Analytics Workshop

25

http://learningemergence.net/events/lasi-dla-wkshp http://learningemergence.net/2014/03/01/ assessing-learning-dispositions-academic-mindsets

Observation informal and formal

Self-Diagnostic informal and formal

Behavioural Analytics

Assessing Learning Dispositions/Mindsets

Future sweetspot... multiple lenses to provoke self-reflection

http://people.kmi.open.ac.uk/sbs/2014/02/assessing-learning-dispositions-academic-mindsets

Self-report through reflective blogging 9-10 yr old EnquiryBloggers • Bushfield School, Wolverton, UK

EnquiryBlogger Wordpress Multisite plugins http://learningemergence.net/tools/enquiryblogger

27

Masters level EnquiryBloggers Graduate School of Education, University of Bristol

EnquiryBlogger: blogging for Learning Power & Authentic Enquiry http://learningemergence.net/2012/06/20/enquiryblogger-for-learning-power-authentic-enquiry

28

29

EnquiryBlogger teacher’s dashboard – direct navigation to learners’ blogs from the visual

analytic

We are quantifying our lives

http://learningemergence.net/2014/03/01/assessing-learning-dispositions-academic-mindsets

Dispositional profile from behavioural traces, to complement self-report?

Shaofu Huang: Prototyping Learning Power Modelling in SocialLearn http://www.open.ac.uk/blogs/SocialLearnResearch/2012/06/20/social-learning-analytics-symposium Simon Knight: http://people.kmi.open.ac.uk/knight/2014/02/knowledge-in-search

Social network patterns, teamwork effectiveness

and initiation of relationships

Questioning, arguing and search behaviours

reveal intrinsic curiosity and epistemic commitments

Tagging/sharing/blogging/social patterns

reveal how you see connections between

ideas

Behavioural and somatic traces associated with

perseverance, grit, tenacity; overcoming

panic/stress when stretched

narra7ve/ discourse

analy7cs 32

The Storymaking Project: PhD by Joanna Kwiat hLp://projects.kmi.open.ac.uk/storymaking

1st International Workshop on Discourse-Centric Learning Analytics

analytics that look beneath the surface, and quantify linguistic proxies for ‘deeper learning’

Beyond number / size / frequency of posts; ‘hottest thread’

http://ww

w.glennsasscer.com

/wordpress/w

p-content/uploads/2011/10/iceberg.jpg

http://solaresearch.org/events/lak/lak13/dcla13

Why are stories so important to formation?

34

“Fundamentally different to logical argument, the story positively demands those things that the other either denies or restricts: emotion, agency, character, perspective, and so on. To understand a story on some level is to become engaged in its telling and this will normally be achieved by allowing oneself to empathise or identify with the story’s characters or its narrator.”

Discourse analytics on webinar textchat

Ferguson, R. and Buckingham Shum, S., Learning analytics to identify exploratory dialogue within synchronous text chat. In: 1st International Conference on Learning Analytics and Knowledge (Banff, Canada, 2011). ACM, pp. 99-103. Open Access Eprint http://oro.open.ac.uk/28955

Can we spot the quality learning conversations in a 2.5 hr webinar?

-60

-40

-20

0

20

40

60

80

9:2

8 9

:32

9:3

6

9:4

0

9:4

1

9:4

6

9:5

0

9:5

3

9:5

6

10:

00

10:

05

10:

07

10:

07

10:

09

10:

13

10:

17

10:

23

10:

27

10:

31

10:

35

10:

40

10:

45

10:

52

10:

55

11:

04

11:

08

11:

11

11:

17

11:

20

11:

24

11:

26

11:

28

11:

31

11:

32

11:

35

11:

36

11:

38

11:

39

11:

41

11:

44

11:

46

11:

48

11:

52

11:

54

12:

00

12:

03

12:

04

12:

05

Average Exploratory


Sheffield, UK not as sunny as yesterday - still warm

Greetings from Hong Kong

Morning from Wiltshire, sunny here!

See you!

bye for now!

bye, and thank you

Bye all for now

Given a 2.5 hour webinar, where in the live textchat were the most effective learning conversations? Not at the start and end of a webinar…

Ferguson, R., Wei, Z., He, Y. and Buckingham Shum, S., An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. In: Proc. 3rd International Conference on Learning Analytics & Knowledge (Leuven, BE, 8-12 April, 2013). ACM. http://oro.open.ac.uk/36664

-60

-40

-20

0

20

40

60

80

9:2

8 9

:32

9:3

6

9:4

0

9:4

1

9:4

6

9:5

0

9:5

3

9:5

6

10:

00

10:

05

10:

07

10:

07

10:

09

10:

13

10:

17

10:

23

10:

27

10:

31

10:

35

10:

40

10:

45

10:

52

10:

55

11:

04

11:

08

11:

11

11:

17

11:

20

11:

24

11:

26

11:

28

11:

31

11:

32

11:

35

11:

36

11:

38

11:

39

11:

41

11:

44

11:

46

11:

48

11:

52

11:

54

12:

00

12:

03

12:

04

12:

05

Average Exploratory


…but if we zoom in on a peak…



-100

0

100

9:2

8 9

:40

9

:50

1

0:00

1

0:07

1

0:17

1

0:31

1

0:45

1

1:04

1

1:17

1

1:26

1

1:32

1

1:38

1

1:44

1

1:52

1

2:03

Averag

Classified as “exploratory

talk”

(more substantive for learning)

“non-exploratory”

…language is used in a manner more akin to “Exploratory Talk” (Neil Mercer)


Rhetorical discourse analytics

41

OPEN QUESTION: “… little is known …” “… role … has been elusive” “Current data is insufficient …”

CONTRASTING IDEAS: “… unorthodox view resolves …” “In contrast with previous hypotheses ...” “... inconsistent with past findings ...”

SURPRISE: “We have recently observed ... surprisingly” “We have identified ... unusual” “The recent discovery ... suggests intriguing roles”

http://technologies.kmi.open.ac.uk/cohere/2012/01/09/cohere-plus-automated-rhetorical-annotation De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052


42

Human analyst


43

Human analyst Computational analyst

http://technologies.kmi.open.ac.uk/cohere/2012/01/09/cohere-plus-automated-rhetorical-annotation De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052


44 Duygu Simsek’s PhD: http://people.kmi.open.ac.uk/simsek/research/

Glimpses of analytics capable of detecting higher order thinking.

But humans will always read

differently to machines

Can we correlate this with “academic writing”, and can such

analytics be used as formative feedback on drafts?


45 Simsek D, Buckingham Shum S, Sándor Á, De Liddo A and Ferguson R. (2013) XIP Dashboard: http://oro.open.ac.uk/37391

CONTRAST

SUMMARY & CONTRIBUTION

Evidence Hub: semantic storytelling for students, practitioners and researchers

Systems Learning & Leadership Evidence Hub: http://sysll.evidence-hub.net

A wizard guides the user through the submission of a structured story: •  What’s the Issue? •  What claim are you making/

addressing? •  What kind of evidence

supports/challenges this? •  Link it to papers/data •  Index it against the NIC’s core

themes

Evidence Hub: Argument Maps

Systems Learning & Leadership Evidence Hub: http://sysll.evidence-hub.net

The wizard then generates a structured Knowledge Tree showing evidence-based claims (and disagreements)

http://learningemergence.net/2013/07/17/deed-elli-ai-ci-systemic-school-learning

An alternative graphical network rendering can be generated to show the current Argument Map of supporting and challenging evidence around an Issue

Evidence Hub: Argument Maps

Evidence Hub: professional development http://learningemergence.net/2013/07/17/deed-elli-ai-ci-systemic-school-learning

Issue

Poten?al Solu?on

Suppor?ng Evidence

(prac??oner story)

epistemic cogni7on analy7cs

50

De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. (2011). Discourse-centric learning analytics. 1st Int. Conf. Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr). ACM: New York. Eprint: http://oro.open.ac.uk/25829

Argumentation analytics (in Debate Hub & Cohere apps) what epistemic contributions are learners making in the network?

51

Rebecca is playing the role of broker,

connecting different peers’ contributions in

meaningful ways We now have the basis for recommending that

you engage with people NOT like you…

Generates novel visualizations to summarise community arguments on a given issue

http://consider.it

Structured Deliberation/Argument Maps

Shifts in epistemic commitments? (Simon Knight, KMi PhD research)

What is it to ‘know’ when we search? http://sjgknight.com/finding-knowledge/2014/02/knowledge-in-search Danish exams permit Net: http://sjgknight.com/finding-knowledge/2013/07/danish-use-of-internet-in-exams-epistemology-pedagogy-assessment Epistemic networks for epistemic commitments: http://oro.open.ac.uk/39254

Learning sciences research into Epistemic Commitments/Cognition combined with

‘researching search’ methodologies.


Dimensions of Epistemic Belief

Certainty The degree to which knowledge is conceived as stable or changing, ranging from absolute to tentative and evolving knowledge

Simplicity The degree to which knowledge is conceived as compartmentalised or interrelated, ranging from knowledge as made up of discrete and simple facts to knowledge as complex and comprising interrelated concepts

Source The relationship between knower and known, ranging from the belief that knowledge resides outside the self and is transmitted, to the belief that it is constructed by the self

Justification What makes a sufficient knowledge claim, ranging from the belief in observation or authority as sources, to the belief in the use of rules of inquiry and evaluation of expertise

Knight, Simon; Buckingham Shum, Simon and Littleton, Karen (2014). Epistemology, assessment, pedagogy: where learning meets analytics in the middle space. Journal of Learning Analytics (In press). http://oro.open.ac.uk/39226


Behavioural Proxies of Epistemic Belief?

Certainty Frequency and density of positive and negative polarity links (e.g. supports; builds on, is consistent with, versus challenges; refutes; is inconsistent with).

Presence of stability markers – (e.g. current sources, bibliographic references, geographical spread).

Frequency and density of meta-discourse markers in the text of nodes, which moderate assertions in scholarly ways (e.g. in contrast to; remains an unresolved issue)

Simplicity Standard network analytics describing the topography of the structure

Semantic analysis of node and link types describing the variety of types used by a learner, the balance in link polarity (positive/neutral/negative), and the balance of propositional (causes; refutes; solves) versus analogical thinking (is a metaphor of; is analogous to).

Semantic analysis of tags on nodes and links

Encapsulation of node clusters or sub-networks within a broader category expressing a higher order construct

The use of different argumentation schemes, which reflect different sources of authority (argument from expertise; argument by analogy; argument from precedent)

In collaborative maps, the social network view showing who is connected to whom, but what kinds of links

Source Presence within the text of nodes of ‘I think’ or other restatements of fact

Challenges to claims through negative link types and meta-discourse markers in node text

Few additional nodes made other than those created as quotations.

Justification Judgements of relevance, and supporting or explanatory notes (‘this evidences/ explains x’). Ties to method ‘ideas’.

Use of well known Argumentation Schemes, especially evidence and arguments that grounded in the attribution of authority to people (‘experts’) or publications based on their status (‘valuing peer review’)

Rhetorical constructions in node text which show a critical use of sources

analytics for wider educational outcomes

Deakin Crick, R., Barr, S., Green, H. and Pedder, D. (2013). Evaluating the Wider Outcomes of Schools: Complex Systems Modelling. Centre for Systems Learning & Leadership, Graduate School of Education, University of Bristol, UK. http://learningemergence.net/library/reports/

Providing a more complete analytic on the health of the educational community

57

“Test grades are just ONE of a dozen indicators in our vision of a healthy educational community”

• life-long/life-wide learners • citizenship • self-confidence • teamwork • emotional

intelligence • “wellbeing”…

Hierarchical Process Modelling (Univ. Bristol Perimeta tool)

58

Seeing a learning community as a complex adaptive system requires the voices of learners, teachers, leaders, parents and ‘external’

stakeholders

The core mission and values of a network of school academies

60

Hierarchical Process Modelling transforms

qualitative and quantitative inputs into a multi-level

visual analytic

‘Italian Flag’ visual analytic

61

Degree of green / white / red reflects current (un)certainty over availability of evidence:

supporting / unknown / challenging

analytics as mirrors to provoke

(trans?)formative reflection

Your most recent mood comment: “Great, at last I have found all the resources that I have been looking for, thanks to Steve and Ellen.

In your last discussion with your mentor, you decided to work on your resilience by taking on more learning challenges

Your ELLI Spider shows that you have made a start on working on your resilience, and that you are also beginning to work on your creativity, which you identified as another area to work on.

1 2 3

45

Envisioning a social learning analytics dashboard

Ferguson R and Buckingham Shum S. (2012) Social Learning Analytics: Five Approaches. Proc. 2nd International Conference on Learning Analytics & Knowledge. Vancouver, 29 Apr-2 May: ACM: New York, 23-33. DOI: http://dx.doi.org/10.1145/2330601.2330616 Eprint: http://oro.open.ac.uk/32910

63

liminal space analy7cs? Use of computational techniques to reflect back to learners apparent shifts in • how they engage in a community of enquiry/practice • how they narrate their learning journeys • how they negotiate meaning • how their worldview seems to have shifted • how they respond cognitively (even somatically) when challenged.

65

towards a networked improvement community

(NIC) for FxD?

hLp://learningemergence.net/2014/05/29/learning-‐analy?cs-‐plus-‐nics-‐for-‐systemic-‐edu-‐improvement

Defining features of a NIC ‘B’ + ‘C’ Level improvement

‘A’ — what prac??oners do in a given ins?tu?on/

context

‘B’ — trying to improve local

prac?ce

‘C’ — the network of ‘B’ people

sharing insight and coordina?ng innova?on

Based on Doug Engelbart’s forma?ve work: hLp://www.dougengelbart.org/about/nics.html

Defining features of an Educational-NIC a sociotechnical infrastructure and modus operandi for coordinated aLack on

educa?onal challenges an inten?onal community facilitated by a coordina?ng hub using agreed representa?ons of the system and improvement strategies

in order to orchestrate ac?on in pursuit of measurable goals by sharing data, insights and arguments in order to build evidence-‐based claims

Bryk et al, Carnegie Founda?on for the Advancement of Teaching: hLp://www.carnegiefounda?on.org/sites/default/files/bryk-‐gomez_building-‐nics-‐educa?on.pdf

A NIC is in essence a “Collaboratory”

“a collaboratory is more than an elaborate collection of ICT; it is

a new networked organizational form that also

includes social processes; collaboration techniques; formal and informal communication; and agreement on norms, principles, values, and rules” (Cogburn, 2003, p. 86).

Cogburn, D. L. (2003). HCI in the so-called developing world: what’s in it for everyone, Interactions, 10(2), 80-87, New York: ACM Press. http://dx.doi.org/10.1145/637848.637866

Building blocks of an Educational-NIC

NIC Data Hub

Collective Sensemaking

Coordinating Maps

(of the system, problem, and

strategies)

Coordinating Maps


strategies)

Coordinating Maps


strategies)

Human + Machine Analytics

Educational Context Educational

Context


NIC Data Hub




Context Coordinating

Maps (of the system, problem, and

strategies)

Coordinating Maps


strategies)

Coordinating Maps


strategies)

Shared Conceptual Maps: of system / problem / strategies

http://www.carnegiefoundation.org/elibrary/getting-ideas-action-building-networked-improvement-communities-in-education

http://www.carnegiealphalabs.org/persistence/

Shared Conceptual Maps: for knowledge navigation

People / Organizations / Projects / Claims / Evidence Systems Learning & Leadership Evidence Hub: http://sysll.evidence-hub.net

Shared Conceptual Maps: for knowledge navigation

People / Organizations / Projects / Claims / Evidence Evidence Hub for Research by Children & Young People: http://rcyp.evidence-hub.net


NIC Data Hub



Context Coordinating

Maps (of the system, problem, and

strategies)

Coordinating Maps


strategies)

Coordinating Maps


strategies)


Impact Map

http://chaos.open.ac.uk/hypothesis-list/

Impact Maps were inspired by the Evidence Hub, and are organised around core hypotheses that the NIC is investigating.

Impact Map

http://oermap.org/hypothesis/578/hypothesis-a-performance/

Impact Map


Impact Maps use Sankey flow diagrams to show how findings from different contexts support or challenge each of the NIC’s core hypotheses

Impact Map
