Suthers & Rosen, Learning Analytics and Knowledge 2011

A Unified Framework for Multi-Level Analysis of Distributed Learning

Dan SuthersDepartment of Information and Computer Sciences

andCommunication and Information Sciences Program

Devan RosenDepartment of Speech

University of Hawaii

Funded by NSF VOSS

Multiple theories of how learning takes place in social settings ▪ From social as stimulus to social entity as learning agent▪ From "networked individualism" to "maintaining a joint

conception of a problem" ▪ From "diffusion of innovations" to "knowledge building"

All involve uptake: when an actor takes (a trace of) another actor's activity as being relevant in some way for his or her current activity ▪ See Suthers (ijCSCL 2006) for discussion of learning

epistemologies; and Suthers et al. (ijCSCL 2010) for uptake

Uptake is evidenced by how individual actions are observably contingent on the actions of others in their socio-technical network contexts

Learning in Social Settings

Fundamental question: how learning takes place through the interplay between individual and collective agency ▪ Situated accomplishments of individuals and small groups▪ How these local accomplishments give rise to larger

phenomena in networksRequires coordinated multi-level analysis

Activity can be distributed across multiple media and sites ▪ Traces of activity may be fragmented across multiple logs ▪ Logs may record activity in the wrong ontology for analysis

(e.g., media-level events rather than interaction or ties) Distributed activity may be analytically cloaked

✓ Abstract transcript representation that collects relevant events into a single analytic artifact

✓ Analytic hierarchy that supports multiple levels of analysis

Analytic Challenges

Contingency Graph (Contextual Action Model) Entity-Relations(Domain Model)

Associograms(Mediation Model)

Discussion 1

m1

m2

m3

m4Threading

Containment

P2 P1

P3

m1

m2

m3 m4

Sociogram (Tie Model)

P2 P1

P3

Uptake Graph (Interaction Model)

Events (Event Model)

w1

w2

w4

w3

r1

r2r1

r2r3

aa.bbb.cc.74 [27/Aug/2007:16:03:24] "POST /workspace/282/note/2080/messages/reply_preview ..."aa.bbb.cc.74 [27/Aug/2007:16:03:25] "GET /workspace/282/note/2080/discussions/tree/756/message/9558 ..."aa.bbb.cc.74 [27/Aug/2007:16:03:38] "POST /workspace/282/note/2080/messages/post_reply?...&message_id=9558 ..."... xx.y.zzz.245 [27/Aug/2007:22:09:47] "POST /workspace/282/note/2080/messages/open/756/message/9558 ..." ...xx.y.zzz.245 [27/Aug/2007:22:18:07] "POST /workspace/282/note/2080/messages/reply_preview/9558 ..."xx.y.zzz.245 [27/Aug/2007:22:18:07] "GET /workspace/282/note/2080/discussions/tree/756/message/9576 ..."xx.y.zzz.245 [27/Aug/2007:22:18:22] "POST /workspace/282/note/2080/messages/post_reply?...&message_id=9576 ..."

Logs

P2

P1

P3

w1

w2

w4

w3

Logs(Process

Trace)

aa.bbb.cc.74 [27/Aug/2007:16:03:24] "POST /workspace/282/note/2080/messages/reply_preview ..."aa.bbb.cc.74 [27/Aug/2007:16:03:25] "GET /workspace/282/note/2080/discussions/tree/756/message/9558 ..."aa.bbb.cc.74 [27/Aug/2007:16:03:38] "POST /workspace/282/note/2080/messages/post_reply?...&message_id=9558 ..."... xx.y.zzz.245 [27/Aug/2007:22:09:47] "POST /workspace/282/note/2080/messages/open/756/message/9558 ..." ...

xx.y.zzz.245 [27/Aug/2007:22:18:07] "POST /workspace/282/note/2080/messages/reply_preview/9558 ..."xx.y.zzz.245 [27/Aug/2007:22:18:07] "GET /workspace/282/note/2080/discussions/tree/756/message/9576 ..."xx.y.zzz.245 [27/Aug/2007:22:18:22] "POST /workspace/282/note/2080/messages/post_reply?...&message_id=9576 ..."

Entity-Relation Types(Domain Model)

Discussions

messages

Threading

Containment Actors

Write, Read

Logs(Process

Trace)



Entity-Relations(Domain Model)

Discussion 1

m1

Containment


P2

w1: P2 writes m1 @27/08/2007 16:03:38

Writes

Logs(Process

Trace)




Discussion 1

m1

Containment


P2

P1

r1: P1 reads w1 @27/08/2007 22:09:47

Reads

Logs(Process

Trace)



w1: P2 writes m1 @27/08/2007 16:03:38


Discussion 1

m1

m2

Containment


P2

P1

r1 w2: P1 writes m2 @27/08/2007 22:18:22

Writes

Logs(Process

Trace)



Threading

w1


Discussion 1

m1

m2

m3

m4Threading

Containment


w2

w4

w3

r1

r2r1

r2r3

P2

P1

P3

Logs(Process

Trace)



w1


Discussion 1

m1

m2

m3

m4Threading

Containment


w2

w4

w3

r1

r2r1

r2r3

P2

P1

P3

Event Model as Abstract Transcript▪ Log files no longer needed (except as we discover new

information needs)

▪ Straightforward extension to include events from other media

▪ Sequence of events serves as unified transcript of distributed events

w1


Discussion 1

m1

m2

m3

m4Threading

Containment


w1

w2

w4

w3

r1

r2r1

r2r3

P2

P1

P3

Contingency Graphs▪ But what about interaction?

??

?? ?

?

??

▪ Identify empirical relationships between events that collectively evidence uptake

▪ Called contingencies after Garfinkel's "contingently achieved accomplishments": how actors draw on the evolving context


Discussion 1

m1

m2

m3

m4Threading

Containment


w1

w2

w4

w3

r1

r2r1

r2r3

P2

P1

P3

Contingencies: Media Dependency ▪ To reply to a message, it must first be written


Discussion 1

m1

m2

m3

m4Threading

Containment


w1

w2

w4

w3

r1

r2r1

r2r3

P2

P1

P3

Contingencies: Media Dependency ▪ To read a message, it must first be written


Discussion 1

m1

m2

m3

m4Threading

Containment


w1

w2

w4

w3

r1

r2r1

r2r3

P2

P1

P3

Contingencies: Temporal Proximity & Same Actor▪ Continuity of agent's activity


Discussion 1

m1

m2

m3

m4Threading

Containment


w1

w2

w4

w3

r1

r2r1

r2r3

P2

P1

P3

Contingencies: Lexical or Semantic Overlap▪ For example, reuse of noun phrases


Discussion 1

m1

m2

m3

m4Threading

Containment

w1

w2

w4

w3

r1

r2r1

r2r3

P2

P1

P3

Contingency Graph as Contextualized Action Model▪ Analytically relevant manifest relationships between

actor's actions and other events have been recorded ▪ Next: raise analytic level of description to latent

relationships and higher order structures


Discussion 1

m1

m2

m3

m4Threading

Containment

w1

w2

w4

w3

r1

r2r1

r2r3

P2

P1

P3

Uptake Graph (Interaction Model)▪ Interpret collections or subgraphs of contingencies as

corroborating evidence for uptake ▪ Supports sequential analysis of interaction

w1

w2

w4

w3

Uptake Graph(Interaction Model)


Discussion 1

m1

m2

m3

m4Threading

Containment

w1

w2

w4

w3

r1

r2r1

r2r3

P2

P1

P3

Uptake Graph (Interaction Model)▪ Finds uptake not manifest in threading structure▪ P3 plays an integrative role in this discussion▪ Other examples in Suthers et al. (ijCSCL 2010)

w1

w2

w4

w3

Uptake Graph(Interaction Model)


Discussion 1

m1

m2

m3

m4Threading

Containment

P2

m1

w1

w2

w4

w3

r1

r2r1

r2r3

P2

P1

P3

Associograms● Directed affiliation network of actors and artifacts● Mediation Model: How actors' associations are mediated



Discussion 1

m1

m2

m3

m4Threading

Containment

P2 P1

m1

w1

w2

w4

w3

r1

r2r1

r2r3

P2

P1

P3



Discussion 1

m1

m2

m3

m4Threading

Containment

P2 P1

m1

m2

w1

w2

w4

w3

r1

r2r1

r2r3

P2

P1

P3



Discussion 1

m1

m2

m3

m4Threading

Containment

P2 P1

m1

m2

w1

w2

w4

w3

r1

r2r1

r2r3

P2

P1

P3

m3



Discussion 1

m1

m2

m3

m4Threading

Containment

P2 P1

P3

m1

m2

m3 m4

w1

w2

w4

w3

r1

r2r1

r2r3

P2

P1

P3

Finding Interaction Patterns: Round Trip

Contingency Graph (Contextual Action Model)


P2 P1

P3

m1

m2

m3 m4

w1

w2

w4

w3

r1

r2r1

r2r3

Recovering Temporal Information

Contingency Graph (Contextual Action Model)


P2 P1

P3

m1

m2

m3 m4

w1

w2

w4

w3

r1

r2r1

r2r3


P2 P1

P3

m1

m2

m3 m4 P2 P1

m1

m2

Dialogue

Relationships▪ Patterns of mediated associations reveal relationships

Pairwise Associations (Relationship Model)


P2 P1

P3

m1

m2

m3 m4 P2 P1

m1

m2

Dialogue P2 P3

m1

m3

Producer/Consumer




P2 P1

P3

m1

m2

m3 m4 P2 P1

m1

m2

Dialogue P2 P3

m1

m3

Producer/Consumer



P1 P2

M1

M2

SharedInterest

P1 P2

M1

M2

SharedAuthorship

Multi-Media Associations▪ Characterize pairwise relationships in terms of

distribution across media ▪ Compare roles of various media in supporting

associations (Suthers & Chu, Networked Learning 2010)

P2 P1

P3

m1

m2

m3 m4

w1

w2


P2 P1

P3

m1

m2

m3 m4

P2

P3

Ties▪ Straightforward to collapse into sociogram by transitive

closure or similar computations

P2 P1

m1

m2

Dialogue P2 P3

m1

m3

Producer/Consumer



P2 P1

P3

m1

m2

m3 m4


P2 P1

m1

m2

Dialogue P2 P3

m1

m3

Producer/Consumer

P2 P1

P3


Ties▪ SNA methods can now be applied

Contingency graphs used for ... ▪ Microanalysis of process through which learners achieved

an insight ▪ Semi-automated analyses of graph manipulations to find

pivotal moments

Prior Research

Tapped In (SRI International) ▪ Network of educators: professional development and peer

support (longest running educational online community)▪ 8 years of data; focusing on 2 year peak▪ 20K educators, 8K user-created spaces; 50 tenant

organizations▪ Chats, threaded discussions, wikis, resource sharing ...

Current Focus ▪ Identifying where significant activity takes place and

characterizing the nature of that activity (talk tomorrow afternoon)

▪ Nonlocal consequences of local activities, e.g., trace contingencies to find whether actors move ideas and other actors to new settings

Current Research

As a data representation ▪ Integration of distributed data: uncloak distributed interaction▪ Common format for reuse of algorithms

As an analytic framework ▪ Multi-Level Multi-Theoretical analysis possible ▪ Multiple ontologies allow for mapping between interaction,

mediated affiliation and tie levels of analysis

Workshop: Connecting Levels of Learning in Networked Communities ▪ July 5th @ CSCL in Hong Kong ▪ http://www.isls.org/cscl2011/ or http://engaged.hnlc.org/

Advantages



Discussion 1

m1

m2

m3

m4Threading

Containment

P2 P1

P3

m1

m2

m3 m4


P2 P1

P3

Uptake Graph (Interaction Model)


w1

w2

w4

w3

r1

r2r1

r2r3

aa.bbb.cc.74 [27/Aug/2007:16:03:24] "POST /workspace/282/note/2080/messages/reply_preview ..."aa.bbb.cc.74 [27/Aug/2007:16:03:25] "GET /workspace/282/note/2080/discussions/tree/756/message/9558 ..."aa.bbb.cc.74 [27/Aug/2007:16:03:38] "POST /workspace/282/note/2080/messages/post_reply?...&message_id=9558 ..."... xx.y.zzz.245 [27/Aug/2007:22:09:47] "POST /workspace/282/note/2080/messages/open/756/message/9558 ..." ...xx.y.zzz.245 [27/Aug/2007:22:18:07] "POST /workspace/282/note/2080/messages/reply_preview/9558 ..."xx.y.zzz.245 [27/Aug/2007:22:18:07] "GET /workspace/282/note/2080/discussions/tree/756/message/9576 ..."xx.y.zzz.245 [27/Aug/2007:22:18:22] "POST /workspace/282/note/2080/messages/post_reply?...&message_id=9576 ..."

Logs

P2

P1

P3

Dan [email protected]

w1

w2

w4

w3

Education

Suthers & Rosen, Learning Analytics and Knowledge 2011