How Well Do Online Forums Facilitate How Well Do Online Forums Facilitate Discussion and Collaboration Among Discussion and Collaboration Among

Novice Animation Programmers?Novice Animation Programmers?

Christopher Scaffidi, Aniket Dahotre, Yan ZhangSchool of Electrical Engineering and Computer Science

Oregon State University

22

Animation programmingAnimation programming

• Effective for motivating students to learn programming• Increasingly used in freshman CS courses

• Many tools available– Logo

– Alice

– KidSim

– AgentSheets

– …

Introduction Study Statistical results Social network results Discussion

33

Scratch as a particular Scratch as a particular programming toolprogramming tool

• Turing-complete language

• Events, loops, conditionals, sprites, sound…

• Drag-and-drop programming

+Online

community

Introduction Study Statistical results Social network results Discussion

44

Massive online communityMassive online community

• 1,019,863 registered members (Feb 2012)– Post animations, discuss animations, use online forum

• Learn by building and reflecting upon shared objects– Using constructionism (à la Papert) to support

social constructivism (à la Vygotsky)

– Essential ingredients:

• Learners

• Learner collaboration to create objects

• Sharing of created objects

• Reflective discussion about the objects

Introduction Study Statistical results Social network results Discussion

55

Key results from previous studyKey results from previous study

• Analysis of user comments about 100 posted animations– 27% compliment, 27% feature suggestions/critique,

26% chit-chat off-topic, 20% replies by animation owner

– 0% of the comments indicated that the animations under discussion were shared creations

Where is the collaboration and social support for learning??

Introduction Study Statistical results Social network results Discussion

66

Which made us start to wonderWhich made us start to wonder

• Maybe the post-animation-for-discussion model isn’t effective for anchoring collaboration– Maybe we need to think harder about what

social constructionism really calls for, in this context?

• Or maybe the model is fine, but the implementation is just getting in the way– E.g., maybe the short, Twitter-like user interface in that

part of the site isn’t conducive to forming collaboration

– Maybe forums might be better; they allow more space

Introduction Study Statistical results Social network results Discussion

77

Specific questions for this studySpecific questions for this study

• Do different kinds of initiating posts lead to differences in ensuing discussions?– Does posting an animation obtain more interaction?

– Does inviting collaboration obtain more interaction?

– Do different topics obtain different levels of interaction?

– Do all users obtain the same level of interaction?

• What kinds of collaboration take place via the forums?– Does interaction resemble collaboration in some sense?

– If so, what does collaboration practically entail?

Introduction Study Statistical results Social network results Discussion

88

% of all forum posts, categorized based % of all forum posts, categorized based on respective forum’s primary focuson respective forum’s primary focus

Word Games, 34%

Show Off, 31%

Introduction Study Statistical results Social network results Discussion

99

Acquiring data for analysisAcquiring data for analysis

• Focused on forums…– Where most discussions were started by users

– And did not simply involve word games

• Which led to…– 11 Show Off, Trouble, and Collab Request forums

– Containing 397,477 posts in 36,886 discussions

– We scraped them all

Introduction Study Statistical results Social network results Discussion

1010

Analysis methods—StatisticalAnalysis methods—Statistical

• Four statistical hypotheses– Link to animation will obtain different levels of interaction

– Inviting collaboration will obtain different level of interaction

– Posts in different kinds of forums (Show Off / Trouble / Collab Requests) will obtain different levels of interaction

– Community managers’ and ordinary users’ posts will obtain different levels of interaction

• Operationalize “level of interaction” with four measures of interaction– # replies, # distinct users involved, # animations posted in

reply, # collaborative words used

Introduction Study Statistical results Social network results Discussion

1111

Analysis methods—Social network Analysis methods—Social network analysis, plus qualitative analysisanalysis, plus qualitative analysis

• Compute social network graph (directed graph)– One node per user

– One edge indicating user responded to another user

• Compute statistics characterizing overall structure– Standard statistics, eg, avg in-degree, connectedness

• Grounded theoretical (qualitative) analysis of typical interactions– Identify 30 moderately sized clusters

– Create a coding scheme for categorizing each cluster

– Another researcher independently categorizes clusters with same coding scheme

Introduction Study Statistical results Social network results Discussion

1212

Do different kinds of initiating posts obtain Do different kinds of initiating posts obtain differences in ensuing discussions?differences in ensuing discussions?

• Link to animation led to different levels of interaction

0

1

2

3

4

5

6

7

8

9

10

n_replies n_users n_anim n_collab_words

Has link to animationNo link to animation

Initiating post (has_anim)

** * ** * ** * ** * ** * ** * **Introduction Study Statistical results

Social network results Discussion

1313

Do different kinds of initiating posts obtain Do different kinds of initiating posts obtain differences in ensuing discussions?differences in ensuing discussions?

• Inviting collaboration led to different levels of interaction

0

2

4

6

8

10

12

14

n_replies n_users n_anim n_collab_words

Uses collaboration word

Does not mention any

Initiating post (collab_word)

* **Introduction Study Statistical results

Social network results Discussion

1414

Do different kinds of initiating posts obtain Do different kinds of initiating posts obtain differences in ensuing discussions?differences in ensuing discussions?

• Posts in different kinds of forums were followed by different levels of interaction

0

2

4

6

8

10

12

14

16

n_replies n_users n_anim n_collab_words

Show OffTroubleCollab Requests

Forum category (frm_catg)

* ** *

Introduction Study Statistical results Social network results Discussion

1515

Do different kinds of initiating posts obtain Do different kinds of initiating posts obtain differences in ensuing discussions?differences in ensuing discussions?

• Different users’ posts led to different levels of interaction

0

2

4

6

8

10

12

14

16

18

20

n_replies n_users n_anim n_collab_words

Community managerOrdinary user

Initiator (mgr)

* ** *

Introduction Study Statistical results Social network results Discussion

1616

What kinds of collaboration take place What kinds of collaboration take place via the forums?via the forums?

• Social network analysis of user interactions within the Collab Request forums– 2074 user nodes

– average in-degree only 6.27

– average of 20.5 persons per weak component

– bi-directional connection to an average of 0.40 people

• Fragmented community, with any collaboration likely within small groups consisting of a few people

Introduction Study Statistical results Social network results Discussion

1717

Social Social networknetwork

One directed edge per pair of users who ever replied to each other

Introduction Study Statistical results Social network results Discussion

1818

Social Social networknetwork

• Exactly same graph; different visualization– Showing only

edges where ≥20% of user’s replies went to other user

– Omitting arrowheads

– Run Gephi layout engines

Introduction Study Statistical results Social network results Discussion

1919

Identifying discussions for analysisIdentifying discussions for analysis

• Users with in-degree approximately in the 80th percentile– In-degree of 10 or 11

– Not dependent on the visualization

• Randomly select 30 discussions initiated by these users• Code with grounded theoretical analysis

Introduction Study Statistical results Social network results Discussion

2020

Forms of interaction observedForms of interaction observedCategory # Example

Requests for media 3 Peanut butter jelly time...I need the banana!

Requests for ideas 4 Need an idea? Or want to share one? Look here!...

Requests for feedback

2 I am currently working on a big animation… can't dicide which ideas to use… A true story about 18 American soldiers in WW2 who fended off 500 German soldiers

Requests for participation

2 This will be a weekly paper that comes in on Sunday.… I need these people for the Scratcher Times.

Tips 2 Does anyone know a program that compresses audio files?

Forum meta-discussions

4 I'm probably going to get banned soon.

Community meta-discussion

2 My new website is opening up soon and I have put a Scratch games section on the Games page.

Non-Scratch topics 11 It's getting pretty cold, and it's snowing nearly the whole day

Introduction Study Statistical results Social network results Discussion

2121

Forms of interaction observedForms of interaction observed

Introduction Study Statistical results Social network results Discussion

2222

Forms of interaction observedForms of interaction observed~ 1/3 of discussionscontributed to creatingspecific projects

~ 1/3 of discussionswere Scratch-related

but not project-specific

~ 1/3 of discussionswere not Scratch-

related at all

Introduction Study Statistical results Social network results Discussion

2323

Putting in perspectivePutting in perspective

• 12% of all forum posts were in Collab Request forums– Of these users, fewer than 20% had in-degree ≥ 10

• Of these users’ discussions, ~ 1/3 contributed to a project

– Requests for feedback– Requests for ideas– Requests for media– Requests for participation

Interpretation: collaboration seems fairly rare(We have found the needle in the haystack,

and this is what it looks like.)

Introduction Study Statistical results Social network results Discussion

2424

Most interesting conclusions Most interesting conclusions

• Requests for collaboration led to highest interaction (vs. posts in forums focused on getting help or showing off)

• Posting a link did not lead to more replies or more users

• Collaboration in the forums is very rare– But when it does occur, it involves requests for

ideas, feedback, media, and participation

Introduction Study Statistical results Social network results Discussion

2525

Possible implications for Possible implications for distance educationdistance education

• Requests for help obtained the lowest level of interaction– Learners might be better able to obtain help if they

explicitly request collaboration, not just information

• Posting an anim link did not lead to more replies or users– More effective ways are needed to structure and

stimulate critical reflection and discussion

• Collaboration was rare– More effective ways are needed to structure and

stimulate and organize collaboration among learners

Introduction Study Statistical results Social network results Discussion

2626

Thank you…Thank you…

• To SIGCSE for this opportunity to present

• To you for your attention, questions, and potential collaboration

Introduction Study Statistical results Social network results Discussion