Shared Rationales in Group Activities

Lu Xiao Facul ty of Informat ion & Media Studies

Depar tment of Computer Science The Univers i ty of Western Ontar io

Ht tp: / /h i i . f ims.uwo.ca

14-03-04

1

Shared Rationales in Group Activities

Shared Rationales in Group Activities

14-03-04

2

Related Studies: Explanations in Knowledge-based Systems (KBS) Types of explanations, Content of explanations, Effects of KBS Explanations (explanation use behavior, learning, perceptions, and judgmental decision making) Shared Information in Group Activities �  How and why group members share information �  The factors of information sharing, information pooling phenomenon �  The information practices and cultures that members develop �  The effects of shared information and aspects of the shared information (e.g.,

representation strategy, the use of language) ¡  Influence the change of people’s attitudes ¡  Shared reflections

Tools to support information sharing in group activities �  Group decision support systems for hidden profiles �  Tools for capturing, archiving, and reusing design rationales �  Tools to foster reflective thinking in group learning activities

Rationale - the information that justifies one’s ideas, approach, and solution in group activities.

Research Gaps

14-03-04

3

1). the effects of shared rationales in group activities 2). Design requirements to promote the processes of articulating, sharing, and managing rationales in group activities

The Role of Shared Rationales in Group Ideation and Deliberation Activities

14-03-04

4

The effects of rationale awareness in �  small group ideation activities �  Large online crowdsourcing ideation activities

The effects of shared rationales in �  Large online deliberation activities

Rationale Awareness, as part of Activity Awareness (Carroll et al., 2003, 2005, 2011; Carroll, Rosson, Farooq, & Xiao, 2009), refers to one’s awareness of the other group members’ rationales in a group activity

14-03-04

5

The Effects of Rationale Awareness in Small Group Ideation Activities

The effects of rationale sharing (Xiao, 2011a, 2012; Xiao & Carroll, 2013)

•  Rationale awareness can contribute to one’s •  awareness of others’ knowledge and intellectual

contribution; can affect the development of his/her reflection skills

•  Explicit rationale sharing has potential downsides such as groupthink

14-03-04

6

Related Work: � Quality measure of different means

¡ Providing real-time assessment ¡ Collecting multiple assessment ¡ Analyzing workers’ behavior ¡ Parallel vs. iterative approach

The Effects of Shared Rationales in Online Crowdsourcing Ideation Activities

14-03-04

7

In an ideation task performed through online crowdsourcing processes, whether and how sharing previous workers’ good rationales of their generated ideas affects the ideas’ quality in the task? Pros: awareness of the others’ knowledge and intellectual contributions Cons: problems with explicit rationale sharing

The Effects of Shared Rationales in Large Online Ideation Activities

Research Design

In an idea evaluation task performed through online crowdsourcing processes, whether and how showing the idea’s rationale affect its evaluation?

Hypothesis: Making the ideas’ rationales available to all of its evaluators reduce the variation between evaluations by multiple raters

Research Design

�  Two iterative conditions in the idea generation task: presence vs. absence of previous workers’ rationales

�  Two idea evaluation conditions in the idea evaluation task: presence vs. absence of the idea’s rationale

�  Manipulation of the rationale’s quality: Experiment 1, Experiment 2, and Experiment 3 ¡  The quality of the ideas and rationales was checked after all the

iterations were completed (Experiment 1) ¡  The quality of the ideas and rationales was checked at the end of each

iteration (Experiment 2 and 3) – stricter and better quality control of ideas and rationales

Research Design (Little et al.,2010)

Brainstorming/Idea Generation Task

• Six company descriptions • Five names for a company description in each iteration • Six iterations for each company

Rating/Idea Evaluation Task

Each name had 10 ratings

Turkit

•  Open source software: Java/JavaScript API for running iterative tasks on Mechanical Turk.

Manipulation Check

14-03-04

12

Results – Average Quality of the Ideas

14-03-04

13

Results – Best Quality of the Ideas

14-03-04

14

Results: Rationale Awareness in Idea Evaluation

Findings: the Shared Rationales in Online Crowdsourcing Ideation Task

�  In an idea generation task, the awareness of previous workers’ rationales may slightly improve the average quality but NOT the best quality of the generated ideas in iterative approach

�  In an idea evaluation task, the awareness of an idea’s rationale can affect the evaluation outcome and the quality of the rationale may play a significant role on the evaluation

(Xiao, CSCW, 2012; Xiao, CI, 2012; Xiao, JASIST, to appear)


14-03-04

17



Shared Rationales in Large Online Deliberation

14-03-04

18

�  Deliberation concept ¡  Habermas (1989) – public sphere ¡  Halpen and Gibbs (2013)

a communication process that involves at least two individuals; that focuses on a social or political issue where the solutions are identifiable by participants; and that values equality among participation and emphasizes rational thinking and logic instead of a power struggle.

14-03-04

19

Wikipedia’s Article for Deletion (AfD) discussions

Step 1 : Types of rationales; factors of deliberation outcome

What are the types of rationales used in the deliberation? Are there any relationships among the kinds of votes, the article’s topic, the discussion situation (unanimous or non-unanimous), and the final decision?


14-03-04

20

Possible outcomes of deletion discussions Coded as…

Outcome Explanation Keep Article is kept. Changes may or may not be suggested as part of debate. Keep Withdrawn Nomination is withdrawn. No consensus Debate is inconclusive. This might result from disagreement, lack of

participation, or other factors. Article defaults to keep. Procedural close Debate is closed because of problems with the nomination. Delete Article is deleted. Delete Speedy delete Article is deleted under the “speedy” criteria outlined at WP:CSD. Userfy Article is deleted but a copy is given to a user to work on as a draft, and may

be recreated as an article later. Other

Incubate As with userfy, but in a communal space rather than related to a single user. Merge Article is deleted but some or all of its content is added into one or more

existing articles. Rename or Move Article’s title is changed. Its scope may or may not be amended. Convert Article is converted into another type of page, usually one with a structural

function. For example, a list might be changed into a category to be added to the list entries.

Transwiki Article is deleted from English Wikipedia but moved to another Wikimedia project as appropriate – for example, a French article to French Wikipedia or an image gallery to Wikimedia Commons.

Redirect Article’s content is replaced with a pointer to another page. Split Article is divided into one or more new pages, or part of the article is moved

to another page.

Research Methodology

�  Qualitative Analysis ¡  Open coding process to classify rationales used in deletion

debates on three selected dates

�  Quantitative Analysis ¡  Chi Square Tests ¡  Relationships among articles’ topics and deliberation

outcomes, discussion situations ¡  Relationship between the SOPA act event and the deliberation

outcome

SOPA (Stop Online Piracy Act) act event: On January 18, 2012, the English Wikipedia, Google, and an estimate of 7,000 other smaller websites coordinated a service blackout, to raise awareness.

Sample

�  Date selection for qualitative analysis

�  Date selection for quantitative analysis ¡  Previous sample ¡  20 dates for chi-square tests that require larger sample size (a priori power

analysis) ÷  Before the SOPA act event: Jan. 1 - 10, 2012 and Nov. 1 - 10, 2011 (N = 1453) ÷  After the SOPA act event: Jan. 20 - 29, 2012 and March 20- 29, 2012 (N = 1202)

Day # of articles

Total votes for “keep”

Total votes for “delete”

Total “other” votes (merge, userfy, etc)

1 Jun. 2010 89 127 280 37

1 Jun. 2011 73 119 212 23

15 Jan. 2012 67 109 200 63

Findings - Types of rationales; factors of deliberation outcome

14-03-04

23

�  Rationales are mainly about the articles’ notability (50%) and credibility (12%); Wikipedia policies are often referred to as well (10%)

�  Relationship between the deliberation outcome ÷ and the type of votes: in the case that the decision is delete, there

tend to have more delete votes than keep votes, whereas in the case that the decision is keep, the delete votes are not more than keep votes; the votes other than keep and delete significantly affect those decisions that would change the articles’ status.

÷ and the topic of article : articles about people, for-profit organizations, and definitions are slightly more likely to be deleted than expected; articles about locations or events are more likely to be kept than expected; and articles about non-profit organizations and media are more likely to be suggested for other options (e.g., merge, redirect, etc) than expected

(Xiao & Askin, JASIST, to appear)

Findings - Types of rationales; factors of deliberation outcome

14-03-04

24

�  Relationship between the discussion situation ¡  and the type of rationale: more agrees in non-unanimous

situations ¡  and the deliberation outcome: in non-unanimous

situations, it is more likely to have final decisions as keep or other solutions (e.g., merge, redirect, etc)

¡  and the community participation: ÷ more unique Wikipedia IDs in non-unanimous discussions ÷ More participants in a non-unanimous discussion; the most

involved participant was more likely to be recognized in the discussion

(Xiao & Askin, JASIST, to appear)

Findings – the Impact of SOPA blackout event on the Deliberation

14-03-04

25

Before the blackout of the site in response to the proposed Stop Online Piracy Act (SOPA) law, there were slightly less keep cases than expected and after the event there were slightly more keep cases than expected. The effect was more significant on the decisions which would change the articles’ status. These articles were more likely to be deleted before the Act, whereas after that Act (or during the discussions about it) they were more likely to be offered suggestions for other options

14-03-04

26

Wikipedia’s Article for Deletion (AfD) discussions Step 1 : Types of rationales; factors of deliberation outcome


Step 2: Computational linguistic approaches to extract the rationales

Rationale Extractions for Knowledge Management

14-03-04

27

�  Motivation – to benefit new editors

�  Approach - extraction of rationales that reflect the needed knowledge on Wikipedia policies in AfD discussions

�  Technique –Illocutionary Act (Searle, 1976) ¡  Representatives ¡  Directives ¡  Commissives ¡  Expressives ¡  Declarations

Rationale Extractions for Knowledge Management

14-03-04

28

�  Directives (Ervin-Tripp, 1976) ¡  Need statements ¡  Imperatives ¡  Imbedded imperatives ¡  Permission directives ¡  Question directives ¡  Hints

Detect Imperatives

14-03-04

29

1.   A verb (in its base form) as the root in the phrase structure and this particular verb has no subject child in the dependency structure.

(ROOT (S (INTJ (VB please)) (VP (VB refrain) (PP (IN from) (S (VP (VBG making) (NP (JJ personal) (NNS attacks)))))) (. .)))

2. A personal pronoun or noun (e.g., you, they, username) followed by a modal verb (e.g., should, must) "You must discuss the matter there, and you need to be specific”

Rationale Extractions for Decision-Making Support

14-03-04

30

�  Motivation – to facilitate efficient final decision-making

�  Approach – elimination of redundancy by identifying representative rationales in the discussion

�  Technique – text similarity and sentiment analysis

classify by sentiment polarity

group similar rationales

Discussion

Group A

positive neutral negative

Group B

positive neutral negative

Rationale Extractions for Decision-Making Support

Select representative rationales – similarity score, number of users,

policies

•  SEMILAR, a semantic similarity toolkit, was used to compute text similarity

�  Compared the performance of similarity measure among algorithms and human evaluation: ¡  Weighted Latent Semantic Analysis (LSA) ¡  Latent Dirichlet Allocation (LDA)

Text Similarity

�  Determine the sentiment polarity of a rationale in our language context (“notable”) ¡ MPQA Subjectivity Lexicon + additional words

Sentiment Analysis

Data Input Stanford Parser

Part-of-speech tagged text

Dependency relations

Check modified MPQA subjectivity lexicon to obtain the prior polarity

(if not in MPQA, marked as ‘non-sentiment’)

MPQA format: type=strongsubj len=1 word1=aberration pos1=adj stemmed1=n priorpolarity=negative

•  Local negation: A not usually modifies the sentiment word. –  “The place is not notable.”

•  Predicate negation: using verbs with negative polarity. –  “I disagree that the place is notable.”

•  Subject negation: a subject leads to the negation of its predicate. –  “Neither one of us agrees that the place is notable.”

Sentiment Analysis

�  Preposition negation: the polarity of the object following the preposition “of” can be changed by the word modified by the preposition. ¡  “It is a viola&on of notability.”

�  Modifier negation: some sentiment word’s polarity can be negated by its modifier. ¡  “The place is of indeterminable notability.”

Sentiment Analysis

•  Modifier negation –  Phrase in the following combination:

Noun modified by adjective Noun modified by noun Adjective modified by adverb Adverb modified by adverb Verb modified by verb

Sentiment Analysis

�  Using machine learning methods to determine the polarity of a phrase that has a modifier and a word

–  Features: •  First word token •  Second word token •  First word polarity •  Second word polarity •  First word part-of-speech •  Second word part-of-speech

l  Performance of Naïve Bayes, k-nearest neighbor (KNN) and decision tree:

÷ Data: 961 instances (phrases)

÷ Evaluation:10 folds cross validation

Sentiment Analysis

Naïve Bayes K-nearest neighbor Decision Tree

Accuracy (%) 77.94 83.77 80.65

� Bottom-up (recursive) algorithm ¡  Based on dependency structure

Sentiment Analysis

�  Evaluation ¡ Data: 236 sentences from discussions in AfD ¡ 3 classes: positive, negative, neutral ¡ Accuracy: 58.47%

Sentiment Analysis


14-03-04

40



Current research plan To automatic detect rationales from online ideation activities and deliberation activities

Current Research Plan

Rhetorical Structure Theory has been recently used to identify justifications in the social Web (Biran, and Rambow, 2011), where the existence of certain discourse structures has been considered argument indicators. Justification is defined as: 1.  Recommendation for action, and motivation for proposed action. 2.  Statement of like or dislike or of desires and longing, and subjective reason for this like

or dislike or desire or longing 3.  Statement of like or dislike or of desires and longing, and claimed objective reason for

this like or dislike or desire or longing 4.  Statement of subjectively perceived fact, with a proposed objective explanation 5.  A claimed general objective statement and a more specific objective statement that

justifies the more general one

Presentational relations from RST Treebank were primarily considered

14-03-04

42

The Effects of Rationale Awareness in Small Group Ideation Activities

Discourse relations in shared rationales in the small group ideation activities (Xiao, 2013):

Most used strategies in justifying one’s ideas in the activities were: providing contextual information (circumstance), additional information (elaboration), and evaluation of the information (evaluation)

We are extending Biran and Rambow’s (2011) approach by conducting further analysis on these discourse relations and their potential connections to different types of reasoning.

Acknowledgement

14-03-04

43

John M. Carroll, Mary Beth Rosson, Craig Ganoe, Shawn Clark, Steven High, Anabel Quan-Haase, Yan Luo, Tatiana Vashchilko, Trina Joyce Sajo, William Klie, Becky Ellis, Mengshuo Chen, Nicole Askin, Jill Kavanaugh, Lindsay Baker, Achchana Nadarajah, Yumo Yin, Vadim Mazalov, Wanting Mao, Taraneh Khazaei

Funding support: NSERC Discovery, NSERC Engage, MITACS, SSHRC, FIMS Internal Funding, SSHRC 4A assistance

Education

Shared Rationales in Group Activities