Automated argumentation mining to the rescue? Envisioning argumentation

and decision-making support for debates in open online collaboration communities

Jodi Schneider

First Workshop on Argumentation Mining at ACL 2014Baltimore, Maryland2014-06-26

Argumentation mining today

• No unified vision of the field. Multiple:– Interrelated problems– Application domains– Tools handling one aspect of annotation

• Few corpora• Need for– Common definition(s) of argumentation– "Challenge problems"– Shared corpora– Applications

Argumentation mining today

• No unified vision of the field. Multiple:– Interrelated problems– Application domains– Tools handling one aspect of annotation

• Few corpora• Need for– Common definition(s) of argumentation– "Challenge problems"– Shared corpora– Our Application: debates in online collaboration

Application: Debates in Open Online Collaboration

• Wikipedia• HTML5• OpenStreetMap• Project Gutenberg• Apache projects• Mozilla Firefox• …

Debates argue for & against a single conclusion

Which content belongs in Wikipedia?

500 discussions per week!!!

Argument-based support

• How can I win an argument? Which arguments sway the community?

• Why were previous decisions made?• Which ongoing debates need more comments?

Argument-based support

• How can I win an argument? Which arguments sway the community?

• Why were previous decisions made?• Which ongoing debates need more comments?

Corpus: 72 discussions started on 1 day

• Each discussion has:3-33 messages2-15 participants

• 741 messages contributed by 244 users.Each message has 3-350+ words.

• 98 printed A4 sheets

Approach

• Compare two argumentation theories• Iterative annotation with multiple annotators– Refine to get good inter-annotator agreement

• 4 rounds of annotation– Rounds 1-2 by me– Rounds 3-4 by 2 assistants

We used two argumentation theories

• Walton’s Argumentation Schemes (Walton, Reed, and Macagno 2008)

– Informal argumentation (philosophical & computational argumentation)

– Identify & prevent errors in reasoning (fallacies)– 60 patterns

• Factors/Dimensions Analysis (Ashley 1991; Bench-Capon and Rissland, 2001)

– Case-based reasoning– E.g. factors for deciding cases in trade secret law,

favoring either party (the plaintiff or the defendant).

Walton’s Argumentation Schemes

Example Argumentation Scheme: Argument from Rules – “we apply rule X”

Critical Questions1. Does the rule require carrying out this type of action?

2. Are there other established rules that might conflict with or override this one?

3. Are there extenuating circumstances or an excuse for noncompliance?

Walton, Reed, and Macagno 2008

Example: "Stop at a red light"

1. Does the rule require carrying out this type of action?Were you driving a vehicle?

2. Are there other established rules that might conflict with or override this one?Did a police officer direct you to continue without stopping?

3. Are there extenuating circumstances or an excuse for noncompliance?Were you driving an ambulance with its siren on?

Critical Questions from Argument from Rules based on Walton, Reed, and Macagno 2008

Walton’s Argumentation Schemes

Jodi Schneider, Krystian Samp, Alexandre Passant, Stefan Decker. “Arguments about Deletion: How Experience Improves the Acceptability of Arguments in Ad-hoc Online Task Groups”. In CSCW 2013.

How to win an argument with a Wikipedian?

• Argument from Evidence to Hypothesis (19%)• Argument from Rules (17%)

How to win an argument (Arucaria)?

Classifying Arguments by Scheme. Vanessa Wei Feng. Master's thesis, Toronto, 2010.

Experts vs. Novices

• Experts were more likely to use – Argument from Precedent

• Novices were more likely to use– Argumentation from Values– Argumentation from Cause to Effect– Argument from Analogy

Unsuccessful arguments from novices

• Emsworth Cricket Club is one of the oldest cricket clubs in the world, and this really is worth a mention. Especially on a website, where pointless people … gets a mention.

• Why just because it is a small team and not major does it not deserve it’s (sic) own page on here?

Newcomers don't understand how to counterargue

Factors/Dimensions Analysis

• Factors (case-based reasoning)– All or nothing• Either present ("applicable") or absent• When present, a factor always favors the same side

• Dimensions– More complex/subtle• Can be applicable to a varying degree ("sliding scale")• Favor plantiff on one extreme; defendant on the other

Ashley 1991; Bench-Capon and Rissland, 2001

Example factors analysis (Aleven 1997)

Wikipedia Factors Analysis

Factors determined by iterative annotation

4 Factors cover– 91% of comments– 70% of discussions

“Other” as 5th catchall

Wikipedia Factors Analysis

Factors determined by iterative annotation

4 Factors cover– 91% of comments– 70% of discussions

“Other” as 5th catchall

Factor Example (used to justify `keep')

Notability Anyone covered by another encyclopedic reference is considered notable enough for inclusion in Wikipedia.

Sources Basic information about this album at a minimum is certainly verifiable, it's a major label release, and a highly notable band.

Maintenance …this article is savable but at its current state, needs a lot of improvement.

Bias It is by no means spam (it does not promote the products).

**Other I'm advocating a blanket "hangon" for all articles on newly-drafted players

Wikipedia Factors Analysis

Factors determined by iterative annotation

4 Factors cover– 91% of comments– 70% of discussions

“Other” as 5th catchall

Factor Example (used to justify `keep')

Notability Anyone covered by another encyclopedic reference is considered notable enough for inclusion in Wikipedia.

Sources Basic information about this album at a minimum is certainly verifiable, it's a major label release, and a highly notable band.

Maintenance …this article is savable but at its current state, needs a lot of improvement.

Bias It is by no means spam (it does not promote the products).

**Other I'm advocating a blanket "hangon" for all articles on newly-drafted players

Factor-based Summarization

Comparison of Annotation

• Cohen’s kappa (Cohen, 1960).48 for Walton’s argumentation schemes.64-.82 for factors, depending on the factor

• Potential for task support– Argumentation schemes • Write effective arguments• Ask critical questions to check others' arguments

– Factors• Summarize debates

Argumentation mining could be the basis for support tools

• Help participants write persuasive arguments– How: provide personalized feedback on drafts– Requires: knowing which arguments are accepted;

identifying argumentation in a drafts • Find weaknesses in others’ arguments– How: suggest & instantiate relevant critical questions– Requires: identifying argumentation schemes

• Summarize the overall conclusions of the debate– How: identify the winning and losing rationales– Requires: identifying rationales and contradictions

Argumentation mining could be the basis for support tools

• Help participants write persuasive arguments– How: provide personalized feedback on drafts– Requires: knowing which arguments are accepted;

identifying argumentation in a drafts • Find weaknesses in others’ arguments– How: suggest & instantiate relevant critical questions– Requires: identifying argumentation schemes

• Summarize the overall conclusions of the debate– How: identify the winning and losing rationales– Requires: identifying rationales and contradictions

Argumentation mining could be the basis for support tools

• Help participants write persuasive arguments– How: provide personalized feedback on drafts– Requires: knowing which arguments are accepted;

identifying argumentation in a drafts • Find weaknesses in others’ arguments– How: suggest & instantiate relevant critical questions– Requires: identifying argumentation schemes

• Summarize the overall conclusions of the debate– How: identify the winning and losing rationales– Requires: identifying rationales and contradictions

Argumentation Mining papersArguing on Wikipedia • “Arguments about Deletion: How Experience Improves the Acceptability of Arguments in Ad-hoc

Online Task Groups” CSCW 2013.• “Deletion Discussions in Wikipedia: Decision Factors and Outcomes” WikiSym2012.Arguing in Social Media• “Dimensions of Argumentation in Social Media" EKAW 2012• “Why did they post that argument? Communicative intentions of Web 2.0 arguments.” Arguing on

the Web 2.0 at ISSA 2014Arguing in Reviews• “Identifying Consumers' Arguments in Text” SWAIE 2012• “Semi-Automated Argumentative Analysis of Online Product Reviews" COMMA 2012• “Arguing from a Point of View” Agreement Technologies 2012Structuring Arguments on the Social Semantic Web• “A Review of Argumentation for the Social Semantic Web” Semantic Web – Interoperability, Usability,

Applicability, 2013.• “Identifying, Annotating, and Filtering Arguments and Opinions in Open Collaboration Systems" 2013

Thesis: purl.org/jsphd• “Modeling Arguments in Scientific Papers” at ArgDiaP 2014

http://jodischneider.com/jodi.html

http://purl.org/jsphd

An argument gives a rationale

What do we mean by an argument?

• The simplest possible argument connects two Statements by means of an Inference Rule

Cases

• Cases support a single conclusion.

Attacks & Debates

• Debates argue for and against a single conclusion.

Open collaboration based on

• Technological infrastructure• People• Social structures: rules, policies, procedures,…

Open collaboration based on

• Technological infrastructure• People• Social structures: – joint decision-making– Importance of rationales: reasons for opinions

argumentation and decision-making support for debates in open online collaboration communities

HTML5

OpenStreetMap

Example from Corpus

Results: Important tasks for consensus discussions

1. Determine one’s personal position2. Express one’s personal position in accordance

with community norms3. Determine the consensus

Related work

• Dissent and rhetorical devices in bug reporting (Ko and Chilana, 2011)

• how Python listservs select enhancement proposals (Barcellini et al., 2005). – role of a participant is related to the kinds of

message they quote (Syntheses, Disagreements, Proposals, or Agreements)

– Syntheses and Disagreements are the most quoted