http://creativecommons.org/licenses/by-sa/3.0

Characterising Citations in Scholarly Articles: an Experiment

Paolo Ciancarini – ciancarini@cs.unibo.itAngelo Di Iorio – diiorio@cs.unibo.it

Andrea Giovanni Nuzzolese – nuzzoles@cs.unibo.itSilvio Peroni – essepuntato@cs.unibo.it

Fabio Vitali – fabio@cs.unibo.itInternational Workshop on Artificial Intelligence and Cognitionco-located with XIII Conference of the Italian Association for Artificial IntelligenceTurin (Italy), December 3, 2013

What and Why

What and Why

• What: to analyse how humans use a particular citation model for the annotation of citations in scientific article

✦ Citation model under investigation: the Citation Typing Ontology (CiTO), an OWL-2 DL ontology describing various functions of citation (e.g. uses method in, extends, critiques)

What and Why

• What: to analyse how humans use a particular citation model for the annotation of citations in scientific article

✦ Citation model under investigation: the Citation Typing Ontology (CiTO), an OWL-2 DL ontology describing various functions of citation (e.g. uses method in, extends, critiques)

• Why: final aims (of future works): ✦ To improve CiTO by adding new properties, and by creating cluster of

semantically-similar properties (e.g. disagrees with and critiques) by analysing how humans use them when dealing with concrete tasks of annotation

✦ To improve CiTalO, a web application that tries to infer citation functions from sentences containing a citation, so as to reflect human behaviour when annotating citations

Who

users

Who

users

The metal m

odelsw

ere derived from

Who

citation model (CiTO)mentalmodel

mentalmodel

mentalmodel

mentalmodel

mentalmodel

users

The metal m

odelsw

ere derived from

Who

citation model (CiTO)mentalmodel

mentalmodel

mentalmodel

mentalmodel

mentalmodel

users

author

“It extends the research outlined in earlier work [3]”

Interpretation of author’s

text

The metal m

odelsw

ere derived fromMapping personal interpretation with citation functions

Who

citation model (CiTO)mentalmodel

mentalmodel

mentalmodel

mentalmodel

mentalmodel

users

author

“It extends the research outlined in earlier work [3]”

Interpretation of author’s

text

How, When and Where

How, When and Where

• How: preliminary user testing session with users to whom we asked to assign CiTO properties to the citations in the Proceedings of Balisage 2011:

✦ 18 papers✦ 104 citations selected (out of 377)✦ 5 users

How, When and Where

• How: preliminary user testing session with users to whom we asked to assign CiTO properties to the citations in the Proceedings of Balisage 2011:

✦ 18 papers✦ 104 citations selected (out of 377)✦ 5 users

• When: well, few months ago, but we are still experimenting involving a larger set of users and using different experimental settings

How, When and Where

• How: preliminary user testing session with users to whom we asked to assign CiTO properties to the citations in the Proceedings of Balisage 2011:

✦ 18 papers✦ 104 citations selected (out of 377)✦ 5 users

• When: well, few months ago, but we are still experimenting involving a larger set of users and using different experimental settings

• Where: the whole test was/is performed online, without any supervision

How, When and Where

• How: preliminary user testing session with users to whom we asked to assign CiTO properties to the citations in the Proceedings of Balisage 2011:

✦ 18 papers✦ 104 citations selected (out of 377)✦ 5 users

• When: well, few months ago, but we are still experimenting involving a larger set of users and using different experimental settings

• Where: the whole test was/is performed online, without any supervision

By the way, would you like to help us?

How, When and Where

• How: preliminary user testing session with users to whom we asked to assign CiTO properties to the citations in the Proceedings of Balisage 2011:

✦ 18 papers✦ 104 citations selected (out of 377)✦ 5 users

• When: well, few months ago, but we are still experimenting involving a larger set of users and using different experimental settings

• Where: the whole test was/is performed online, without any supervision

By the way, would you like to help us?

Please ... ;-( ...

Results

• Users have selected 34 different CiTO properties over 40✦ average: 22.4 properties per user

• Properties used many times✦ cites for information (110)✦ cites as related (39)✦ cites as data source (38)

• Low positive agreement for the 5 raters over all 104 subjects✦ k = 0.16

• Moderate positive local agreement on ✦ cites as data source: k = 0.5✦ cites as potential solution: k = 0.45✦ cites as recommended reading: k = 0.34✦ includes quotation from: k = 0.49

Results

• Users have selected 34 different CiTO properties over 40✦ average: 22.4 properties per user

• Properties used many times✦ cites for information (110)✦ cites as related (39)✦ cites as data source (38)

• Low positive agreement for the 5 raters over all 104 subjects✦ k = 0.16

• Moderate positive local agreement on ✦ cites as data source: k = 0.5✦ cites as potential solution: k = 0.45✦ cites as recommended reading: k = 0.34✦ includes quotation from: k = 0.49

Summarising: it is a difficult task for

humans

Results

• Users have selected 34 different CiTO properties over 40✦ average: 22.4 properties per user

• Properties used many times✦ cites for information (110)✦ cites as related (39)✦ cites as data source (38)

• Low positive agreement for the 5 raters over all 104 subjects✦ k = 0.16

• Moderate positive local agreement on ✦ cites as data source: k = 0.5✦ cites as potential solution: k = 0.45✦ cites as recommended reading: k = 0.34✦ includes quotation from: k = 0.49

Summarising: it is a difficult task for

humans

Which reminds me...

... that we are looking for users for

additional tests

Results

• Users have selected 34 different CiTO properties over 40✦ average: 22.4 properties per user

• Properties used many times✦ cites for information (110)✦ cites as related (39)✦ cites as data source (38)

• Low positive agreement for the 5 raters over all 104 subjects✦ k = 0.16

• Moderate positive local agreement on ✦ cites as data source: k = 0.5✦ cites as potential solution: k = 0.45✦ cites as recommended reading: k = 0.34✦ includes quotation from: k = 0.49

Summarising: it is a difficult task for

humans

Which reminds me...

Would you like to help us?Please ... ;-( ...

... that we are looking for users for

additional tests

Thanks for your attention