COMPUTATIONAL ANALYSES OF THE EFFECTS OF A …...COMPUTATIONAL ANALYSES OF THE EFFECTS OF A STRUCTURE STRATEGY ON COLLEGE-LEVEL SUMMARIES: COHESION AND RHETORICAL STRUCTURE, Winterthur

COMPUTATIONAL ANALYSES OF THE EFFECTS OF A STRUCTURE STRATEGY ON COLLEGE-LEVEL SUMMARIES:

COHESION AND RHETORICAL STRUCTURE

, Winterthur Switzerland, 5-6 September 2019, https://writinganalytics.zhaw.ch

Tamara Sladoljev-Agejev, University of Zagreb

Jan Šnajder, University of Zagreb

Svjetlana Kolić-Vehovec, University of Rijeka

THE EIGHTH INTERNATIONAL CONFERENCE ON WRITING ANALYTICS https://writinganalytics.zhaw.ch

Winterthur, Switzerland, 5-6 September 2019

https://writinganalytics.zhaw.ch/

GENERAL RESEARCH QUESTION

Can the effects of structure strategy training be

identified automatically in student summaries?

THE EIGHTH INTERNATIONAL CONFERENCE ON WRITING ANALYTICShttps://writinganalytics.zhaw.ch


STRUCTURE STRATEGIES HELP CONTENT INTEGRATION

Structure strategies: e.g. notes/graphic organizers (e.g. Jiang, 2012), summaries (e.g. Kirkland & Saunders, 1991)

Structure strategies ‘enable students to ...

a. follow the logical structure of text to understand how an author organized and emphasized ideas;

b. increase their own learning and thinking (e.g.,comparing, finding causal relationships, looking for solutions to block causes of problems);

c. use these text structures to organize their own writing ...’ (Meyer & Ray, 2011, p. 128)

THE EIGHTH INTERNATIONAL CONFERENCE ON WRITING ANALYTICS,https://writinganalytics.zhaw.ch


Note-making and summaries are coherence-building exercises

TEXT MACROSTRUCTUREKintsch & van Dijk (1978); Lorch (2001); Louwerse & Graesser (2005); global coherence (source author’s plan/intention, Hobbs, 1993, Grosz & Sidner, 1986)

READING-FOR-UNDERSTANDINGintegrating segments of a text into a coherent

whole (Sabatini et al., 2013)

READING-TO-WRITEDelaney (2008)

conveying information – ‘real-life skill’ (Folz, 2016)

van den Broek et al., 1995; Zwaan & Singer, 2003

COHERENCE

IDEAS

concepts/propositions

RELATIONSe.g. causality, listing,

comparison, problem-solution

OUR RESEARCH

based on teaching rhetorical structure strategy (RSS)

↓

identifying the rhetorical structure of a text (Moore & Wiemer-Hastings, 2003) to achieve deep comprehension(i.e. ideas & rhetorical relations)


Winterthur. Switzerland, 5-6 September 2019

Quasi-experimental research design

PRE-TEST

C-E

I BEFORE READING (TEXT 1)

1. Biodata

2. Metacognitive self-assessment

3. EL2 proficiency

a) grammar

b) vocabulary (Text 1)

4. Prior knowledge (Text 1)

II READING & NOTE MAKING

III AFTER READING (TEXT 1)

5. Summary writing

RSS INTERVENTION

E

READING AND NOTE-MAKING (GRAPHIC ORGANIZERS)

1. Paragraph-level comprehension (key words and headings to paragraphs)

2. Text segmentation (paragraph grouping)

3. Establishing rhetorical relations witin and between paragraphs

4. Making notes with explicitly indicated rhetorical relations (graphic organizers)

POST-TEST

C-E

I BEFORE READING (TEXT 2)

1. Prior knowledge (Text 2)

2. Vocabulary (Text 2)

II READING & NOTE MAKING

III AFTER READING (TEXT 1)

6. Summary writing



Pretest/posttest: A three-step procedure

SOURCE TEXT

reading

RHETORICAL STRUCTURE (MACRO, GLOBAL)

making structured notes, i.e. graphic organizers)

SUMMARY

writing



Part of a broader research study

COHERENCE?

(assessed by human raters)

COHESION?


RHETORICAL STRUCTURE?

Šnajder, Sladoljev-Agejev & Kolić-Vehovec (2019)

COH-METRIX INDICES?

Crossley & McNamara, 2009, 2010 & 2011; Graesser et al., 2004; McNamara & Graesser, 2012

EFFECTS OF RHETORICAL STRUCTURE STRATEGY

(RSS)



Research questions in this study

COHERENCE?


COHESION?


RHETORICAL STRUCTURE?

Šnajder, Sladoljev-Agejev & Kolić-Vehovec (2019)

COH-METRIX INDICES?

Crossley & McNamara, 2009, 2010 & 2011; Graesser et al., 2004; McNamara & Graesser, 2012

EFFECTS OF RHETORICAL STRUCTURE STRATEGY

(RSS)

THE EIGHTH INTERNATIONAL CONFERENCE ON WRITING ANALYTICS, https://writinganalytics.zhaw.chWinterthur Switzerland, 5-6 September 2019

Instruction to participants

‘Write a summary in the note form which will clearly convey the ideas of the text to a third person. Then write the summary as connected (linear) text.’



DATASETTot. 225 text-present summaries (300+/-10% w.), Sladoljev-Agejev & Šnajder (2017)

113 first-year business/economics undergraduates (English-L2, mostly upper intermediate and advanced)

Pretest: C(N=55), E (N=58), source text - 901 w.(The Economist)

Posttest: C (N=55), E(N=58), source text - 981 w. (The Economist)

Raters: -coherence/cohesion scoring (coherence/cohesion breaks)

-scores independently assigned first, then discussed and agreed

-weighted kappa: Chr-0.69, Chs-0.83

RESEARCH QUESTION 1

Are there effects of RSS on summaries measured by

Coh-Metrix indices of coherence and cohesion?

a) Referential cohesion (CRF)?b) Semantic similarity (LSA)?c) Text connectives (CNC)?d) Situation model (SM)?



Coh-Metrix indices

CRF referential cohesion

LSA semantic similarity

CNC connectives

SM situation model-related indices

Source: http://cohmetrix.com/

PRE-TEST POST-TEST

COH-METRIX C E E-C C E E-C

CRFAO1 0,261 0,263 0,002 0,303 0,360 0,056*

CRFAOa 0,261 0,263 0,002 0,303 0,360 0,056*

CRFCWO1 0,048 0,047 <-0,001 0,049 0,064 0,015**

CRFCWOa 0,037 0,037 <0,001 0,037 0,044 0,007**

CRFANP1 0,149 0,157 0,008 0,249 0,242 -0,007

CRFANPa 0,032 0,029 -0,003 0,053 0,061 0,008

LSASS1 0,170 0,173 0,004 0,130 0,145 0,015*

LSASSp 0,155 0,158 0,003 0,103 0,137 0,035****

LSAGN 0,274 0,278 0,004 0,248 0,255 0,007

CNCAll 77,798 78,186 0,388 88,021 96,630 8,606**

CNCCaus 25,183 24,800 -0,383 20,194 25,710 5,516**

CNCLogic 28,642 24,869 -3,773* 32,549 39,034 6,485*

CNCADC 5,011 5,643 0,631 12,634 12,033 0,602

CNCTemp 16,375 16,181 -0,195 14,494 17,048 2,554

CNCTempx 20,066 18,830 -1,236 10,933 11,320 0,387

CNCAdd 42,592 43,350 0,758 56,598 62,376 5,778*

CNCPos 73,727 74,194 0,467 79,736 89,680 9,944**

CNCNeg 4,472 4,945 0,473 9,053 8,795 -0,259

SMINTEp 21,177 18,655 -2,523 23,986 17,896 -6,090***

SMCAUSr 0,309 0,286 -0,023 0,335 0,514 0,179**

SMINTEr 0,840 0,956 0,116 0,627 0,958 0,330***

SMCAUSlsa 0,095 0,090 -0,004 0,086 0,114 0,028****

SMTEMP 0,679 0,665 -0,014 0,670 0,728 0,058**

RQ1: RESULTS

• effects of RSS in 16/23 features

• E summaries: more cohesion

CRF more referential cohesion

LSA more semantic similarity

CNC more connectives

SM more relatedness

Welch’s t-test

Boldface: statistical significance *p<0,05, **p<0,01, ***p<0,001, ****p<0,0001

RQ1: RESULTSMore cohesion devices found in E summaries

CRF

• more overalapping arguments locally and globally

• more overlapping content words locally and globally

LSA

• more semantically similar sentences locally and globally

CNC

• more connectives (all, causal andlogical operators)

SM

• fewer events/actions, more explicitrelations

• more semantically overlapping verbs

• more tense/aspect repetitions

RESEARCH QUESTION 2

Are there RSS effects on the computationally analysed rhetorical structure of student summaries in comparison with expert-written summaries (Šnajder, Sladoljev-Agejev & Kolić-Vehovec, 2019)?



Rhetorical structure may be linked to coherencevan den Broek et al., 1995; Moore & Wiemer-Hastings, 2003; Zwaan & Singer, 2003

COHERENCE

IDEAS

concepts/propositions

RELATIONSe.g. causality, listing,

comparison, problem-solution

Computational analysis of rhetorical structure (RS) Šnajder, Sladoljev-Agejev & Kolić-Vehovec (2019)

COHERENCE

Arg 1 – R – Arg2

Comparing rhetorical structuresŠnajder, Sladoljev-Agejev & Kolić-Vehovec (2019)

RHETORICAL STRUCTURE OVERLAPOVERLAPPING RELATIONS + OVERLAPPING ARGUMENTS

discourse parsing + semantic similarity measures

student summary expert-written summary

AUTOMATED SUMMARY SCORINGŠnajder, Sladoljev-Agejev & Kolić-Vehovec (2019)

1. DISCOURSE PARSING (Prasad et al., 2008; Lin et al., 2014)

2. COMPARING RHETORICAL STRUCTURES (SS vs REFS)

a) Equivalent rhetorical relations? If yes, then b) and c)

b) Argument similarity between pairs of rhetorical relations(Mikolov et al., 2013)

c) Total overlap scores between pairs of summaries (Kuhn, 1955)

RQ2: RESULTS

Rhetorical structure of E summaries - higher overlap with expert summaries

Average computational scores of RS overlap

PRETEST Precision Recall F-measure

C (N=55) 0.270 ± 0.086 0.284 ± 0.107 0.255 ± 0.058

E (N=58) 0.270 ± 0.076 0.263 ± 0.108 0.243 ± 0.057

E - C <0.0004 -0.021 -0.012

POSTTEST Precision Recall F-measure

C (N=55) 0.378 ± 0.064 0.234 ± 0.114 0.270 ± 0.101

E (N=58) 0.368 ± 0.056 0.331 ± 0.055 0.343 ± 0.041

E - C -0.010 +0.097**** +0.072****

CONCLUSION

Effects of teaching RSS can be detected automatically.

The following RSS effects are revealed:

• more cohesion devices (e.g. more overlapping words, more connectives, fewer actions/events, more semantically similar sentences, ...)

• higher overlap with the rhetorical structure of expert-writtensummaries



REFERENCES

Crossley, S. A. & McNamara, D. S. (2009). Computational assessment of lexical differences in L1 and L2 writing. Journal of Second Language Writing, 18(2), 119-135.

Crossley, S. A. & McNamara, D. S. (2010). Cohesion, coherence, and expert evaluations of writing proficiency. U S. Ohlsson & R. Catrambone (eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society, 984–989. Austin, TX: Cognitive Science Society.

Crossley, S. A. & McNamara, D. S. (2011). Understanding expert ratings of essay quality: Coh-Metrix analyses of first and second language writing. International Journal of Continuing Engineering Education and Life Long Learning, 21(2-3), 170-191

Delaney, Y. A. (2008). Investigating the reading-to-write construct. Journal of English for Academic Purposes, 7(3), 140-150.

Foltz, P. W. (2016). Advances in Automated Scoring of Writing for Performance Assessment. In Handbook of Research on Technology Tools for Real-World Skill Development, 659-678.

Graesser, A. C., McNamara, D. S., Louwerse, M. M. & Cai, Z. (2004). Coh-Metrix: Analysis of text on cohe-sion and language. Behavior Research Methods, Instruments & Computers, 36(2), 193-202.



Grosz, B. J. & Sidner, C. L. (1986). Attention, intentions, and the structure of discourse. Computational Linguistics, 12(3), 175-204.

Hobbs, J.R. (1993). Intention, information, and structure in discourse: A first draft. In Burning Issues in Discourse, NATO Advanced Research Workshop, 41-66.

Jiang, X. (2012). Effects of discourse structure graphic organizers of EFL reading comprehension. Reading in a Foreign Language, 24(1), 84.

Kintsch, W., & Van Dijk, T. A. (1978). Toward a model of text comprehension and production. Psychological Review, 85(5), 363.

Kirkland, M. R. & Saunders, M. A. P. (1991). Maximizing student performance in summary writing: Managing cognitive load. Tesol Quarterly, 25(1), 105-121.

Kuhn, H.W. (1955). The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1-2):83–97.


Winterthur, Switzerland,, 5-6 September 2019

Lin, Z., Ng, H. T. & Kan, M. Y. (2014). A PDTB-styled end-to-end discourse parser. Natural Language Engineering, 20(2), 151-184.

Lorch Jr., R. F. (2001). Psychology of macrostructure in discourse comprehension. In N. J. Smelser and P. B. Baltes (eds.) International Encyclopedia of the Social & Behavioral Sciences, 9122–9125. Oxford: Pergamon.

Louwerse, M. M. & Graesser, A. C.,(2005). Macrostructure. In K. Brown (ed.), Encyclopedia of Language and Linguistics, 7, 2005.

McNamara, D. & Graesser, A. (2012). Coh-Metrix: An automated tool for theoreticaland applied natural language processing. In P. M. McCarthy and C. Boonthum(eds.), Applied Natural Language Processing and Content Analysis: Identification, Investigation, and Resolution,188-205. Hershey, PA: IGI Global.

Meyer, B. J. & Ray, M. N. (2011). Structure strategy interventions: Increasingreading comprehension of ex-pository text. International Electronic Journal ofElementary Education, 4(1), 127.

Mikolov,T., Sutskever, I., Chen, K., Corrado, G.S. & Dean, J. (2013). Distributedrepresentations of words and phrases and their compositionality. In Advances inNeural Information Processing Systems, 3111–3119.



Moore, J. D., & Wiemer-Hastings, P. (2003). Discourse in computational linguistics andartificial intelligence. In Handbook of Discourse Processes, 438-484. Routledge.

Prasad, R., Dinesh, N., Lee, A., Miltsakaki,E., Robaldo, L., Joshi, A. K., Webber, B.L. (2008). The Penn Discourse Treebank.2.0. In LREC. Citeseer.

Sabatini, J., O'Reilly, T. & Deane, P. (2013). Preliminary reading literacy assessmentframework: Foundation and rationale for assessment and system design. ETS Research Report Series.

Sladoljev-Agejev, T. & Šnajder, J. (2017). Using Analytical Scoring Rubrics in the Automatic Assessment of College-Level Summary Writing Tasks in L2. Proceedings of the 18th International Joint Conference on Natural Language Processing - IJCNLP 2017, Taipei, Taiwan (Volume 2: Short papers), 181-186.

Šnajder, J., Sladoljev-Agejev, T. & Kolić-Vehovec, S. (2019). Analysing Rhetorical Structure as a Key Feature of Summary Coherence. Proceedings of the 14th Workshop on InnovativeUse of NLP for Building Educational Applications – BEA 2019, Firenze, Italy, 46-51.

Van den Broek, P., Risden, K. & Husebye-Hartmann, E. (1995). The role of readers’ standards for coherence in the generation of inferences during reading. In Sources ofCoherence in Reading, 353–373. Hills-dale, NJ: Lawrence Erlbaum.

Weigle, S. C. (2002). Assessing Writing. Stuttgart: Ernst Klett Sprachen.

Zwaan, R. A., & Singer, M. (2003). Text comprehension. In Handbook of Discourse Processes89-127. Routledge.

THE EIGHTH INTERNATIONAL CONFERENCE ON WRITING ANALYTICS https://writinganalytics.zhaw.ch


Documents

COMPUTATIONAL ANALYSES OF THE EFFECTS OF A …...COMPUTATIONAL ANALYSES OF THE EFFECTS OF A STRUCTURE STRATEGY ON COLLEGE-LEVEL SUMMARIES: COHESION AND RHETORICAL STRUCTURE, Winterthur