1
RESEARCH POSTER PRESENTATION DESIGN © 2012 www.PosterPresentations.com This study compared the use of cohesion devices and lexical knowledge in L1 and L2 blog entries, using the Coh-Metrix tool (http://tool.cohmetrix.com). The present study is based on the research described in Crossley and McNamara (2009) who studied the lexical differences in L1 and L2 writing by using Coh-Metrix. The topic of automated measures of L2 writing has attracted considerable interests recently, due to potentially better efficiencies and higher effectiveness of computational measures, compared to human raters. In the present study, the Coh-Metrix tool (http://tool.cohmetrix.com) is used to automatically evaluate lexical knowledge and the use of cohesive devices in the L2 (English) writing of an Indonesian writer, and to compare these measures with those generated for L1 writers. In my study, written entries from an L2 writer’s (http://tantrialmira.wordpress.com/) and an L1 writer’s (http://jeremyharmer.wordpress.com/) blogs are analysed for indices of lexical knowledge and cohesion, and the results are compared. Introduction Research Question: Design & Methodology Results Implications Limitations A case study subjects of the study one L1 writer and one L2 writer Sample texts: relatively low in number, raising the question of generalizability Future studies using Coh-Metrix may benefit from using: a) large corpus b) not only cohesion and lexical indices References Crossley, S.A. & McNamara, D. S. (2009). Computational assessment of lexical differences in L1 and L2 writing. Journal of Second Language Writing, 18, 119–135 Crossley, S. A., & McNamara, D. S., (2010). Predicting second language writing proficiency: the roles of cohesion and linguistic sophistication. Journal of Research in Reading, 35 (2), 1–21. DOI: 10.1111/j.1467-9817.2010.01449.x Crossley, S. A., Salsbury, T., McNamara, D. S., & Jarvis, S. (2010). Predicting lexical proficiency in language learner texts using computational indices. Language Testing, 20 (10), 1–20. doi:10.1177/0265532210378031 Graesser, A. C. & McNamara, D. S. (2011). Computational analyses of multilevel discourse comprehension. Topics in Cognitive Science, 3, 371-398 Kyle, K. (2011). Objective measures of writing quality. Master Thesis. Colorado State University, Fort Collins: USA McNamara, D. S., Crossley, S.A., & McCarthy, (2010). Linguistic features of writing quality. Written Communication, 27 (1), 57-86. doi: 10.1177/0741088309351547 McNamara, D. S., Louwerse, M. M., McCarthy, P.M., & Graesser, A. C. (2010). Coh-Metrix: Capturing linguistic features of cohesion. Discourse Processes, 47 (4), 292-330 O’Reilly, T., & McNamara, D. S. (2007). Reversing the reverse cohesion effect: Good texts can be better for strategic, high-knowledge readers. Discourse Processes, 43 (2), 121-152 http://jeremyharmer.wordpress.com/ http://tantrialmira.wordpress.com/ Can indices of cohesion and lexical use generated by Coh-Metrix be used to distinguish between L1 and L2 blog writers? School of Linguistics and Applied Language Studies, Victoria University of Wellington Anik Wulyani, PhD candidate Automatic measures of cohesion and lexical proficiency in L2 writing: A case study Indices Expected Indices # 29: Argument overlap L1 > L2 Indices # 46: Latent Semantic Analysis (LSA) give/new L1 < L2 Indices # 85: Verb incidence L1 > L2 Indices # 94: CELEX word frequency for content words, mean L1 < L2 Indices # 97: Age of acquisition L1 < L2 Indices # 102: Polysemy L1 > L2 Indices # 103: Hypernymy for nouns L1 > L2 Indices Expected Indices # 48: Lexical diversity, Type-Token Ratio, content word lemmas L1 > L2 Indices # 49: Lexical Diversity, Type-Token Ratio, all words L1 > L2 Indices # 50: Lexical Diversity, MTLD, all words L1 > L2 Indices # 51: Lexical Diversity, VOCD, all words L1 > L2 Indices # 95: CELEX Log frequency for all words, mean L1 < L2 Indices # 99: Concreteness for content words L1<L2 Indices # 104: Hypernymy for verbs L1 > L2 Indices # 105: Hypernymy for noun and verbs L1 > L2 1. Practicality Coh-Metrix is able to distinguish L1 and L2 texts using linguistics features (cohesion and lexical indices) 2. Assessment Coh-Metrix is a powerful tool to measure L2 writing L 1 writer: 13 blog posts L2 writer: 13 blog posts Cohesion indices: 5 indices Lexical indices: 10 indices Cohesion indices: 5 indices Lexical indices: 10 indices Coh-Metrix Tool 3.0 compared MANOVA Are they significantly different? Can you see the difference? Automatic measures of cohesion and lexical proficiency in L2 writing: A case study Blogging among in-service teachers Main research EFL teachers in Indonesia Pilot study Coh-Metrix Tool 3.0 One L1 blog writer, native speaker of English, the UK One L2 blog writer, EFL teacher, Indonesia Those 15 indices from Coh-Metrix were chosen because previous related studies showed that they were able to differentiate L1 writing to L2 writing (Crossley & McNamara, 2009; Crossley & McNamara, 2010; Kyle, 2011; McNamara, Crossley & McCarthy, 2010; & Crossley, Salsbury, McNamara, & Jarvis, 2010). Today I guide my students to continue design their summative task and formative tasks. I ask my students to list down possible things they can do for their summative and formative tasks. I asked them to list down as many action as possible (actions which they have done in their previous summative or their lies). Then they need to classify the actions which can be feasibly done for their summative tasks. In my blog I have tried, so far, to address general issues to do with presenting, conferences, writing abstracts etc etc. As with every other blogger, my ‘thoughts’ have been personal, of course, but I have tried to exercise some dispassion. But not this time. Just for once I want to tell you how I feel – or rather what it felt like (and then see if there is anything to learn from that). What I am trying to say is that this post is going to be incredibly personal, and I hope you will forgive me for that. Sample 1 Blog post Sample 2 Blog post Future Research Future research English Language Teaching in second language context L2 writing proficiency L2 blog writers cohesion and lexical indices Before discussing the findings of the study, please check the samples of the blog posts I have put in the next section! In terms of cohesion and vocabulary use which one do you think is the blog post of an L1 writer? How do you know? What does Coh-Metrix 3.0 and MANOVA with alpha level .05 tell us about the cohesion and lexical use? Current study versus Crossley & McNamara (2009) Indices Current study Crossley & McNamara (2009) Argument overlap, #29 L1 < L2 L1 > L2 LSA given/new, #46 L1 < L2 L1 < L2 Verb Incidence, #85 L1 < L2 L1 > L2 Lexical diversity , #50 L1 > L2 L1 > L2 Lexical diversity, #51 L1 > L2 L1 > L2 Hypernymy for nouns & verbs, #105 L1> L2 L1 > L2 Crossley & McNamara (2009) Present study Seven variables (indices) Fifteen variables (indices) Corpus of L1 (undergraduates in USA) & of L2 (the International Corpus of Learner English) Blog posts of L1 (a teacher and trainer in English Language Teaching) & of L2 (an Indonesian English teacher) Academic writing/argumentative essays Blog writing Table 2b Expected Results, Additional Indices Table 1 Differences between Crossley & McNamara’s study & present study Table 2a Expected Results (Crossley & McNamara, 2009) 1. Three cohesion indices showed significant differences: argument overlap (#29), LSA given/new (#46), and verb incidence (#85) 2. Three lexical indices reached statistical differences: lexical diversity #50, #51, and hypernymy for nouns and verbs (#105) 3. Cohesion: a) The L1 writer appeared to produce more cohesive texts than the L2 writer b) The L2 writer was more likely to produce texts with more new information than the L1 writer. c) The L2 writer seemed to use more verbs or more spatial cohesion than the L1 writer 4. Lexical Proficiency: a) The L1 writer seemed to have more diverse lexicon than the L2 writer b) The L2 writer appeared to use less diverse vocabulary than the L1 writer c) The L1 writer used significantly more conceptually abstract and hierarchically connected words than the L2 writer Why are the cohesion indices of the present study different from Crossley & McNamara (2009)? Because: a) The L1 writer appeared to be more casual in organizing his texts (few argument overlap) and to expect his readers to have high knowledge to understand his texts (few verb incidences) b) The L2 writer seemed to have a formal writing or text organization (greater argument overlap) and to use more cohesion devices (more verb incidences) to assist her readers to better understand her texts Graesser and McNamara, 2011; McNamara, Louwerse, McCarthy, & Graesser, 2010; & O’Reilly & McNamara, 2007 Email address: [email protected]

Automatic measures of cohesion and lexical proficiency

Embed Size (px)

Citation preview

Page 1: Automatic measures of cohesion and lexical proficiency

RESEARCH POSTER PRESENTATION DESIGN © 2012

www.PosterPresentations.com

(—THIS SIDEBAR DOES NOT PRINT—)

D E S I G N G U I D E

This PowerPoint 2007 template produces a 36”x48”

presentation poster. You can use it to create your research

poster and save valuable time placing titles, subtitles, text,

and graphics.

We provide a series of online tutorials that will guide you

through the poster design process and answer your poster

production questions. To view our template tutorials, go online

to PosterPresentations.com and click on HELP DESK.

When you are ready to print your poster, go online to

PosterPresentations.com

Need assistance? Call us at 1.510.649.3001

Q U I C K S TA RT

Zoom in and out As you work on your poster zoom in and out to the level that

is more comfortable to you.

Go to VIEW > ZOOM.

Title, Authors, and Affiliations Start designing your poster by adding the title, the names of the authors,

and the affiliated institutions. You can type or paste text into the

provided boxes. The template will automatically adjust the size of your

text to fit the title box. You can manually override this feature and

change the size of your text.

TIP: The font size of your title should be bigger than your name(s) and

institution name(s).

Adding Logos / Seals Most often, logos are added on each side of the title. You can insert a logo

by dragging and dropping it from your desktop, copy and paste or by going

to INSERT > PICTURES. Logos taken from web sites are likely to be low

quality when printed. Zoom it at 100% to see what the logo will look like

on the final poster and make any necessary adjustments.

TIP: See if your school’s logo is available on our free poster templates

page.

Photographs / Graphics You can add images by dragging and dropping from your desktop, copy and

paste, or by going to INSERT > PICTURES. Resize images proportionally by

holding down the SHIFT key and dragging one of the corner handles. For a

professional-looking poster, do not distort your images by enlarging them

disproportionally.

Image Quality Check Zoom in and look at your images at 100% magnification. If they look good

they will print well.

ORIGINAL DISTORTED Corner handles

Go

od

pri

nti

ng

qu

alit

y

Bad

pri

nti

ng

qu

alit

y

Q U I C K S TA RT ( c on t . )

How to change the template color theme You can easily change the color theme of your poster by going to the

DESIGN menu, click on COLORS, and choose the color theme of your

choice. You can also create your own color theme.

You can also manually change the color of your background by going to

VIEW > SLIDE MASTER. After you finish working on the master be sure to

go to VIEW > NORMAL to continue working on your poster.

How to add Text The template comes with a number of pre-

formatted placeholders for headers and text

blocks. You can add more blocks by copying and

pasting the existing ones or by adding a text box

from the HOME menu.

Text size Adjust the size of your text based on how much content you have to

present. The default template text offers a good starting point. Follow the

conference requirements.

How to add Tables To add a table from scratch go to the INSERT menu and

click on TABLE. A drop-down box will help you select rows and

columns.

You can also copy and a paste a table from Word or another PowerPoint

document. A pasted table may need to be re-formatted by RIGHT-CLICK >

FORMAT SHAPE, TEXT BOX, Margins.

Graphs / Charts You can simply copy and paste charts and graphs from Excel or Word.

Some reformatting may be required depending on how the original

document has been created.

How to change the column configuration RIGHT-CLICK on the poster background and select LAYOUT to see the

column options available for this template. The poster columns can also

be customized on the Master. VIEW > MASTER.

How to remove the info bars If you are working in PowerPoint for Windows and have finished your

poster, save as PDF and the bars will not be included. You can also delete

them by going to VIEW > MASTER. On the Mac adjust the Page-Setup to

match the Page-Setup in PowerPoint before you create a PDF. You can also

delete them from the Slide Master.

Save your work Save your template as a PowerPoint document. For printing, save as

PowerPoint of “Print-quality” PDF.

Print your poster When you are ready to have your poster printed go online to

PosterPresentations.com and click on the “Order Your Poster” button.

Choose the poster type the best suits your needs and submit your order. If

you submit a PowerPoint document you will be receiving a PDF proof for

your approval prior to printing. If your order is placed and paid for before

noon, Pacific, Monday through Friday, your order will ship out that same

day. Next day, Second day, Third day, and Free Ground services are

offered. Go to PosterPresentations.com for more information.

Student discounts are available on our Facebook page.

Go to PosterPresentations.com and click on the FB icon.

© 2013 PosterPresentations.com 2117 Fourth Street , Unit C Berkeley CA 94710

[email protected]

This study compared the use of cohesion devices and lexical knowledge

in L1 and L2 blog entries, using the Coh-Metrix tool

(http://tool.cohmetrix.com). The present study is based on the

research described in Crossley and McNamara (2009) who studied the

lexical differences in L1 and L2 writing by using Coh-Metrix.

The topic of automated measures of L2 writing has attracted

considerable interests recently, due to potentially better efficiencies

and higher effectiveness of computational measures, compared to

human raters.

In the present study, the Coh-Metrix tool

(http://tool.cohmetrix.com) is used to automatically evaluate lexical

knowledge and the use of cohesive devices in the L2 (English) writing

of an Indonesian writer, and to compare these measures with those

generated for L1 writers.

In my study, written entries from an L2 writer’s

(http://tantrialmira.wordpress.com/)

and an L1 writer’s (http://jeremyharmer.wordpress.com/) blogs

are analysed for indices of lexical knowledge and cohesion, and the

results are compared.

Introduction

Research Question:

Design & Methodology

Results

Implications

Limitations

• A case study subjects of the study one L1 writer and one L2 writer

• Sample texts: relatively low in number, raising the question of generalizability

• Future studies using Coh-Metrix may benefit from using:

a) large corpus

b) not only cohesion and lexical indices

References

Crossley, S.A. & McNamara, D. S. (2009). Computational assessment of lexical differences in L1 and L2 writing. Journal of Second Language Writing, 18, 119–135

Crossley, S. A., & McNamara, D. S., (2010). Predicting second language writing proficiency: the roles of cohesion and linguistic sophistication. Journal of Research in Reading, 35 (2), 1–21. DOI: 10.1111/j.1467-9817.2010.01449.x

Crossley, S. A., Salsbury, T., McNamara, D. S., & Jarvis, S. (2010). Predicting lexical proficiency in language learner texts using computational indices. Language Testing, 20 (10), 1–20. doi:10.1177/0265532210378031

Graesser, A. C. & McNamara, D. S. (2011). Computational analyses of multilevel discourse comprehension. Topics in Cognitive Science, 3, 371-398

Kyle, K. (2011). Objective measures of writing quality. Master Thesis. Colorado State University, Fort Collins: USA

McNamara, D. S., Crossley, S.A., & McCarthy, (2010). Linguistic features of writing quality. Written Communication, 27 (1), 57-86. doi: 10.1177/0741088309351547

McNamara, D. S., Louwerse, M. M., McCarthy, P.M., & Graesser, A. C. (2010). Coh-Metrix: Capturing linguistic features of cohesion. Discourse Processes, 47 (4), 292-330

O’Reilly, T., & McNamara, D. S. (2007). Reversing the reverse cohesion effect: Good texts can be better for strategic, high-knowledge readers. Discourse Processes, 43 (2), 121-152

http://jeremyharmer.wordpress.com/

http://tantrialmira.wordpress.com/

Can indices of cohesion and lexical use generated by

Coh-Metrix be used to distinguish between L1 and L2

blog writers?

School of Linguistics and Applied Language Studies, Victoria University of Wellington

Anik Wulyani, PhD candidate

Automatic measures of cohesion and lexical proficiency in L2 writing: A case study

Indices Expected

Indices # 29: Argument overlap L1 > L2

Indices # 46: Latent Semantic Analysis (LSA) give/new L1 < L2

Indices # 85: Verb incidence L1 > L2

Indices # 94: CELEX word frequency for content words, mean L1 < L2

Indices # 97: Age of acquisition L1 < L2

Indices # 102: Polysemy L1 > L2

Indices # 103: Hypernymy for nouns L1 > L2

Indices Expected

Indices # 48: Lexical diversity, Type-Token Ratio, content word

lemmas

L1 > L2

Indices # 49: Lexical Diversity, Type-Token Ratio, all words L1 > L2

Indices # 50: Lexical Diversity, MTLD, all words L1 > L2

Indices # 51: Lexical Diversity, VOCD, all words L1 > L2

Indices # 95: CELEX Log frequency for all words, mean L1 < L2

Indices # 99: Concreteness for content words L1<L2

Indices # 104: Hypernymy for verbs L1 > L2

Indices # 105: Hypernymy for noun and verbs L1 > L2

1. Practicality Coh-Metrix is able to distinguish L1 and L2 texts using linguistics features (cohesion and lexical indices) 2. Assessment Coh-Metrix is a powerful tool to measure L2 writing

L 1 writer: 13 blog posts

L2 writer: 13 blog posts

Cohesion indices: 5 indices

Lexical indices: 10 indices

Cohesion indices: 5 indices

Lexical indices: 10 indices

Co

h-M

etrix Too

l 3.0

compared MANOVA

Are they significantly different?

Can you see the difference?

Automatic measures of cohesion and lexical proficiency in L2 writing: A case

study

Blogging among in-service teachers • Main research • EFL teachers in Indonesia

• Pilot study • Coh-Metrix Tool 3.0 • One L1 blog writer, native

speaker of English, the UK • One L2 blog writer, EFL

teacher, Indonesia

Those 15 indices from Coh-Metrix were chosen because previous related studies showed that they were able to differentiate L1 writing to L2 writing (Crossley & McNamara, 2009; Crossley & McNamara, 2010; Kyle, 2011; McNamara, Crossley & McCarthy, 2010; & Crossley, Salsbury, McNamara, & Jarvis, 2010).

Today I guide my students to continue design their summative task and formative tasks. I ask my students to list down possible things they can do for their summative and formative tasks. I asked them to list down as many action as possible (actions which they have done in their previous summative or their lies). Then they need to classify the actions which can be feasibly done for their summative tasks.

In my blog I have tried, so far, to address general issues to do with presenting, conferences, writing abstracts etc etc. As with every other blogger, my ‘thoughts’ have been personal, of course, but I have tried to exercise some dispassion. But not this time. Just for once I want to tell you how I feel – or rather what it felt like (and then see if there is anything to learn from that). What I am trying to say is that this post is going to be incredibly personal, and I hope you will forgive me for that.

Sample 1 Blog post

Sample 2 Blog post

Future Research

Future research English Language Teaching in second language context L2 writing proficiency L2 blog writers cohesion and lexical indices

Before discussing the findings of the study, please check the samples of the blog posts I have put in the next section!

In terms of cohesion and vocabulary use which one do you think is the blog post of an L1 writer? How do you know?

What does Coh-Metrix 3.0 and MANOVA with alpha level .05 tell us about the cohesion and lexical use?

Current study versus Crossley & McNamara (2009)

Indices Current study

Crossley & McNamara (2009)

Argument overlap, #29 L1 < L2 L1 > L2

LSA given/new, #46 L1 < L2 L1 < L2

Verb Incidence, #85 L1 < L2 L1 > L2

Lexical diversity, #50 L1 > L2 L1 > L2

Lexical diversity, #51 L1 > L2 L1 > L2

Hypernymy for nouns & verbs, #105

L1> L2 L1 > L2

Crossley & McNamara (2009) Present study

Seven variables (indices) Fifteen variables (indices)

Corpus of L1 (undergraduates in USA) & of L2 (the International Corpus of Learner English)

Blog posts of L1 (a teacher and trainer in English Language Teaching) & of L2 (an Indonesian English teacher)

Academic writing/argumentative essays

Blog writing

Table 2b Expected Results, Additional Indices

Table 1 Differences between Crossley & McNamara’s study & present study

Table 2a Expected Results (Crossley & McNamara, 2009)

1. Three cohesion indices showed significant differences: argument overlap (#29), LSA given/new (#46), and verb incidence (#85)

2. Three lexical indices reached statistical differences: lexical diversity #50, #51, and hypernymy for nouns and verbs (#105)

3. Cohesion: a) The L1 writer appeared to produce more cohesive texts than the L2

writer b) The L2 writer was more likely to produce texts with more new

information than the L1 writer. c) The L2 writer seemed to use more verbs or more spatial cohesion

than the L1 writer 4. Lexical Proficiency: a) The L1 writer seemed to have more diverse lexicon than the L2

writer b) The L2 writer appeared to use less diverse vocabulary than the L1

writer c) The L1 writer used significantly more conceptually abstract and

hierarchically connected words than the L2 writer

Why are the cohesion indices of the present study different from Crossley & McNamara (2009)? Because: a) The L1 writer appeared to be more casual in organizing his texts (few

argument overlap) and to expect his readers to have high knowledge to understand his texts (few verb incidences)

b) The L2 writer seemed to have a formal writing or text organization (greater argument overlap) and to use more cohesion devices (more verb incidences) to assist her readers to better understand her texts

Graesser and McNamara, 2011; McNamara, Louwerse, McCarthy, & Graesser, 2010; & O’Reilly & McNamara, 2007

Email address: [email protected]