Exploring the impact of using automated writing evaluation in English as a foreign language university students' writing

This article was downloaded by: [Southern Taiwan University of Science and Technology]On: 16 December 2014, At: 20:23Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Computer Assisted Language LearningPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/ncal20

Exploring the impact of usingautomated writing evaluation inEnglish as a foreign language universitystudents' writingYing-Jian Wang a , Hui-Fang Shang a & Paul Briody aa Department of Applied English , I-Shou University , KaohsiungCity , Taiwan , R.O.CPublished online: 23 Mar 2012.

To cite this article: Ying-Jian Wang , Hui-Fang Shang & Paul Briody (2013) Exploring the impact ofusing automated writing evaluation in English as a foreign language university students' writing,Computer Assisted Language Learning, 26:3, 234-257, DOI: 10.1080/09588221.2012.655300

To link to this article: http://dx.doi.org/10.1080/09588221.2012.655300

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the“Content”) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoever as tothe accuracy, completeness, or suitability for any purpose of the Content. Any opinionsand views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Contentshould not be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilities whatsoeveror howsoever caused arising directly or indirectly in connection with, in relation to orarising out of the use of the Content.

This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

http://www.tandfonline.com/loi/ncal20

http://www.tandfonline.com/action/showCitFormats?doi=10.1080/09588221.2012.655300

http://dx.doi.org/10.1080/09588221.2012.655300

http://www.tandfonline.com/page/terms-and-conditions

http://www.tandfonline.com/page/terms-and-conditions

© 2013 Taylor & Francis

Computer Assisted Language Learning, 2013Vol. 26, No. 3, 234–257, http://dx.doi.org/10.1080/09588221.2012.655300

Exploring the impact of using automated writing evaluation in English asa foreign language university students’ writing

Ying-Jian Wang, Hui-Fang Shang* and Paul Briody

Department of Applied English, I-Shou University, Kaohsiung City, Taiwan, R.O.C

A period of expanding globalization has emphasized the significant role played bythe English language in both communications and information sourcing. In suchan environment, it is understandable that many have sought to enhance availablewriting skills and the assessment of said writing skills. In recent decades,automated writing evaluation (AWE) has been applied with significant frequencyto the evaluation and assessment of English writing performance in EFLenvironments. The purpose of this study is to examine the impact and effect ofusing AWE on EFL students’ writing. In this study, the subjects were 57 EFLfreshmen from the Department of Applied English at one university in the southof Taiwan. Both quantitative and qualitative research methods were used toconduct a quasi-experimental research by employing a t-test technique and asemi-structured interview technique, and the nonequivalent groups’ pre-test/post-test control and comparison group design were applied to explore the overalleffect of using AWE on the improvement of student writing in terms of accuracy,learner autonomy, and interaction. The research results reveal a significantdifference between the experimental group and the control group in terms ofwriting accuracy following the adoption of AWE. Regarding the overall effectand the exploration of students’ perceptions toward their usage of the AWEsoftware, it shows that students who used AWE display obvious writingenhancement in terms of writing accuracy and learner autonomy awareness.The pedagogical implications of fully understanding students’ usage of AWE inthe writing process in order to improve EFL learners’ writing performance arefully discussed.

Keywords: automated writing evaluation (AWE); accuracy; learner autonomy;interaction; EFL writing

Introduction

The expansion of globalization has led to an observed increase in significance for therole of written English. It has been a major source of information for a multitude offields, both popular and academic. It has also become much more important in therole of global communication. As a matter of fact, writing is an indispensable part inthe mastery of a language (Li, 2005). Writing is an important tool in evaluationwhere English proficiency needs to be assessed, but it also allows measurements ofmany other skills and subjects (Rezaei & Lovorn, 2010). Therefore, how to enhance

*Corresponding author. Email: [email protected]

Dow

nloa

ded

by [

Sout

hern

Tai

wan

Uni

vers

ity o

f Sc

ienc

e an

d T

echn

olog

y] a

t 20:

23 1

6 D

ecem

ber

2014

Computer Assisted Language Learning 235

Exploring the impact of using automated writing evaluation in English asa foreign language university students’ writing

Ying-Jian Wang, Hui-Fang Shang* and Paul Briody

Department of Applied English, I-Shou University, Kaohsiung City, Taiwan, R.O.C

A period of expanding globalization has emphasized the significant role played bythe English language in both communications and information sourcing. In suchan environment, it is understandable that many have sought to enhance availablewriting skills and the assessment of said writing skills. In recent decades,automated writing evaluation (AWE) has been applied with significant frequencyto the evaluation and assessment of English writing performance in EFLenvironments. The purpose of this study is to examine the impact and effect ofusing AWE on EFL students’ writing. In this study, the subjects were 57 EFLfreshmen from the Department of Applied English at one university in the southof Taiwan. Both quantitative and qualitative research methods were used toconduct a quasi-experimental research by employing a t-test technique and asemi-structured interview technique, and the nonequivalent groups’ pre-test/post-test control and comparison group design were applied to explore the overalleffect of using AWE on the improvement of student writing in terms of accuracy,learner autonomy, and interaction. The research results reveal a significantdifference between the experimental group and the control group in terms ofwriting accuracy following the adoption of AWE. Regarding the overall effectand the exploration of students’ perceptions toward their usage of the AWEsoftware, it shows that students who used AWE display obvious writingenhancement in terms of writing accuracy and learner autonomy awareness.The pedagogical implications of fully understanding students’ usage of AWE inthe writing process in order to improve EFL learners’ writing performance arefully discussed.

Keywords: automated writing evaluation (AWE); accuracy; learner autonomy;interaction; EFL writing

Introduction

The expansion of globalization has led to an observed increase in significance for therole of written English. It has been a major source of information for a multitude offields, both popular and academic. It has also become much more important in therole of global communication. As a matter of fact, writing is an indispensable part inthe mastery of a language (Li, 2005). Writing is an important tool in evaluationwhere English proficiency needs to be assessed, but it also allows measurements ofmany other skills and subjects (Rezaei & Lovorn, 2010). Therefore, how to enhance

*Corresponding author. Email: [email protected]

and evaluate the writing ability of Taiwanese EFL learners has gained muchattention. Giving feedback is a key element in the process of evaluating students’English writing performance, and it is also one of the most vital sources ofinformation helping students to reconstruct knowledge, remedy misconception,enhance motivation, and improve academic performance (Foote, 1999; Warden,2000). Giving and receiving feedback has been a main issue in the field of writingbecause the quality of feedback may greatly influence students’ writing (Yang, 2010;Yu, 2006). Traditionally, the authority to provide feedback to students has been inthe hands of teachers. However, it is time-consuming for teachers to correct allstudent errors and to give individualized feedback (Yu, 2006).

With technological advancement and the expansion of internet use, computer-ized feedback, provided by automated writing evaluation (AWE) software, hasexerted an increasing influence on writing instruction (Hirvela, 2005; Huot, 1996;Tuzi, 2004; Warschauer & Ware, 2006). Unlike traditional feedback approachessuch as feedback produced by teachers and peers, AWE can be operatedindependently, that is, by oneself, and it can give students near-instant feedback.By adopting this artificial intelligence (AI) system, an abundance of researchershave asserted that students can achieve significant improvements in their writing inthree dimensions. First of all, AWE can offer students more writing opportunitieswith attendant error feedback and explanations (Kern & Warschauer, 2000;Milton, 2006; Ware & Warschauer, 2006). Secondly, such a system can facilitatestudents’ learner autonomy awareness in order to improve their writing by guidingstudents with different functions and resources, concurrent with immediatefeedback (Dikli, 2006; Ding, 2008; Griffiths & Nicolls, 2010; Hoon, 2006; Li,2007). Thirdly, this technology claims to be more consistent and more objectivethan human raters since students who write more neatly and who demonstrate‘‘better’’ writing mechanics, often get higher scores than students who do not havesuch skills, even though their content may be better (Li, 2005; Rezaei & Lovorn,2010). In contrast, some researchers still doubt the effectiveness of the usage ofAWE (Beatty, 2003; Ding, 2008; Warschauer, 1999). For example, the AWEsoftware is often opposed because it falls short of any meaningful interaction thatmight be encountered in the real world (Ferris & Hedgcock, 2005; Hamp-Lyons,2001; Leki, 1990; Reid, 1994; Ware & Warschauer, 2006). Besides this, results insome studies reveal that AWE can actually have a negative influence on students’writing in terms of accuracy or learner autonomy (Chen & Cheng, 2008; Kepner,1991). In Taiwan, EFL learners have been misusing grammar, usage, style, etc.when writing in English, due to an over-reliance on translating directly fromChinese (Liu, 2002). Under such circumstances, learners display a propensity totranslate the phrases literally and thus exhibit numerous, nonnative expressions(Chang, Chang, Chen, & Liou, 2008). Since it is time-consuming and indeedunrealistic to expect teachers to correct all student errors and to give individualizedfeedback (Yu, 2006), alternative approaches may suggest more appropriatesolutions. The aim of this study is to investigate one such alternative, AWE,and how such a method might be applied in a Taiwanese context despiteacknowledging certain, above-mentioned, drawbacks. To state this more clearly,this research investigates the impact of AWE usage on Taiwanese EFL students’writing development. It is hoped that the automatic detection and correction toolsfor mechanical errors will help learners to acquire increasingly appropriate usageand to transfer this incremental knowledge into the development of their writing.

Dow

nloa

ded

by [

Sout

hern

Tai

wan

Uni

vers

ity o

f Sc

ienc

e an

d T

echn

olog

y] a

t 20:

23 1

6 D

ecem

ber

2014

236 Y.-J. Wang et al.

Literature review

Computerized feedback is regarded as a crucial component of AWE; it greatly affectsstudents’ writing in second language writing studies (Ferris & Roberts, 2001; Huang,2001). Numerous research on AWE has shown both positive and negative effects onstudents’ writing development in terms of accuracy (Dikli, 2006; Kern &Warschauer, 2000; Milton, 2006; Ware & Warschauer, 2006; Yeh, Liou, & Yu,2007), learner autonomy (Chen & Cheng, 2008; Diab, 2010; Dikli, 2006; Griffiths &Nicolls, 2010; Hoon, 2006; Lee, 2004; Li, 2007), and interaction (Ferris & Hedgcock,2005; Hamp-Lyons, 2001; Leki, 1990; Reid, 1994; Ware & Warschauer, 2006). Moredetails are presented in the following.

Accuracy

AWE can enhance students’ writing quality because it can offer an untiring source ofwriting practice opportunities for students by generating individualized errorfeedback along with explanations of grammar, spelling, sentence, and word usage(Kern & Warschauer, 2000; Milton, 2006; Ware & Warschauer, 2006). For instance,Milton (2006) conducted research involving 323 first-year university students lastingfor 28 weeks to examine how Check My Words helped students improve their writingskills. Milton stated that in most cases, the computerized feedback focuses thestudents’ attention on sentence-level errors by helping students to resolve theirinaccurate usage, as well as to identify and to reformulate errors. Besides, in Li’s(2007) observation, AWE possesses not only the features of grammatical and spellingcheck but also the function to guide learners to create correct sentences. By applyingthis software, students can enhance their writing and overcome ‘‘Chin-glish’’ usagebecause they are guided to choose better words and sentence structures provided bytheir computer. A similar conclusion is made by Yeh et al. (2007) and Dikli (2006).Yeh et al. considered that feedback from MY Access is helpful for revision in termsof accuracy. Dikli also mentioned that first, the feature of Writer’s handbook, whichis one of the functions included in Criterion, provides students with an opportunityto view feedback definitions, examples of accurate and inaccurate use, and anexplanation of every error reported. Secondly, the ‘‘Standardized Assessment’’ modeconsisted in IntelliMetric provides an overall score and detailed diagnostic feedbackon grammar, usage, and spelling on the basis of various rhetorical and analyticaldimensions of an essay. What is more, participants involved in Hoon’s (2006)research responded that they felt they had improved their grammar and spelling byusing My Editor, and word meanings can also be improved by using the Thesaurusfunction. The feedback provided by My Tutor had helped them identify their owndefects with explanations which are provided with examples. Therefore, it can beconcluded that AWE is helpful in terms of students’ writing performance, in terms ofaccuracy, by providing immediate error feedback with clear explanations.

However, research designed by Kepner (1991) argued that the contribution ofdifferent types of writing feedback to the development of L2 writing skills yieldssimple error corrections which mainly focus on accuracy and are only effective forstudents with low-verbal ability, and almost of no use to students with high-verbalability. The same conclusion is also confirmed by Chen and Cheng (2008) whoimplemented the AWE software, My Access, in three different ways in three writingclasses with third-year English majors in Taiwan in order to explore the effectivenessof using AWE in improving students’ writing. The results revealed that a number of

Y.-J. Wang

Dow

nloa

ded

by [

Sout

hern

Tai

wan

Uni

vers

ity o

f Sc

ienc

e an

d T

echn

olog

y] a

t 20:

23 1

6 D

ecem

ber

2014


Literature review

Computerized feedback is regarded as a crucial component of AWE; it greatly affectsstudents’ writing in second language writing studies (Ferris & Roberts, 2001; Huang,2001). Numerous research on AWE has shown both positive and negative effects onstudents’ writing development in terms of accuracy (Dikli, 2006; Kern &Warschauer, 2000; Milton, 2006; Ware & Warschauer, 2006; Yeh, Liou, & Yu,2007), learner autonomy (Chen & Cheng, 2008; Diab, 2010; Dikli, 2006; Griffiths &Nicolls, 2010; Hoon, 2006; Lee, 2004; Li, 2007), and interaction (Ferris & Hedgcock,2005; Hamp-Lyons, 2001; Leki, 1990; Reid, 1994; Ware & Warschauer, 2006). Moredetails are presented in the following.

Accuracy

AWE can enhance students’ writing quality because it can offer an untiring source ofwriting practice opportunities for students by generating individualized errorfeedback along with explanations of grammar, spelling, sentence, and word usage(Kern & Warschauer, 2000; Milton, 2006; Ware & Warschauer, 2006). For instance,Milton (2006) conducted research involving 323 first-year university students lastingfor 28 weeks to examine how Check My Words helped students improve their writingskills. Milton stated that in most cases, the computerized feedback focuses thestudents’ attention on sentence-level errors by helping students to resolve theirinaccurate usage, as well as to identify and to reformulate errors. Besides, in Li’s(2007) observation, AWE possesses not only the features of grammatical and spellingcheck but also the function to guide learners to create correct sentences. By applyingthis software, students can enhance their writing and overcome ‘‘Chin-glish’’ usagebecause they are guided to choose better words and sentence structures provided bytheir computer. A similar conclusion is made by Yeh et al. (2007) and Dikli (2006).Yeh et al. considered that feedback from MY Access is helpful for revision in termsof accuracy. Dikli also mentioned that first, the feature of Writer’s handbook, whichis one of the functions included in Criterion, provides students with an opportunityto view feedback definitions, examples of accurate and inaccurate use, and anexplanation of every error reported. Secondly, the ‘‘Standardized Assessment’’ modeconsisted in IntelliMetric provides an overall score and detailed diagnostic feedbackon grammar, usage, and spelling on the basis of various rhetorical and analyticaldimensions of an essay. What is more, participants involved in Hoon’s (2006)research responded that they felt they had improved their grammar and spelling byusing My Editor, and word meanings can also be improved by using the Thesaurusfunction. The feedback provided by My Tutor had helped them identify their owndefects with explanations which are provided with examples. Therefore, it can beconcluded that AWE is helpful in terms of students’ writing performance, in terms ofaccuracy, by providing immediate error feedback with clear explanations.

However, research designed by Kepner (1991) argued that the contribution ofdifferent types of writing feedback to the development of L2 writing skills yieldssimple error corrections which mainly focus on accuracy and are only effective forstudents with low-verbal ability, and almost of no use to students with high-verbalability. The same conclusion is also confirmed by Chen and Cheng (2008) whoimplemented the AWE software, My Access, in three different ways in three writingclasses with third-year English majors in Taiwan in order to explore the effectivenessof using AWE in improving students’ writing. The results revealed that a number of

participants consider that their writing was not enhanced because this program onlyprovides vague and repeated feedback in the area of mechanical accuracy. Therefore,these participants suggest that this software with machine-controlled rules might bemore suitable for beginners and intermediate learners since those participants wantto write more flexibly and creatively.

Learner autonomy

The argument remains, however, that students’ writing would be enhanced withthe use of AWE, because it can help students learn how to manipulate the wholewriting process by themselves, in terms of the combination of web resources(Griffiths & Nicolls, 2010; Hoon, 2006), guidance of planning (Chandler, 2003;Griffiths & Nicolls, 2010; Hoon, 2006; Li, 2007), and the management ofstudents’ first and subsequent drafts, which can be saved as a portfolio online(Dikli, 2006). In depth analysis of 34 volunteers at university-level in Malaysiawas done by Hoon (2006) for a whole semester. The aim was to explore theeffectiveness of My Access in automated essay assessment and students’ writingdevelopment. The findings from Hoon’s interview and student feedback supportthe proposition that MyAccess helps them organize their ideas and produce abetter essay by showing learners graphically organized writing steps. Griffiths andNicolls (2010) advocate that the blended e-activities from e-Support4U provide auseful scaffold that guides students through the process of academic writing in astructured way which contributes to successful completion. The same assertion issupported by Li (2007), who states that AWE helps students to improve theirwriting proficiency by extending students’ writing ideas. Additionally, in Dikli’s(2006) study, the electronic portfolio feature involved in AWE facilitates students’learner autonomy awareness, because the electronic portfolio allows students tostore their first and subsequent drafts online, giving students the opportunity tosee their weekly progress, as well as allowing them to view a list of completedassignments, scores, reports, comments, etc. Moreover, Lee (2004) conductedresearch with 42 international college students, comparing the difference betweenpaper- and pencil-based writing courses and a computer-based one. The resultsdemonstrate that 60% of participants answered positively to the assertion thattheir writing was greatly improved, and that they were more willing to write onthe computer due to the extended time allowed for planning and the opportunityfor correction and revision.

In addition, AWE can motivate students to produce more and better writingrevisions with an opportunity to learn on their own, by providing learners withimmediate feedback, which is convenient and accessible without the usual limitationsof time and usage frequency (Chen & Cheng, 2008; Denton, 2001, 2003; Denton,Madden, Roberts, & Rowe, 2008; Dikli, 2006; Griffiths & Nicolls, 2010; Lee, 2004;Li, 2005). For example, Grimes and Warschauer (as cited in Chen & Cheng, 2006)inspected howMY Access and Criterion were implemented in US high school writingclasses. It was found that the speed of responsiveness was a strong motivator forstudents to practice writing. Efficient composition skills can be nurtured throughfunctions provided in AWE which saves time, because students can write morequickly and produce a greater number of revisions (Ferris & Hedgcock, 2005). Thesame conclusions were confirmed by Lee (2004), namely that students expressed theirsatisfaction about saving more time during the editing process, thus allowing them to

Dow

nloa

ded

by [

Sout

hern

Tai

wan

Uni

vers

ity o

f Sc

ienc

e an

d T

echn

olog

y] a

t 20:

23 1

6 D

ecem

ber

2014


focus more on content development in computer writing. Likewise, Griffiths andNicolls (2010) observed 17 students from pre-registration nurse education atBuckinghamshire New University, and carried out a four-month experiment byapplying e-Support4U with the aim of extending academic writing support and intothe practice arena. Their findings showed a 100% positive response to the web-basedacademic support via e-Support4U. Students replied that they can learn how toimprove their writing work without time and space limitations because they study bythemselves and as a result of adopting feedback afforded by the e-moderator which isvisible all the time. By using this convenient, accessible, and effective tool, studentsare able to take a step forward in their mastery of the language and theirdevelopment as writers.

In contrast, some researchers found that AWE was not conducive toimprovements in attitudes toward learner autonomy and thus to eventualimprovements in writing (Beatty, 2003; Chen & Cheng, 2008; Diab, 2010; Ding,2008; Warschauer, 1999). For example, Warschauer (1999) found that AWE whichcombines various web sources would impair students’ writing performance because‘‘critics say the Web drowns students in a sea of unreliable information andencourages mindless net-surfing’’ which reduces students’ self-learning motivationdue to its endless nature (p. 157). Besides, Chen and Cheng (2008) claimed that theresults concerned with whether students can develop greater autonomy in theirwriting revision through computerized feedback or in the use of self-help writing andediting tools are uncertain. It has also been found that writers can easily fool orhoodwink AWE if an essay is lengthy or involves certain lexical-grammatical featuresfavored by the AWE scoring systems. Such software bias restricts students’ creativityand ideas development, and encourages the adoption of short-cuts instead.Essentially, this means that students become far more motivated in devising variousmethods by which they might circumvent the veracity of scoring, namely beating thesoftware, rather than making a real effort to learn and improve by themselves. Thesame concern is supported by Beatty (2003) who cautions that AWE users oftenfollow the closed steps and sequences which encourage learners to have ‘‘only limitedopportunities to organize their own learning or tailor it to their special needs,’’ andthus their writing cannot be improved (p. 10). Moreover, Ding (2008) expressed thatmonolingual feedback, which refers to English feedback offered by AWE, can neitherprovide much help to less skillful students nor increase their learner autonomy toimprove their writing, due to the frustration of understanding the nature of suchlimited feedback.

Interaction

A number of researchers support the argument that student writing can be improveddue to the immediate nature of feedback provided by AWE. Such immediacy, it isargued, creates synchronous communication, which refers to simultaneous interac-tion between users and computers in the completion of the writing tasks (Hirvela,2005; Hoon, 2006; Huot, 1996; Li, 2007). The majority of the students involved inthe depth analysis done by Hoon (2006) readily perceived that AWE was an effectivemeans by which to enhance their writing as it is interactive and the feedback is near-instant and helpful. The same assertion is also supported by Li (2007) who said thatAWE contains instancy, interaction, and objectivity. Moreover, Slatin (1990) hasessentially described interactive writing which can enhance student writing as an

Dow

nloa

ded

by [

Sout

hern

Tai

wan

Uni

vers

ity o

f Sc

ienc

e an

d T

echn

olog

y] a

t 20:

23 1

6 D

ecem

ber

2014


focus more on content development in computer writing. Likewise, Griffiths andNicolls (2010) observed 17 students from pre-registration nurse education atBuckinghamshire New University, and carried out a four-month experiment byapplying e-Support4U with the aim of extending academic writing support and intothe practice arena. Their findings showed a 100% positive response to the web-basedacademic support via e-Support4U. Students replied that they can learn how toimprove their writing work without time and space limitations because they study bythemselves and as a result of adopting feedback afforded by the e-moderator which isvisible all the time. By using this convenient, accessible, and effective tool, studentsare able to take a step forward in their mastery of the language and theirdevelopment as writers.

In contrast, some researchers found that AWE was not conducive toimprovements in attitudes toward learner autonomy and thus to eventualimprovements in writing (Beatty, 2003; Chen & Cheng, 2008; Diab, 2010; Ding,2008; Warschauer, 1999). For example, Warschauer (1999) found that AWE whichcombines various web sources would impair students’ writing performance because‘‘critics say the Web drowns students in a sea of unreliable information andencourages mindless net-surfing’’ which reduces students’ self-learning motivationdue to its endless nature (p. 157). Besides, Chen and Cheng (2008) claimed that theresults concerned with whether students can develop greater autonomy in theirwriting revision through computerized feedback or in the use of self-help writing andediting tools are uncertain. It has also been found that writers can easily fool orhoodwink AWE if an essay is lengthy or involves certain lexical-grammatical featuresfavored by the AWE scoring systems. Such software bias restricts students’ creativityand ideas development, and encourages the adoption of short-cuts instead.Essentially, this means that students become far more motivated in devising variousmethods by which they might circumvent the veracity of scoring, namely beating thesoftware, rather than making a real effort to learn and improve by themselves. Thesame concern is supported by Beatty (2003) who cautions that AWE users oftenfollow the closed steps and sequences which encourage learners to have ‘‘only limitedopportunities to organize their own learning or tailor it to their special needs,’’ andthus their writing cannot be improved (p. 10). Moreover, Ding (2008) expressed thatmonolingual feedback, which refers to English feedback offered by AWE, can neitherprovide much help to less skillful students nor increase their learner autonomy toimprove their writing, due to the frustration of understanding the nature of suchlimited feedback.

Interaction

A number of researchers support the argument that student writing can be improveddue to the immediate nature of feedback provided by AWE. Such immediacy, it isargued, creates synchronous communication, which refers to simultaneous interac-tion between users and computers in the completion of the writing tasks (Hirvela,2005; Hoon, 2006; Huot, 1996; Li, 2007). The majority of the students involved inthe depth analysis done by Hoon (2006) readily perceived that AWE was an effectivemeans by which to enhance their writing as it is interactive and the feedback is near-instant and helpful. The same assertion is also supported by Li (2007) who said thatAWE contains instancy, interaction, and objectivity. Moreover, Slatin (1990) hasessentially described interactive writing which can enhance student writing as an

emerging type of L2 writing class where students get synchronous feedback fromspelling and grammar checks.

However, many studies show that in the automated environment, writing isframed as a piece designed to evaluate student mastery of grammar, usage, andorganization, but it has yet to be modeled on real interaction for providingmeaningful communications or negotiations (Ferris & Hedgcock, 2005; Hamp-Lyons, 2001; Leki, 1990; Reid, 1994; Wang & Brown, 2008; Ware & Warschauer,2006). Ferris and Hedgcock (2005) pointed out that the shortcoming of AWEsoftware is its ‘‘lack of meaning negotiation in real-world context’’ (p. 109). Theyquestion the authenticity of such writing, asking whether students really becomestronger writers, thinkers, and communicators by practicing their writing on such asystem, or if they merely apply the same kinds of test-taking strategy skills to theirwriting assignments. Scholars, such as Leki (1990) and Reid (1994), pinpointedthat computers cannot replace the interactions with teachers because novicewriters are in need of instruction, modeling, and practice in various perspectivesupon writing, and these needs are especially salient for ESL writers who needmore guidance in the linguistic and rhetorical forms of written English, muchmore than the native English speakers. Likewise, AWE systems have beencriticized for eliminating the human element in writing assessment (Warschauer &Ware, 2006) and falling short of human interaction as well as any sense of thewriter as an actual person (Hamp-Lyons, 2001). Hence, students are rarelyinspired during the learning process and are thus lacking when it comes tointeracting with others in terms of thinking, gaining ideas, and negotiatingmeanings. Moreover, since such feedback is unable to provide responses involvingrich negotiations of meaning, AWE systems can be misused to reinforce artificial,mechanistic, and formulaic writing which is disconnected from communication inreal-world contexts. The very same concern is expressed by Drechsel (1999) andHerrington and Moran (2001) who believe that students expect to receive anessentially human reaction while writing. If machine scoring is adopted, studentsmight not be able to understand the power of writing because they develop apropensity to feel they are merely writing for the benefit of the machines. Such aperception can lead to students losing their own voice as writers. Therefore,students’ writing proficiency cannot be improved by applying AWE since it lacksthe force and impact of real interactions and meaningful negotiations.

Recent AWE technology: CorrectEnglish

Inspired by the results provided by the previous researchers about various AWEsoftwares, the interest in investigating the effects of using CorrectEnglish hasincreased somewhat. With the recent development of technology provided byVantage Learning, CorrectEnglish claims to help EFL writers improve their criticalwriting and revision skills by checking grammar, style, and proper word usage, andby providing the holistic score and immediate feedback on content, focus,organization, style, and overall performance. This system (http:// www.correcten-glish.com) which consists of AI also gives instruction concerned with providing anoverview of the task, an annotated model which students follow to the completion ofthe final draft. It comes with a checklist for students to view the elements andinformation in their own writing. Likewise, this editing and word-processingprogram provides English grammar instruction in seven languages, and makes

Dow

nloa

ded

by [

Sout

hern

Tai

wan

Uni

vers

ity o

f Sc

ienc

e an

d T

echn

olog

y] a

t 20:

23 1

6 D

ecem

ber

2014


complex publishing formats simpler to follow: American Psychology Association(APA), Modern Language Association (MLA), Council of Biology Editors (CBE),and Chicago Manual of Style. Moreover, this technology blends other resources,such as Lexipedia Word Web and multiple reference volumes, which make writingeasier by saving students time in searching for information.

To operate this electronic technology requires the student to submit their essayfor checking. The essay once uploaded, generates the diagnostic feedback, which willbe reported back to the student immediately. The diagnostic feedback, whichcontains highlighted errors along with a list of explanations, mainly focuses on threeportions: grammar and style, writing and revision, and word choice. The function ofthe grammar and style portion is to provide feedback on highlighted errors as a basisfor assessing word level (see Figure 1) and grammar. In terms of grammar, theemphasis is upon standard grammatical rules (see Figure 2), while style choice pointsout any incorrect expressions in the student’s text (see Figure 3). The function ofwriting and revision offers a holistic score which divides students’ writing proficiencyinto three levels from poor to great in terms of overall performance, organization,focus, content, and style (see Figure 4).

Following the perusal of previous studies, it is discernable that AWE has beendeveloped since the mid-1960s and that there are various discriminations in the usageof AWE software in students’ writing. This discriminated usage applies particularlyto terms of accuracy, learner autonomy, and interaction. It shows both the positiveand negative sides of AWE software. The software is asserted to improve students’writing because of its linguistic error feedback with explanations, the function aimsto help students self-manipulate the whole writing process, and the instant feedbackwhich offers synchronous interaction; however, some researchers doubt itseffectiveness due to limitations in its suitability for skillful students as well asbeginners, and in the creation of realistic meaning negotiations between students and

Figure 1. The basic check interface of CorrectEnglish.

Dow

nloa

ded

by [

Sout

hern

Tai

wan

Uni

vers

ity o

f Sc

ienc

e an

d T

echn

olog

y] a

t 20:

23 1

6 D

ecem

ber

2014


complex publishing formats simpler to follow: American Psychology Association(APA), Modern Language Association (MLA), Council of Biology Editors (CBE),and Chicago Manual of Style. Moreover, this technology blends other resources,such as Lexipedia Word Web and multiple reference volumes, which make writingeasier by saving students time in searching for information.

To operate this electronic technology requires the student to submit their essayfor checking. The essay once uploaded, generates the diagnostic feedback, which willbe reported back to the student immediately. The diagnostic feedback, whichcontains highlighted errors along with a list of explanations, mainly focuses on threeportions: grammar and style, writing and revision, and word choice. The function ofthe grammar and style portion is to provide feedback on highlighted errors as a basisfor assessing word level (see Figure 1) and grammar. In terms of grammar, theemphasis is upon standard grammatical rules (see Figure 2), while style choice pointsout any incorrect expressions in the student’s text (see Figure 3). The function ofwriting and revision offers a holistic score which divides students’ writing proficiencyinto three levels from poor to great in terms of overall performance, organization,focus, content, and style (see Figure 4).

Following the perusal of previous studies, it is discernable that AWE has beendeveloped since the mid-1960s and that there are various discriminations in the usageof AWE software in students’ writing. This discriminated usage applies particularlyto terms of accuracy, learner autonomy, and interaction. It shows both the positiveand negative sides of AWE software. The software is asserted to improve students’writing because of its linguistic error feedback with explanations, the function aimsto help students self-manipulate the whole writing process, and the instant feedbackwhich offers synchronous interaction; however, some researchers doubt itseffectiveness due to limitations in its suitability for skillful students as well asbeginners, and in the creation of realistic meaning negotiations between students and

Figure 1. The basic check interface of CorrectEnglish.

the AWE system. Additionally, with respect to an understanding of AWE software,CorrectEnglish is introduced to address the interests of different AWE systems andthe results of applying them. Inspired by previous reports, the present study adopts

Figure 2. The grammatical usage interface of CorrectEnglish.

Figure 3. The style choice interface of CorrectEnglish.

Dow

nloa

ded

by [

Sout

hern

Tai

wan

Uni

vers

ity o

f Sc

ienc

e an

d T

echn

olog

y] a

t 20:

23 1

6 D

ecem

ber

2014


an AWE system in an EFL writing class in Taiwan, to investigate whether there is asignificant difference between students’ pre-test and post-test scores after theemployment of AWE, and to explore the students’ perceptions toward the usage ofAWE. In this study, four research questions are specifically addressed:

(1) Is there a significant difference in the writing accuracy of the experimentalgroup compared with the control group after the application of AWE?

(2) Is there a significant difference in the writing accuracy before and after theemployment of AWE?

(3) What is the overall self-report of using AWE on the improvement of studentwriting in terms of accuracy, learner autonomy, and interaction?

(4) What are students’ attitudes toward the usage of AWE on their writingdevelopment?

Methodology

Subjects

The subjects in this study were 57 voluntary freshmen, including 21 males and 36females, majoring in the Department of Applied English at one university in thesouth of Taiwan. Freshmen were chosen in the present study for the reason that theyhad no previous experience with automated essay evaluation. A demographicquestionnaire was provided to collect subjects’ background information, includingtheir age, gender, and years of learning English. The results of the demographicinformation show that all of the subjects were ranged in ages from 19 to 20 years old,with an average of 19.15 years old, and they have been receiving English learning for9.58 years. In order to evaluate the effectiveness of employing AWE in EFL

Figure 4. The writing score interface of CorrectEnglish.

Dow

nloa

ded

by [

Sout

hern

Tai

wan

Uni

vers

ity o

f Sc

ienc

e an

d T

echn

olog

y] a

t 20:

23 1

6 D

ecem

ber

2014


an AWE system in an EFL writing class in Taiwan, to investigate whether there is asignificant difference between students’ pre-test and post-test scores after theemployment of AWE, and to explore the students’ perceptions toward the usage ofAWE. In this study, four research questions are specifically addressed:

(1) Is there a significant difference in the writing accuracy of the experimentalgroup compared with the control group after the application of AWE?

(2) Is there a significant difference in the writing accuracy before and after theemployment of AWE?

(3) What is the overall self-report of using AWE on the improvement of studentwriting in terms of accuracy, learner autonomy, and interaction?

(4) What are students’ attitudes toward the usage of AWE on their writingdevelopment?

Methodology

Subjects

The subjects in this study were 57 voluntary freshmen, including 21 males and 36females, majoring in the Department of Applied English at one university in thesouth of Taiwan. Freshmen were chosen in the present study for the reason that theyhad no previous experience with automated essay evaluation. A demographicquestionnaire was provided to collect subjects’ background information, includingtheir age, gender, and years of learning English. The results of the demographicinformation show that all of the subjects were ranged in ages from 19 to 20 years old,with an average of 19.15 years old, and they have been receiving English learning for9.58 years. In order to evaluate the effectiveness of employing AWE in EFL

Figure 4. The writing score interface of CorrectEnglish.

university students’ writing, the subjects were divided into an experimental groupand a control group. One class, as the experimental class, was trained how to utilizeAWE to complete their writing tasks in the spring semester of 2011. The other class,as the control group, received traditional teacher-led writing instruction, whichmeans that teachers were the main source of control during the writing process. Awriting composition test which is related to Taiwanese culture was conducted within30 minutes before this study for the purpose of confirming an equivalent proficiencylevel between the experimental group and the control group. Also, the AWEsoftware, CorrectEnglish, was used to evaluate subjects’ proficiency level based onthe Flesch–Kincaid Grade Level which is a valid and reliable language readabilityformula to test the readability of written texts. According to the previous studies,lower scores characterize a text that is more difficult to read and roughly correspondsto students’ lower writing ability (Cleaveland & Larkins, 2004), lower literacy level(Perin, Keselman, & Monopoli, 2003), and poorer writing quality (Darus, Ismail, &Ismail; 2008; Shang, 2007). The results of the t-test reveal that the mean scores of theexperimental group and the control group were 7.65 (SD ¼ 2.20) and 7.61(SD ¼ 2.58), with an insignificant difference between these two groups(p ¼ 0.956); in other words, the subjects involved in this study had the equivalentEnglish writing proficiency level. The detailed information is shown in Table 1.

Sampling strategies

Quantitative sampling

In this study, 57 EFL freshmen majoring in applied English at one university in thesouth of Taiwan were selected to be involved in this study, and 31 students who werein the experimental group were chosen to do the questionnaire for attaining theirself-reports toward the usage of AWE by using multistage sampling strategies. Thecluster sampling, which is useful to save time by collecting data at a specific site, wasused first to randomly select writing classes among freshmen at the university.Subsequently, random sampling was applied to select a sample with 57 students fromtwo English writing classes and then dividing them into the experimental group andthe control group.

Qualitative sampling

In addition, in this study, 15 out of 31 students in the experimental group wererandomly chosen for exploring their in-depth perceptions toward the usage of AWEon their writing performance. Purposeful random sampling was applied for

Table 1. Demographic information of subjects.

Group N

Gender

Age Years of learning EnglishMean and SDof writing testM F

Experimental 31 12 19 19.13 9.60 7.65 (SD ¼ 2.20)Control 26 9 17 19.17 9.56 7.61 (SD ¼ 2.58)Total 57 21 36 19.15 9.58 7.63 (SD ¼ 3.49)

Dow

nloa

ded

by [

Sout

hern

Tai

wan

Uni

vers

ity o

f Sc

ienc

e an

d T

echn

olog

y] a

t 20:

23 1

6 D

ecem

ber

2014


conducting the semi-structured interview because 15 participants involved in thepresent study had experience of utilizing AWE in terms of accuracy, learnerautonomy, and interaction.

Instrumentation

A writing composition test

For the purpose of examining the effectiveness of applying AWE on theenhancement of students’ writing accuracy between the experimental group andthe control group, a writing composition pre-test was conducted in both groups atthe beginning of the spring semester of 2011 before the subjects in the experimentalgroup were trained how to use AWE. A post-test with the same writing topic wasalso conducted in both groups at the end of the spring semester of 2011 after thosesubjects in the experimental group had had the experience of using the AWEsoftware, CorrectEnglish. The topic of the writing composition in the pre-test andpost-test, My Favorite Chinese Festival, is extracted from Yu’s (2006) study becauseit is related to Taiwanese culture, and thus it is easier for the subjects to pool ideasand to produce their own work. Both the pre-test and post-test were of 30 minutesduration in total to ensure that the subjects would have enough time to complete thewriting task with the requested 300 words minimum. Dictionaries were allowed forbasic reference. A pilot test was done by the first three subjects in the experimentalgroup and the control group before conducting this research in order to confirm thatall of the subjects would not have difficulties in finishing the tests under pressure oftime.

AWE software

CorrectEnglish was applied in the present study for three reasons. First, this softwarewas used to provide instructions for the task, a holistic score, and immediatefeedback of grammar, style, and proper word usage to those subjects in theexperimental group for helping them complete the final draft and produce a betteressay. Secondly, such software can easily be utilized to categorize the subjects intothe experimental group and the control group with equivalent proficiency level basedon the Flesch–Kincaid Grade Level. Thirdly, this software was also employed toinvestigate whether there was a significant difference after the treatment of applyingthe AWE in the writing process in terms of accuracy.

A questionnaire

A questionnaire was designed and adopted by the studies of Chen and Cheng (2008),Lai (2009), Yang (2010), and Yu (2006). Thirty multiple-choice questionnaire itemswere composed regarding the subjects’ self-reports of using the AWE to influencetheir writing accuracy, learner autonomy attitude, and interaction with thecomputer. Thirty items were divided into three major directions, including 10 itemsin the accuracy section, 10 items in the learner autonomy section, and 10 items in theinteraction section. Likewise, the questionnaire consisted of a five-point Likert scale,ranging from 5 (strongly agree) to 1 (strongly disagree). A pilot study was done byfive freshmen, majoring in applied English at one university in the south of Taiwan

Dow

nloa

ded

by [

Sout

hern

Tai

wan

Uni

vers

ity o

f Sc

ienc

e an

d T

echn

olog

y] a

t 20:

23 1

6 D

ecem

ber

2014


conducting the semi-structured interview because 15 participants involved in thepresent study had experience of utilizing AWE in terms of accuracy, learnerautonomy, and interaction.

Instrumentation

A writing composition test

For the purpose of examining the effectiveness of applying AWE on theenhancement of students’ writing accuracy between the experimental group andthe control group, a writing composition pre-test was conducted in both groups atthe beginning of the spring semester of 2011 before the subjects in the experimentalgroup were trained how to use AWE. A post-test with the same writing topic wasalso conducted in both groups at the end of the spring semester of 2011 after thosesubjects in the experimental group had had the experience of using the AWEsoftware, CorrectEnglish. The topic of the writing composition in the pre-test andpost-test, My Favorite Chinese Festival, is extracted from Yu’s (2006) study becauseit is related to Taiwanese culture, and thus it is easier for the subjects to pool ideasand to produce their own work. Both the pre-test and post-test were of 30 minutesduration in total to ensure that the subjects would have enough time to complete thewriting task with the requested 300 words minimum. Dictionaries were allowed forbasic reference. A pilot test was done by the first three subjects in the experimentalgroup and the control group before conducting this research in order to confirm thatall of the subjects would not have difficulties in finishing the tests under pressure oftime.

AWE software

CorrectEnglish was applied in the present study for three reasons. First, this softwarewas used to provide instructions for the task, a holistic score, and immediatefeedback of grammar, style, and proper word usage to those subjects in theexperimental group for helping them complete the final draft and produce a betteressay. Secondly, such software can easily be utilized to categorize the subjects intothe experimental group and the control group with equivalent proficiency level basedon the Flesch–Kincaid Grade Level. Thirdly, this software was also employed toinvestigate whether there was a significant difference after the treatment of applyingthe AWE in the writing process in terms of accuracy.

A questionnaire

A questionnaire was designed and adopted by the studies of Chen and Cheng (2008),Lai (2009), Yang (2010), and Yu (2006). Thirty multiple-choice questionnaire itemswere composed regarding the subjects’ self-reports of using the AWE to influencetheir writing accuracy, learner autonomy attitude, and interaction with thecomputer. Thirty items were divided into three major directions, including 10 itemsin the accuracy section, 10 items in the learner autonomy section, and 10 items in theinteraction section. Likewise, the questionnaire consisted of a five-point Likert scale,ranging from 5 (strongly agree) to 1 (strongly disagree). A pilot study was done byfive freshmen, majoring in applied English at one university in the south of Taiwan

to ensure that each item was fully comprehensible. After pilot testing, items 8, 12, 13,17, and 27 were modified for better understanding.

To testify the validity of the questionnaire, each item of the questionnaire wasexamined by using the software, SPSS 17.0 version, showing that there was asignificant difference (p 5 0.05) in each questionnaire item, so that all the itemsshould be retained. For reliability, the results of the internal-consistency coefficientsshowed that Cronbach’s alpha ( a ) for accuracy, learner autonomy, and interactionwere 0.81, 0.85, and 0.79, respectively, indicating that there is a high reliabilityamong the questionnaire items.

A semi-structured interview

Fifteen participants who were selected from the experimental group were interviewedusing a semi-structured interview technique to explore students’ in-depth perceptionstoward their usage of AWE software while writing. In terms of getting access to theparticipants, a consent form delineating the purpose of this study and the rights ofthe respondents, including their right to refuse to answer questions, deemed sensitive.The interview process was also explained prior to obtaining permission from theparticipants. Each participant was interviewed for about 15–20 minutes, and theywere informed that their personal identities would not be revealed, instead beingreplaced by unidentifiable numbers. During the interview process, each participantcould answer the questions in either Chinese or English depending on whichlanguage the participants felt could best express their opinions and perceptions.Fifteen questions were composed to explore participants’ opinions toward the effectof using the AWE software in writing. Additional follow-up questions were asked inorder to build up a more complete picture. Before conducting the interview with theparticipants, a pilot testing was conducted by two participants from those 15 chosenstudents in order to modify the interview questions. Some adjustments were madecontaining revisions of the proper tense, the modification of some distortion ofquestions, and clarification of any ambiguous questions.

Data collection procedures

In the present study, the nonequivalent group pre-test/post-test control andcomparison group design were applied; that is, the subjects in the experimentalgroup are given a pre-test, then the treatment, and then a post-test which is the sameas the pre-test. The overall experiment was conducted with a group of freshmen whowere separated with nonequivalent numbers into the experimental group and thecontrol group from two English writing classes, and the experiment was divided intothree phases: the pre-treatment phase, the during-treatment phase, and the post-treatment phase.

First, in the pre-treatment phase (Week 1 and Week 2), subjects in theexperimental group received instruction on how to utilize AWE, and both theexperimental group and the control group received a writing composition pre-test.

Secondly, in the during-treatment phase (Week 3 to Week 16), the subjects in theexperimental group received the AWE instruction for three hours per week, lastingfor 14 weeks. The whole writing process was divided into three sessions:‘‘Introductory Session,’’ ‘‘Writing Session,’’ and ‘‘Revising Session’’ (Lai, 2009). Inthe ‘‘Introductory Session,’’ the instructor introduced the topic and gave students

Dow

nloa

ded

by [

Sout

hern

Tai

wan

Uni

vers

ity o

f Sc

ienc

e an

d T

echn

olog

y] a

t 20:

23 1

6 D

ecem

ber

2014


guidelines for the topic. After that, the instructor left 40–50 minutes for students topool and to organize their ideas by themselves, and to produce their own work in the‘‘Writing Session.’’ As for the ‘‘Revising Session,’’ the activities of AWE wereapplied in the experimental group through 14 weeks, and the researcher individuallyarranged a specific time (1 hour per week) for each subject after class in order toensure that all of the subjects could fully utilize the AWE software. During theactivities of AWE, students were asked to submit their compositions to the AWE,CorrectEnglish, to read the immediate feedback through CorrectEnglish, and torevise their draft based on the feedback in CorrectEnglish. After being assessed byCorrectEnglish, students had to save the revised drafts in their computerizedportfolios as proof. The control group meanwhile, received the same content ofinstruction as the experimental group in both the ‘‘Introductory Session’’ and the‘‘Writing Session.’’ With regard to the ‘‘Revising Session,’’ subjects in the controlgroup turned in their compositions to the instructor for correction and timelyfeedback.

Finally, in the post-treatment phase (Week 17 and Week 18), all the subjects inthe experimental group and the control group were requested to do the writingcomposition post-test with the same topic as the pre-test. Afterward, the subjects inthe experimental group were asked to fill in the questionnaire and to take part in aninterview about their perceptions toward the effect of AWE usage. To be morespecific, after a week of doing the post-tests, a questionnaire was delivered, and anindividual interview was performed to obtain more in-depth information from theexperimental group regarding the effects of AWE on accuracy, learner autonomy,and interaction. As for the qualitative data collection, tape-recording was adoptedduring the interview, and full transcription was used for further content analysis.

Data analysis procedures

In this study, both quantitative and qualitative research methods were employed toexplore the following four research questions: (1) Is there a significant difference inthe writing accuracy of the experimental group compared with the control groupafter the adoption of AWE? (2) Is there a significant difference in writing accuracybefore and after the employment of AWE? (3) What is the overall self-report of usingAWE on the improvement of student writing in terms of accuracy, learnerautonomy, and interaction? (4) What are students’ attitudes toward the usage ofAWE on their writing development?

First of all, for the purpose of scrutinizing the differences of the usage of AWE onstudents’ writing accuracy between the experimental group and the control group, anindependent-samples t-test technique was computed to examine the mean differenceof the post-test writing scores to answer research question 1.

Secondly, in order to evaluate the effect of applying the AWE software whilewriting, the results from the pre-test and post-test scores in the experimental groupwere calculated to answer the research question. To analyze the data, a paired-samples t-test technique was applied to investigate whether students’ writingaccuracy would have a significant difference (p 5 0.05) after the usage of the AWEsoftware, CorrectEnglish.

Thirdly, for the purpose of gathering subjects’ self-reports toward the usage ofAWE in their writing, frequency was computed from the questionnaire survey toanswer research question 3.

Dow

nloa

ded

by [

Sout

hern

Tai

wan

Uni

vers

ity o

f Sc

ienc

e an

d T

echn

olog

y] a

t 20:

23 1

6 D

ecem

ber

2014


guidelines for the topic. After that, the instructor left 40–50 minutes for students topool and to organize their ideas by themselves, and to produce their own work in the‘‘Writing Session.’’ As for the ‘‘Revising Session,’’ the activities of AWE wereapplied in the experimental group through 14 weeks, and the researcher individuallyarranged a specific time (1 hour per week) for each subject after class in order toensure that all of the subjects could fully utilize the AWE software. During theactivities of AWE, students were asked to submit their compositions to the AWE,CorrectEnglish, to read the immediate feedback through CorrectEnglish, and torevise their draft based on the feedback in CorrectEnglish. After being assessed byCorrectEnglish, students had to save the revised drafts in their computerizedportfolios as proof. The control group meanwhile, received the same content ofinstruction as the experimental group in both the ‘‘Introductory Session’’ and the‘‘Writing Session.’’ With regard to the ‘‘Revising Session,’’ subjects in the controlgroup turned in their compositions to the instructor for correction and timelyfeedback.

Finally, in the post-treatment phase (Week 17 and Week 18), all the subjects inthe experimental group and the control group were requested to do the writingcomposition post-test with the same topic as the pre-test. Afterward, the subjects inthe experimental group were asked to fill in the questionnaire and to take part in aninterview about their perceptions toward the effect of AWE usage. To be morespecific, after a week of doing the post-tests, a questionnaire was delivered, and anindividual interview was performed to obtain more in-depth information from theexperimental group regarding the effects of AWE on accuracy, learner autonomy,and interaction. As for the qualitative data collection, tape-recording was adoptedduring the interview, and full transcription was used for further content analysis.

Data analysis procedures

In this study, both quantitative and qualitative research methods were employed toexplore the following four research questions: (1) Is there a significant difference inthe writing accuracy of the experimental group compared with the control groupafter the adoption of AWE? (2) Is there a significant difference in writing accuracybefore and after the employment of AWE? (3) What is the overall self-report of usingAWE on the improvement of student writing in terms of accuracy, learnerautonomy, and interaction? (4) What are students’ attitudes toward the usage ofAWE on their writing development?

First of all, for the purpose of scrutinizing the differences of the usage of AWE onstudents’ writing accuracy between the experimental group and the control group, anindependent-samples t-test technique was computed to examine the mean differenceof the post-test writing scores to answer research question 1.

Secondly, in order to evaluate the effect of applying the AWE software whilewriting, the results from the pre-test and post-test scores in the experimental groupwere calculated to answer the research question. To analyze the data, a paired-samples t-test technique was applied to investigate whether students’ writingaccuracy would have a significant difference (p 5 0.05) after the usage of the AWEsoftware, CorrectEnglish.

Thirdly, for the purpose of gathering subjects’ self-reports toward the usage ofAWE in their writing, frequency was computed from the questionnaire survey toanswer research question 3.

Finally, in order to answer research question 4, a semi-structured interviewtechnique was utilized in the experimental group to explore more in-depthinformation about students’ perceptions toward the usage of AWE in Englishwriting. Regarding the qualitative data analysis, tape recording was adopted duringthe process of interview. After the interview, all of the participants’ expressions weretranscribed in order to explore students’ perceptions regarding their usage of AWEwhile writing. After transcribing, all the data were studied carefully to find out thesimilarities and differences, and then the data were divided into smaller and moremeaningful units related to accuracy, learner autonomy, and interaction of AWE inthe present study. Likewise, positive and negative patterns were discovered bydefining conceptual similarities and finding negative evidence from those categories.What is more, a grounded theory was inductively formed by identifying andsynthesizing these two patterns.

Results

Research question 1: Is there a significant difference in the writing accuracy of theexperimental group compared with the control group after the treatment of usingAWE?

An independent-samples t-test technique was conducted to analyze whether there is asignificant difference in terms of students’ writing accuracy between the experimentalgroup and the control group. Sentence-level errors including run-on sentences,sentence fragments, capitalization errors,missing articles, punctuation errors, etc.wereidentified by the AWE software and then counted by the researcher. As indicated inTable 2, the experimental group (M ¼ 4.84, SD ¼ 3.02) made fewer errors than thecontrol group did (M ¼ 15.58, SD ¼ 9.90), with a significant difference at the 0.05probability level (p ¼ 0.000), demonstrating that there is a statistically significantdifference in the post-test score between the experimental group and the control groupon the variable of accuracy after the treatment of utilizing the AWE software.

Research question 2: Is there a significant difference in the writing accuracy before andafter the employment of AWE?

A paired-samples t-test technique was implemented to investigate whether there is asignificant difference in terms of students’ writing accuracy between the pre-test andpost-test scores in the experimental group. As indicated in Table 3, comparing withthe pre-test score (M ¼ 14.81, SD ¼ 7.62), students in the experimental group madefewer errors in their post-test (M ¼ 4.84, SD ¼ 3.02), with a significant difference atthe 0.05 probability level (p ¼ .010), showing that there is a statistically significantdifference between the pre-test and post-test scores in terms of accuracy afterstudents in the experimental group received the treatment of using AWE in their

Table 2. Results from the t-test for the experimental and control groups.

Group N M SD F t Sig

Experimental group 31 4.84 3.02 16.69 75.78* 0.000*Control group 26 15.58 9.90

Note: N ¼ number of subjects, M ¼ mean of errors,*p 5 0.05.

Dow

nloa

ded

by [

Sout

hern

Tai

wan

Uni

vers

ity o

f Sc

ienc

e an

d T

echn

olog

y] a

t 20:

23 1

6 D

ecem

ber

2014


Table 4. Mean scores and standard deviations toward the self-report of writing accuracywith AWE.

Writing accuracy

(Strongly)Agree, Frequency

(%)

Neutral,Frequency

(%)

(Strongly)Disagree,Frequency

(%)

1. AWE helps me figure outgrammatical errors which Ihave not noticed before.

27 (87%) 4 (13%) 0

2. With the support from AWE, Itry to use different sentencestructures to improve mywriting skills.

22 (71%) 8 (26%) 1 (3%)

3. I had a hard time usingappropriate words insentences. After using AWE, Ilearn to use accurate words inspecific contents.

27 (87%) 4 (13%) 0

4. I do not know how to revisemy writing accurately due tothe unclear feedback fromAWE.

1 (3%) 10 (32%) 20 (65%)

5. AWE provides me with greathelp of using proper tense.

21 (68%) 7 (23%) 3 (10%)

6. I think the feedback fromAWE is not useful because it istoo simple for me.

1 (3%) 12 (39%) 18 (58%)

7. With the support from AWE, Itry to use different vocabulariesto improve my writing skills.

21 (68%) 7 (23%) 3 (10%)

8. My writing cannot beenhanced because the feedbackfrom AWE is vague.

3 (10%) 6 (19%) 22 (71%)

9. My writing becomes moreaccurate after receivingfeedback from AWE.

18 (58%) 11 (35%) 2 (6%)

10. AWE provides me inaccuratefeedback which makes me getlower grades.

1 (3%) 3 (10%) 27 (87%)

Table 3. Results from the t-test for the pre-test and the post-test in the experimental group.

Test N M SD

95% confidenceinterval of the

difference

t SigLower Upper

Pre-test 31 14.81 7.62 7.47 12.46 8.16 0.010*Post-test 31 4.84 3.02

Note: N ¼ number of subjects, M ¼ mean of errors, *p 5 0.05.

Dow

nloa

ded

by [

Sout

hern

Tai

wan

Uni

vers

ity o

f Sc

ienc

e an

d T

echn

olog

y] a

t 20:

23 1

6 D

ecem

ber

2014


Table 4. Mean scores and standard deviations toward the self-report of writing accuracywith AWE.

Writing accuracy

(Strongly)Agree, Frequency

(%)

Neutral,Frequency

(%)

(Strongly)Disagree,Frequency

(%)

1. AWE helps me figure outgrammatical errors which Ihave not noticed before.

27 (87%) 4 (13%) 0

2. With the support from AWE, Itry to use different sentencestructures to improve mywriting skills.

22 (71%) 8 (26%) 1 (3%)

3. I had a hard time usingappropriate words insentences. After using AWE, Ilearn to use accurate words inspecific contents.

27 (87%) 4 (13%) 0

4. I do not know how to revisemy writing accurately due tothe unclear feedback fromAWE.

1 (3%) 10 (32%) 20 (65%)

5. AWE provides me with greathelp of using proper tense.

21 (68%) 7 (23%) 3 (10%)

6. I think the feedback fromAWE is not useful because it istoo simple for me.

1 (3%) 12 (39%) 18 (58%)

7. With the support from AWE, Itry to use different vocabulariesto improve my writing skills.

21 (68%) 7 (23%) 3 (10%)

8. My writing cannot beenhanced because the feedbackfrom AWE is vague.

3 (10%) 6 (19%) 22 (71%)

9. My writing becomes moreaccurate after receivingfeedback from AWE.

18 (58%) 11 (35%) 2 (6%)

10. AWE provides me inaccuratefeedback which makes me getlower grades.

1 (3%) 3 (10%) 27 (87%)

Table 3. Results from the t-test for the pre-test and the post-test in the experimental group.

Test N M SD

95% confidenceinterval of the

difference

t SigLower Upper

Pre-test 31 14.81 7.62 7.47 12.46 8.16 0.010*Post-test 31 4.84 3.02

Note: N ¼ number of subjects, M ¼ mean of errors, *p 5 0.05.

writing. One sample of student writing in the experimental group (before and afterthe employment of AWE) is shown in the appendix.

Research question 3: What is the overall self-report of using AWE on the improvementof student writing in terms of accuracy, learner autonomy, and interaction?

The questionnaire in this study was designed in three parts in order to evaluatestudents’ perspectives upon whether the usage of AWE can improve their English

Table 5. Mean scores and standard deviations toward the self-report of learner autonomywith AWE.

Learner autonomy(Strongly)

Agree, Frequency (%)

Neutral,Frequency

(%)

(Strongly)Disagree,

Frequency (%)

1. I can examine my own writingweaknesses by myself afterusing AWE.

17 (55%) 12 (39%) 2 (6%)

2. I do not have to learn hard butget high score because it is easyto fool the AWE just by usingsome certain grammaticalrules.

0 4 (13%) 27 (87%)

3. I think that I cannot organizemy own learning by using theAWE.

2 (6%) 8 (26%) 21 (68%)

4. The functions of AWE, such asgrammar, tips for formats, andbilingual dictionary, can guideme to self-control the wholeprocess of writing.

10 (32%) 17 (55%) 4 (13%)

5. I can easily arrange my ownlearning by utilizing AWE.

13 (42%) 13 (42%) 5 (16%)

6. AWE arouses my awareness tobe responsible for writing sinceI can have a second chance toreview my own work.

22 (71%) 7 (23%) 2 (6%)

7. It depresses my self-learningattitude to surf in the endlessweb sources in the AWE.

3 (10%) 7 (23%) 21 (68%)

8. AWE makes me understandmore about writing becauseduring the process of usingAWE, I learn how to find outthe major points by myself.

18 (58%) 11 (35%) 2 (6%)

9. I think its hard to learn bymyself in the AWE because Ido not understand all of theEnglish feedback.

5 (16%) 11 (35%) 15 (48%)

10. I am willing to produce morewriting because of theconvenience of using AWEwithout the limitation of timeand frequency of usage.

23 (74%) 7 (23%) 1 (3%)

Dow

nloa

ded

by [

Sout

hern

Tai

wan

Uni

vers

ity o

f Sc

ienc

e an

d T

echn

olog

y] a

t 20:

23 1

6 D

ecem

ber

2014


writing in terms of accuracy, learner autonomy, and interaction. The results in Table4 manifestly show that students in the experimental group get help from the AWEnot only on the aspect of figuring out grammatical errors, such as tenses andsentence structures, but also in providing appropriate word usage. Generallyspeaking, most of the students reveal a positive attitude toward the AWE applicationin relation to its enhancing their writing accuracy.

As for the concern of students’ learner autonomy awareness, the results exhibitedin Table 5 display that students are more willing to produce more and better writingdue to an uplift in their learner autonomy attitude. The results shown in Table 5 alsouncover that students’ learner autonomy awareness is increased for two mainreasons. First, the beneficial functions of the AWE, such as the guidance formats andgrammar, can help students not only to organize their own writing easily but also tore-examine their productions in order to find out their own writing weaknesses.Secondly, students’ learner autonomy attitude is enhanced because there is norestriction of time and frequency of usage; therefore, students can complete theirwriting task and get immediate feedback at any time and at any place.

With regard to the interaction between students and AWE, however, there is notmuch interaction between them. According to the results shown in Table 6, most ofthe students consider that their writing cannot be improved when they confront somewriting difficulties which cannot be discussed with the AWE software in a real andmeaningful way because the feedback from the AWE is too rule-based for them interms of sentence meaning. This is despite the admission that interacting with theAWE is faster. In sum, the majority of students revealed a negative attitude towardthe AWE due to lack of meaningful interaction with the computer.

Research question 4: What are students’ attitudes toward the usage of AWE on theirwriting development?

To elicit more in-depth information regarding participants’ perceptions of the impactof using AWE in their English writing in terms of accuracy, learner autonomy, andinteraction, a semi-structured interview technique with 15 interview questions wasused. Fifteen participants from the experimental group were randomly selected to dothe individual interview. According to the qualitative interview results, several majorfindings are summarized in the following. First of all, students accentuated that theirwriting accuracy is obviously enhanced by utilizing the AWE software, includingtheir use of tenses and their sentence structures with respect to grammar and usage ofappropriate words. However, some writing difficulties might occur due to therepeated and vague feedback which confuses students during revision. Secondly,regarding students’ learner autonomy attitude, the majority of students stated thatby implementing the AWE software, their learner autonomy awareness was inspired.Part of the reason is that students are more willing to take more responsibilities fortheir learning, focusing on how to produce better work because the AWE providesstudents with more opportunities, guiding functions, and openness of time and usageto self-evaluate and self-correct their own writing. However, a small group ofstudents declared that they confronted some difficulties while using this software,such as feeling tired of extensive web searching, being restricted by a shortfall infeedback in relation to creativity, as well as understanding the English feedback.Thirdly, according to students’ perceptions toward their usage of AWE in terms ofinteraction, the majority of students are clear that there is no meaningful interaction

Dow

nloa

ded

by [

Sout

hern

Tai

wan

Uni

vers

ity o

f Sc

ienc

e an

d T

echn

olog

y] a

t 20:

23 1

6 D

ecem

ber

2014


writing in terms of accuracy, learner autonomy, and interaction. The results in Table4 manifestly show that students in the experimental group get help from the AWEnot only on the aspect of figuring out grammatical errors, such as tenses andsentence structures, but also in providing appropriate word usage. Generallyspeaking, most of the students reveal a positive attitude toward the AWE applicationin relation to its enhancing their writing accuracy.

As for the concern of students’ learner autonomy awareness, the results exhibitedin Table 5 display that students are more willing to produce more and better writingdue to an uplift in their learner autonomy attitude. The results shown in Table 5 alsouncover that students’ learner autonomy awareness is increased for two mainreasons. First, the beneficial functions of the AWE, such as the guidance formats andgrammar, can help students not only to organize their own writing easily but also tore-examine their productions in order to find out their own writing weaknesses.Secondly, students’ learner autonomy attitude is enhanced because there is norestriction of time and frequency of usage; therefore, students can complete theirwriting task and get immediate feedback at any time and at any place.

With regard to the interaction between students and AWE, however, there is notmuch interaction between them. According to the results shown in Table 6, most ofthe students consider that their writing cannot be improved when they confront somewriting difficulties which cannot be discussed with the AWE software in a real andmeaningful way because the feedback from the AWE is too rule-based for them interms of sentence meaning. This is despite the admission that interacting with theAWE is faster. In sum, the majority of students revealed a negative attitude towardthe AWE due to lack of meaningful interaction with the computer.

Research question 4: What are students’ attitudes toward the usage of AWE on theirwriting development?

To elicit more in-depth information regarding participants’ perceptions of the impactof using AWE in their English writing in terms of accuracy, learner autonomy, andinteraction, a semi-structured interview technique with 15 interview questions wasused. Fifteen participants from the experimental group were randomly selected to dothe individual interview. According to the qualitative interview results, several majorfindings are summarized in the following. First of all, students accentuated that theirwriting accuracy is obviously enhanced by utilizing the AWE software, includingtheir use of tenses and their sentence structures with respect to grammar and usage ofappropriate words. However, some writing difficulties might occur due to therepeated and vague feedback which confuses students during revision. Secondly,regarding students’ learner autonomy attitude, the majority of students stated thatby implementing the AWE software, their learner autonomy awareness was inspired.Part of the reason is that students are more willing to take more responsibilities fortheir learning, focusing on how to produce better work because the AWE providesstudents with more opportunities, guiding functions, and openness of time and usageto self-evaluate and self-correct their own writing. However, a small group ofstudents declared that they confronted some difficulties while using this software,such as feeling tired of extensive web searching, being restricted by a shortfall infeedback in relation to creativity, as well as understanding the English feedback.Thirdly, according to students’ perceptions toward their usage of AWE in terms ofinteraction, the majority of students are clear that there is no meaningful interaction

between the AWE and themselves, not only because they cannot discuss their writingdifficulties with the computer, but also because students’ true meaning cannot beunderstood while using the AWE software.

Discussion and conclusion

The purpose of this study aimed to examine whether there is a significant differencebetween the experimental and control groups after the employment of AWE. Thisstudy also focused on investigating whether there is a significant difference betweenthe pre-test and post-test scores after the application of AWE in the experimentalgroup. This was in addition to measuring the overall effect of using the AWE in theEFL writing classes in terms of accuracy, learner autonomy, and interaction asanalyzed from a questionnaire survey. Several major findings are discussed in thefollowing.

Table 6. Mean scores and standard deviations toward the self-report of interaction withAWE.

Interaction(Strongly) Agree,Frequency (%)

Neutral,Frequency (%)

(Strongly)Disagree,

Frequency (%)

1. I like feedback from AWEbecause I can get more ideas inwriting.

6 (19%) 11 (35%) 14 (45%)

2. I think my writing cannot beimproved due to the lack ofreal, meaningful interactionfrom AWE.

15 (48%) 11 (34%) 5 (16%)

3. I regard AWE as a realaudience.

2 (6%) 20 (65%) 9 (29%)

4. I think the feedback fromAWE is mechanistic, nothumanistic.

19 (61%) 12 (39%) 0

5. My writing ability cannot beimproved because in the AWE,I cannot discuss my ownwriting problems with it.

19 (61%) 12 (39%) 0

6. I think AWE is interactivebecause the feedback is instant.

9 (29%) 17 (55%) 5 (16%)

7. I think my writing can beimproved due to thesynchronous interaction fromAWE.

10 (32%) 17 (55%) 4 (13%)

8. I think the feedback fromAWE is not comparable tohuman interaction.

23 (74%) 7 (23%) 1 (3%)

9. It is faster to interact withAWE software than withhuman beings.

12 (39%) 12 (39%) 7 (23%)

10. My writing ability cannot beimproved because the AWEsoftware cannot understandwhat I want.

7 (23%) 15 (48%) 9 (29%)

Dow

nloa

ded

by [

Sout

hern

Tai

wan

Uni

vers

ity o

f Sc

ienc

e an

d T

echn

olog

y] a

t 20:

23 1

6 D

ecem

ber

2014


First of all, regarding the difference between the experimental group and thecontrol group, and the difference before and after the employment of AWE in theexperimental group in terms of writing accuracy, the findings of the present studydemonstrate that students in the experimental group make significantly fewererrors than those in the control group. Additionally, students involved in theexperimental group also make significantly fewer errors after receiving thetreatment of AWE. It is obvious to find out that students in the experimentalgroup perform better than those in the control group in terms of writing accuracyafter receiving the assistance of AWE. Such results match the previous studies(Dikli, 2006; Hoon, 2006; Kern & Warschauer, 2000; Li, 2007; Milton, 2006;Ware & Warschauer, 2006; Yeh et al., 2007), indicating that by receiving detailedand diagnostic feedback with clear explanations provided from the AWE,students can significantly improve their grammar, word usage, and spelling, andthus gain greater writing accuracy.

Secondly, overall findings from the questionnaire survey indicate that studentsin the experimental group generally hold positive attitudes toward the effect ofusing AWE on the improvement of their writing accuracy. Such a findingconfirms the t-test result of the present study, illustrating that it is effective forstudents to utilize the AWE to improve their usage of grammar, vocabulary, andspelling because students’ writing can be checked and guided by the AWEsoftware, so as to get immediate feedback for self-revision. Regarding the resultof students’ learner autonomy awareness, it is also shown that students whoreceive AWE increase their self-learning motivation and their attitude to theirresponsibility for completing their writing task. Such a result coincides withprevious studies (Chandler, 2003; Chen & Cheng, 2008; Denton, 2001, 2003;Denton et al., 2008; Dikli, 2006; Griffiths & Nicolls, 2010; Hoon, 2006; Lee, 2004;Li, 2007), suggesting that first, students learn how to manipulate the wholewriting process by themselves with the guidance of web resources and through thefunction of guided planning drawn from leading questions, and the managementof portfolios. Secondly, students are automatically motivated to produce moreand better written texts due to the convenience and easy accessibility of the AWEwithout the restriction of time and frequency of the usage. It is, nevertheless,exposed that there is no obvious interaction between students and the AWE.Such a result confirms the previous studies (Drechsel, 1999; Ferris & Hedgcock,2005; Hamp-Lyons, 2001; Herrington & Moran, 2001; Leki, 1990; Reid, 1994;Wang & Brown, 2008; Ware & Warschauer, 2006), showing that there is a lack ofreal communication when writing in the automated environment. No wonder themajority of EFL students still prefer getting feedback from human beings becausethey cannot communicate with the computer to solve their writing problems.

In short, according to the research results of the present study, students whouse the AWE experienced obvious enhancement in terms of writing accuracy andlearner autonomy awareness. Part of the reason is that the AWE software canguide students not only to resolve and reformulate wrong usage of grammar,words, and spelling by providing individualized error feedback with explanationsand examples but also to help students self-manipulate the whole writing processby using miscellaneous functions and immediacy. However, as for the interactionbetween students and the AWE, it was found that AWE has limitations inproviding specific feedback, especially with regard to aspects of content andorganization.

Dow

nloa

ded

by [

Sout

hern

Tai

wan

Uni

vers

ity o

f Sc

ienc

e an

d T

echn

olog

y] a

t 20:

23 1

6 D

ecem

ber

2014


First of all, regarding the difference between the experimental group and thecontrol group, and the difference before and after the employment of AWE in theexperimental group in terms of writing accuracy, the findings of the present studydemonstrate that students in the experimental group make significantly fewererrors than those in the control group. Additionally, students involved in theexperimental group also make significantly fewer errors after receiving thetreatment of AWE. It is obvious to find out that students in the experimentalgroup perform better than those in the control group in terms of writing accuracyafter receiving the assistance of AWE. Such results match the previous studies(Dikli, 2006; Hoon, 2006; Kern & Warschauer, 2000; Li, 2007; Milton, 2006;Ware & Warschauer, 2006; Yeh et al., 2007), indicating that by receiving detailedand diagnostic feedback with clear explanations provided from the AWE,students can significantly improve their grammar, word usage, and spelling, andthus gain greater writing accuracy.

Secondly, overall findings from the questionnaire survey indicate that studentsin the experimental group generally hold positive attitudes toward the effect ofusing AWE on the improvement of their writing accuracy. Such a findingconfirms the t-test result of the present study, illustrating that it is effective forstudents to utilize the AWE to improve their usage of grammar, vocabulary, andspelling because students’ writing can be checked and guided by the AWEsoftware, so as to get immediate feedback for self-revision. Regarding the resultof students’ learner autonomy awareness, it is also shown that students whoreceive AWE increase their self-learning motivation and their attitude to theirresponsibility for completing their writing task. Such a result coincides withprevious studies (Chandler, 2003; Chen & Cheng, 2008; Denton, 2001, 2003;Denton et al., 2008; Dikli, 2006; Griffiths & Nicolls, 2010; Hoon, 2006; Lee, 2004;Li, 2007), suggesting that first, students learn how to manipulate the wholewriting process by themselves with the guidance of web resources and through thefunction of guided planning drawn from leading questions, and the managementof portfolios. Secondly, students are automatically motivated to produce moreand better written texts due to the convenience and easy accessibility of the AWEwithout the restriction of time and frequency of the usage. It is, nevertheless,exposed that there is no obvious interaction between students and the AWE.Such a result confirms the previous studies (Drechsel, 1999; Ferris & Hedgcock,2005; Hamp-Lyons, 2001; Herrington & Moran, 2001; Leki, 1990; Reid, 1994;Wang & Brown, 2008; Ware & Warschauer, 2006), showing that there is a lack ofreal communication when writing in the automated environment. No wonder themajority of EFL students still prefer getting feedback from human beings becausethey cannot communicate with the computer to solve their writing problems.

In short, according to the research results of the present study, students whouse the AWE experienced obvious enhancement in terms of writing accuracy andlearner autonomy awareness. Part of the reason is that the AWE software canguide students not only to resolve and reformulate wrong usage of grammar,words, and spelling by providing individualized error feedback with explanationsand examples but also to help students self-manipulate the whole writing processby using miscellaneous functions and immediacy. However, as for the interactionbetween students and the AWE, it was found that AWE has limitations inproviding specific feedback, especially with regard to aspects of content andorganization.

Pedagogical implications

In light of some students’ opinions, three pedagogical implications are provided toEFL instructors for better improvement in teaching writing with the appropriateusage of AWE. First of all, though there is a significant effect after the treatment ofAWE in the experimental group, several students still mention that some feedbackprovided by the AWE is either too simple or too vague, so that they cannot improvetheir writing accuracy based on those mechanical feedback. It is, therefore, suggestedthat EFL writing teachers should also teach sentence structures and grammaticalrules in the writing class, so that students can figure out their own mistakes andknow how to correct them more easily while seeing the simple and vague feedbackwhich only shows ‘‘clause error’’ or ‘‘run-on sentence.’’

Secondly, although the majority of students hold positive attitudes toward theeffect of using AWE on the enhancement of their learner autonomy awareness, somestudents state that the AWE decreases their motivation because of the restrictions oncreativity, the tedium of endless searches of web resources, and the English-onlyfeedback. Therefore, EFL writing instructors should design some brainstormingactivities to help students get more writing ideas and creativity in the ‘‘IntroductorySession.’’ Likewise, it is also important for teachers to encourage students if they facedifficulties in using such software because after all, the AWE is an innovative tool forTaiwanese students. Moreover, teachers should show and review all of the English-only feedback in the pre-treatment phase while introducing this software, so thatstudents will not feel unfamiliar and anxious about the English-only feedback.

Finally, according to students’ self-reports toward interaction, the majority ofparticipants expressed the feeling that there was no real and meaningfulcommunication between students and the AWE software. To be more specific,even though the AWE provides the function of synchronous feedback, students stillexperience the problem while writing, particularly on the aspects of content andorganization, because of lack of mutual communication. Therefore, it is suggestedthat teachers should take this into account and provide students with some models ofwritten texts related to the topics, so that students can compare their own writingwith those models, in order to get more concrete ideas about how to compose awritten text with better content and organization.

Limitations of the study

Based on the research design of the present study, several limitations need to beameliorated and improved in any future study. First of all, the sample size with only57 subjects is too small to generalize. Secondly, the research results derived from thesubjects who are only English-major students may not be applicable to non-Englishmajors. Thirdly, the research results might not be valid and reliable enough becausethe duration of the experiment was for only one semester, this could well be too shorta period. Finally, some issues regarding gender and individual differences such asdifferent English proficiency levels are not taken into consideration.

In any future research, it is suggested to enlarge the sample size with the inclusionof other non-English majors, and to lengthen the duration of the experiment whichwould also permit an increased number of writing episodes. Consideration ofindividual differences such as students’ attitudes who participated in the research,gender, class, etc. may form the basis of variables used to assess the influence of theuse of AWE in the future.

Dow

nloa

ded

by [

Sout

hern

Tai

wan

Uni

vers

ity o

f Sc

ienc

e an

d T

echn

olog

y] a

t 20:

23 1

6 D

ecem

ber

2014


Notes on contributors

Ying-Jian Wang is a MA student in the Department of Applied English at I-Shou Universityin Taiwan.

Hui-Fang Shang is a full professor in the Department of Applied English at I-Shou Universityin Taiwan.

Paul Briody is a lecturer in the Department of Applied English at I-Shou University inTaiwan.

References

Beatty, K. (2003). Teaching and researching computer-assisted language learning. New York:Longman.

Chandler, J. (2003). The efficacy of various kinds of error feedback for improvement in theaccuracy and fluency of L2 student writing. Journal of Second LanguageWriting, 12, 267–296.

Chang, Y.C., Chang, J.S., Chen, H.J., & Liou, H.C. (2008). An automatic collocation writingassistant for Taiwanese EFL learners: A case of corpus-based NLP technology. ComputerAssisted Language Learning, 21, 283–299.

Chen, C.F., & Cheng, W.Y. (2006). The use of a computer-based writing program:Facilitation or frustration? Proceedings of the 23rd International Conference on EnglishTeaching and Learning in ROC, 1, 112–129.

Chen, C.F., & Cheng, W.Y. (2008). Beyond the design of automated writing evaluation:Pedagogical practices and perceived learning effectiveness in EFL writing classes.Language Learning & Technology, 12, 94–112.

Cleaveland, M.C., & Larkins, E.R. (2004). Web-based practice and feedback improve taxstudents’ written communication skills. Journal of Accounting Education, 22, 211–228.

Darus, S., Ismail, K., & Ismail, M.B.M. (2008). Effects of word processing on Arabpostgraduate students’ essays in EFL. European Journal of Social Sciences, 7, 63–77.

Denton, P. (2001). Generating coursework feedback for large groups of students using MSExcel & MS Word. University Chemistry Education, 5, 1–8.

Denton, P. (2003). Evaluation of the ‘electronic feedback’ marking assistant & analysis of anovel collusion detection facility. In J. Christie (Ed.), Proceedings of the 7th internationalcomputer aided assessment conference (pp. 157–173). Loughborough: Learning andTeaching Development.

Denton, P., Madden, J., Roberts, M., & Rowe, P. (2008). Students’ response to traditional andcomputer-assisted formative feedback: A comparative case study. British Journal ofEducation Technology, 39, 486–500.

Diab, N.M. (2010). Effects of peer-versus self-editing on students’ revision of language errorsin revised drafts. System, 38, 85–95.

Dikli, S. (2006). An overview of automated scoring of essays. The Journal of Technology,Learning, and Assessment, 5(1), 1–35.

Ding, T. (2008). A study of an E-Comment generating system: Development & application(Unpublished master’s thesis). National Tsinghua University of Education, Beijing, China.

Drechsel, J. (1999). Writing into silence: Losing voice with writing assessment technology.Teaching English in the Two-Year College, 26, 380–387.

Ferris, D., & Roberts, B. (2001). Error feedback in L2 writing classes: How explicit does itneed to be? Journal of Second Language Writing, 10, 161–184.

Ferris, D.R., & Hedgcock, J.S. (2005). Teaching ESL composition: Purpose, process, andpractice (2nd ed.). Mahwah, NJ: Lawrence Erlbaum Associates.

Foote, C.J. (1999). Attribution feedback in the elementary classroom. Journal of Research inChildhood Education, 13, 155–166.

Griffiths, L., & Nicolls, B. (2010). E-Support4U: An evaluation of academic writing skillssupport in practice. Nurse Education in Practice, 10, 341–348.

Hamp-Lyons, L. (2001). Fourth generation writing assessment. In T. Silva, & P.K. Matsuda(Eds.), On second language writing (pp. 117–125). Mahwah, NJ: Lawrence ErlbaumAssociates.

Herrington, A., & Moran, C. (2001). What happens when machines read our students’writing? College English, 63, 480–499.

Dow

nloa

ded

by [

Sout

hern

Tai

wan

Uni

vers

ity o

f Sc

ienc

e an

d T

echn

olog

y] a

t 20:

23 1

6 D

ecem

ber

2014


Notes on contributors

Ying-Jian Wang is a MA student in the Department of Applied English at I-Shou Universityin Taiwan.

Hui-Fang Shang is a full professor in the Department of Applied English at I-Shou Universityin Taiwan.

Paul Briody is a lecturer in the Department of Applied English at I-Shou University inTaiwan.

References

Beatty, K. (2003). Teaching and researching computer-assisted language learning. New York:Longman.

Chandler, J. (2003). The efficacy of various kinds of error feedback for improvement in theaccuracy and fluency of L2 student writing. Journal of Second LanguageWriting, 12, 267–296.

Chang, Y.C., Chang, J.S., Chen, H.J., & Liou, H.C. (2008). An automatic collocation writingassistant for Taiwanese EFL learners: A case of corpus-based NLP technology. ComputerAssisted Language Learning, 21, 283–299.

Chen, C.F., & Cheng, W.Y. (2006). The use of a computer-based writing program:Facilitation or frustration? Proceedings of the 23rd International Conference on EnglishTeaching and Learning in ROC, 1, 112–129.

Chen, C.F., & Cheng, W.Y. (2008). Beyond the design of automated writing evaluation:Pedagogical practices and perceived learning effectiveness in EFL writing classes.Language Learning & Technology, 12, 94–112.

Cleaveland, M.C., & Larkins, E.R. (2004). Web-based practice and feedback improve taxstudents’ written communication skills. Journal of Accounting Education, 22, 211–228.

Darus, S., Ismail, K., & Ismail, M.B.M. (2008). Effects of word processing on Arabpostgraduate students’ essays in EFL. European Journal of Social Sciences, 7, 63–77.

Denton, P. (2001). Generating coursework feedback for large groups of students using MSExcel & MS Word. University Chemistry Education, 5, 1–8.

Denton, P. (2003). Evaluation of the ‘electronic feedback’ marking assistant & analysis of anovel collusion detection facility. In J. Christie (Ed.), Proceedings of the 7th internationalcomputer aided assessment conference (pp. 157–173). Loughborough: Learning andTeaching Development.

Denton, P., Madden, J., Roberts, M., & Rowe, P. (2008). Students’ response to traditional andcomputer-assisted formative feedback: A comparative case study. British Journal ofEducation Technology, 39, 486–500.

Diab, N.M. (2010). Effects of peer-versus self-editing on students’ revision of language errorsin revised drafts. System, 38, 85–95.

Dikli, S. (2006). An overview of automated scoring of essays. The Journal of Technology,Learning, and Assessment, 5(1), 1–35.

Ding, T. (2008). A study of an E-Comment generating system: Development & application(Unpublished master’s thesis). National Tsinghua University of Education, Beijing, China.

Drechsel, J. (1999). Writing into silence: Losing voice with writing assessment technology.Teaching English in the Two-Year College, 26, 380–387.

Ferris, D., & Roberts, B. (2001). Error feedback in L2 writing classes: How explicit does itneed to be? Journal of Second Language Writing, 10, 161–184.

Ferris, D.R., & Hedgcock, J.S. (2005). Teaching ESL composition: Purpose, process, andpractice (2nd ed.). Mahwah, NJ: Lawrence Erlbaum Associates.

Foote, C.J. (1999). Attribution feedback in the elementary classroom. Journal of Research inChildhood Education, 13, 155–166.

Griffiths, L., & Nicolls, B. (2010). E-Support4U: An evaluation of academic writing skillssupport in practice. Nurse Education in Practice, 10, 341–348.

Hamp-Lyons, L. (2001). Fourth generation writing assessment. In T. Silva, & P.K. Matsuda(Eds.), On second language writing (pp. 117–125). Mahwah, NJ: Lawrence ErlbaumAssociates.

Herrington, A., & Moran, C. (2001). What happens when machines read our students’writing? College English, 63, 480–499.

Hirvela, A. (2005). Computer-based reading and writing across the curriculum: Two casestudies of L2 writers. Computers and Composition, 22, 337–356.

Hoon, T.B. (2006). Online automated essay assessment: Potentials for writing development. InA. Treloar & A. Ellis (Eds.),Making a difference with Web technologies. Proceedings of AusWeb 2006 (pp. 230–234). Australia: Southern Cross University.

Huang, Y.P. (2001). A review of composition teachers’ written feedback from learners’perspectives. In Selected Papers from the Tenth International Symposium on EnglishTeaching (pp. 456–465). Taipei, Taiwan: Crane.

Huot, B. (1996). Computers and assessment: Understanding two technologies. Computers andComposition, 13, 231–243.

Kepner, C.G. (1991). An experiment in the relationship of types of written feedback to thedevelopment of second-language writing skills. The Modern Language Journal, 75, 305–313.

Kern, R., & Warschauer, M. (2000). Introduction. In M. Warschauer & R. Kern (Eds.),Network-based language teaching: Concepts and practice (pp. 1–19). London: CambridgeUniversity Press.

Lai, Y.H. (2009). Which do students prefer to evaluate their essays: Peers or computerprogram? British Journal of Educational Technology, 41, 432–454.

Lee, H.K. (2004). A comparative study of ESL writers’ performance in a paper-based and acomputer-delivered writing test. Assessing Writing, 9, 4–26.

Leki, I. (1990). Coaching from the margins: Issues in written response. In B. Kroll (Ed.),Second language writing: Research insights for the classroom (pp. 57–68). Cambridge:Cambridge University Press.

Li, M. (2005). A tentative study of e-grading mechanisms in the teaching of writing(Unpublished master’s thesis). National Tsinghua University, Beijing, China.

Li, Y. (2007). The application of computers in English writing teaching and learning. Journalof Hubei Normal University, 27(5), 139–141.

Liu, L.E. (2002). A corpus-based lexical semantic investigation of verb-noun miscollocations inTaiwan learners’ English. Master’s thesis, Tamkang University, Taipei.

Milton, J. (2006). Resource-rich Web-based feedback: Helping learners become independentwriters. In K. Hyland & F. Hyland (Eds.), Feedback in second language writing: Contextand issues (pp. 123–139). New York: Cambridge University Press.

Perin, D., Keselman, A., & Monopoli, M. (2003). The academic writing of community collegeremedial students: Text and learner variables. Higher Education, 45(1), 19–42.

Reid, J.M. (1994). Responding to ESL students’ texts: The myths of appropriation. TESOLQuarterly, 28, 273–292.

Rezaei, A.R., & Lovorn, M. (2010). Reliability and validity of rubrics for assessment throughwriting. Assessing Writing, 15, 18–39.

Shang, H.F. (2007). An exploratory study of e-mail application on FL writing performance.Computer Assisted Language Learning, 20(1), 79–96.

Slatin, J.M. (1990). Reading hypertext: Order and coherence in a new medium. CollegeEnglish, 52, 870–883.

Tuzi, F. (2004). The impact of e-feedback on the revisions of L2 writers in an academic writingcourse. Computers and Composition, 21, 217–235.

Wang, J., & Brown, M.S. (2008). Automated essay scoring versus human scoring: Acorrelational study. Contemporary Issues in Technology and Teacher Education, 8.Retrieved September 26, 2010, from http://www.citejournal.org/vol8/iss4/languagearts/article1.cfm.

Warden, C.A. (2000). EFL business writing behaviors in differing feedback environments.Language Learning, 50, 573–616.

Ware, P.D., & Warschauer, M. (2006). Electronic feedback and second language writing. InK. Hyland & F. Hyland (Eds.), Feedback in second language writing: Context and issues(pp. 104–122). New York: Cambridge University Press.

Warschauer, M. (1999). Electronic literacies: Language, culture, and power in online education.Mahwah, NJ: Lawrence Erlbaum.

Warschauer, M., & Ware, P. (2006). Automated writing evaluation: Defining the classroomresearch agenda. Language Teaching Research, 10, 1–24.

Yang, Y.F. (2010). Students’ reflection on online self-correction and peer review to improvewriting. Computers & Education, 55, 1202–1210.

Dow

nloa

ded

by [

Sout

hern

Tai

wan

Uni

vers

ity o

f Sc

ienc

e an

d T

echn

olog

y] a

t 20:

23 1

6 D

ecem

ber

2014


Yeh, Y., Liou, H.C., & Yu, Y.T. (2007). The influence of automated essay evaluation andbilingual concordancing on EFL students’ writing. English Teaching & Learning, 31(1),117–160.

Yu, Y.T. (2006). Effects of automatic essay grading system and bilingual concordancer on EFLcollege students’ writing (Unpublished master’s thesis). National Tsinghua University ofEducation, Hsinchu, Taiwan.

Dow

nloa

ded

by [

Sout

hern

Tai

wan

Uni

vers

ity o

f Sc

ienc

e an

d T

echn

olog

y] a

t 20:

23 1

6 D

ecem

ber

2014


Yeh, Y., Liou, H.C., & Yu, Y.T. (2007). The influence of automated essay evaluation andbilingual concordancing on EFL students’ writing. English Teaching & Learning, 31(1),117–160.

Yu, Y.T. (2006). Effects of automatic essay grading system and bilingual concordancer on EFLcollege students’ writing (Unpublished master’s thesis). National Tsinghua University ofEducation, Hsinchu, Taiwan.

Appendix. Sample of student’s writing in the experimental group

My Favorite Chinese Festival (pre-test)

My favorite Chinese festival is Chinese New Year. Chinese New Year is the most interestingfestival for Chinese. It is a very happy festival and people can have a long vacation with theirfamilies. In the New Year, everyone is with happiness. There have aura of happiness aroundeverywhere.

The Chinese New Year’s source is from thousands years ago. There are many kind oflegend and many customs about Chinese New Year. The most famous source is about thebeast. People put the red paper on the door and let off the fireworks to scare away the beast,because the beast is afraid of red. In the end they cast the beast out. The next day, everyonecelebrate it, and say ‘‘congratulation’’ to each other and enjoy about the victory and therenascent breath.

My favorite Chinese festival is Chinese New Year because grandparents would givechildren the red envelopes. And in the Chinese New Year’s eve, everyone would come homeand get together to have a great dinner. Grandmother and aunts would make lots of deliciousfood. It is interesting that each food has different mean about it. For example, we eat fishbecause it means we are superabundant of life next year.

Every Chinese New Year, my father, mother and I will go to the countryside to visit mygrandmother, and we always have a nice dinner together. We can see uncles and aunts and mycousins, and we chat with others for many hours. In the vacation, we also can go outside or gohiking together and enjoy the works of god. All of us love the time we get along with eachother, because after the vacation, we have to go back to our home and work.

Nowadays, most of people just can see their family on the Chinese New Year vacation.Because the globalization, many people live around the world. Therefore, from ancient timesto the present, the Chinese New Year is not only a new year but also it is become a moreimportant festival in Chinese’s heart.

My Favorite Chinese Festival (post-test)

My favorite Chinese festival is Chinese New Year. Chinese New Year is the most interestingfestival for Chinese. It is a very happy festival, and people can have a long vacation with theirfamilies. In the New Year, everyone is full of happiness. There are the aura of happinessaround everywhere.

The Chinese New Year’s source is from thousands years ago. There are many kinds oflegend and many customs about Chinese New Year. The most famous source is about thebeast. People put the red paper on the door and let off the fireworks to scare away the beast,because the beast is afraid of red. In the end they cast the beast out. The next day, everyonecelebrates it, and say ‘‘Congratulation’’ to each other and enjoy about the victory and therenascent breath.

My favorite Chinese festival is Chinese New Year because grandparents would givechildren the red envelopes. And in the Chinese New Year’s Eve, everyone would come homeand get together to have a great dinner. Grandmother and aunts would make much deliciousfood. It is interesting that each food has different mean about it. For example, we eat fishbecause it means we are superabundant of life next year.

Every Chinese New Year, my father, mother and I will go to the countryside to visit mygrandmother, and we always have a great dinner together. We can see uncles and aunts andmy cousins, and we chat with others for many hours. In the vacation, we also can go outside orgo hiking together and enjoy the works of god. All of us love the time we get along with eachother, because after the vacation, we have to go back home and work.

Nowadays, most people just can see their family on the Chinese New Year vacation.Because of the globalization, many people live around the world. Therefore, from ancienttimes to the present, the Chinese New Year is not only a new year but also it becomes a moreimportant festival in Chinese’s heart.

Dow

nloa

ded

by [

Sout

hern

Tai

wan

Uni

vers

ity o

f Sc

ienc

e an

d T

echn

olog

y] a

t 20:

23 1

6 D

ecem

ber

2014

Documents

Exploring the impact of using automated writing evaluation in English as a foreign language university students' writing