Concurrent and retrospective verbal reports as tools to better

CONCURRENT AND RETROSPECTIVE VERBAL REPORTS 201

© Blackwell Publishing Ltd. 2003

INTERNATIONAL JOURNAL OF APPLIED LINGUISTICS, Vol. 13, No. 2, 2003 201

© Blackwell Publishing Ltd. 2003, 9600 Garsington Road, Oxford OX4 2DQ, UK and350 Main Street, Malden, MA 02148, USA

Concurrent and retrospectiveverbal reports as tools to better

understand the role of attention insecond language tasks

JOAQUIM CAMPS

Universtiy of Florida

This study investigates how the use of think-aloud protocols, bothconcurrent and retrospective, can contribute to the study of the roleof attention in second language acquisition. It is based on the analysisof think-aloud protocols produced by 74 first-year learners of Spanishduring and immediately after a reading and multiple-choice activity.The activity consisted of a text with direct object pronouns and 16blanks for which a choice of three possible antecedents was given.The key to making the right choice was attending to both form andmeaning in the input. The data in the think-aloud protocols wasclassified into mentions of the pronouns in the text as well as referencesto gender and number agreement. The results showed that mention ofthe targeted structure in the think-aloud protocols was related tobetter performance on the task for second-semester students, but notfor first-semester students. There was some difference in the resultsfor the concurrent and retrospective protocols. The possible com-plementary nature of these two sources of data is discussed.

Introduction

Recent studies have shown the usefulness of think-aloud protocols in under-standing learners’ cognitive processes as they perform tasks designed to helpthem make form–meaning connections when processing input (Alanen 1995;Jourdenais et al. 1995; Jourdenais 1998; Leow 1997, 1998a, 1998b, 2000, 2001;Rosa and O’Neill 1999). Think-aloud protocols, described by Ericsson andSimon (1993), have been used in psychology for over a century. In the past few

202 JOAQUIM CAMPS


decades they have also been applied to a variety of areas in second languageacquisition, such as writing (Cumming 1989, 1994; Raimes 1994), reading (Davisand Bistodeau 1993; Hosenfeld 1977), strategies for language use and interpreta-tion (A.D. Cohen 1984; Mangubhai 1991), and discourse (A.D. Cohen andOlshtain 1993). Jourdenais (2001), Kasper (1998), and Kormos (1998) offerrecent discussions of the application of verbal reports to L2 research.

In concurrent think-aloud protocols, participants are asked to verbalizetheir thought processes at the same time as they are carrying out a certainactivity (reading, writing, a problem-solving task, etc.). As Ericsson and Simon(1993: xiii) indicate: “It is important to note that subjects verbalizing theirthoughts while performing a task do not describe or explain what they are doing– they simply verbalize the information they attend to while generating theanswer” (emphasis in the original).

Despite the fact that think-aloud protocols have been commonly used inother areas of second language acquisition (SLA), their use in analyzing the roleof attention in SLA is relatively recent. Empirical studies on the effect ofinstruction and the role of attention in SLA have usually relied on a designconsisting of pretest, instructional treatment and posttest(s). The assumptionwas that if the results in the posttest were significantly better than those in thepretest, we could safely conclude that learners had paid attention to thestructures targeted during the instructional treatment. The introduction ofthink-aloud protocols in studies on the role of attention in SLA has enrichedthe experimental designs in this line of research by offering the possibility ofcollecting online evidence of the actual processes learners go through when theyare exposed to linguistic input. By collecting data through verbal reports, wecan obtain more accurate and detailed information that was impossible to obtainby means of the pretest–treatment–posttest experimental design mentionedabove.1 Studies that have used think-aloud protocols to investigate the role ofattention in SLA have analyzed the data provided in the verbal reports that theparticipants produced while they performed an experimental task. This wasdone in order to assess the level of awareness of the participants in a taskdesigned to help them attend to certain aspects of the input. This informationhas allowed researchers to compare the posttest results of different groups oflearners based on the levels of awareness they showed in their verbal reports.This methodological improvement has strengthened the empirical support forthe “noticing hypothesis” (Schmidt 1990, 1994, 1995).2

Think-aloud protocols in studies on the role of attentionto input

Think-aloud protocols have been used in studies on the role of attention to theinput in second language acquisition in order to analyze the effect of a varietyof experimental tasks or treatments. Most of these studies suggest that thelearner’s awareness of the target forms is positively related to recognition and



learning of those forms. Alanen (1995) studied the acquisition of a modifiedversion of Finnish and reported that awareness of the target forms (as reflectedin think-aloud protocols) was positively correlated to learning in some, but notall, experimental groups. Another example is provided by Leow (1997, 1998a,1998b), where learners performed think-aloud protocols while doing a cross-word puzzle involving irregular forms of the Spanish preterit. Leow found thatthose participants who showed a higher awareness of the target forms in theirprotocols performed better in the later recognition and production of thoseforms. Similar results were obtained by Rosa and O’Neill (1999) in a study thatused multiple-choice jigsaw puzzles with five different treatment groups usingconditional sentences in Spanish. In that study, learners whose protocolsindicated levels of awareness classified as understanding obtained better resultsthan those who showed awareness at the level of noticing or no awareness at all.Rosa and O’Neill applied the distinction between noticing and understandingproposed by Schmidt (1990, 1995). Their study analyzed intake but not pro-duction data. Leow (2001) also found a significant positive relationship betweenreported noticing of the target form (Spanish formal commands) during a readingactivity and its subsequent recognition. His study found no difference betweenthe enhanced and the unenhanced condition. Overall, the studies mentionedabove support Schmidt’s position that noticing, that is, attention to a particularform in the input, together with some level of awareness of that form, allows forfurther processing of that input.

Concurrent and retrospective verbal reports

Most of the research on the role of attention in SLA that used verbal reportshas relied on concurrent protocols. However, concurrent protocols are not theonly type of verbal report available to researchers: the collection of retrospec-tive reports is another interesting option. Retrospective protocols are typicallyperformed immediately after the completion of a task, and participants areasked to report on the thoughts they had while they were completing the task.While concurrent think-aloud protocols reflect thought processes as they occur,retrospective protocols may draw from short- and long-term memory dependingon the length of the task performed and the amount of time between the end ofthe task and the beginning of the verbal report. In their discussion of think-aloud protocols, Ericsson and Simon (1993) favor the use of concurrent protocols,collected as the processing task is being performed, but they also recognize thebenefits of retrospective protocols as complementary data and recommend theuse of both. They indicate that even though in long cognitive tasks (beyond twominutes in length) the retrospective reports may be incomplete, having datafrom two sources may be very helpful. An important methodological point theyemphasize is that retrospective reports need to be collected as soon as possibleafter the cognitive task. This would minimize the possibility that participantsmay start relying on inferences rather than reporting what happened.

204 JOAQUIM CAMPS


Next, we will discuss a few studies that have used retrospective measures toassess the learners’ level of awareness of the particular target forms analyzed.P. Robinson (1996, 1997a, 1997b) asked participants a few questions, in writing,regarding their level of awareness of the rules and target forms they had beenexposed to. This allowed him to classify his participants according to whetherthey noticed any rules, were looking for rules, or were able to verbalize therules. In his (1997a) study of dative alternation in English, only instructedlearners were able to refer to the key features of the rule, but many incidentaland enhanced learners mentioned the position of direct and indirect objects inthe sentence. On the other hand, few implicit learners verbalized any aspect ofthe rules. The methodology used by P. Robinson departs from that typicallyused in verbal reports, as described by Ericsson and Simon (1993), in that thereports were written, rather than oral, and the questions he asked were verydirect (some were yes/no questions). This may have prompted specific responsesfrom the participants. Ericsson and Simon favor more open-ended methods ofletting participants retrospect on their experience.

In a study on the role of output in promoting noticing, Izumi and Bigelow(2000) conducted retrospective interviews with four of the nine participants intheir experimental group (they do not provide details on how the interviewswere carried out). They attested great individual variation in what learnersfound problematic in their production and in what they noticed in the inputthey were subsequently exposed to. To assess noticing, they asked the learnersto underline those elements that they thought would be necessary for theirsubsequent production.

Kormos (2000) followed more closely the methodology proposed by Ericssonand Simon for retrospective reports. The goal of Kormos’ study was to analyzethe self-correction behavior in the oral production of Hungarian learners ofEnglish as a foreign language. In her study, participants listened to a recordingof their production, which included examples of self-correction, and wereencouraged to stop the tape and comment on their production whenever theyhad something to say. The researcher also paused the tape when she felt therewas something worth discussing.

A few studies on the role of attention in SLA have complemented the use ofconcurrent think-aloud protocols with retrospective measures. In addition tousing concurrent think-aloud protocols, Alanen (1995) measured her participants’awareness by means of a questionnaire in which they were asked to state therules that governed the use of the linguistic forms in the input they received.The participants who received textual enhancement were also asked whetherthey had noticed the highlighting used. Leow (2000) is a recent study thatcombined a variety of measurements of learners’ awareness. He conductedconcurrent think-aloud protocols not only during the exposure task but alsoduring the post-exposure assessment task. Before and after the assessment task,participants answered, in writing, two probe questions to further assess theirawareness of the target forms (the irregular preterit in Spanish). Finally, threeweeks after the completion of the experiment, Leow interviewed participants



who had been classified as being unaware of the target forms and asked them toelaborate on their answers to the probe questions and to explain their choicesin the assessment task. Although the measures Leow used in addition to theconcurrent think-aloud protocols would not match the ideal for retrospectivereports described by Ericsson and Simon (written responses, a long delay betweenthe end of the experiment and the interviews), they are an interesting exampleof how to gather data from a variety of sources in order to increase the reliabilityof the results obtained.

Several studies in SLA have successfully implemented the use of bothconcurrent and retrospective verbal reports. Mangubhai (1991) conducted astudy that investigated and classified the different behaviors learners engagein when they extract meaning from the input they receive. Learners receivedindividualized Total Physical Response instruction in Hindi while wearing lapelmicrophones that recorded everything they said. They were encouraged toconstantly verbalize whatever was going through their minds as they performedthe TPR activities. Mangubhai collected concurrent think-aloud protocols during20 teaching sessions. At some points during a lesson, when a comprehensionproblem was observed, participants were probed to report on those problems.These reports produced immediate retrospective data. Further retrospectiveverbal reports were conducted at the end of some of the teaching sessions.

M. Robinson’s (1991) goal was to evaluate verbal reports as a means ofinvestigating interlanguage pragmatics knowledge. She gathered concurrentthink-aloud protocols from 12 female native speakers of Japanese as they plannedhow to fill out written discourse completion items using refusals. Participantswere instructed not to think aloud while they were writing but only before theystarted and whenever they paused to plan what to write. Immediately after thetask the researcher interviewed each participant, basing the discussion on thetape recording of the think-aloud protocol the participant had just produced.M. Robinson concluded that the combination of concurrent and retrospectivereports enhances the informative value of verbal reports, and that data fromverbal reports provide information regarding language-processing strategies thatwould not be available from the analysis of discourse completion tasks alone.

Rationale and research questions

In the review presented above, we saw that research on the role of attention insecond language acquisition supports a positive relationship between awarenessof the target forms and their subsequent recognition and learning, thus favoringSchmidt’s (1990, 1994, 1995) “noticing hypothesis”. This research points to theusefulness of think-aloud protocols as a tool to gather data regarding the actualonline processing of input by language learners, and it is mostly based onconcurrent verbal reports to establish awareness. However, we also saw howretrospective verbal reports have been successfully used in combination withconcurrent verbal reports in studies in other areas of SLA research. The

206 JOAQUIM CAMPS


discussion of these different procedures suggests that using multiple data-collection procedures (in this case, concurrent and retrospective verbal reports)may greatly enrich our knowledge of how learners attend to input.

By designing a study that requires participants to complete both concurrentand retrospective protocols based on one given task, we will not only benefitfrom two different sources of information but will also be able to compare theinformation obtained through each type of verbal report.

The present study was designed to explore whether those participants whonoticed the targeted forms performed differently in the experimental task thanthose who did not. In this study, the learners’ noticing was operationalized asmentioning in their verbal protocols the targeted object pronouns, the objectpronouns and their agreement features, or just the agreement features as thebasis for their choice of answer in a reading and multiple-choice task.

The scope of this project was limited to analyzing how learners processedthe input present in one particular task. It did not use any post-exposuremeasures to test the recognition or production of the target forms included inthe experimental task, since the aim was to carefully investigate the character-istics of this specific type of task and to analyze how learners processed theinput it provided. This study also sought to research a methodological question:whether the concurrent and retrospective protocols, collected during theexperimental task and right after it was completed, produced similar results.The following research questions were investigated:

a) Will the participants who notice the target forms obtain better scores in theexperimental task than those who do not?

b) Will the results vary depending on the type of think-aloud protocol considered(concurrent or retrospective)?

c) Will the results vary depending on whether students are in first- or second-semester courses?

The experimental study

PARTICIPANTSA total of 74 first-year Spanish college students, all native speakers of English,participated in the study. The pool of participants was drawn from five sectionsof first-semester Spanish classes and seven sections of second-semester Spanish.The final distribution by groups was the following: 29 in the first-semester groupand 45 in the second-semester group.3 All participants took part in the studyduring the same week in July of 1998. The first-semester students performedthe task the week after they had been introduced to the target structure as partof their regular Spanish class, while the second-semester students had beenintroduced to the target structure several weeks earlier, during their firstsemester of study. Although no independent measures of proficiency were usedto compare the two groups, the second group had had many more opportunities



to be exposed to those forms in the input they received and to produce themduring their regular class activities.

TARGETED LINGUISTIC FORMThe study focused on the processing of third-person object clitic pronouns inSpanish (lo, la, los, las). Sentence (1) is an example of the structure considered,where la refers to una carta:

1) Juan escribe una carta y después la envía por correo.‘John writes a letter, and then he mails it.’

The structural characteristics of the constructions with clitic pronouns, andespecially the fact that clitic pronouns can never appear in isolation, makethem an appropriate target for studies that deal with form–meaning connections.The clitic pronouns considered in this study must always have a referent (anantecedent), either overtly present in the linguistic context or understood in amore general, situational context. Thus in example (1), the agreement features(gender and number) of the clitic pronoun la have to match with those of itsantecedent una carta.

MATERIALSAll participants received the same text, a narration of events which containedfour examples each of the four third-person direct object clitic pronouns (lo, la,los, las) and 16 blanks. The learners’ task was to read the text and choose oneof the three options provided for each blank (see Appendix). Sentence (2) is anexample taken from the text, with the possible answers available to the learners:

2) Recoge ____ del mes pasado que tiene por el suelo y las lleva a reciclar.‘(She) picks up ____ from last month, which are lying on the floor, andtakes them out for recycling.’

a. las revistas (‘magazines’)b. los periódicos (‘newspapers’)c. las blusas (‘blouses’)

The participants had to check both how well the meaning of a possible answerfit in the context and whether the gender and number of the noun phraseagreed with the clitic pronoun related to it. In the example given, if the learnerjust focused on meaning, both answers (a) las revistas and (b) los periódicoswould appear to be correct, since they can be from last month and can berecycled. However, only las revistas agrees in gender and number with the directobject pronoun las. If the learner just focused on the form, the agreementfeatures of the pronoun las, then both (a) las revistas and (c) las blusas wouldappear to be correct choices. However, blouses is not a good choice, becausethe text talks about something from last month that María is going to recycle.

208 JOAQUIM CAMPS


The order of the three possible types of answers was varied randomly from itemto item.

PROCEDUREThe entire data collection process took place in a language laboratory wherestudents sat at individual recording booths. After the researcher introducedhimself and the general purpose of the study, students read and signed the consentform, which contained detailed information about the study. Next, participantscompleted a questionnaire with personal and language background information.

The researcher read aloud instructions on how to perform think-aloudprotocols for the activities that were going to follow. The participants wereasked to perform both concurrent and retrospective think-aloud protocols. Theywere told about the retrospective protocol only after the experimental task wascompleted. The script used for the concurrent protocol, adapted from Ericssonand Simon (1993), was the following:

In this experiment we are interested in what you think about when you perform areading and multiple-choice activity in Spanish. In order to find out, I am going toask you to THINK ALOUD while you perform the task. What I mean by “thinkaloud” is that I want you to tell me EVERYTHING you are thinking while you aredoing this activity. I would like you to talk CONSTANTLY while you perform theactivity. I don’t want you to try to plan out what you say or to try to explain whatyou are saying. Just act as if you were alone in the room speaking to yourself.What’s most important is that you keep talking, and talk clearly and loudly enoughinto the microphone. If you are silent for any period of time I will remind you to talkby saying “What are you thinking?” Similarly, if you begin to speak softly, I may askyou to speak a little more loudly. I will not be able to help you in any way, but youcan use the vocabulary list provided. You will be audiotaped while you work on thetask. Do you understand what I want you to do?

After that, the researcher provided an example of a think-aloud protocolby performing a multiplication task (24 × 26) on the board for the entire classto see. Next, the participants were asked to perform another multiplicationtask (14 × 34) while thinking aloud and recording their voices on tape. Theseexamples were included in order to provide the participants with some trainingon the procedure involved in think-aloud protocols, as suggested by Ericssonand Simon (1993).

In the next step, participants were given a bilingual vocabulary list withitems relevant to the main activity that would follow. They received anotherpage with five different categories, e.g. Cosas que se pueden comer (‘Things youcan eat’). Their task was to classify the Spanish words into their correspondingcategories. They were asked to perform a think-aloud protocol while doing thistask and were given eight and a half minutes to carry it out. This vocabularyactivity served a double purpose. On the one hand, it provided additionalpractice in performing a think-aloud protocol before the main task. On theother hand, it helped to control for vocabulary familiarity by providing a reviewof the vocabulary participants would need in order to carry out the main task.



After the vocabulary activity, the participants moved on to the main activity,for which they were asked to provide a think-aloud protocol while completingthe reading and multiple-choice activity. They were allowed to check with thevocabulary list as many times as needed and to complete the task at their ownpace. Most of the participants finished in 7 to 12 minutes.

As each participant finished the main task, they were given the following setof written instructions for a retrospective think-aloud protocol: “Please, talkabout what you remember with regard to how you went about performing themultiple-choice activity you just completed.” Most participants completed thisfinal task in approximately 2 minutes.

SCORING PROCEDURE AND ANALYSISIn scoring the multiple-choice task, 1 point was given for every correct answer.The maximum score possible was 16. A coding system was developed to classifythe different types of comments found in the think-aloud protocols. Two slightlydifferent sets of categories were developed: one for the concurrent think-aloudprotocols and one for the retrospective think-aloud protocols.

The analysis of the concurrent protocols identified specific mentions of theobject pronoun forms which indicated that the choice of answer was based onthat pronoun, as well as general mentions of gender and number. The specificmentions of pronouns were further subclassified as making reference to genderor to number features. Below are the categories used in the concurrent think-alouds, with some examples.

Mention of a pronoun:3) I’ll say lettuce ’cause of la . . . yeah, la.

Mention of a pronoun and reference to gender:4) Let’s see, lo is masculine so it would have to be plato [dish].4

Mention of a pronoun and reference to number:5) I thought bananas ’cause it’s plural, so it’s las pone [puts them], so it can’t

be un tenedor [a fork] . . . and spoon . . . so, that makes sense.

General mentions of gender or number:6) Let’s go back to these. One, feminine, and this is still feminine, oh

man . . . that’s not plural though.5

The data in the retrospective protocols were classified in terms of generalmentions of object pronouns and agreement (gender and number). These typeswere considered to be equivalent to the categories used in the concurrentprotocols, because these mentions made reference to either the object pronouns orthe agreement features that make possible the matching of pronouns and theirantecedents. Below are the categories used in the retrospective think-alouds,followed by some examples.

210 JOAQUIM CAMPS


Reference to object pronouns:7) I would . . . look at the direct object pronouns and determine whether it was

male, I mean masculine or feminine.

Reference to gender:8) I looked for what went with the masculine and feminine.

Reference to number:9) Usually I had to finish the sentence to see to, like, see what it is they are

referring to later on . . . if it’s, like, plural or singular.

While in the concurrent protocols, most references were directly related tospecific examples of pronouns in the input, although there were also generalmentions of gender and number, the comments in the retrospective protocolswere more general. They included references to paying attention to pronounsand to gender and number features in order to solve the multiple-choice task,but they did not focus on specific items in the task.6 In view of the limitednumber of mentions of pronouns and their agreement features, and given themethodological difficulty of assigning specific examples to categories reflectingdifferent levels of awareness (e.g. noticing vs. understanding), a conservativeapproach to the analysis of the data was taken. Thus the participants weresorted into two groups: those who mentioned pronouns and their agreementfeatures and those who did not.

Results

The number of participants who mentioned pronouns, pronouns and theiragreement features, or just the agreement features in their protocols as thebasis for their choice of answer was fairly limited. Only 6 out of 29 students infirst semester and 10 out of 45 in second semester did so in the concurrentthink-aloud protocols. This represents only about 20% of the participants, asseen in Table 1. The percentage was slightly higher in the retrospective protocols(around 27%).

Table 1. Participants who mentioned pronouns and agreement in protocols

1st semester 2nd semester Total(N ===== 29) (N ===== 45) (N ===== 74)

no. % no. % no. %

Concurrent 6 20.7 10 22.2 16 21.6Retrospective 8 27.6 12 26.7 20 27.0



Table 2. 2 × 2 ANOVA for Group × Level based on concurrent and retrospectiveprotocols

df SS MS F-value p

ConcurrentGroup 1 9.35 9.35 2.63 .109Level 1 12.36 12.36 3.48 .066Group × Level 1 20.27 20.27 5.70* .020Error 70 248.76 3.55

RetrospectiveGroup 1 5.47 5.47 1.42 .237Level 1 6.09 6.09 1.58 .213Group × Level 1 8.91 8.91 2.32 .133Error 70 269.22 3.85

* statistically significant

The scores in the multiple-choice task for each of the 74 participants werecomputed. Recall that the maximum score possible was 16, corresponding to the16 blanks. Table 2 shows the results of the 2 × 2 ANOVA performed on the databased on the concurrent and the retrospective protocols. For the concurrentprotocols, neither Group (mention vs. no mention) nor Level (1st vs. 2ndsemester) were significant, although Level showed a trend towards significance.However, the interaction Group × Level was significant. Similar results wereobtained based on the retrospective protocols: neither of the two independentvariables showed significance, but in this case the interaction between Groupand Level was not significant, either.

Table 3 shows a comparison of mean scores and the results of t-tests basedon the type of protocol, semester and mention of pronouns. When comparingthose who mentioned pronouns and those who did not, there were significantdifferences in accuracy on the multiple-choice task only for second-semesterparticipants; this was true for both types of protocol, but the effect size waslarger for the concurrent protocols.7 Looking at all those who mentionedpronouns, within the concurrent protocols the mean for the second-semesterparticipants was significantly higher than for those in the first-semester, with alarge effect size. There were no other significant effects.

Table 4 shows the individuals in each group who mentioned pronouns andagreement during at least one of the two types of protocols, how many timesthey did so, and the types of mentions they made. For instance, participant X27from the first-semester group mentioned pronouns and their agreement threetimes in her concurrent think-aloud protocol and two times in her retrospectiveprotocol. We can see that in the first-semester group only three participantsmentioned pronouns or their agreement features in both their concurrent andretrospective protocols, while seven of the second-semester students did so.With regard to what features those participants made reference to in their

212J

OA

QU

IM C

AM

PS

© B

lackwell P

ublishing Ltd. 2003

Table 3. Comparison of results on multiple-choice task by type of protocol, mention of pronouns and course level

Concurrent Retrospective

No. of Mean s.d. No. of Mean s.d.students score students score

1st semester Mention 6 10.67 1.97 8 10.88 2.17No mention 23 11.09 2.13 21 11.05 2.09

p = .666, d = .20 p = .850, d = .082nd semester Mention 10 13.00 1.76 12 12.33 2.02

No mention 35 10.80 1.73 33 10.91 1.81p = .001*, d = 1.27 p = .045*, d = .76

Mention 1st semester 6 10.67 1.97 8 10.88 2.172nd semester 10 13.00 1.76 12 12.33 2.02

p = .028*, d = 1.27 p = .151, d = .70No mention 1st semester 23 11.09 2.13 21 11.05 2.09

2nd semester 35 10.80 1.73 33 10.91 1.89p = .575, d = .15 p = .797, d = .07

* = statistically significant; d = effect size



Table 4. Frequency of mention of pronouns, gender and number by participant

1st semester 2nd semester

Stud. Concurrent Retrospective Stud. Concurrent RetrospectiveX27 3 (O, G) 2 (O, G, N) + Y11 3 (O, G) 1 (G) –X31 1 (G) 1 (O, G) + Y53 1 (O) 1 (G) ≠X67 2 (G, N) 2 (G, N) = Y55 1 (N) 1 (G) ≠3 6 5 Y57 2 (G, N) 1 (G, N) =

Y61 3 (O, G) 1 (O, G) =Y93 1 (G) 1 (G) =Y105 1 (N) 1 (N) =7 12 7

X11 2 (N) Y13 1 (G)X53 2 (G, N) Y33 1 (O)X63 1 (O) Y97 1 (O)3 5 0 3 3 0

X7 1 (G, N) Y15 1 (N)X9 1 (G, N) Y23 1 (G, N)X21 1 (G) Y79 1 (G)X35 1 (N) Y83 2 (G, N)X49 1 (O, G) Y101 1 (N)5 0 5 5 0 611 11 10 15 15 13

O = Object; G = Gender; N = Number; totals in italics+ increase in features; – decrease in features; = same features; ≠ different features

comments, half of the participants mentioned the same types of elements inboth protocols (e.g. X67 referred to both gender and number), two othersprovided additional information in the retrospective protocol (e.g. X27 hadnot mentioned number in the concurrent protocol), and only one providedless information in the retrospective report (Y11). All but one of those 10participants made reference to gender.

There were the same number of participants at both levels who mademention of pronouns or agreement in only one of their protocols. Overall, moresecond-semester students mentioned pronouns or agreement. The total numberof mentions of relevant features was slightly higher in the concurrent protocolsfor both groups.8

When considering the cases where the learners mentioned the pronounsor some of their characteristics in the concurrent think-aloud protocols, it isimportant to look into the relationship between mentioning a pronoun or itsagreement features and getting the corresponding item right. The learnersmentioned pronouns on 26 occasions in the concurrent protocols, and on 24of those occasions they chose the correct answer. That is an accuracy level of

214 JOAQUIM CAMPS


92%, compared to only 69% accuracy for those items that were not mentioned(802 out of 1158). This gives a clear indication that when the learnersmentioned a pronoun, they made the right choice of antecedent in the vastmajority of the cases.

The research questions in this study explored the possible effect of threedifferent factors on the results obtained: (a) the difference between noticing andnot noticing the target forms, (b) the effect of the type of protocol considered,and (c) the course level at which the learners were enrolled. The resultsobtained show an interaction of factors that lead us to address the answers tothe three questions in an interrelated manner. In summary, when consideringall participants, those who mentioned the object pronouns and their agreementfeatures in their think-aloud protocols did not obtain higher scores than theparticipants who did not mention the pronouns and their features. However,among the second-semester students, those who mentioned the pronouns andtheir features did better than those who did not, and that was the case withboth types of protocols. Additionally, among all those who mentioned the targetforms, the scores for the second-semester group were significantly higher thanfor the first-semester group when we considered the concurrent protocols.

Discussion

The results of this study are in line with those of Leow (1997, 1998a, 1998b,2000, 2001), and Rosa and O’Neill (1999), in that the second-semester studentswho noticed the targeted forms and mentioned them in their protocols alsoobtained the best results in the experimental task. However, that was not thecase with the first-semester students. This finding, together with the lack ofdifference between first- and second-semester students within the group of thosewho did not mention pronouns or their features, may lead us to postulate adifference in the ability of the two groups to make use of the information theyattended to. This could be due to their ability to rehearse in short-term memorythe information they had just attended to (P. Robinson 1995). We could arguethat the first-semester students who mentioned the pronouns did not benefitmuch from paying attention to that particular aspect of the input because theywere not able to process the information provided deeply enough to activateits rehearsal in short-term memory. On the other hand, the second-semesterstudents who mentioned the pronouns may have been able to process theinformation at a deeper level and activate its rehearsal in short-term memory.This may have helped them obtain significantly different results from those ofthe second-semester students who did not mention the pronouns in the input.The first-semester students’ more limited processing capacity (VanPatten 1996;Wickens 1989) may have prevented them from using the input they noticed withthe same success as the second-semester students.9

Regarding the comparison of the results based on the concurrent and theretrospective reports, in the statistical comparisons the results for concurrent



and retrospective protocols agreed in all but two of the measures considered.One of the differences was in the analysis of variance of the whole database,where no interaction was found between Group and Level based on theretrospective protocols. However, in the descriptive analysis we saw that manyparticipants provided as much information, if not more, in the retrospectiveprotocols as in the concurrent protocols.

Despite the absence of complete agreement between the two different meas-ures, it seems that using both types of verbal reports in the same study mayhelp us obtain richer data. This is particularly important when we try to assessthe attentional processes of the participants in a study. As Leow (2000) points out,exploring the learners’ internal processes by means of multiple data-elicitationmeasures may offer the necessary evidence to better understand how participantsgo about completing a specific task. If only concurrent protocols had been usedin this particular study, we would have counted as many as ten of the participantsamong those who did not notice the targeted forms, because they only commentedon those forms in their immediate retrospective protocols. In this study, thatwould have meant undercounting the number of participants who noticed thetargeted forms by 37%.

A final point regarding retrospective protocols is their level of cost-effectiveness. There are various ways to use them. In some research designsresearchers may need to meet individually with participants, replay a recordingof their performance, and ask them to comment.10 The present study sought toobtain general comments about the task that the participants had performed.Since the data were collected from all the participants at the same time(recording their comments in their individual laboratory booths), the collectionof retrospective reports only added three or four minutes to the total data-collection time.

An issue that needs to be discussed here is the relatively low frequencyof mentions of the target forms that appear in the verbal reports (none ofthe participants made more than three mentions in the concurrent protocols).A possible explanation is the nature of the task. Although the task consisted of16 blanks with three choices for each, which could technically be consideredsimilar to the problem-solving tasks in other previous research, such as thecrossword puzzle used in Leow (1997, 1998a, 1998b, 2000) and the jigsawpuzzle used in Rosa and O’Neill (1999), there was one crucial difference. Inthe current study the 16 blanks were embedded in a long text (237 words).While tasks in previous studies generated much more evidence of awarenessfrom the participants, possibly because they may have focused on each elementof the puzzles one at a time as discrete units, the learners in the present studymay not have felt the need to focus and report on each individual item, sincethey may have approached the task more globally and not as a series of tasks tobe completed one item at a time. A similar explanation is offered in Leow(2001), where a lower frequency of reported noticing was explained by thefact that the target forms were part of a text and not part of a discrete-pointproblem-solving task.

216 JOAQUIM CAMPS


With regard to verbalization of thought processes, it is important to keepin mind a potential limitation of any study involving think-aloud protocols:although such data can provide evidence of awareness of the elements targetedon the part of the learners, lack of verbalization of their thought processes doesnot mean that learners are not aware of the elements they are processing (A.D.Cohen 1987; Seliger 1983). In connection to this point, the effectiveness ofthink-aloud protocols may be limited if participants do not provide enoughuseful data due to a lack of practice with the think-aloud procedure. This studyprovided learners with training on how to perform think-aloud protocols inorder to address this limitation and produce more reliable results. However, itis difficult to assess the degree of familiarity with the procedure the participantsachieved with the practice examples carried out before the main task.

Another factor to keep in mind when carrying out concurrent think-aloudprotocols is the issue of reactivity, that is, the possibility that performing athink-aloud protocol may affect the learners’ thought processes. Jourdenais(2001) offers a discussion of such concerns, while Ericsson and Simon (1993),based on a review of studies in psychology, state that, when carried out properly,think-aloud protocols do not influence the sequence of thought, although theymay increase the amount of time needed to complete a task.11

There are other limitations which are more specific to this study. First, itwas purposely designed to explore only how learners processed information inone particular experimental task, without measuring the possible effects ofexposure to this task in terms of subsequent recognition or production of thetargeted forms. The decision to limit the scope of the study was based on thebelief that it is important to carefully investigate the characteristics of a specifictask before attempting to explore its effects as a possible instructional task in apretest–posttest experimental design. Another limitation is that the discussionin this article is mostly confined to a quantitative analysis of the data. Futurework will benefit from an analysis of the comments in the participants’ verbalreports from a qualitative point of view by comparing the remarks in theirconcurrent protocols to those in their retrospective reports. This type of analysisshould further develop our understanding of the differences between the twotypes of reports and how they can complement each other in investigating howlearners attend to input.

Conclusion

This study adds further evidence to research that has shown the connectionbetween noticing targeted forms and success in linguistic tasks involving theseforms. The results of the study suggest a possible difference between two levelsof beginning learners with regard to the benefits they obtain from attending toforms targeted in the input. From a methodological point of view, this study hasdiscussed the differences and similarities between the results obtained fromconcurrent and retrospective protocols, both at the level of statistical analysis



and at the descriptive level. The evidence shows that retrospective protocolsare an effective complement to the use of concurrent protocols. It would beto the benefit of the field if more studies on the role of attention in SLAimplemented the use of both concurrent and retrospective verbal reports inorder to more fully assess the potential advantages of using both types togetherwhen studying attentional processes.

Acknowledgement

This research was supported by a grant from the College of Liberal Arts and Sciencesat the University of Florida. An early version of this paper was presented at theConference on Form–Meaning Connections in Second Language Acquisition held inChicago in February, 2002. I am grateful to Renée Jourdenais and Gillian Lord fortheir comments on earlier versions of this article. I would also like to thank the twoanonymous reviewers for their suggestions.

Notes

1. Leow (1998b) pointed out this internal validity issue as something affecting manystudies in SLA. Further discussion can be found in Leow (1999).

2. Other on-line measures used to assess learners’ attention to input include eye-tracking and some applications of computer-based instruction. See Tanenhaus andSpivey-Knowlton (1996) and Blake (1998) for respective descriptions.

3. In all, data were collected from 155 participants, but 81 of them were exposed to atype of treatment that will not be discussed in this article.

4. Translation of Spanish words is provided in brackets.5. In some cases, participants made reference to several traits in a given mention, for

example, mentioning gender and number at the same time.6. The two types of protocols also differed in that the retrospective protocols elicited

comments on how the participants performed the task, while the concurrent protocolsdid not.

7. An effect size of .80 or greater is considered a large effect. Values between .50 and.80 indicate a medium effect, while anything below .50 indicates a small effect(Norris and Ortega 2000, following J. Cohen’s 1988 recommendation).

8. See note 5.9. As one of the reviewers pointed out, the differences in processing may also be due to

individual differences between learners. It is plausible that a specific first-semesterlearner may have benefited more from attending to the input than a second-semesterstudent, despite the fact that the group scores are higher for the second semester.For example, in the analysis based on the participants who mentioned pronouns intheir concurrent protocols, the top four learners in the first-semester group had higherscores than the two weakest learners in second-semester group. However, 8 of the10 learners in the second-semester group had higher scores than all the learners inthe first-semester group.

10. An example of this would be the application of stimulated recall methodology (Gassand Mackey 2000).

218 JOAQUIM CAMPS


11. Leow and Morgan-Short (2002) empirically addressed the issue in the context of areading task in Spanish as a foreign language and found no reactivity effects – thatis, no difference between learners who performed think-alouds and those who did not.

References

Alanen, R. (1995) Input enhancement and rule presentation in second languageacquisition. In R. Schmidt, Attention and awareness in foreign languagelearning (Technical Report #9). Honolulu: University of Hawai’i, SecondLanguage Teaching and Curriculum Center. 259–302.

Blake, R.J. (1998) The role of technology in second language learning. InH. Byrnes, Learning foreign and second languages. New York: The ModernLanguage Association of America. 75–104.

Cohen, A.D. (1984) On taking language tests: what the students report. LanguageTesting 1: 70–81.

—— (1987) Using verbal reports in research on language learning. In C. Faerchand G. Kasper, Introspection in second language research. Clevedon:Multilingual Matters. 82–95.

—— and E. Olshtain (1993) The production of speech acts by ESL learners.TESOL Quarterly 27: 33–56.

Cohen, J. (1988) Statistical power analysis for the behavioral sciences (2nded.). Hillsdale, NJ: Erlbaum.

Cumming, A. (1989) Writing expertise and second language proficiency.Language Learning 39: 81–141.

—— (1994) Writing expertise and second-language proficiency. In A. Cumming,Bilingual performance in reading and writing. Ann Arbor: John Benjamins.173–221.

Davis, J. and L. Bistodeau (1993) How do L1 and L2 reading differ? Evidencefrom think aloud protocols. The Modern Language Journal 77: 459–72.

Ericsson, K.A. and H.A. Simon (1993) Protocol analysis. Cambridge, MA: MITPress.

Gass, S. and A. Mackey (2000) Stimulated recall methodology in secondlanguage Research. Mahwah, NJ: Lawrence Erlbaum.

Hosenfeld, C. (1977) A preliminary investigation of the reading strategies ofsuccessful and nonsuccessful second language learners. System 5: 110–123.

Izumi, S. and M. Bigelow (2000) Does output promote noticing and secondlanguage acquisition? TESOL Quarterly 34: 239–78.

Jourdenais, R. (1998) The effects of textual enhancement on the acquisition ofthe Spanish preterit and imperfect. Unpublished doctoral dissertation,Georgetown University, Washington, DC.

—— (2001) Cognition, instruction and protocol analysis. In P. Robinson,Cognition and second language instruction. Cambridge University Press.354–75.

—— , M. Ota, S. Stauffer, B. Boyson, and C. Doughty (1995) Does textualenhancement promote noticing? A think-aloud protocol analysis. In R.



Schmidt, Attention and awareness in foreign language learning (TechnicalReport #9). Honolulu: University of Hawai’i, Second Language Teachingand Curriculum Center. 183–216.

Kasper, G. (1998) Analysing verbal protocols. TESOL Quarterly 32: 358–62.Kormos, J. (1998) Verbal reports in L2 speech production research. TESOL

Quarterly 32: 353–8.—— (2000) The role of attention in monitoring second language speech

production. Language Learning 50: 343–84.Leow, R.P. (1997) Attention, awareness, and foreign language behavior.

Language Learning 47: 467–505.—— (1998a) The effects of amount and type of exposure on adult learners’ L2

development in SLA. Modern Language Journal 82: 49–68.—— (1998b) Toward operationalizing the process of attention in second language

acquisition: evidence for Tomlin and Villa’s (1994) fine-grained analysis ofattention. Applied Psycholinguistics 19: 133–59.

—— (1999) The role of attention in second/foreign language classroom research:methodological issues. In J. Gutiérrez-Rexach and F. Martínez-Gil, Advancesin Hispanic linguistics. Somerville, MA: Cascadilla Press. 60–71.

—— (2000) A study of the role of awareness in foreign language behavior:aware versus unaware learners. Studies in Second Language Acquisition22: 557–84.

—— (2001) Do learners notice enhanced forms while interacting with the L2?An online and offline study of the role of written input enhancement inL2 reading. Hispania 84: 496–509.

—— and K. Morgan-Short (2002) To think aloud or not to think aloud: theissue of reactivity in SLA Research methodology. Paper presented at theConference on Form and Meaning Connections in SLA, Chicago, IL.

Mangubhai, F. (1991) The processing behaviours of adult second languagelearners and their relationship to second language proficiency. AppliedLinguistics 12: 268–98.

Norris, J.M. and L. Ortega (2000) Effectiveness of L2 instruction: a researchsynthesis and quantitative meta-analysis. Language Learning 50: 417–528.

Raimes, A. (1994) Language proficiency, writing ability, and composing strategies:a study of ESL college student writers. In A. Cumming, Bilingual performancein reading and writing. Ann Arbor: John Benjamins. 139–72.

Robinson, M. (1991) Introspective methodology in interlanguage pragmaticsresearch. In G. Kasper, Pragmatics of Japanese as native andtarget language (Technical Report #3). Honolulu: University of Hawai’i,Second Language Teaching and Curriculum Center. 29–84.

Robinson, P. (1995) Attention, memory, and the “noticing” hypothesis. LanguageLearning 45: 283–331.

—— (1996) Learning simple and complex second language rules under implicit,incidental, rule-search, and instructed conditions. Studies in SecondLanguage Acquisition 18: 27–67.

220 JOAQUIM CAMPS


—— (1997a) Generalizability and automaticity of second language learningunder implicit, incidental, enhanced, and instructed conditions. Studies inSecond Language Acquisition 19: 223–47.

—— (1997b) Individual differences and the fundamental similarity of implicitand explicit adult second language learning. Language Learning 47: 45–99.

Rosa, E. and M. O’Neill (1999) Explicitness, intake, and the issue of awareness.Studies in Second Language Acquisition 21: 511–56.

Schmidt, R. (1990) The role of consciousness in second language learning.Applied Linguistics 11: 129–56.

—— (1994) Deconstructing consciousness in search of useful definitions forapplied linguistics. AILA Review 11: 11–26.

—— (1995) Consciousness and foreign language learning: a tutorial on therole of attention and awareness in learning. In R. Schmidt, Attention andawareness in foreign language learning (Technical Report #9). Honolulu:University of Hawai’i, Second Language Teaching and Curriculum Center.1–63.

Seliger, H.W. (1983) The language learner as a linguist: of metaphors andrealities. Applied Linguistics 4: 179–91.

Tanenhaus, M.K. and M.J. Spivey-Knowlton (1996) Eye-tracking. Languageand Cognitive Processes 11.6: 583–8.

VanPatten, B. (1996) Input processing and grammar instruction. Norwood,NJ: Ablex.

Wickens, C.D. (1989) Attention and skilled performance. In D.H. Holding,Human skills. New York: John Wiley. 71–105.

[Received 7/9/02; revised 10/12/02]

Joaquim CampsDepartment of Romance Languages and LiteraturesUniversity of Florida170 Dauer Hall, Box 117405Gainesville, Florida 32611-7405USAe-mail: [email protected]



Appendix

Text and multiple-choice task.Read the text and circle the letter of the items below that would better fill in

the blank in every case.

María tiene mucho trabajo en casa los sábados por la mañana. Primero lava____(1) y luego la plancha. Después arregla ____(2) porque lo quiere tener enorden. Ordena ____(3) que tiene en su escritorio y los pone en montones distintos.Recoge ____(4) del mes pasado que tiene por el suelo y las lleva a reciclar.Limpia ____(5) que tiene en su escritorio y la cambia de lugar. Al mediodíaalmuerza. Toma ____ (6) y lo pone en la mesa de la cocina. Saca ____(7) quetiene en el refrigerador, lo pone en el plato, y prepara un sándwich. Toma ____(8) y la pone en el sándwich. También pone tomate y jamón. A María le gustacomprar ____(9) que hay en el mercado del barrio, porque los venden a buenprecio. Toma también ____(10) y las pone al lado del sándwich. Entonces va ala sala de estar, toma ____(11) que está al lado de la mesa, la coloca al lado dela ventana, y se sienta. Mira ____(12) en el balcón, los llama y les da migas depan. Después llega la hora de ____(13). María las mira todos los días en el canal4 de televisión. Se sienta delante de ____(14) que le regaló su madre, y loenciende. Le gusta mucho ver a ____ (15) tan importantes que actúan en esosprogramas y las admira por su belleza. También le parecen muy guapos ____(16), y los contempla con emoción. En fin, María pasa una tarde muy agradablelos sábados.

(1)a. los pantalonesb. la carac. la ropa

(2)a. la habitaciónb. el cuartoc. el almuerzo

(3)a. las cosasb. los teléfonosc. los libros

(4)a. las revistasb. los periódicosc. las blusas

(13)a. las tareasb. las telenovelasc. los noticieros

(14)a. la radiob. el periódicoc. el televisor

(15)a. las sirvientasb. las actricesc. los actores

(16)a. los actoresb. los librosc. las actrices

(5)a. las fotosb. la lámparac. la mesa

(6)a. una botellab. una tazac. un plato

(7)a. la mantequillab. el vinoc. el pan

(8)a. (la) lechugab. (la) cervezac. (el) atún

(9)a. las manzanasb. los tomatesc. los cuchillos

(10)a. unas bananasb. unas cucharasc. un tenedor

(11)a. una revistab. una sillac. un sillón

(12)a. los pájarosb. los dibujosc. las golondrinas

Documents

Concurrent and retrospective verbal reports as tools to better