Qualitative Research in Language Teaching and Learning Journals, 1997–2006

Qualitative Research in LanguageTeaching and Learning Journals,1997–2006PHIL BENSON,1 ALICE CHIK,2 XUESONG GAO,3 JING HUANG,4 and WENFENG WANG5

In the light of widespread discussion of qualitative approaches to language teaching andlearning research in recent years, this article reports on a survey of qualitative data-basedarticles published in 10 major journals between 1997 and 2006. The survey revealed that 477qualitative articles (22% of the total number published) had appeared in the 10 journals overthis period, that the year-by-year totals were relatively stable over the 10-year period of thesurvey, and that there was considerable variation among the journals. An analysis of the articlesin terms of approaches to qualitative research showed that methodological eclecticism, ratherthan adherence to established traditions, is now the dominant characteristic of publishedqualitative work in our field.

IN A REVIEW OF QUALITATIVE RESEARCHin foreign and second language teaching andlearning published a little over a decade ago,Lazaraton (1995) commented that although thenumber of qualitative data-based articles pub-lished in major journals in the early 1990s wassmall, the prospects for qualitative research werepromising. “It remains to be seen,” she wrote,“whether 10 years hence qualitative research willbe on an equal footing with quantitative researchin how frequently it is employed and how it is re-ceived by the profession” (p. 467). More than adecade later, it seems that qualitative research iswell on the way to attaining this status, if it hasnot done so already. Research manuals now rou-

1The Hong Kong Institute of Education, Department ofEnglish, 10 Lo Ping Road, Tai Po, Hong Kong, Email:[email protected] University of Hong Kong, Department of English, TatChee Avenue, Kowloon Tong, Hong Kong, Email: [email protected] Hong Kong Institute of Education, Department ofEnglish, 10 Lo Ping Road, Tai Po, Hong Kong, Email:[email protected] of Hong Kong, Centre for Applied EnglishStudies, Pokfulam Road, Hong Kong, Email: [email protected] of Hong Kong, Centre for Applied EnglishStudies, Pokfulam Road, Hong Kong, Email: [email protected]

The Modern Language Journal, 93, i, (2009)0026-7902/09/79–90 $1.50/0C©2009 The Modern Language Journal

tinely cover both quantitative and qualitative ap-proaches (Allison, 2002; Brown & Rodgers, 2002;Burns, 1999; Dornyei, 2007; McDonough & Mc-Donough, 1997; McKay, 2006; Wallace, 1998), andsome have been entirely devoted to qualitative re-search (Holliday, 2002; Markee, 2000; Richards,2003). There have also been numerous reviewand discussion articles on aspects of qualitative re-search in journals (Bell, 2002; Block, 2000; Davis,1995; Edge & Richards, 1998; Goldstein, 1995;Holliday, 1996, 2004a, 2004b; Kouritzin, 2000;Lazaraton, 1995, 2003; Pavlenko, 2002; Ramathan& Atkinson, 1999; Seedhouse, 2005; Shohamy,2004), as well as in handbooks and encyclope-dias (see, e.g., contributions to Cummins & Davi-son, 2007; Hinkel, 2005; Hornberger & Corson,1997).

Language teaching and learning journals alsoappear to be far more open to the publicationof qualitative data-based studies than they werein the early 1990s. Introducing a special issue ofThe Modern Language Journal (MLJ ) on method-ology, epistemology, and ethics in research, Mag-nan (2005), for example, observed that “the MLJtoday reflects the discipline in accepting a vari-ety of methodologies in both quantitative andqualitative paradigms” (p. 315). After a decadein which reliance on quantitative methods wasquestioned, she argued, “our discipline now em-braces a variety of qualitative methods as accepted,or even preferred, methods of inquiry” (p. 315).Published surveys of journal articles lend some

80 The Modern Language Journal 93 (2009)

support to this view. Surveying the contents ofLanguage Learning (LL), MLJ , Studies in SecondLanguage Acquisition (SSLA), and TESOL Quar-terly (TQ ) from 1991 to 1997, Lazaraton (2002)found that 10% of the data-based articles usedqualitative approaches. Extending the survey upto 2001, Lazaraton (2005) found that this figurehad risen to 14%, with the proportion of qual-itative articles reaching 40% in TQ . CoveringApplied Linguistics (AL), International Review ofApplied Linguistics in Language Teaching (IRAL),MLJ , and TQ from 1985 to 1997, Gao, Li, andLu (2001) found that approximately 32% of thedata-based articles used qualitative approachesand that the proportion had risen from 1991 on-ward. Estimates of the proportions of qualitativeresearch articles in language teaching and learn-ing journals will clearly vary according to thejournals selected for review. The differences be-tween Lazaraton’s (2002, 2005) figures and thosereported in Gao et al., for example, are largelyexplained by the replacement of LL and SSLAwith AL and IRAL in the latter survey. Never-theless, both of these surveys provide some evi-dence of the more frequent employment of qual-itative research methods that Lazaraton (1995)anticipated.

We know far less, however, about patternswithin published qualitative research on languageteaching and learning. Commentators on quali-tative research also vary in their assessments ofthe most important approaches within the field.Davis (1995), for example, identified qualitativeresearch on language teaching and learning withwork in the traditions of language socializationand the ethnography of communication, whereasLazaraton (2003) identified ethnography andconversation analysis as “the two forms of quali-tative research . . . in which applied linguists cur-rently engage” (p. 2). Gao et al. (2001), however,referred to a much broader range of approaches,including text analysis, ethnography, narrativeaccounts, verbal reports, and classroom interac-tion research. Richards (2003) discussed seven“core traditions” (p. 13) in qualitative researchthat are relevant to language teaching and learn-ing: ethnography, grounded theory, phenomenol-ogy, case study, life history, action research, andconversation analysis. Richards’s extensive biblio-graphy, however, included very few references todata-based language teaching and learning stud-ies. Like many other contributors to the literatureon qualitative research, Richards pays more atten-tion to the potential applications of approachesused in the humanities and social sciences than

he does to their actual use in research on lan-guage teaching and learning.

This article therefore sets out to achieve twogoals. First, it aims to update and expand uponearlier surveys through a 10-year survey of qual-itative research in 10 major language teachingand learning journals. Second, it aims to assesspatterns of approach within the qualitative stud-ies published in these journals. By adopting a“bottom-up” approach to the identification ofthese patterns, based on a detailed examinationof 477 published articles, we aim to explore thesense that researchers in our field are currentlymaking of the idea of qualitative research.

METHODOLOGY

Selection of Journals

The survey reported in this article grew out ofa project to compile a bibliography of qualitativeinvestigations of language teaching and learningfor use by postgraduate research students. Suchstudies are published in a variety of venues andformats, including international, regional, and lo-cal journals, books, conference proceedings, andcollections of working articles. We decided to fo-cus on international journals for two main rea-sons: First, they are now widely (although by nomeans universally) accessible through universitylibrary subscriptions to full-text databases and,second, articles published in these journals arefiltered through rigorous peer-review processesand are therefore more likely to exemplify prac-tices that are acceptable to a wide range of re-searchers. The selection of journals for the sur-vey was initially dependent on our definition ofthe field of research with which we are con-cerned. In this article, we refer to this field us-ing the term “second and foreign language teach-ing and learning” (or, for the sake of brevity,“language teaching and learning”), a term thatrefers strictly to an object of inquiry rather thana field of research. Our interest, in other words,lies in investigations of the people, situations, pro-cesses, and texts involved in language teachingand learning. According to our understanding,this is one area of research within the broaderfield of applied linguistics, which also covers ar-eas like lexicography, translation, and speechtherapy, as well as interactional sociolinguistics,language socialization, systemic linguistics andgenre theory, critical discourse analysis, socialpsychology of language, and language in institu-tional settings (Rampton, 1997; van Lier, 1997).

Phil Benson et al. 81

Although the boundaries of the various branchesof applied linguistics are porous, our selection ofjournals is confined to those whose editorial poli-cies give prominence to language teaching andlearning.

The selection was further narrowed down us-ing Jung’s (2004) list of the “top-most” languageteaching and learning journals, assessed by ab-stracts published in the British journal LanguageTeaching between 1996 and 2002. Jung arguedthat because Language Teaching reviewers followrigorous selection procedures, the number of ab-stracts selected from a particular journal is a goodindication of its quality. (This argument may seemsomewhat contentious, if only because some jour-nals publish a greater number of articles annu-ally than others.) Nevertheless, Jung’s list of the12 journals with more than 100 abstracts in Lan-guage Teaching between 1996 and 2002 seemed areasonable starting point from which to samplethe major language teaching and learning jour-nals. Of these 12 journals, ELT Journal and Lan-guage Learning Journal were ultimately excludedfrom our survey because they tend to publish rel-atively short articles, in which research methodsare not always clearly and completely described.Etudes de Linguistique Appliquee was also excludedbecause only one of the members of the researchteam knew French. In order to cover the six jour-nals surveyed by Lazaraton (2002, 2005) and Gaoet al. (2001), IRAL—the 13th journal on Jung’slist (personal communication)—was added to ourlist. Our sample of 10 journals thus included the 6appearing in these surveys—AL, IRAL, LL, MLJ ,SSLA, and TQ —plus Canadian Modern LanguageReview (CMLR), Foreign Language Annals (FLA),Language Testing (LT ), and System.

Whether these journals are, in fact, truly rep-resentative of the “top-most” language teachingand learning journals remains a matter for de-bate; for example, the inclusion of Etudes de Lin-guistique Appliquee in Jung’s (2004) list notwith-standing, there seems to be a bias toward English-language journals published in North Americaand the United Kingdom. Furthermore, althoughour sample includes at least one specialized jour-nal within the field of language teaching andlearning, LT , it excludes many others. More re-cently, Egbert (2007) has used a combination ofdifferent quality indicators to compile a list ofthe top seven language teaching and learningjournals. Although Egbert did not explicitly re-fer to Jung’s article, her list included 6 journalsfrom his top 10 (AL, ELTJ, LL, MLJ , SSLA, TQ )and only 1 journal that does not appear on his

list (Journal of Second Language Writing). Her listdoes seem to offer some support to the selectionof journals in our study. Once again, however,we make no greater claim for this selection thanthat it is a reasonable starting point from whichto undertake a survey of the contents of majorlanguage teaching and learning journals. Difficul-ties in interpreting the qualitative research data inthese journals relative to the wider context of pub-lications on language teaching and learning willbe discussed after the data have been presented.

The Survey

The starting point for the survey was set at 1997,the earliest year for which the full texts of all10 journals were available in online databases. Theabstracts and methods sections of all substantiveresearch published in English in these journalsover the period 1997–2006 were examined in de-tail. Following Gao et al. (2001), we began fromthe assumption that this research falls into threemajor groups: nonempirical studies (mainly the-oretical or position articles based on secondarydata), quantitative empirical studies, and qualita-tive empirical studies. Within this framework, ar-ticles were identified as qualitative if the authorsclaimed to have adopted a qualitative approach,or if they adopted comparable approaches with-out making this claim explicitly. This second cat-egory is an important one because authors oftendescribe their research methods without attach-ing a formal label (i.e., quantitative or qualitative)to them.

In the first round of review, each of the fiveresearchers engaged in the project reviewed thecontents of two journals. Using the criteria out-lined earlier, we focused on identifying articlesthat could be counted as qualitative for the pur-poses of the survey. Problematic cases were dis-cussed in a meeting of the whole team and de-cisions were reached by consensus. Articles thatproved difficult to assess were of two main kinds:those in which the sources of data or methodsof analysis were difficult to identify and those inwhich the balance between quantitative and qual-itative data was in question. Although these ar-ticles generated a great deal of discussion, theyturned out to be relatively few in number. In theend, we decided to count articles using a mix ofqualitative and quantitative approaches if thequalitative data were reported and discussed indetail. As we tended to err on the side of inclu-sion, we have more likely overestimated, ratherthan underestimated and, based on the number


of articles that proved difficult to assess, we esti-mate a possible “margin of error” of around 5%.

Three further rounds of review focused on theapproaches to research and methods of data coll-ection and analysis used in the articles we hadidentified as qualitative. In the first of these threerounds, articles were placed into broad categories(e.g., ethnography or conversation analysis) basedon stated approaches to data collection and anal-ysis in the title, abstract, or methods sections. As amajority of the articles remained uncategorized,another round of review, which involved a closerreading of methods sections, was conducted. Thisled to a categorization of all of the articles in termsof their overall approach. Our final round of re-view involved specification of the means of datacollection and analysis described in the methodssections of the articles, as compared to actual dataanalysis reported in the findings sections. Again,meetings were held after each round to resolve un-certainties concerning individual articles as well asto refine criteria. The journals also rotated fromresearch team member to team member aftereach round, so that by the end of the three roundsof review, each researcher had examined in somedetail all of the qualitative articles published in atleast 6 of the 10 journals.

RESULTS

An Overview

The raw survey data from the first round ofreview are shown in Table 1, which lists qualita-

TABLE 1Data-Based Qualitative Articles as a Proportion of Total Research Articles Published in 10 Major Journals,1997–2006

Journal 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 1997–2006 %

AL 4/20 9/19 7/19 4/19 5/18 7/16 6/21 5/19 6/20 9/23 62/194 32%CMLR 3/15 6/15 6/16 10/18 4/23 11/19 8/15 6/20 7/23 5/14 66/178 37%FLA 6/35 6/36 5/30 3/42 8/35 8/39 6/41 13/41 6/43 13/48 74/390 19%IRAL 4/13 2/13 2/15 0/19 2/13 4/13 1/16 1/19 0/19 7/16 23/156 15%LL 3/16 2/26 1/28 0/19 0/19 3/21 3/25 2/24 2/21 1/16 17/215 8%LT 1/10 1/10 0/15 1/9 3/9 1/11 2/12 5/15 0/19 3/18 17/128 13%MLJ 4/26 4/23 6/22 4/24 7/24 7/24 5/22 10/22 4/26 4/24 55/237 23%SSLA 0/20 0/19 2/23 3/18 2/19 1/27 0/18 0/20 0/16 1/19 9/199 5%System 6/37 8/33 6/33 6/33 7/29 7/27 5/28 6/34 9/33 16/37 76/324 23%TQ 10/19 7/13 7/18 8/16 8/14 4/18 7/17 12/17 9/22 6/27 78/181 43%Total 41/211 45/207 42/219 39/217 46/203 53/215 43/215 60/231 43/242 65/242 477/2202 22%

Total% 19% 22% 19% 18% 23% 25% 20% 26% 18% 27% 22%

Note. Figures for CMLR and IRAL exclude articles published in French.

tive articles as a proportion of the total numberof research articles (qualitative, quantitative, ornonempirical) published in each of the 10 jour-nals year by year. The total number of articlesidentified as qualitative amounts to 477, or 22%of the 2,202 articles examined. This figure in-cludes 24 articles using both qualitative and quan-titative methods. Once again, because the surveywas intended to assess the overall contribution ofqualitative research to journals as well as to iden-tify patterns within qualitative research, we didnot focus on the relative contributions of quan-titative and nonempirical articles. For this rea-son, our percentages are not directly comparableto those from Lazaraton (2002, 2005) and Gaoet al. (2001) reported earlier in this article, all ofwhich referred to proportions of qualitative arti-cles among data-based articles only. Nevertheless,it is worth noting that our figure for qualitativearticles as a percentage of the total number ofarticles (22%) is actually higher than Lazaraton’sfigures for percentages of data-based articles only(10% in 2002, 14% in 2005). It is also higherthan the percentage of the total number of ar-ticles that Gao et al. surveyed (18%). For the fourjournals covered by Gao et al., our figure risesto 28%. Such comparisons suggest an overall risein the number of qualitative articles publishedin major language teaching and learning journalssince the early 1990s. We state this conclusion withsome caution, however, because there appears tobe no clear upward trend in the data within the10 years of the survey. Rather, year-by-year per-centages tend to fluctuate around the 20% mark,with the highest recorded in 2004 (26%) and 2006


(27%) and the lowest in 2000 and 2005 (both18%).

The most striking feature of Table 1 is, per-haps, the variation among journals, with 6 of the10 journals (AL, CMLR, FLA, MLJ, System, andTQ ) accounting for 86% of the qualitative arti-cles. Among the 6 journals that appear to favorqualitative articles, TQ displays both the highestpercentage (43%) as well as the highest overallnumber of articles (78). In percentage terms, TQis followed by CMLR (37%) and AL (32%), butin terms of the total number of articles published,System (76) and FLA (74) rank second and third,respectively. (Without access to data on rejectionrates for qualitative submissions, however, it is dif-ficult to assess the significance of these figures.)On the surface, it would seem that editorial poli-cies and refereeing processes may favor qualita-tive research in some journals and work againstit in others. However, it is also likely that qualita-tive researchers will have chosen to submit theirarticles to journals that publish qualitative arti-cles regularly and to avoid those that do not. Ifthis is the case, the emerging patterns of variationamong journals we surveyed could well becomepolarized. Indeed, this may help to explain thefigures for 2005—a year in which the 6 journalsmost favorable to qualitative research accountedfor 41 of the 43 total qualitative articles published.In this same year, 3 of the other 4 journals sur-veyed (IRAL, LT , SSLA) published no qualitativearticles at all.

In spite of these variations, qualitative researchplays some role in all of the journals we surveyed.Furthermore, if these journals can be consideredto be collectively representative of publicationtrends in the field, we must conclude that qual-itative studies now play an important role in lan-guage teaching and learning research. Althoughthe year-by-year numbers of qualitative researcharticles have been relatively constant over the10 years of the survey, their publication has hada cumulative effect. The most striking figure wehave to report, then, is the total of 477 qualita-tive studies published in major journals between1997 and 2006. This 10-year period represents abank of studies that is readily available as a re-source for the investigation of methodological is-sues concerning the use of qualitative approachesand methods in investigations of language teach-ing and learning.

Approaches to Qualitative Research

We now turn to the patterns that can be ob-served within the 477 articles that we identified as

qualitative. This is addressed through the notionof “approach,” which refers simply to any meansof collecting data that is distinct and recogniz-ably different from other methods of carrying outresearch. From this perspective, the labels qualita-tive and quantitative research refer to two broad,separate approaches, distinguishable from eachother by a number of features (not all of whichneed be present in a particular study). McKay(2006), for example, listed nine contrasts betweenqualitative and quantitative types of research, in-cluding assumptions each type holds about thenature of reality, the roles of researchers, thepurposes of research, research questions, and avariety of issues concerned with research designand methods. We further assume that particularqualitative approaches can be identified in simi-lar ways. Ethnography and Conversation Analysis,for example, are grounded in well-established re-search traditions and are identifiable as distinctivequalitative approaches by features concerned withphilosophy, purpose, and methods that ethnog-raphers and conversation analysts adopt in theirresearch designs. However, distinct approachesto qualitative research are not always so easilyidentified in published work. There is a cleardifference in approach, for example, betweenqualitative studies based on interviews or on ob-servation. However, this is more a question ofdata collection methods than of identificationwith an established research tradition. As we havementioned, our attempt to identify patterns ofapproach within the articles identified as qualita-tive, therefore, involved three separate cuts intothe data. The first was based on explicit identifi-cation with established qualitative research tradi-tions, the second was based on comparability ofapproach, and the third was based on methods ofdata collection and analysis.

Qualitative Traditions

As we noted earlier, the articles we identifiedas qualitative met one of two criteria: Either theauthors claimed to have used a qualitative ap-proach or the research methods were compara-ble to those used in articles in which similar claimswere made. In articles matching the first criterion,authors identified the study either with a partic-ular qualitative research tradition or with qualita-tive research more generally in the title, abstract,or methods section. The number of articles in thisgroup was 154, a little less than one third of the to-tal; this calculation included 51 articles identifiedas ethnographic studies and 14 as conversationanalysis studies. These figures lend some support


to Lazaraton’s (2003) observation that ethnogra-phy and conversation analysis are the two formsof qualitative research in which language teach-ing and learning researchers engage most often,although articles explicitly identifying with thesetraditions turned out to be a minority among the477 we identified as qualitative. Certainly, our fig-ures suggest that ethnography plays an importantrole in language teaching and learning research.This was less clear in the case of conversation anal-ysis, but, as we will see in the next subsection, therewere also a number of articles that used conver-sation analysis methods but were not labeled asconversation analysis studies. The word qualita-tive was used to describe 70 articles. We shouldemphasize, however, that this refers to the use ofthe word qualitative to describe the overall ap-proach of the study rather than particular meth-ods of data collection or analysis within the study.(Notably, some 24 articles employ a combinationof qualitative and quantitative methods in theirresearch.)

Comparability of Approach

The fact that so many of the articles exam-ined did not explicitly identify their approaches asqualitative posed a slight problem for our attemptto establish patterns of approach within the arti-cles surveyed. Initially, it seemed, overcoming thisdifficulty would depend ultimately on a detailedexamination of methods of data collection andanalysis. In this subsection, however, we present anintermediate solution to the problem, one basedon other types of identification and comparabilityof approach.

In the previous subsection we indicated thatthe articles explicitly identified with qualitativeresearch traditions made up less than one thirdof the total. This figure is somewhat misleadingbecause many articles (again, around one third ofthe total) identified with approaches that cross thequalitative–quantitative divide. These includedcase studies, longitudinal studies, discourse analy-ses, and classroom interaction research and think-aloud studies. Think-aloud studies in which theprotocols were analyzed qualitatively were, for ex-ample, usually described simply as “think-aloud”studies. Strictly speaking, these terms do not, inand of themselves, imply a qualitative approachto research, and using them to describe a studydoes not mean that study was carried out by quali-tative means. Nevertheless, we came to believe theterms could be interpreted as designations of theapproach to qualitative research taken in many ofthe studies investigated.

Furthermore, we observed that the remainingthird of the articles, which did not specify theiroverall approach, could be placed in one categoryor another based on comparability of approach;for example, there were a number of articles thatused a different label entirely—that of “narrativestudy,” or other closely related terms—to describetheir approaches. Finally, there were a number ofarticles in which the approaches were unnamedbut seemed very similar to the approaches andmethods used in the articles described as narra-tive studies. Both of these groups of articles (the“named” and the “unnamed” narrative studies)were therefore brought together under the sameheading.

Table 2 represents the outcome of an analysisbased on the three assumptions outlined earlier.Each of the categories listed serves as an explicitcategory of identification for some articles; how-ever, the figures for each category also includeseveral articles that did not explicitly state theirapproaches. The total number of conversationanalysis articles, for example, rose from 14 to 20with the addition of 6 articles that did not iden-tify themselves as conversation analysis studies butwere clearly comparable to those that did. The fig-ures reported in Table 2 are those we were left withafter a protracted series of counts and recounts;we believe they should be treated as approxima-tions, not absolutely certain, final counts.

We also found that similarities between ap-proaches allowed us group them together, leav-ing us with the two broad categories representedby the two columns in Table 2. The main distinc-tion is between investigation of the people, situ-ations, and social processes involved in languageteaching and learning (column A) and investiga-tion of spoken or written texts (column B). This

TABLE 2Approaches to Qualitative Research

A B

Case Study 225 Discourse Analysis 53Ethnography 49 Classroom Interaction 49Longitudinal 19 Conversation Analysis 20Think-Aloud 16 Corpus Study 6Narrative 12 Genre Analysis 4Self-Study 6 Systemic Functional 1

AnalysisStimulated Recall 7Action Research 4Diary Study 4Phenomenology 2Total 344 133


distinction may at first seem somewhat artificial.The studies in column A often make use ofrecorded spoken interactions and written texts ofvarious kinds, whereas several of those in columnB hold the understanding of people, situations,and social processes as their overarching goal. Forthe purposes of this article, however, we believethat the distinction is useful. Making it allows usto separate off a substantial number of articles incolumn B that are strongly focused on text analysisand typically employ single methods. The articlesin column A are generally focused more on peo-ple, situations, and processes involved in languageteaching and learning and use multiple methods.Within this group, articles labeled “case studies”turned out to be something of a catch-all cate-gory, accounting for a little under half of the totalnumber of articles (225 out of 477). Of these 225,only 60 articles explicitly described themselves ascase studies. In effect, we made this a holding cat-egory for articles that clearly employed qualitativemethods but could not be associated with any well-established qualitative research tradition. Becausethe case study group could not be broken downfurther without resorting to detailed specificationof the methods or combinations of methods em-ployed therein, it will be the focus of attentionin the next subsection. In column B, the labelof discourse analysis similarly became a holdingcategory for articles that used text analysis proce-dures without explicitly subscribing to a particularapproach.

The ratio between the articles in columns Aand B in Table 2 is around 2.5 to 1. Recast-ing Lazaraton’s (2003) observation, we thereforesuggest that “case study” and “discourse analy-sis,” rather than “ethnography” and “conversa-tion analysis,” might be considered the two ma-jor forms of qualitative research in which lan-guage teaching and learning researchers engage.(Within this framework, we argue that case stud-ies play the more prominent role, at least interms of the number of articles published.) Wealso note, however, that many of the articles wecategorized as “case studies” are methodologi-cally comparable to those described elsewhereas ethnographic. Similarly, many of the articlesdescribed as classroom interaction studies aremethodologically comparable to those describedas conversation analysis studies. Although explicitidentification with ethnography and conversationanalysis as qualitative research traditions appearsto be the minority tendency, we nevertheless notethat these traditions may be important poles ofattraction.

Methods of Data Collection and Analysis

Our third attempt to identify patterns of ap-proach was based on a more detailed examina-tion of the methods of data collection and analysisreported in each article. The 133 articles in Ta-ble 2, column B can be dealt with briefly becausethe relationship between approach and methodtends to be relatively straightforward: Most makeuse of a single type of data. The 69 articles de-scribed as conversation analysis or classroom in-teraction studies were typically based on audiorecordings, although a small number used videoinstead. In most cases, these articles employed tra-ditional conversation analysis methods, with somevideo studies employing innovative methods de-signed to record and analyze gesture as well asspeech. The remainder used written or printedtexts, which were analyzed using a variety of estab-lished discourse analysis techniques.

The articles in Table 2, column A, on the otherhand, present a more complex picture. Table 3shows the breakdown of these articles in terms ofthe number of different types of data as reportedin the methods sections of the original studies. Ofa total of 344 articles, 103 were based on a sin-gle data collection type, 157 used a combinationof two or three types, and 71 used four or moretypes (in 13 articles the sources of data were un-clear). Among the studies that use a single datacollection type, we identified 12 distinct types: in-terview transcripts; observation records and fieldnotes; samples of student work; audio recordingsof interactions; diaries and journals; policy, cur-riculum, and course documents; think-aloud pro-tocols; stimulated recall protocols; open-endedquestionnaire responses; video recordings; courseevaluation data; and teachers’ ratings of studentwork. We also identified two additional sources ofdata that had been used only in combination withothers: audio recordings of class and group discus-sions, and focus group interviews. As single datasources, interview transcripts (29), samples of stu-dent work (17), and observation records and fieldnotes (11) were the most frequently employed.

TABLE 3Single-Method and Multimethod Studies

Types of Data Collection Total Number of Articles

1 1032/3 1574+ 71Unclear 13


Data collection patterns in multimethod studiesproved difficult to identify for two reasons. Amongthe 157 articles using two or three methods, wefound 50 different combinations of methods. Al-though we did not attempt to count them, therewere almost as many combinations among the 71articles using four or more methods as there werearticles. An additional difficulty in interpretingthese data was that in many of the multimethodstudies, the findings section covered only one ortwo of the data types mentioned in the methodssection. The pattern that emerges once again isthe prevalence of interviews, observations, andsamples of student work. Among the 260 articlesemploying three or fewer data collection types, in-terviews were used in 109, observations in 77, andstudent work in 58. Interviews and observationswere the most frequently combined pair, with atotal of 59 articles. Interviews, observations, andstudent work were used in combination in many ofthe 71 articles using more than three methods ofdata collection; audio-recorded interactions wereused in more than 30 of these studies. In an effortto simplify a rather complex picture, we reiter-ate that qualitative research on language teach-ing and learning tends to draw on four main datacollection types—interviews, observation records,samples of student work, and audio recordings—and that these can be used both singly and in avariety of combinations.

Patterns in data analysis proved yet more dif-ficult to identify. At this point, in fact, we wereforced to abandon attempts at strict categoriza-tion. One additional complication was that in con-trast to data collection, data analysis proceduresare often described rather vaguely (if at all) inmost of the articles surveyed. A second difficultyis that, often, data discussed in the findings areonly a fraction of the data collected and describedin the methods section of a study. The criteria forselecting the data to be discussed are also oftenleft unexplained. A final complication was in theuse of terms such as “grounded theory” and “con-versation analysis” for procedures that were notalways explained in detail. As we noted earlier,however, articles concerning spoken and writtentext did tend to employ well-established methodsof analysis (if these were not explained in advancein the methods section, the findings section usu-ally spoke for itself).

In case studies and ethnographic articles, wefound that there is often a rather loose fit be-tween the methods and findings sections. Thesearticles often refer to “grounded theory” or a sim-ilar approach, but the analytical procedures lead-ing to the identification of specific conclusions

or themes are neither explained in the findingssection nor apparent on their own. Although itdoes make categorizing such articles a bit morecomplicated, this should not be interpreted ascriticism; qualitative data analysis frequently andlegitimately involves processes of interpretationand insight that are difficult to describe in strictlyprocedural terms. We also observed that in arti-cles based on large quantities of data, refereedjournal format (i.e., length restrictions) meantthe authors summarized findings and illustratedthem with data extracts, rather than developingarguments through data analysis.

One striking pattern that did emerge in rela-tion to data analysis and presentation, however,was a tendency across all of the types of paperswe have discussed so far to base analysis on writ-ten data extracts. Where the data do not origi-nally come in the form of written text, they aretypically transcribed before analysis. One effectof this is a tendency in multimethod case studiesto use transcriptions of audio or video record-ings, rather than observation schedules or fieldnotes, as observational evidence. And althoughinterview and observation are the most frequentlycombined data collection methods, the observa-tional data usually play a supplementary role; find-ings sections are built around extracts from inter-view transcripts, while observational data tend tobe used as a source of contextual information.We believe that can be attributed to two domi-nant writing strategies that we observed. The firstbegins from data extracts, which are analyzed “be-fore the reader’s eyes,” as it were, in order to gen-erate findings—the strategy most often used indiscourse analysis and conversation analysis stud-ies. The second begins from findings, often pre-sented as themes emerging from the data, whichare produced “behind the scenes” and illustratedin the published article by data extracts—a strat-egy that we found to be more typical of case stud-ies and ethnographies. The preference for textualdata noted earlier could arise from the fact thatboth of the strategies mentioned seem to workbest with data that can be represented in quotableform.

Mixing Quantitative and Qualitative Methods

Finally, we would like to make a brief com-ment on multimethod studies that use both quan-titative and qualitative data. The number of ar-ticles in this category was initially quite large;most quantitative studies we surveyed appearedto make use of at least some qualitative data.Questionnaire studies typically included a few


open-ended items in the questionnaire, whereasexperimental studies often made use of rudimen-tary observational data on the circumstances un-der which the experiment was conducted. As wehave previously stated, our criterion for decidingwhether such articles should be considered quali-tative had to do with whether qualitative data werereported in detail. In the vast majority of articlessurveyed, this was not the case. Typically, in oneof these articles, the methods section would men-tion the collection of qualitative data, which wouldbe discussed minimally in the findings section(if at all). Extracts from answers to open-endedquestionnaire items might be included, but onlyto illustrate or back up quantitative evidence. Al-though we do not dispute that many of these stud-ies make use of qualitative data, we felt that doingso was not sufficient for the articles to be catego-rized as qualitative research reports.

What is of interest, however, is that these stud-ies were sharply distinguishable from the 24quantitative–qualitative studies that we did ulti-mately categorize as qualitative. In the latter, com-bining quantitative and qualitative methods wastypically a deliberate strategy used in order toweigh the two types of evidence against each other.In the context of journal publication, therefore,what we have found is not so much a continuumfrom quantitative to qualitative methods as twoquite distinct tendencies. In some cases, thereseems to be a tendency for quantitative studiesto employ qualitative data, but in ways that usuallyhave little influence on the findings of the study.In other cases, the tendency is to combine quanti-tative and qualitative methods in ways that do havean influence on the findings. We believe that thissecond tendency, which was represented by a rela-tively small number of articles, is worthy of furtherinvestigation using a broader base of quantitative–qualitative studies in the literature.

DISCUSSION

The total number of 477 qualitative articlespublished in the 10 journals that we reviewedappears to be a fairly robust estimate, accordingto the criteria we have used. These criteria wereessentially that the articles should be based on pri-mary data and make use of qualitative methods ofdata collection or analysis. Disagreements withinthe research team arose over two kinds of articles:those in which the data were not clearly identi-fied or the findings were only loosely based onthe data described in the methods section, andthose in which the balance between qualitativeand quantitative data seemed to be in question.

These disagreements were usually resolved in fa-vor of inclusion (rather than exclusion) of thearticles in question. As a result, the number ofqualitative articles may have been overestimatedby up to 5%; in other words, according to ourcriteria, we identified around 450 indisputablyqualitative articles. This relatively high level ofagreement among members of the research teamcame as something of a surprise, given that theboundaries between qualitative and quantitativeresearch have often been represented as some-what fuzzy. We found that these boundaries tendto be more sharply drawn in mainstream journalsthan they are in more methodological literature(as well as, possibly, in the field overall, wherethere may be greater tolerance for experimentalor unorthodox approaches to research).

An assessment of the overall contribution ofqualitative approaches to language teaching andlearning research based on our survey would de-pend on two factors: the validity of the criteriathat we used to identify qualitative research andthe relationship of the 10 journals reviewed to thewider body of publications in the field. The mainobjection to our criteria could be that we chose tofocus on studies using qualitative research meth-ods rather than studies representing qualitativeresearch. Qualitative research, it might be argued,is more a matter of underlying philosophy andpurpose than it is a matter of methods of data col-lection or analysis. By this definition, a qualitativestudy is one that is situated within a qualitativetheoretical framework or adheres to the princi-ples of an established qualitative approach, suchas ethnography or conversation analysis. This is-sue was addressed to some degree in our analy-sis of approaches to qualitative research, whichbegan by isolating articles that explicitly identi-fied their approaches as qualitative. We believethat these articles are qualitative in a strongersense than articles that make use of qualitativemethods without explicitly orienting themselvestoward qualitative research as a tradition. Further,if we were to have limited our selection to articlesthat met this criterion, the number of qualitativearticles would have fallen to around 150, or 7% ofthe total number of articles published.

In the context of this survey, we found thatstricter views of what constitutes qualitative re-search became problematic. This was true for sev-eral reasons. First, we often observed a ratherloose relationship between stated theoretical ormethodological frameworks and actual treatmentof data; for example, we observed that multi-method studies often dealt with only a subset ofthe data in the findings section and that extracts


from some data types were used only illustratively.In their review of longitudinal research in lan-guage teaching and learning, Ortega and Iberri-Shea (2005) pointed out that these tendencieswere related both to the constraints of the journalarticle format and academic pressure to publisharticles before full completion of the long-termprojects from which they arise. Ortega and Iberri-Shea also discussed cases where an ethnographicreport “spanned long enough to be consideredlongitudinal in its own right, yet it was unclearwhether it should be considered longitudinal be-cause the results were presented with little atten-tion illuminating change over time” (p. 36) Thepoint here is simply that qualitative journal arti-cles are not always quite what they claim to be.After detailed examination of numerous articles,we have concluded that in terms of treatment ofdata, the differences between articles that stronglyidentify with qualitative research traditions andthose that use qualitative data without drawingsuch connections are often so small that the dis-tinction lacks any real meaning.

Furthermore, we believe that attempts to for-mulate more stringent criteria for identifyingqualitative studies would run the risk of becom-ing too subjective (i.e., of imposing standardsfor qualitative research that make more senseto some researchers than to others). The “Qual-itative Research Guidelines” that first appearedin the Winter 1994 edition of TQ , for exam-ple, were criticized for precisely this reason; theywere largely based on ethnography and set upsomewhat arbitrary epistemological guides forwould-be authors to follow (Lazaraton, 2003). Dis-cussing the expanded guidelines that appearedin the Winter 2003 edition of TQ (TESOL Quar-terly, 2003), which included separate sections on“case study research,” “conversation analysis,” and“(critical) ethnography,” Shohamy (2004) urgedresearchers to be “cautious that the guidelines donot restrict innovations in research designs by im-posing fixed categories and forcing research intosealed boxes” (pp. 728–729). Researchers should,she argued, “feel free to examine a variety ofmodes, to mix and blend different methods inthe long journey towards answering research ques-tions” (p. 729). Our attempt to construct patternsamong the articles identified as qualitative led usto the conclusion that this mixing and blendingis exactly what language teaching and learning re-searchers do with qualitative research methods.Such a conclusion only becomes apparent, how-ever, when casting the net as widely as possiblewith regard to what actually constitutes qualitativeresearch.

Our figure of 477 articles published in 10 jour-nals over a 10-year period hints at a much largerfigure for the field of language teaching andlearning as a whole. The “Publications Received”section of the journal Language Teaching , forexample, regularly lists more than 60 refereedresearch journals and numerous collections of re-search articles are published by commercial pub-lishers and university presses each year. However,although these no doubt include many qualita-tive articles, the variations among the journals wesurveyed suggest that it would be unwise to of-fer an estimate of the number qualitative articlesthey contain based on the figures produced inour research. Our overall estimate of 477 articlesrepresents 22% of the total number of articlespublished in the 10 journals, but the figures forindividual journals range from 43% (TQ ) to 5%(SSLA). We believe that similar variations wouldlikely be found among other journals and publish-ing houses, but we cannot be absolutely certain.We therefore hypothesize that the 477 qualitativestudies identified in this article represent the tipof an iceberg, whose exact size and dimension weare unable to estimate.

Finally, we would like to comment on our resultsin regard to approaches to qualitative researchand their significance in relation to commen-tary on qualitative research methods. Althoughthe results themselves are a bit uncertain, thestrongest pattern to emerge appears to be oneof diversity; in the light of this diversity, thereappears to be a certain mismatch between writ-ing on qualitative research methods and the data-based qualitative research that actually appears inprint in journals. As we have already noted, forexample, Lazaraton (2003) observed that ethnog-raphy and conversation analysis were the twomain approaches in the field. These approachesalso figured prominently among the contribu-tions to the research methods literature cited atthe beginning of this article. Our own survey sug-gests, however, that what these concise labels ac-tually represent is the more diffuse categories of“case study” and “discourse study.” Similarly, wefound that Richards’s (2003) seven core traditionsof qualitative research—ethnography, groundedtheory, phenomenology, case study, life history,action research, and conversation analysis—wereunevenly represented among the qualitative ar-ticles we examined. Commentary on qualitativeresearch methods, it seems, has tended to fore-ground established traditions in the social sci-ences and to neglect the methodological eclecti-cism that appears to be most characteristic of ourfield. Whether such eclecticism is desirable or not,


we propose that at the very least it deserves to berecognized as an important feature of the senseresearchers are attempting to make of qualitativeapproaches within our field at the present time.

CONCLUSION

It seems to us that any survey that aims to as-sess the contribution of qualitative approaches tolanguage teaching and learning research in quan-titative terms is likely to be open to a range ofobjections concerned with sampling and criteriaof selection. It might even be objected that theresults of a survey of this kind can never be mean-ingful in view of the long-standing assumptionthat “consensus on the definitions, principles, andvalue of qualitative research is not necessary, desir-able, or even possible” (Lazaraton, 1995, p. 468).However, we feel that just as rudimentary count-ing can be useful from time to time in qualitativeresearch, it can also be useful in assessing the con-tribution of qualitative research to our field. Thesize of this contribution, measured in terms of thenumber of qualitative articles published in a rela-tively small number of journals, turned out to bea strong confirmation of the opening up of lan-guage teaching and learning research to a varietyof approaches, as Magnan (2005) anticipated.

So far, perhaps, we may have missed an essen-tial point. The contribution of qualitative researchis not to be measured by the number of qualita-tive articles published, but by the ways in whichthe field changes as a result. Although our sur-vey does not cover the period, even the mostcursory review of the contents of major languageteaching and learning journals in the 1970s and1980s shows the clear dominance of quantitativemethods. (We believe that this dominance mayhave contributed to the construction of languageteaching and learning within applied linguistics asa distinct field that was particularly amenable toquantitative methods.) There was, in other words,a narrowing both of the methodological scope ofthe field and of conceptualizations of its objectof inquiry. In consequence, researchers adopt-ing qualitative approaches to the investigation ofissues that were relevant to, but not directly con-cerned with, language teaching and learning pro-cesses may well have chosen to publish elsewhere.

From the early 1990s onward, however, therehave been repeated arguments within the ma-jor journals for less rigid boundaries within thebroader field of applied linguistics as well as formore transdisciplinary approaches to the investi-gation of language teaching and learning (e.g.,Block, 1996; Firth & Wagner, 1997; Rampton,

1997; van Lier, 1997). The growing presence ofqualitative research within mainstream languageteaching and learning journals may therefore be asign not only of greater methodological opennessbut also of an increased awareness of the poten-tial contributions of other disciplines within andbeyond applied linguistics. Two of the journalssurveyed, LL and SSLA, appear to be resistant tothis process. (It may be that the relative absenceof qualitative research in their pages has moreto do with the maintenance of a narrow cogni-tive view of language teaching and learning asan object of inquiry than it has to do with fixedmethodological preferences.) In other journals,on the other hand, the growth in the numbersof qualitative studies appears to correspond to awidening of the scope of research. In effect, thesejournals may now be accepting qualitative articlesthat would earlier have been published in journalsin other branches of applied linguistics.

These comments are largely speculative, how-ever, and go somewhat beyond the scope of thisarticle, in which we have confined our objectivesto establishing the extent to which qualitativeresearch has a presence within major languageteaching and learning journals and to identify-ing the patterns of approach within this research.With these objectives in mind, we feel that themajor contribution of this article has been to es-tablish a basis on which commentary on qualita-tive inquiry into issues of language teaching andlearning might be built. Without losing transdisci-plinary inspiration and perspectives, we hope thatfurther efforts will take into greater account thegrowing body of research published within thefield of language teaching and research itself.

REFERENCES

Allison, D. (2002). Approaching English language research.Singapore: Singapore University Press.

Bell, J. S. (2002). Narrative inquiry: More than justtelling stories. TESOL Quarterly, 36, 207–218.

Block, D. (1996). Not so fast: Some thoughts on theoryculling, relativism, accepted findings and the heartand soul of SLA. Applied Linguistics, 17 , 63–83.

Block, D. (2000). Problematizing interview data: Voicesin the mind’s machine? TESOL Quarterly, 34, 757–763.

Brown, J. D., & Rodgers, T. S. (2002). Doing second lan-guage research. Oxford: Oxford University Press.

Burns, A. (1999). Collaborative action research for Englishlanguage teachers. Cambridge: Cambridge Univer-sity Press.

Cummins, J., & Davison, C. (Eds.). (2007). The interna-tional handbook of English language teaching (Vols.1 & 2). Norwell, MA: Springer.


Davis, K. A. (1995). Qualitative theory and methods inapplied linguistics research. TESOL Quarterly, 29,427–453.

Dornyei, Z. (2007). Research methods in applied linguistics.Oxford: Oxford University Press.

Edge, J., & Richards, K. (1998). May I see your warrant,please? Justifying outcomes in qualitative research.Applied Linguistics, 19, 334–356.

Egbert, J. (2007). Quality analysis of journals in TESOLand Applied Linguistics. TESOL Quarterly, 41,157–171.

Firth, A., & Wagner, J. (1997). On discourse, commu-nication and some fundamental concepts in SLAresearch. Modern Language Journal , 81, 285–300.

Gao, Y., Li, L., & Lu, J. (2001). Trends in research meth-ods in applied linguistics: China and the West. En-glish for Specific Purposes, 20, 1–14.

Goldstein, T. (1995). Interviewing in multicul-tural/multilingual settings. TESOL Quarterly, 29,587–593.

Hinkel, E. (Ed.). (2005). Handbook of research in secondlanguage learning . Mahwah, NJ: Erlbaum.

Holliday, A. (1996). Developing a sociological imagina-tion: Expanding ethnography in international En-glish language education. Applied Linguistics, 17 ,234–255.

Holliday, A. (2002). Doing and writing qualitative research.Thousand Oaks, CA: Sage.

Holliday, A. (2004a). Issues of validity in progressiveparadigms of qualitative research. TESOL Quar-terly, 38, 731–734.

Holliday, A. (2004b). The value of reconstruction inrevealing hidden or counter cultures. Journal ofApplied Linguistics, 1, 275–294.

Hornberger, N. H., & Corson, D. (Eds.). (1997). Researchmethods in language and education (Encyclopedia ofLanguage and Education, Volume 8). Dordrecht,Netherlands: Kluwer.

Jung, U. O. H. (2004). Paris in London revisited or theforeign language teacher’s topmost journals. Sys-tem, 32, 357–361.

Kouritzin, S. (2000). Bringing life to research: Lifehistory research and ESL. TESL Canada Journal ,17(2), 1–35.

Lazaraton, A. (1995). Qualitative research in applied lin-guistics: A progress report. TESOL Quarterly, 29,455–472.

Lazaraton, A. (2002). Quantitative and qualitative ap-proaches to discourse analysis. Annual Review ofApplied Linguistics, 22, 32–51.

Lazaraton, A. (2003). Evaluative criteria for qualitativeresearch in applied linguistics: Whose criteria andwhose research? Modern Language Journal , 87 , 1–12.

Lazaraton, A. (2005). Quantitative research methods.In E. Hinkel (Ed.), Handbook of research in secondlanguage learning (pp. 209–224). Mahwah, NJ: Erl-baum.

Magnan, S. S. (2005). From the editor: Presenting thespecial issue. Modern Language Journal , 89, 315–316.

Markee, N. (2000). Conversation analysis. Mahwah, NJ:Erlbaum.

McDonough, J., & McDonough, S. (1997). Research meth-ods for English language teachers. London: Arnold.

McKay, S. L. (2006). Researching second language class-rooms. Mahwah, NJ: Erlbaum.

Ortega, L., & Iberri-Shea, G. (2005). Longitudinal re-search in second language acquisition: Recenttrends and future directions. Annual Review of Ap-plied Linguistics, 25, 26–45.

Pavlenko, A. (2002). Narrative study: Whose story is it,anyway? TESOL Quarterly, 36, 213–218.

Ramathan, V., & Atkinson, D. (1999). Ethnographic ap-proaches and methods in L2 writing research: Acritical guide and review. Applied Linguistics, 20,44–70.

Rampton, B. (1997). Returning in applied linguistics.International Journal of Applied Linguistics, 7 , 3–25.

Richards, K. (2003). Qualitative inquiry in TESOL. Lon-don: Palgrave Macmillan.

Seedhouse, P. (2005). Conversation analysis and lan-guage learning. Language Teaching , 38, 165–187.

Shohamy, E. (2004). Reflections on research guidelines,categories and responsibilities. TESOL Quarterly,38, 728–731.

TESOL Quarterly. (2003). Some guidelines for con-ducting quantitative and qualitative research inTESOL. TESOL Quarterly, 37 , 147–178.

van Lier, L. (1997). Apply within, apply without? In-ternational Journal of Applied Linguistics, 7 , 95–105.

Wallace, M. J. (1998). Action research for language teachers.Cambridge: Cambridge University Press.

Retraction of Article from MLJ 92.3

The article, “Gender Identity and Homophobia: The Impact on Adolescent Males Studying French” byScott Kissau and Edward Wierzalis, published in The Modern Language Journal , Vol. 92, No. 3, has beenretracted by agreement between the authors, the journal Editor in Chief, Prof. Leo van Lier, and thepublisher, Wiley-Blackwell. The decision to retract was made because of duplicate publication in theCanadian Modern Language Review, Vol. 62, No. 3.

Documents

Qualitative Research in Language Teaching and Learning Journals, 1997–2006