747984378

PLEASE SCROLL DOWN FOR ARTICLE

This article was downloaded by: [PERI Pakistan]On: 22 April 2010Access details: Access Details: [subscription number 778684090]Publisher RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

International Journal of Social Research MethodologyPublication details, including instructions for authors and subscription information:http://www.informaworld.com/smpp/title~content=t713737293

Paradigm Peace and the Implications for QualityAlan Bryman

To cite this Article Bryman, Alan(2006) 'Paradigm Peace and the Implications for Quality', International Journal of SocialResearch Methodology, 9: 2, 111 126To link to this Article: DOI: 10.1080/13645570600595280URL: http://dx.doi.org/10.1080/13645570600595280

Full terms and conditions of use: http://www.informaworld.com/terms-and-conditions-of-access.pdf

This article may be used for research, teaching and private study purposes. Any substantial orsystematic reproduction, re-distribution, re-selling, loan or sub-licensing, systematic supply ordistribution in any form to anyone is expressly forbidden.

The publisher does not give any warranty express or implied or make any representation that the contentswill be complete or accurate or up to date. The accuracy of any instructions, formulae and drug dosesshould be independently verified with primary sources. The publisher shall not be liable for any loss,actions, claims, proceedings, demand or costs or damages whatsoever or howsoever caused arising directlyor indirectly in connection with or arising out of the use of this material.

Int. J. Social Research MethodologyVol. 9, No. 2, April 2006, pp. 111126

ISSN 13645579 (print)/ISSN 14645300 (online) 2006 Taylor & Francis DOI: 10.1080/13645570600595280

Paradigm Peace and the Implications for QualityAlan Bryman

Taylor and Francis LtdTSRM_A_159511.sgm10.1080/13645570600595280International Journal of Social Research Methodology1364-5579 (print)/1464-5300 (online)Original Article2006Taylor & Francis92000000April [email protected]

In this article it is shown that the paradigm wars that raged concerning the incompatibilityof quantitative and qualitative research have largely subsided. In the process, discussions ofepistemological and ontological issues have become less prominent. The peace that hasbroken out has proved to be more favourable to research combining quantitative and qual-itative research than was the case during the paradigm wars. Drawing on interviews withsocial researchers who employ a mixed-methods approach and on the literature, it is shownthat a spirit of pragmatism with regard to combining quantitative and qualitative researchprevails which encourages researchers to consider using mixed-methods research when theresearch question is suited to it. However, the issue of which quality criteria should beemployed in investigations combining quantitative and qualitative research has not beengiven a great deal of consideration. The author argues for a contingency approach, in whichissues to do with quality are decided in relation to the nature of the study.

Introduction

The debate about quantitative and qualitative research has oscillated between philo-sophical and technical levels of discussion (Bryman, 1984). At the former level,authors have drawn attention to the different epistemological and ontologicalassumptions underpinning quantitative and qualitative approaches to social researchand the research methods with which they are associated. Such discussions are oftenconstrued in terms of a clash between positivism or post-positivism, on the onehand, and a broadly interpretivist approach, founded on such traditions as phenome-nology, symbolic interactionism, hermeneutics and Verstehen, on the other. At the

Alan Bryman is Professor of Organizational and Social Research in the Management Centre, University of Leices-ter, England. His main research interests are in research methodology and leadership studies. He is the author ofSocial Research Methods (Oxford University Press, 2001, 2004). Correspondence to: Alan Bryman, ManagementCentre, Ken Edwards Building, University of Leicester, University Road, Leicester LE1 7RH, UK. Tel.: +44 (0)116252 2790; Email: [email protected]

Downlo

aded

By:

[PE

RI P

akis

tan]

At:

08:

30 2

2 Ap

ril

2010

112 A. Bryman

technical level, quantitative and qualitative research refer to collections of techniques.Technique most often refers to the methods employed for collecting data and to theapproach taken to the analysis of data, but could be taken (and indeed has beentaken by some writers) to include other aspects of the research process, such assampling.

The significance of the distinction between philosophical and technical levels ofdiscussion is that they represent contrasting perspectives in connection with thefeasibility of combining the two approaches. Typically, writers who emphasizeepistemological issues have depicted quantitative and qualitative research as based onincompatible principles, and therefore as not capable of being combined. Any apparentcombination tended to be portrayed as superficial and therefore as not constituting agenuine rapprochement (e.g. Smith & Heshusius, 1986). By contrast, the technicalversion of the account of the differences between quantitative and qualitative researchtypically allowed for, and indeed frequently encouraged, the merging of the twoapproaches within single research projects.

In this article, I examine some of the literature concerning the debate about quanti-tative and qualitative research and in particular the suggestion in several quarters thatthe cleavages between them have been bridged, so that investigations that combinethem become feasible. I then examine some of the implications for quality criteria ofthe dtente that has largely replaced the antimonies. Specifically, given that it is some-times suggested that quantitative and qualitative research should be judged accordingto different criteria, what are the implications of quality criteria for studies thatcombine the two approaches?

Research Methods

Side by side with this examination of the literature, I present the views of 20 socialresearchers working in the UK whom I interviewed in 2004. These researchers areessentially a purposive sample generated from an examination of articles published inbooks and journals during the period 19942003. The interviews were conducted witha semi-structured interview guide. The interviews were concerned with the researchersviews on and practices in relation to the integration of quantitative and qualitativeresearch. The typical interview lasted around 45 minutes. Interviewees varied betweensenior figures in the field and relatively new researchers. They conducted their researchin at least one of the following fields: sociology, social psychology, human and culturalgeography, media and cultural studies, and organization studies.

In addition, I draw very briefly upon a content analysis of articles using bothquantitative and qualitative research in refereed journals. I searched the Social SciencesCitation Index (SSCI) for articles in which relevant key words or phrases such as quan-titative and qualitative, multi(-)method, mixed method or triangulationappeared in the title, key words or abstract. The sample therefore comprises articleswhich to some degree foreground the fact that the study is based on both quantitativeand qualitative research. Searches using other kinds of key words, such as survey andethnography/ic, produced a far larger sample of articles than could be dealt with

Downlo

aded

By:

[PE

RI P

akis

tan]

At:

08:

30 2

2 Ap

ril

2010

International Journal of Social Research Methodology 113

within the purview of this investigation. In conducting the search, the same five fieldsof social research as those referred to in relation to the interviews were the focus for thesearches. The analysis was restricted to 19942003. The fact that the findings are basedon a large corpus of articles suggests that the sample is unlikely to be overly unusual,although it is impossible to make any claims of representativeness. Judgments aboutwhether an article was relevant to the investigation, in terms of whether it could beregarded as deriving from one of the five fields previously mentioned, were made onthe basis of the journal title or information supplied in abstracts. In this way, a total of232 articles was generated and content analysed.

The Paradigm Wars

The debate about quantitative and qualitative research at the epistemological levelbecame known in some quarters as the paradigm wars. The term stemmed from aperception of quantitative and qualitative research as distinct and to a large extentcompeting paradigms based on fundamentally different principles. The use of the termparadigm was frequently taken to entail, following Kuhn (1970), the implication thatthe two approaches cannot be combined because paradigms are incommensurable.Interestingly, the terms of this debate were, to a large extent, set by qualitative research-ers; quantitative researchers tended not to become entangled in the philosophicaldistinctions that were being demarcated. To the extent that others, such as methodol-ogists, became embroiled in the tussles, it was largely in terms of the battle lines drawnup by qualitative researchers. One possible reason for qualitative researchers being themain contributors to the epistemological version of the debate is that drawing out adistinctive philosophical position for qualitative research gave an intellectual rationalefor their preferred approach in the face of the hegemony of quantitative investigationsin journals and elsewhere. Alternatively, it may be that the growth of interest inqualitative research grew out of an awareness of philosophical positions that weremarginalized by the quantitative research orthodoxy.

Paradigm Peace

To a very large extent, the paradigm wars can be considered over and peace can beregarded as having broken out. There are occasional skirmishes, as authors occasionallyrevive the old debates, perhaps placing them in a new context or with a fresh gloss. Forexample, in the field of educational research, there has been some concern aboutattempts emanating largely from policy makers and others concerned about policy issuesto introduce standards for the conduct of investigations that are predominantly asso-ciated with quantitative research (Hodkinson, 2004). By implication, such a position isinimical to qualitative research and has prompted debates about the desirability of suchstandards and about what the criteria should be. This is also a field where skirmishesstill occur (e.g. Smith, 1997), but it is also clear that not all of those who work in thisfield are participants in the battles (e.g. Rocco et al., 2003). Also, while classic statementsof the epistemological divisions are often cited (e.g. Guba & Lincoln, 1982; Smith &

Downlo

aded

By:

[PE

RI P

akis

tan]

At:

08:

30 2

2 Ap

ril

2010

114 A. Bryman

Heshusius, 1986), particularly in terms of their arguments regarding the incompatibilityof quantitative and qualitative research, such references are usually used to provide abackcloth to the newer thinking that allows for and even promotes mixed-methodsresearch. Thus, the editors of a handbook concerned with mixing methods state confi-dently that the incompatibility thesis has now been largely discredited (Teddlie &Tashakkori, 2003, p. 19). It could be argued that one problem with this characterizationis that it fails to reflect fully the diversity of positions among qualitative researchers(Denzin & Lincoln, 2005). These positions vary in their stances towards, and prepared-ness to accommodate, quantitative research. However, in general there is a perceptionthat by and large quantitative and qualitative research can be meaningfully integrated.

Further evidence on this point derives from the previously mentioned content anal-ysis of journal articles. One of the dimensions on which these articles were coded wasin terms of whether there was any reference to paradigm issues. The criteria for an arti-cle to be deemed to have made a reference to paradigm issues were extremely nominal:all that was needed was for the author(s) to have referred to the possible relevance ofepistemological or ontological issues or paradigm conflicts to the combined use ofquantitative and qualitative research, even if the reference was one that entailed adismissal of the relevance of such issues. In other words, the reference to the possiblerelevance of such philosophical issues to the combination of quantitative and qualita-tive research could be one which portrayed those issues in a positive or a negative light.In fact, such issues were addressed in just 14 (6 per cent) of the 232 articles. Again, thisfinding suggests that at least so far as practising researchers are concerned, paradigmwars issues have little if any relevance to their work.

The view that quantitative and qualitative research can no longer be regarded asincompatible is significant for three reasons. First and most obviously, it removes anylingering doubts concerning whether it is intellectually legitimate to integrate the twoapproaches. Second, the compatibility view marginalizes the epistemological issues andconcerns that were at the heart of the paradigm wars, though, as will be shown, that isnot to suggest that philosophical issues disappear completely. Third, the view thatquantitative and qualitative research can be combined tended to be associated with anuncoupling of research methods from philosophical positions. Writers often arguedthat the association of particular methods with philosophical stances was based onconvention and that methods are in fact independent of epistemology (e.g. Bryman,1988) or that the connections between them were more contingent than is oftenassumed (e.g. Hammersley, 1992, p. 142).

Researchers in applied fields such as evaluation research and nursing have been inthe vanguard of the paradigm peace movement. For example, Twinn writes that thenursing literature generally has accepted the idea of mixed methods research and thatthe paradigm wars have been resolved within nursing (2003, p. 549). As Tashakkoriand Teddlie put it: Most investigators using these methods have not been interested indelving deeply into the philosophical orientations that supposedly underlie theapplication of their research studies (2003a, p. x).

Moreover, there may be a cultural side to the issues. Writing as the editor of a newjournal, Evaluation, Elliot Stern noted in connection with his editorial board:

Downlo

aded

By:

[PE

RI P

akis

tan]

At:

08:

30 2

2 Ap

ril

2010


There were those present, probably more from among North American participants thanothers, who believed that the paradigm wars were over. Not everyone was convinced,however. Continuing appeals to the positivist, constructivist and scientific realist canonwere noted (Stern, 1995, pp. 78)

However, the position may well be changing from that identified by Stern. I asked myinterviewees, all of whom were working in the UK and none of whom were in appliedfields like evaluation research, whether they felt the view that quantitative and qualita-tive research cannot or should not be combined because they derive from differentepistemological and ontological positions concerned them in their own research.Virtually all of my interviewees indicated that such issues did not affect them in theirresearch. In other words, they were not concerned about the possibility that their useof mixed-methods research transgressed philosophical principles. Examples of theirviews on this issue are:

No, not at allI mean it doesnt. Ive just decided that thats not aI mean Im not reallyprepared to worry about it, it does seem to me, from my experience er that you need touseto use both, and if you have an epistemology that says you can only use one, thenthats a very narrow and pointless epistemology because clearlyclearly you can generateunderstandings from different kinds of methods. (Respondent 1)

Er, no, not at all. (laughter) Im aware of Smith and Heshusius and you knowI justcant be bothered with that sort of attitude really. (laughter) I dont really want tothink a lot about philosophy, you know (yeah). I feel its a separatea somewhat separaterealm. Erm, but I dont think you have to subscribe to all the the sort of foundation orphilosophical assumptions in order to do exciting and interesting things in your researchpractice, which I think is a sort of quasi autonomous realm from philosophy and so on.(Respondent 3)

Sometimes, the interviewees acknowledged that they may be transgressing fundamen-tal epistemological issues, but felt that in order to get on with their research, they hadto sideline such concerns:

I mean I can see intellectually, that there are kind of contradictions there, erm, but IIguess I tend to do research in more practical terms and I think, you know, you can use thedata from contradictory, structured and contradictory kind of traditions, to shed light onthe same question. So Iit doesnt really bother me in the way that some purists might, Iguess, be less happy with it. (Respondent 16)

So I suppose what Im saying is I know that if Iwhen I go back and read those epistemo-logical debates, I know that Im going to be reminded of the impossible chasm and I dontbelieve in that impossible chasm in my own research practice Im afraid I put my headin the sand. (Respondent 12)

Such comments display a lingering anxiety about the neglect of the philosophicalissues.

It could be argued that there is a certain inevitability about such views. After all,researchers who have actually carried out investigations that bring together quantita-tive and qualitative research are less likely to fret about the epistemological nicetiesinvolved. Indeed, one interviewee noted that some of her colleagues were opposed onprinciple to the combination of quantitative and qualitative research, but also hinted

Downlo

aded

By:

[PE

RI P

akis

tan]

At:

08:

30 2

2 Ap

ril

2010

116 A. Bryman

that they were not empirical researchers. In other words, it is quite feasible that there isa small cadre of social scientists for whom the paradigm wars have not been resolved.These may be either those who are relatively uninvolved in empirical research or thosewho maintain a principled objection to combining quantitative and qualitativeresearch in their investigations. What can be said is that this cadres voices cannot bevery loud, as those of the peaceniks are far more frequently encountered in theliterature.

Occasionally, the social researchers I interviewed gave an indication that they werenot totally at ease with the way in which they had been forced (or felt forced) to sidelinethe epistemological debates. Such debates were felt by a few interviewees to be bothinteresting and unresolved, and it was felt that it was a shame that the philosophicalissues had been forced into a realm that had become divorced from their own researchpractice. They felt that the issues should not be lost, as suggested by the followingcomment: Erm, but an intellectual familiarity with the broad problems erm isabsolutely indispensable, I think (Respondent 5).

However, the interviewees typically saw themselves as needing to ignore such issuesto get on with the kind of research they wanted to do or as somehow transcending thedebates, as suggested by the following remark:

Yeah, well certainly I think ones got to keep asking oneself those questions all the time andthe discipline Im inis interesting from that point of view, because it does combinepeople who are at both ends of the spectrum, in terms of the work they do and others, likemyself, who are in the middle. (Respondent 11)

By and large, in spite of occasional glimmers of discomfort, my interviewees sawthemselves as having to disregard the philosophical issues that were a feature of someversions of the debate about quantitative and qualitative research.

The Rise of Pragmatism

Research combining quantitative and qualitative research has become imbued with anoverwhelmingly pragmatic attitude. In some versions of this attitude, the notion ofpragmatism is being construed with reference to pragmatism as a distinctive philoso-phy of the kind with which Charles Peirce is associated. In other versions, it is less todo with an allegiance to a distinct set of philosophical principles than a generalizedcommitment to needing to conduct research that will answer certain kinds of researchquestions. The specific issue of the role of research questions will be returned to below.

The apparent commitment to pragmatism has become a way of rationalizing theconjoint use of quantitative and qualitative research while simultaneously recognizingthat there have been debates about their supposed epistemological incompatibility.Maxcy suggests that pragmatism seems to have emerged as both a method of inquiryand a device for the settling of battles between research purists and more practical-minded scientists (2003, p. 79). Even when pragmatism is not explicitly mentioned, itsbroad principles can be detected in remarks such as: there is growing acknowledgmentthat complex social phenomena can usefully be understood by looking at them both

Downlo

aded

By:

[PE

RI P

akis

tan]

At:

08:

30 2

2 Ap

ril

2010


quantitatively and qualitatively (Rossman & Wilson, 1994, p. 315). Smith recalls beingtold by the editor of a major educational research journal that most researchers hadbecome bored with philosophical debates and were more interested in getting onwith the task of doing their research (1996, pp. 162163). Thus, in place of the primacyof philosophical principles that was associated with the era of the paradigm wars,pragmatism is taken to advocate the pre-eminence of technical decisions about theappropriate use of different methods (either singly or in tandem with other methodsregardless of whether they are quantitative or qualitative ones) according to particularcircumstances.

The adoption of disciplined pragmatism was also discernible in the interviews thatI carried out. Respondent 17 said: Erm, to some extent. I mean I think thatsyeah,there are major theoretical questions there, you know, but in the end I take a morepragmatic stance. In similar fashion, Respondent 16 replied: Erm, so I guess I donttend to think of it as aat a philosophical level, I tend to think of it in terms of theoutcomes and what you can do with what comes out of it. I mean Im a really prag-matic academic. Respondent 15 took a similar view: So its a fairly pragmaticapproach really. I mean I suppose I dont feel dogmatically signed up to one epistemol-ogy either, you know. Respondent 18 expressed a similar view regarding an investiga-tion in which he was then involved: So weve taken that pragmatic decision to do itthat way because thatll generate something that either method, standing alone, is notgonna give us.

Often, the need for pragmatism was justified in terms of the requirement to putaside the intellectual debates in order to apply for research grants from funding bodies.Debating the niceties of epistemological positions was seen as inconsistent with theneed for a fairly hard-nosed approach to getting financial support for ones research.For example, Respondent 14 said: because I suppose here, traditionally, weve beenyou know a self-funding, entrepreneurial centre as we were, theres always been somuch more of a pragmatic approach to doing things and perhaps there isnt in otherdepartments. Respondent 6 took the view:

Im much more focused on the pragmatic issues of designing a research design which, insome ways, is strong enough to overcome thatthat gatekeeper who will referee theproposal I guess. Its become much more strategic

He went on to add later in the interview: Erm, and again its probably thebeing verypragmatic, that in order to get funding from x or y, you need to fall into line to theirexpectations as to what is a good research design

Even when the term pragmatism was not employed, its influence was frequentlydetectable. For example, Respondent 9 replied:

To me, the answersis much more an answer ofof just attempting to better understandwhat it is youre trying to understand, and in that way, you then have to ask how appropri-ate are the sorts of methods Im using and are they going to give me the information tounderstand what it is Im researching?

In suggesting that such views are indicative of pragmatism, regardless of whether theterm is mentioned, it has to be recognized that the term is typically being employed in

Downlo

aded

By:

[PE

RI P

akis

tan]

At:

08:

30 2

2 Ap

ril

2010

118 A. Bryman

a general way, rather than as a commitment to a clearly defined philosophical posi-tion. As such, pragmatism tends to denote a no-nonsense practical approach toresearch.

The Centrality of the Research Question

One of the chief manifestations of the pragmatic approach to the matter of mixingquantitative and qualitative research is the significance that is frequently given to theresearch question. In the context of the debate about quantitative and qualitativeresearch and the matter of mixing the two approaches, two issues follow from this.First, it is frequently suggested, in the new climate of pragmatism, that issues to do withthe adequacy of particular methods for answering research questions are the crucialarbiter of which methodological approach should be adopted rather than a commit-ment to a paradigm and the philosophical doctrine on which it is supposedly based. Forexample, Teddlie and Tashakkori write that: Pragmatist researchers consider theresearch question to be more important than either the method they use or theparadigm that underlies the method (2003, p. 20). Similarly, Erzberger and Kelle arguethat: The selection of adequate methods should not be made mainly on the basis ofsympathies toward a certain methodological camp or school. Methods are tools for theanswering of research questions and not vice versa (2003, p. 482). This position withregard to the debate about quantitative and qualitative research prioritizes the researchquestion and relegates epistemological and ontological debates to the sidelines. Indoing so, it clears the path for research that combines quantitative and qualitativeresearch.

Second, the primacy of the research question means that research that bringstogether quantitative and qualitative research is not only feasible, but more impor-tantly desirable or even required for answering certain kinds of research question orcombinations of research questions. A writer from the field of population geographytypifies this position:

It may seem common sense to suppose that any choice of research method must be appro-priate to the particular research problem in hand and the data available for addressing thatresearch problem. The notion that we must adopt ways of doing research best suited to theresearch question and consider all methods available is not uncommon. On this view, amulti-method research design may be appropriate to some research projects but not toothers, with appropriateness being judged in relation to the nature of the research questionand the sources of information we have at our disposal to answer that question. (Graham,1999, p. 78)

Teddlie and Tashakkori put the matter simply: Mixed methods research can answerresearch questions that the other methodologies cannot (2003, p. 14).

Similar positions were also very evident among a few of my interviewees. Respondent16 put it this way:

Erm I dont really have a kind of problem, with a kind of clashing philosophy of using allkinds of so-called scientific method and a more qualitative approach. I mean I can seeintellectually, that there are kind of contradictions there, erm, but II guess I tend to do

Downlo

aded

By:

[PE

RI P

akis

tan]

At:

08:

30 2

2 Ap

ril

2010


research in more practical terms and I think, you know, you can use the data fromcontradictory, structured and contradictory kind of traditions, to shed light on the samequestion. So Iit doesnt really bother me in the way that some purists might, I guess, beless happy with it.

Respondent 6 felt that it was crucial to ask is that the appropriate research design forthis research problem or this research issue?.

A variation of this kind of argument was to suggest that mixing quantitative andqualitative research was only acceptable if it was relevant to the research question inwhich one was interested. In other words, my interviewees, all of whom were users ofa mixed-methods research approach, were often keen not to present it as a panacea orthemselves as its unequivocal advocates. Their view of mixing quantitative and quali-tative research was typically that it should be viewed as something that can be adoptedto fit some research questions but not others. While discussing mixed-methodsresearch, Respondent 2 said:

I think itsImean, I dont think its necessarily a good or a bad thing, I think it dependson how you want to access research questions and answer them and whether its appropriatefor the particular questions that you have.

Respondent 4 suggested:

So effectively I would say its task driven and its what you might call methodologicallyappropriate that Im concerned with, and I dont like what I would call ideologicalcommitments to any particular kind of methodology.

Respondent 14 maintained that youve got to have a completely clear idea in your headabout why youre combining them .

These views carry the suggestion that mixed-methods research is helpful for answer-ing certain kinds of research question. By implication, some research questions orcombinations of research questions can only be answered by combining quantitativeand qualitative research.

Implications for Quality

In this section, I consider the implications of paradigm peace and the issues addressedin the previous sections for the matter of the quality of research that combines quanti-tative and qualitative approaches. Two main areas are considered: the issue of theappropriateness of a mixed-methods approach and the extent to which there should bealternative criteria for such research.

The Appropriateness of Mixing Quantitative and Qualitative Research

In some of the comments by the interviewees can be detected the glimmer of a concernabout quality. For example, Respondent 2 argued:

I dont think any is gonna be better quality than anything else, it depends on have youfound a way of interrogating your research questions thats going to give you in-depth

Downlo

aded

By:

[PE

RI P

akis

tan]

At:

08:

30 2

2 Ap

ril

2010

120 A. Bryman

insights. And you can do that quantitatively, qualitatively or using both. So I wouldnt saythat any one is particularly better than any other really.

Interviewees frequently expressed a concern that mixed-methods research should notbe seen as a universally superior or appropriate strategy. Two kinds of reservationsabout the appropriateness of mixed-methods research and the possibility that qualitymay be jeopardized could be discerned in interviewees remarks.

First, there was a concern that, following a period when quantitative and qualitativeresearch were seen as irreconcilable, combining them was increasingly being seen asfashionable. This was viewed as raising the prospect of researchers seeking to combinequantitative and qualitative research in ways that may not be appropriate to theresearch question(s). Respondent 1 maintained:

Erm, yeah, so I do think its a bit of a fashion er I dont think its always necessary. I oftenlook at research proposals that Im reviewing and think you dont need this bit, you dontneed this bit, its a waste of your time, why is it there? You know, concentrate on the 20interviews, in-depth interviews and find outyou can answer all the questions you wantfrom that, why bother with all the rest of it? Erm, yeah.

Respondent 6 observed:

theres now expectation that, to have aa research design which is gonna be robust, whichis going to stand up to critique and which is representativeof whatever, whatever thatmay be, you need to adopt a mixed method approach. And that isnt necessarily the rightway to approach a particular research problem because it might not be the right approachat all. I think people are pigeon holed or you pick, as I said, youyou have to, the structurepushes you to making decisions about picking off, picking off the shelf the tailor mademixed method approach and it might not necessarily be appropriate. But because youalmostyoure not obliged to do it but you feel constrained that you have to do thisapproach to convince the gatekeepers and itsas you said, I think, you know, there areexpectations there. And of course, bodies like the Economic and Social Research Council,you knowin some ways their quantitative turn over theover recent times, you know,reproduces that mixed method approach.

Such comments imply a concern that mixed-methods research has become a fad andthat there is perception that it is more likely to be favoured by funding bodies, whothemselves are implicated in that fad, because it offers the best of all worlds. AsRespondent 2 succinctly put it: I think the disadvantages arise from when people justdo it because its expected or do it without thinking about how it relates to the researchquestions. The problem that interviewees are expressing is that it may result in quan-titative and qualitative research being combined regardless of the research question.

Second, the concern that the rationale for mixing quantitative and qualitativeresearch may not be fully thought through, because it is sometimes being employedregardless of the research question, was seen by several interviewees as having implica-tions for the quality of mixed-methods research. In particular, it was argued by severalinterviewees that the quality of mixed-methods research is diminished if it is notsoundly grounded in a clear sense of purpose. For example, Respondent 7 suggested:

And I suppose if you had a kind of evangelistic belief, which I dont quite have, I wouldntgo quite that far but you know, if you were really a fervent advocate of multiple methods,

Downlo

aded

By:

[PE

RI P

akis

tan]

At:

08:

30 2

2 Ap

ril

2010


regardless of the research problem and you always applied them, then that would really bequite inappropriate.

Respondent 8 provided a similar position:

So, the combination of methods, I think is reallyit needs to be seen in the light of answer-ing the particular research problem and just throwing all the methods in together is arecipe for disaster, I think. Im sure theres probably projects which have tried to do that.

One interviewee felt that if, in order to implement a mixed-methods researchapproach, researchers were having to conduct research using methods with which theywere unfamiliar (and perhaps unsympathetic to), the quality of the ensuing researchmight jeopardized. Similarly, Respondent 7 averred that with mixed-methods researchtheres more opportunities to make bad mistakes. Indeed, the problem of skills was arecurring motif, with many interviewees voicing a concern that the training and skillsof researchers (including their own) were often a barrier to combining quantitative andqualitative research. It was felt that too often researchers are skilled in and comfortablewith one approach rather than the other. Respondent 17 expressed a related view whiletalking about some classic qualitative studies in his field:

you know, those are such sensitive studies, the meanings, the subtle fluid meanings ofpeoples lives, that if youd have asked those researchers to be quantitative as well, I dontthink it would have been as high quality, you know. If it was somehow a requirement ofany study that it included both, I think it would have detracted because those people, doingit by themselves, would have lost some bitsanalytical power and excitement.

On the other hand, there was a prevalent feeling that all things being equal, mixed-methods research would deliver quality research. As Respondent 14 succinctly put it:

it does depend on the parameters of the research and on the aims and objectives erm andhow theyre set and how you relate to them. But, instinctively Id say yes it woulditwould provide better quality . I think there are certain things that you can learn byadopting different approaches, you know

Thus, for most of the interviewees, mixed-methods research is not intrinsically supe-rior in terms of quality to mono-method research, and there are even suggestions thatits quality may be endangered as a result of a slavish devotion to fashion or to a (possi-bly erroneous) belief that it is favoured by research funding bodies.

Quality Criteria for Research Combining Quantitative and Qualitative Research

One of the most significant developments in research methodology, as qualitativeresearch has become more accepted and has developed a momentum of its own, is thefocus upon quality criteria (Seale, 1999). Qualitative researchers have expressed theconcern that traditional criteria, that is, those associated largely with quantitativeresearchreliability, validity, generalizability, replicability etc.are inappropriate totheir approach. There have been two main approaches to addressing this issue. One isfor qualitative researchers to adapt traditional criteria (e.g. LeCompte & Goetz, 1982).The other is to develop alternative criteria. In the process, there has been a proliferation

Downlo

aded

By:

[PE

RI P

akis

tan]

At:

08:

30 2

2 Ap

ril

2010

122 A. Bryman

of different ways of expressing quality criteria for qualitative research (e.g. Lincoln &Guba, 1985; Spencer et al., 2003; Yardley, 2000). There is also a prior issue of what thequestion of quality relates to, that is, quality of what? By and large, assessments of qual-ity tend to relate to the overall research process from formulation of research questionsto the generation of implications for theory. As such, it encapsulates all aspects of aninvestigation, including problem formulation, sampling, research instruments, analy-sis and inferences.

What are typically given less attention in the course of these discussions are qualitycriteria in relation to research that includes both quantitative and qualitativeapproaches. In fact, Sale and Brazil (2004) concluded, following an examination ofappraisal criteria through journal databases, that no criteria with respect to mixed-methods studies could be found. There are several ways of conceiving of the ways inwhich quality criteria should be used in mixed-methods research. The following mightbe envisaged:

(1) Convergent criteriause the same criteria for both the quantitative and thequalitative components of the research.

(2) Separate criteriause separate criteria for the quantitative and qualitativecomponents.

(3) Bespoke criteriadevise new criteria specifically for mixed-methods research.

One problem with all three of these solutions is that they are likely to generate contro-versy that may hinder rather than help the development of mixed-methods research.Such a prospect is quite feasible in the light of the substantial and sometimes heatedarguments surrounding quality criteria for qualitative research (Sparkes, 2001). Onlyone of the 20 interviewees raised the issue of criteria in relation to mixed-methodsresearch, implying that they have not become a major issue yet for practitioners of theapproach. Respondent 18 noted in connection with the analysis of both quantitativeand qualitative data that it can be very difficult to synchronize the two because of thedifferent standards of evidence youre applying and rigour. However, it may not bedesirable to think in universalistic terms when discussing or using quality criteria andinstead a contingency position might be more appropriate. Each of the threeapproaches may be suitable in some circumstances but not others. For example, inmixed-methods research in which qualitative research is used to generate hypothesesor to assist with devising measurement devices for quantitative research, the investiga-tion is operating more or less exclusively within a quantitative research framework. Inthese circumstances, convergent criteria entailing the use of the traditional criteria ofreliability, validity and generalizability may be more appropriate than either separateor bespoke criteria. Similarly, when quantitative research is being employed simply tocollect background data within a predominantly qualitative research project or whenthe amount of quantitative data is minimal, criteria associated with interpretivist orconstructionist ideas, such as trustworthiness criteria (Lincoln & Guba, 1985), could bemore appropriate than either separate or bespoke criteria.

Bespoke criteria might appear to be the ideal solution insofar as they either fuse thedifferent quality criteria associated with quantitative and qualitative research or entail

Downlo

aded

By:

[PE

RI P

akis

tan]

At:

08:

30 2

2 Ap

ril

2010


the development of criteria that transcend these. However, it is unlikely, given thecontroversy that has surrounded criteria in qualitative research (e.g. Sparkes, 2001),that, at least in the near future, criteria could be developed that would satisfy the qual-itative research community. More fundamentally, it is being argued here that suchcriteria may not necessarily be relevant to a large proportion of studies using bothquantitative and qualitative research, namely, ones in which either quantitative orqualitative is a dominant approach. In such studies, the qualitative or the quantitativecomponent is a procedural adjunct to a dominant quantitative or qualitative approachrespectively. It is common for writers on mixed-methods research to distinguishbetween studies in which either quantitative or qualitative research is dominant or inwhich they are equal (e.g. Creswell, 2003), so that the view being presented here isconsistent with such a distinction.

A further issue concerns whether the quantitative and qualitative research compo-nents are genuinely integrated or whether they stand alone. Sometimes, mixed-meth-ods research is conducted to explore very different research questions within an overallproject. When there is little integration of the different components, separate criteriamight be appropriate. Where there is a genuine integration of quantitative andqualitative research, whereby the quantitative and the qualitative findings are system-atically related to each other rather than allowed to stand alone as separate sets offindings, a bespoke criteria approach, if one could be successfully produced given thecontroversies involved, would be appropriate.

Tashakkori and Teddlie (2003b) have proposed an approach to bespoke criteria formixed-methods research that is based on the notion of inference quality, which is meantto include both the traditional notion of internal validity and the notions of trustwor-thiness and credibility that are associated with qualitative research (Lincoln & Guba,1985). Inference quality is to do with the issue of whether an inference meets the funda-mental criteria required for it to be credible and/or defensible. This is an interestingapproach but it clearly only captures portions of the kinds of criteria that have come tobe associated with quantitative and qualitative research criteria. It does not, forexample, address issues to do with external validity/transferability (Lincoln & Guba,1985). More fundamentally, it is being argued here that such a criterion may only berequired when there is no dominant quantitative or qualitative approach or when thequantitative and the qualitative components relate to distinctly separate research ques-tions. As such, a bespoke criteria approach of the kind that Tashakkori and Teddlieoffer would primarily apply only to those studies where there is genuine integration.Sale and Brazil (2004) have also proposed bespoke criteria based on commonalitiesamong criteria associated with both quantitative and qualitative research. Theypropose: truth value (quantitative research criterion of internal validity vs. qualitativeresearch criterion of credibility); applicability (external validity/generalizability vs.transferability/fittingness); consistency (reliability vs. dependability); and neutrality(objectivity vs. confirmability). Interestingly, however, these authors proffer a remarkthat is very much in tune with the chief argument of this section: future guidelines forcriteria might be more specific depending on the mixed-methods design (Sale &Brazil, 2004, p. 361). As such, bespoke criteria such as those offered by Tashakkori and

Downlo

aded

By:

[PE

RI P

akis

tan]

At:

08:

30 2

2 Ap

ril

2010

124 A. Bryman

Teddlie and by Sale and Brazil would be employed in the context of mixed-methodsinvestigations in which quantitative and qualitative research components are inte-grated, that is, where one is not a procedural adjunct of the other and the quantitativeand qualitative components are fully integrated. However, these considerationsessentially sidestep the issues of whether there is agreement over the content andmeaning of quality criteria for quantitative research and whether it is desirable to haveparallel criteria for qualitative research (quite aside from what the criteria should be).

One further issue that is likely to make the creation of criteria for mixed-methodsresearch difficult arises from the fact that the outcomes are often unplanned. Researchof all kinds has the capacity to offer surprising or unexpected findings, but when quan-titative and qualitative research are combined the possibilities of unplanned or unan-ticipated outcomes are magnified considerably. For example, including bothquantitative and qualitative components in a study may result in a realization that itoffers the opportunity to compare findings in the spirit of triangulation (that is, tocross-check quantitative and qualitative findings), even though the investigation wasnot devised with triangulation in mind (Deacon, Bryman, & Fenton, 1998; Waysman& Savaya, 1997). Alternatively, triangulation may have been the goal of combining thetwo approaches, but other implications of putting the two sets of data together maybecome apparent when the data are analysed (Bryman, 2006; Rocheleau, 1995). Suchunpredictability may pose problems to mixed-methods researchers seeking to specifyquality criteria. This may arise because the kinds of criteria foreseen at the outset mayappear less appropriate towards the later stages of an investigation when the implica-tions of combining the quantitative and qualitative components appear to be differentfrom the original design.

Furthermore, several writers have observed that articles reporting the results ofstudies mixing quantitative and qualitative research frequently provide little explicitdiscussion of the rationale for using both approaches (Bryman, 2006; Niglas, 2004;Rocco et al., 2003). Unless mixed-methods researchers become a great deal morecandid about such issues, it will be difficult to move forward the specification andapplication of criteria for mixed-methods research.

Conclusion

I have sought to show that the so-called paradigm wars, which emphasized the episte-mological and ontological differences between quantitative and qualitative research,have been replaced by a period of paradigm peace. In this new era, there is a tendencyto stress the compatibility between quantitative and qualitative research and a prag-matic viewpoint which prioritizes using any approach that allows research questions tobe answered regardless of its supposed philosophical presuppositions. Within this moreaccommodating climate, the research question and the appropriateness of particularresearch methods or approaches to the research question become the hub for the possi-ble integration of quantitative and qualitative research. The issue of which quality crite-ria should be applied to investigations that mix quantitative and qualitative research hasnot been given a great deal of attention. I show that practitioners of mixed-methods

Downlo

aded

By:

[PE

RI P

akis

tan]

At:

08:

30 2

2 Ap

ril

2010


research are concerned that the approach should not be seen as a panacea for allresearch problems. They feel that this could be to the detriment of the quality of somemixed-methods research, since it would mean that the approach was sometimes beingapplied with insufficient attention to research questions. I also consider different waysin which quality might be assessed in studies combining quantitative and qualitativeresearch and it is suggested that a contingency position might be one way forward. Thismeans that rather than adopting one approach to assessing the quality of all mixed-methods research, it is suggested that the approach to considering quality issues shouldbe influenced by the nature and goals of the investigation. Probably the most difficultkind of mixed-methods research to fit into this approach is that which fully integratesthe two approaches, since neither quantitative nor qualitative research would sufficeand there are few generally agreed criteria that transcend the quantitativequalitativedivide. The suggestions concerning inference quality provided by Tashakkori andTeddlie (2003) represent a starting point in this connection but their work only dealswith a limited domain of issues. This is an area that will require further attention.

Acknowledgements

The author wishes to thank the three referees of this article for their constructivecomments. He also wishes to thank the Economic and Social Research Council forfunding the research project Integrating Quantitative and Qualitative Research:Prospects and Limits (Award number H333250003) which made possible the researchon which this article is based.

References

Bryman, A. (1984). The debate about quantitative and qualitative research: A question of method orepistemology? British Journal of Sociology, 35, 7592.

Bryman, A. (1988). Quantity and quality in social research. London: Routledge.Bryman, A. (2006). Integrating quantitative and qualitative research: How is it done? Qualitative

Research, 6, 97113.Creswell, J. W. (2003). Research design: Qualitative, quantitative, and mixed methods approaches (2nd

ed.). Thousand Oaks, CA: Sage.Deacon, D., Bryman, A., & Fenton, N. (1998). Collision or collusion? A discussion of the unplanned

triangulation of quantitative and qualitative research methods. International Journal of SocialResearch Methodology, 1, 4763.

Denzin, N. K., & Lincoln, Y. S. (2005). The SAGE handbook of qualitative research (3rd ed.). ThousandOaks, CA: Sage.

Erzberger, C., & Kelle, U. (2003). Making inferences in mixed methods: The rules of integration. InA. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioralresearch (pp. 45788). Thousand Oaks, CA: Sage.

Graham, E. (1999). Breaking out: The opportunities and challenges of mixed-methods research inpopulation geography. Professional Geographer, 51, 7689.

Guba, E. G., & Lincoln, Y. S. (1982). Epistemological and methodological bases of naturalisticinquiry. Educational Communication and Technology Journal, 3, 233252.

Hammersley, M. (1992). The paradigm wars: Reports from the front. British Journal of Sociology ofEducation, 13, 131143.

Downlo

aded

By:

[PE

RI P

akis

tan]

At:

08:

30 2

2 Ap

ril

2010

126 A. Bryman

Hodkinson, P. (2004). Research as a form of work: Expertise, community and methodologicalobjectivity. British Educational Research Journal, 30, 926.

Kuhn, T. S. (1970). The structure of scientific revolutions (2nd ed.). Chicago, IL: University of ChicagoPress.

LeCompte, M. D., & Goetz, J. P. (1982). Problems of reliability and validity in ethnographicresearch. Review of Educational Research, 52, 3160.

Lincoln, Y. S., & Guba, E. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage.Maxcy, S. J. (2003). Pragmatic threads in mixed methods research in the social sciences: The search

for multiple modes of enquiry and the end of the philosophy of formalism. In A. Tashakkori& C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 5189).Thousand Oaks, CA: Sage.

Niglas, K. (2004). The combined use of qualitative and quantitative methods in educational research.Tallinn, Estonia: Tallinn Pedagogical University Dissertation on Social Sciences.

Rocco, T. S., Bliss, L. A., Gallagher, S., Perez-Prado, A., Alacaci, C., Dwyer, E. S., Fine, J. C., &Pappamihiel, N. E. (2003). The pragmatic and dialectical lenses: Two views of mixed methodsuse in education. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in socialand behavioral research (pp. 595615). Thousand Oaks, CA: Sage.

Rocheleau, D. (1995). Maps, numbers, text, and context: Mixing methods in feminist political ecol-ogy. Professional Geographer, 47, 458466.

Rossman, G. B., & Wilson, B. L. (1994). Numbers and words revisited: Being shamelessly eclectic.Quality and Quantity, 28, 315327.

Sale, J. E. M., & Brazil, K. (2004). A strategy to identify critical appraisal criteria for primary mixed-methods studies. Quality and Quantity, 38, 351365.

Seale, C. (1999). The quality of qualitative research. London: Sage.Smith, J. K. (1996). An opportunity lost? In L. Heshusius & K. Ballard (Eds.), From positivism

to interpretivism and beyond: Tales of transformation in educational and social research(pp. 1617). New York: Teachers College Press.

Smith, J. K. (1997). The stories educational researchers tell about themselves. Educational Researcher,26, 411.

Smith, J. K., & Heshusius, L. (1986). Closing down the conversation: The end of the quantitativequalitative debate among educational researchers. Educational Researcher, 15, 412.

Sparkes, A. (2001). Myth 94: Qualitative health researchers will agree about validity. QualitativeHealth Research, 11, 538552.

Spencer, L., Ritchie, J., Lewis, J., & Dillon, L. (2003). Quality in qualitative evaluation: A frameworkfor assessing research evidence. London: Government Chief Social Researchers Office.

Stern, E. (1995). Editorial. Evaluation, 1, 59.Tashakkori, A., & Teddlie, C. (2003a). Preface. In A. Tashakkori & C. Teddlie (Eds.), Handbook of

mixed methods in social and behavioral research (pp. ixxv). Thousand Oaks, CA: Sage.Tashakkori, A., & Teddlie, C. (2003b). The past and future of mixed methods research: From data

triangulation to mixed model designs. In A. Tashakkori & C. Teddlie (Eds.), Handbook ofmixed methods in social and behavioral research (pp. 671701). Thousand Oaks, CA: Sage.

Teddlie, C., & Tashakkori, A. (2003). Major issues and controversies in the use of mixed methods inthe social and behavioural sciences. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixedmethods in social and behavioral research (pp. 350). Thousand Oaks, CA: Sage.

Twinn, S. (2003). Status of mixed methods research in nursing. In A. Tashakkori & C. Teddlie(Eds.), Handbook of mixed methods in social and behavioural research (pp. 54156). ThousandOaks, CA: Sage.

Waysman, M., & Savaya, R. (1997). Mixed method evaluation: A case study. Evaluation Practice, 18,227237.

Yardley, L. (2000). Dilemmas in qualitative health research. Psychology and Health, 15, 215228.

Downlo

aded

By:

[PE

RI P

akis

tan]

At:

08:

30 2

2 Ap

ril

2010

Documents

747984378