This article was downloaded by: [McMaster University]On: 11 November 2014, At: 06:43Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Nordic Journal of Music TherapyPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/rnjm20

Do We Need General Criteria for theEvaluation of Qualitative ResearchArticles, and If We Do, How CouldSuch Criteria Be Formulated?Brynjulf StigePublished online: 04 Jul 2009.

To cite this article: Brynjulf Stige (2002) Do We Need General Criteria for the Evaluation ofQualitative Research Articles, and If We Do, How Could Such Criteria Be Formulated?, NordicJournal of Music Therapy, 11:1, 65-71, DOI: 10.1080/08098130209478048

To link to this article: http://dx.doi.org/10.1080/08098130209478048

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the“Content”) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoeveras to the accuracy, completeness, or suitability for any purpose of the Content. Anyopinions and views expressed in this publication are the opinions and views of theauthors, and are not the views of or endorsed by Taylor & Francis. The accuracy ofthe Content should not be relied upon and should be independently verified withprimary sources of information. Taylor and Francis shall not be liable for any losses,actions, claims, proceedings, demands, costs, expenses, damages, and other liabilitieswhatsoever or howsoever caused arising directly or indirectly in connection with, inrelation to or arising out of the use of the Content.

This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden. Terms

& Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

Dow

nloa

ded

by [

McM

aste

r U

nive

rsity

] at

06:

43 1

1 N

ovem

ber

2014

DIALOGUES

Do We Need General Criteria for the Evaluation of Qualitative Research Articles, and If We Do, How Could Such Criteria Be Formulated?1

Brynjulf Stige

Acknowledgement

The idea of starting a discussion in the web Forum of the Nordic Journal of Music Therapy (NJMT) grew out of the Fourth International Symposium for Qualitative Music Therapy Research, organized by Mechtild Jahn-Langenberg in Sauen, April 2002.1 want to thank all participants for these stimulating days, and especially the members of the discussion group I belonged to (Kenneth Aigen, Gary Ansdell, and Eckhard Weymann) for helping me to shape these initial ideas. I hope that they, and others, will contribute in a discussion about this important topic. Eventually then, after a process of discussion in this Forum, a pre-conference seminar about criteria for the evaluation of qualitative music therapy research will be arranged in Bergen on 21 May 2003, in collaboration with the Fourth Nordic Music Therapy Conference, possibly leading to the establishment of defined standards for this journal.

Introduction

The Nordic Journal of Music Therapy publishes quantitative and qualitative research studies. For quantitative studies there are quite clear criteria for the evaluation of submitted texts: there are some firm methodological rules to follow, there are specific procedures for statistic analysis, and there is a high degree of consensus on what adequate means of reporting the results are. In qualitative research this is different, and there are many good reasons for proposing that it should be different. Qualitative research is about exploring particularity and diversity, and there is a plethora of qualitative research approaches around with rather different answers to basic ontological, epistemological, and methodological questions.

This diversity creates some problems though in the process of evaluating articles, maybe not because there is a lack of criteria for evaluation, but because there is a lack of consensus as to which criteria to use. Most traditions of qualitative research have their

1 Editor's note: Republished from NJMT's online discussions at Forum (http://www.hisf.no/njmt/forumqualart_l.html)

Nordic Journal of Music Therapy, 11(1) 2002

Dow

nloa

ded

by [

McM

aste

r U

nive

rsity

] at

06:

43 1

1 N

ovem

ber

2014

DIALOGUES

own criteria of evaluation, but how could a journal receiving articles from several traditions best deal with this issue? As there is little consensus as to which criteria to use, one may in fact imagine a situation where four sets of criteria operate in the evaluation of one text: the author is writing according to the criteria of his or her tradition of research, the two referees use two other sets of criteria, and the editor a fourth. While ideally one could think that each research project deserves to be evaluated according to the criteria of the tradition it belongs to, this is not always easy to exercise practically. First of all, authors are not always clear about what tradition their research belongs to (or their methodology may be eclectic, using bits and pieces from several traditions.) Second, it is not always realistic to think that a referee should be equally updated and qualified in the latest criteria proposed in any tradition or branch of qualitative research.

Even if these practical problems were possible to overcome completely, which I doubt they are, another more basic problem would remain: if criteria idiosyncratic to each research tradition are used only, one runs the risk of inhibiting a process where different positions may challenge each other and produce a healthy discussion among qualitative researchers about what the most relevant and important criteria of evaluation are.

What General Criteria Should Not Be, and What They Possibly Could Be To me it is obvious that the general criteria for the evaluation of qualitative research could not be based on the specific procedures advocated by one or another tradition of qualitative research. While phenomenological researchers may advocate the importance of triangulation and Grounded Theorists request systematic procedures of categorization, to base general criteria of evaluation on such research procedures certainly could alienate a researcher working for instance in a hermeneutic tradition. From this simple example I conclude that the general criteria could not be formulated as a rule-book in relation to research procedures. Neither could they be formulated as prescriptive rules as to how to

£ £. Nordic Journal of Music Therapy, 11(1) 2002

present a study, since many qualitative researchers advocate that not only the research method should be adapted to the object or phenomenon of study, but also to the form and format of presentation of the research findings.

As an alternative to criteria based upon research procedures etc., I would like to consider criteria based upon more basic metatheoretical assumptions related to ontology and epistemology, one could call them metacriteria if one likes. Let me - by playing with Wittgenstein's (1953/1967) notions of language games and family resemblances - use the rules of games such as basket, soccer, and rugby as an analogy. It makes sense to group these games together, as they share certain characteristics. They are all games where two teams compete according to certain rules, manipulating a ball in order to have more points than the other team. It would not make sense for a referee to evaluate a soccer team by the rules of a basketball team though. In the first case you are supposed to move the ball with your feet and you are - within certain limits - allowed to be in physical contact with the players on the other team, while in the other case you are supposed to move the ball with your hands and no physical contact between players is allowed. If rugby is added when making this comparison, we have a game where both kicking and throwing is allowed, as well as physical contact at a much rougher level than what is allowed in soccer. In other words: a referee of a basket or soccer game would create chaos if using the rules of rugby, and vice versa. Still, at another level, there are some rules that seem to be shared by games such as soccer, basket, and rugby, and which are necessary in order for them to make any sense. Four examples could be: a) there is a set of rules for the particular type of game, and this set is known and accepted by both teams and all players, b) a player plays on one team only, and tries to help his or her own team to win the game, c) you are not allowed to hurt other players or deliberately put their health to risk, d) the referee has the final word when events and rules are disputed.

In using this analogy we could propose that qualitative research represents a family of "games," with particular rules for each game, but also with some "shared rules" for the whole family. (From this analogy it follows that quantitative research may

Dow

nloa

ded

by [

McM

aste

r U

nive

rsity

] at

06:

43 1

1 N

ovem

ber

2014

DIALOGUES

be another - more or less related - family, and that there is not necessarily one dividing line between the two families. Grounded Theory, for instance, may be an approach sharing some characteristics with certain quantitative post-positivistic approaches and other characteristics with certain qualitative interpretive approaches). I will proceed by proposing a set of "shared rules" or metacriteria that could be relevant for the evaluation of qualitative music therapy research. The proposal is based on the premise that reflexivity is a basic characteristic of qualitative research. With reflexivity in relation to research I mean: the researcher acknowledges that she is not separated from the field she studies, she is herself positioned in it and must therefore reflect upon this position, which includes self-inquiry and examination of the assumptions guiding the research process (Ruud, 1998). I have discussed the concept of reflexivity in relation to music therapy and research in a forthcoming book (Stige, in press/ 2002), and some of the arguments below paraphrase arguments in this book.

Reflexive Methodology

Alvesson and Skoldberg (2000) have developed a conceptualization of four elements of reflexive qualitative research methodology that I find relevant to use as a starting point. The conceptualizations are based upon certain ontological and epistemological assumptions. For instance, the authors try to avoid the polarized positions of realism with little awareness of the problems of representation of reality and social constructivism where the representations may lose contact with reality. The conception of data as merely reflections of the world is therefore rejected by these authors, and in this respect the primacy of interpretations is accepted. In order to communicate this reservation, these authors therefore use the term empirical material instead of data. Following these basic assumptions it is proposed that the basic principle for qualitative research methodology is reflexivity. There are no fixed rules prescribing what to do and what not to do, rather, to be reflexive in relation to research means to reflect upon the preconditions of one's own activity and upon how one's own personal

and professional involvement influences the interaction with the phenomenon under scrutiny. Reflexivity in the research process, then, is awareness of the need for critical and sensitive interpretations of one's interpretations of the empirical material. Part of this is the awareness that as researchers we construct ourselves while constructing the objects or phenomena that we study, which then suggests that self-reflexivity and reflexivity converge.

Based upon these assumptions Alvesson and Skoldberg (2000) then propose four elements in reflexive research methodology: 1) Systematics and techniques in research procedures: There must be some kind of logic in the way the researcher interacts with the empirical material, and some kind of systematic approach when collecting it. 2) Clarification of the primacy of interpretation: A reflective researcher acknowledges the primacy of interpretation (that even "raw data" are interpretations), which implies that research cannot be disengaged from either theory or self-reflection. 3) A wareness of the political-ideological character of research: What is explored, and how it is explored, cannot but support or challenge some values and interests in society. The political and ideological aspects of research must therefore be acknowledged. 4) Reflection in relation to the problem of representation and authority: There are problems of representation and authority connected to any research text, and awareness of rhetoric elements and the relationships between text, author, and world need to be examined.

These four issues have been treated differently by different traditions of qualitative research, and a possibility for learning exists in examining these traditions. Consider for instance the concern about systematics and techniques found in Grounded Theory, empirical phenomenology, and other inductive approaches. Similarly, hermeneutics has much to offer concerning clarification of the primacy of interpretation, while Critical Theory, feminism, and related metatheories have given valuable inputs concerning the political-ideological character of research and knowledge. Postmodern (text-oriented) approaches, admired by some and held in contempt by others, should also have something to offer to all researchers, by sensitizing them to the problems of

Nordic Journal of Music Therapy, 11(1) 2002

Dow

nloa

ded

by [

McM

aste

r U

nive

rsity

] at

06:

43 1

1 N

ovem

ber

2014

DIALOGUES

representation and authority in research texts. Alvesson and Skoldberg (2000) therefore argue that reflexivity in research involves some kind of search for a balance between these four perspectives, and that this is necessary in order to avoid reductionism with focus only on data, interpretation, critique, or language. The researcher thus must search for some kind of interplay between the four levels, and avoid totalization of one of them. These suggestions also have a practical aspect: reflexivity in research means to restrict the time you use for data collection and analysis, to be able to make room for interpretations of the interpretations that you make.

An integration of the four concerns and orientations outlined above - from concerns about the empirical material to concerns about interpretation and critique - is what Alvesson and Skoldberg (2000) call "reflexive interpretation." Again, their main message is the need to counteract any one of those orientations to totalize and "take over" operations in the field. There are several levels of interpretation to deal with, from data-collection and analysis ("low-abstract" interpretations) to critical and self-critical interpretations that focus upon ideology and the selectivity of voices represented in the research text (Alvesson & Skoldberg, 2000, p. 238ff.).

ERIC: Metacriteria for the Evaluation of Qualitative Music Therapy Research? I believe Alvesson and Skoldberg's (2000) eclectic approach is relevant for a multifaceted field like music therapy. The four metacriteria that I will propose below are therefore in many respects based upon their contribution. I have rephrased them from being descriptions of elements of reflexive research to being metacriteria for the evaluation of qualitative music therapy research. I have also changed the sequence somewhat. This is partly for practical reasons, in order to be able to create an acronym that researchers could remember more easily than abstract principles, and partly to communicate how central the problem of representation is for researchers in a discipline where music is a major element. (It must be noted, though, that the criteria

Nordic Journal of Music Therapy, 11(1) 2002

proposed are not set up according to any hierarchy. They are all equally relevant and important.) The acronym I propose is ERIC, composed of the following criteria:

"The researcher should demonstrate awareness of and sensitivity to:

• the relevance and solidity of the Empirical material

• the problem of Representation • the primacy of Interpretation • the obligation to produce pertinent Critique."

(Even though ERIC is a good old Nordic name, I would rather have preferred an acronym that could characterize qualitative research in some way, such as EPIC. I have not been able to produce that without stretching words and arguments though, so ERIC is my proposal at this point. Can anyone help?)

I will briefly exemplify the issues lurking behind each letter of the acronym, with a few references to some music therapy literature where such issues have been discussed. The list of references is in no way intended to be complete. The references are included just to demonstrate that the criteria proposed here relate to ongoing discussions in the discipline:

E — the relevance and solidity of Empirical material: Empirical materials - for instance musical recordings, field notes from therapy sessions, and interview transcriptions - will be the result of musical and/or verbal interactions with clients or other agents of the setting. While qualitative researchers may not need to have representative data according to the conventions of quantitative research, empirical material should not be arbitrary and fragmented, or colored by the preferences, personality, or values of the researcher with a lack of discussion about such issues. A high degree of sensitivity of the researcher-as-instrument is thus asked for (Aigen, 1996).

R —the problem of Representation: This domain is multifaceted. As Ansdell (1999) has argued, problems of language and discourse are at the heart of a discipline with a non-verbal phenomenon such as music as a major element. In addition, there is the more general problem of representation in research, based on the assumption that language does not simply reflect reality. Any representation is a

Dow

nloa

ded

by [

McM

aste

r U

nive

rsity

] at

06:

43 1

1 N

ovem

ber

2014

DIALOGUES

selection, distortion, and addition. Related to this are issues of power, discourse, and rhetoric. What is required of the researcher is of course not a particular stance on the above-mentioned debates, but awareness of and relevant discussions about these dilemmas.

/ - the primacy of Interpretation: Interpretation starts already with the choice and collection of data. The researcher therefore needs to demonstrate awareness of the hermeneutic problems inherent in all research, such as part-whole relationships when working with the empirical material, and the influence of the researcher's pre-understanding on the understanding that is developed. As a general criteria, what is asked for is of course not the use of hermeneutics as a specific research method, but awareness of the general problems of understanding and interpretation that have been discussed in philosophical hermeneutics (Ruud, 1998; Stige, in press/2002).

C-the obligation to produce pertinent Critique: This criterium is based upon the assumption that all research is situated in social and political contexts, and in some ways is influenced by and also influences such contexts. While some researchers-such as participatory action researchers - make the implication that research should contribute to social change and empower participants (Stige, in press/ 2002), this could hardly work as a general principle for evaluating qualitative music research. A minimum of awareness of this dimension could be requested though, ensuring that researchers at least do what they can to hinder that their research contributes to repression and dis-empowerment.

The DaRMI-principle

The above criteria could be summarized as a request for refiexivity. Based upon Alvesson and Skoldberg's (2000) discussion of reflexive methodology I have created the acronym ERIC that could denote four domains of major relevance in qualitative music therapy research. These criteria are not meant to "replace" other standards proposed for the field, such as Bruscia's (1996,1998) discussion of authenticity and integrity. Rather, what is proposed are some domains in which the researcher must use his

authenticity and integrity in order to develop the required refiexivity.

For researchers and for referees evaluating submitted research articles, one question remains: how should refiexivity be demonstrated and documented? To produce thorough discussions of the issues implicit in the ERIC acronym could easily turn every research article into a book. I think there is no general answer to this question. Solutions must be found in relation to each research project. I have one general comment though, and that is that qualitative researchers - to a higher degree than quantitative researchers - need to know the history of ideas quite well. Since there are no fixed methodological rules to play by, qualitative researchers need to position themselves in relation to philosophical and metatheoretical discussions. This has, in fact, clear implications for the content and organization of qualitative research courses in music therapy training programs.

Going back to the concrete question of how to demonstrate and document refiexivity, the issue of how to write becomes essential. My proposal is that at this concrete level, the DaRMI-principle could be helpful. DaRMI is a term I have made up, by spelling the established IMRaD-principle for quantitative research backwards. What I am trying to communicate is that the IMRaD-principle in many ways encapsulates what makes scholarly communication effective (for explication, see the NJMT's Guidelines for Authors.) The principle could therefore not be easily dismissed. At the same time, a literal application of the IMRaD-principle, as it has been explicated within the quantitative tradition, would be limiting for many qualitative research projects. For instance, in most qualitative research studies it does not make sense to present the methods used with the purpose of making replication possible. On the other hand, the explication of metatheoretical assumptions may need to be developed to a higher level than what is usual in quantitative research articles.

With the acronym DaRMI I then try to communicate that the sections expressed by the IMRaD acronym should be present in most qualitative research articles (Introduction, Methods, Results, and Discussion), but that there could be considerable freedom as to their sequence and as to

Nordic Journal of Music Therapy, 11(1) 2002

Dow

nloa

ded

by [

McM

aste

r U

nive

rsity

] at

06:

43 1

1 N

ovem

ber

2014

DIALOGUES

how to develop each section. The basic story of a research report is, according to the four sections suggested by the IMRaD-principle: "This is what I looked for and why I wanted to find it, This is how I did it, This is what I found, and This is what I think it means." Are the story lines of qualitative research articles very different? I don't think so. I do think, however, that some phenomena require flexibility and creativity in writing, so that the plot may be arranged differently. In this respect qualitative researchers may sometimes come close to authors of fiction. In a detective novel, for instance, the author may choose to disclose who the murderer is on page one, only to use the rest of the text to elaborate on how and why this happened. Similarly I have read good research articles organized in other sequences than I-M-R-D (to rearrange the sequences may sometimes be effective, as in the detective novel, by leading to a closer examination of how and why instead of who and what).

But: if I read a research article with no discussion of results, or with a lot of discussion, but with no presentation of results, or with no clarification of method, or with no positioning of the study in relation to existing theory, do I read a good research article? I don't think so. I think the I, the M, the R, and the D need to be there, in some way or other, which I then have chosen to call the DaRMI-principle.

Conclusion

In the literature of qualitative research several sets of criteria for evaluation have been proposed. As there is no consensus, this is part of the problem. Authors may feel that their text is evaluated in accordance with the wrong criteria, and journal editors may feel that the evaluation process becomes vague, because of a lack of shared criteria. In the long run this is a potential problem for the discipline and profession, as the situation may impede the development of high quality qualitative research. As soccer cannot be refereed by the rules of basketball, a participatory action research project informed by critical theory could not be refereed by the coding rules of Grounded Theory. My proposal

Nordic Journal of Music Therapy, 11(1) 2002

is therefore not a set of specific rules, but a set of metacriteria denoting domains with a complexity of issues related to them. These issues are under continuous debate and have for a large part been discussed for centuries. The purpose of having general criteria of evaluation would then not be to have prescriptive rules for how exactly qualitative research should be done. Such a thing would be limiting for the content- and context-sensitivity that characterizes good qualitative research. Instead the purpose of having general criteria would be to develop consensus about a few major domains which the researcher should demonstrate awareness of and sensitivity to.

The reflexivity implicit in the ERIC acronym needs to be communicated effectively in a research article. I have tried to encapsulate a request for the combination of rigor and flexibility through making up the term "the DaRMI-principle." The story told by the researcher may be organized in many ways, but we need to know what was looked for and why, how the (re)searching was done, what was found, and what the researcher thinks this means. If reflexivity - as outlined by the ERIC acronym - is integrated in the writing process, I think the result may be a good and interesting research article. This is one of the reasons why qualitative researchers need to know the history of ideas quite well. They need to be able to position their work in relation to existing traditions of thought (which could often be done in a few sentences and with a few references instead of re-inventing the wheel by producing pages of explications of every basic assumption).

One could of course counter my arguments for ERIC by stating that researchers may disagree about metacriteria based on metatheoretical assumptions as much as they could disagree about specific procedure prescriptions. I still think a journal like the NJMT has reason to explore whether such metacriteria could be established, for at least the following reasons:

a) The existence of such criteria would ensure that the author knows the criteria used by the journal, which should be better than a hidden set of criteria (to imagine no criteria is impossible, unless one imagines a process without evaluation, which is incompatible with the notion of a peer reviewed journal.)

Dow

nloa

ded

by [

McM

aste

r U

nive

rsity

] at

06:

43 1

1 N

ovem

ber

2014

DIALOGUES

b) When the author knows the criteria, he has the freedom of choosing to accept them, trying to change them (through initiating a debate), or simply choosing another journal. Again this must be preferable to being evaluated by criteria that are not explicit.

c) The fact that there could be disagreement about metacriteria such as those communicated by the acronym ERIC could in itself be fruitful for the discipline, as it could stimulate metatheoretical debate and discussion, which will be helpful for the future refinement of theory, research and practice.

References

Aigen, Kenneth (1995). "Principles of Qualitative Research." In: Wheeler, Barbara (ed.). Music Therapy Research. Quantitative and Qualitative Perspectives. Phoenixville, PA: Barcelona Publishers.

Alvesson, Mats & Kaj Skoldberg (2000). Reflexive Methodology: New Vistas for Qualitative Research. London: Sage

Publications. Ansdeil, Gary (1999). Music Therapy as

Discourse & Discipline. A Study of 'Music Therapist's Dilemma' . London: Doctoral thesis, City University, Department of Music.

Bruscia, Kenneth (1996). "Authenticity Issues in Qualitative Music Therapy Research." In: Langenberg, Mechtild, Kenneth Aigen & Jorg Frommer (eds.). Qualitative Research in Music Therapy. Beginning Dialogues. Gilsum, NH: Barcelona Publishers.

Bruscia, Kenneth (1998). "Standards of Integrity for Qualitative Music Therapy Research." Journal of Music Therapy, XXXV (3), 176-200.

Ruud, Even (1998). Music Therapy: Improvisation, Communication and Culture. Gilsum, NH: Barcelona Publishers.

Stige, Brynjulf (in press/2002). Culture-Centered Music Therapy. Gilsum, NH: Barcelona Publishers.

Wittgenstein, Ludwig (1953/1967). Philosophical Investigations. Oxford: Blackwell.

Nordic Journal of Music Therapy, 11(1) 2002

Dow

nloa

ded

by [

McM

aste

r U

nive

rsity

] at

06:

43 1

1 N

ovem

ber

2014