Upload
mike-crabb
View
44
Download
2
Embed Size (px)
Citation preview
Managing Quality in Qualitative Research
How to Address Research Quality
Standards, Criteria, Checklists and Guidelines
Concepts of Triangulation
Triangulation of Qualitative and Quantitative Research
Focusing on Process and Transparency
How to Address Research Quality
The issue of quality in qualitative research is located at the crossroads of internal needs and external challenges
• Researchers interest • Can I trust the results that I have created? • Are the methods that I have used in this situation
correct? • If I work with others will our results be similar?
• Funding Institution approach • How is one qualitative approach more better than
another?
• Journal Editor / Conference Editor decisions • Can I trust the rigour of this piece of work? • How do I know that this is the best approach?
• Readers questions • Can I trust what I’m reading? • How much detail did the researcher go into when
they were creating this?
It has become a crucial issue with the establishment of qualitative research as a valid method and in competition with other forms of work
The Independent Samples t-test compares the means of two independent groups in order to determine whether there is statistical evidence that the associated population means are significantly different.
What is the qualitative equivalent?
• Evaluation built on Standardisation • One-Size-Fits-All Approach • Criteria Appropriate to Research or
Research Appropriate to Criteria • How to Assess Research Quality in a Sensitive Way
Different potential ways to approach this
The formulation of research ‘criteria’ is only one solution to the problem
Standards, Criteria, Checklists and Guidelines
Reformulating Traditional CriteriaAim here is to make the whole research process more transparent
• Exact and Coherent Guidelines are used • Interviews are transcribed in a certain
way using specific methods • Distinction between verbatim
statements in field notes and summaries are given
• Reliability of whole process can be seen in reflexive documentation
Reformulating Traditional Criteria
Validating an Interview Situation• Are interviewees given any cause to construct a biased version
of their experiences that doesn’t fit with their views? • Script is analysed to look for these specific points and to
see whether the data is reliable
Communication Validation• Participants can be involved in a secondary research study to
assist in validation of data • Participants can reflect on previous answers and then
better articulate their thoughts
Reformulating Traditional Criteria
Procedural Validation• Validation is looked at for the entire research process and the
different relationships that take place: • Relationship between what is being observed and the larger
cultural, historical, and organisational contexts within which the observations are made
• The relationship among the observer, the observed, and the setting • The perspectives that are used to render an interpretation of any
ethnographic data • The role of the reader in the final product • The issue of author style to render the description or interpretation
Alternative, method-appropriate criteriaInstead of using traditional metrics, we use those that are more credible for qualitative work
• Are findings grounded in the data? • Is sampling appropriate • Are data weighed correctly?
• Are inferences logical? • Analytic strategies applied correctly? • Alternative explanations accounted for?
• Is the category structure appropriate? • Can decisions be justified? • What is the degree of researcher bias? • What strategies were used for increasing credibility?
Traditional criteria tend to miss the specific qualities of qualitative research.
Alternative criteria mostly come without benchmarks for distinguishing good from bad.
Evaluating Grounded TheorySpecific suggestions on evaluating Grounded Theory studies come from Charmaz (2006) who suggests breaking this down into four criteria
• Credibility • Originality • Resonance • Usefulness
Evaluating Grounded Theory
• Has the research achieved familiarity with the setting or topic? • Is the data sufficient to merit your claims?
• Think about range, number and depth of observations contained in the data
• Have you made systematic comparisons between observations and between categories
• Do the categories cover a wide range of empirical observations • Are there strong links between the gathered data and your
argument and analysis? • Has the research provided enough evidence for your claims to
allow the reader to form an independent assessment, and agree with your claims?
Credibility
Evaluating Grounded Theory
• Are your categories fresh, do they offer new insights? • Does your analysis provide a new conceptual rendering of the
data? • What is the social and theoretical significance of this work? • How does your theory challenge, extend, or refine current
ideas, concepts and practices?
Originality
Evaluating Grounded Theory
• Do the categories portray the fullness of the studied experience?
• Have you revealed both liminal and unstable taken-for-granted meanings
• Have you drawn links between larger collectivities or institutions and individual lives?
• Does your grounded theory make sense to your participants or people who share their circumstances? Does your analysis offer them deeper insights about their lives and world?
Resonance
Evaluating Grounded Theory
• Does your analysis offer interpretations that people can use in their everyday worlds?
• Do your analytic categories suggest any generic processes? • If so, have you analysed these processes for
implications? • Can the analysis spark further research in other substantive
areas? • How does your work contribute to knowledge, how does it
contribute to making a better world!?
Usefulness
A Framework for Analysing Qualitative Research Data
Research should be: • Contributory in advancing wider knowledge
or understanding • Defensible in design by providing a research
strategy • Rigorous in conduct through the systemic
collection, analysis, and interpretation of qualitative data
• Credible in Claim through offering well-founded and plausible arguments about the significance of the data generated
Quality in Qualitative Evaluation:A framework for assessingresearch evidence
A Quality Framework
Liz Spencer, Jane Ritchie, Jane Lewis and Lucy DillonNational Centre for Social Research
Government Chief Social Researcher’s Office
418771_CaboffQTY_SHORTV3_AW2 11/7/03 12:39 Page c3
Standards will only be helpful if they apply to qualitative research in general and not to specific approaches
Concepts of Triangulation
Triangulation is the method of location of a point from two others of known distance apart, given the angles of the triangle formed by the three points
x
y°
z°
Triangulation includes researchers taking different perspectives on an issue under study or more generally in answering research questions.
These perspectives can be substantiated by using several methods or theoretical approaches,
Triangulation should produce knowledge at different levels, which means they go beyond the knowledge made possible by one approach and thus contribute to promoting quality in research.
Types of Triangulation
• Data Triangulation • The use of different sources of data • Allows the researcher to reach maximum profit when using the
same methods • Investigator Triangulation
• The use of different observers or interviewers • This is not the sharing of work, it is a systemic comparison of
researchers influences on the issue • Theory Triangulation
• Approaching the data with multiple perspectives and hypothesis in mind
• Points are put side by side to assess their utility and power. • Methodological Triangulation
• Different methods are used to look at the same issue
Within-Methods Triangulation
Method Issue of Research
Approach 1
Approach 2
Example of this technique is the episodic interview which combines questions and narratives.
Within-Methods Triangulation
Interviews can be used to understand the everyday knowledge that people have
Semi-Structured interviews can include narratives where participants talk at length about specific aspects that they have encountered
BUT there is the chance for participants to switch from a narrative discussion to a descriptive, argumentative, or other non-native form of presentation
Episodic Interviews
Within-Methods TriangulationEpisodic Interviews
Situation 1
Situation 3
Situation 2
Episodic Knowledge
Episodic Interview
Narrative Presentation
Argumentative- Theoretical
Presentation
Within-Methods TriangulationEpisodic Interviews
Situation 1
Situation 3
Situation 2
Episodic Knowledge
Concept 1 Concept 2
Sub-Concept 1 Sub-Concept 3Sub-Concept 2
Episodic Interview
Semantic Knowledge
Narrative Presentation
Argumentative- Theoretical
Presentation
Within-Methods TriangulationEpisodic Interviews
Situation 1
Situation 3
Situation 2
Episodic Knowledge
Concept 1 Concept 2
Sub-Concept 1 Sub-Concept 3Sub-Concept 2
Episodic Interview
Semantic Knowledge
Narrative Presentation
Argumentative- Theoretical
Presentation
Argumentations
Subjective Definitions
Examples
Repisodes
Narratives of Situations
Stereotypes
Situational Narratives based on different levels of concreteness
Repisodes regularly occurring situations, no longer based on a clear local and temporal reference
Examples abstracted from concrete situations, and metaphor also ranging to clichés and stereotypes
Subjective Definitions which are asked for within sessions
Linked to Argumentative Definitions of terms, explanations of concepts
Within-Methods Triangulation
Between-Methods Triangulation
Method 1
Issue of Research
Choose at least one method which is specifically suited to exploring the structural aspects of the problem and at least one which can capture the essential elements of its meaning to those involved
Method 2
Triangulation of Qualitative and Quantitative Research
In this module we’ve dealt exclusively with qualitative work. However, it is possible to combine qualitative and quantitative methods in order to increase research quality through triangulation
Quantitative
Qualitative
continuous collection of both sorts of data
Qualitative
Quantitative
continuous field research
wave 1 wave 2 wave 3
Qualitativeexploration
QualitativeQuantitativequestionnaire deepening and
assessing results
Quantitativesurvey
QuantitativeQualitativefield study experiment
Linking Qualitative and Quantitative ResultsResults may converge, that is, are consistent completely, in general, by tendency or partially. For example - answers in a representative survey may match with statements from semi-structured interviews
Results may be complementary. Interviews can provide deeper, more detailed explanations to complement results from a questionnaire
Results may diverge. For example, interviews may produce views that are different when compared to questionnaires. This would cause the need for further research.
Triangulation of qualitative and quantitative research is not per se a quality indicator for qualitative research, but it can contribute to overall quality
Qualitative Research Quantitative Research
Data Sets
Single Cases
Triangulation
Focus Groups
Interviews
Questionnaires
Lab Studies
Positioning Triangulation in the Research Process
Exploration Data Collection Data Interpretation
Generalisation
Focusing on Process and Transparency
Quality in research is produced through the whole process.
Questions to ask for selecting a qualitative research method
I. What do I know about the issue of my study or how detailed is my knowledge already?
II. How developed is the theoretical or empirical knowledge in the literature about this issue?
III. Is my interest in more generally exploring the field and the issue of my study?
IV. What is the background of my study and which methods fit with this?
V. What do I want to get close to in my study? I. Personal Experiences of a group of people / social process II. Reconstruction of underlying research structures
Questions to ask for selecting a qualitative research method
VI. Do I start with a focused research question right away or do I start with an unfocused approach in order to develop the question?
VII. Which aggregate do I want to study: personal experience, interactions or situations, or bigger entities like organisations or discourse?
VIII.Is it more the single case I am interested in or the comparison of various cases?
IX. Which resources (time, money, manpower, skills etc.) are available for my study?
Questions to ask for selecting a qualitative research method
X. What are the characteristics of the field I want to study and of the people in it? What can you request of them and what not?
XI. What is the claim of generalisation of my study?
XII. What are the ethical issues to take into account that are affected by selecting a specific method?
Decide and reflect carefully whether you should use qualitative or quantitative research
• Why qualitative research?
• Which reasons do you have for the one or the other?
• What are your expectations for the qualitative research that you are planning?
Reflect on the theoretical background of your knowledge interest
• What is the impact of your setting on the research?
• How open and closed is your access to what you want to study?
Plan your study carefully, but allow for reconsidering the steps and modifying according to the state of play
• What are the resources available for the study?
• How realistic are the aims of your research in relation to the available resources
• What are necessary and appropriate shortcuts
x
y
z
a
b
Plan your sampling carefully!
• What are your cases?
• What do they stand for?
Think about whom in the field you should contact and inform about your research. Reflect about the relation to establish to field subjects
• What can you learn about your research field and issues from the way you get into the field or are rejected?
Think about why you chose your methods for collecting data
• Is it a decision for a favourite method or for habitual reasons?
• What could or would alternative methods provide?
• What are the impacts of the methods you use on your data and your knowledge?
Plan carefully how to document your data and research experiences
• How exactly should you write your notes?
• What are the influences of the documentation on your research and on your field subjects?
• What are the impacts of the documentation on your methods of collection and analysis?
Think about the way that you want to present what you have experienced in the field and found in your research
• What are the target audiences of your writing?
• What is it mainly you want to convince them about your research?
• What is the impact of the format of your writing on your research and its findings?
Plan how to establish the quality of your research
• What are the quality criteria your research should meet?
• How should these criteria be realised?
• What is their impact on your research and your field subjects or relationships?
How to Address Research Quality
Standards, Criteria, Checklists and Guidelines
Concepts of Triangulation
Triangulation of Qualitative and Quantitative Research
Focusing on Process and Transparency
Information in this presentation was
based on…