Achieving Appropriate Rigor in Qualitative Research

Research Day, February 4, 2011

Mary Katherine O’Connor, Ph.D.School of Social Work,

Virginia Commonwealth Universitymkoconno@vcu.edu

Presentation Goals Propose a multi-paradigmatic heuristic for

understanding variety in qualitative research

Detail differential standards for research quality depending upon paradigmatic perspective

Aid in matching research questions, analytical needs and computer-based qualitative analysis packages

Two Dimensions of Analysis

Subjective/objective relates to assumptions about the nature of knowing (and, therefore, of science)

Regulation/radical change relates to assumptions about the nature of society (and, therefore, appropriate goals for science)

Burrell & Morgan’s Framework

FunctionalistInterpretive

Radical Humanist

Radical Structuralist

Sociology of Regulation

Sociology of Radical Change

ObjectivitySubjectivity

Multiple Paradigms

Fundamentally different perspectives for analysis of social phenomena

Generating different concepts and analytical tools

Different standards for good research practice

Four different types of qualitative research building from different types of research questions and research goals

Multiple Paradigms in Qualitative Research

Interpretive

Critical Interpretive Critical Positivist

Radical Change

Regulation

ObjectiveSubjective

Positivist/Post Positivist

Research Goals

Radical Change

Regulation

ObjectiveSubjective

Positivist/Post Positivist

Positivist Research:UNDERSTANDING FOR THEORY BUILDING

THAT CAN RESULT IN THEORY TESTING

Assumptions: Rational, pragmatic methods of natural science are appropriate to study human affairs

Questions: Useful for theory building in the early stages of theory development

Standards: understanding for generalizability including random assignment, control groups, valid and reliable data collection, standardized data analysis

Critical Positivist

Radical Change

Regulation

ObjectiveSubjective

Critical Positivist Research:CONSCIOUSNESS RAISING AND CHANGE AT

THE SYSTEM LEVEL

Assumptions: Knowledge is for radical/transformative change, emancipation and potentiality; realist but concentrating on structural relationships to understand and generate fundamental conflicts that will be the basis for radical change at the class/structural level

Questions: Focus on structural relationships seeking to provide explanations of the basic interrelationship within the context of total social formation for the purpose of consciousness raising

Designs: critical ethnographies, cooperative or collaborative inquiry, participatory action research, appreciative inquiry, empowerment evaluation

Standards: same as functionalist qualitative inquiry, but in addition, measurable presence of increased sophistication with change or change potential

Interpretive

Radical Change

Regulation

ObjectiveSubjective

Interpretive Research:MEANING MAKING

Assumptions: Understanding at the level of subjective experience from perspective of participants, not observer. Relativist in that the world, though ordered, is an emergent social enterprise that is continually being created.

Questions: seeking explanation within individual consciousness and subjectivity from standpoint of the participants rather than the observer

Designs: ethnographies, case studies, phenomenological studies, constructivist studies

Standards: sampling for maximum variation, emergent design, multiple data collection approaches, triangulation, reflexivity, inductive data analysis, thick description of research product

Critical Interpretive

Radical Change

Regulation

ObjectiveSubjective

Critical Interpretive Research: CONSCIOUSNESS RAISING AND CHANGE AT

THE INDIVIDUAL LEVEL

Assumptions: Critical interpretive with radical change focus, but from a subjectivist, individualistic standpoint. Concentrating on human consciousness and the social structures that inhibit true fulfillment. Goal is release of constraints that hamper individual human development.

Questions: how individual makes sense of life and experiences of societal constraints

Designs: interview studies, narrative research, life histories, autoethnographies, collective biographies

Standards: more artistic or aesthetic including creativity, aesthetic quality, interpretive vitality, and degree of stimulation for change

Research Goals: Implications All qualitative research is not based on the same

ontological and epistemological assumptions All qualitative research cannot be expected to be

held to a universal standard of rigor, quality or worth

Key to justification of qualitative processes and products is selection of appropriate research design including data collection and analysis techniques able to answer research question lodged within specific paradigmatic perspective

QUESTIONS?

Research Design

Positivist Design: An appropriate research design could include structured

interviewing, focus group, grounded theory, etc. and should be identified and justified along with a data collection mechanism.

The sampling plan is preferably randomized or at least looking for maximum variation along identifiable dimensions.

An “n” of at least 30 or a justification for a lesser number that is similar to a justification for a lesser sample number in a quantitative project.

A data analytic plan (thematic, content, constant comparison, linguistic, etc.) that matches the question, including whether or not computer based text analysis is being used. When computer analysis is present, the selected program must be able to do the analysis proposed. In most cases the variables for analysis should be identified beforehand. The exception would be in grounded theory designs.

Inter-rater reliability should also be addressed.

Critical Positivist Design: An appropriate research design could include

empowerment, appreciative, focus group, etc. and should be identified and justified along with a data collection mechanism.

The sampling plan is preferably randomized or at least looking for maximum variation along identifiable dimensions.

An “n” of at least 30 or a justification for a lesser number that is similar to a justification for a lesser number in a quantitative project.

A data analytic plan (thematic, content, constant comparison, linguistic, etc.) that matches the question, including whether or not computer based text analysis is being used. When computer analysis is present, the selected program must be able to do the analysis proposed. In most cases the variables for analysis should be identified beforehand.

Inter-rater reliability should also be addressed. Change resulting from the research will need to be

measured.

Interpretive Design: An appropriate design could include ethnomethodological,

constructivist, phenomenological, etc. and should be identified and justified. If the design is emergent, benchmarks for the emergence should be clear including how sampling and data collection will begin and might change.

Sampling for maximum variation is preferred and should identify the dimensions of variation of interest, the stakeholding groups to be sampled, and what might constitute saturation in determining the end of sampling and data collection.

A data analytic plan and the type of analysis should match the question and if computer-based analysis is being used, the program should be geared to interpretive analysis. Variables for analysis will rarely be identified beforehand; therefore, content analysis and constant comparison would be preferred.

Inter-rater reliability is not favored. Instead, member checking with participants is preferred.

Critical Interpretive Design: An appropriate design could include an interview study,

narrative research, life history, etc. and should be identified and justified. If the design is emergent, benchmarks for the emergence should be clear including how sampling and data collection will begin and might change.

Sampling will tend to be for a single case; but if the “n” is greater, maximum variation should identify the dimensions of variation of interest, the stakeholding groups to be sampled, and what might constitute saturation in determining the end of sampling and data collection.

A data analytic plan and the type of analysis should match the question and if computer-based analysis is being used, the program should be geared to interpretive analysis. Variables for analysis will rarely be identified beforehand; therefore, content analysis and constant comparison would be preferred.

Inter-rater reliability is not favored. Instead, member checking with participants is preferred.

Plans for documenting individual change as a result of the process can be noted.

Data Analysis & Reporting

Paradigmatic Implications Content Analysis

Thematic AnalysisLinguistic AnalysisSemioticsConstant Comparison

ReportingCountCase StudyNarratives

Content Analysis Thematic Analysis - reading “chunks” of data for

meaning and attaching a meaning label (can then move to word count/distribution analysis)

Linguistic Analysis - assessing word usage, message style (can then move to word count/distribution analysis)

Semiotics - reading for meaning through use of metaphor (can then move to word count/distribution analysis)

Constant Comparison - deconstruction and reconstruction through unitization and categorization (can then move to word count/distribution analysis)

Reporting

Count- data display, distribution tables, conceptual or causal maps

Case Study – to chronicle (to record temporally and sequentially as in a history); to render (as in a description or to provide a vicarious experience); to teach (as instructional material); to test (using case as a trial for certain theories or hypotheses)

Narrative – factual (he said/she said); interpretive (novel like); evaluative (elaborative judgments for evaluative testing)

Computer Assisted Qualitative Data Analysis Software

(CAQDAS)

Misconceptions about software:

A primary research instrument A separate form of analysis Enhances rigor Packages have similar paradigmatic

assumptions

Data Analysis with CAQDAS

Practical Considerations Available technology Skills and abilities Time

Political Considerations Institutional expectations (IRB, etc.) Peer review/professional/funding expectations

Paradigmatic Considerations Basic design assumptions Package similarities and nuanced differences

CAQDAS Disciplinary Backgrounds

Atlas.ti – Interdisciplinary, Psychology, Linguistics, Computer Science

NVivo – Sociology MAXqda – Political Science

CAQDAS Similarities

Unitizing Coding Annotating Retrieving Querying Graphic representation Writing Support

CAQDAS Differences

Types of data Coding structure Contextualization

However…

Access + Awareness

Paradigmatic Congruence

…When considering qualitative data analysis software:

Conclusion

When there is congruence between paradigmatic perspective, research goal and selected processes, articulation of what can and should be expected for accountability standards are possible and usefulness can be asserted.

Positivist work cannot provide deep, individualized meaning, but can be basis for appropriately targeted quantitative work or provide evidence to support causal hypotheses.

Critical Positivist work must engender class or structural changes from objective, generalizable data.

Interpretive work is not generalizable, but can provide complex, context-based deep understanding.

Critical Interpretive work must engender individual, subjective changes at the intersection of art, spirituality and scientific ways of knowing.

Protocol Reviews

(or…How do you defend what you are doing? Or assess student work?)

Try to determine underlying assumptions in order to establish the paradigmatic location of the research by evaluating the aims or goals.

Look for mixing of assumptions that suggest an incorrect understanding that qualitative research is always interpretive research

Apply the standards for design and analysis quality consistent with the paradigm.

Words to the Wise for the Researcher

Be aware of underlying assumptions in project that establish the paradigmatic location of the research, generally best articulated by project aims or goals.

Avoid mixing of assumptions that suggest an incorrect understanding that qualitative research is always interpretive research.

Articulate and demonstrate the standards for design and analysis quality consistent with the paradigmatic perspective of the project…then choose your software