Research Quality in Qualitative Methods

Class Session 6ADLT 673, March 17, 2014

Assumptions of Qualitative Design

• Are concerned with process over outcomes

• Are interested in the meaning of experience – how people make sense of their lives

• Act as the primary instrument for data collection and interpretation

Qualitative Researchers ….

Assumptions of Qualitative Design

• Involves fieldwork – the researcher physically goes to the setting or site to observe or record behavior

• Is descriptive—the researcher seeks to understand in terms of words or pictures

• Is inductive—the researcher builds abstractions, concepts, hypotheses, and theories from details

Qualitative Research ….

Characteristics of a Qualitative Research Problem

The concept is “immature” due to a lack of theory or previous research

The sense that the available theory may be inaccurate, inappropriate, incorrect, or biased

A need exists to explore and describe the phenomena and develop theory

The nature of the phenomenon may not be suited to quantitative measures

In a paper or proposal, the researcher should

• Specifically describe the type of design and its approach to data collection, analysis, and report writing*

• Narrative Inquiry

• Ethnography

• Case Studies

• Phenomenology

• Grounded Theory* Not an inclusive list – these are some of the more common

In a paper or proposal, the researcher should describe design characteristics

The discipline or field where it originated

A good definition of the design

The typical unit of analysis for the design

Types of problems investigated by the design

Types of data collection

Data analysis processes & formats for reporting

Any other special characteristics of the design

The Researcher’s RoleResearcher biases, background, steps taken re: reflexivity

Why this site? What was done at the site?

Will it be disruptive? How minimize? What effects might that have on the quality of the data?

How will results be reported? What will the “gatekeeper” gain from the study?

Indicate steps taken to obtain IRB permission

Comment on sensitive / ethical issues: confidentiality of data, anonymity of participants, and intentions to use research for intended purposes

Data Collection Steps• Setting the boundaries for the study• Parameters for data collection• Purposefully select participants who are best suited to

provide insight into the research question

• Collecting information through observations, interviews, documents, and visual materials

• Establishing the protocol for recording information

• Indicate the type of data to be collected and provide a rationale for it

• Review Table 9.3 on pp. 150-151 of Qual Procedures article for advantages and disadvantages of different methods

Thoughts on Interviewing • NOT to get answers, test hypotheses, or evaluate• In-depth interviewing is a means to understand the

experience of other people, and the meaning that they make of their experience

• “Tell me about…”• “Can you describe for me…” • What, how, and why questions• Interviewer: fewer words (short questions)• Interviewee: many words (allow for expansive

answers)• Use probes• Watch for markers and follow them up with a probe

Conducting Focus Groups• Requires a highly skilled facilitator to draw people out,

listen carefully, and encourage others to build on the topic

• Goal is to obtain in-depth, “rich, thick” descriptive data

• Important to build rapport with group – food helps, but finish it before you start to record

• Pose question to the group, then wait for first response. Build off that response by asking, “Can anyone else relate to that experience” or “Has anyone else had a different experience?”

• Encourage as many diverse responses as possible

• If the participant gives a short, one or two word answer, ask for an example, and keep asking for examples

Tricks of the TradeUse digital recorder(s) – need 2 or 3 for a table of 12 participants

No more than 4-5 questions per hour/ 6 to 8 in 1.5 hours

Round robin number your participants (no names) and have them use their number before speaking

If you conduct the intervention, ask another (qualified) person to serve as your facilitator

Use a transcriptionist familiar with qualitative data ; decide on degree of clean-up in transcription (verbatim / pauses and hesitancies)

Always listen to your data; do not rely on transcripts alone

Characteristics: Narrative StudiesStories from individuals and groups (can include documents) about individuals’ lived and told experiences

Stories are either told to the researcher, or co-constructed between the researcher and participant

Strong collaborative feature in all narrative inquiry

Types of narratives: Autoethnography, life history, oral history

Example: Seasons of a Man’s Life (1978)

• Developed the biographical interview method for narrative studies

• Explored the male life cycle: his 1978 book has remained the most comprehensive psychosocial understanding of male adult development to date

• Enabled him to propose a theory of the life structure as a series of stages, marked by the “mid-life” crisis

• Research Question: What does it mean to be an adult?

• Levinson and his colleagues at Yale studied 40 men from a sample drawn from four different occupational groups

Example: Seasons of a Man’s Life (1978)

• Unique features of the biographical interview method: following, rather than leading

• Interviewer is sensitive to feelings expressed by interviewees and attempts to explore themes in ways that have meaning for each study participant

• Interviewer and interviewee have defined roles, yet the relationship between them is one of equality, both being able to comment on the researcher on the basis of personal experience.

• Not simply an interviewing technique or procedure, but development of a relationship that permits greater understanding of the interviewee’s experience

Triangulation: An Important Feature in Qualitative Studies

Use of multiple measures of the same variable to increase confidence that the data reflect the phenomenon under study

• Source triangulation: use of a variety of data sources

• Investigator triangulation: Seven different researchers contributed to the research team in Levinson’s work; four co-authored the book that resulted

• Methodological triangulation: use of multiple methods to study a problem

Characteristics: EthnographyPurpose is to study cultural phenomena in a natural setting

Originated in anthropology

Researcher spends long periods of time living in the host culture to study it

Methods are primarily direct observations of activities of group studied; formal and informal interviews

Quality often depends upon time spend in observing and interacting with informants

Use of extensive field notes (taken or recorded and transcribed immediately after observations)

Quality in Ethnography (Patton, 1990)

Comprehensiveness of field notes

Variety in information gathered from different perspectives

Cross-validation and triangulation of different data sources

Representation of participants in their own terms; capture experiences in their own words

Selection of key informants wisely

Ability to synthesize vast amounts of data to compose a complete picture

Provide carefully timed feedback as part of the verification process of fieldwork; observe its impact

Field notes should include your own experiences, thoughts, and feelings as the researcher

Examples from Medicine

• Boys in White (1961) by Becker, Geer, Hughes, & Strauss – an early ethnography

• Original problem framed as a question: What does medical school do to medical students other than give them a technical education?

• Evolved to become a study of medical school culture; reflective of emergent design

Examples from Medicine

• Getting Rid of Patients (1985) by Terry Mizrahi – (based on dissertation research)

• Study of Internal Medicine residents’ experience in GME at SAMS (southern area medical school); suspected to be here at VCU

• Cultural study – 3 years; involved extensive participant observation and interviews

Characteristics: Case StudyBest used to study “how” and “why” questions within a bounded system

Ideal approach when the variables are inextricably linked to the context in which they occur

The goal of a case study is exploratory, does not include evaluation of an intervention

Tells a story, with intricate details, with special attention paid to specifics about the setting, culture, and the participants in the study

Individual can be the unit of analysis; can then conduct a cross-case analysis of multiple individuals studied

Case Study of Eight Physicians at MCV earning the M.Ed.

1. What impact has participating in the M.Ed. in Adult Learning program had on how you teach?

2. What changes have occurred in your beliefs, values, and assumptions about teaching and learning that you can attribute to your participation in the program?

3. To what extent has participating in the program helped you to realize both personal and professional goals as a medical educator?

Case Study of Eight Physicians at MCV earning the M.Ed.• Bounded system: cohort of physician learners

within one institution

• Methods included

• Interviews (beginning of program, mid-point, upon completion)

• Focus group (someone else conducted)

• Inductive analysis of data from class learning products: papers written, presentations made; CV analysis; projects, etc.

Characteristics: Phenomenology

Purpose is to identify phenomena as they are perceived by the study participants

What is the meaning, structure, and essence of the lived experience of this phenomenon by individual(s)?

Direct investigation and description of phenomena without a priori theories about causal explanations

Has many derivatives, such as phenomenological heuristic inquiry (Moustakas): researcher must have experienced the phenomena him/herself

Example: Research Question: How do midcareer women perceive and describe transformative learning through developmental relationships? (data collection: participants kept journals for 6 months; in-person and telephone interviews; focus group session)

Research Question: What do medical students experience to be the essence or essential structure of empathy? (interviews)

• Focus on the lived experiences of participants

Characteristics: Grounded TheoryInvolves the discovery of theory through the analysis of data

Involves both inductive and deductive thinking through a constant comparison of data at different levels of abstraction

Not a descriptive qualitative method

Researcher does not formulate hypotheses in advance since preconceived hypotheses result in theory ungrounded in data

Goal is to generate explanatory concepts: the unit of analysis is the incident as reported by individuals

Key Concepts in Grounded TheoryContinual questioning of gaps, omissions, inconsistencies, and incomplete understandings – this informs the need for additional data on the situation being studied

Open processes in conducting research, rather than fixed methods and procedures

Generate theory and data from interviewing, rather than observation practices

Data collection, coding and analysis occur simultaneously, not as separate components of research design

Inductive: theory must grow out of the data and be grounded in the data

How does grounded theory work?

Initial (or open) coding and categorization of data

• Identify important words, or groups of words in the data and then label them• In vivo codes, words taken verbatim from participants

Concurrent data collection and data analysis

• Researcher collects data with an initially purposive sample• These data are coded before more data are collected • Researcher constructs a theoretical proposition and then collects data to test the hypothesis • Engages in a constant comparison analysis of incident to incident, incident to codes, codes to codes, codes to categories, and categories to categories• Result is theory built up from the data themselves

Data CodingCoding is analysis – Reviewing field notes or transcripts and dissecting them meaningfully while keeping the relationships among the parts intact

Codes are labels for assigning meaning

Codes are attached to “chunks” of data: either words, phrases, sentences, or paragraphs

It is not the words that matter, but their meaning

Codes are used to organize and categorize data into themes and patterns

Coding can be done manually or with the aid of qualitative data software

Two Methods for Coding • Can begin with a provisional “start list” of codes based on

conceptual framework prior to fieldwork. This is called a priori coding

• List comes from list of research questions, literature review, problem areas studied, key variables to create an initial codebook which is modified through successive iterations of data analysis

• Coding can be done using inductive methods solely (grounded theory)

• Initial data are transcribed and reviewed line by line, typically within a paragraph. In the margins next to the paragraph, categories or labels are generated, and the list grows. The list is reviewed, modified, and continuously examined.

Want to learn more about software for coding qualitative data?• Available YouTube Videos from manufacturers:• Check out Nvivo 10 Tutorial• http://www.youtube.com/watch?v=oelXFnJ-7Ms

• Another option is Atlas.ti• http://www.youtube.com/watch?v=0skIbvmScsE (intro)• http://www.youtube.com/watch?v=_f1jO3B4Z18 (coding)• http://www.youtube.com/watch?v=YibqDB0iO-0 (coding and

analysis)

Rigor and Quality in Qualitative Research Methods

Quantitative Qualitative

• Internal Validity

• External Validity

• Reliability

• Objectivity

• Credibility (truth value)

• Transferability (applicability)

• Dependability or Trustworthiness (consistency)

• Confirmability (neutrality)

CredibilityAddressed by three issues: (1) the techniques and methods used to ensure integrity and accuracy of the findings; (2) the qualifications, experience, and perspective that the primary researcher brings to the study, and (3) the paradigm orientation and assumptions that undergird the study (Patton, 1990)

Trustworthiness When researcher describes in detail how successive interpretations of the data are carried out and makes this available for public scrutiny in publications; primary data should also be made available to participants for their verification (Reissman, 1993)

You TryTake a marker and use it as you review the three pages of transcript material if you want to use it

Create codes in the margin as labels you assign codes to “chunks” of meaning

Review codes with others at your table

Calculate inter-rater reliability of the first 10 codes you (collectively) determine

Discuss discrepancies as a group and see if you can come to agreement in meaning of a passage or group of words

What kind of coding guidelines can you, as a group, make to help you do better as a coding team next time?

Next Week’s Meeting Times

• Monday, March 24• 5 pm• 6 pm• 7 pm

• Thursday, March 27• 5:15• 6:30

• Blue (Fulco, Paletta, Zulfiqur

• Red (McIntosh, Lee, Ferrada

• Purple (DiGiovanni, McGinn, Carlyle, Marko)

• Orange (Dragoescu, Vinnikova, Woleben, Call)

• Green (Sabo, Massey, deWit, Uram-Tuculescu