Introduction to Focus Groups, Odum Institute, October 30

RTI International

RTI International is a trade name of Research Triangle Institute. www.rti.org

Intro to Focus Groups

Emily GeisenAmanda Wilmot

10/30/2013

RTI International

Ground Rules

Cell phones turned to vibrate or off Refreshments and restrooms any time Several bio-breaks throughout Questions are encouraged at end of each section

RTI International

Course Outline

1. The role and use of focus groups2. Focus group demonstration3. Developing the protocol guide4. Moderating focus groups5. Recruiting and scheduling6. Participant selection7. Data management/analysis8. Reporting findings9. Drawing wider inferences10. Qualitative Research Ethics

RTI International


1. The Role and Use of Focus Groups

RTI International

What is a Focus Group?

“Focus group interviews typically have five characteristics or features: (a) people, who (b) possess certain characteristics, (c) provide data (d) of a qualitative nature (e) in a focused discussion.”

-Focus Groups: A Practical Guide for Applied Research (Krueger)

RTI International

Examples

A group of coal miners sit around a small conference room discussing mine safety. They all agree that safety is important, but some argue that existing safety regulations are too unrealistic or impractical to be implemented.

A group of employees sit in a conference room discussing access to email and other work functions from mobile devices. One participant says answering emails on his phone will save him time. Another participant worries that her supervisor will expect her to reply to emails at all hours if she has access on her phone.

RTI International

What is a Focus Group, again?

Researchers have a topic they want to know about:– Why aren’t coal miners following safety precautions?– What concerns do employees have about mobile email access?

These topics are communicated to a group of participants to discuss– Pre-scripted questions to generate discussion– Targeted moderation to focus the discussion

Researchers summarize and interpret what they have learned

RTI International

Uses of Focus Groups

Exploration and Discovery– Collect info that will be used to guide development of a survey– Collect info for a needs assessment– Test a new product before development

Context and Depth– Adding new topics or population groups to a survey– Delving deeper into a survey topic– Testing a product with a new audience

Interpretation– Deciding how to use survey results– Discuss applications with end users

-Focus Group Guidebook (Morgan)

RTI International

Uses of Focus Groups*, Continued

Stand-alone data collection method, or Supplement other qualitative or quantitative

research methods– In-depth personal interviews– Survey development– Needs assessment– Product development

RTI International

Focus Groups for Survey Research

Identify topics/themes that are of interest to population– Use this to add questions on these topic/themes– Cut questions that are not relevant, not applicable, or show no

variation Get more detail on a topic so that you don’t have to ask

open-ended questions– Which of the following are reasons why you do not have health

insurance? Too expensive, No health problems, … Get reaction to survey materials such as advanced

letters, envelopes, etc. Explore methods to boost cooperation: what type of

incentives would people prefer?

RTI International

Strengths of Focus Groups (over other qual./quan. methods)

If you’re unfamiliar with a topic, you can learn a lot quickly (often used as the first step in research)

Can be used to observe interactions or group dynamics

Generate hypotheses (that can be tested using quantitative methods)

Cheaper/quicker than other methods

RTI International

Limitations of Focus Groups

Some topics are too sensitive or controversial to be discussed in a group setting (social desirability)

Discussions can get easily sidetracked/derailed

Limited generalizability of findings/results

Cannot quantify findings or determine statistical significance

RTI International

Thinking about your research…

How are you going to use focus groups? – Exploration and discovery? How so?– Context and depth? How so?– Interpretation? How so?

Will your focus groups be stand-alone or will you use them with other research methods?

What are the strengths/weaknesses of using focus groups for your research?

RTI International

Questions/Discussion

RTI International


2. Focus Group Demonstration

RTI International


3. Developing the Protocol Guide

RTI International

The What, Why, Who, How, Where

What Review research objectivesWho Determine number and type of participants

Recruit and schedule participantsWhere Determine test location and equipment

Why Identify major topic areas of interestHow Script focus group questions

Develop focus group guide

RTI International

The What, Why, Who, How, Where

What Review research objectivesWho Determine number and type of participants

Recruit and schedule participantsWhere Determine test location and equipment

Why Identify major topic areas of interestHow Script focus group questions

Develop protocol guide

RTI International

What’s the Focus Group About: Review Research Objectives

Start with the overall research objectives / study goals Identify what do you already know about the topic Determine what new information you want to learn Determine what information you want to explore in more

detail Decide how are you going to use this information

RTI International

Review Research Objectives: Example

Step Personal Health Records ExampleStudy goals: Learn about healthcare consumers

understanding, experiences, needs, and concerns about the ways technology can be used within the healthcare system

What do you already know about the topic:

Research has shown that use of electronic PHRs can save consumers money and improve health, yet they are not being used extensively.

What do we want to learn through focus groups:

Why aren’t more consumers using electronic PHRs to manage their health information?

How will we use this info: Provide recommendations for improving PHRs and promoting awareness.

RTI International

Review Research Objectives: Your Turn

Step Your StudyStudy goals:

What do you already know about the topic:

What do we want to learn through focus groups:

How will we use this info:

RTI International

Why: Identify Major Discussion Topics

Once you identify what you want to learn, identify major discussion topics (brainstorm)

Why are we doing these focus groups: – What is the most important question your research must answer?– What’s the next most important question?– And so on …

Consult with project team members as needed Consider how topic areas may need to differ by

participant selection

RTI International

Identify Major Discussion Topics: Example

Why aren’t more consumers using electronic PHRs to manage their health information?

Consumers who have not used electronic PHRs How familiar are consumers with electronic PHRs? How do consumers currently manage their health

information? What are the most important factors consumers consider

when deciding how to manage their health information What are their biggest concerns/fears about using PHRs?

Narrow list of discussion topics to about 4 major topic questions per hour (more if questions are more specific)

RTI International

How: Scripting Questions

Avoid yes/no or short-answer questions, unless: Open-ended follow up is planned (e.g., why/why not?) Used to tally group (e.g., those in favor/opposed)

Open-ended questions generate discussion, however:– Keep questions focused, one-dimensional– Be careful of “Why?” questions– Use scripted probes to focus or narrow topics

Questions should be easily comprehensible: use familiar words/terms

Questions should be reflective, not hypothetical Move from general to more specific

RTI International

Scripting Questions: Example 1

Topic: How do consumers currently manage their health information? Bad: How do you manage your family’s health

information? Good: How do you currently maintain or store your and

your family’s medical records, such as records of vaccinations or prescriptions?– Do you keep paper or electronic records? – How does this differ for you compared to your children (or elderly

parents)?

RTI International

Scripting Questions: Example 2

Bad: Why do you use the mobile device you have?

Better: What kind of mobile device do you use at work?

Best: How did you decide what kind of mobile device to use at work?

RTI International

Other aspects of Protocol Guide

Welcome/Introduction Icebreaker Opening (or Engagement) questions:

– Easy and inviting, comfortable to discuss– Introduces the topic of discussion, provides context/background

Exploration questions (main study questions) Exit questions:

– Used to wrap-up the discussion– Check to see if there is anything else people would like to

mention that they did not get a chance to Pre/Post Questionnaires

RTI International

Welcome/Introduction

Welcome– Introduce yourself, note-taker AND people behind the glass– Make participants feel comfortable

Informed consent (if required)– Read or summarize consent, ask participants to sign– Note audio-recording if applicable– Explain confidentiality

Go over ground rules– Refreshments, restrooms, cell phones off or on vibrate– No right or wrong answers – Like to hear from everyone

RTI International

Icebreaker

Sometimes it’s helpful to start with an ice-breaker Ask participant to provide their name (first name only) Have them answer an easy question

– Participants should be able to answer it briefly, quickly– Should not be sensitive– Can be related to focus group topic or unrelated

Example– Employee focus group: How long at company?

Department/Division?– Focus group of moms: How many children and their ages

RTI International

Intro Questions

Introduce Topic– Goal/Purpose of the study– Why participants were selected (if not obvious)

Provide any needed context– For focus group on Personal Health Records, we had to explain

what PHRs were– Show participants related materials or products– Provide background

Opening Question– Should be easy/simple to answer– Non-sensitive

RTI International

Pre/Post Questionnaires

Pre questionnaire:– Use to collect information about your participants without wasting

valuable time during the focus group– Use to determine how familiar/knowledgeable participants are

with topic at start of focus group Post questionnaire:

– Can be same as pre-questionnaire to monitor changes in opinion or knowledge

– Can be a handy way to summarize participants’ opinions on discussion topics

Keep questionnaires short (5 minutes)

RTI International

Putting it all together

Allow time for welcome, explaining the purpose of the study, and consent procedures (if required)– At least 5 minutes

Allow time for introductory or ice-breaker question– At least 5 minutes

Time for your main discussion questions– About 4 question topics (per hour)– 10-15 minutes per question

Exit/Wrap-up– At least 5 minutes

RTI International

Determining Question Flow

What order would you put these questions in?1. Describe the benefits of using a mobile device for the

work you do?2. How did you decide what kind of mobile device to use at

work?3. What are the biggest limitations or barriers to using a

mobile device for the work you do?4. How often do you use your mobile device for the work

you do?

RTI International

Question Flow

2. How did you decide what kind of mobile device to use at work?1. Describe the benefits of using a mobile device for the work you do?4. How often do you use your mobile device for the work you do?3. What are the biggest limitations or barriers to using a mobile device for the work you do?

RTI International

Protocol Guide Examples: Handouts

Handout 1: Mine Contractors Protocol Guide Handout 2: Healthcare Consumers Protocol Guide Handout 3: Tobacco Panel Protocol Guide Handout 4: Post questionnaire

RTI International

Individual Practice (15 minutes)

Use your study or class example Identify 1-2 major discussion topics for your study Prepare an introductory question Prepare 1-2 discussion questions with probes as needed

RTI International


RTI International


4. Moderating Focus Groups

RTI International

Understanding the group Process

Model of group phases (Ritchie and Lewis

2003)

Forming Testing and dependence

Dependence on the leader

Storming

Intragroup conflict Criticism

Norming Development of group cohesion

Optimism

Performing Functional role relatedness

Cohesiveness

Adjourning (Mourning) Termination, Separation

RTI International

The Ideal Moderator:

Has adequate knowledge of topic Has good communication skills Is similar to / can build a rapport with participants Can maintain balanced viewpoint Is prepared for surprises Reports accurately, even if it’s uncomfortable Has Experience with:

– group dynamics– neutral probing/feedback

RTI International

What the Moderator Does

Welcome/Introductions Leads discussion Concludes discussion

RTI International

Welcome/Introduction

Make participants feel comfortable Read or paraphrase this welcome/introduction in

protocol guide Make sure to collect informed consent Notify participants of observers and/or audio recording Go over any ground rules Let participants get refreshments and get situated before

starting actual focus group discussion

RTI International

Leads Discussion

Provides unobtrusive control Manages group dynamics Asks spontaneous probes as needed to gather more

information Provides unbiased feedback Summarize/verify information Manages time

RTI International

Provide Unobtrusive Control

Know when to stop talking– Don’t constantly interject, let participants discuss back and forth

Learn to focus the discussion– You said X, can you tell me more about that?– Note topics said, that you want to circle around to

Manage the flow of the discussion– Don’t have to follow script, allow natural segues– If a topic you plan to cover later is brought up, indicate that you’ll

talk more about that later. Encourage different point of views

– Has anyone had a different experience?– Does anyone view it differently?

RTI International

Tips for Managing Group Dynamics

Dominant talkers:– Don’t look at them when you ask a question.– (Politely) cut them off: “Thank you Amanda. Does anyone else

have an experience they’d like to share?” Ramblers/Off-topic:

– Avoid eye contact after so long off topic, look down or at clock– Be prepared to interject at next pause

Shy participants: – Maximize eye contact– Call by name: I’ve heard from some of you on this and I want to

hear from the rest of you. Sarah, what is your experience with X?

RTI International

Tips on Spontaneous Probing

Leading probes:– So you had a hard time with that then?– You didn’t want to learn more about that?– Was that difficult for you?

Neutral probing:– Can you tell me more about that?– Would you explain that further?– Can you give me an example?– Is there anything else?

RTI International

47

When you hear yourself asking a leading question, balance it

Leadingquestion

“So you think that’s difficult then?”

Balanced question

“...or was it easy?”

RTI International

Tips on Neutral Feedback

Monitor your reaction to participants/discussion Be careful of nonverbal cues (leaning back, crossed

arms) Limit head nodding: shows you’re listening, but also

implies agreement Provide quick verbal responses periodically (uh-huh,

yes, mmm, okay, go on) Avoid loaded feedback (which implies agreement)

– Loaded feedback: “That’s good!” “Excellent.”– Neutral feedback: “That’s helpful” “That’s interesting.”

RTI International

Summarize/Verify Information

As needed, summarize/paraphrase what has been said– I understand that work-life balance is your most important

concern.– Safety is critical, but some of the precautions you’re asked to

follow are pointless or unrealistic. Is that correct? See if anything has been missed?

– Did I miss anything?– Does anyone have anything else to add?

Probe on any subtopics you didn’t cover– We talked about X, but I’m also interested in Y

Move on to next topic

RTI International

Conclusion / Wrap-up

Be sure to leave time for wrap-up, especially for sensitive topics or intense discussions

Ask participants to identify most important issues– Of everything we have discussed, what is the most important?– If you could make one request to X, what would it be?

Summarize major themes (ask note-taker for help)– Ask if its an adequate summary– Ask if anything has been missed, anything to add

Check with observers to see if they have any questions/clarifications

Pay participants!

RTI International

Dealing with Difficult Situations

Too many people show up Participants bring their children Only a few attend Group is reluctant to talk Can’t get them to stop talking

RTI International

Plan Your Response…

20 minutes into the focus group, and Sarah has not said anything…

You can’t get Joe to stop talking…

John mentions something interesting, but you don’t want to interrupt because there’s a good discussion going…

RTI International

Role of Note-Taker

Can Handle logistics & refreshments Collects signed informed consent (if required) Takes careful notes Does not participate in discussion Can recap major themes at end of discussion (used

before wrap-up question) Monitors recording equipment Liaison between moderator and observers/clients Debriefs with moderator after session Assist with analysis and reports

RTI International

Role of Co-Moderator

Not required, but can be useful in some situations Balance out strengths/weaknesses in moderator Use to match moderator (without being obvious) Switch leading focus group (good for long or intense

focus groups) Support leader by keeping on track, recapping major

themes, etc.

RTI International

Moderating style

Different styles for different people– Spontaneous flow– Follow guide strictly

Adapt to your style, but plan ahead– Spontaneous: Be sure to note what you’ve covered and what you

haven’t so that you don’t forget anything. – Strict: “That’s a very interesting point, and we’re going to get to

that in a minute, but right now I want to focus on X:

RTI International


Group Practice Moderating (30 min)

RTI International


RTI International


5. Recruiting and Scheduling(review on your own)

RTI International

At Least a Month Before Focus Group

Decide on what you’re testing Decide on the participants (see Section 6) :

– Who you want to recruit– How many– How you will find them

Plan the test date– Find a date when your stakeholders can observe– Decide on your location

RTI International

How to Find Your Participants

Frame/list if available Advertisements: Craigslist, flyers, newspapers

– Pros: quick, easy, cheap– Cons: yields younger, higher educated users

Participant databases/recruiting orgs– Pros: quick and easy, pay per user recruited– Cons: can lead to “professional respondents”

Other sources: word of mouth; sample lists; clubs, churches, and societies; snowball recruiting

RTI International

What to Pay Participants?

Are participants allowed to accept monetary incentives?– Federal employees– Participants who are being paid by their employer

Is it appropriate for your organization to pay participants? Is topic interesting/important enough that incentives are

not necessary How much should you pay them?

– Trade-off between incentive amount and recruitment effort– Tailor amount to population and location– Will participants need to pay for parking?

RTI International

Recruiting Tips

Recruit extras due to no-shows or cancellations– Example: Recruit 12 to ensure at least 8 show up

Schedule sessions about 3-4 weeks ahead– Any earlier and they may forget or make new plans– Any later and you may not be able to recruit enough

participants Send them an email or letter confirmation Remind them the day before (by email or phone)

RTI International

Testing Location

At a minimum, use a large conference room (at your organization’s offices, hotel conference room)

For numerous focus groups: formal observation rooms are nice (see picture on next slide)– One-way mirror so you can see them, but they can’t see you– Microphones in ceiling to pick up voices, but not other sounds

(paper shuffling)– Video-taping capability

Rent a focus group facility– Usually very nice but can be expensive ($1,000 for 2-hour

session)

RTI International

Observation Room Example

RTI International

Create a Virtual Observation Room using screen-sharing software

Examples: Skype, Go to Meeting

Fosters collaboration– Can accommodate observers from any location– Facilitate discussions in conference setting

Improved schedule– Stakeholders get information immediately– No waiting for recorded videos or report

Cheaper: Inexpensive compared to travel costs

RTI International

Note-Taking

Recommend having a note-taker and audio or video-recording the session

Note-takers should be in the room or observation room For audio-recording, consider getting additional

microphones so that you can hear everyone Video-recording can be helpful, but makes some

participants uncomfortable Participants usually forget they’re being recorded/

observed after 5-10 minutes

RTI International

At Least Three Weeks Before Focus Group

Get started on recruitment Organize your incentives Develop your focus group guide

RTI International

One Week Before the Focus Group Final your focus group guide Organize roles in the test:

– Meet and greet– Observers/Stakeholders– Moderator– Note-takers

Do a practice run on any equipment Arrange any refreshments

– At a minimum provide beverages– Tailor food to participants / time of day– Nothing messy or loud

RTI International

Prepare your materials

Develop consent forms, screeners Instructions/directions for participants Prepare any visual materials for participants Pretest/posttest questionnaires

RTI International

The Day Before / Morning Of

Send out reminders:– Phone or email to respondents– Email to observers, stakeholders, note-takers

Equipment/Facility– Make sure the room you’ll use is tidy– Make sure your meet/greet person

has the final list of participants’ names– Incentives are available– Check any equipment (video/audio recording)

RTI International


6. Participant Selection

RTI International

Purpose of the session

To understand the principles of participant selection for focus group research

To consider the practicalities of participant selection for focus group research

RTI International

Focus groups: a qualitative data collection method

Focus groups are a qualitative data collection method When reporting the findings the rationale should be

provided for the research along with the methods used Is the research…

– Justified– Rigorous– Systematic– Transparent

RTI International

Sampling for qualitative research

The design of a sampling strategy for qualitative research is as important as that for quantitative research

Qualitative research uses non-probability sampling

RTI International

Informing the design of a qualitative sampling strategy

What are the research objectives? What is the scope of the research? Who is out of scope and should be excluded from the sample? Who is in scope and should be included in the sample? What is the budget? What is the reporting time period? What sampling technique will be employed? How are the data to be analyzed? What data collection methods should be employed? What are the sample criteria? What size should the sample be? What will be used as the sampling frame? How are potential respondents/participants recruited?

RTI International

Sampling technique

Quantitative sampling Probability sampling

Members of the research population are chosen at random and have a known probability of selection

The aim is to produce a statistically representative sample

Qualitative sampling Purposive non-probability sampling

The number of people interviewed is less important that the criteria used to select them

Members of the research population are chosen on the basis of their characteristics to reflect breadth and diversity of the research population

We do not aim to produce a statistically representative sample or draw statistical inference

RTI International

Sample criteria

What characteristics will need to be reflected in the sample population to ensure breadth and diversity?

Criteria used may be based on demographic characteristics or behaviours or attitudes

Some criteria may be considered more important than others

RTI International

Sample size

Small sample sizes for qualitative researchThere is no need for scale because there is no need for

statistical inference

Sample size determinants: Heterogeneous or homogenous nature of sample population Number of selected criterion

Scale 10 to 50 for one-to-one investigation 40 to 100 for group interview

RTI International

Sampling frames

A sampling frame is a list that identifies units within the target population

Frame evaluation: Comprehensive Sufficient numbers Geographical dispersion Respondent contact details correct

RTI International

Sampling frames

Existing frames– Administrative sources– Survey samples

Constructed frames– Direct and/or indirect methods

RTI International

Constructed frames

Focussed enumeration Snowballing Screening questionnaires Organisations Advertisements

RTI International

Study about attitudes and behaviors surrounding dental attendance

Selection criteria initially considered Age to ensure demographic balance Gender because patterns of attendance differ between men and women Family Unit composition because attendance by others in the family might

influence the respondents attitudes or behaviours Employment activity because attendance might be affected by time constraints

during working hours Income as a known factor affecting dental attendance patterns Regional location as dental attendance varies across the country Ethnic origin as may influence attitudes or behaviours Type of area as urban/rural location may affect attitudes and behaviours Dental health to explore how attitudes vary among people with different dental

health Current pattern of dental attendance (regular/irregular/occasional) for comparative

analysis

Illustration from Ritchie J and Lewis J ‘Qualitative Research Practice’ (2006)

RTI International

Prioritised selection criteria

Primary criteria Secondary criteriaDental attendance pattern Dental HealthAge Ethnic OriginGender Type of AreaRegionFamily Unit Employment activityIncome


RTI International

Sample Matrix

Area 1 of 6

Group 1 Group 2 Group 318-29 30-44 45+

4 men 4 men 4 men1-2 regular attenders 1 regular attender 1-2 regular attenders1-2 irregular attenders 1-2 irregular attenders 1 irregular attender1 occasional attender1-2 occasional attenders 1-2 occasional attenders

4 women 4 women 4 women1-2 regular attenders 1 regular attender 1-2 regular attenders1-2 irregular attenders 1-2 irregular attenders 1 irregular attender1 occasional attender1-2 occasional attenders 1-2 occasional attenders

Spread of family unit type and employment activity


RTI International

Any Questions?

RTI International

Excercise

Groups Designing a sampling strategy for focus groups on the

use of electronic personal health records 15 mins Feedback

RTI International


7. Data Management and analysis

RTI International


To understand the principles of good qualitative focus group analysis

To consider the practicalities of analysing qualitative data

RTI International

Acknowledgement

Some of the slides contributed by the UK National Centre for Social Research (NatCen).

The Framework methodology was developed by Jane Ritchie and Liz Spencer in the 1980s at NatCen.

http://www.natcen.ac.uk/events-and-training/our-training

RTI International

Qualitative data

Focus group recording

Focus group transcription

Notes

Admin data

Photographs, videos etc

RTI International

Transcription Transcription is time consuming The focus group transcription should capture the

discussion in it’s entirety Focus groups are more difficult to transcribe than one-to-

one interviews as there are multiple participants to distinguish between

It may be important for the analysis for the transcriber to be able to identify each participant’s words. The transcriber will use the notetaker’s notes to help determine who is speaking

Check on quality of the transcription Anonymize transcript

RTI International

Different approaches to data management, analysis and reporting

Summary report

Qualitative analysis

RTI International

Nature of qualitative analysis

Quantitative research– Fixed categories aiming to

enumerate– Requires enumeration in

order to be meaningful– Test hypotheses and

provide explanation

Qualitative research- Creating categories- Developing explanations- Does not need enumerating

- Range and diversity key

RTI International

Aim of analysis - analytical outputs

Categories of things (thematic analysis) Reasons for gambling Sources of debt advice

Categories of people or processes (typologies) Types of parents of children with learning difficulties Types of welfare benefit claimants

Explanations of attitudes, choices or impacts (explanatory analysis) Factors influencing how people save for retirement What helps achieve positive outcomes from an employment

programme

RTI International

The nature of qualitative data

The data are:

Voluminous

Unwieldy

Multi-stranded with overlapping categories and theme

RTI International

7.1 Data management

“Chaos is merely order waiting to be deciphered” Jose Saramago

Introduction to Framework © NatCen Learning

RTI International

What is Framework?Developed by Jane Ritchie and Liz Spencer in the 1980s

at NatCen Social Research

Key characteristics Approach to data management which facilitates case

and theme based analysis of qualitative data Case and theme based approach Matrix display Reduces data through summarisation and synthesis Retains links to original data Output allows comprehensive and transparent data

analysis


RTI International

Objectives of data management

Primary objectiveRe-order Make data accessible

Secondary objectiveReduce, prioritize


RTI International

Data management• Data management involves ‘cutting’ up data

• Aim is to create useful ‘piles’ of data

• Three ways you can ‘cut’ data Thematically By case By case and theme


RTI International

Theme-based approach

Crime Contact with CJS Impact


RTI International

Case-based approach

Jim

BobWendy

Robbery

Contact with police

Financial impact


RTI International

Framework is a case and theme based approach

Name

Crime Contact with CJS

Impacts

Jim

Bob

Wendy


RTI International

The Framework approachFamiliarisation

Read proposal and transcriptsReview topic guide and field notes

Identify descriptive categoriesDevelop working analytical framework of key

themes and sub-themes

Pilot ChartingChart a few transcripts using working

framework and the revise

IndexingApply numerical series to framework, label

data sources and revise framework

ChartingSummarise/synthesise data

Investigation and interpretation

Reporting findings


RTI International

Stage 1: Data familiarization process

• Familiarize yourself with the data by reading through transcripts and other documentation

• Immerse yourself in the data

• Begin to construct your analytical framework


RTI International

Stage 2: Creating the ‘framework’

Multitude of potential descriptive categories different kinds and levels of category

Choice of categories depends on interview guide research questions emergent themes recurrence across cases

Data management interpretation

RTI International

Frame construction: individual data

One chart represents one main theme

Chart each individual case– Each individual as one row– Subthemes in each column– Give cases same space on each thematic chart

RTI International

Frame construction: group data

Chart as a whole group– Each group as one case– Several groups on each thematic chart

One group per page, tracing individual cases– Order cases in the same way– Give cases same space on each thematic chart– Don’t be alarmed if every cell is not filled for each case

Guidelines for entering data– May be useful to keep a record of group process– Levels of participation, context in which comments made, how

views evolve/develop/change


RTI International

Respondent

Employment situation/ activity at time of contact

Employment history/work activity in past

Hopes & aims regarding work before contact

Barriers perceived in achieving aims

Efforts made to overcome barriers before contacting NDLP

Perception of needs before contact

Overview of financial situation & views about at time

Other/Misc

# 1

# 2

# 3

# 4

A chart with sub-categories

Before contact with employment scheme

RTI International

Stage 3: Testing the analytic framework

Testing framework is important– multi-stranded & overlapping categories– different experiences

Test by coding or indexing– apply numerical code to categories and sub-categories– code using software or write codes in margin of

transcript Test by piloting

– enter data from small number of transcripts

Checking for: overlap / gaps

RTI International

Data management coding / indexing

Example of analytical framework index for sexual identity project1. Demographic details2. Defining sexual identity

2.1 Conceptualization2.2 Salience2.3 Self categorization2.4 Changes over time

3. Language used3.1 Terminology used3.2 Categories used

4. Acceptability4.1 Acceptance by society4.2 Understanding purpose of questioning4.3 Trust in data security4.4 Trust in data collector

5. Other

RTI International

Developing an analytical framework exercise

Groups to discuss analytic framework that could be used for the morning focus group session

Create main themes and subthemes

Post-it notes

15 mins

Feedback – group spokesperson


RTI International

Stage 4: Summarising data (‘charting’)

• Condensing data• retains richness and flavour of

data • Process

• theme by theme (if indexed) or by transcript

• Takes practice – an art

RTI International

RTI International


RTI International


RTI International


RTI InternationalEntering data (‘charting’) Process theme by theme (only when indexed) by transcript Guidelines for entering data summarise material from transcripts/other data

sources retain language mark but don’t recite quotations note page references / create links between

summaries and the original transcript within CAQDAS software

use agreed abbreviations/conventions avoid repetition by cross referencing data


RTI International

Data Management: Good practice

Reading the transcript beforehand structure of account and nature of the data repetition, clarification, contradiction

Reading chart afterwards clarity balance and emphasis check blank cells Use ‘other’ column for interpretative notes note ‘flavor’ of discussion note overarching issues alert others to important issues

RTI International

7.2 Interpretation

“All meanings, we know, depend on the key of interpretation”

George Elliot

RTI International

Descriptive or explanatory analysis

Descriptive accounts

Explanatory accounts


RTI InternationalThere are three key steps…Detection

familiarisation extraction (highlight/ summarise)

Categorisationcreating meaningful conceptual boxesassigning data

Classificationcreating higher order categoriesassessing relationships between categories


RTI International

Categorization and classificationLone parents orientation to the Labor Market• Relevant dimensions

Attitudes to work Barriers to work Stage of job search activity

• Typology Work not currently an option Beginning to think about work Personal issue barriers Labor market barriers Close to work

RTI International

Categorization and classification (2)

Classification of sexual identity Relevant dimensions

– Conceptualization– Salience– Self-categorization– Changes over time

Typology– Latent identifiers– Conscious identifiers– Reluctant identifiers

RTI International

Associations and explanations

Linkages between:– Two attitudes– Attitude and behavior– Circumstances and need

Verified through explanation– Explicit respondent accounts– Implicit identified by the researcher

RTI International

Making explanations

Informed by– Hunches and hypothesis– Reflections during field work and analysis– Other research or theories

Involves– Detailed within case analysis– Comparison between cases– Repeated interrogation of data

Comprehensive Expect multiplicity

RTI International

Levels of classification and interpretation

In summary: Descriptive categories

– Factors, reasons, impacts Classifications or typologies Explained associations Meaning Theory/Strategy generation

RTI International

Any Questions?

RTI International


8. Reporting the Findings

RTI International

Purpose of session

To provide a basic overview of the principles involved in reporting qualitative findings and constructing a final report.

RTI International

Aims of the final report

To complete and document the research process To inform key stakeholders of the findings To communicate the findings in a clear and coherent way To guide readers in the interpretation of findings (avoid

misinterpretations)

RTI International

Basic principles for writing

Tell the story Structure the report Know your reader Write in plain English Use visual representation

RTI International

Reporting the Findings

Revisit the objectives (ensure the report covers these) Consider the audience Check report format Consider time available Consider length of the report (focus on relevant points

only) Report findings in past tense

RTI International

Structuring the report

Title Contents Executive summary Introduction / Background Methodology Findings Conclusions Recommendations Appendices e.g. sample matrix, interview guide

RTI International

Nature of the reporting

Descriptive

Explanatory

RTI International

Defining the boundaries Research rationale Methodology

– Sampling strategy– Data collection– Method of analysis– What can or cannot be inferred form the findings

Examples of relevant documentation– Interview guide– Contact letter– Screening questionnaire– Sample matrix

Substantiate conclusions by grounding the data in the findings– Examples– Quotations– Case illustrations

RTI International

Using Quotations

Avoid over-use Avoid under-use Present range and balance Edit sparingly Should provide illustration only Amplify but do not repeat a point Avoid reliance on a few articulate respondents Avoid very long quotations Identify relevant characteristics Preserve participant confidentiality

RTI International

Quotation examples

“I don’t feel you have to be out in all contexts…” [Gay man, 35-44]

“The Government uses statistics to show what it wants to show, for example unemployment, they’ve used all sorts of different measures for employment to show it’s going down…rather than it necessarily going down itself.” [MHE Wales]

“They don’t XXX care about us. It’s everyone for themselves” [Male, 18-24, Central London]

RTI International

Quotation examples

“I think it depends quite a lot on the way you live and how old you are. For example, I was born down [South West county], and my parents being extremely religious, have very definite views on sexuality.” [Bisexual woman, aged 35-44]

“I’ve lived in [Muslim country], nobody would ever admit that [being gay], so they all choose to be heterosexual. Somebody’s not been here that long, it’s even more of an issue.” [Heterosexual woman, aged 35-44]

RTI International

Is a quotation necessary?

The actual presence of the question was considered to have an important purpose in its own right. It was thought that as the questioning became more commonplace it would demonstrate to the wider public that non-heterosexuality was unremarkable. “It’s having it on there, it’s beginning to usualise the question, it’s a cultural shift, it’s enabling people to gradually recognise it’s no bloody big deal. But if we don’t have it there, we are setting up this whole process of we must be quiet about these people and all the rest of it.” [Gay/lesbian woman, 55+]

RTI International

Avoiding quantitative language

Avoid tendency to use numbers Do not discuss qualitative results in terms of proportions,

percentages or statistics Indicate strength of finding by discussing in terms of:

– A common finding, A recurring problem, Problems were observed…. etc

Use diagrams for illustration

RTI International

Principles to remember

Present balanced findings Present grounded conclusions (avoid assumptions) Distinguish between findings and your own conclusions Provide commentary alongside the analysis (do not

leave the reader to do this for you) Support with evidence from other research findings (if

appropriate)

RTI International

Editing and reviewing

Allow some time before editing (if possible) Review the content Check accuracy of statements Peer Review Proof read

RTI International

Any Questions?

RTI International


9. Drawing Wider Inferences

RTI International


To understand whether wider inference can be made when using qualitative data

RTI International

Concerns

Small samples

Not statistically representative

Interviews/discussions not standardized

Proximity of researcher / too personal

Biased reporting

RTI International

Generalization

Can qualitative findings be generalized beyond the sample and context of the study research?

Different definitions of ‘generalization’.– Representational generalization.

RTI International

Representational generalization

Meaning Inference to parent or sampled population Whether range and diversity (experiences, explanations

etc) can be matched with the sampled population Whether list (of experiences, explanations etc) is

inclusive of what would be found in the sampled population

Basis Depends on validity and reliability of findings

RTI International

Validity

Validity: Accuracy of representation

Has the researcher understood the issues from the respondent’s perspective?

Have the issues been fully articulated and explained

Are interpretations underpinned by the data

Alternative terms: Credibility and Plausibility

RTI International

Reliability

Reliability: Replicability Internal reliability

– Extent to which assessments and judgments are replicated between researchers

External reliability– Extent to which findings would be replicated if the study were repeated with the same or different sample

Alternative terms: Confirmability, Consistency, Dependability

RTI International

Drawing wider inference

Depends on: Quality of sampling

– Reflects diversity of sample population

– Includes all key constituents– Includes outliers and a-typical

cases Quality of data collection

– Free from interference: neutral and objective

– Probing of meaning– Exploration of all relevant

explanations

Quality of analysis– Systematic, comprehensive and

inclusive– Within and between case

analysis– Displays diversity

Quality of interpretation and reporting– Comprehensive– Clear– Underpinned by data– Displays multiplicity of accounts

and explanations– Non-quantitative

Documentation– Transparency – Research process

RTI International

Validation of the inference

Validation– Comparison and check of fit– Deviant cases analysis– Member or respondent validation– Triangulation

RTI International

In summary

Validity: do the data and findings accurately reflect the behavior , views etc of respondents?– Content validity: the respondent’s expression or articulation of

the issue– Validity of interpretation: the researchers interpretation of the

issue Reliability: would the data and findings be replicated if

the study were repeated? Representational generalization: are the data and

findings applicable to the wider population from which the sample is drawn?

RTI International

Any Questions?

RTI International


10. Qualitative Research Ethics

RTI International

Qualitative Research Ethics

Check the legal regulations for the State / Country Check your organisations Institutional Review Board

(IRB) policies Understanding “informed consent” (verbal/written) Understanding “confidentiality” Basic principles of good social research

– Participant safety (Physical/Emotional/Psychological)– Researcher safety

Recompense

RTI International


References and Reading List

RTI International

Reading List Quality in Qualitative Evaluation: A Framework for assessing research

evidence. A quality framework. A report produced on behalf of the Cabinet Office by Liz Spencer, Jane Ritchie, Jane Lewis and Lucy Dillon, National Centre for Social Research. ISBN: 07715 04465 8. August 2003. Government Chief Social Researcher’s Office Crown Copyright 2003.

Assessing the Quality of Qualitative Research. Patient Education and Counseling 90 (2013) 1–3.

Qualitative Research Practice: A guide for Social Science Students and Researchers. Edited by Jane Ritchie and Jane Lewis. SAGE 2003.

Focus Groups: Theory and Practice. D. W. Stewart and P.N. Shamdasani. Applied Social Research Methods, Volume 20.

Focus Groups as Qualitative Research. David Morgan. Sage. 1989. Focus Groups: A Practical Guide for Applied Research. Richard Krueger.

Sage. 1989. What Are Qualitative Research Ethics? Rose Wiles. Bloomsbury Academic.

2013.

RTI International

Other Resources

http://www.eiu.edu/~ihec/Krueger-FocusGroupInterviews.pdf

http://www.tgci.com/magazine/How%20to%20Conduct%20a%20Focus%20Group.pdf

http://managementhelp.org/businessresearch/focus-groups.htm

http://assessment.aas.duke.edu/documents/How_to_Conduct_a_Focus_Group.pdf

http://www.rowan.edu/colleges/chss/facultystaff/focusgrouptoolkit.pdf











RTI International

Final Questions and Discussion

Technology

Introduction to Focus Groups, Odum Institute, October 30