Field Research and Analytic Transparency:Collecting Data, Analyzing Evidence, and
Drawing Inferences in Qualitative Research
Short Course #2APSA 2013
Instructors:Diana Kapiszewski, Georgetown University
Naomi Levy, Santa Clara UniversityColin Elman, Syracuse University
Building on a course initially developed and taught by Melani Cammett (Brown University),
Marc MorjĂŠ Howard (Georgetown University), Evan S. Lieberman (Princeton University),
Julia F. Lynch (University of Pennsylvania), Lauren Morris MacLean (Indiana University),
Benjamin L. Read (UC Santa Cruz), Scott Straus (University of Wisconsin, Madison) &
Sara Watson (Ohio State University)
Agenda
Part I â Conceptualizing, Designing, and Preparing for Fieldwork (2:00-3:25)
Part II â Data Collection (3:25-4:10)~ Break (4:10-4:20) ~
Part III â Interviewing (4:20-5:15)Part IV â Organizing and Analyzing Data &
Assessing Progress (5:15-6:00)~ Break (6:00-6:10) ~
Part V â Analytic Transparency (6:10-7:00)
Instant Field Research Survey
⢠Any prior fieldwork experience?â If so:
⢠how many months total ?⢠in what regions?
â If not: ⢠fieldwork planned for when?⢠and for where?
Where We Are in the Course
Part I â Conceptualizing, Designing, and Preparing for Fieldwork (2:00-3:25)1. Borders and Varieties of Fieldwork 2. Research Design and Fieldwork 3. Preparing for Fieldwork
Part II â Data Collection (3:25-4:10) * Break (4:10-4:20) Part III â Interviewing (4:20-5:15)Part IV â Organizing, Analyzing, Assessing (5:15-6:00) * Break (6:00-6:10)Part V â Analytic Transparency (6:10-7:00)
Fieldwork in Political Science: What Is It?
⢠Our working definition of field research:â Leaving oneâs home institution to collect data
or information that significantly informs a research project.
⢠Fieldwork is not delimited to oneâs time in the field
⢠Fieldwork is a very iterative process
Fieldwork in Political Science: Why Do It?
⢠Is fieldwork still necessary? (Yes!!)⢠Sometimes you just have to BE THERE⢠Allows building of networks⢠Provides opportunities ⢠Facilitates path-breaking empirical
scholarship⢠Less tangible reasons
Fieldwork in Political Science: What Does it Look Like?
⢠Heterogeneous!
⢠Different epistemological approaches
⢠Different types of settings
⢠Done independently or as part of a larger project
⢠Many types of data-collection techniques
⢠No âstandard fieldworkâ!
Fieldwork in Political Science: When Does it Happen?
⢠Selecting the topic⢠Reading existing literature⢠Defining the research question
â Scoping trip⢠Completed proposal
â âSurgical strikeâ tripsâ Three-four month tripsâ Long-haul stays
⢠Writing upâ Follow-up data-gathering
Loosely Structured: Open-Ended Research
Highly Structured: Narrowly Focused Research
Fieldwork in Political Science:Trade-offs Among Types
⢠Long stays:- Pros
⢠Experience another culture
⢠In-depth research⢠Contacts/network⢠Ideas⢠More relaxed & fun
- Cons⢠May be far away⢠Lack of urgency⢠âToo muchâ data
⢠Short trips: â Pros
⢠Efficiency⢠Forces you to take
stock/think analytically
⢠Identify comparisons/ contrasts quicker
⢠Less time away â Cons
⢠Can be more costly⢠Scheduling
difficulties⢠Not enough time
Emotional and Psychological Challenges
⢠Loneliness / isolation⢠NEVER being alone⢠Language/cultural
barriers⢠Family stresses⢠Financial stresses⢠NERVOUSNESS!
⢠Identify your concerns⢠Network⢠Balance⢠Know where to turn⢠Donât be stoic!
Where We Are in the Course
Part I â Conceptualizing, Designing, and Preparing for Fieldwork (2:00-3:25)1. Borders and Varieties of Fieldwork 2. Research Design and Fieldwork 3. Preparing for Fieldwork
Part II â Data Collection (3:25-4:10) * Break (4:10-4:20) Part III â Interviewing (4:20-5:15)Part IV â Organizing, Analyzing, Assessing (5:15-6:00) * Break (6:00-6:10)Part V â Analytic Transparency (6:10-7:00)
Design-Driven Fieldwork and
Fieldwork-Driven Design
⢠Field work must fit your research design
⢠Your design must accommodate field realities
⢠Competing Imperatives
â Your ideal research design
â Practicality
Design-Driven Fieldwork and
Fieldwork-Driven Design
⢠Design âdo-ableâ researchâ Think about your own limitations
⢠Find a wayâ Think positively
⢠Reviseâ Think in terms of variables
Fieldwork and the Research Design
⢠Nomotheticâ Thinking in terms of variables
⢠Idiographicâ Thinking in terms of cases
Data Matrix
Variable 1 Variable 2 Variable 3
Case 1
Case 2
Case 3
Case 4
Case Selection and Sampling
⢠Selection decisions arise at many stagesâ Macro-level
⢠Country casesâ Meso-level
⢠Regions or Towns⢠Time periods⢠Sectors, etc.
â Micro-level⢠Individuals for interviews⢠Documents for content analysis
Small-N
⢠Use case selection to provide causal leverage
⢠Use a variable-centered approachâ Hold rival explanatory variables constantâ Allow your primary explanation to vary
⢠Example: Dan Posner (2004)â Macro-level: Zambia & Malawiâ Meso-level: Town selection held rival variables
constant
Posner (2004) Meso-level Selection
Large-N
⢠Where possible, use random samplingâ This increases the generalizability of your findingsâ Need a list of the universe of casesâ Consider cluster sampling
⢠Sometimes not possible or desirableâ Special cases you want to includeâ Important variation you want to capture
Where We Are in the Course
Part I â Conceptualizing, Designing, and Preparing for Fieldwork (2:00-3:25)1. Borders and Varieties of Fieldwork 2. Research Design and Fieldwork 3. Preparing for Fieldwork
Part II â Data Collection (3:25-4:10) * Break (4:10-4:20) Part III â Interviewing (4:20-5:15)Part IV â Organizing, Analyzing, Assessing (5:15-6:00) * Break (6:00-6:10)Part V â Analytic Transparency (6:10-7:00)
The Months and Weeks BeforeâŚAdministration
⢠Fundingâ Develop a strategy far in advanceâ Apply for lots of money
⢠Dealing with your Institutional Review Boardâ Does not have to be difficult!â Follow the directions, adhere to deadlinesâ Find out about exemptions and consent
The Months and Weeks BeforeâŚIntellectual Prep
⢠Dig into your topic
⢠Background research on your country/ies
⢠Begin to write documents
⢠Collect documents youâll use in the field
⢠Brush up on methods
⢠Think about your foreign language skills
The Months and Weeks BeforeâŚReach Out!
⢠Develop your network of scholars and contact them!
⢠How about a host institution (research affiliation)?
Converting your Research Design into a Data Collection Plan
Concept/Sub-dim/ Other info
Indicators
WHAT data needed to measure
WHERE (source for data) and HOW to collect
WHEN to get/HOW LONG needed
DV
IV 1
IV 2
Other info.
Example Data Collection PlanConcept/Sub-dim/ Other info
Indicators WHAT data needed to measure
WHERE (source for data) and HOW to collect
WHEN to get/HOW LONG needed
Legal infra-
structure governin
g inf. work
1. # laws regulating inf. sector2. # const. rts. re: IWs
1. Lists of natâl., state, local laws2. Const. rights
1. Govât. ministries/offices/ on-line2. Const.
1. Early (1-2 mos.)
2. Early
Use of legal
system by
informal workers
1. IWsâ court cases2. Protest based on law/const rights
1. Record of cases filed with courts
2. History of protest
1. Courtsâ records/ on-line2. News-papers
1. Early (2-3 mos.)
2. Later
Converting your Research Design into a Data Collection Plan
Other aspects of res des/data collection
Data / info needed
Source for data/info. and how to collect
When to get/time needed
Information to inform case selectionPermissionsNames of people to interviewCausal process observations
About the Data Collection Plan
⢠You can head to the field before you have your entire Data Collection Plan filled out!
⢠Objectives of the Data Collection Plan â Helps make your project manageableâ Is a measure of your progressâ Is the link between all the millions of tasks
you carry out in the field, and your larger project
Converting your Research Design into a Data Collection Plan
Variable
Indicators
Data needed to measure
Where/how to get data
When to get/time needed
DV
IV 1
IV 2
IV 3
Where We Are in the Course
Part I â Conceptualizing, Designing, and Preparing for Fieldwork (2:00-3:25)
Part II â Data Collection(3:25-4:10)1. Forms of Data Collection2. Data Collection: Choices, Challenges,
Assistants * Break (4:10-4:20) Part III â Interviewing (4:20-5:15)Part IV â Organizing, Analyzing, Assessing (5:15-6:00) * Break (6:00-6:10)Part V â Analytic Transparency (6:10-7:00)
Non-Interactive Data Collection
⢠Observations
⢠Following the local mediaâ Newspapers, magazines, radio and TVâ Books & articles
⢠Obtaining Documents & Existing Datasetsâ Government agencies & Ministriesâ NGOs
⢠Archival Work
Interactive Data Collection
⢠Ethnographyâ Participant Observation
⢠Experimentsâ Laboratory experimentsâ Survey experimentsâ Natural experimentsâ Field experiments
⢠Large-scale surveys⢠Interviews⢠Focus Groups
Choices in Data Collection
⢠Strengths and Weaknessesâ Non-interactive
⢠Documents donât âreactâ⢠Documents and archives can be biased
â Interactive⢠Respondents can provide first-hand account⢠Respondents might be inaccurate
⢠Triangulate!
Triangulation
Where We Are in the Course
Part I â Conceptualizing, Designing, and Preparing for Fieldwork (2:00-3:25)
Part II â Data Collection (3:25-4:10)1. Forms of Data Collection2. Data Collection: Choices, Challenges, and Assistants
* Break (4:10-4:20) Part III â Interviewing (4:20-5:15)Part IV â Organizing, Analyzing, Assessing (5:15-6:00) * Break (6:00-6:10)Part V â Analytic Transparency (6:10-7:00)
Sequencing and Prioritizing Data Collection
⢠Distinguish necessary from desirable dataâ Centrality to core hypotheses or key variables
⢠Factors to consider in ordering data collectionâ Overlapping strategiesâ Harder-to-get vs. easier-to-get dataâ âLow-riskâ vs. âhigh-riskâ contactsâ Temporally tied events â Depth vs. breadth
Anticipate Data Collection Challenges
⢠Roadblocks to accessing elites, ordinary citizens, archives, datasetsâ Affiliateâ Network â find a âconnectionââ Think empatheticallyâ BUT consider investment: time, effort, $
Methods for Capturing Data
⢠Documentsâ Take notesâ Reproduce
⢠Interviews & Focus Groups â Field notes after the factâ Jotted notes duringâ Audio/Video tapeâ Combo
⢠How do decide?â How much data do you want to capture?â How are you going to use the data?
Methods for Capturing Data (contâd.)
⢠Reproducing/Recording Prosâ Taking home much more dataâ You arenât just relying on your notes and memoryâ Capture verbatim quotesâ Richer data
⢠Reproducing/Recording Consâ Taking home much more dataâ Costly
⢠Equipment⢠Storage⢠Transportation⢠Transcription
â Recording can make people nervous
Hiring Research Assistants
⢠Pros:â âGiving backâ to countriesâ Can be a great source of information â Building long-term relationships
⢠Consâ Training = time consumingâ May cause problems/quit/not follow instructnsâ May introduce bias
⢠Recruitmentâ How to find RAsâ Interviewâ What are you offering?â Contract
Where We Are in the Course
Part I â Conceptualizing, Designing, and Preparing for Fieldwork (2:00-3:25)
Part II â Data Collection (3:25-4:10) * Break (4:10-4:20) Part III â Interviewing (4:20-5:15)Part IV â Organizing, Analyzing, Assessing
(5:15-6:00) * Break (6:00-6:10)Part V â Analytic Transparency (6:10-7:00)
Where We Are in the Course
Part I â Conceptualizing, Designing, and Preparing for Fieldwork (2:00-3:25)
Part II â Data Collection (3:25-4:10) * Break (4:10-4:20)
Part III â Interviewing (4:20-5:15)1. Preparing and Writing Questions 2. Conducting the Interview/Follow-up
Part IV â Organizing, Analyzing, Assessing (5:15-6:00) * Break (6:00-6:10)Part V â Analytic Transparency (6:10-7:00)
InterviewingGeneral Guidelines
⢠Do your homework ⢠Interview âthose who studyâ before âthose
who doâ
⢠Varieties of interviewsâ Structured? Semi-structured? Informal chat?
⢠Scheduling interviews⢠Where to conduct the interview
Interviewing Writing Interview Protocols (I)
⢠Whatâs the goal of the interview? Will your questions produce useful data?
⢠Language⢠Question sequencing⢠Theoretically motivated/in colloquial terms⢠What are you asking?! ⢠Asking sensitive questions ⢠Weeding questions
Interviewing Writing Interview Protocols (II)
⢠Transitional language⢠Keep questions simple and direct⢠Get local input on your questions⢠Pretest⢠Crucial questions: how are you going to
analyze and use the data?
Where We Are in the Course
Part I â Conceptualizing, Designing, and Preparing for Fieldwork (2:00-3:25)
Part II â Data Collection (3:25-4:10) * Break (4:10-4:20)
Part III â Interviewing (4:20-5:15)1. Preparing and Writing Questions 2. Conducting the Interview/Follow-up
Part IV â Organizing, Analyzing, Assessing (5:15-6:00) * Break (6:00-6:10)Part V â Analytic Transparency (6:10-7:00)
Conducting the Interview
⢠Introducing yourself and your research⢠Interacting with your respondent
â Conversation vs. Interviewâ DOs and DONâTs
⢠Know your protocol⢠Prioritize your questions
Conducting the Interview
⢠Probesâ follow-up questions that are used to deepen a
response to a question⢠Types
â Basic signalsâ Detail-oriented Questions
⢠who, what, where, when, howâ Elaboration Probes
⢠âTell me moreââ Clarification Probes
⢠âWhat do you mean?â
Conducting the Interview
⢠Using Silenceâ One of the most useful âprobesââ Gives the subject the space to talkâ Wait longer than is comfortable for youâ Builds constructive tension
⢠Taking notesâ A form a body languageâ What to write
⢠Wrapping upâ Be thankful, be very very thankful!
After the Interview
⢠Write up your notes!!!â Worth being VERY disciplined about this!â Do it before you conduct another interviewâ Include a description of the person & locationâ Note the most important new informationâ Remember: writing notes is a form of data
reduction⢠If you record, dealing with your recordings
â Transcribe? âListen throughâ? Put aside? ⢠Send a thank you note!⢠Getting better
Activity: Interviewing
⢠Write out your 30 second elevator pitch. ⢠Try it out on your neighbors and get their
input.
⢠Before you startâŚâ What is an elevator pitch?â Questions elevator pitch should answer?â Language
Where We Are in the Course
Part I â Conceptualizing, Designing, and Preparing for Fieldwork (2:00-3:25)
Part II â Data Collection(3:25-4:10) **Break (4:10-4:20) Part III â Interviewing (4:20-5:15)
Part IV â Organizing and Analyzing Data & Assessing Progress (5:15-6:00)1. Organizing and Analyzing Data2. Assessing Progress and Wrapping Up
**Break (6:00-6:10)Part V â Analytic Transparency (6:10-7:00)
Data Organization:Questions to Consider
⢠Youâre Project Manager of your study â organize and systematize from the start!
⢠Why organize?â Helps you to see the progress youâve made â Eases re-entry
⢠What needs to be organized?â dataâ contactsâ thoughts
Organizing Your Data
Sources of Data
Interviews
Archives
Written Primary andSecondary Sources
Datasets
Observations
Misc. Documents
Forms of Data
Notes, Tapes, Transcripts
Notes, Copies
Notes, Copies
Electronic Files, Printed Copies
Notes, Tapes
Notes, Copies
Establish a System for Filing Physical and Electronic Documents
S A M P L E F IL IN G S Y S TE M
C on tac t lis tsB ib liog rap h yL e tte rs O u tL e tte rs In
A d m in is tra t ion
S u rveys (fo rm s)E con om ic d a taO th er c od ed d a tase ts
Q u an t D ata
In te rview tran sc rip tsN otes from a rch ivesS c h o la rly A rt ic lesR ead in g n o tesN ew sp ap er c lip p in g sR ep orts / B roch u res
Q u alit D a ta
D ata
O u tlin eS u m m ariesC h artsG rap h s
A n a lys is
Th es is ch ap te rsA rtic les
O u tp u t
R es earch P ro jec t
Organizing Your Contacts (I)
⢠Develop a contact management systemâ What matters most is that it works for you.
⢠General suggestionsâ Separate from your general address book. â Lots of fieldsâ Set up to automate thank-you letters â Easy retrieval for follow-up, thank you notes,
future projects â Make it the beginning of a database
Organizing Your Contacts (II)Chart for Potential Respondents
Status Respondent and contact info.
Position/Comments
Suggestion of whom
Progress (e-mails; phone calls, etc.)
X Fernando Prieto 513-5887âŚSec: Claudia
Head of Central Bank⌠was importantâŚ
Gustavo Sainz (said talk to him esp. about...)
02Jan2005 â spoke with secretaryâŚ
Status:X = need to findX = found, need to get goingX = goingX = set and/or doneX = donât do anything for nowX = give up/declined
Organizing Your Contacts (III)Interview Clean-up
Intrvw. code
Intervw. done
Names given added tocontactlist
Type up notes / things said off record /thoughts
Send thank you note
Still need/respondent promised/need to follow-up
EC-01 22 sept. 2005
22 Sept. 2005
22 Sept. 2005
22 Sept. 2005
Promised document onâŚ
Digesting Your Data
⢠Develop a strategy for keeping track of your thoughtsâ Make a habit of writingâ Organize your thoughts
⢠Begin your analysis while in the fieldâ Periodically read through your documents and notesâ Write tentative memos
⢠Prepare your data for analysisâ Think about your coding strategyâ Import data into software programs
Where We Are in the Course
Part I â Conceptualizing, Designing, and Preparing for Fieldwork (2:00-3:25)
Part II â Data Collection (3:25-4:10) **Break (4:10-4:20) Part III â Interviewing (4:20-5:15)
Part IV â Organizing and Analyzing Data & Assessing Progress (5:15-6:00)1. Organizing and Analyzing Data2. Assessing Progress and Wrapping Up
**Break (6:00-6:10)Part V â Analytic Transparency (6:10-7:00)
Assessing Your Progress (I)
⢠Challenge: simultaneously thinking micro (trees) and thinking macro (forest)
⢠Where are you on the âto getâ list?â How much data have you collected?â How many of the people you consider crucial have
you spoken with?⢠How about in terms of your argument?
â Have evidence to support your hypotheses?â Have evidence to address âalternative
hypothesesâ?⢠Get help to correct for âfield gogglesâ -- to
push larger questions to the forefront
Assessing Your Progress III
⢠Different vehicles for telling your evolving story â Periodic written reports or memos to advisors â Write conference papers.â Present your work in-country informally
⢠Upsidesâ Helps you do some early analysisâ Gets you feedback on your analysis and storyâ Helps you see where you really are!
Making Adjustments
⢠Itâs common for people to (want to) make some adjustments
⢠Carefully consider any change to your project â Be patient and set some reasonable
deadlines.â Carefully diagnose the problem.â Talk with others about your potential âfixââ Make sure your âfixâ addresses the problem.
What seems to be the problem?
Why is this happening?
Possible fixes
Should I change my topic?
I canât answer the question
Unfalsifiable question;Access issues
Triangulate;Come back later;Do without
Only if multiple, central hypotheses are untestable
Iâm surprised by the answer
Your initial guesses were wrong;Youâre not getting the full story
Make it a good surprise
Only if nothing of value to be gained from pursuing question in its current framing
Iâm testing the wrong hypotheses
Theories were inapplicable;You made a good guess, but guessed wrong
Come up with new hypotheses and ways to test them
Generally not a reason to change topics
Iâm asking the wrong question
Youâre bored;Your question got dated;It really was never the right question
Remind yourself why you asked this question;Take a historical view; Link to different theories
Radical change usually not advisable
Tweaking Your Project
⢠Micro-level Matrix -- Individuals
Association # of Grants/yr Sources
Person 1 Org 1 12 Government
Person 2 Org 1 10 Internationa
lPerson 3 Org 1 13 Government
Person 4 Org 2 2 Private
Person 5 Org 2 1 Private
Person 6 Org 2 3 Private
Tweaking Your Project
⢠Meso-level Matrix
Funding Sources Connectivity
Organization 1 Well-funded Govât &
IntâlOrganization 2 Struggling Private
Knowing When to Wrap Things Up
⢠Develop criteria to help you determine when you have âenoughââ Check your âto getâ listâ What information is only available abroad?
⢠Consider how much paper to cart home (or between countries)
⢠Return trips are an optionâ Research sites wonât disappearâ You will be able to hit the ground running
Preparing for âLife After Field Researchâ While in the Field
⢠Consider reintegration strategies â Request office space from your home deptâ Join a dissertation-writing groupâ To TA or not to TA?
⢠Try to plan out your first month backâ Unpack your boxes ASAPâ Deal with unfinished field businessâ Data to transcribe, code, enter, or clean upâ Reconnect with your advisors
⢠Plan to go easy on yourself!
Where We Are in the Course
Part I â Conceptualizing, Designing, and Preparing for Fieldwork (2:00-3:25)
Part II â Data Collection (3:25-4:10) * Break (4:10-4:20) Part III â Interviewing (4:20-5:15)Part IV â Organizing, Analyzing, Assessing (5:15-6:00)
* Break (6:00-6:10)Part V â Analytic Transparency (6:10-7:00)