Field Research and Analytic Transparency: Collecting Data, Analyzing Evidence, and Drawing Inferences in Qualitative Research

Field Research and Analytic Transparency:Collecting Data, Analyzing Evidence, and

Drawing Inferences in Qualitative Research

Short Course #2APSA 2013

Instructors:Diana Kapiszewski, Georgetown University

Naomi Levy, Santa Clara UniversityColin Elman, Syracuse University

Building on a course initially developed and taught by Melani Cammett (Brown University),

Marc Morjé Howard (Georgetown University), Evan S. Lieberman (Princeton University),

Julia F. Lynch (University of Pennsylvania), Lauren Morris MacLean (Indiana University),

Benjamin L. Read (UC Santa Cruz), Scott Straus (University of Wisconsin, Madison) &

Sara Watson (Ohio State University)

Agenda

Part I – Conceptualizing, Designing, and Preparing for Fieldwork (2:00-3:25)

Part II – Data Collection (3:25-4:10)~ Break (4:10-4:20) ~

Part III – Interviewing (4:20-5:15)Part IV – Organizing and Analyzing Data &

Assessing Progress (5:15-6:00)~ Break (6:00-6:10) ~

Part V – Analytic Transparency (6:10-7:00)

Instant Field Research Survey

• Any prior fieldwork experience?– If so:

• how many months total ?• in what regions?

– If not: • fieldwork planned for when?• and for where?

Where We Are in the Course

Part I – Conceptualizing, Designing, and Preparing for Fieldwork (2:00-3:25)1. Borders and Varieties of Fieldwork 2. Research Design and Fieldwork 3. Preparing for Fieldwork

Part II – Data Collection (3:25-4:10) * Break (4:10-4:20) Part III – Interviewing (4:20-5:15)Part IV – Organizing, Analyzing, Assessing (5:15-6:00) * Break (6:00-6:10)Part V – Analytic Transparency (6:10-7:00)

Fieldwork in Political Science: What Is It?

• Our working definition of field research:– Leaving one’s home institution to collect data

or information that significantly informs a research project.

• Fieldwork is not delimited to one’s time in the field

• Fieldwork is a very iterative process

Fieldwork in Political Science: Why Do It?

• Is fieldwork still necessary? (Yes!!)• Sometimes you just have to BE THERE• Allows building of networks• Provides opportunities • Facilitates path-breaking empirical

scholarship• Less tangible reasons

Fieldwork in Political Science: What Does it Look Like?

• Heterogeneous!

• Different epistemological approaches

• Different types of settings

• Done independently or as part of a larger project

• Many types of data-collection techniques

• No “standard fieldwork”!

Fieldwork in Political Science: When Does it Happen?

• Selecting the topic• Reading existing literature• Defining the research question

– Scoping trip• Completed proposal

– “Surgical strike” trips– Three-four month trips– Long-haul stays

• Writing up– Follow-up data-gathering

Loosely Structured: Open-Ended Research

Highly Structured: Narrowly Focused Research

Fieldwork in Political Science:Trade-offs Among Types

• Long stays:- Pros

• Experience another culture

• In-depth research• Contacts/network• Ideas• More relaxed & fun

- Cons• May be far away• Lack of urgency• “Too much” data

• Short trips: – Pros

• Efficiency• Forces you to take

stock/think analytically

• Identify comparisons/ contrasts quicker

• Less time away – Cons

• Can be more costly• Scheduling

difficulties• Not enough time

Emotional and Psychological Challenges

• Loneliness / isolation• NEVER being alone• Language/cultural

barriers• Family stresses• Financial stresses• NERVOUSNESS!

• Identify your concerns• Network• Balance• Know where to turn• Don’t be stoic!




Design-Driven Fieldwork and

Fieldwork-Driven Design

• Field work must fit your research design

• Your design must accommodate field realities

• Competing Imperatives

– Your ideal research design

– Practicality

Design-Driven Fieldwork and

Fieldwork-Driven Design

• Design “do-able” research– Think about your own limitations

• Find a way– Think positively

• Revise– Think in terms of variables

Fieldwork and the Research Design

• Nomothetic– Thinking in terms of variables

• Idiographic– Thinking in terms of cases

Data Matrix

Variable 1 Variable 2 Variable 3

Case 1

Case 2

Case 3

Case 4

Case Selection and Sampling

• Selection decisions arise at many stages– Macro-level

• Country cases– Meso-level

• Regions or Towns• Time periods• Sectors, etc.

– Micro-level• Individuals for interviews• Documents for content analysis

Small-N

• Use case selection to provide causal leverage

• Use a variable-centered approach– Hold rival explanatory variables constant– Allow your primary explanation to vary

• Example: Dan Posner (2004)– Macro-level: Zambia & Malawi– Meso-level: Town selection held rival variables

constant

Posner (2004) Meso-level Selection

Large-N

• Where possible, use random sampling– This increases the generalizability of your findings– Need a list of the universe of cases– Consider cluster sampling

• Sometimes not possible or desirable– Special cases you want to include– Important variation you want to capture




The Months and Weeks Before…Administration

• Funding– Develop a strategy far in advance– Apply for lots of money

• Dealing with your Institutional Review Board– Does not have to be difficult!– Follow the directions, adhere to deadlines– Find out about exemptions and consent

The Months and Weeks Before…Intellectual Prep

• Dig into your topic

• Background research on your country/ies

• Begin to write documents

• Collect documents you’ll use in the field

• Brush up on methods

• Think about your foreign language skills

The Months and Weeks Before…Reach Out!

• Develop your network of scholars and contact them!

• How about a host institution (research affiliation)?

Converting your Research Design into a Data Collection Plan

Concept/Sub-dim/ Other info

Indicators

WHAT data needed to measure

WHERE (source for data) and HOW to collect

WHEN to get/HOW LONG needed

DV

IV 1

IV 2

Other info.

Example Data Collection PlanConcept/Sub-dim/ Other info

Indicators WHAT data needed to measure

WHERE (source for data) and HOW to collect

WHEN to get/HOW LONG needed

Legal infra-

structure governin

g inf. work

1. # laws regulating inf. sector2. # const. rts. re: IWs

1. Lists of nat’l., state, local laws2. Const. rights

1. Gov’t. ministries/offices/ on-line2. Const.

1. Early (1-2 mos.)

2. Early

Use of legal

system by

informal workers

1. IWs’ court cases2. Protest based on law/const rights

1. Record of cases filed with courts

2. History of protest

1. Courts’ records/ on-line2. News-papers

1. Early (2-3 mos.)

2. Later


Other aspects of res des/data collection

Data / info needed

Source for data/info. and how to collect

When to get/time needed

Information to inform case selectionPermissionsNames of people to interviewCausal process observations

About the Data Collection Plan

• You can head to the field before you have your entire Data Collection Plan filled out!

• Objectives of the Data Collection Plan – Helps make your project manageable– Is a measure of your progress– Is the link between all the millions of tasks

you carry out in the field, and your larger project


Variable

Indicators

Data needed to measure

Where/how to get data

When to get/time needed

DV

IV 1

IV 2

IV 3



Part II – Data Collection(3:25-4:10)1. Forms of Data Collection2. Data Collection: Choices, Challenges,

Assistants * Break (4:10-4:20) Part III – Interviewing (4:20-5:15)Part IV – Organizing, Analyzing, Assessing (5:15-6:00) * Break (6:00-6:10)Part V – Analytic Transparency (6:10-7:00)

Non-Interactive Data Collection

• Observations

• Following the local media– Newspapers, magazines, radio and TV– Books & articles

• Obtaining Documents & Existing Datasets– Government agencies & Ministries– NGOs

• Archival Work

Interactive Data Collection

• Ethnography– Participant Observation

• Experiments– Laboratory experiments– Survey experiments– Natural experiments– Field experiments

• Large-scale surveys• Interviews• Focus Groups

Choices in Data Collection

• Strengths and Weaknesses– Non-interactive

• Documents don’t “react”• Documents and archives can be biased

– Interactive• Respondents can provide first-hand account• Respondents might be inaccurate

• Triangulate!

Triangulation



Part II – Data Collection (3:25-4:10)1. Forms of Data Collection2. Data Collection: Choices, Challenges, and Assistants

* Break (4:10-4:20) Part III – Interviewing (4:20-5:15)Part IV – Organizing, Analyzing, Assessing (5:15-6:00) * Break (6:00-6:10)Part V – Analytic Transparency (6:10-7:00)

Sequencing and Prioritizing Data Collection

• Distinguish necessary from desirable data– Centrality to core hypotheses or key variables

• Factors to consider in ordering data collection– Overlapping strategies– Harder-to-get vs. easier-to-get data– “Low-risk” vs. “high-risk” contacts– Temporally tied events – Depth vs. breadth

Anticipate Data Collection Challenges

• Roadblocks to accessing elites, ordinary citizens, archives, datasets– Affiliate– Network – find a “connection”– Think empathetically– BUT consider investment: time, effort, $

Methods for Capturing Data

• Documents– Take notes– Reproduce

• Interviews & Focus Groups – Field notes after the fact– Jotted notes during– Audio/Video tape– Combo

• How do decide?– How much data do you want to capture?– How are you going to use the data?

Methods for Capturing Data (cont’d.)

• Reproducing/Recording Pros– Taking home much more data– You aren’t just relying on your notes and memory– Capture verbatim quotes– Richer data

• Reproducing/Recording Cons– Taking home much more data– Costly

• Equipment• Storage• Transportation• Transcription

– Recording can make people nervous

Hiring Research Assistants

• Pros:– ‘Giving back’ to countries– Can be a great source of information – Building long-term relationships

• Cons– Training = time consuming– May cause problems/quit/not follow instructns– May introduce bias

• Recruitment– How to find RAs– Interview– What are you offering?– Contract



Part II – Data Collection (3:25-4:10) * Break (4:10-4:20) Part III – Interviewing (4:20-5:15)Part IV – Organizing, Analyzing, Assessing

(5:15-6:00) * Break (6:00-6:10)Part V – Analytic Transparency (6:10-7:00)



Part II – Data Collection (3:25-4:10) * Break (4:10-4:20)

Part III – Interviewing (4:20-5:15)1. Preparing and Writing Questions 2. Conducting the Interview/Follow-up

Part IV – Organizing, Analyzing, Assessing (5:15-6:00) * Break (6:00-6:10)Part V – Analytic Transparency (6:10-7:00)

InterviewingGeneral Guidelines

• Do your homework • Interview “those who study” before “those

who do”

• Varieties of interviews– Structured? Semi-structured? Informal chat?

• Scheduling interviews• Where to conduct the interview

Interviewing Writing Interview Protocols (I)

• What’s the goal of the interview? Will your questions produce useful data?

• Language• Question sequencing• Theoretically motivated/in colloquial terms• What are you asking?! • Asking sensitive questions • Weeding questions

Interviewing Writing Interview Protocols (II)

• Transitional language• Keep questions simple and direct• Get local input on your questions• Pretest• Crucial questions: how are you going to

analyze and use the data?



Part II – Data Collection (3:25-4:10) * Break (4:10-4:20)

Part III – Interviewing (4:20-5:15)1. Preparing and Writing Questions 2. Conducting the Interview/Follow-up

Part IV – Organizing, Analyzing, Assessing (5:15-6:00) * Break (6:00-6:10)Part V – Analytic Transparency (6:10-7:00)

Conducting the Interview

• Introducing yourself and your research• Interacting with your respondent

– Conversation vs. Interview– DOs and DON’Ts

• Know your protocol• Prioritize your questions


• Probes– follow-up questions that are used to deepen a

response to a question• Types

– Basic signals– Detail-oriented Questions

• who, what, where, when, how– Elaboration Probes

• “Tell me more”– Clarification Probes

• “What do you mean?”


• Using Silence– One of the most useful “probes”– Gives the subject the space to talk– Wait longer than is comfortable for you– Builds constructive tension

• Taking notes– A form a body language– What to write

• Wrapping up– Be thankful, be very very thankful!

After the Interview

• Write up your notes!!!– Worth being VERY disciplined about this!– Do it before you conduct another interview– Include a description of the person & location– Note the most important new information– Remember: writing notes is a form of data

reduction• If you record, dealing with your recordings

– Transcribe? “Listen through”? Put aside? • Send a thank you note!• Getting better

Activity: Interviewing

• Write out your 30 second elevator pitch. • Try it out on your neighbors and get their

input.

• Before you start…– What is an elevator pitch?– Questions elevator pitch should answer?– Language



Part II – Data Collection(3:25-4:10) **Break (4:10-4:20) Part III – Interviewing (4:20-5:15)

Part IV – Organizing and Analyzing Data & Assessing Progress (5:15-6:00)1. Organizing and Analyzing Data2. Assessing Progress and Wrapping Up

**Break (6:00-6:10)Part V – Analytic Transparency (6:10-7:00)

Data Organization:Questions to Consider

• You’re Project Manager of your study – organize and systematize from the start!

• Why organize?– Helps you to see the progress you’ve made – Eases re-entry

• What needs to be organized?– data– contacts– thoughts

Organizing Your Data

Sources of Data

Interviews

Archives

Written Primary andSecondary Sources

Datasets

Observations

Misc. Documents

Forms of Data

Notes, Tapes, Transcripts

Notes, Copies

Notes, Copies

Electronic Files, Printed Copies

Notes, Tapes

Notes, Copies

Establish a System for Filing Physical and Electronic Documents

S A M P L E F IL IN G S Y S TE M

C on tac t lis tsB ib liog rap h yL e tte rs O u tL e tte rs In

A d m in is tra t ion

S u rveys (fo rm s)E con om ic d a taO th er c od ed d a tase ts

Q u an t D ata

In te rview tran sc rip tsN otes from a rch ivesS c h o la rly A rt ic lesR ead in g n o tesN ew sp ap er c lip p in g sR ep orts / B roch u res

Q u alit D a ta

D ata

O u tlin eS u m m ariesC h artsG rap h s

A n a lys is

Th es is ch ap te rsA rtic les

O u tp u t

R es earch P ro jec t

Organizing Your Contacts (I)

• Develop a contact management system– What matters most is that it works for you.

• General suggestions– Separate from your general address book. – Lots of fields– Set up to automate thank-you letters – Easy retrieval for follow-up, thank you notes,

future projects – Make it the beginning of a database

Organizing Your Contacts (II)Chart for Potential Respondents

Status Respondent and contact info.

Position/Comments

Suggestion of whom

Progress (e-mails; phone calls, etc.)

X Fernando Prieto 513-5887…Sec: Claudia

Head of Central Bank… was important…

Gustavo Sainz (said talk to him esp. about...)

02Jan2005 – spoke with secretary…

Status:X = need to findX = found, need to get goingX = goingX = set and/or doneX = don’t do anything for nowX = give up/declined

Organizing Your Contacts (III)Interview Clean-up

Intrvw. code

Intervw. done

Names given added tocontactlist

Type up notes / things said off record /thoughts

Send thank you note

Still need/respondent promised/need to follow-up

EC-01 22 sept. 2005

22 Sept. 2005

22 Sept. 2005

22 Sept. 2005

Promised document on…

Digesting Your Data

• Develop a strategy for keeping track of your thoughts– Make a habit of writing– Organize your thoughts

• Begin your analysis while in the field– Periodically read through your documents and notes– Write tentative memos

• Prepare your data for analysis– Think about your coding strategy– Import data into software programs



Part II – Data Collection (3:25-4:10) **Break (4:10-4:20) Part III – Interviewing (4:20-5:15)

Part IV – Organizing and Analyzing Data & Assessing Progress (5:15-6:00)1. Organizing and Analyzing Data2. Assessing Progress and Wrapping Up

**Break (6:00-6:10)Part V – Analytic Transparency (6:10-7:00)

Assessing Your Progress (I)

• Challenge: simultaneously thinking micro (trees) and thinking macro (forest)

• Where are you on the “to get” list?– How much data have you collected?– How many of the people you consider crucial have

you spoken with?• How about in terms of your argument?

– Have evidence to support your hypotheses?– Have evidence to address “alternative

hypotheses”?• Get help to correct for “field goggles” -- to

push larger questions to the forefront

Assessing Your Progress III

• Different vehicles for telling your evolving story – Periodic written reports or memos to advisors – Write conference papers.– Present your work in-country informally

• Upsides– Helps you do some early analysis– Gets you feedback on your analysis and story– Helps you see where you really are!

Making Adjustments

• It’s common for people to (want to) make some adjustments

• Carefully consider any change to your project – Be patient and set some reasonable

deadlines.– Carefully diagnose the problem.– Talk with others about your potential “fix”– Make sure your “fix” addresses the problem.

What seems to be the problem?

Why is this happening?

Possible fixes

Should I change my topic?

I can’t answer the question

Unfalsifiable question;Access issues

Triangulate;Come back later;Do without

Only if multiple, central hypotheses are untestable

I’m surprised by the answer

Your initial guesses were wrong;You’re not getting the full story

Make it a good surprise

Only if nothing of value to be gained from pursuing question in its current framing

I’m testing the wrong hypotheses

Theories were inapplicable;You made a good guess, but guessed wrong

Come up with new hypotheses and ways to test them

Generally not a reason to change topics

I’m asking the wrong question

You’re bored;Your question got dated;It really was never the right question

Remind yourself why you asked this question;Take a historical view; Link to different theories

Radical change usually not advisable

Tweaking Your Project

• Micro-level Matrix -- Individuals

Association # of Grants/yr Sources

Person 1 Org 1 12 Government

Person 2 Org 1 10 Internationa

lPerson 3 Org 1 13 Government

Person 4 Org 2 2 Private



Tweaking Your Project

• Meso-level Matrix

Funding Sources Connectivity

Organization 1 Well-funded Gov’t &

Int’lOrganization 2 Struggling Private

Knowing When to Wrap Things Up

• Develop criteria to help you determine when you have “enough”– Check your “to get” list– What information is only available abroad?

• Consider how much paper to cart home (or between countries)

• Return trips are an option– Research sites won’t disappear– You will be able to hit the ground running

Preparing for “Life After Field Research” While in the Field

• Consider reintegration strategies – Request office space from your home dept– Join a dissertation-writing group– To TA or not to TA?

• Try to plan out your first month back– Unpack your boxes ASAP– Deal with unfinished field business– Data to transcribe, code, enter, or clean up– Reconnect with your advisors

• Plan to go easy on yourself!



Part II – Data Collection (3:25-4:10) * Break (4:10-4:20) Part III – Interviewing (4:20-5:15)Part IV – Organizing, Analyzing, Assessing (5:15-6:00)

* Break (6:00-6:10)Part V – Analytic Transparency (6:10-7:00)