Embed Size (px)
Citation preview
DESIGNING AND CONDUCTING CASE STUDIES
FOR DEVELOPMENT EVALUATIONS
A Preconference Workshop for the IDEAS Global Assembly in
JohannesburgMarch 17, 2009
Linda Morra [email protected] [email protected]© IPDET 2009
2
Workshop Objectives• Understand defining features of
case studies• Know when to use case studies and
when each type of case study is appropriate
• Contrast case study methods• Know how to increase case study
reliability and validity, and determine sample size
• Apply some case study methods
© IPDET 2009
3
BOOKS TO READ
1.Robert E. Stake (1995). The Art of Case Study Research, SAGE Pub., Thousand Oaks: CA.
2. Robert K. Yin (2005). Case Study Research: Design & Methods, Third Ed. Applied Social Research Methods Series, Volume 5. SAGE Pub., Thousand Oaks: CA.
3. Robert E. Stake (2006). Multiple Case Study Analysis, Guilford Press, New York: New York.
www.amazon.com
© IPDET 2009
4
Experience with Case Studies
• Who has conducted case studies?
• How did you know it was a case study?
© IPDET 2009
5
Definition
A case study is a method of learning about a complex instance, based on a comprehensive understanding of that instance obtained through extensive description and analysis of that instance taken as a whole and in its context. © IPDET 2009
6
Discussion• A case study is a method of learning
about a complex instance…..– Simple instance vs. complex instance
• Based on a comprehensive understanding – Fly through qualify?
• Obtained through extensive description and and analysis – Qualitative, not quantitative?
• Taken as a whole and in its context – ???????
© IPDET 2009
7
Case Studies versus Other Approaches
• Histories: no access or control– rely on documents and artifacts (the past).
• Experiments: done when behavior can be manipulated– a social experiment.
• Surveys: large # of cases; thin descriptions; empirical generalizations© IPDET 2009
8
DEFINING THE CASE STUDY
Case studies emphasize thick descriptions
– Not just descriptions of people, places, and events
– But also the interpretations of those people most knowledgeable
– Evaluator applies principle of relativity in identifying the best explanations
© IPDET 2009
© IPDET 2009
9
CASE STUDY PURPOSE “The real business of case study is
particularization, not generalization. We take a particular case and come to know it well…There is emphasis on uniqueness…we emphasize placing an observer in the field to observe the workings of the case, one who records objectively what is happening but simultaneously examines its meaning and redirects observation to refine or substantiate those meanings.”
Robert E. Stake. 1995, pg. 8-9
© IPDET 2009
10
THE APPROACH
• Holistic view of complex instances• Observation• Progressive focusing• Searching for patterns• Developing assertions
© IPDET 2009
11
When to Use A Case Study
• Preferred strategy for “how” or “why” questions.
• Can use for “what” questions, but …. – “Who” or “where” questions
or “how many” or “how much” or “extent” questions favor survey strategies or analysis of archival records.
© IPDET 2009
12
Would You Use Case Study ?
• If you want to know “how frequently” project sites are obtaining monthly target outputs, would you likely use a case study?
• If you want to know more about cases that illustrate a certain pattern (e.g. sites not meeting targets), is case study appropriate?
© IPDET 2009
13
Would You Use Case Study ?
• If you want to know what a best or worst case looks like and what makes it different, would case study be appropriate?
• Would you use case study to answer “How many clients did the program serve?” Or “What kinds of benefits were received most often?”
• If you want to know what a typical case looks like, would you use case study?
© IPDET 2009
14
Use Case Study• If you want to know “why” or
“how” the program worked (or did not).
• Key: exploratory or explanatory purpose; not a frequency or extent purpose.
© IPDET 2009
15
CASE STUDIESMyth or Fact
© IPDET 2009
16
#1 Myth or Fact?
Case studies take a long time.
© IPDET 2009
17
# 2 Myth or Fact?
Case studies only use qualitative methods.
© IPDET 2009
18
# 3 Myth or Fact?
Case studies provide little basis for generalization.
© IPDET 2009
19
# 4 Myth or Fact?
You can use case studies to draw causal inferences.
© IPDET 2009
20
THREE CASE STUDY CATEGORIES
© IPDET 2009
21
Three Case Study Categories
• Descriptive: describe an intervention and the context in which it occurred.
• Explanatory: explain causal links in interventions; link program implementation with program effects.
• Combined : Brings together findings from several case studies to answer an evaluation question.
Overlap in real life!© IPDET 2009
22
3 Types of Descriptive Case Studies
1) Illustrative– Add realism and in-depth examples
to other information about a program, project, or policy.
– Describe what is happening and why, to show what a situation is like.
– Often used to help interpret survey data.
– Generally selected as typical or representative of important variations.
– Number generally kept small.© IPDET 2009
23
Types of Descriptive Case Studies
2) Exploratory To explore those situations where little is known about the intervention or its potential outcomes.
© IPDET 2009
24
Types of Descriptive Case Studies
3) Critical Instance To examine a single
instance of unique interest or serve as a critical test of an assertion about a program, project, problem, or strategy.
© IPDET 2009
25
2 Types of Explanatory Case Studies
1) Program Implementation: The case study investigates operations, often at several sites, and often normatively.
2) Program Effects: The case study examines causality and usually involves multisite and multimethod assessments.
© IPDET 2009
26
1 Type of Combined Case Study
1. Cumulative: Brings together findings from many case studies to answer an evaluation question, whether descriptive, normative, or cause and effect.
THINK SYNTHESIS!
© IPDET 2009
© IPDET 2009
27
EXERCISE
1. Is case study appropriate? 2. If yes, what type?
© IPDET 2009
28
DESIGNING THE CASE STUDY EVALUATION
© IPDET 2009
29
Doing Case Studies
Design
Prepare
Define
Collect
Analyze Conclude
© IPDET 2009
30
Develop design
Select cases
Write data
collectionprotocol
Conduct 1st
Case study
Conduct 2nd
case study
Conduct remaining
case studies
Write casereport
Write casereport
Write casereports
Draw cross-caseconclusions
Modify theory
Develop policyimplications
Write cross-case report
Case Study Method: Adapted from Yin, 2003. Pg. 50.
Developtheory
Identify Evaluation
issue
31
DEFINING THE EVALUATION ISSUE
• Why is this particular program being evaluated at this particular time?
• Who is the main client and what are their major concerns and questions?
• How will they use the evaluation results?
• Who are the major stakeholders and what are their concerns and questions?
© IPDET 2009
32
DEVELOP THEORY OF CHANGE
• Conduct a literature review• Talk with experts• Engage stakeholders in the
process of articulating the program theory
• Develop conceptual model of how program/project/intervention is supposed to work = TOC
• Highlight assumptions and relationships between the key elements
© IPDET 2009
33
Develop Design• Design: the overall strategy for
systematically gathering and analyzing data to address the main evaluation purpose or issue.
• Begins with questions identified through stakeholder analysis and TOC/logic model.
• Is case study indicated? – General approach? – Depends also on the types of questions
posed.© IPDET 2009
34
Questions and Case Study Design
• How does the new village well typically affect the lives of the women in the village?
• What types of outcomes both intended and non-intended are associated with the Rural Roads Expansion Program?
Illustrative Case Study
Exploratory Case Study
© IPDET 2009
35
Questions and Case Study Design
• How critically important is the support group to the success of the women’s Microenterprise Program?
• How are women using the food and nutritional supplements provided by the provincial health clinics?
Critical Instance Case Study
Program Implementation
Case Study
© IPDET 2009
36
Questions and Case Study Design
• Did educational services provided by the agriculture extension center result in planting of the new grain seed and increased yields?
• Does the body of evaluation literature indicate that the building of roads increases trade and benefits villages economically?
Program Effects Case Study
Cumulative Case Study
© IPDET 2009
37
Using Multiple Strategies• More than one strategy can be
used in a study.
• Which comes first, the survey or the case study? [Sam D. Sieber. “The Integration of Fieldwork and Survey Methods”. The American Journal of Sociology, Vol. 78, No. 6, (May, 1973), pp. 1335-1359. http://www.jstor.org/stable/2776390]
© IPDET 2009
38
DEFINING THE SAMPLE
© IPDET 2009
39
MULTIPLE CASES NEEDED WHEN
– Bracketing. What is happening at the extremes? What explains differences?
– Best cases/Worst cases. Why is (isn’t) it working?
– Cluster. How do different types of instances compare with each other?
– Representative. How do different types of instances chosen to represent important variations compare and why?
© IPDET 2009
40
Multiple Case Studies• Same study contains more than a
single case.– For example, Study of school
innovations. Independent innovations occur at different sites.
– Every case should have a specific purpose.
• Often needed for explanatory case studies. Replication - same results predicted for each case or contrasting results for predictable reasons.
© IPDET 2009
41
SINGLE CASE APPROPRIATE
– Exploratory. In this particular circumstance, what is happening and why? Could be an extreme or unique case.
– Typical. In a typical site, what is happening and why?
– Critical instance. Does this single instance support the theory?
© IPDET 2009
42
Embedded Case Studies
• Within a single case (single public program), the analysis includes outcomes from individual projects within the program.
• Embedded units selected through sampling or cluster techniques. (Project characteristics)
• More frequent versus holistic approach.
© IPDET 2009
43
Other Sampling Strategies• Probability Sample: What is
happening as a whole and why?– Not appropriate for case study
selection, but can use sampling for embedded units.
• Convenience Sample: In this site, selected for data collection convenience, what is happening and why?– Not a good sampling strategy
© IPDET 2009
44
Defining the Sampling Unit
• Family/kinship group• Small group
– E.g. What is a community?
• Geographic area– E.g. How define boundaries of a
neighborhood?
• Time period
© IPDET 2009
45
GENERALIZING FROM THE CASE STUDY SAMPLE
© IPDET 2009
46
GENERALIZING FROM THE SINGLE CASE STUDY?
• Naturalistic generalization . Not scientific, but intuitive, based on personal or vicarious direct experience (Stake)
• Transferability and fittingness. The “fit” or similarity between two cases determines the transferability of conclusions from one to the other (Lincoln & Guba)
© IPDET 2009
47
GENERALIZING FROM MULTIPLE CASE STUDIES
Replication• A finding emerging repeatedly
across study of numerous sites yields a good working hypothesis
• The more different the sites, the stronger the working hypothesis
© IPDET 2009
48
Analytic Generalization
Robert Yin: Ruling out rival hypotheses
• If two or more cases are shown to support the same theory, replication may be claimed.
• The empirical results are stronger if two or more cases support the same theory but do not support an equally plausible, rival theory.
© IPDET 2009
49
Illustration• Proposition: Increase in number of days of
school attendance per year for girls in rural areas will occur when both extra income is provided to their families and the girls’ personal safety is assured.
• Select 3-4 sites in which both interventions are present.
• Select 3-4 sites where only extra income is provided to the family, with the prediction being little increase in girls’ school attendance.
• Might select 3-4 sites where only physical safety is assured, with the prediction being little increase in girls’ school attendance. © IPDET 2009
50
DATA COLLECTION METHODS
© IPDET 2009
51
Data Collection Methods
• See handout on data collection options
• Three key approaches – Observation– Interview– Document analysis
© IPDET 2009
52
Data Collection Methods
OBSERVATION– Participant– Non-Participant– Obtrusive– Unobtrusive
Group Review of Observation DCI
© IPDET 2009
53
Form & Exercise
Data Collection Methods: Paired Observation Parts I & II
© IPDET 2009
54
Data Collection Methods
• INTERVIEW– Individual– Small Group– Large Group
© IPDET 2009
55
Data Collection Methods
A GOOD INTERVIEW IS A GOOD CONVERSATION! AND, WHAT MAKES FOR A GOOD CONVERSATION?
© IPDET 2009
56
Data Collection Methods
• DOCUMENT ANALYSIS– Contemporary records– Historical records– Personal information– Public information
© IPDET 2009
57
TRIANGULATION: Checking for Consistency Increases
Credibility• Methods triangulation
– Most common– Using different data collection methods
• Data triangulation– Using different data sources
• Investigator triangulation– Using several evaluators
© IPDET 2009
58
Convergence of Multiple Sources of Evidence
Archival Records
Documents Interviews
FACTFACT
Observations
Physical Artifacts
Focus Groups
FACT
Source: Adapted from Yin, 2003. Pg. 100
59
EXERCISE
Case study Critiques
© IPDET 2009
60
PREPARING FOR THE CASE STUDY
© IPDET 2009
61
Increasing Reliability• Reliability: if the study was repeated
the same results• Increase: use case study protocols• TRAIN & TRAIN!• Help develop case study data bases
© IPDET 2009
62
Selecting Data Collectors
• Field Investigators and Skills Needed– Asking questions– Listening– Flexibility– Understanding of issues being
studied– Objectivity
© IPDET 2009
63
Training
READREAD DISCUSSDISCUSS PRACTICE PRACTICE!! – why the study is being done– theoretical background – basic concepts – terminology– study issues– evidence being sought– case study methods and tactics– data collection and analysis procedures– reporting requirements
© IPDET 2009
65
Case Study Protocol
Contains the procedures and general rules to be followed.
Increases the reliability of the evaluation.
Anticipates problems that might arise and identifies solutions or strategies.
© IPDET 2009
66
Case Study Protocol “Must Haves”
2 Page Overview
1. Study background
2. Study purpose and objectives
3. Study scope and methodology
4. Case study issues, relevant readings
1 Page Letter of Introduction
1. Sponsoring org., official, title and contact detail
2. Study title
3. Data collector name, org., and assignment© IPDET 2009
70
DATA ANALYSIS
© IPDET 2009
71
Data Analysis• Coding schemes• Cross validation• Disconfirming evidence• Themes• Patterns
Group Review of Observation Analysis
© IPDET 2009
72
DATA ANALYSIS
• Holistic fallacy• Using both qualitative and quantitative
evidence• Computer software• Within and between sites
© IPDET 2009
73
Exercise
Data Analysis: Paired Observation III
© IPDET 2009
74
Content Analysis• Ethnograph 5.03 www.qualisresearch.com
– Searches, marks with code words, runs analysis
• atlas.ti 5.2 www.atlasti.com– Searches recorded multimedia plus print
• Nvivo 8 (formerly NUD*IST) www.qsrinternational.com/products_nvivo.aspx– Also multimedia, PDF files, and most languages
• MAXqda 2007 (formerly winMAX) www.maxqda.com
© IPDET 2009
75
Content Analysis Continued
• Lewins, A. & Silver, C. (2007) Using Software in Qualitative Research: A Step-By-Step Guide, SAGE Pub., London
• Comparison of software
© IPDET 2009
76
PRESENTING FINDINGS
© IPDET 2009
77
STRATEGIES FOR PRESENTING FINDINGS
• Natural history• Critical incident• Thematic• Within and between
© IPDET 2009
78
PRESENTATION
FROM WHOSE VANTAGE POINT DOES ONE REPORT?
© IPDET 2009
79
PRESENTATION
• The power of pictures• The power of video, but is it worth
it?
© IPDET 2009
80
SUMMARY The Good Case Study…
• Design follows operational specification of the issue (e.g., a logic model)
• Discovery and flexibility are accompanied by re-design
• Rival explanations are addressed• Evidence (data) and interpretation
are clearly distinguished• Evidence is from multiple sources© IPDET 2009
81
GET REAL LINDA!
OK, SO NOW I KNOW ALL ABOUT DOING GOOD CASE STUDIES. BUT REALITY IS THAT I HAVE ABOUT ONE WEEK MAX TO SPEND AT EACH OF 4 SITES. WHAT CAN I DO?
© IPDET 2009
82
Use your new knowledge to strengthen your methodology but note limitations
© IPDET 2009
83
How Did We Do?
1. Understand defining features of case studies
2. Know when to use case studies and when each type of case study is appropriate
3. Contrast case study methods4. Know how to increase case study
reliability and validity, and determine sample size
5. Apply some case study methods
© IPDET 2009
