Philosophy of the Social Sciences-2012-Ruzzene-99-120.pdf

8/9/2019 Philosophy of the Social Sciences-2012-Ruzzene-99-120.pdf

1/22

Philosophy of the Social Sciences42(1) 99 –120

© The Author(s) 2012Reprints and permission:

sagepub.com/journalsPermissions.nav

DOI: 10.1177/0048393111426683http://pos.sagepub.com

POS 42 1 10.1177/0048393111426683RuzzenePhilosophy of theSocial Sciences

Received 21 September 20111Erasmus University Rotterdam, The Netherlands

Corresponding Author:Attilia Ruzzene, Erasmus University, Faculty of Philosophy, EIPE Office, Room H5-23 P.O. Box1738, 3000 Dr Rotterdam, The NetherlandsEmail: [email protected]

Drawing Lessons fromCase Studies by EnhancingComparability

Attilia Ruzzene 1

AbstractExternal validity is typically regarded as the downside of case study researchby methodologists and social scientists; case studies, however, are often aimedat drawing lessons that are generalizable to new contexts. The gap betweenthe generalizability potential of case studies and the research goals demandscloser scrutiny. I suggest that the conclusion that case study research is weak inexternal validity follows from a set of assumptions that I term the “traditionalview,” which are disputable at best. In this view, external validity is treated as amatter of mere representativeness. I argue that it is best understood insteadas a problem of inference and that the emphasis should be placed on thecomparability of the study rather than on the typicality of the case. By makingcase studies highly comparable, their external validity can be reliably andefficiently assessed and, in this way, their generalizability potential enhanced.

Keywords

external validity, case study research, comparability

In an influential book on the principles and practice of the case study method,John Gerring defines the case study as “the intensive study of a single casewhere the purpose of that study is—at least in part—to shed light on a larger

at Ministry of Higher Education on April 9, 2015pos.sagepub.comDownloaded from

http://pos.sagepub.com/http://pos.sagepub.com/http://pos.sagepub.com/


2/22

100 Philosophy of the Social Sciences 42(1)

class of cases (a population)” (2007, 20). And, in fact, case studies are often performed with the purpose of “drawing lessons” in the form of conclusionsthat apply beyond the single case and explain other outcomes in addition tothe one studied directly. Case studies in fields such as economics, politicalscience, and educational research are also used to suggest hypotheses thathelp inform policy decisions in other contexts. In the former case, issues ofmere generalizability arise: they regard the range of conditions under whichthe conclusions of the case study are expected to hold. In the latter case, theseconcerns about generalizability are further complicated by the need to formu-late guidelines on how to intervene in unstudied contexts.

In the philosophical literature, issues of generalizability are usually dis-cussed under the name of external validity (Campbell and Stanley 1963;Cook and Campbell 1979; Guala 2005, 2010; Steel 2008, 2010). Externalvalidity is more easily understood by way of its opposite. A scientifichypothesis is said to be “internally valid” when it is true of the studied con-text. It is said to be “externally valid” when it is also true of contexts that havenot been studied yet. These concepts were first introduced by Campbell andStanley (1963) in their work on experimental designs. And, indeed, the rel-evance of the distinction is immediately apparent when thinking of labora-

tory sciences. Experiments are usually set up with the ultimate purpose ofteaching lessons about the nonexperimental world. Results that are onlytrue of the studied sample and have no bearing on nonexperimental popula-tions would be of questionable relevance. Knowing the conditions underwhich these results are applicable outside the laboratory becomes thereforea prominent concern. Though a less pressing concern, generalizing alsocounts among the goals of case study researchers. External validity is thusan issue for this methodology as well.

Philosophers, however, so far have hardly worried about the generaliz-ability of case study research. For their part, social scientists that make use ofcase studies proved rather timid in addressing this issue. This tendency is partlychanging nowadays. The interest in case studies and their methodological rid-dles experienced an upsurge in the last decade especially among politicalscientists (Brady and Collier 2004; Gerring 2004, 2007; George and Bennett2005; Mahoney and Goertz 2006b). And the renewed attention on the method

brought to the forefront the problems of generalizability it encounters.External validity is here addressed in terms of a trade-off. These authorsemphasize, in fact, the specificity of case study research by describing itsadvantages and disadvantages with respect to other empirical methods.Furthermore, they usually assume the existence of a trade-off between internaland external validity and, in the latter, find the downside of the case study




3/22

Ruzzene 101

design. Treated as a comparative weakness of case study research, externalvalidity is, however, only shortly discussed and quickly dismissed. 1 This situ-ation generates an interesting tension and calls for attention: there is, in fact,a gap to bridge between purposes and means. Generalizing is set forth by casestudy researchers as a prominent goal, but methodological discussions relatedto it seemingly conclude with a gloomy perspective.

In this article I first examine the reasons that led most scholars to the con-clusion that case study research is a weak methodology with regard to estab-lishing external validity. As we shall see, this conclusion is based on assumptionsthat are, at best, disputable. In the second section, I revisit this line of reasoningthat underwrites what I term “the traditional view” on external validity andoutline some of the objections raised against it. I focus in particular on the rolethat the concept of typicality plays within this view and argue that the centralitygiven to it diverts debate from the real issue of external validity. One unfortu-nate result of this has been to lead debate to the dead end where it stands now. I

propose to refocus the debate on the external validity of case studies by bringingthe concept of comparability to the forefront. This refocusing has two major

beneficial effects. First, my analysis demonstrates why it would be best tosituate external validity as a problem of inference rather than mere represen-

tativeness, as in the traditional view. Second, the approach that I developsuggests strategies for strengthening the generalizability potential of casestudies. The goal, in short, is not so much a refutation of the traditional view’saccount of the pitfalls of case study research with respect to external validity

but a shifting of perspective that reveals unnoticed room for improvement.

The Traditional View on the External

Validity of Case Study ResearchThe use of case studies is common in the social sciences and apparentlyincreasing (Gerring 2007). Interestingly, the case study method starts to beused as an autonomous tool of investigation even in fields that typically rele-gated it to an ancillary position, such as economics (Rodrik 2003; Bates et al.1998). John Gerring (2004, 2007) quite surprisingly notices that, even thoughwidely employed across the sciences, the case study method is still regarded

1Gerring (2007) is an interesting case in point. His book on the principles and practiceof case study research devotes two chapters to investigate techniques to strengthen theinternal validity of case studies; external validity, after being briefly mentioned in theintroductory chapter, is only discussed indirectly in the chapter on case selection.


http://pos.sagepub.com/http://pos.sagepub.com/http://pos.sagepub.com/http://pos.sagepub.com/


4/22


as a weak methodology, and he attributes the low consideration in which it isheld to the general lack of understanding that still surrounds it. Several schol-ars lately tried to rehabilitate this methodology by providing a thorough analysisof its specificity. Brady and Collier (2004), George and Bennett (2005), Gerring(2004, 2007), Mahoney and Goertz (2006b), Ragin (1992, 2000) all contributeto the methodological reflections on the case-based method by emphasizing itsdistinctiveness with respect to the other research designs. These works findsome convergence in their understanding of what case study research is goodfor. Nonetheless, they tend to agree on the fact that external validity counts asa weakness of the method. This conclusion is supported by a set of assump-tions on what external validity is and how it should be evaluated. I term thesethe “traditional view” on external validity.

Some of these beliefs are widely shared in the generic literature on externalvalidity and are thus not confined within the debate among case study research-ers. At the same time, not all scholars above would probably endorse each ofthese assumptions with the same degree of confidence. Even if it is not fullyexpressed by any of these authors, I take George and Bennett (2005), Mahoneyand Goertz (2006b), and Gerring (2004, 2007) as holding firmly to this view.The assumptions on which it rests are, in fact, traceable in the following

excerpts:

Recurrent trade-offs [of the case-study methods] include . . . the relatedtension between achieving high internal validity and good historicalexplanations of particular cases versus making generalizations thatapply to broad populations. The inherent limitations include a relativeinability to render judgments on the frequency or representativeness of

particular cases. (George and Bennett 2005, 22)

Questions of validity are often distinguished according to those thatare internal to the sample under study and those that are external (i.e.,applying to a broader—unstudied—population). The latter may beconceptualized as a problem of representativeness between sampleand population. Cross-case research is always more representative ofthe population of interest than case study research. . . . Case studyresearch suffers problems of representativeness because it includes, bydefinition, only a small number of cases of some more general phe-nomenon. Are the men chosen by Robert Lane typical of white, immi-grant, working-class American males? Is Middletown representativeof other cities in America? These sorts of questions forever haunt casestudy research. This means that case study research is generally




5/22

Ruzzene 103

weaker with respect to external validity than its cross case cousin. Thecorresponding virtue of case study research is its internal validity.(Gerring 2007, 43)

In qualitative research, it is common for investigators to define thescope of their theories narrowly such that inferences are generalizableto only a limited range of cases. Indeed, in some qualitative works, thecases analyzed in the study represent the full scope of the theory. Bycontrast, in quantitative research, scholars usually define their scopemore broadly and seek to make generalizations about large numbers ofcases. Quantitative scholars often view the cases they analyze simplyas a sample of a potentially much larger universe. (Mahoney andGoertz 2006b, 237)

Even though not fully developed and thorough discusses by the authorswho endorse it, the traditional view displays some internal and externalcoherence. Internal coherence among the assumptions enables the conclusionthat external validity is a comparative weakness of case study research.External coherence is granted by the fact that this conclusion sits comfortably

in a theory of case study research that ascribes to the method comparativeadvantages and disadvantages with respect to the other research designs.Specific normative implications are then derived regarding when the casestudy design is the appropriate method to use and how to make it stronger.

I discuss below these assumptions and their normative implications. Someof them have been independently challenged in the extant literature on externalvalidity. I mention them and the related criticisms only shortly. I focus insteadon the assumptions whose normative implications, to the best of my knowl-

edge, have not been challenged yet. The set of beliefs that constitutes thetraditional view is the following:

1. External validity is a property of research designs and of the scien-tific results they deliver.

2. Internal and external validity stand in a trade-off relation.3. External validity is a matter of representativeness.4. External validity is a quantifiable property. Whether it is high or

low depends on the scope (breadth) of the population to which theresults of the study apply.

In virtue of these assumptions, the case study method is characterized ascomparatively weak in external validity. Assumption 1 treats external validity




6/22


as depending on intrinsic features of the research design: a given method isthus characterized as good or bad at providing generalizable results. This is atodds with the original formulation by Cook and Campbell (1979), whereexternal validity is used to qualify solely the result of an experiment. In thisformulation, an experiment is externally valid if its results can be generalizedto a broader population. More generally, recent literature now commonlytreats external validity as a property of a whole design rather than of a par-ticular application of it (Lucas 2003). Assumption 2 asserts a trade-off

between internal and external validity. It follows from assumption 2 that adesign described as having a comparative advantage in the former respect is,in virtue of the trade-off, comparatively weaker in the latter. Assumptions 1and 2 enabled the scholars above to qualify case study research as high ininternal validity and low in external validity. 2 These assumptions rationalizethe methodological prescription that recommends the use of case studyresearch when the main goal is achieving internal validity and other designs,such as the statistical methods, when the goal is deriving broad generaliza-tions instead.

The soundness of this methodological principle that counsels the use ofcase studies is certainly disputable once assumptions 1 and 2 are also dis-

puted. Assumption 1 has been criticized by Lucas (2003) in the context of adebate on the external validity of the experimental design. Lucas mounts adefense of the experimental method. Although it is dismissed by severalscholars as poor in external validity, Lucas rejects the criticism as essen-tially misdirected. Specifically, he responds that “critiques of investigativetechniques as being low in external validity because findings cannot be gen-eralized quite often should be directed at the theory under test, rather than atthe methodology employed to test it” (238). Assumption 2 has been addressed

by Jimenez-Buedo and Miller (2010). They notice a tension between the belief widely held that internal and external validity stand in a trade-off rela-tion and, at the same time, that the former is a prerequisite for the latter. Yetthey conclude upon analysis of the experimental practice that the allegedtrade-off relation is far less cogent than the traditional view would have one

believe. These criticisms suggest that the methodological norm based onassumptions 1 and 2 does not have the self-evident status that the traditionalview presumes.

Let us now turn to assumptions 3 and 4. They can be rephrased asfollows:

2They are not, by themselves, sufficient for this conclusion.




7/22

Ruzzene 105

3.1. A case study is externally valid if the case it studies is representa-tive of a broader population.

4.1. The broader the target population is the higher is the external valid-ity of the research design.

Assumption 3 sets a condition for the generalizability of scientific results.Results that are obtained within a study apply outside of it only if the contextstudied represents the target context in some sense to be specified. The tra-ditional view borrows its idea of representativeness from the statistical dis-course. The external validity inference is here conceived as an inference fromsample to population legitimized by the former being a statistical representa-tive of the latter.

Translated into a qualitative framework, a case is said to stand in a sample-to-population relation with the target universe of cases when it is a typical case within that universe. Typicality is therefore understood as the keyrequirement for ensuring external validity to the case study within the tra-ditional view. In this perspective, methodological precepts oriented tostrengthen the external validity of case studies would all go in the directionof giving rules for the selection of the cases. The following excerpt is an

example:

The typical case study focuses on a case that exemplifies a stable,cross-case relationship. By construction, the typical case may also beconsidered a representative case, according to the terms of whatevercross-case model is employed. . . . One may identify a typical casefrom a larger population of potential cases by looking for the smallest

possible residuals . . . for all cases in a multivariate analysis. In a large

sample, there will often be many cases with almost identical near-zeroresiduals. . . . Thus researchers may randomly select from the set ofcases with very high typicality. (Seawright and Gerring 2008, 299) 3

Reduced to a matter of representativeness, the problem of external validitythus amounts to adopting the selection procedure that maximizes the probabilityof choosing the case most typical of the target of interest.

In the traditional view, two major problems threaten the external validityof case study research. The first lies on the difficulty of reliably establishingthe typicality of the case selected, the solution to which consists in further

3See also Mahoney and Goertz (2006b).




8/22


refining the selection procedure of the case to study. The second is the intrin-sic limitation to the degree of external validity case study research can reach.According to assumptions 4 and 4.1, the degree of external validity depends onthe breadth of the population to which the results are generalizable. In virtue ofassumptions 3 and 4, case study research is low in external validity becauseits capacity for being representative of a broad universe of cases is very lim-ited indeed. Even if one succeeds in identifying typical cases, so the argumentgoes, their typicality is always confined to a small population. Case studyresearch, in fact, studies intensively either one case or a very small set whosedegree of representativeness is not only hard to establish but also very lim-ited. Representativeness, it is said, increases with the size of the sample andso in turn does external validity.

External Validity in Case Study Research:From Typicality to ComparabilityThe traditional view treats the problem of external validity as a problem ofrepresentativeness. This has two major normative implications. First, itsmethodological precepts are all and only oriented to guide the selection of

the “right” case, understood as a typical one. Second, the traditional viewascribes the difficulty that case study research has in putting together a rep-resentative sample as the source of the method’s incapacity, or the extremeweakness, in achieving external validity. But this reasoning goes wrongalready at the first step, and this makes problematic its gloomy conclusion.External validity is not , I argue, essentially a problem of representativeness butrather one of inference and so a problem to which the representativeness of thecase might offer (one) possible solution. The challenge of external validity

actually consists in fact in identifying correctly the circumstances under whichthe results of a study can be generalized to other cases. The inference from thestudied case to some new contexts needs to be thus justified by some factorsthat give us reason to believe that what was found true of the former is most

probably true of the latter as well. Typicality might be one of these factors.The information that the case at hand is typical, in fact, backs up the infer-ence through which we conclude that what is true of the case is also true ofthe target. Finding the typical case is therefore a pragmatic solution to whattruly is an epistemic problem. Typicality is a solution to the problem ofgeneralizability; typicality per se is not the ultimate problem to solve. The tradi-tional view conflates these two concepts—the typicality of a case and its gener-alizability and, in so doing, not only fails to capture the essential distinction butalso confuses one solution with the entire problem. As a consequence, the




9/22

Ruzzene 107

methodological norms it imparts to guide the selection of the case cannotrespond to the epistemic challenge of external validity

The traditional view, in fact, confines the methodological discourse onexternal validity to the stage of the selection of the cases and in so doing implic-itly suggests that the problem of external validity is fully solved by singling outthe representative case from the target universe. Representativeness, however,only offers a solution if the strategy used to establish it properly responds to theepistemic challenges posed by external validity. That is, the typical case cannot

be identified by presupposing knowledge that its identification is expected todeliver in the first place. The problem with the strategy described by Gerring inthe excerpt above is exactly this one. The way he suggests for the selection ofthe typical case presupposes a knowledge of the cases that we are not supposedto possess when the problem at hand is correctly described as one of externalvalidity. If we already know the causal relationship that we are interested ingeneralizing beforehand, there is nothing left to generalize in the first place.Gerring’s strategy probably solves successfully issues of representativeness

but cannot double as a solution to an inferential problem.The scholars from the traditional view failed to respond properly to this chal-

lenge because they failed to distinguish conditions for the external validity of the

results and epistemic criteria that help establish whether these conditions hold.The conditions for external validity are the circumstances that justify the gener-alization; typicality is the one explicitly acknowledged by these scholars: 4

CEV: 5 If the case is typical of a broader universe of cases, the resultobtained in the former is generalizable to the latter.

Typicality, however, cannot double as an epistemic criterion for the assess-

ment of external validity. Once the conditions for the generalizability of theresults have been defined, independent strategies should be devised that helpestablish whether those conditions hold. These are epistemic criteria thatinform us about the representativeness of the case and, at the same time, donot presuppose the knowledge of the target that we are expected to extractfrom the case study itself. This criterion is comparability.

4Typicality might be taken as a restrictive condition. In the literature on externalvalidity, several scholars endorse similarity instead, which is a broader concept. Idiscuss the relation between typicality and similarity below.5CEV stands for condition for external validity .




10/22


EC:6 Comparability of the study is required to establish whether the case istypical of the target universe of cases and the result hence generalizable.

If the case study is comparable in the appropriate respects to the target, itenables us to elicit from the case both the information that is to be general-ized and the information that is required to decide about the generalizabilityof the same results.

The notion of comparability has been introduced by LeCompte and Goetz(1982) in a work on the validity of the ethnographic methods. LeCompte andGoetz understand external validity in terms of typicality and comparability:

The fieldworker’s problem is to demonstrate what Wolcott conceptual-izes as the typicality of a phenomenon, or the extent to which it com-

pares and contrasts along relevant dimensions with other phenomena.Consequently, external validity depends on the identification anddescription of those characteristics of phenomena salient for compari-son with other, similar types. Once the typicality of a phenomenon isestablished, bases for comparison may be assumed. (51)

LeCompte and Goetz have the merit of hinting to the epistemic problemat the core of external validity but still fail to disentangle fully conditions forvalidity and criteria of assessment. Typicality refers to the condition that thecase has to satisfy for having results from the study that are generalizable.Typicality, however, cannot be established a priori, nor it can be inferredfrom knowledge of the target universe of cases that the generalization itselfis expected to provide. The typicality of the case and the generalizability ofthe results are established upon comparison with other/new cases. As already

widely discussed in the philosophical literature, external validity is truly anempirical hypothesis and has to be settled on a case-by-case basis (Guala2005, 2010; Steel 2008, 2010). Comparability is therefore the epistemicrequirement to be imposed on the design of the study in such a way that, bycontrasting its results with what we observe in other situations, we are capa-

ble to adjudicate the typicality of the case at hand and the generalizability ofits results.

By rendering the case study comparable in the appropriate respects, the problem of its external validity becomes ultimately solvable—that is, decid-able. This does not mean that external validity is in this way granted, only

6EC stands for epistemic criterion for assessing the external validity of the study.




11/22

Ruzzene 109

that it can be reliably established. The discourse on external validity so fardeveloped among case study researchers is misdirected and therefore nothelpful to this end. Required in addition are strategies that are epistemicallyviable for assessing the typicality of the case and the generalizability of thefindings. Disentangling the two issues by distinguishing neatly between typi-cality and comparability is a first step in this direction.

A second move that needs to be introduced involves making the notion ofcomparability more precise. I return to this point below. Before turning to thisaspect, however, I want to emphasize a final point in relation to the traditionalview. Its focus on representativeness as if it was the ultimate challenge toestablishing external validity has biased the debate and led its current deadend. That is, the consensus has it that external validity constitutes an irreme-diable weakness of case study research. And this is attributed to the fact that itstudies a very limited number of cases and is in this way subject to an inherentlimit on the achievable degree of external validity. This conclusion, however,links high external validity to a capacity for offering broad generalizations.This assumption about the requisite breadth of the generalization, however, isdisputed in the literature.

There are scholars who hold the view that the generalizations that science

allows are always very limited in scope. In a discussion on the external valid-ity of experiments, Francesco Guala (2002) mentions Bruno Latour, DavidGooding, and Andy Pickering as promoting a form of radical localism and IanHacking and Nancy Cartwright as defending milder positions in a similarspirit. In its extreme version, radical localism denies any external validity toscientific hypotheses except when the outside world can be carefully engi-neered and made alike to the laboratory such that the experimental results can

be directly exported (1196). Guala defends a less skeptical position that admits

of several ways to solve the problem of external validity. According to Guala,7

the problem amounts to minimize the error in the inference from the experi-ment to the outside world; this is achieved by making the two contexts as simi-lar as possible. One way to this end is what he calls “engineering the world.”Another strategy is adapting the experimental setting to the outside conditions.For instance, the former can be modified as to reproduce more accurately non-experimental settings. Even though various strategies exist to generalize reli-ably from experiments to the outer world, external validity is bound to remaina “local” matter. That is, the generalizations that science allows never traveltoo far and never apply too broadly. In this perspective, case studies pose nospecial problem; all studies possess only limited generalizability.

7Guala refers to Debra Mayo as a defender of this view.




12/22


Moreover, from the perspective of this alternative approach to externalvalidity, similarity between the two contexts is the condition that grants gen-eralizability to the results:

CEV1: If the case is similar to the target case/cases, the results obtained

in the former are generalizable to the latter.

Similarity is a broader concept than typicality. Typicality presupposessimilarity between the case and its target but further requires a sample-to-

population kind of relation between the two. This idea, originating as it doesin a statistical context, badly fits case study research where random samplingis not a feasible strategy. Furthermore, it is restrictive in that it asks that therelevant population be clearly defined before engaging in the study of thecase that is supposedly representative of it. This led to the type of discourseon external validity I discussed above. Similarity instead does not require anya priori definition of the relevant target and leaves the issue of generalizabil-ity open to the empirical analysis that would follow the case study research.Unlike experiments, in case study research, the studied context (the case) andits target cannot be “made” similar as an experiment and the nonexperimentalsetting are. The case, in fact, cannot be adapted in any meaningful sense to

the target, whereas, as Guala suggests, the experimental settings can beslightly modified to fit some features of the outside world. And, in general,given the complexity of the phenomena that case studies examine, the idea ofengineering the outside world as to reproduce the study conditions is simplynot practicable. The way that is open to case study research is finding similar-ity between the studied system and the target in vivo.

Improving the External Validity of Case StudyResearch: Enhancing ComparabilityOn the basis of what is said above, we can thus relax EC as to encompass the

broader condition of similarity:

EC1: Comparability of the study is required to establish whether the

case is similar to the target case/cases and the result hence gener-alizable.

Even though intuitively appealing, comparability as described by EC1 is

too vague to be compelling. We thus need to refine the criterion further todistinguish what qualifies as a comparable case study and what does not.Furthermore, one needs to specify what makes a case study highly compara-

ble and what detracts from it. In this way we would hint to some principles




13/22

Ruzzene 111

that might help strengthen the external validity of case study research by mak-ing its assessment more reliable. To evaluate comparability, it is worthkeeping in mind that when assessing the generalizability of a result, onefaces severe epistemic constraints. The external validity hypothesis is, infact, empirically settled by the comparison between the case studied andthe target case, of which we know very little. If we knew of the target whatwe already know of the case, there would be no worries for external valid-ity in the first place. Certainly we do not know whether the result or hypoth-esis that is true of the case is also true of the target, since this is what oneaims to establish. But, in general, any inference of external validity is

bounded by the limited knowledge of the target case (Steel 2008). Hence, provided that it is correct, the inferential strategy that is epistemicallycheaper is the one to be preferred.

With this proviso in mind, I suggest that comparability is of the right kindif it is effective. Effectiveness requires that the study render available theinformation necessary to establish whether, upon comparison, the case is suf-ficiently similar to the target context so as to justify the generalization of theresults obtained. Take, for instance, the most common case in which the resultto be generalized is a causal relationship. The case and the target have to be

similar in the respects that are causally relevant to the hypothesis for this to be valid in the latter as well (Guala 2010; Steel 2010). The study then needsto inform us about the respects that matter to the causal relation in the casesuch that we can proceed to the comparison with the target context and even-tually to the inference.

In case study research where engineering the world is not an option noris adapting the case to the target, 8 knowledge of the relevant causal factorsis what enables the inference. Without it, even a tentative assessment of

external validity would not be possible. Complete knowledge of the rele-vant causal factors is, however, only an epistemic ideal. Ideal aside, one cansay that the more complete this knowledge is, the more reliable the infer-ence will be.

Provided that comparability is effective in the sense described above, wecan then distinguish between high and low comparability depending onwhether the inference from the study is more or less epistemically efficient.The more information about target case required to establish the externalvalidity of the hypothesis, the lower the comparability of the case study will

8Guala says that in experimental economics, this causal knowledge can be sometimes black-boxed without implications for the external validity inference (2010, 1080).




14/22


be. If the case study only describes the causal factors that are relevant to thecausal relationship, then the comparison between the studied case and thetarget needs to be fully articulated—that is, contrasted along all relevantrespects before the hypothesis can be generalized to the new case. Since epis-temic efficiency is a virtue when external validity is at stake, a case study thatrequires full comparison is low in comparability. If the study describes thecausal structure instead, then comparison can be partial and only regard certainelements in it. A causal structure, or causal mechanism, is the set of intercon-nected, regularly operating causal relationships that generate one or moreregularities between (observable and unobservable) events (Guala 2010,1072). It requires that the causal relations among the relevant factors be fullyspelled out. If the study also describes the causal structure, its comparabilityis high indeed because limited comparison between the case and the target ina subset of the causal factors is sufficient to draw conclusions about the

behavior of the whole mechanism.The examples that follow illustrate the distinction just sketched between a

“full” causal relationship, on the one hand, and a “causal structure” or “causalmechanism,” on the other. The first case study, on Botswana, is a case of highcomparability and enables in fact partial comparison between the causal

structures of the case and those of the targets. The second case study is anexample of low comparability, where comparison between the cases needsinstead to be full before confirming any hypotheses of external validity.

* * *In a case study on Botswana, Acemoglu, Johnson, and Robinson (2003)

explain the unusually good economic performance observed in the country inthe last decades. If compared with the average in sub-Saharan Africa, in fact,Botswana performed outstandingly in terms of per capita income growth rate

in the last 35 years. The authors start with the assumption based on previousstudies that proximate determinants of its economic success are the institu-tions and the related policies that the country developed over time. Institutionsare conducive to growth when they correspond to a social organization thatensures effective property rights to a broad cross section of the society. Theauthors refer to this cluster as property right institutions .

What they aim to explain is, however, why Botswana was able to developthe institutions it now possesses and thus search for what can be defined asthe deep determinants of growth (Rodrik 2003, 3). To this end, they adopt acase-oriented methodology. They finally offer a country narrative in whichthey reconstruct the processes through which Botswana developed the insti-tutions it actually has.




15/22

Ruzzene 113

The examination of the country history suggests five (structural) featuresas plausibly responsible for its property right institutions and good economic

policies:1. Botswana’s is very rich in natural-resource wealth.2. It had unusual precolonial political institutions that enabled an

unusual degree of participation in the political process and placedrestrictions on the political power of elites ( Kgotla ).9

3. British colonial rule in Botswana was limited. This allowed the pre-colonial institutions to survive to the independence era.

4. Exploiting the comparative advantage of the nation after 1966directly increased the incomes of the members of the elite.

5. The political leadership of BDP, 10 particularly that of SeretseKhama, 11 inherited the legitimacy of these institutions, which gaveit a broad political base.

The causal influence of each of these features on Botswana modern insti-tutions is justified by describing the mechanisms through which this influ-ence is conveyed. These mechanisms, theorized in the background literatureon development economics to which the authors refer, are used to explain

why these factors are causally relevant to the emergence of property rightsinstitutions and how. Consider, for instance, the mechanism of political los-ers (Acemoglu and Robinson 2000):

An institutional setup encouraging investment and adoption of newtechnologies may be blocked by elites when they fear that this processof growth and social change will make it more likely that they will bereplaced by other interests—that they will be political losers. Similarly,

a stable political system where the elites are not threatened is lesslikely to encourage inefficient methods of redistribution as a way ofmaintaining power. (Acemoglu, Johnson, and Robinson 2003, 103)

According to the mechanism of political losers, elites do not oppose theadoption of institutions and policies favorable to growth if they feel their

9 Kgotla is an assembly of adult males in which issues of public interest are discussed(Acemoglu, Johnson, and Robinson 2003, 93).10BDP is the dominant party in the country and stands for Botswana Democratic Party.11Seretse Khama was BDP political leader and president of Botswana from 1965 until1980.




16/22


power not being threatened by the change. In Botswana (feature 2) precolo-nial institutions ensured some degree of political stability and went almostunaffected by the imposition of British colonial rule (feature 3). In addition,the legitimacy of Seretse Khama and the broad coalition he formed furtherstrengthened it (feature 5). The authors conclude that features 2, 3, and 5influenced the building of property right institutions by setting the mecha-nism of political losers at work and thus ensuring a high degree of politicalsecurity to the existing elites. A similar use is made of the other two theoreti-cal mechanisms: they trace the causal relations among the factors that jointlydetermine the emergence of property rights institutions and the ensuing eco-nomic growth.

In the final section of the work, Botswana is compared with four Africancountries—namely, Somalia, Lesotho, Cote d’Ivoire, and Ghana. The com-

parison is, however, only partial. It has the purpose of checking the hypothesisthat the difference in one factor is sufficient to disrupt one of the mechanismsconducive to property right institutions and therefore to growth. Each coun-try is thus compared to Botswana with respect to one or two of the featuresabove and never along all five dimensions. For instance, consider the thirdmechanism that Acemoglu, Johnson, and Robinson (2003) describe, which

they call constraints .

When (precolonial) institutions limit the powers of rulers and the rangeof distortionary policies that they can pursue, good policies are morelikely to arise (see Acemoglu and Robinson 1999). Constraints on

political elites are also useful through two indirect channels. First, theyreduce the political stakes and contribute to political stability (mecha-nism of political losers ), since, with such constraints in place, it

becomes less attractive to fight to take control of the state apparatus.Second, these constraints also imply that other groups have less reasonto fear expropriation by the elites and are more willing to delegate

power to the state. (104)

The mechanism of constraints is set into operation by factor 2—that is, by the type of precolonial institutions. If these institutions have the right properties, as kgotla had in Botswana, then they are effective in placingconstraints on rulers. These, in turn, affect the emergence of property rightinstitutions both directly and indirectly. At the same time, factor 3 can havethe opposite effect of inhibiting the mechanism of constraints. In fact, strongBritish colonial rule alters precolonial institutions and disrupts the mecha-nism of constraints if it was in place before.




17/22

Ruzzene 115

Acemoglu, Johnson, and Robinson thus compare Botswana and Somaliawith respect to factors 2 and 3 and find them similar in factor 3 but differentin factor 2. Similarly to Botswana, British government had in fact onlymarginal interest in Somalia and imposed very soft colonial rule. Somaliahad, however, precolonial institutions that induced intense factional conflictand were therefore incapable of placing constraints on political elites. Fromthis partial difference in the type of precolonial institutions between the twocountries (factor 2), Acemoglu, Johnson, and Robinson infer that the mecha-nism of constraints was not operating properly in the country and it thusimpeded also the proper working of the mechanism of political losers. Asa consequence, property right institutions did not emerge and in turn itseconomic performance faltered.

A similar line of reasoning is then applied to the comparison with the othercountries. The comparison is limited to one or two structural features whosedifference is taken as evidence that the corresponding mechanism is not oper-ating properly in the context of comparison. These observations license theinference that this explains the absence of appropriate institutions and, so, the

bad economic performance.It seems that Acemoglu, Johnson, and Robinson reason in a way akin to

what Daniel Steel (2008) calls comparative process tracing. Comparative process tracing consists in comparing the mechanisms in the study and thetarget at the “critical junctures.” From this limited comparison, one canestablish the presence (or absence) of the mechanism in the target and, thus,the presence (or absence) of the causal relationship that ensues. In this casestudy, Botswana is compared to each target case in only a limited number offactors involved in the working of the mechanisms that are conducive togrowth through the emergence of property right institutions. Partial com-

parison is possible because the causal structure (or causal mechanisms)through which the relevant factors convey their causal influence is specified.If the causal relations among the factors are described, the difference in asubset of them is sufficient to formulate conclusions about the behavior ofthe causal mechanism and the causal factors connected to it. This studytherefore can be said to attain a high degree of comparability because a lim-ited amount of information of the target case is sufficient for formulating anassessment of its external validity. In particular, it shows higher comparabilitythan the one I illustrate below. Nevertheless, even in this case, there is stillroom for improvement. Mechanisms are, in fact, sketched rather than fullyspecified. Epistemic efficiency increases when the causal structure is more

precisely displayed.* * *




18/22


In the second case, the hypothesis whose generalizability is at stake is the policy hypothesis about the effectiveness of community-based programs todefeat malnutrition. It draws on a report by the World Bank in the early 1990son successful nutritional programs in Africa (Kennedy 1991). It aims to identifythe factors that make programs against malnutrition work. To this end, itcombines the use of two qualitative methodologies—namely, large samplesurvey and case-oriented studies. Eileen Kennedy (1991) motivates the study

by appealing to the fact that the literature on malnutrition in the 1970s and1980s focused on only the types of interventions implemented. The 1989meeting of the International Nutritional Planners in Seoul, however, con-cluded that “how” a program is implemented is as important as, or maybemore important than, the type of intervention for successful programming(1). In line with the recommendations of nutritional planners in Seoul,Kennedy thus uses survey and case studies jointly to identify what factorsmatter for the effective implementation of programs against malnutrition.The ultimate goal is to learn lessons that can be generalized to other Africancontexts (2). The evidence is combined in the following way.

The survey offers prima facie evidence of what factors are required forsuccessful implementation. It combines the findings from 110 answers

received from policy makers and program implementers. The respondentsanswered two types of questions: whether the program was successful andwhat factors they thought were key to its success. 12 Furthermore, 6 programsamong the 66 found to be successful were selected for an in-depth analysis:the Macina Child Health Project, in Segou Region, Mali; the Infant FeedingProject, in Togo; the Imo State Child Survival Project, in Nigeria; the Applied

Nutrition Program, in Ghana; the Mali Institutional Development Enterpriseand Nutrition Program, in Mali; and the Nutrition Project, in Kinshasa, (for-

merly) Zaire. These projects were selected out of the 66 because they repre-sent different types of community-based programs that can succeed incombating malnutrition. The survey and additional interviews singled outseven factors as important for successful implementation: community participa-tion, program flexibility, institutional structure, recurrent cost recovery, multi-faceted program activities, training and staff qualifications, and infrastructure(Kennedy 1991, 7). The same factors were found to be present in almost all 6cases. The case studies then focus on how the seven factors were actuallyimplemented in the specific context.

12In sum, 110 individuals/institutions responded to the mail survey, out of the 330 ini-tially contacted. In addition to this first pool, 78 individuals involved in various waysin the implementation of programs against malnutrition were interviewed.




19/22

Ruzzene 117

The main conclusion drawn from the report is that different types of pro-grams can succeed in defeating malnutrition. Whether they actually succeed,however, depends on a set of conditions regarding how the programs are infact implemented. Even though the study cannot be considered in itself suf-ficient to definitely establish which of the factors identified are ultimatelynecessary to the effectiveness of the program (to this end, a quantitative anal-ysis is further required), it offers prima facie evidence for it. It thus givessupport to the idea that there is no such a thing as a one-size-fit-all intervention,

but, rather, different ways can be pursued. The strength of this study derivesfrom the effort it makes to single out the causal factors that matter to the suc-cessful implementation of the programs against malnutrition. Whether theones identified are all and only the factors that are jointly necessary for thecausal relationship to ensue is a problem of internal validity. Nevertheless,we can consider the study effectively comparable in the sense describedabove. By listing the set of causal respects that matter to the causal hypothe-sis, it enables the comparison with new cases along the relevant dimensions.This allows, then, for the formulation of conclusions about its external valid-ity. Whether these conclusions are reliable depends in the end on the confi-dence we have in the causal inference drawn within the studied case.

These case studies, however, fail to develop a narrative that explains whythe factors identified matter to the success of the program and how. Theircontribution is, in fact, limited to a more or less exhaustive list in which therelevant factors are described in detail, and so are the modifications in thespecifics of implementation that occurred over time. An analysis of thecausal processes at work is however lacking, despite the fact that this was setas a research goal at the very beginning of the paper (Kennedy 1991).Differently from the case discussed above, the causal relationships among

the factors of implementation are not specified, and the underlying causalstructure is thus not even hinted at. As a consequence, even though we canconsider these studies effectively comparable to new target contexts, theyhave low epistemic efficiency. The comparison needs in fact to be full. Thiswould provide an identification of the work done by each factor in all rele-vant causal respects in the case before any attempt to evaluate the generaliz-ability of the hypothesis of interest.

ConclusionThe debate on the external validity of case study research is stunted and, sofar, developed under the influence of the statistical perspective. The result-ing approach was biased in an unfruitful direction that ultimately led thedebate to the dead end, where it seems to stand now. Reaching external




20/22


validity in case study research was, in fact, essentially regarded as a hope-less endeavor. This conclusion stands in a stark contrast with the strugglefor generalizations in which case study researchers engage at the same time.One way to solve this tension is by making external validity a decidableissue. This can be done once we abandon the old paradigm, the traditionalview, and its focus on representativeness as the problem of external validity.External validity becomes decidable only if the study first renders availablethe evidence that is necessary to circumvent the epistemic impasse in whichany inference of this kind finds itself. Making case studies stronger in exter-nal validity therefore means strengthening the design in such a way that ithelps researchers reach judgments of external validity with a higher confi-dence and epistemic efficiency. One way to this end is by enhancing itscomparability.

External validity should be right on the agenda of philosophers and meth-odologists who worry about making case study research a design betterunderstood and better used. Generalizing is not only a valuable goal in itselffor scientific practice but also the middle step on the way to sound policymaking. Drawing the right lessons from the contexts we know already is partand parcel of planning effective interventions in contexts with which we are

not acquainted yet. One way to learn how to use the knowledge obtainedfrom the former in the latter is by discussing issues of external validity. Theinferential problem that it underpins is in fact the first the scholar encounterswhen transferring the knowledge gained from epistemically privileged sys-tems to less privileged ones. Experimentalists were the first to worry about it,and they still reflect upon it extensively. I think case study researchers shoulddo the same.

Acknowledgment

I want to thank Paul Roth and Stephen Turner for their valuable comments on earlierversions of this article and the audiences of the Philosophy of Social ScienceRoundtable in Paris (March 2011), the Erasmus Institute for Philosophy andEconomics PhD seminar in Rotterdam (May 2011), and the conference of theInternational Network for Economic Methodology in Helsinki (September 2011). Iam also grateful to Lara Kutschenko, René Lazcano, and Luis Mireles Flores formuch helpful discussion. Remaining errors and omissions are mine.

Declaration of Conflicting InterestsThe author(s) declared no potential conflicts of interest with respect to the research,authorship, and/or publication of this article.




21/22

Ruzzene 119

Funding

The author(s) received no financial support for the research, authorship, and/or pub-lication of this article.

References

Acemoglu, D., S. Johnson, and J. A. Robinson. 2003. An African success story:Botswana. In In search of prosperity. Analytic narratives on economic growth ,edited by D. Rodrik, 80-119. Princeton, NJ: Princeton University Press.

Acemoglu, D., and J. A. Robinson. 1999. The political economy of institutions anddevelopment. Background paper for World Development Report 2001. Washington,DC: World Bank.

———. 2000. Political losers as a barrier to economic development. American Economic Review 90:126-30.

Bates, Robert H., Avner Greif, Margaret Levi, and Jean-Laurent. 1998. Analytic nar-ratives . Princeton, NJ: Princeton University Press.

Brady, H. E., and D. Collier. 2004. Rethinking social inquiry: Diverse tools, shared standards . Lanham, MD: Rowman & Littlefield.

Campbell, D. T., and J. C. Stanley. 1963. Experimental and quasi-experimental

designs for research . Chicago: Rand McNally.Cook, T. D., and D. T. Campbell, eds. 1979. Quasiexperimentation: Design and analysis

issues for field settings . Chicago: Rand McNally.George, A. L., and A. Bennett. 2005. Case studies and theory development in the

social sciences . Cambridge, MA: MIT Press.Gerring, J. 2004. What is a case study and what is it good for? American Political

Science Review 98:341-54. ———. 2007. Case study research: Principles and practices . Cambridge, UK: Cambridge

University Press.Guala, F. 2002. Experimental localism and external validity. Philosophy of Science

70:1195-205. ———. 2005. The methodology of experimental economics . Cambridge, UK:

Cambridge University Press. ———. 2010. Extrapolation, analogy, and comparative process tracing. Philosophy

of Science 77:1070-82.Jimenez-Buedo, M., and L. M. Miller. 2010. Why a trade-off? the relationship

between the external and internal validity of experiments. THEORIA 25:301-21.Kennedy, E. T. 1991. Successful nutrition programs in Africa: What makes them

work? Washington, DC: World Bank.LeCompte, M. D., and J. P. Goetz. 1982. Problems of reliability and validity in ethno-

graphic research. Review of Educational Research 52:31-60.




22/22


Lucas, J. W. 2003. Theory-testing, generalization, and the problem of external valid-ity. Sociological Theory 21:236-53.

Mahoney, J., and G. Goertz. 2006a. Scope in case study research. Unplublished work-ing paper.

———. 2006b. A tale of two cultures: Contrasting quantitative and qualitativeresearch. Political Analysis 14:227-49.

Ragin, C. 1992. What is a case? Exploring the foundations of social inquiry . Cam- bridge, UK: Cambridge University Press.

———. 2000. Fuzzy-set social science . Chicago: University of Chicago Press.Rodrik, D. 2003. In search of prosperity: Analytical narratives on economic growth .

Princeton, NJ: Princeton University Press.Seawright J., and J. Gerring. 2008. Case selection techniques in case study research.

Political Research Quarterly 61:294-308. ———. 2008. Across the boundaries: Extrapolation in biology and social science .

New York: Oxford University Press. ———. 2010. A new approach to argument by analogy: Extrapolation and chain

graphs. Philosophy of Science 77:1058-69.

Bio

Attilia Ruzzene is a graduate student at the Erasmus Institute for Philosophy andEconomics, Rotterdam, Netherlands. Her dissertation centers on methodologicalquestions related to the use of the case study method: causal inference, internal andexternal validity, and the use of case studies as evidence for policy making.

Documents

Philosophy of the Social Sciences-2012-Ruzzene-99-120.pdf