1

Historical-Comparative Methods

CSLS/Boalt Miniseries on Empirical Research Methods, Workshop 2, Oct. 23, 2007 [See accompanying reference list]

© 2007 Robin Stryker, Prof. of Sociology,Affiliated Prof. of Law, Scholar of College, 2004-07, Univ. of Minnesota

2

Introduction:

Some continuity of themes & techniques from empirical methods for field research to historical-comparative methods

Especially with respect to case selection, research design

Field research sometimes combined with archival data collection & analysis, and both often combined with in-depth semi-structured interviewing techniques

Social science empirical research about law –qualitative as well as quantitative -- stands or falls on systematic procedures for combining ideas & evidence

3

Historical comparative methods

also known as Case-oriented comparative methods &

Case-oriented narrative & comparative methods &

Comparative narratives

Qualitative interpretive strategies for making sense of and explaining some empirical phenomenon or set of phenomena

4

Law as “dependent variable”

Law as object of inquiry – that which we want to understand and explain

E.g. how and why all capitalist democracies except US have abolished death penalty, how and why US remains exception

5

Law as “independent variable”

When something about legal action, doctrine or institutions helps us understand and explain something else about social life, i.e., law as part of explanatory apparatus.

E.g. research stream showing how 19th & early 20th century criminal conspiracy & antitrust doctrine & also labor injunction shape organization, strategies and ideology of American labor movement (in comparison with Europe).

6

Qualitative-interpretive strategies

specific, systematic methods for determining whether you have arrived at a better or worse interpretation & explanation

Better means well supported by empirical data conducted in such a way as to provide the maximal opportunity to know if your ideas are wrong.

Trying to explain how world works, how & why social processes, including legal processes work differently under different sets of conditions

Since empirical world ultimate arbiter of ideas, methods for knowing empirical world become very important.

7

Qualitative-interpretive strategies (2)

Regardless of agreement/disagreement about moral valence of what we find, other empirical researchers should be able to follow our methodological logic and replicate it.

Getting accurate picture of how world works essential for coming up with good policies/legal rules.

Even if ultimate goal is to get people to adopt particular normative approach, good empirical research essential.

8

Outline of Workshop

Define key terms, techniques show where they fit in general logic of historical-comparative research strategies.

Flesh out general logic of with examples & practical “how-tos” appropriate for the different stages of this type of empirical research project.

-- case selection & research design--data sources & data collection--data management, coding & analysis

Issues, questions, dilemmas you want to raise for collective discussion & advice

9

Cases, Narratives and Comparisons

Narratives – sequential accounts organizing material into chronological order to tell stories about what happened. Conceptual wholes built by selecting, linking otherwise discrete parts, so each takes on meaning in light of whole.

Narratives compel focus on individual, institutional &/or collective actors, actions they take, when, where, why, how & with what consequences they take them.

Implicitly or explicitly, narratives tell not just what happened, also explain how & why happened as it did and not otherwise.

10

Narrative as term of art to refer to particular analytic method

Not any old story counts as analytic narrative Other people’ voices/stories may be raw material

but not in themselves the analytic narrative. Some stories more than others, constructing stories

in some ways rather than others better, if goal is to produce valid & reliable knowledge about general social processes (Stryker SMR 1996)

Part of what historical-comparative methods do is construct sequentially ordered accounts of who did what, when, where, how, under what conditions & with what consequences.

11

Sometimes analytic narrative part of comp-historical research called “process tracing”

Especially useful in considering contingency points built into any sequence of events.

Could things have been different? How & at what point(s) could they have been different? Why weren’t they? What difference would it have made later down the line in the event sequence if some of earlier events had not happened or had happened differently?

“Action coding” – type of data coding facilitating analytic narrative/process tracing (Stryker, SMR 1996)

12

For more on narrative

See additional reference list on Workshop Handout, especially:

Griffin 1993; Quadagno and Knapp 1992; Stryker 1996.

13

Cases

Term of art to refer to what analyst constructs narratives & comparisons of/about

Not same thing as legal case

To “get” what social science case is, and is not, note ways in which it is, and is not, analogous to the legal case.

14

Legal case vs. social science case

Legal case is basic unit of legal research & analysis Apply legal argumentation skills to making sense of

cases, finding similarities and differences among cases.

Social science case denotes basic unit of empirical observation &/or analysis.

Social scientists collect, analyze empirical information on cases.

Social science cases can take on almost infinite array of concrete forms (from 1 person or legal dispute to an organization (Boalt, WTO, ECJ) to American legal system to global capitalist economy)

15

Ex: Legal case as point of departure for social science case

Beth Harris, “Representing Homeless Families: Repeat Player Implementation Strategies,” Law & Society Review 33: 911-939 (1999)

Constructed narratives of course & outcome of implementation for 3 “right to home” class action lawsuits.

Identified amount/kind of success in terms of providing concrete benefits to disadvantaged

Lined up similarities & differences in implementation among the three cases to identify conjunction of factors/conditions most likely to maximize benefits for the disadvantaged.

16

From Harris to more general point

Harris’ units of observation & analysis come from three major legal disputes resulting in consent decrees. She selected these 3 empirical cases because of analytic leverage they provided on her particular research question

Key general point: What is an appropriate case can only be seen/decided form point of view of an empirical research question & empirical research design. And what is a good empirical research question & design can only be seen/decided when refracted against scholarly literature.

17

Case-oriented legal reasoning

refers to standard interpretive techniques suitable for making judicial arguments.

Reasoning by analogy from precedent, distinguishing cases, constructing legislative intent, etc. leave much room for creativity, discretion but also channel creativity/discretion according to set of methodological rules for legal argumentation.

In law, there are methodological rules for putting together legal ideas with factual evidence.

Asking “what general principles does the case stand for” uses the legal case as empirical unit to convert the legal case into the representation of an analytic construct.

18

Analogy to social science case

When doing case-oriented social science empirical research about law, the standard reasoning techniques are analytic narrative & comparative techniques.

Like the legal case, social science case both a concrete empirical unit of observation & analysis AND the representation of an analytic construct standing for some more general pattern.

Unlike the legal case, the social science cases become analytic constructs relative to social science theories about general patterns of association or cause-effect in social life, combined with appropriate methods of social scientific empirical research.

19

Cases within cases strategy

Strategy to get maximum explanatory leverage out of small numbers of cases

Small-N studies –rich, deep information on cases, whether they are field sites or historical events & processes

In small-N studies, research findings serve as empirical grounding for modifying social science concepts & theories, constructing new concepts, theories & research hypotheses, & constructing new operational measures of key concepts

Trade offs b/w small-N and large-N studies

20

For more reading on cases

See references in Workshop Handout, especially:

Ragin and Becker (1997) Amenta (1991) Ragin (1987) Stryker (1996, 2003)

21

Comparison

Integral to reasoning by analogy

In empirical research, use comparisons at two different stages of research inquiry-- initial case selection--additional comparisons among the cases selected for in-depth empirical research.

22

Comparison as “congruence testing”

Comparison for congruence testing involves analogical reasoning, but NOT analogical reasoning from precedent.

Both during case selection and in doing the resulting research, we compare to search for patterns of similarities and differences in the cases, based on the empirical information we have gathered.

23

How to know what data to collect

Guided by a pre-existing conceptual framework Some information so essential that if it can’t be

gotten, better move on to another research project (the “fatal analytic flaw” test)

Go beyond a pre-existing conceptual framework in data gathering so that if our empirical research warrants, we can use it to suggest new explanatory concepts and research hypotheses that can guide us in our next empirical research project or can guide someone else in their empirical research.

24

Comparisons usually done both

With respect to concrete information we gather about specific aspects of our empirical cases AND

With respect to the more abstract & general

conceptual meaning of the cases when refracted against the conceptual/analytic frameworks and research hypotheses that are the starting point for the empirical research.

25

Extended Example (from Stryker 1990)

Compare political attacks on economic division of NLRB, 1935-40 and on economic division of FTC, 1915-20, examining how they were generated, their nature and outcomes/consequences.

Strategic comparison in part because of things both cases shared, e.g., both adjudicative rule-making agencies YET economic division at NLRB abolished, FTC economic unit remains.

This difference in outcomes/consequences in two cases of political attack is the dependent variable (object of inquiry) to understand and explain.

26

Extended Example (from Stryker 1990), continued (2)

Key similarity that both NLRB & FTC are adjudicative rule making agencies meant cases selected held constant/controlled for legal-institutional framework in which economic expertise & knowledge operated.

Rules out an otherwise plausible explanation that could account for different outcomes in the two cases.

Another key similarity – first 5 years of each agency = early institutionalization as analytic construct. This similarity becomes important in light of scholarly literature in organizational sociology.

27

Extended Example (from Stryker 1990), continued (3)

Similarity on early institutionalization, just like similarity in legal institutional framework for economic expertise (and just like other similarities not discussed in workshop but available from Stryker 1990), gives set of potential explanatory factors/independent variables held constant in NLRB/FTC comparison.

If you “control for” or hold some things constant in multiple cases, these things can not account for some difference in the cases that you want to explain.

Neither differences in legal-institutional framework or differences in timing relative to agency institutionalization can account for different outcomes for NLRB & FTC econ. units.

28

In sum, with respect to example

Haven’t told you what my own explanation for NLRB/FTC difference was, nor how my own empirical research on the 2 cases led to a new, more general analytic framework and set of explanatory hypotheses.

Have told you about: 1) why chose to conduct in-depth research on NLRB and FTC; 2) how began to analytically construct my cases to justify this 2 case comparative design.

2 cases were cases of regulatory agencies whose economic units subject to political attacks by Congress and important business organizations during the initial institutionalization period AND in which economic expertise had the same legal function in adjudicative rule making…

29

To review:

Legal cases vs. social science cases Historical-comparative research requires identifying and

conceptually/analytically constructing cases based on empirical information.

Historical-comparative methods provide narrative accounts emphasizing sequences of actions & events, actors involved, what they did, when and under what conditions, to whom or what, and with what consequences

Historical-comparative methods compare features of cases to identify empirical patterns of similarities and differences that are consistent & inconsistent with various causal explanations for social (including legal) behavior, processes or institutions

30

Research design & Case selection – some key points

1. What makes for a good research design and a good case or cases to study can only be assessed relative to the particular research question asked and especially how that question fits within extant theoretical & empirical literature in a field

2. Historical comparative research strategies involve “concurrent construction and mutual adjustment of theory and empirical evidence” (Stryker 1996). This is an iterative procedure in which needed evidence defined with respect to starting ideas. Then starting ideas become modified in light of new, detailed empirical evidence that is collected and analyzed.

31

For more reading on using narrative with comparative methods

See reference list in Workshop Handout, especially:

Goldstone 2003Quadagno & Knapp 1992Stryker 1996Ragin 1987

32

Research design & Case selection – some key points (2)

3. Research design/case selection typically time consuming because selecting cases already requires substantial empirical work at the front end of the research project.

4. Here is the typical research progression that results in research design and case selection:

a) start reading previous lit. on general research topic

b) reading & synthesizing what’s been said in prior lit. allows you to use your literature review & synthesis to find a useful specific research question (e.g., directed to questioning & improving on received wisdom, addressing controversies in field, etc.)

33

Research design & Case selection – some key points (3)

More steps on the typical research progression resulting in case selection & research design:

c) As you work toward a research question that can be addressed with historical comparative research, you simultaneously figure out some key analytic characteristics the cases you choose to research must have.

d) As you figure out these analytic characteristics, do preliminary research on larger # of cases than those you ultimately pursue. Also explore what data & data sources available to learn about these cases.

e) Some of the potential cases will not be feasible because of lack of data sources or data.

34

Research design & Case selection – some key points (4)

More steps in the typical research progression resulting in case selection & research design:

f) Other of the potential cases get eliminated because they turn out not to satisfy key analytic criteria for case selection (Recall: criteria come from research question situated in prior scholarship).

g) End result: preliminary research & elimination process will provide very strong rationale for the finalized research design and case selection.

“Historical comparative researchers select and construct empirical cases in response to a clearly developed general theoretical backdrop.” (Stryker 1996)

35

Research design & Case selection – some key points (5)

5. Because historical comparative research requires much empirical depth on all cases on which you focus, typically researchers start by examining very small number of cases that both: a) meet their analytic criteria; and b) meet their data source and availability criteria

6. Researchers may start with one or two cases. If start with one case only, choose it so it represents a puzzle or anomaly in light of past theory and compared with other empirical cases.

36

Running illustration 1

NLRB/FTC comparative research design (Stryker 1990) the second project in research program examining what accounts for variation in role of economic expertise across regulatory arenas.

Starting point: case study on NLRB took form when identified NLRB as an anomaly/puzzle (Stryker 1989)

Comparative data to establish empirical anomaly Empirical anomaly likewise a theoretical

anomaly relative to social science literature on how government & science inter-relate.

37

Running illustration 1(continued)

Explaining what happened at NLRB, how and under what conditions, provides analytic narrative useful in own right.

Also useful to suggest needed modifications to more general social science theories given NLRB a case in which then extant general theoretical predictions of increasing government reliance on science did not hold.

In-depth strategic or analytic narrative of NLRB is then used to suggest some new general hypotheses about conditions under which US government expands or contracts its use of social scientific-technical expertise

Case study, but within a broader comparative framework – enough comparative analysis to show NLRB is an anomaly.

38

Running Illustration 2 – Gulseren Kozak-Isik’s dissertation, Univ. of Minnesota*

Archival, participant observation and in-depth interviewing to study how Islamic legal, religious and political institutions reconstructed in west.

Comparative research focused on two major umbrella organizations, Muslim Council of Britain (MCB) and Islamic Society of North America (ISNA).

Moving from general topic to specific research question required identifying focused set of Islamic principles likely to be contested, renegotiated with implications for Islamic immigrants, host countries.

*See associated reading list for full citation.

39

Running Illustration 2 – Gulseren Kozak-Isik’s dissertation (continued)

Umma, caliphate, shari’a Citizenship in & primary loyalty to secular nation-

states, separation of church & state, individualism, gender equality, state legal hegemony.

Preliminary empirical research identifying major Muslim organizations, how, by whom created, how interconnected, what constituencies served in US and Europe

Preliminary empirical research on key aspects of receiving country politics & policies with regard to Muslim immigrants

Ended up with MCB/ISNA comparison. Why?

40

Running Illustration 2 – Key analytic & empirical features of Isik research design

MCB & ISNA largest, most comprehensive Muslim organizations in US and UK.

While MCB does have small number Shia affiliates, both MCB and ISNA overwhelmingly Sunni organizations.

MCB & ISNA have similar origins in various respects yet have evolved quite differently with respect to their interpretation of Islamic legal, political & religious institutions.

Choosing two cases that have evolved differently despite many important similarities should permit identifying in an especially clear way key factors that shaped MCB/ISNA differences in interpreting key Islamic concepts, doctrine.

41

Running Illustration 2 – Potential explanatory factors highlighted by research design

Differences in internal constituencies of MCB and ISNA

Differing government policies and organizational environments in the two host countries.

Much empirical work to do to see to what extent her specific hypotheses are right – and what other factors not prefigured but discovered in course of the empirical research, might also be operating.

Kozk-Isik research design and case selection is an example of a most similar cases strategy.

42

Most similar cases strategy outlined

Researcher concentrates empirical research on cases as similar as possible on some set of key factors that would otherwise be expected to include the process s/he is investigating

Preliminary empirical research to identify small number of “most similar” cases. Having found them, researcher has held constant factors that otherwise would be raised as good explanations for what s/he wants to explain.

43

Most similar cases strategy outlined (continued)

With some factors eliminated as potential explanations, researcher can move from research design and case selection to additional data collection & analysis that:

a) provide much more information on exactly how, under what conditions, and with what consequences the varying object of explanation in the otherwise similar cases happened

b) compare in systematic & highly detailed way, exactly how & why the two cases diverged.

c) is maximally likely to provide insight into the role that explanatory factors which do vary b/w the cases play in explaining how and why cases evolved differently.

44

Note:

Most similar cases research designs and also case study designs of anomalous cases require empirical research on front end AND on back end of your project.

Unlike in the study of the anomalous case, the back end of a study using a fully comparative design will involve comparative analyses of the multiple (most similar) cases that are as rich and detailed as the analytic narrative needed for each case.

45

Illustration 3: Cases within cases research design

Stryker, “Social Science in Government Regulation of Equal Employment Opportunity” (NSF SES#0514700, 2005-08)

Role of economics, psychology, sociology & statistics in enforcement of Title VII of CRA of 1964 and Equal Pay Act of 1963.

Enforcement defined very broadly. Outcomes to be explained differ across regulatory

arenas. Labor law/employment discrimination comparison

especially useful in showing whether & how it matters that labor law is class politics and EEO-AA is race & gender politics.

46

Illustration 3: But what about cases within cases?

Are there any patterns of similarities and differences in outcomes and potential explanatory factors to exploit within employment discrimination law?

From my prior research: If use of social science viewed as consistent rather than inconsistent with market values and principles, it’s less likely to be subject to political attack spilling over boundaries of formal adversary enforcement and more likely to be firmly institutionalized in enforcement.

47

Illustration 3, continued – within employment discrimination law

Compare role & politics of statistics in legal proof, remedies for intentional, “bad actor” discrimination with role and politics of statistics in group, effects-oriented ways of proving & remedying discrimination; the first more easily framed as consistent with merit-oriented, competitive individualism.

7 key event sequences in EEO-AA identified, all involve important role for social science & in all, some aspect or use of social science subject to broad politicization. Outcomes vary in e.g., breadth, duration, intensity of conflict, government and societal fault lines, coalitions, whether conflicts spill over from litigation to broader politics, whether enforcement role, presence, influence of social science is restricted.

48

Data sources & data collection – Some tips

1. Typically historical comparative research rests on collecting, analyzing many types of existing public, private docs.

2. Today’s “detective work” much easier than it used to be especially for recent periods because of electronic data bases, indexes, web sites.

3. Small grants for on-site research at many archives, each has own rules for scanning, own cost structure for copying – usually contact person listed on web.

4. Given electronic legal data bases, it’s possible to handle very systematically much larger amount of documentary data than previously.

49

Data sources & data collection – Some tips (continued)

5. Computer-assisted coding & analysis only as good as the thought that goes into organizing it, de-bugging it, interpreting, it & figuring out what other patterns in your data (e.g., of similarity & difference among your cases) should hold for the interpretive understanding and causal explanation you are developing to hold up as most consistent with your empirical evidence.

5. Reading secondary treatments of actors, events in your cases helps identify primary documents & discover where they are & from whom you need permission to gain access.

5. Gaining access to private documents usually is not a trivial matter – use preliminary research, contacts.

50

Data sources & data collection – Some tips (continued)

8. In whatever personal or public archive non-public docs are located, there will be procedures to gain access. Finding out from whom you need permission & getting it can take awhile. Start early.

9. Except when point is, e.g., analyzing published legal opinions, analyzing discursive framing in public discourse alone, gaining access to private docs of some sort usually essential. Otherwise, lack information about totality of actor motivations, perceptions, experiences.

51

Data sources & data collection – Some tips (continued)

To construct analytic narrative, need to know how individual and organizational actors who were involved perceived their environment & conditions they were operating under at the time their actions occurred.

Non-public documents that take you back in time to when the events you are researching were happening will be key.

52

Data sources & data collection – More tips

11. In study of something sufficiently recent, do in-depth interviews with people who were involved in historical sequences studied.

12. Select interviewees known to – or likely to have – different points of view on sequences of events being researched. Try & find people who represent different organizations or institutional locations involved, also people who represent different time points in the unfolding processes to be understood & explained.

13. All participants have biases – getting different vantage points is a very, very good thing. Do it.

53

Data sources & data collection – More tips

14. Because interviews are retrospective, what interviewees say is shaped by: a) memory failures; & b) 20-20 hindsight about what happened & its consequences.

15. Best practice to conduct interviews & archival or documentary research in tandem. When you collect documents and interview data in tandem, each informs the other so you get the most out of both.

E.g., docs help identify interviewees, interviews help fill gaps in doc record, docs used to construct better interview questions, anchor memories, use docs to cross-check, validate, interpret sensibly what interviewees say.

54

Data sources & data collection – More tips on interviews

16. Desirable to have detailed list of specific topics, often possible and desirable to have complete interview schedule containing specific questions for each topic.

17. Some questions same for all, others specific to particular interviewee to capitalize on his/her experience, knowledge

18. Start fairly open-ended to not bias or pre-judge how interviewee interprets question & what s/he wants to say but then move to series of more specific questions focusing interviewee on things that bring evidence to bear on analyst’s developing interpretations and explanations.

19. List series of possible probes to elicit details, elaborations.

55

Semi-structured, in-depth interviewing

Semi-structured, because you have flexibility to move to issues & topics previously not considered or known about, but that are interjected by interviewee because he/she knows relevant to research topic and wants to tell you about it.

In-depth, because interviewee gets to elaborate in her own language and terms.

Interviews typically of different duration, depending on purpose of interview and tolerance/forbearance of interviewee.

Jumping around in interview schedule to match way things are flowing is typical, but pause for “stock taking,” so you don’t leave out something important.

56

Ending an interview – questions from Stryker’s NSF 2005-08 project

“Based on your knowledge of my research interests and all the things we did talk about, are there other things that I should have asked you but did not? If so, what are they?”[If there are some of these, ask for more information on each]

“Are there additional people you would suggest I interview?”

“Are you in possession of any documents that you feel would be helpful & that you are willing to share with me?”

57

How do you know on what to collect data?

Beyond knowing you want to collect sufficient data to construct chronologies of actions & events & to show how these are inter-linked in an unfolding process, what else?

Can’t know without having developed a detailed conceptual/ theoretical framework for your narratives & comparisons.

Stryker 1996 provided as reading in part because it gives detailed example of the framework for my data collection, management, coding & analyses relative to my research question. Shows step by step how I arrived at politics of science theoretical framework.

58

Appendix A in Stryker 1996

All the topics on which data collected and which are starting points for content coding once documentary and interview data gathered.

Today’s Handout, pp. 2-12 reproduces Appendix A and provides specific examples of application of coding techniques.

Short hand run-down of the topic list, explanatory and outcome factors.

59

Data Management, Coding & Analysis

Data management essential, otherwise drown in data

Need to coordinate multiple person project, ensure all project assistants use same techniques in data collection, management, coding.

Much more detail on data management, techniques for action coding and content coding for analytic narrative and comparative analyses available in Stryker, SMR 1996.

More detail as it pertains to specific examples of research questions & corresponding analyses in:Pedriana & Stryker 1997, 2004, Stryker, Scarpellino & Holtzman 1999, Stryker & Wald 2007.

60

In general:

Best practice data management & coding techniques designed to maximize validity & capacity to replicate the research.

Degree of precision needed in coding any particular doc or set of docs depends on what role data will serve in overall analysis.

Transcribed interview data can be content coded using topic codes in theoretical framework.

Whether or not your documents are electronic, you’ll want to fully catalogue them electronically.

For many projects, some documents will be readily available electronically, but others not (at least at reasonable time & cost).

61

Example: four document groups for data management & coding*

1. Secondary & primary docs read, summarized & content coded (by me or RAs) for general context, flow of employment discrimination law enforcement, role of social science, and controversies over that role.

2. All primary docs pertinent to key event sequences are action coded & content coded in some detail.

3. For all Title 7 and EPA cases selected (both purposive & random sample) – electronic text is coupled with detailed content coding based on highly specific coding scheme for assessing nature, amount & role of social science in court.

*Also from R. Stryker, “Social Science in Government Regulation o Equal Employment Opportunity.” funded by National Science Foundation Sociology Program, SES#0514700 (2005-08).

62

Example: four document groups, continued

4. Docs especially useful for analyzing how actors discursively frame enforcement role of social science as consistent or inconsistent with univeralist, competitive individualist market values.

--- In addition to being summarized, these are input, coded & analyzed using Atlas.ti. Computer assisted techniques for qualitative data allow quantitative counts of relative frequency & analyses of associations among discursive frames, how frames associated with factors like professional background, expertise or employing agency, or political party of actor mobilizing the frame.

63

Note:

Quantitative evidence on relative frequency, patterns of association is not goal in itself. Rather, it is helpful in arriving at interpretive understanding of discursive frames and of their explanatory role relative to, or combined with institutional factors.

Special programs like Atlas.ti are helpful, but not essential as long as your raw text exists within an electronic database allowing you to search for, select, sort and count textual items fulfilling criteria you, the analyst, set.

64

My project:

To facilitate data management, narrative & comparative analyses and writing up the results as a set of synthetic interpretations and causal explanations, all summaries done by any member of project team, with attached content & action coding, produced and archived as MS Word files.

Complete electronic text files likewise stored for many documents, but sometimes only hard copy text.

65

Action coding -- Example

For each document, action coding records the participating persons, organizations, what they did, when, where, with what resources & consequences, & pertaining to what policy arena event sequence.

Handout, p. 13 provides example. Cong.

Quarterly docs & Civil Rights Act of 1991. Especially useful for constructing chronologies,

analytic narratives.

66

Content coding -- Example

Attaches topic codes for factors in politics of science framework; this is especially useful for constructing analytic comparisons – comparative narratives – to provide causal explanation.

See Handout, pp. 2-5 for original topic codes. During course of research, more topic codes

added – some examples are in Handout, pp. 6-7

Periodic need to clarify criteria for applying particular topic codes to ensure consistency among coders.

67

Training & supervision

Includes reviewing RA work from week to week to spot coding errors or problems, keep track of where things are, what is done, what still needs to be done, & what is the next step.

Periodically emerge from details to big picture, doing some writing working toward interpretive & explanatory conclusions (interim memos and reports)

Document summaries & coding by RAs often achieved

interactively, through responses to my requests for further clarification and elaboration based on initial summaries & coding.

When I make a decision to have new topic code, or clarify the rules for applying a topic code, this is documented in written form.

68

Example of iterative research process

Pp. 8-10 Example of document that was summarized and content coded by a project RA; Pp. 11-12 my response.

BNA Pay Equity and Comparable Worth 1984. First Appendix A categories, then Main Actors, then Supplemental Materials/Documents. Then summary proper starts, including running page numbers for both summarized material and quotes.

My response adds additional content codes & analytic points I wanted to flag; also asks questions, specifies some follow ups.

Afterwards, project RA produces finalized version of Work Memo 79.

69

Some general points about content coding

1. Content coding of this sort already has some interpretation & analysis contained within it.

2. Coding & analysis are separate research phases in quantitative res., but in qualitative research coding more part of analysis.

3. But checking, validation, requires summary material & iterative procedure among RA coders and PI.

4. General topic codes provide basis for search and sorts to gather all material pertaining to particular analytic points, e.g., everything pertaining to relationship EEOC and CRD, everything pertaining to legal institutional framework for use of social science at EEOC or CRD, without prejudging what analytic judgment will be made based on all the evidence contained in all the interpretively-linked summary material.

70

More general points about content coding

5. Topic codes are like variables (age) without variable “values.” (36, 47, 52).

6. Coders should only do what they are able to do consistent with the instructions & training you provide & their expertise – if you ask more of them, you will sacrifice validity &reliability.

7. Reliability basically measures consistency in applying coding rules. Do different coders using the same set of rules to code the same text come up with the same coding results?

8. Inter-coder reliability assesses degree to which they do. You want it to be as close to 100 percent as possible.

71

More general points about content coding

6. Example: coding legal cases – an RA who is not legally trained could not apply Monahan & Walker’s categories for role of social science as “social facts,” “social authority” or “social framework,” nor could such a coder apply my elaboration, modification on these categories.

Such a coder can, however, be trained in what is meant by “social science sources,” if PI supplies an explicit, precise operational definition. Then coder can count up whether and how many of these appear in a legal opinion (majority, concurrence, dissent).

72

Example, continued:

Project coding sheets & instructions designed to have coders answer a series of explicit questions, then provide the relevant data by elaborating in response to specific further probes & cutting & pasting relevant parts of text of legal opinion. E.g., Does the majority opinion explicitly mention one or more expert consultants, witnesses, information providers for P? If so [continue with additional battery of questions?] When these coding sheets are reviewed by PI in conjunction with full text of document, can be efficiently converted into analytic categories reflecting role of social science as e.g., “social facts” vs. “social frameworks.”

73

Coding discursive frames

Based on preliminary research through secondary literature and summaries of primary documents, analyst can select feasible subsets of documents for computer-assisted analysis of cultural framing.

To enhance reliability, use multiple, independent coders applying same set of coding rules, including word & phrase searches, & working under close supervision, to identify & assess relative prominence of diverse frames.

74

Frames Defined

Central organizing ideas attributing meaning to phenomena/events.

Frames have “signature elements” (Gamson & Modigliani 1987) or catch-phrases used to signal them and to promote one or more frames over others.

E.g., “quotas,” “preferential treatment” are catch phrases often used by those who argue against affirmative action

75

Example of coding for frame analysis

Workshop handout, pp. 13-16, final criteria for frames identified in Stryker & Wald 2007, to examine how general value of compassion shaped debate over PRWORA of 1996.

Final criteria identified iteratively through repeated interactions with data until the necessary level of explicitness, precision is achieved for validity & reliability.

Provide sufficient information on coding frames so that if another analyst replicates the procedures you used on same data, will identify the same frames.

Provide sufficient information so that another analyst can take issue with, question your procedures.

76

Analysis -- Recall

Cases as analytic constructs. Narrative and comparison as analytic techniques.

Identification & further analyses of discursive frames may play a role within a historical-comparative research project.

If so, frame analyses informed by placing frames in historical & comparative context, frame analyses in turn further inform analytic narrative & comparisons.

Examples on Handout reference list: Stryker & Wald 2007, Stryker et al 1999