1VCMJD1PMJDZOBMZTJT %VOO 'JGUI&EJUJPOirpublicpolicy.ir/wp-content/uploads/2017/09/Public-Policy-Analysis... · normative, a term that refers to value judgments about what ought to

Public Policy AnalysisDunn

Fifth Edition

www.irpublicpolicy.ir

Pearson Education LimitedEdinburgh GateHarlowEssex CM20 2JEEngland and Associated Companies throughout the world

Visit us on the World Wide Web at: www.pearsoned.co.uk

© Pearson Education Limited 2014

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without either the prior written permission of the publisher or a licence permitting restricted copying in the United Kingdom issued by the Copyright Licensing Agency Ltd, Saffron House, 6–10 Kirby Street, London EC1N 8TS.

All trademarks used herein are the property of their respective owners. The use of any trademark in this text does not vest in the author or publisher any trademark ownership rights in such trademarks, nor does the use of such trademarks imply any affi liation with or endorsement of this book by such owners.

British Library Cataloguing-in-Publication DataA catalogue record for this book is available from the British Library

Printed in the United States of America

ISBN 10: 1-292-02021-0ISBN 13: 978-1-292-02021-1

ISBN 10: 1-292-02021-0ISBN 13: 978-1-292-02021-1


Table of Contents

P E A R S O N C U S T O M L I B R A R Y

I

1. The Process of Policy Analysis

1

1William N. Dunn

2. Policy Analysis in the Policy-Making Process

31

31William N. Dunn

3. Structuring Policy Problems

65

65William N. Dunn

4. Forecasting Expected Policy Outcomes

117

117William N. Dunn

5. Prescribing Preferred Policies

189

189William N. Dunn

6. Monitoring Observed Policy Outcomes

247

247William N. Dunn

7. Evaluating Policy Performance

311

311William N. Dunn

8. Developing Policy Arguments

339

339William N. Dunn

9. Communicating Policy Analysis

383

383William N. Dunn

10. Appendix: The Policy Issue Paper

425

425William N. Dunn

11. Appendix: The Policy Memorandum

433

433William N. Dunn

12. Appendix: Planning Oral Briefings

441

441William N. Dunn

451

451Index


II www.irpublicpolicy.ir

The Process of PolicyAnalysis

From Chapter 1 of Public Policy Analysis, Fifth Edition.William N. Dunn. Copyright © 2012 byPearson Education, Inc. All rights reserved.

1www.irpublicpolicy.ir

O B J E C T I V E S

By studying this chapter, you should be able to

The Process of PolicyAnalysis

� Define and illustrate phases ofpolicy analysis.

� Describe elements of integratedpolicy analysis.

� Distinguish four strategies of policyanalysis.

� Contrast reconstructed logic andlogic-in-use.

� Distinguish prospective andretrospective policy analysis.

� Describe the structure of a policyargument and its elements.

� Understand the role of argumentmapping in critical thinking.

� Interpret scorecards, spreadsheets,influence diagrams, decision trees,and argument maps.

Policy analysis is a process of multidisciplinary inquiry aiming at thecreation, critical assessment, and communication of policy-relevant infor-mation. As a problem-solving discipline, it draws on social science methods,

theories, and substantive findings to solve practical problems.1

1For a sample of alternative definitions see Harold D. Lasswell, A Pre-view of Policy Sciences(New York: American Elsevier Publishing, 1971); Yehezkel Dror, Ventures in Policy Sciences: Conceptsand Applications (New York: American Elsevier Publishing, 1971); Edward S. Quade, Analysis forPublic Decisions, 3d rev. ed., ed. Grace M. Carter (New York: North Holland Publishing, 1989); DavidL. Weimer and Aidan R. Vining, Policy Analysis: Concepts and Practice, 2d ed. (Englewood Cliffs, NJ:Prentice Hall, Inc., 1992); Duncan Mac Rae Jr., The Social Function of Social Science (New Haven, CT:Yale University Press, 1976).

2 www.irpublicpolicy.ir

The Process of Policy Analysis

METHODOLOGY OF POLICY ANALYSISAs used here, the word methodology refers to a process of reasoned inquiryaimed at finding solutions to practical problems. The aim of methodology is tohelp us understand not only the products of policy inquiry but also the processesemployed to create these products.2 The methodology of policy analysis is notconfined to the analytical routines of specialized social science fields—forexample, benefit-cost analysis in economics or implementation analysis inpolitical science—because none of these holds a privileged place in policy inquiry.Nor is the methodology of policy analysis constrained by the doctrines andprinciples of obsolescent philosophies of science such as logical positivism, whichmistakenly claimed that scientific knowledge, properly understood, is objective,value free, and quantitative.3 On the contrary, policy analysis is methodologicallyeclectic; its practitioners are free to choose among a wide range of scientificmethods, qualitative as well as quantitative, as long as these yield reliableknowledge. In this context, policy analysis includes art, craft, and reasonedpersuasion, all of which are scientific to the extent that they succeed in producingreliable knowledge.4 Ordinary commonsense knowing and well-winnowedpractical wisdom—both products of evolutionary learning across generations of

2Abraham Kaplan, The Conduct of Inquiry: Methodology for Behavioral Science (San Francisco, CA:Chandler Publishing Company, 1964), pp. 23–24.3Logical positivism (or logical empiricism) was abandoned by most philosophers of science more than50 years ago, although its epistemological pillars—the correspondence theory of truth, the empiricalcriterion of meaning, and quantificationism—are still venerated by many social scientists. For alterna-tives to logical positivism in economics and political science see Daniel Bromley, Sufficient Reason:Volitional Pragmatism and the Meaning of Economic Institutions (Princeton, NJ: PrincetonUniversity Press, 2006); Henry E. Brady and David Collier, eds. Rethinking Social Inquiry: DiverseTools, Shared Standards (Lanham, MD: Rowman Littlefield, 2004); Deirdre N. McCloskey, TheRhetoric of Economics, 2nd ed., Madison: University of Wisconsin Press, 1998; Stephen ThomasZiliak and Deirdre N. McCloskey, The Cult of Statistical Significance: How the Standard Error CostsUs Jobs, Justice, and Lives. Ann Arbor: University of Michigan Press, 2008. Paul Diesing, How DoesSocial Science Work? Reflections on Practice (Pittsburgh, PA: University of Pittsburgh Press, 1991);and Mary Hawkesworth, Theoretical Issues in Policy Analysis (Albany: State University of New YorkPress, 1988).4Larry Laudan has argued that the demarcation between science and non-science, including art andcraft, is a pseudo-problem that should be replaced by focusing on the distinction between reliable andunreliable knowledge. It is not necessary to ask whether knowledge is “scientific,” only whether it isreliable. “The Demise of the Demarcation Problem,” in R.S. Cohen and L. Laudan, Physics, Philosophyand Psychoanalysis: Essays in Honor of Adolf Grünbaum. Boston Studies in the Philosophy of Science,Vol.76 (Dordrecht: D. Reidel, 1983), pp. 111–127. Aaron Wildavsky and others have used the terms artand craft to characterize policy analysis. See Aaron Wildavsky, Speaking Truth to Power: The Art andCraft of Policy Analysis (Boston, MA: Little Brown, 1979); and Iris Geva-May and Aaron Wildavsky,An Operational Approach to Policy Analysis: The Craft, Prescriptions for Better Analysis (Boston, MA:Kluwer, 1997). The term policy science(s) is Harold Lasswell’s. See the short methodological history ofthe policy sciences in Ronald Brunner, “The Policy Movement as a Policy Problem,” in Advances inPolicy Studies since 1950, vol. 10, Policy Studies Review Annual, ed. W. N. Dunn and R. M. Kelly(New Brunswick, NJ: Transaction Books, 1992), pp. 155–97 and contributions to Michael Moran,Martin Rein, and Robert E. Goodin, eds. The Oxford Handbook of Public Policy (Oxford: OxfordUniversity Press, 2006).



problem solvers—often permit conclusions that are more trustworthy andreliable than those produced by means of policy analysis and other specializedforms of professional and scientific inquiry.5

The rationale for policy analysis is pragmatic. For this reason, it is unmistakablydifferent from social science disciplines that prize knowledge for its own sake. Thepolicy-relevance of these disciplines depends not on their status as sciences but onthe extent to which they are successful in illuminating and alleviating practicalproblems, problems that come in complex bundles that are at once economic,political, cultural, ethical, and more. Practical problems do not arrive in separatedisciplinary packages addressed to departments of economics and politicalscience—to name two of the most important policy disciplines. In today’s world,multidisciplinary policy analysis seems to provide the best fit with the manifoldcomplexity of public policy making.

POLICY ANALYSIS—A MULTIDISCIPLINARY FRAMEWORKPolicy analysis is partly descriptive. It relies on traditional social science disciplinesto describe and explain the causes and consequences of policies. But it is alsonormative, a term that refers to value judgments about what ought to be, in con-trast to descriptive statements about what is.6 To investigate problems of efficiencyand fairness, policy analysis draws on normative economics and decision analysisas well as ethics and other branches of social and political philosophy—all of whichare about what ought to be. This normative orientation stems from the fact thatanalyzing policies demands that we choose among desired consequences (ends) andpreferred courses of action (means). The choice of ends and means requires contin-uing trade-offs among competing values of efficiency, equity, security, liberty, anddemocracy.7 The importance of normative reasoning in policy analysis was wellstated by a former undersecretary in the Department of Housing and UrbanDevelopment: “Our problem is not to do what is right. Our problem is to knowwhat is right.”8

5On the contrasts between scientific and professional knowledge on one hand, and ordinarycommonsense knowing on the other, see Charles E. Lindblom and David K. Cohen, UsableKnowledge: Social Science and Social Problem Solving (New Haven, CT: Yale University Press,1979). On the frequent soundness of evolved practical knowledge—but the periodic need forsupplemental scientific testing—see Donald T. Campbell, “Evolutionary Epistemology,” inMethodology and Epistemology for Social Science: Selected Papers, ed. E. S. Overman (Chicago:University of Chicago Press, 1989).6One classic statement of the difference between positive and normative knowledge in economics isMilton Friedman, Essays in Positive Economics (Chicago, IL: University of Chicago Press, 1953). Thissame positive-normative distinction is present throughout the social sciences.7Deborah Stone, Policy Paradox: The Art of Political Decision Making, rev ed. (New York: W. W.Norton, 2001).8Robert C. Wood, “Foreword” to The Study of Policy Formation, ed. Raymond A. Bauer and KennethJ. Gergen (New York: Free Press, 1968), p. v. Wood is quoting President Lyndon Johnson.



Policy-Relevant InformationPolicy analysis is designed to provide policy-relevant information about five types ofquestions:

� Policy problems. What is the problem for which a potential solution is sought?Is global warming a human-made consequence of aircraft and motor vehicleemissions? Or is global warming a consequence of periodic fluctuations in thetemperature of the atmosphere? What alternatives are available to mitigateglobal warming? What are the potential outcomes of these alternatives and whatis their value or utility?

� Expected policy outcomes. What are the expected outcomes of policies designedto reduce harmful emissions? Because periodic natural fluctuations are difficult orimpossible to control, what is the likelihood that emissions can be reduced byraising the price of gasoline and diesel fuel, compared with requiring that aircraftand motor vehicles use biofuels?

� Preferred policies. Which policies should be chosen, considering not onlytheir expected outcomes in reducing harmful emissions, but also the value ofreduced emissions in terms of economic costs and benefits? Should distribu-tional criteria involving environmental justice be used along with criteria ofeconomic efficiency?

� Observed policy outcomes. What policy outcomes are observed, as distin-guished from the outcomes expected before a preferred policy is implemented?Did the preferred policy actually result in reduced emissions? Were otherfactors such as political opposition to governmental regulation responsible forthe limited achievement of emissions targets?

� Policy performance. To what extent do observed policy outcomes con-tribute to the reduction of global warming through emissions controls? Whatare the benefits and costs of government regulation to present and futuregenerations?

Answers to these questions yield five types of information, which are policy-informational components. These components are shown as rectangles inFigure 1.9

A policy problem is an unrealized need, value, or opportunity for improvementattainable through public action.10 Knowledge of what problem to solve requiresinformation about a problem’s antecedent conditions (e.g., school dropouts as anantecedent condition of unemployment), as well as information about values (e.g.,safe schools or a living wage) whose achievement may lead to the problem’ssolution. Information about policy problems plays a critical role in policy analysis,

9The framework was originally suggested by Walter Wallace, The Logic of Science in Sociology(Chicago: Aldine Books, 1971). Wallace’s framework addresses research methodology in sociology,whereas Figure 1 addresses the methodology of policy analysis.10Compare James A. Anderson, Public Policymaking: An Introduction, 7th ed. (Boston, MA:Wadsworth, 2011); Charles O. Jones, An Introduction to the Study of Public Policy, 2d ed. (NorthScituate, MA: Duxbury Press, 1977), p. 15; and David Dery, Problem Definition in Policy Analysis(Lawrence: University of Kansas Press, 1984).



FIGURE 1The process of integrated analysis

because the way a problem is defined shapes the search for available solutions.Inadequate or faulty information may result in a fatal error: defining the wrongproblem.11

Expected policy outcomes are likely consequences of one or more policy alterna-tives designed to solve a problem. Information about the circumstances that gave riseto a problem is essential for producing information about expected policy outcomes.Such information is often insufficient, however, because the past does not repeat itselfcompletely, and the values that shape behavior may change in the future. For thisreason, information about expected policy outcomes is not “given” by the existingsituation. To produce such information may require creativity, insight, and the use oftacit knowledge.12

11Defining the wrong problem is a type III error, as contrasted with type I and type II errors committedwhen the level of statistical significance (alpha) is set too high or too low in testing the null hypothesis.An early statement of this contrast is Ian I. Mitroff and Thomas R. Featheringham, “On SystematicProblem Solving and the Error of the Third Kind,” Behavioral Sciences 19, no. 6 (1974): 383–93.12Dror, Ventures in Policy Sciences; Sir Geoffrey Vickers, The Art of Judgment: A Study of PolicyMaking (New York: Basic Books, 1965); and C. West Churchman, The Design of Inquiring Systems;Basic Concepts of Systems and Organization (New York: Basic Books, 1971).

ProblemStructuring

Forecasting

Monitoring Prescription

POLICYPROBLEMS

PREFERREDPOLICIES

Evaluation

EXPECTEDOUTCOMES

OBSERVEDOUTCOMES

PracticalInference

POLICYPERFORMANCE



A preferred policy is a potential solution to a problem. To select a preferred policy,it is necessary to have information about expected policy outcomes as well as informa-tion about the value or utility of these expected outcomes. Another way to say this isthat factual as well as value premises are required for policy prescriptions. Fact alone—for example, the fact that one policy produces more of some quantity than another—donot justify the choice of a preferred policy. Factual premises must be joined with valuepremises involving efficiency, equality, security, democracy, or some other value.

An observed policy outcome is a present or past consequence of implementing apreferred policy. It is sometimes unclear whether an outcome is actually an effect ofa policy, because some effects are not policy outcomes; many outcomes are the resultof other, extra-policy factors. It is important to recognize that the consequences ofaction cannot be fully stated or known in advance, which means that many conse-quences are neither anticipated nor intended. Fortunately, information about suchconsequences can be produced ex post (after policies have been implemented), notonly ex ante (before policies are implemented).

Policy performance is the degree to which an observed policy outcome con-tributes to the solution of a problem. In practice, policy performance is never perfect.Problems are rarely “solved”; most often, problems are resolved, reformulated, andeven “unsolved.”13 To know whether a problem has been solved, resolved, reformu-lated, or unsolved requires information about observed policy outcomes, as well asinformation about the extent to which these outcomes contribute to the opportunitiesfor improvement that gave rise to a problem.

Policy-Informational TransformationsThe five types of policy-relevant information are interdependent. The arrows connect-ing each pair of components represent policy-informational transformations, wherebyone type of information is changed into another, so that the creation of information atany point depends on information produced in an adjacent phase. Information aboutpolicy performance, for example, depends on the transformation of prior informationabout observed policy outcomes. The reason for this dependence is that any assess-ment of how well a policy achieves its objectives assumes that we already have reliableinformation about the outcomes of that policy. The other types of policy-relevantinformation are dependent in the same way.

Information about policy problems is a special case. Information about policyproblems usually includes some problem elements—for example, potential solutionsor expected outcomes—and excludes others. What is included or excluded affectswhich policies are eventually prescribed, which values are appropriate as criteria ofpolicy performance, and which potentially predictable outcomes warrant or do notwarrant attention. At the risk of being overly repetitious, it is worth stressing againthat a fatal error of policy analysis is a type III error—defining the wrong problem.14

13Russell L. Ackoff, “Beyond Problem Solving,” General Systems 19 (1974): 237–39.14Type I and type II errors are also known as false positives and false negatives. Other sources on type IIIerrors include A. W. Kimball, “Errors of the Third Kind in Statistical Consulting,” Journal of the AmericanStatistical Association 52 (1957): 133–42; Howard Raiffa, Decision Analysis (Reading, MA: Addison-Wesley, 1968), p. 264; and Ian I. Mitroff, The Subjective Side of Science (New York: Elsevier, 1974).



Policy-Analytic MethodsThe five types of policy-relevant information are produced and transformed byusing policy-analytic methods. All methods involve judgments of differentkinds:15 judgments to accept or reject an explanation, to affirm or dispute therightness of an action, to prescribe or not prescribe a policy, to accept or reject aprediction, and to formulate a problem in one way rather than another.

In policy analysis, these procedures have special names:

� Problem structuring. Problem-structuring methods are employed to produceinformation about which problem to solve. One example of problem-structuringmethods is the influence diagram and decision tree presented in Case 3 of thischapter (The Influence Diagram and Decision Tree—Structuring Problems ofEnergy Policy and International Security). Other examples of problem-structuringmethods include critical thinking tools such as argument mapping (Case 4: The Argument Map—Problem Structuring in National Defense and EnergyPolicy).

� Forecasting. Forecasting methods are used to produce information aboutexpected policy outcomes. An example of a simple forecasting tool is thescorecard described in Case 1 (The Goeller Scorecard—Monitoring andForecasting Technological Impacts). Scorecards, which are based on thejudgments of experts, are particularly useful in identifying expectedoutcomes of science and technology policies.

� Prescription. Methods of prescription are employed to create informationabout preferred policies. An example of a prescriptive method is the spread-sheet (Case 2: The Spreadsheet—Evaluating the Benefits and Costs of EnergyPolicies). The spreadsheet goes beyond the identification of expected policyoutcomes by expressing consequences in terms of monetary benefits andcosts.

� Monitoring. Methods of monitoring are employed to produce informationabout observed policy outcomes. The scorecard (Case 1) is a simple methodfor monitoring observed policy outcomes as well as for forecasting expectedpolicy outcomes.

� Evaluation. Evaluation methods are used to produce information about thevalue or utility of observed policy outcomes and their contributions to policyperformance. The spreadsheet (Case 2) may be used for evaluation as well asprescription.

The first method, problem structuring, is about the other methods. For this reason, itis a metamethod (method of methods). In the course of structuring a problem, analyststypically experience a “troubled, perplexed, trying situation, where the difficulty is, as

15John O’Shaughnessy, Inquiry and Decision (London: George Allen & Unwin, 1972).



it were, spread throughout the entire situation, infecting it as a whole.”16 Problemsituations are not problems; problems are representations of problem situations.Hence, problems are not “out there” in the world, but they stem from the interactionof thought and external environments. Imagine a graph showing the growth ofdefense expenditures as a percentage of gross domestic product. The graph representsa problem situation, not a problem, because one analyst will see the graph as evidenceof increasing national security (more of the budget is allocated to defense), whileanother interprets the graph as an indication of a declining budget for social welfare(less of the budget can be allocated to social services). Problem structuring, a proce-dure for testing different representations of a problem situation, is the centralguidance system of policy analysis.

Policy-analytic methods are interdependent. It is not possible to use onemethod without first having used others. Thus, although it is possible to monitorpast policies without forecasting their future consequences, it is usually not possi-ble to forecast policies without first monitoring them.17 Similarly, analysts canmonitor policy outcomes without evaluating them, but it is not possible to evaluatean outcome without first establishing that it is an outcome in the first place. Finally,to select a preferred policy requires that analysts have already monitored,evaluated, and forecasted outcomes.18 This is yet one more way of saying thatpolicy prescription is based on factual as well as value premises.

Figure 1 supplied a framework for integrating methods from different policy-relevant disciplines. Some methods are used solely or primarily in some disciplines,and not others. Program evaluation, for example, employs monitoring to investigatewhether a policy is causally relevant to an observed policy outcome. Althoughprogram evaluation has made extensive use of interrupted time-series analysis,regression discontinuity analysis, causal modeling, and other techniques associatedwith the design and analysis of field experiments,19 implementation research withinpolitical science has not. Instead, implementation researchers have relied mainly ontechniques of case study analysis.20 Another example comes from forecasting.Although forecasting is central to both economics and systems analysis, economicshas drawn almost exclusively on econometric techniques. Systems analysis has madegreater use of qualitative forecasting techniques for synthesizing expert judgment,for example, the Delphi technique.21

16John Dewey, How We Think (Boston, MA: D.C. Heath and Company, 1933), p. 108. The originalstatement of the difference between a problem and a problem situation is attributable to philosophicalpragmatists including Charles Sanders Peirce.17An exception is predictions made on the basis of expert judgment. The explanation of a policy is notnecessary for predicting its future consequences. Strictly speaking, a prediction is a causal inference,whereas a projection, extrapolation, or “rational forecast” is not.18Causation may be assumed but not understood. Recipes claim only that a desired result is a conse-quence of action. Joseph L. Bower, “Descriptive Decision Theory from the ‘Administrative’ Viewpoint,”in The Study of Policy Formation, ed. Bauer and Gergen, p. 10.19See, for example, William R. Shadish, Thomas D. Cook, and Donald T. Campbell, Experimental andQuasi-Experimental Designs for Generalized Causal Inference (Boston, MA: Houghton Mifflin, 2002).20Paul A. Sabatier and Hank C. Jenkins-Smith, “The Advocacy Coalition Framework: An Assessment,”in Theories of the Policy Process, ed. P. A. Sabatier (Boulder, CO: Westview Press, 1999), pp. 117–66.21See the chapter “Prescribing Preferred Policies.”



FIGURE 2Forms strategies of policy analysis

FOUR STRATEGIES OF ANALYSISRelationships among policy-informational components, policy-analytic methods,and policy-informational transformations provide a basis for contrasting fourstrategies of policy analysis (Figure 2).

Prospective and Retrospective AnalysisProspective policy analysis involves the production and transformation of informa-tion before policy actions are taken. This strategy of ex ante analysis, shown as theright half of Figure 2, typifies the operating styles of economists, systems analysts,operations researchers, and decision analysts.

The prospective strategy is what Williams means by policy analysis.22 Policyanalysis is “a means of synthesizing information to draw from it policy alternativesand preferences stated in comparable, predicted quantitative and qualitative termsas a basis or guide for policy decisions; conceptually, it does not include thegathering of information [emphasis in original].” Policy research, by contrast,

22Walter Williams, Social Policy Research and Analysis: The Experience in the Federal Social Agencies(New York: American Elsevier, 1971), p. 8.

ProblemStructuring

Forecasting


EXPECTEDOUTCOMES

OBSERVEDOUTCOMES

POLICYPERFORMANCE

RETROSPECTIVE (ex post):What happened and whatdifference does it make?

PROSPECTIVE (ex ante):What will happen andwhat should be done?

PROBLEM FINDING:What problemshould be solved?

PROBLEM SOLVING:What is the solutionto the problem?

POLICYPROBLEMS

PREFERREDPOLICIES

Evaluation

PracticalInference



refers to “all studies using scientific methodologies to describe phenomena and/ordetermine relationships among them.” Prospective analysis often creates wide gapsbetween preferred solutions and actual efforts to implement them. Perhaps no morethan 10 percent of the work actually required to achieve a desired set of policyoutcomes is carried out before policies are implemented: “It is not that we have toomany good analytic solutions to problems. It is, rather, that we have more goodsolutions than we have appropriate actions.”23

Retrospective policy analysis is displayed as the left half of Figure 2. This strat-egy of ex post analysis involves the production and transformation of informationafter policies have been implemented. Retrospective analysis characterizes the oper-ating styles of three groups of analysts:

� Discipline-oriented analysts. This group, composed mainly of politicalscientists, economists, and sociologists, seeks to develop and test discipline-based theories that describe the causes and consequences of policies. Thisgroup is not concerned with the identification of specific policy goals or withdistinctions between “policy” variables that are subject to policy manipula-tion and those that are not.24 For example, the analysis of the effects of partycompetition on government expenditures provides no information aboutspecific policy goals; nor is party competition a variable that policy makerscan manipulate to change public expenditures.

� Problem-oriented analysts. This group, again composed mainly of politicalscientists, economists, and sociologists, seeks to describe the causes and conse-quences of policies. Problem-oriented analysts, however, are less concerned withthe development and testing of theories believed to be important in social sciencedisciplines than with identifying variables that may explain a problem. Problem-oriented analysts are not overly concerned with specific goals and objectives,primarily because the practical problems they analyze are usually general innature. For example, the analysis of aggregate data on the effects of gender,ethnicity, and social inequality on national achievement test scores providesinformation that helps explain a problem (e.g., inadequate test performance) butdoes not provide information about policy variables that can be manipulated.

� Applications-oriented analysts. A third group includes applied economists,applied sociologists, applied psychologists, and applied anthropologists, aswell as analysts from professions such as public administration, social work,and evaluation research. This group also seeks to describe the causes andconsequences of public policies and programs and is not concerned with thedevelopment and testing of discipline-based theories. This group is concernednot only with manipulable policy variables but also with the identification ofspecific policy goals and objectives. Information about specific goals andobjectives provides a basis for monitoring and evaluating outcomes and

23Graham T. Allison, Essence of Decision: Explaining the Cuban Missile Crisis (Boston, MA: Little,Brown, 1971), pp. 267–68.24James S. Coleman, “Problems of Conceptualization and Measurement in Studying Policy Impacts,” inPublic Policy Evaluation, ed. Kenneth M. Dolbeare (Beverly Hills and London: Sage Publications,1975), p. 25.



impacts of policies. For example, applications-oriented analysts may addressearly childhood reading readiness programs that can be manipulated in orderto achieve higher scores on reading tests.

The operating styles of the three groups reflect their characteristic strengthsand limitations. Discipline-oriented as well as problem-oriented analysts seldomproduce information that is directly useful to policy makers. Even when problem-oriented analysts investigate important problems such as educational opportunity,energy conservation, crime control, or national security, the resultant informationis often macronegative. Macronegative information describes the basic (or “root”)causes and consequences of policies, usually by employing aggregate data to showwhy policies do not work. By contrast, micropositive information shows whatpolicies and programs do work under specified conditions.25 It is of little practicalvalue to policy makers to know that the crime rate is higher in urban than rural areas, but it is practically important to know that a specific form of guncontrol reduces the commission of serious crimes or that intensive policepatrolling is a deterrent.

Even when applications-oriented analysts provide micropositive information,they may find it difficult to communicate with practitioners of ex ante policyanalysis, who in most cases are professional economists. In agency settings, ex ante analysts, whose job it is to find optimally efficient solutions, often havelimited access to information about policy outcomes produced through retrospec-tive analysis. For their part, practitioners of ex ante analysis often fail to specifyin sufficient detail the kinds of policy-relevant information that will be mostuseful for monitoring, evaluating, and implementing their recommendations.Often, the intended outcomes of a policy are so vague that “almost any evalua-tion of it may be regarded as irrelevant because it missed the ‘problem’ towardwhich the policy was directed.”26 Legislators, for example, usually formulateproblems in general terms in order to gain acceptance, forestall opposition, ormaintain neutrality.

Contrasts among the operating styles of policy analysts suggest that disci-pline-oriented and problem-oriented analysis are inherently less useful thanapplications-oriented analysis—that retrospective (ex post) analysis as a wholeis perhaps less effective in solving problems than prospective (ex ante) analysis.Although this conclusion may have merit from the point of view of policymakers who want advice on what actions to take, it overlooks several importantbenefits of retrospective analysis. Retrospective analysis, whatever its shortcom-ings, places primary emphasis on the results of action and is not content withinformation about expected policy outcomes, as is the case with prospectiveanalysis. Discipline-oriented and problem-oriented analysis may offer newframeworks for understanding policy-making processes, challenging conven-tional formulations of problems, questioning social and economic myths, andshaping the climate of opinion in a community or society. Retrospective analysis,

25Williams, Social Policy Research and Analysis, p. 8.26Ibid. p. 13; and Alice Rivlin, Systematic Thinking for Social Action (Washington, DC: Brookings,1971).



however, “has been most important in its impact on intellectual priorities andunderstandings, and not nearly so effective in offering solutions for specificpolitical problems.”27

Descriptive and Normative AnalysisFigure 2 also captures another important contrast, the distinction betweendescriptive and normative strategies of policy analysis. Descriptive policy analysisparallels descriptive decision theory, which refers to a set of logically consistentpropositions that describe or explain action.28 Descriptive decision theories may betested against observations obtained through monitoring and forecasting.Descriptive theories, models, and conceptual frameworks originate for the most partin political science, sociology, and economics. The main function of these theories,models, and frameworks is to explain, understand, and predict policies by identify-ing patterns of causality. The principal function of approaches to monitoring such asfield experimentation is to establish the approximate validity of causal inferencesrelating policies to their presumed outcomes.29 In Figure 2, the descriptive form ofpolicy analysis can be visualized as an axis moving from the lower left (monitoring)to the upper right (forecasting).

Normative policy analysis parallels normative decision theory, which refers to a setof logically consistent propositions that evaluate or prescribe action.30 In Figure 2,the normative strategy of policy analysis can be visualized as an axis running from thelower right (prescription) to upper left (evaluation). Different kinds of information arerequired to test normative and descriptive decision theories. Methods of evaluation andprescription provide information about policy performance and preferred policies, forexample, policies that have been or will be optimally efficient because benefitsoutweigh costs or optimally equitable because those most in need are made better off.One of the most important features of normative policy analysis is that its propositionsrest on disagreements about values such as efficiency, equity, responsiveness, liberty,and security.

Problem Finding and Problem SolvingThe upper and lower halves of Figure 2 provide another important distinction. Theupper half points to methods that are designed for problem finding, whereas thelower designates methods for problem solving. The problem-finding strategy has todo with the discovery of elements that go into the definition of problems, and not totheir solution. How well do we understand the problem? Who are the most impor-tant stakeholders who affect and are affected by the problem? Have the appropriateobjectives been identified? Which alternatives are available to achieve objectives?

27Janet A. Weiss, “Using Social Science for Social Policy,” Policy Studies Journal 4, (Spring 1976): 237.28Bower, “Descriptive Decision Theory,” p. 104.29See Thomas D. Cook and Donald T. Campbell, Quasi-Experimentation: Design and Analysis Issuesfor Field Settings (Boston, MA: Houghton Mifflin, 1979); Shadish, Cook, and Campbell, Experimentaland Quasi-Experimental Designs for Generalized Causal Inference.30Bower, “Descriptive Decision Theory,” pp. 104–05.



Which uncertain events should be taken into account? Are we solving the “right”problem rather than the “wrong” one?

Problem-solving methods, located in the lower half of Figure 2, are designed tosolve rather than find problems. The problem-solving strategy is primarily technicalin nature, in contrast to problem finding, which is more conceptual. Problem-solvingmethods such as econometrics are useful in answering questions about policy causa-tion, statistical estimation, and optimization. How much of the variance in a policyoutcome is explained by one or more independent variables? What is the probabilityof obtaining a coefficient as large as that obtained? Another problem-solving methodis benefit-cost analysis. What are the net benefits of different policies? What is theirexpected utility or payoff?

Segmented and Integrated AnalysisIntegrated policy analysis links the four strategies of analysis displayed in Figure 2.Retrospective and prospective strategies are joined in one continuous process.Descriptive and normative strategies are also linked, as are methods designed to findas well as solve problems. Practically speaking, this means that policy analysts bridgethe several main pillars of multidisciplinary policy analysis, especially economics andpolitical science. Today, this need is not being properly met by specialized socialscience disciplines, which tend to practice segmented policy analysis. The job ofbridging segmented disciplines—to convert intellectual knowledge into practicalknowledge—is carried out by multidisciplinary professions including public adminis-tration, planning, management, and policy analysis. The American Society for PublicAdministration (ASPA), the National Association of Schools of Public Affairs andAdministration (NASPAA), the American Planning Association (APA), theInternational Association of Schools and Institutes of Administration (IASIA), theAcademy of Management (AM), the Operations Research Society of America(ORSA), and the Association for Public Policy and Management (APPAM) areorganizations that represent these professions. So far, these professions have beenmore open to the disciplines of economics and political science than those disciplineshave been open to them, notwithstanding a consensus among policy scholars andpractitioners that the substance and methods of these and other disciplines areessential for producing policy-relevant information.

In summary, the framework for integrated policy analysis (Figure 1) helpsexamine the assumptions, strengths, and limitations of methods employed in disci-plines that tend to be overly segmented and excessively specialized to be useful inpractical problem solving. The framework identifies and relates major elements ofpolicy analysis—policy-informational components, policy-analytic methods, andpolicy-informational transformations—enabling us to see the particular rolesperformed by methods of problem structuring, monitoring, evaluation, forecasting,and prescription. The framework (Figure 2) identifies different strategies of policyanalysis: prospective (ex ante) and retrospective (ex post), descriptive and normative,and problem finding and problem solving. The framework integrates these strategiesof analysis and explains why we have defined policy analysis as a problem-solvingdiscipline that links social science theories, methods, and substantive findings to solvepractical problems.



THE PRACTICE OF POLICY ANALYSISReconstructed Logic versus Logic-in-UseThe process of integrated policy analysis is a logical reconstruction (reconstructedlogic). The process of actually doing policy analysis never completely conforms to thisreconstruction, because all logical reconstructions are abstract representations ofstylized practices endorsed by the scientific community.31 By contrast, the logic-in-useof practicing analysts, as distinguished from the logical reconstruction of their use ofreason and evidence to solve practical problems, always varies from methodological“best practices” due to personal characteristics of analysts, their professional social-ization, and the institutional settings in which they work.

� Cognitive styles. The personal cognitive styles of analysts predispose themtoward different modes of acquiring, interpreting, and using information.32

Corporations, nonprofit organizations, and public agencies such as the U.S.Department of Corrections and the National Science Foundation use theMyers-Briggs test as a training and personnel selection diagnostic.

� Analytic roles. In agency settings, most analysts are largely insulated frompolitics. As such, they are primarily “technicians.” Others perform roles that,in addition to technical content, are political. These “politicians” are activelycommitted to advancing the interests of political leaders or officials to whomthey report. Other activist analysts are “entrepreneurs” who seek greaterinfluence in policy making.33

� Institutional incentive systems. Policy “think tanks” encourage differentorientations toward analysis, including the “humanistic-value-critical” andthe “scientific.”34 Institutional rewards and punishments affect the validityof conclusions and recommendations.35

� Institutional time constraints. Analysts working in governmental settings areoften subject to tight institutional time constraints (three to seven days is typical).They work with much greater speed, and perhaps greater efficiency, than analystsin academic settings or think tanks. Understandably, government analysts rarelycollect original data; nor do they employ complex and time-consumingtechniques.36

31On reconstructed logic and logic-in-use, see Kaplan, Conduct of Inquiry, pp. 3–11.32Studies using the Myers-Briggs type indicator (Jungian personality types) suggest different cognitivestyles among scientists, managers, and analysts. References provided by the Myers and BriggsFoundation at www.myersbriggs.org. See also Ian I. Mitroff and Ralph H. Kilmann, MethodologicalApproaches to Social Science (San Francisco: Jossey-Bass, 1978).33Arnold Meltsner, Policy Analysts in the Bureaucracy (Berkeley: University of California Press, 1976);Robert A. Heineman, William T. Bluhm, Steven A. Peterson, and Edward N. Kearney, The World of thePolicy Analyst. Chatham, NJ: Chatham House, 1990.34Pamela Doty, “Values in Policy Research,” in Values, Ethics, and the Practice of Policy Analysis, ed.William N. Dunn (Lexington, MA: D.C. Heath, 1983).35Donald T. Campbell, “Guidelines for Monitoring the Scientific Competence of Preventive InterventionResearch Centers: An Exercise in the Sociology of Scientific Validity,” Knowledge: Creation, Diffusion,Utilization 8, no. 3 (1987): 389–430.36See P. J. Cook and J. W. Vaupel, “What Policy Analysts Do: Three Research Styles,” Journal of PolicyAnalysis and Management 4, no. 3 (1985): 427–28.



� Professional socialization. The different disciplines and professions that makeup policy analysis socialize their members into different norms and values.Analyses of published papers suggest that analysts employ formal-quantitative aswell as informal-narrative approaches, although sound policy recommendationssometimes require formal-quantitative procedures.37

� Multidisciplinary teamwork. Much of the analysis conducted in publicagencies is carried out by multidisciplinary teams. Some members haveprimary responsibility for the particular types of analysis displayed inFigure 2. Team members trained in economics and decision analysis are typ-ically more qualified to perform prospective (ex ante) analysis, whereasteam members trained in applied sociology, applied political science, andprogram evaluation are usually better at retrospective (ex post) analysis.The effectiveness of teams depends on everyone acquiring an operationalunderstanding of analytic methods employed throughout the process ofintegrated policy analysis.

Methodological Opportunity CostsIntegrated analysis has opportunity costs. Given limited time and resources, it isdifficult to conduct systematic economic, political, and organizational analysessimultaneously. Multiple triangulation,38 or what Cook calls critical multiplism,39

responds to some of the inadequacies of logical positivism.40 Positivism nowappears as a one-sided methodology and epistemology claiming that true statementsabout the world must be logically and empirically verifiable, expressed in a formal(ideal) language such as mathematical statistics, and confirmed by means of state-ments that correspond to objective reality. Objective reality, rather than a realityconstituted by subjective meaningful actions and institutions, is the foundation oftrue statements. Logical positivism, as Cook argues, was the dominant methodologyof policy analysis and program evaluation during the era of President LyndonJohnson’s War on Poverty. The advantage of critical multiplism over logicalpositivism is that multiplism provides a better approximation of what is true byemploying procedures that triangulate from a variety of perspectives on what isworth knowing and what is known about policies.41

37An early but representative overview of approaches is Janet A. Schneider, Nancy J. Stevens, and LouisG. Tornatzky, “Policy Research and Analysis: An Empirical Profile, 1975–1980,” Policy Sciences 15(1982): 99–114.38The methodology of triangulation is analogous to practices employed in geodesic surveys; cartography;navigation; and, more recently, satellite tracking. The position or location of an object is found by meansof bearings from two or more fixed points or electronic signals a known distance apart.39Cook advanced critical multiplism as an alternative to logical positivism. See Thomas D. Cook,“Postpositivist Critical Multiplism,” in Social Science and Social Policy, ed. R. Lane Shotland andMelvin M. Mark (Beverly Hills, CA: Sage Publications, 1985), pp. 21–62.40A critical assessment of logical positivism is Mary E. Hawkesworth, “Epistemology and PolicyAnalysis,” in Advances in Policy Studies since 1950, ed. Dunn and Kelly, pp. 293–328; andHawkesworth, Theoretical Issues in Policy Analysis.41Cook, “Postpositivist Critical Multiplism,” p. 57.



A disadvantage of multiplism lies in its costs. Triangulation among multipledisciplinary perspectives, along with the use of multiple methods, measures, anddata sources, involves trade-offs and opportunity costs.42 When single methodssuch as econometric modeling are employed to achieve measurement precisionand statistical generalizability, analysts forgo opportunities to acquire a deeperunderstanding of policies that is possible through ethnographic interviews, casestudies, and other qualitative methods. A leading econometrician, noting thateconomists are unique among social scientists because they are trained only toanalyze data, not to collect it, observes that “empirical work can be greatlyenhanced by being sensitive to the context of the problem (the data-generatingprocess) and knowing a lot about one’s data.”43 Similar trade-offs apply tomethods of research synthesis, or meta-analysis, which purchase measurementprecision and generalized policy causation at the expense of a deeper understand-ing of contexts of policy-making.44

Ethnographic interviews, by contrast, involve high information costs becausethey require the collection of substantial primary data through interviews.However, they also lack precision and seldom permit the generalization of policycausation to other settings. Although greater precision and generalizability can beobtained by means of field studies and field experiments, these are expensive,especially when they are employed in conjunction with mixed (quantitative andqualitative) methods. To be sure, triangulation among convergent (and divergent)perspectives, methods, and measures may enhance the validity of policy analysisand other applied social sciences.45 But the time and financial constraints maketrade-offs inevitable.

CRITICAL THINKING AND PUBLIC POLICYThe world of the policy analyst is complex. Analysts must sift through and evalu-ate a large volume of available quantitative and qualitative data, make difficultchoices among sources of information, select appropriate methods and techniques,and employ effective strategies for communicating the results of analysis throughoral briefings and documents. These practical challenges place a premium oncritical thinking—that is, the capacity to organize, synthesize, and evaluate diversesources of reasoning and evidence. One method available for this purpose is the

42See David Brinberg and Joseph E. McGrath, Validity and the Research Process (Beverly Hills, CA:Sage Publications, 1985). For Brinberg and McGrath and other methodological pragmatists, the choiceof methods is similar to an optimization problem in decision analysis. See C. West Churchman,Prediction and Optimal Decision: Philosophical Issues of a Science of Values (Englewood Cliffs, NJ:Prentice Hall, 1961); and Russell Ackoff, Scientific Method: Optimizing Applied Research Decisions(New York: John Wiley, 1962).43Peter Kennedy, A Guide to Econometrics, 4th ed. (Cambridge, MA: MIT Press, 1998), pp. 83–84.44See Lawrence Rudner, Gene V. Glass, David L. Evartt, and Patrick J. Emery, A User’s Guide to theMeta-Analysis of Research Studies. ERIC Clearinghouse on Assessment and Evaluation, University ofMaryland, College Park, 2002. http://echo.edres.org45The case for triangulation in its many forms is found in Campbell, Methodology and Epistemologyfor Social Science, ed. Overman.



analysis of policy arguments. By analyzing policy arguments, we are able to iden-tify and probe the assumptions underlying competing policy claims, recognize andevaluate objections to these claims, and synthesize policy-relevant informationfrom different sources.

The Structure of Policy ArgumentsPolicy arguments are the main vehicle carrying debates about public policies.46

Although social scientists may rightly pride themselves on methodological special-ization, they too often forget that “public policy is made of language. Whether inwritten or oral form, argument is central to all stages of the policy process.”47

The structure of a policy argument can be represented as a set of seven elements(Figure 3):48

Policy claim (C). A policy claim is the conclusion of a policy argument.Arguments also include other elements, including policy-relevant infor-mation (I), warrants (W), backings (B), qualifiers (Q), objections (O),and rebuttals (R). The movement from policy-relevant information toclaim implies therefore, thus, or so. Policy claims are of different types.Some are normative: “Congress should pass the amendments to theFair Employment Practices Act.” Some are descriptive: “The use of theInternet will double in the next ten years.”

Policy-relevant information (I). Policy-relevant information provides thegrounds for a policy claim. These grounds may be statistical data, experi-mental findings, expert testimony, common sense, or political judgments.Policy-relevant information is a response to the question: What informationis relevant to the claim? Information is the starting point of a new argumentand the end of a previous one. Policy arguments may lead to complexargument chains, trees, or cycles.

Warrant (W). The warrant is a reason to support a claim. Warrants may beeconomic theories, ethical principles, political ideas, professional authority,and so forth.49 A warrant answers the question: Why does this reasonsupport the claim? Different types of warrants are related to arguments

46See Frank Fischer and John Forester, ed., The Argumentative Turn in Policy Analysis and Planning(Durham, NC: Duke University Press, 1993). Earlier works on policy argumentation are Ian I. Mitroffand Richard O. Mason, Creating a Dialectical Social Science (Boston: D. Reidel, 1981); William N.Dunn, “Reforms as Arguments,” Knowledge: Creation, Diffusion, Utilization 3 (1982): 293–326;Donald T. Campbell, “Experiments as Arguments,” Knowledge: Creation, Diffusion, Utilization 3(1982): 327–47; Giandomenico Majone, Evidence, Argument, and Persuasion in the Policy Process(New Haven, CT: Yale University Press, 1989); and Stone, Policy Paradox and Political Reason.47Majone, Evidence, Argument, and Persuasion, p. 1.48This structural model of argument is part of the computer software called Rationale 2, which was de-veloped by Tim van Gelder and his colleagues in Australia. URL: www.austhink.com. The classic struc-tural model is presented in Stephen Toulmin, The Uses of Argument (Cambridge: Cambridge UniversityPress, 1958); and Stephen Toulmin, A. Rieke, and A. Janik, An Introduction to Reasoning (New York:Macmillan, 1984).49Different kinds of warrants yield the “modes” of policy argument presented in the chapter“Developing Policy Arguments.”



FIGURE 3Elements of a policy argument

Source: Created with Rationale 2. Melbourne: Austhink Consulting, 2010. www.austhink.com

made in different disciplines and professions. For example, law uses casecomparisons and rules of evidence, whereas economics uses theories andcomponent laws such as the law of diminishing utility of money. Policymakers as well as social scientists employ causal warrants such as “Ethniccleansing will be deterred by air strikes that establish NATO’s credibility in

A policy claim is theconclusion of a policyargument. There are fourtypes of policy claims:definitional, descriptive,evaluative, andadvocative.

CLAIM

Policy-relevant information providestaken-for-granted facts to support apolicy claim. Policy-relevantinformation may be statistical data,experimental findings, experttestimony, common sense, orpolitical judgments.

INFORMATIONA warrant is a reason tosupport a policy claim.Warrants may be economictheories, ethical principles,political ideas, authority, andso forth. Most arguments havemultiple warrants.

WARRANTA qualifier expresses the approximatetruth of a claim, considering the strengthof information, warrants, backings,objections, and rebuttals. Qualifiers maybe stated statistically (p < 0.01) or ineveryday language (“probably,” “notlikely,” “apparently,” “unlikely”).

QUALIFIER

A backing justifies or“backs up” a warrantby providing goodreasons for believingthe warrant.

BACKINGAn objection opposes or challengesa qualifier by identifying specialconditions or exceptions whichreduce confidence in the strengthof the qualifier.

OBJECTION

An objection opposes or challengesa backing by identifying specialconditions or exceptions thatreduce confidence in the truth ofthe backing.

OBJECTION

A rebuttal opposes orchallenges anobjection byidentifying specialconditions orexceptions thatreduce confidencein the truth of theobjection,

REBUTTAL

An objection opposesor challengesinformation byidentifying specialconditions orexceptions thatreduce confidence inthe truth of theinformation.

OBJECTION



the region.” The warrant, which provides a justification for accepting aclaim, answers the question: Considering the information, what reasonsmake the claim true?

Qualifier(Q). The qualifier expresses the degree to which a claim is approxi-mately true, given the strength of the information, warrants, and backings,as well as objections and rebuttals. Although social scientists may statequalifiers in the language of formal probability (p � 0.01 or t � 2.24),ordinary language is the normal mode of qualifying claims with such termsas certainly, absolutely, necessarily, probably, in all likelihood, presumably,apparently, and barring unforeseen circumstances. The qualifier answers thequestion: How strong or credible is the claim? It is primarily throughprocesses of argumentation and debate that policy makers, policy analysts,and other policy stakeholders adjust or even abandon arguments. Suchchanges, when they occur, are motivated by the strength of objections andrebuttals offered by those who have a stake in policies.

Backing (B). The backing is an additional reason to support or “back up” thewarrant. The backing answers the question: Why does the warrant supportthe claim? with a more general reason, assumption, or argument that beginswith because. Different kinds of backings are characteristically employed bymembers of different disciplines and professions. Backings may be scientificlaws, appeals to the authority of experts, or ethical and moral principles.For example, consider the warrant presented earlier: “Ethnic cleansing willbe deterred by air strikes that establish NATO’s credibility in the region.”The backing for warrants advocating the use of coercive force is frequentlyan informal statement of the law of diminishing utility: “The greater thecost of an alternative, the less likely it will be pursued.”

Objection (O). An objection opposes or challenges the information, warrant,backing, or qualifier by identifying special conditions or exceptions thatreduce confidence in the truth of the information, warrant, backing, orqualifier. An objection answers the question: Are there special circum-stances or exceptions that threaten the credibility of the warrant? Analystswho pay attention to objections are more likely to take a critical perspec-tive toward a policy argument, identifying weak or hidden assumptions,anticipating unintended consequences, or questioning possible rebuttals toobjections. Thereby, analysts can be self-critical, challenging their ownassumptions and arguments.

Rebuttal(R). A rebuttal is an objection to an objection. Rebuttals oppose orchallenge objections by identifying special conditions or exceptions thatreduce confidence in the truth of the objection. Rebuttals answer the ques-tion: Are there special circumstances or exceptions that threaten the cred-ibility of the objection? Most policy arguments have objections and rebut-tals, because policy making involves bargaining, negotiation, competition,and compromise among opponents and proponents of policies.

The frames of reference, perspectives, and reasons of policy makers and analystsare found in their underlying warrants, backings, objections, and rebuttals.Therefore, identical policy-relevant information is interpreted in distinctly different



ways. A decrease in crime rates in urban areas may be welcomed by the urban poor,viewed with skepticism by owners of central city businesses, rejected by criminolo-gists who attribute urban crime rates to changes in unemployment and homelessness,and hailed as an achievement by elected officials. By examining contendingarguments and their underlying assumptions, analysts can uncover and criticallyassess reasoning and evidence that otherwise goes unnoticed. Equally important iswhat it brings to analysts themselves—they can probe their own assumptions byexamining the objections, qualifications, and exceptions to their own conclusions.

REVIEW QUESTIONS1. What does it mean to define policy analysis as

a process of inquiry as distinguished from a setof methods?

2. Describe the dynamics of policy-informationalcomponents, policy-analytic methods, andpolicy-informational transformations.

3. Contrast segmented and integrated policyanalysis. Give examples.

4. How does normative decision theory differfrom descriptive decision theory?

5. List some of the key differences between prob-lem solving and problem finding.

6. Contrast retrospective and prospective analy-sis. Which social science disciplines tend to spe-cialize in prospective analysis? Retrospectiveanalysis?

7. Discuss the strengths and limitations of criticalmultiplism.

8. Contrast the “logic-in-use” and the “recon-structed logic” of policy analysis. Provideexamples.

9. How can argumentation mapping assist analyststo become critical thinkers?

DEMONSTRATION EXERCISES1. Scorecards provide a useful overview of the

observed and expected outcomes of differentpolicies. When using the scorecard for monitor-ing and forecasting the outcomes of the twospeed limits (Case 2), the 55 mph speed limitseems preferable to the 65 mph speed limit.Compare the scorecard (Figure C2) with thespreadsheet (Figure C3). Does the comparisonchange your conclusions about the performanceof the 55 mph speed limit? What does this tell us

about the differences between monitoring andevaluation in policy analysis? What are theimplications for the distinctions among differenttypes of policy analysis?

2. Influence diagrams and decision trees are usefulmethods for structuring policy problems. Thediagram and tree displayed in Figure 2 helpidentify policy stakeholders, policy alternatives,uncertain outcomes and events, probabilities ofthese outcomes and events, and valued

CHAPTER SUMMARYThis chapter has provided a framework forpolicy analysis that identifies the role ofpolicy-analytic methods in creating and trans-forming policy-relevant information. The fourregions of this framework call attention tosimilarities and differences among methods ofpolicy analysis and point to the origins of thesemethods in different social science disciplinesand professions, thus clarifying the meaning of

multidisciplinary inquiry. No one methodol-ogy is appropriate for all or most problems.Given the need to choose among methods,methodological choices can be viewed as anoptimization problem involving trade-offs andopportunity costs. The actual work of practic-ing analysts demands critical thinking. Theanalysis of policy arguments is well suited forthis purpose.



outcomes (valued outcomes are objectives).Consider the influence diagram that representsthe problem of energy supply in 1973 and1974, when the OPEC oil embargo posed sig-nificant challenges both to U.S. energy supplyand national security. How does Figure C3 helpus formulate the problem? What do the arrowssuggest about the causes of the energy shortageas well as the causes of the decline in traffic fa-talities? What does the influence diagram sug-gest about the conditions that gave rise to the55 mph speed limit and its effectiveness? Aftercomparing the influence diagram with the deci-sion tree (which is based on the influence dia-

gram), describe why these two problem repre-sentations are good examples of descriptive andnormative decision theory.

3. Create an argument map based on the influ-ence diagram presented in Case 3. Begin withthe following claim: “The United States shouldreturn to the 55 mph speed limit in order toconserve fuel and save lives.” Include in yourmap as many warrants, backings, objections,and rebuttals as you can. Assuming that theoriginal qualifier was certainly, indicatewhether the qualifier changes as we move froma simple, static, uncontested argument to acomplex, dynamic, and contested argument.

BIBLIOGRAPHYCampbell, Donald T. Methodology and Epistemology

for Social Science: Selected Papers. Edited by E. Samuel Overman. Chicago: University ofChicago Press, 1988.

Diesing, Paul. How Social Science Works: Reflectionson Practice. Pittsburgh, PA: Pittsburgh UniversityPress, 1991.

Dunn, William N., and Rita Mae Kelly. Advancesin Policy Studies since 1950. New Brunswick,NJ: Transactions Books, 1992.

Fischer, Frank, and John Forester. The ArgumentativeTurn in Policy Analysis and Planning. Durham,NC: Duke University Press, 1993.

Hawkesworth, Mary E. Theoretical Issues inPolicy Analysis. Albany: State University ofNew York Press, 1988.

Kaplan, Abraham. The Conduct of Inquiry:Methodology for Behavioral Science. SanFrancisco, CA: Chandler, 1964.

Mac Rae, Duncan Jr. The Social Function of SocialScience. New Haven, CT: Yale University Press,1976.

Stone, Deborah. Policy Paradox: The Art ofPolitical Decision Making. Rev Ed. New York:W. W. Norton, 2001.

Toulmin, Stephen R. Return to Reason. Cambridge,MA: Harvard University Press, 2001.

Van Gelder, Tim. “The Rationale for Rationale.”Law, Probability, and Risk 6 (2007): 23–42.

When advanced technologies are used to achievepolicy goals, sociotechnical systems of considerablecomplexity is created. Although it is analyticallytempting to prepare a comprehensive economicanalysis of the costs and benefits of such policies,most practicing analysts do not have the time or

the resources to do so. Given the time constraintsof policy making, many analyses are completed in aperiod of several days to a month, and in mostcases policy analyses do not involve the collectionand analysis of new data. Early on in a project,policy makers and their staffs typically want an

CASE 1 THE GOELLER SCORECARD—MONITORING AND FORECASTINGTECHNOLOGICAL IMPACTS



50E.S. Quade, Analysis for Public Decisions (New York: American Elsevier, 1975), p. 65.

TABLE C1

Scorecard

Social Impacts CTOL VTOL TACV

TRANSPORTATION

Passengers (million miles) 7 4 9Per trip time (hours) 2 1.5 2.5Per trip cost ($) $17 $28 $20Reduced congestion (%) 0% 5% 10%

FINANCIAL

Investment ($ millions) $150 $200 $200Annual subsidy ($ millions) 0 0 90

ECONOMIC

Added jobs (thousands) 20 25 100Added sales ($millions) 50 88 500

COMMUNITY

Noise (households) 10 1 20Added air pollution (%) 3% 9% 1%Petroleum savings (%) 0% –20% 30%Displaced households 0 20 500Taxes lost ($millions) 0 0.2 2Landmarks destroyed None None Fort X

DISTRIBUTIONAL

Low-income trips (%) 7% 1% 20%Low-income householdNoise annoyance (%) 2% 16% 40%

SOurce: Goeller (1974); Quade, Analysis for Public Decisions (1975), p. 60.

NOte: Conventional takeoff and landing aircraft (CTOL); vertical takeoff and landing aircraft (VTOL);tracked air-cushion vehicle (TACV).

overview of the problem situation and the potentialimpacts of alternative policies. Under thesecircumstances, the scorecard is appropriate.

The Goeller scorecard, named after Bruce Goellerof the RAND Corporation, is appropriate for thispurpose.Table C1 shows the impacts of alternativetransportation systems. Some of the impacts involvetransportation services used by members of thecommunity, whereas others involve impacts on low-

income groups. In this case, as Quade observes, thelarge number of diverse impacts are difficult tovalue in dollar terms, making a benefit-cost analysisimpractical and even impossible.50 Other impactsinvolve financial and economic questions such asinvestments, jobs created, sales, and tax revenues.Other impacts are distributional because theyinvolve the differential effects of transportation. �

CASE 2 THE CASE 3


In 1972 and 1973, the United States and otherpetroleum-dependent countries experienced thefirst of several oil crises precipitated by adramatic increase in the price of crude oil by theOrganization of Petroleum Exporting Countries(OPEC).The response of American and Europeanleaders was to adopt maximum speed limits of 55mph and 90 kph, respectively. In the UnitedStates, the National Maximum Speed Limit(NMSL) was designed to reduce the consumptionof gasoline by requiring that all vehicles oninterstate highways travel at a maximum of 55mph, a speed that would maximize fuel efficiencyfor most vehicles.

Soon after the implementation of the 55 mphspeed limit, it was discovered that the new policynot only reduced fuel consumption, but apparentlycaused a dramatic decline in traffic fatalities andinjuries as well. Therefore, long after the OPEC oilcrisis was over, the speed limit was retained,although it was no longer needed to respond to theenergy crisis that prompted its passage in 1973.Indeed, the 55 mph speed limit was retained formore than 20 years until it was officially repealedin November 1995.51

Heated debates preceded the repeal. SenatorJohn C. Danforth of Missouri, an influentialadvocate of the policy, argued that the repeal wouldsave one minute per day per driver but result in an

additional 600 to 1,000 deaths. The Washington Post

and the New York Times joined the opposition,reporting that, although fatalities would surely rise,the savings in time was trivial. Later, Secretary ofTransportation Pena announced that the Clintonadministration was firmly opposed to abandoning thespeed limit.

This was the right moment for an evaluation ofthe benefits and costs of the NMSL. A spreadsheet isa simple but powerful tool for doing so. Thescorecard, as we saw in Case 1, is a useful tool formonitoring and forecasting impacts when benefit-cost analysis is not feasible or desirable. On thescorecard, policy alternatives are arrayed in columnsalong the top of the matrix and policy impacts arelisted in each row. Spreadsheets, by contrast, areappropriate and useful for prescribing preferredpolicies and evaluating their outcomes. Spreadsheetsdisplay the benefits and costs of observed orexpected policy outcomes, creating informationabout policy performance as well as preferredpolicies (see Figure 1).

Table C2 displays a spreadsheet used toevaluate the effects of the 55 mph speed limitat the end of 1974, one year after the policy wasimplemented. To show the differences betweenthe spreadsheet and the scorecard, Table C2also displays the same information as a scorecard. �

51On April 2, 1987, Congress enacted the Surface Transportation and Uniform Relocation AssistanceAct, permitting 40 states to experiment with speed limits up to 65 mph.

CASE 2 SPREADSHEET—EVALUATING THEBENEFITS AND COSTS OF ENERGY POLICIES



(continued)

TABLE C2

Scorecard and Spreadsheet

(a) Scorecard

65 MPHOUTCOMES (Base Case)* 55 MPH

Fatalities 54,052 45,196Miles traveled (billions) 1,313 1,281Hours driving (billions) 20.2 21.9Gallons fuel consumed (billions) 46.8 43.3Fuel efficiency (mpg) 14.9 16.1Traffic citations (millions) 5,711 7,425Property damage (million cases) 25.8 23.1

*The base case is the policy against which the new policy is compared.

(b) Spreadsheet

OBJECTIVES 65MPH 55MPH Difference Value $ Billions

I Fatalities (000s) 54.1 45.2 8.856 $240,000.00 $ 2.13

II Hours driving (billions) 20.2 21.9 �1.7 5.05 �8.59

III Gallons fuel consumed (billions)

46.8 43.3 3.5 0.53 1.86

IV Traffic citations (000s) 5,711 7,425 �1,714 3.94 �0.0068

V Property damage cases (000s)

25,800 23,100 2,700 363.00 0.98

Benefits (I � III � V) 4.97

Costs (II � IV) �8.60

Net Benefits (B–C) $ �3.63

Along with other policy-analytic methods discussedearlier in this chapter (Figure 1), the influence

diagram and decision tree are useful tools forstructuring policy problems.52 The influence

diagram (Figure C3) displays the policy, theNational Maximum Speed Limit, as a rectangle.A rectangle always refers to a policy choice ordecision node, which in this case is the choice

52The diagram and tree were created with the Decision Programming Language (DPL), which is available from SyncopationSoftware at http://www.syncopation.com. Educational, professional, and commercial versions of DPL 7.0 are available.

CASE 3 THE INFLUENCE DIAGRAM ANDDECISION TREE—STRUCTURING PROBLEMS OFENERGY POLICY AND INTERNATIONAL SECURITY



TravelTime

FuelUsed

NetBenefits

Injuries

Fatalities

MilesTraveled

Employment

RecessionOPEC

Oil Crisis

NMSL

FIGURE C3Influence diagram and decision tree

between adopting and not adopting the nationalmaximum speed limit of 55 mph. To the right andabove the decision node are uncertain events,represented as ovals, which are connected to thedecision node with arrows showing how the speedlimit affects or is affected by them. The rectangleswith shaved corners represent valued policyoutcomes or objectives. The objectives are to lowerfuel consumption, reduce travel time, reduceinjuries, and avert traffic fatalities. To the right ofthe objectives is another shaved rectangle, whichdesignates the net benefits (benefits less costs) ofthe four objectives.The surprising result of usingthe influence diagram for problem structuring isthe discovery of causally relevant economic events,

such as the recession and unemployment, whichaffect miles driven, which in turn affect all fourobjectives. The “root cause” appears to be theOPEC oil embargo.

The decision tree is another representation of theinfluence diagram. Whereas the influence diagramshows how policy choices and uncertain events affectthe achievement of objectives, the decision tree displaysthe monetary value of these objectives. In this abridgedand simplified decision tree, there are two branchesthat represent the alternatives but also the OPEC oilembargo, the recession, the costs of miles traveled, andthe dollar benefits of reducing fatalities. The boldedbranches show the events with the greatest likelihoodof occurring or that already have occurred. �

YesYes

OPECOil Crisis

RenewNMSL

Recession FatalitiesMilesTraveled

Yes

Down 10% (32 billion miles)

$2.77 per 100 Miles Traveled

$2,77 per 100 Miles TraveledDown 3%

$2.77 per 100 Miles TraveledUnchanged

Down 20% (8,900 fatalities)

$240,000 per Fatality Averted

$240,000 per Fatality AvertedDown 5%

$240,000 per Fatality AvertedDown 10%

NoNo No

(b) Decision Tree

(a) Influence Diagram


CASE 4 THE ARGUMENT MAP—PROBLEMSTRUCTURING IN NATIONAL DEFENSE ANDTRANSPORTATION POLICY

The role of causal arguments in transforming policy-relevant information into policy claims may beillustrated by Allison’s well-known study of foreignpolicy decision making during the Cuban missilecrisis of October 1962.53 Showing how differentexplanatory models yield different conclusions,Allison argues that government policy analysts thinkabout problems of foreign policy in terms of implicitconceptual models that shape their thought; mostanalysts explain the behavior of governments interms of a model that assumes the rationality ofpolitical choices (rational actor model); alternativemodels, including those that emphasizeorganizational processes (organizational process model)and bureaucratic politics (bureaucratic politics model),provide a basis for improved explanations.

In 1962, the policy alternatives open to theUnited States ranged from no action and diplomaticpressure to secret negotiations, invasion, surgical airstrikes, and blockade. Among the several claimsmade at the time of the Cuban missile crisis, let usconsider the policy actually adopted by the UnitedStates: “The United States should blockade Cuba.”In this case, the policy-relevant information (I) is“The Soviet Union is placing offensive missiles inCuba.” The warrant states that “the blockade willforce the withdrawal of missiles by showing theRussians that the United States is determined to useforce.” In providing reasons to accept the warrant,the backing (B) supports the warrant by stating that“an increase in the cost of an alternative reduces thelikelihood of that alternative being chosen.”54 Thebacking (B) represents a general theoretical

proposition, or law, within the rational policy model.After the objection (O) has successfully challengedthe warrant, the qualifier (Q) changes from absolutely

to doubtful.

Allison’s account shows how the use of multiplecompeting explanations can facilitate criticalthinking. The use of multiple competing modelsmoves the analysis from a simple uncontestedargument (Figure C4.1) to a new argument that iscomplex, contested, and dynamic (Figure C4.2).Thischange occurs because a serious objection has beenraised about the warrant and the backing of theclaim. The objection states: “But Soviet leaders mayfail to convince their naval units to depart fromestablished organizational routines.” The warrantfor this objection is: “The bulk of research onorganizations shows that major lines oforganizational behavior tend to be straight. Behaviorat time t+1 differs little from behavior at time t.55

The blockade will not work.” The warrant for theobjection is again a general proposition or lawwithin the organizational process model, otherwiseknown as the disjointed incremental theory of policychange.

Simple uncontested maps of arguments aboutthe 55 mph speed limit are presented alongside ofthe arguments about the Cuban missile crisis(Figures C4.1a and C4.1b).The comparisons showthat the simple argument maps represent uncritical

thinking. By contrast, the complex, dynamic, andcontested maps of the same crises (Figures C4.2aand C4.2b) illustrate what is meant by critical

thinking. �

53Graham T. Allison, “Conceptual Models and the Cuban Missile Crisis,” American Political ScienceReview 3002, no. 3 (1969): 689–718.54Allison, “Conceptual Models,” p. 694.55Ibid., p. 702.

(continued)



FIGURE C4.1Simple argument maps are static and uncontested

Reliable intelligencereports confirm that theSoviet Union is placingoffensive missiles in Cuba.

IA blockade will show Sovietleaders that the States meansbusiness and is prepared touse force.

WVeryprobably

Qsupports supports

The United States should forcethe Soviet Union to withdraw themissiles by blockading Cuba.

C

(a) The Cuban Missile Crisis—Simple Uncontested Argument

There was a declineof 8,300 traffic fatalitiesin the year following theimplementation of the55 MPH speed limit.

IThe speed limit wasresponsible for thedecline in trafficfatalities.

WCertainlyQ

support

Congress shouldreinstate the 55MPH speed limit.

C

(b) The 55 mph Speed Limit—Simple Uncontested Argument




C

Because an increasein the cost of an actionreduces the likelihoodthat it will be taken.

BBut considering theobjections, the claimis doubtful.

O

Given that reliable intelli-gence reports confirm thatthe Soviet Union is placingoffensive missiles in Cuba.

IBecause a blockade will showSoviet leaders that the Statesmeans business and isprepared to use force.

WAbsolutelyQ weakly

support

weaklysupport

stronglyopposes

But Soviet leaders may fail toconvince their naval units to depart from established routines.

Ostronglyopposes

Because the bulk of research onorganizations shows that major lines ofbehavior tend to be straight: Behavior attime t + 1 differs little from behavior attime t. The blockade will not work.

Wstronglysupports

(a) The Cuban Missile Crisis—Dynamic Contested Argument

(continued)



FIGURE C4.2Complex argument maps are dynamic and contested

Congress shouldreinstate the 55MPH speed limit.

C

Since it is obvious that the decline isdue to the speed limit. No other factorscan explain the decline, which is thelargest in U.S. history.

BBut this is not certain at all.

O

Given that there was a decline of8,300 traffic fatalities in the yearfollowing the implementationof the 55 MPH speed limit.

IBecause the speedlimit was responsiblefor the decline intraffic fatalities.

WCertainlyQ

support

support opposes

But most of the decline was due to the 1974recession, the rise in unemployment, the doubling of gasoline prices, and the consequent sharp decline in miles driven, which drasticallyreduced the exposure to accidents.

Oopposes

(b) The 55 mph Speed Limit—Simple Uncontested Argument


O B J E C T I V E S


Policy Analysis in thePolicy-Making Process

� Understand policy analysis as anintellectual activity embedded in apolitical process.

� Explain the historical developmentof policy analysis as a response topractical problems and crises.

� Describe policy making as acomplex, nonlinear process

composed of multiple phasesordered in time.

� Distinguish competing explanationsof policy change.

� Contrast potential and actual usesof analysis.

� Describe the composition, scope, andexpected effects of information use.

Policy analysis creates, critically assesses, and communicates informationabout and in the policy-making process.1 This distinction between about andin marks an essential difference between policy analysis, on one hand, and

political science and economics, disciplines that specialize in developing and testingdescriptive and normative theories of policy making. Although some members ofthese disciplines do work on concrete problems facing policy makers, most aremotivated by incentive systems that demand the production of knowledge for itsown sake. By contrast, policy analysts work under incentives designed to promotethe creation and application of practical knowledge—that is, knowledge about

1Harold D. Lasswell, A Pre-view of Policy Sciences (New York: American Elsevier Publishing, 1971),pp. 1–2. Information about refers to “systematic, empirical studies of how policies are made and putinto effect,” whereas information in refers to the fact that “the realism of a decision depends in part onaccess to the stock of available information.”

From Chapter 2 of Public Policy Analysis, Fifth Edition.William N. Dunn. Copyright © 2012 by PearsonEducation, Inc. All rights reserved.


what works.2 Although policy analysis draws on social science methods, theories,and substantive findings, the aim is to solve practical problems. This requires thecommunication and use of information in as well as about the policy-makingprocess.3

THE HISTORICAL CONTEXTPolicy analysis is as old as civilization itself. It includes diverse forms of inquiry,from mysticism and the occult to modern science. Etymologically, the term policycomes from the Greek, Sanskrit, and Latin languages. The Greek polis (city-state)and Sanskrit pur (city) evolved into the Latin politia (state) and later into the MiddleEnglish policie. The latter referred to the conduct of public affairs or the administra-tion of government. The etymological origins of policy are the same for two otherimportant words: police and politics. These multiple connotations are found inGermanic and Slavic languages, which have only one word (Politik, politika, respec-tively) to refer to both policy and politics. This is among the reasons for the porousboundaries among political science, public administration, and policy analysis, andthe resultant confusion about their substance and aims.

Early OriginsThe term policy analysis need not be restricted to its contemporary meaning, whereanalysis refers to breaking problems into basic elements or parts, much as wedisassemble a clock or machine. This is the sense of “analysis” when it is argued thatdecision problems may be decomposed into alternatives, outcomes, and objectives.A related view is that policy analysis is a collection of quantitative techniques usedby systems analysts, decision analysts, and economists to examine the likelihoodand utility of policy outcomes.

Understood in a wider sense, however, policy analysis may be seen to haveemerged at a point in the evolution of human societies when practical knowledgewas consciously cultivated, thereby prompting an explicit and self-reflective exami-nation of links between knowledge and action. The development of specializedprocedures for analyzing policies was related to the emergence of urban civilization

2On differences between intellectual and practical knowledge see the treatise by Fritz Machlup,Knowledge: Its Creation, Distribution, and Economic Significance. Vol. 1: Knowledge and KnowledgeProduction (Princeton, NJ: Princeton University Press, 1980). Machlup also discusses spiritual, pastime,and unwanted knowledge.3Carol H. Weiss’s Social Science Research and Decision Making (New York: Columbia University Press,1980) is a comprehensive synthesis of research and theory on the uses of social science research by policymakers. Other syntheses include William N. Dunn and Burkart Holzner, “Knowledge in Society: Anatomyof an Emerging Field,” Knowledge in Society (later titled Knowledge and Policy) 1, no. 1 (1988): 1–26;and David J. Webber, “The Distribution and Use of Policy Information in the Policy Process,” in Advancesin Policy Studies since 1950, ed. William Dunn and Rita Mae Kelly (New Brunswick, NJ: Transaction,1991), pp. 415–41. Journals that focus on these problems include Science Communication (formerlyKnowledge: Creation, Diffusion, Utilization), Knowledge and Policy: The International Journal ofKnowledge Transfer and Utilization, and Science, Technology and Human Values.

Policy Analysis in the Policy-Making Process



out of scattered and largely autonomous tribal and folk societies.4 As a specializedactivity, policy analysis followed changes in social and, above all, political organiza-tion that accompanied new production technologies and stable patterns of humansettlement.

One of the earliest recorded efforts to cultivate policy-relevant knowledgeoccurred in Mesopotamia, in what is now the Basra region of southern Iraq. In theancient Mesopotamian city of Ur, one of the first legal codes was produced in thetwenty-first century B.C., some two thousand years before Aristotle (384–322 B.C.),Confucius (551–479 B.C.), and Kautilya (ca. 300 B.C.) produced their classic treatiseson government and politics. In the eighteenth century B.C., the ruler of Babylon, withthe assistance of professionals whom we would now call policy analysts, created theCode of Hammurabi. The Code was designed to establish a unified and just publicorder in a period when Babylon was in transition from a small city-state to a largeterritorial state. The Hammurabian Code, a set of policies that parallel Mosaic laws,reflected the economic and social requirements of stable urban settlements whererights and obligations were defined according to social position. The Code coveredcriminal procedures, property rights, trade and commerce, family and maritalrelations, physicians’ fees, and what we now call public accountability.5

The early Mesopotamian legal codes were a response to the growing complexityof fixed urban settlements, where policies were needed to regulate the distribution ofcommodities and services, organize record keeping, and maintain internal securityand external defense. A growing consciousness of relations between knowledge andaction fostered the growth of educated strata that specialized in the production ofpolicy-relevant information. These “symbol specialists,” as Lasswell called them,were responsible for policy forecasting; for example, they were expected to foreseecrop yields at the onset of the planting season, or predict the outcomes of war.6

Because analysts used mysticism, ritual, and the occult to forecast the future, theirmethods were unscientific by present-day standards. Although such procedureswere based in part on evidence acquired through experience, any reasonable defini-tion of science requires that knowledge claims be assessed against observations thatare independent of the hopes of analysts, or of those who hire them.7 Then as now,policy-relevant knowledge was ultimately judged according to its success (or failure)in shaping better policies, not simply because special methods were used to produceit. Even the ancients seemed to know what some contemporary analysts forget—when methods are used for ritualistic purification, political persuasion, andsymbolic legitimation, analysts and their clients eventually must face the decisivetest of whether the methods produce reliable results.8 Although statements such as

4Lasswell, A Pre-view of Policy Sciences, pp. 9, 13.5The Code of Hammurabi, trans. Robert F. Harper (Chicago: University of Chicago Press, 1904).6Lasswell, A Pre-view of Policy Sciences, p. 11.7Donald T. Campbell, “A Tribal Model of the Social System Vehicle Carrying Scientific Information,”in Methodology and Epistemology for Social Science: Selected Papers, ed. E. Samuel Overman(Chicago: University of Chicago Press, 1988), pp. 489–503.8Edward A. Suchman, “Action for What? A Critique of Evaluative Research,” in Evaluating ActionPrograms, ed. Carol H. Weiss (Boston, MA: Allyn and Bacon, 1972), p. 81; and Martin Rein andSheldon H. White, “Policy Research: Belief and Doubt,” Policy Analysis 3, no. 2 (1977): 239–71.



“drug policy is based on good science” are in vogue, the invocation of “science”in these contexts may represent little more than ritualistic purification.

In India, Kautilya’s Arthashastra, written in the fourth century B.C., is a system-atic guide to policy making, statecraft, and government administration. TheArthashastra synthesized much that had been written up to that time on materialsuccess, or what we now call economics. Kautilya, an adviser to the MauryanEmpire in northern India, has been compared to Plato (427–327 B.C.), Aristotle(384–322 B.C.), and Machiavelli (1469–1527). In addition to their work as politicaltheorists, all were deeply involved in the practical aspects of policy making. Platoserved as adviser to the rulers of Sicily, whereas Aristotle tutored Alexander ofMacedonia from the time Alexander was fourteen years old until he ascended thethrone at the age of twenty. Although Aristotle, like many social and behavioralscientists, found practical politics repugnant, he seemed to have accepted the assign-ment because he wanted to bring knowledge to bear on policy issues of the day. Inthis respect, he followed his teacher Plato, who said that good government wouldnot occur until philosophers were kings, or kings philosophers. The opportunity toinfluence policy by instructing the heir apparent was an offer that in goodconscience he could not refuse.9

These are examples of preeminent individual producers of specialized knowledge,not of entire classes of educated persons who later would influence policy makingin Europe and Asia. In the Middle Ages, the gradual expansion and differentiationof urban civilization brought with it an occupational structure that facilitated thedevelopment of specialized knowledge. Princes and kings recruited policy specialists toprovide advice and technical assistance in areas where rulers were least able to makeeffective decisions: finance, war, and law. German sociologist Max Weber describedthe development of a class of educated policy specialists as follows:

In Europe, expert officialdom, based on the division of labor, has emerged in agradual development of half a thousand years. The Italian cities andseigniories were the beginning, among the monarchies, and states of theNorman conquerors. But the decisive step was taken in connection withthe administration of the finances of the prince. . . . The sphere of financecould afford least of all a ruler’s dilettantism—a ruler who at that time wasstill above all a knight. The development of war technique called forth theexpert and specialized officer; the differentiation of legal procedure calledforth the trained jurist. In these three areas—finance, war, and law—expertofficialdom in the more advanced states was definitely triumphant during thesixteenth century.10

The growth of expert officialdom (what Weber called “professional politicians”)assumed different forms in different parts of the world. In medieval Europe, India,China, Japan, and Mongolia, the clergy were literate and therefore technically useful.

9J. A. K. Thompson, The Ethics of Aristotle: The Nichomachean Ethics Translated (Baltimore, MD:Penguin Books,1955), p. D. 11.10Max Weber, “Politics as a Vocation,” in From Max Weber: Essays in Sociology, ed. Hans C. Gerthand C. Wright Mills (New York: Oxford University Press, 1946), p. 88.



Christian, Brahmin, Buddhist, and Lamaist priests, much like some modern socialand behavioral scientists, earned a reputation for impartiality and disinterestednessinsofar as they stood above practical politics and temptations of political power andeconomic gain. Educated men of letters—whose modern counterpart is the specialpresidential adviser—influenced policy making until court nobles, who later came todominate the political and diplomatic service, replaced them. In England, pettynobles and urban rentiers (investors) were recruited without compensation tomanage local governments in their own interest. Jurists trained in Roman law andjurisprudence had a strong influence on policy making, particularly in ContinentalEurope. They were largely responsible for the transformation of the late medievalstate and the movement toward modern government.

The age of the Industrial Revolution was also that of the Enlightenment, aperiod in which a belief in human progress through science and technology becamean ever more dominant theme among policy makers and their advisers. The develop-ment and testing of scientific theories of nature and society gradually came to beseen as the only objective means for understanding and solving social problems. Forthe first time, policy-relevant knowledge was produced according to the canons ofempiricism and the scientific method.

The Nineteenth-Century TransformationIn nineteenth-century Europe, producers of policy-relevant knowledge began tobase their work on the systematic recording of empirical data. Earlier, philosophersand statesmen had offered systematic explanations of policy making and its role insociety. Yet for several thousand years, there was an essential continuity in methodsfor investigating and solving human social, economic, and political problems. Ifevidence for a particular point of view was provided, it was typically based on appealsto authority, ritual, or philosophical doctrine. What was new in the nineteenthcentury was a basic change in the procedures used to understand society and itsproblems, a change reflected in the growth of empirical, quantitative, and policy-oriented research.11

The first censuses were conducted in the United States (1790) and England(1801). It was at this time that statistics (“state arithmetic”) and demographybegan to develop as specialized fields. The Manchester and London StatisticalSocieties, established in the 1830s, helped shape a new orientation toward policy-relevant knowledge. Organized by bankers, industrialists, and scholars, the soci-eties sought to replace traditional ways of thinking about social problems withempirical analyses of the effects of urbanization and unemployment on the livesof workers and their families. In the Manchester Society, an enthusiasm forquantification was coupled with a commitment to social reform, or “progress ofsocial improvement in the manufacturing population.”12 The London Society,

11Daniel Lerner, “Social Science: Whence and Whither?” in The Human Meaning of the Social Sciences,ed. Daniel Lerner (New York: World Publishing, 1959), pp. 13–23.12Nathan Glazer, “The Rise of Social Research in Europe,” in The Human Meaning of the SocialSciences, ed. Lerner, p. 51.



under the influence of Thomas Malthus (1766–1834) and other academics, tooka more disinterested approach:

The Statistical Society will consider it to be the first and most essential rule of itsconduct to exclude carefully all opinions from its transactions and publications—to confine its attention rigorously to facts—and, as far as it may be found possible,to facts which can be stated numerically and arranged in tables.13

The London and Manchester societies used questionnaires to carry out studies,and paid “agents” were the counterpart of today’s professional interviewer. Therewere similar developments in France, Germany, and the Netherlands.

A preeminent contributor to the methodology of social statistics and surveyresearch was Adolphe Quetelet (1796–1874), a Belgian mathematician and astronomerwho was the major scientific adviser to the Dutch and Belgian governments.14 Queteletaddressed most topics on survey design and analysis found in contemporary texts:questionnaire design; data collection, analysis, and interpretation; data organizationand storage; and identification of conditions under which data are collected. Inthe same period, Frederic Le Play (1806–82) wrote Les Ouvriers Europeans[The European Workers], a detailed empirical investigation of family income andexpenditures of European workers in several countries. In Germany, Ernst Engel(1821–96) sought to derive laws of “social economics” from empirical data expressedin statistical form.

In England, the work of Henry Mayhew and Charles Booth, who studied thelife and employment conditions of the urban poor in natural (what we now call“field”) settings, is representative of the new empirical approach to the study ofsocial problems. Mayhew’s London Labour and the London Poor (1851) describedthe lives of the laborers, peddlers, performers, and prostitutes who comprisedLondon’s urban underclass. In writing Life and Labour of the People in London(1891–1903), Booth employed school inspectors as key informants. Using what wenow call participant observation, Booth lived among the urban poor, gaining first-hand experience of actual living conditions. A member of the Royal Commission onthe Poor Law, he was an important influence on the revision of policies on old-agepensions. Booth’s work also served as something of a model for policy-orientedresearch in the United States, including the Hull House Maps and Papers (1895)and W. E. B. Dubois’s The Philadelphia Negro (1899), both of which sought todocument the scope and severity of poverty in urban areas.

The nineteenth-century transformation was not the result of declarations ofallegiance to canons of logical empiricism and the scientific method. Declarations tothis effect did not and could not occur until the next century, when Vienna Circlephilosophers engaged in the logical reconstruction of physics to propose formalprinciples and rules of successful scientific practice (few natural or social scientistshave actually followed these principles and rules). The transformation came, rather,from the uncertainty accompanying the shift from agrarian to industrial societies,

13Ibid., pp. 51–52.14A significant history of statistics and statisticians is Stephen M. Stigler, The History of Statistics: TheMeasurement of Uncertainty before 1900 (Cambridge, MA: Harvard University Press, 1990).



a shift that preceded the Industrial Revolution. Later, industry and industrializedscience required a politically stable order to operate efficiently. Political stability wasassociated with profound social instability.15

Thus, for the most part science and technology were not responsible for thegrowth of newly centralized systems of political control. Although science andtechnology contributed to problems of a newly uprooted, uneducated, and displacedclass of urban workers and their families, rulers and dominant social groups valuedpolicy-oriented research as a means to achieve political and administrative control.In the sphere of factory production, for example, the political organization of workin assembly lines preceded scientific and technological developments that laterculminated in efficiency-enhancing machinery and the specialization of tasks.16 Inthe sphere of public policy, we see a parallel development. Methods of empirical,quantitative, and policy-oriented analysis were a product of the recognition bybankers, industrialists, politicians, and the Victorian middle class that older methodsfor understanding the natural and social world were no longer adequate. The keyquestions of the day were practical: How much did members of the urban prole-tariat need to earn to maintain themselves and their families? How much did theyhave to earn before there was a taxable surplus? How much did they have to savefrom their earnings to pay for medical treatment and education? How much shouldcapitalist owners and the state invest in day care facilities so that mothers mightput in an effective day’s work? How much investment in public works projects—sanitation, sewage, housing, roads—was required to maintain adequate publichealth standards, not only to maintain a productive workforce but also to protectthe middle and upper classes from infectious diseases cultivated in urban slums?

The Twentieth CenturyAn important feature of the twentieth century, as compared with the nineteenth, isthe institutionalization of the social sciences and professions. Twentieth-centuryproducers of policy-relevant knowledge were no longer the heterogeneous group ofbankers, industrialists, journalists, and academics who guided the early statisticalsocieties and other institutions of policy research. They were graduates with firstand advanced degrees in policy-relevant disciplines and professions who, along withprofessors, occupied important positions in governments or worked as consultantsor researchers under grants and contracts. In background, experience, and motiva-tion, they were members of established professions that, more or less, were guidedby commonly accepted scientific and professional norms.

The new professionals played an active role in the administration of WoodrowWilson, particularly during World War I. Later, under the Republican administrationof Herbert Hoover, social scientists carried out two major social surveys, RecentEconomic Trends and Recent Social Trends. The largest influx of social scientists into

15J. H. Plumb, The Growth of Political Stability in England, 1675–1725 (Baltimore, MD: PenguinBooks, 1973), p. 12.16Stephen A. Marglin, “What Do Bosses Do? The Origins and Functions of Hierarchy in CapitalistProduction,” Review of Radical Political Economy 6, no. 2 (1974): 33–60.



government came, however, with Franklin Roosevelt’s New Deal. Large numbers ofsocial scientists staffed the numerous new agencies established during the Rooseveltadministration (e.g., National Recovery Administration, Work Projects Administration,Public Works Administration, Securities and Exchange Commission, and FederalHousing Administration).

The primary function of social scientists in the 1930s was to investigate policyproblems and broad sets of potential solutions, and not, as in later periods, toemploy economic modeling, decision analysis, or policy experimentation toidentify and select specific solutions to problems. The Roosevelt administration’sNational Planning Board (later the National Resources Planning Board), a majorityof whose members were professional social scientists, provides a good illustrationof the approach to policy questions characteristic of the 1930s. The board wasconceived as “a general staff gathering and analyzing facts, observing theinterrelation and administration of broad policies, proposing from time to timealternative lines of national procedure, based on thorough inquiry and matureconsideration.”17 This same general orientation toward problems was evidentamong economists working for the Department of Agriculture, political scientistsinvolved in the reorganization of the executive branch, and anthropologistsconducting studies for the Bureau of Indian Affairs. Social scientists also contributedto methodological innovations; for example, the Department of Agriculture led infurther developing the sample survey as a research tool and instrument of govern-ment census policy.18

World War II and the postwar readjustment that followed provided social scien-tists with opportunities to demonstrate their value in solving practical problems.Interwar achievements in the area of survey research had laid a foundation for theuse of interviews by the Office of War Information, the War Production Board, andthe Office of Price Administration. Military and civilian agencies relied on socialscientists to investigate problems of national security, social welfare, defense, warproduction, pricing, and rationing. The activities of agencies such as the Office ofStrategic Services were continued after the war by the Office of Naval Research; bythe Department of the Air Force; and, later, by the Research and Development Board(subsequently RAND) of the Department of Defense, and the Central IntelligenceAgency. Special research institutes were established by the federal government,including the Operations Research Office at Johns Hopkins University and theHuman Resources Research Office at George Washington University. Among theseminal contributions to policy research in this period was The American Soldier(1950), a four-volume study produced by many of the most able applied social scien-tists in the country. The director of the Army Morale Division commissioned thislarge-scale project in 1941, under the general direction of sociologist Samuel Stouffer.The project is significant, not only because of its scale but also because it was part ofan emerging pattern of extensive governmental support for policy research andanalysis. Military policy makers turned to the social researcher, not only for facts but

17Gene Lyons, The Uneasy Partnership: Social Science and the Federal Government in the TwentiethCentury (New York: Russell Sage Foundation, 1969), p. 65.18Harry Alpert, “The Growth of Social Research in the United States,” in The Human Meaning of theSocial Sciences, ed. Lerner, pp. 79–80.



also for causal inferences and conclusions that would affect the lives of millions oftroops.19 This large research program contributed to the development and refine-ment of multivariate analysis and other quantitative techniques that are now widelyused by researchers in social science disciplines.

After World War II, the first systematic effort to develop an explicit policyorientation within the social and behavioral sciences was The Policy Sciences:Recent Developments in Scope and Method (1951), edited by political scientistsDaniel Lerner and Harold D. Lasswell. The “policy sciences,” as stated by Lasswellin the introduction, are not confined to theoretical aims of science; they also have afundamentally practical orientation. Moreover, their purpose is not simply toprovide a basis for making efficient decisions but also to provide knowledge“needed to improve the practice of democracy. In a word, the special emphasis isupon the policy sciences of democracy, in which the ultimate goal is the realizationof human dignity in theory and fact.”20

The systematic study of public policy also grew out of public administration,then a field within political science. In 1937, Harvard University established theGraduate School of Public Administration, which focused in part on public policy.In the late 1940s, an interuniversity committee was established to develop publicpolicy curricular materials, a major product of which was Harold Stein’s PublicAdministration and Policy Development: A Case-Book (1952). The interuniversitycommittee, composed of professors and practitioners of public administration,speaks for the close relationship between policy analysis and public administrationbefore and after World War II.21

The impetus for developing methods and techniques of policy analysis—asdistinguished from its theory and methodology—did not originate in political scienceor public administration. The technical side of policy analysis rather grew out ofengineering, operations research, systems analysis, applied mathematics, and to a lesserextent applied economics. Most of those responsible for developing methods andtechniques had received their formal training outside the social sciences. World War IIhad prompted the involvement of specialists whose orientation toward policy wasprimarily analytical, in the narrow sense of that term. The idea of “analysis” came tobe associated with efforts to separate or decompose problems into their fundamentalcomponents, for example, decomposing problems of national defense into policyalternatives (nuclear warheads, manned bombers, conventional ground troops) whoseconsequences for the attainment of policy objectives could be estimated. This analycen-tric perspective22 tends to preclude or restrict concerns with political, social, andadministrative aspects of public policy, for example, concerns with the political feasibilityof alternatives or their implications for democratic processes. Although the analycen-tric turn represents a movement away from the multidisciplinary vision of Lasswell’s

19Howard E. Freeman and Clarence C. Sherwood, Social Research and Social Policy (Englewood Cliffs,NJ: Prentice Hall, 1970), p. 25.20Harold D. Lasswell, “The Policy Orientation,” in The Policy Sciences: Recent Developments in Scope andMethod, ed. Daniel Lerner and Harold D. Lasswell (Stanford, CA: Stanford University Press, 1951), p. 15.21H. George Frederickson and Charles Wise, Public Administration and Public Policy (Lexington, MA:D. C. Heath, 1977).22Allen Schick, “Beyond Analysis,” Public Administration Review 37, no. 3 (1977): 258–63.



policy sciences,23 it has supplied more systematic procedures—from decision analysisto applied microeconomics—for selecting policy alternatives.24

The analycentric turn was accompanied by the growing influence of nonprofitresearch organizations (“think tanks”) such as the RAND Corporation, which fosteredthe spread of systems analysis and related techniques to government agencies and theacademic community.25 The development of program-planning-budgeting systems(PPBS) was due in large measure to the efforts of operations researchers and economistsworking under Charles Hitch at RAND. The RAND group wanted to find out “howthe country could ‘purchase’ national security in the most efficient manner—how muchof the national wealth should be devoted to defense, how the funds allocated to defenseshould be distributed among different military functions, and how to assure the mosteffective use of these funds.”26 Although PPBS was introduced into the Department ofDefense in 1965 and was later mandated for use in all federal agencies, it was difficultto implement. After 1971 it became discretionary and soon fell into disuse. Despitemixed conclusions about its success as a tool of policy analysis, PPBS did capture theattention of government and university analysts who value systematic procedures forselecting and evaluating policy alternatives.27

The analycentric turn has been offset to some extent by the rapid growth of privatefoundations whose mission is to support traditional lines of research in the socialsciences and humanities. More than three-fourths of these foundations wereestablished after 1950.28 In the same period, the federal government began to set asidefunds for applied and policy-related research in the social sciences, although the naturalsciences continued to receive the bulk of government research support. Although in1972 the social sciences received approximately 5 percent of all available federalresearch funds, in the period 1980–90 funding for applied and basic research in thesocial sciences fell by approximately 40 percent in constant dollars.29 At the same time,it is noteworthy that more than 95 percent of all research funded by governmental,nonprofit, and private organizations is applied research on practical problems.

By the 1970s, many social science disciplines had established institutions expresslycommitted to applied and policy-related research. These include the Policy StudiesOrganization (political science), the Society for the Study of Social Problems (sociology),and the Society for the Psychological Study of Social Issues (psychology). Each has its

23Peter de Leon, Advice and Consent: The Development of the Policy Sciences (New York: Russell SageFoundation, 1988), ch. 2.24Martin Greenberger, Matthew A. Crenson, and Brian L. Crissey, Models in the Policy Process: PublicDecision Making in the Computer Era (New York: Russell Sage Foundation, 1976), pp. 23–46.25Bruce L. R. Smith, The Rand Corporation: A Case Study of a Nonprofit Advisory Corporation(Cambridge, MA: Harvard University Press, 1966).26Greenberger, Crenson, and Crissey, Models in the Policy Process, p. 32.27For example, Alice Rivlin, Systematic Thinking for Social Action (Washington, DC: BrookingsInstitution, 1971); and Walter Williams, Social Policy Research and Analysis: The Experience in theFederal Social Agencies (New York: American Elsevier Publishing, 1971).28Irving Louis Horowitz and James E. Katz, Social Science and Public Policy in the United States(New York: Praeger Publishers, 1975), p. 17.29National Science Foundation, Federal Funds for Research, Development, and Other ScientificActivities (Washington, DC: NSF, 1973); and National Science Board, Science and EngineeringIndicators—1989 (Washington, DC: NSB, 1989).



own journal of record. In the 1980s the process of institutionalizing policy-orientedsocial science was carried a step further by the creation of multidisciplinary professionalassociations such as the Association for Public Policy and Management, which holdsannual research conferences and publishes a journal of record, the Journal of PolicyAnalysis and Management. The new journal brought a more technical focus than themainstream policy journals, including Policy Sciences, the Policy Studies Journal, andPolicy Studies Review. In addition to the mainstream journals were hundreds of othersthat focused on specific issues involving health, welfare, education, criminal justice,education, science and technology, and other areas.30

In the same period, universities in the United States and Europe founded newgraduate programs and degrees in policy analysis. In the United States, a number ofnew programs were established with the support of the Ford Foundation’s Programin Public Policy and Social Organization. Most research universities in the UnitedStates have policy centers or institutes listed in the Encyclopedia of Associations,along with thousands of freestanding nonprofit policy research organizations andadvocacy groups. Most were established after 1950. In Washington and most statecapitals, and in the European Union, “policy analyst” is a formal job description.The National Governor’s Association and the National League of Cities have policyanalysis units. There are similar units throughout the U.S. government; in direc-torates of the European Union; and in international organizations, including theUnited Nations and the World Bank. Even a brief search of the World Wide Webyields scores of policy think tanks in all regions of the world.31

Since the beginning of the twenty-first century, there has been increasing recognitionthat the complexity of problems faced by governments today requires the systematic useof natural and social scientists to help develop policies and assess their consequences.The call for evidence-based policy making in the United Kingdom, the United States,and the European Union is a response to this complexity; it is also a recognition thatideological, religious, and political influences—usually hidden and lacking in trans-parency—have exerted a harmful effect on policy making in areas ranging from health,education, and welfare to national security and the environment. In the words of a recentBritish House of Commons report titled Scientific Advice, Risk, and Evidence BasedPolicy Making (2006), evidence-based policy making

has its roots in Government’s commitment to “what works” over ideologicallydriven policy. . . . This Government expects more of policy makers. More newideas, more willingness to question inherited ways of doing things, better use ofevidence and research in policy making and better focus on policies that willdeliver long-term goals.32

30Michael Marien, editor of Future Survey (journal of record of the World Future Society), estimates thenumber of policy journals to exceed four hundred. Marien, “The Scope of Policy Studies: ReclaimingLasswell’s Lost Vision,” in Advances in Policy Studies since 1950, vol. 10, Policy Studies Review Annual,ed. William N. Dunn and Rita Mae Kelly (New Brunswick, NJ: Transaction, 1991), pp. 445–88.31See, for example, www.nira.go.jp (World Directory of Think Tanks), a Japanese directory, andwww.policy.com.32United Kingdom, House of Commons, Science and Technology Committee. Scientific Advice, Riskand Evidence Based Policy Making. Seventh Report of Session 2005–06, Volume I (London: HMOPrinting House, 2006).



Evidence-based policy making in the United Kingdom and the European Uniontakes several forms including regulatory impact assessment (RIA), which refers tothe use of scientific analyses to examine the benefits, costs, risks, and consequencesof newly introduced policies before they are adopted. In the United States,evidence-based policy has been promoted by leading program evaluators andpolicy analysts who founded the Coalition for Evidence-Based Policy of theCouncil for Excellence in Government. Some procedures of the Office ofManagement and Budget are based on evidence-based policy analysis, especiallymethods and standards of program evaluation.33 Although some see the movementtoward evidence-based policy making as a continuation of an ostensibly harmfullogical positivist (scientistic) approach to questions of public policy anddemocracy,34 as yet it is unclear whether this negative assessment is itself based onsound reasoning and evidence.

THE POLICY-MAKING PROCESSThe development of policy analysis has been a response to practical problems.Although recognition of these practical origins is important, historical awarenessalone does not tell us much about the characteristics of policy making and howit works.

Policy analysis is a fundamentally intellectual activity embedded in a politicalprocess. This process, which includes economic, cultural, and organizational factors,is usually described as a policy-making process, or policy process for short. It isuseful to visualize this process as a series of interdependent activities arrayed throughtime—agenda setting, policy formulation, policy adoption, policy implementation,policy assessment, policy adaptation, policy succession, and policy termination(Table 1).35 Depending on circumstances, analysts produce information relevant toone, several, or all phases of policy making.

The policy process is composed of complex rounds or cycles (Figure 1). Eachphase of the policy cycle is linked to the next, in backward and forward loops, andthe process as a whole has no definite beginning or end. Individuals, interestgroups, bureaus, offices, departments, and ministries are involved in policy cycles

33Council for Evidence Based Policy. 1301 K Street, NW, Suite 450 West, Washington, DC 2005.www.excelgov.org/evidence; www.evidencebasedprograms.org.34Wayne Parsons, “From Muddling Through to Muddling Up: Evidence Based Policy-Makingand the Modernisation of British Government.” Unpublished paper (London: University of London, 2004).35The policy cycle, or stages approach, is summarized by Peter de Leon, “The Stages Approachto the Policy Process: What Has It Done? Where Is It Going?” in Theories of the Policy Process, ed. Paul A. Sabatier (Boulder, CO: Westview Press, 1999). See also Charles O. Jones,An Introduction to the Study of Public Policy, 2d ed. (North Scituate, MA: Duxbury Press,1977); James A. Anderson, Public Policy Making (New York: Praeger, 1975); Gary Brewer and Peter de Leon, Foundations of Policy Analysis (Homewood, IL: Dorsey Press, 1983). The classicwork that influenced all works cited here is Harold D. Lasswell, The Decision Process: SevenCategories of Functional Analysis (College Park: Bureau of Governmental Research, Universityof Maryland, 1956).



TABLE 1

Phases of the Policy-Making Process

Phase Characteristics Illustration

Agenda Setting Elected and appointed officials placeproblems on the public agenda. Manyproblems are not acted on at all, whereasothers are addressed only after long delays.

A state legislator and her cosponsor preparea bill that goes to the Health and WelfareCommittee for study and approval. The billstays in committee and is not voted on.

Policy Formulation

Officials formulate alternative policies to deal with a problem. Alternative policiesassume the form of executive orders, courtdecisions, and legislative acts.

A state court considers prohibiting theuse of standardized achievement testssuch as the SAT on grounds that the testsare biased against women and minorities.

Policy Adoption

A policy is adopted with the support of alegislative majority, consensus amongagency directors, or a court decision.

In Roe v. Wade, Supreme Court justicesreach a majority decision that womenhave the right to terminate pregnanciesthrough abortion.

PolicyImplementation

An adopted policy is carried out byadministrative units that mobilize financialand human resources to comply with thepolicy.

The city treasurer hires additional staffto ensure compliance with a new lawthat imposes taxes on hospitals that nolonger have tax-exempt status.

Policy Assessment

Auditing and accounting units ingovernment determine whether executiveagencies, legislatures, and courts are incompliance with statutory requirements ofa policy and achieving stated objectives.

The General Accounting Office monitorssocial welfare programs such as Aid toFamilies with Dependent Children(AFDC) to determine the scope ofwelfare fraud.

Policy Adaptation

Auditing and evaluation units report to agencies responsible for formulating,adopting, and implementing policies thatadaptations in poorly written regulations,insufficient resources, inadequate training,etc. are needed.

A state Department of Labor andIndustry evaluates an affirmative actiontraining program, finding that employeeswrongly believe that complaints againstdiscrimination should be made toimmediates.

Policy Succession

Agencies responsible for evaluatingpolicies acknowledge that a policy is nolonger needed because the problemdissolved. Rather than terminate thepolicy, it is maintained and redirectedtoward a new goal.

The National Highway Traffic SafetyAdministration (NHTSA) convincesCongress to maintain the 55 mph speedlimit because it is achieving the new goalof reducing traffic fatalities, injuries, andproperty damage.

Policy Termination

Agencies responsible for evaluation andoversight determine that a policy or anentire agency should be terminatedbecause it is no longer needed.

The U.S. Congress terminates the Office ofTechnology Assessment (OTA) on groundsthat other agencies and the private sectorare able to assess the economic and socialeffects of technologies.



AgendaSetting Formulation

Implementation Adoption

AGENDAS

POLICIES

Assessment

PROPOSALSOUTPUTS

AdaptationSuccessionTermination

IMPACTS

FIGURE 1The policy-making process has multiple functions and stages

through cooperation, competition, and conflict. One form of the cycle involvespolicy adaptation, whereby a feedback loop connects later phases of the process toearlier ones. Other forms of cycles are policy succession, where new policies andorganizations build on old ones, and policy termination. Policy termination maymean the end of a policy or program, although even termination affects what is-sues are placed on the public agenda, and in this sense represents another kind ofcycle.

In some cases, a policy is adopted first and then justified by working backwardto agenda setting, when a problem is formulated or reformulated to fit or justify thepolicy. Adjacent phases may be linked, or skipped altogether, creating “shortcircuits.” Solutions and problems are in continuous flux, creating a degree ofcomplexity that prompts metaphors of “garbage cans,” “primeval policy soups,”and “organized anarchies.”36

36Michael Cohen, James March, and Johan Olsen, “A Garbage Can Model of Organizational Choice,”Administrative Science Quarterly 17 (March 1972): 1–25; John W. Kingdon, Agendas, Alternatives, andPublic Policies, 2d ed. (New York: HarperCollins, 1995).



MODELS OF POLICY CHANGEMetaphors such as garbage cans, primeval policy soups, and organized anarchiesare difficult to grasp, because it seems that policy making under these conditionswould have no structure or organization at all. That this is untrue may be illustratedby a story:37

Some years ago, on a military base, there were reports of the possible theft ofmilitary secrets. Every day, at about the same time, a worker pushing alarge wheelbarrow would attempt to leave the base by passing through thesecurity gate.

On the first day, the guards at the gate asked the worker what was in thewheelbarrow. He replied “Just dirt.” The guard poked into the wheelbarrowwith his baton. Satisfied, he told the worker to pass through. The same proce-dure was repeated for the rest of the week. By the second week, the guardswere growing increasingly suspicious. They made the worker empty thecontents of the wheelbarrow on the road. This time, the guards used a rake toexamine the contents. Again, they found nothing but dirt. This continued foranother two weeks.

In the third week, the guards called in a special investigations unit. Theynot only emptied the contents; they also used a special scanning device. Theyfound nothing. Subsequently, the worker was allowed to pass freely throughthe gate.

At the end of the month, dozens of wheelbarrows were reported missing.

Structures are just as important as their contents. Conceptual models of policymaking are mental structures or maps that help us understand policy-makingprocesses. These conceptual models are abstract representations based on metaphorssuch as “garbage can,” “anarchy,” and “primeval soup.” Other metaphors are “policyobservatory,” “problems as infectious diseases,” “policy decay,” “war on poverty,”and “rationality” itself.38

Comprehensive RationalityThe comprehensive rationality model portrays policy making as an exhaustivestriving for efficiency. A rational economic actor is seen as Homo economicus,an individual or collective decision maker who weighs the costs and benefits ofall available alternatives and takes actions that are motivated by a concern withthe efficient use of resources. The fundamental proposition of the economicrationality model is as follows: The greater the net efficiency (perceived benefitsless perceived costs) of an alternative selected from a full (comprehensive) set of

37John Funari, personal communication.38Rein and Schon call them “generative metaphors” because they generate a framework for representingpolicy problems. See Martin Rein and Donald A. Schon, “Problem Setting in Policy Research,” inUsing Social Research in Public Policymaking, ed. Carol Weiss (Lexington, MA: D.C. Heath, 1977), pp. 240–43.



potential solutions, the more likely it will be chosen as a (rational) basis for policyaction and change.39 Policy change occurs when an individual or collective deci-sion maker:

� Identifies a policy problem on which there is sufficient consensus amongrelevant stakeholders that the decision maker can act on their behalf.

� Specifies and ranks consistently the objectives whose attainment wouldconstitute a resolution of the problem.

� Identifies the policy alternatives that may best contribute to the attainment ofeach goal and objective.

� Forecasts the consequences that will result from the selection of eachalternative.

� Compares the efficacy of these consequences in attaining each objective.� Chooses the alternative(s) that maximizes the attainment of objectives.� Intervenes in the policy-making process by acting on the choice.

Other versions of rational choice involve the incorporation of institutional trans-action costs into choice situations;40 the redefinition of benefits and costs in political,social, organizational, or moral terms (e.g., Homo politicus);41 and the proviso thatdecision makers are fallible learners with imperfect information, limited computa-tional capabilities, and a propensity for error in institutional contexts.42

Second-Best RationalityAn important criticism of the rational economic model is based on and known bythe name of Arrow’s impossibility theorem. The theorem states that it is impossiblefor decision makers in a democratic society to meet the requirements of theeconomic rationality model.43 Individual rational choices cannot be aggregated bymeans of majority voting procedures to create a single best choice for all parties.The impossibility of creating a collective decision that involves transitive preferences(if A is preferred to B, and B is preferred to C, then A is preferred to C) is describedas the “voters’ paradox.”

Consider a committee composed of three members: Brown, Jones, and Smith.The committee wants to decide which of three forms of energy—solar, coal, andnuclear—should be adopted to resolve the energy crisis. Brown, the leader of anenergy rights organization, prefers solar to coal and coal to nuclear, reasoning thatthis ranking creates least risk to citizens. Because Brown’s choice is transitive, it

39Rational choice models are reviewed by Elinor Ostrom, “Institutional Rational Choice: AnAssessment of the Institutional Analysis and Development Framework,” in Theories of the PolicyProcess, ed. Sabatier, pp. 35–72. The classic critique of comprehensive rational choice is Charles E.Lindblom, The Policy-Making Process (Englewood Cliffs, NJ: Prentice Hall, 1968).40Oliver E. Williamson, The Economic Institutions of Capitalism (New York: Free Press, 1985).41David J. Silverman, The Theory of Organisations (New York: Free Press, 1972).42Elinor Ostrom, Governing the Commons: The Evolution of Institutions for Collective Action(New York: Cambridge University Press, 1990); and Mancur Olson, The Logic of Collective Action:Public Goods and the Theory of Groups (Cambridge, MA: Harvard University Press, 1965).43Kenneth J. Arrow, Social Choice and Individual Values (New York: John Wiley, 1963).



conforms to the rule: If A is preferred to B, and B is preferred to C, then A ispreferred to C. Jones and Smith, who represent the coal and nuclear industries,also have transitive preferences. Jones prefers coal to nuclear, nuclear to solar, andcoal to solar, reasoning that coal is most profitable, followed by nuclear and solar.In turn, Smith prefers nuclear to solar, solar to coal, and nuclear to coal, reasoningthat nuclear is most profitable. Solar energy is less profitable than coal but createsfewer dangers to the environment. Smith sees coal as the least desirable of thethree alternatives.

The three choices are rational and transitive from each individual’s point ofview. Yet, once the three members attempt through majority rule to reach ademocratic decision, there is a paradox (Table 2). When we ask them to choose be-tween solar and coal, we see that solar is preferred to coal by a margin of 2 to 1(Brown and Smith versus Jones). Similarly, when we ask them to choose betweencoal and nuclear, we observe that coal is preferred to nuclear by a 2 to 1 margin(Brown and Jones versus Smith). However, if we now apply the transitivity rule tothese collective preferences, we should find that if A is preferred to B (solar to coal),and B is preferred to C (coal to nuclear), then A is preferred to C (solar to nuclear).However, this is not the collective result, because two committee members (Jonesand Smith) prefer C to A (nuclear to solar). Hence, individual preferences aretransitive, whereas the collective preference is cyclic, which means that alternativescannot be ranked consistently. For this reason, an economically rational choiceis impossible.

Arrow’s impossibility theorem proves by logical deduction that it is impossible toapply democratic procedures (e.g., majority rule) to reach collective decisions that aretransitive. There are five “reasonable conditions” of any democratic decision proce-dure: (1) nonrestriction of choices, that is, all possible combinations of individualpreferences must be taken into account; (2) nonperversity of collective choice, that is,

TABLE 2

The Voters’ Paradox

Committee Member Preference

Brown A (solar) preferred to B (coal)B (coal) preferred to C (nuclear)A (solar) preferred to C (nuclear)

Jones B (coal) preferred to C (nuclear)C (nuclear) preferred to A (solar)B (coal) preferred to A (solar)

Smith C (nuclear) preferred to A (solar)A (solar) preferred to B (coal)C (nuclear) preferred to B (coal)

Majority A (solar) preferred to B (coal)B (coal) preferred to C (nuclear)C (nuclear) preferred to A (solar)



collective choices must consistently reflect individual choices; (3) independence ofirrelevant alternatives, that is, choices must be confined to a given set of alternativesthat are independent of all others; (4) citizens’ sovereignty, that is, collective choicesmust not be constrained by prior choices; and (5) nondictatorship, that is, no individualor group can determine the outcome of collective choices by imposing theirpreferences on others.

To avoid the dilemma of intransitive preferences, we might delegate collectivechoices to a few decision makers (e.g., political or technical elites) who can beexpected to reach consensus and, hence, a transitive choice. Although this solves theproblem of intransitive preferences, it nevertheless violates conditions of citizens’sovereignty and nondictatorship. Alternatively, we might introduce additional alter-natives (e.g., wind power) in the hope that this will foster consensus. However, thisviolates the condition of independence of irrelevant alternatives. In practice,political systems based on majority rule employ both procedures to reach collectivechoices.44 These choices are described as second-best decisions.

Disjointed IncrementalismThe disjointed-incremental model of policy change holds that policy choices seldomconform to the requirements of the economic rationality model.45 The fundamentalproposition of disjointed-incremental theory is that policy changes occur at themargin with the status quo, so that behavior at time t is marginally different frombehavior at time t � 1. According to disjointed incrementalism, change occurs whenpolicy makers:

� Consider only those alternatives that differ incrementally (i.e., by smallamounts) from the status quo—alternatives that differ by large amounts areunlikely to result in successful policy change.

� Limit the number of consequences forecast for each alternative.� Make mutual adjustments in goals and objectives, on one hand, and policy

alternatives on the other.� Continuously reformulate problems and alternatives in the course of acquiring

new information.� Analyze and evaluate alternatives sequentially, so that choices are continuously

amended over time, rather than made at a single point prior to action.� Continuously remedy existing problems, rather than attempt to solve problems

completely at one point in time.� Share responsibilities for analysis and evaluation with many groups in society,

so that the process of making choices is fragmented or disjointed.� Make incremental and remedial policy changes.

44The setting of the sequence of issues on agendas is an important example of violating conditions ofcitizens’ sovereignty and nondictatorship. See Duncan Black, The Theory of Committees and Elections(Cambridge, MA: Cambridge University Press, 1958). For a review of these problems, see NormanFrohlich and Joe A. Oppenheimer, Modern Political Economy (Englewood Cliffs, NJ: Prentice Hall,1978), ch. 1.45Charles E. Lindblom and David Braybrooke, A Strategy of Decision (New York: Free Press, 1963).



Bounded RationalityAnother alternative to the rational economic model is bounded rationality.According to this model, policy makers do not attempt to be economically rationalin the full, or comprehensive, sense of considering and weighing all alternatives.46

Although choices are rational, they are bounded by the practical circumstances inwhich policy makers work. The fundamental proposition of bounded rationality isthat policy change occurs when policy makers use “rules of thumb” to make choicesthat are minimally acceptable. The originator of the bounded rationality model,multidisciplinary political scientist Herbert Simon, argues that “[i]t is impossible forthe behavior of a single, isolated individual to reach any high degree of rationality.The number of alternatives he must explore is so great, the information he wouldneed to evaluate them so vast that even an approximation to objective rationality ishard to conceive [emphasis added].”47

This formulation, although it retains the idea of rationality, recognizes the limitsof comprehensive, economically rational choice when decision makers seek to maxi-mize some valued outcome. In contrast to this type of maximizing behavior, Simonproposes the concept of satisficing behavior. Satisficing refers to acts of choicewhereby decision makers seek to identify a course of action that is just “goodenough,” that is, when the combination of satisfactory and suffice produce a “satis-ficing” choice. In other words, decision makers do not consider all the many alterna-tives that, in principle, might produce the greatest possible increase in the benefits ofaction (i.e., maximizing behavior). Decision makers need consider only the mostevident alternatives that will produce a reasonable increase in benefits (i.e., “satisficing”behavior). Bounded rationality, which is about limitations on individual rationalchoices, is closely related to disjointed incrementalism, which is about limitations incollective rational choices.

Satisficing behavior may be seen as an effort to maximize valued outcomeswhile concurrently recognizing constraints imposed by the costs of information. Inthe words of two proponents of this model of choice,

It is doubtful whether any analyst of rational choice procedures would claimthat a decision maker should systematically investigate and evaluate all ofthe alternatives available to him. Such search is both time-consuming andcostly and an optimal decision procedure should take these factors into account . . . the cost of decision-making should be incorporated into the maxi-mizing model.48

Here, rationality is qualified by a requirement to take into account the costs andbenefits of searching for and assessing new alternatives.

46See Herbert A. Simon, Administrative Behavior (New York: Macmillan, 1945). Simon received theNobel Prize in 1978 for his contributions to the study of decision making in economic organizations.Some of Simon’s other important works are Models of Man (New York: Wiley, 1957) and The Sciencesof the Artificial (New York: Wiley, 1970).47Simon, Administrative Behavior, p. 79.48Richard Zeckhauser and Elmer Schaefer, “Public Policy and Normative Economic Theory,” in TheStudy of Policy Formation, ed. Raymond A. Bauer (Gencoe, IL: Free Press, 1966), p. 92.



49See Amitai Etzioni, “Mixed-Scanning: A ‘Third’ Approach to Decision Making,” PublicAdministration Review 27 (December 1967): 385–92.50Yehezkel Dror, Ventures in Policy Sciences (New York: American Elsevier, 1971).51Etzioni, “Mixed-Scanning,” p. 389.

Mixed ScanningAnother model of change is mixed scanning. It is an alternative to economic ration-ality, disjointed incrementalism, and bounded rationality.49 Although Etzioni andothers50 accept these criticisms of the economic rationality model, they also stressthe limitations of the incremental model. Incrementalism is seen to have a conserva-tive and status quo orientation that is difficult to reconcile with needs for creativityand innovation in policy making. For example, incrementalism suggests that themost powerful interests in society make many of the most important policy choices,because it is these interests that have the most to gain from policies that differ aslittle as possible from the status quo. Further, incrementalism appears to ignore thatpolicy choices differ in scope, complexity, and importance. Major strategic choicesare different from day-to-day operational decisions, a distinction that theincremental theory does not adequately take into account.

Mixed scanning distinguishes between the requirements of strategic choicesthat set basic policy directions, and operational choices that help lay the ground-work for strategic choices or contribute to their implementation. The fundamentalproposition of mixed scanning is that policy change occurs when problems ofchoice are adapted to the nature of problems confronted by policy makers. Becausewhat is rational in one context may not be so in another, mixed scanning selectivelycombines elements of comprehensive rationality and disjointed incrementalism.In Etzioni’s words,

Assume we are about to set up a worldwide weather observation system usingweather satellites. The rationalistic approach [i.e., the rational-comprehensivetheory] would seek an exhaustive survey of weather conditions by usingcameras capable of detailed observations and by scheduling reviews of theentire sky as often as possible. This would yield an avalanche of details, costlyto analyze and likely to overwhelm our action capabilities (e.g., “seeding” cloudformations that could develop into hurricanes or bring rain to arid areas).Incrementalism would focus on those areas in which similar patterns developedin the recent past and, perhaps, in a few nearby regions; it would thus ignore allformations which might deserve attention if they arose in unexpected areas.51

Mixed scanning, in contrast to either of the two approaches taken alone,provides for choices based both on comprehensive economic rationality and ondisjointed incrementalism. The particular combination depends on the nature ofthe problem. The more those problems are strategic in nature, the more that thecomprehensive economic rationality approach is appropriate. Conversely, the morethose problems are operational in nature, the more appropriate the incrementalapproach. In all circumstances, some combination of the two approaches is neces-sary, because the problem is not to adopt one approach and reject the other, but tocombine them in a prudent way.



Erotetic RationalityA challenge to the models of policy change described earlier is erotetic rationality,which refers to a process of questioning and answering. Erotetic rationality,although it may be unsatisfying to those who demand well-specified models inadvance of analysis, lies at the core of inductive processes of many kinds.52 Alberthas stated the major principle of erotetic rationality succinctly in his critique of theuses of benefit-cost analysis in judicial settings: “Ignorance is the sine qua non ofrationality.”53 In many of the most important cases, analysts simply do not knowthe relationship among policies, policy outcomes, and the values in terms of whichsuch outcomes should be assessed. Here, the frank acknowledgment of ignorance isa prerequisite of engaging in a process of questioning and answering, a process thatmay yield rationally optimal answers to questions that “transcend accreted experi-ence and outrun the reach of the knowledge already at our disposal.”54 Eroteticrationality is closely related to problem structuring as the central guidance system ofpolicy analysis. It is also at the core of recent innovations in physics, as representedby the work of Ilya Prigogine, the originator of chaos theory: “Twentieth-centuryphysics is no longer the knowledge of certainties, it is one of interrogation. . . . Andeverywhere, instead of uncovering the permanence and immutability that classicalscience has taught us to seek out in nature, we have encountered change, instability,and evolution.”55

Critical ConvergenceCritical convergence refers to policy processes that are like a complex river deltawith multiple streams converging and diverging as they cross the flood plain andmeander toward the sea.56 If we try to understand the “outcomes” of the process by using a bucket of water to sample from the sea, we will not be able to discoverthe various routes by which the water reaches the sea. However, if we were able tomonitor the entire process and its structure, we would find that from time to timesome streams converge to form deep channels that, at least temporarily, guide theprocess in predictable ways.

The metaphor of the river delta and its multiple streams conveys some of thecomplexity of the policy-making process, but without abandoning the responsi-bility to identify structures that shape the process. In policy-making contexts,individuals and groups interact over time to set agendas and formulate policies.Their success, however, depends on the ability to recognize critical moments(“policy windows”) when three kinds of streams—problems, policies, and

52Nicholas Rescher, Induction (Pittsburgh, PA: University of Pittsburgh Press, 1980), pp. 6–7.53Jeffrey M. Albert, “Some Epistemological Aspects of Cost-Benefit Analysis,” George Washington LawReview 45, no. 5 (1977): 1030.54Rescher, Induction, p. 6.55Ilya Prigogine, “A New Model of Time, a New View of Physics,” in Models of Reality, ed. Jacques Richardson (Mt. Airy, MD: Lomond Publications, 1984). Quoted by Rita Mae Kelly andWilliam N. Dunn, “Some Final Thoughts,” in Advances in Policy Studies since 1950, ed. Dunnand Kelly, p. 526.56Alex Weilenman attributes the metaphor to systems theorist Stafford Beer.



politics—converge.57 The fundamental proposition of the critical convergencemodel is that policy change occurs at these critical moments. The recognition ofthese moments of convergence is part of the task of the policy analyst.

Punctuated EquilibriumOne of the difficulties of many of these models is that none satisfactorily accounts formajor departures from the dominant pattern of slow, gradual change predicted by thedisjointed-incremental and bounded rationality models. For example, the abrupt anddiscontinuous changes that occurred in environmental policy making under theClean Air Act are relatively rare.58 Recognizing that significant policy change mayoccur only every twenty-five years or so, Sabatier and Jenkins-Smith59 call attentionto the importance of external perturbations, or exogenous shocks, in effecting policychanges that are discontinuous and broad in scope. Such changes include relativelyrapid socioeconomic changes represented by recessions, depressions, and energy crises;sudden shifts in public opinion such as those that occurred late in the Vietnam War;and the dramatic rise in nationalism and personal insecurity that occurred after theSeptember 11, 2001 attacks on the World Trade Center and the Pentagon.

The punctuated equilibrium model likens the process of policy change to biologicalevolution.60 Most policies are relatively stable, changing by small increments over longperiods. There is a dynamic equilibrium among competing policies, much like theprocess of partisan mutual adjustment identified by Lindblom and Braybrooke.61

Partisan mutual adjustment can and often does involve a competitive process amongrelatively autonomous bureaucracies. Bureaucratic leaders compete for recognition andresources in accordance with the institutional incentives and reward structure of theirorganizations (“Where you stand depends on where you sit.”). Periodically, abruptchanges (punctuations) in policy occur as a consequence of new political images,which, in turn, are products of “policy earthquakes” and other external shocks.

The fundamental proposition of the punctuated equilibrium model is that externalshocks are a necessary but not sufficient condition of major policy change. The sufficientcondition is that new political images and understandings of the political world suddenlyarise in response to these shocks. However, when new political images, beliefs, andvalues develop gradually, over long periods of time, the process is not “punctuated.”62

57In contrast to Kingdon, Agendas, Alternatives, and Public Policies (see note 36), I am using “riverdelta” in place of “garbage can.” The combination of “streams,” “garbage cans,” and “windows”supplies a badly mixed metaphor.58See Charles O. Jones, Clean Air (Pittsburgh, PA: University of Pittsburgh Press, 1975). Jones calls hismodel of discontinuous change “speculative augmentation.”59Paul A. Sabatier and Hank C. Jenkins-Smith, “The Advocacy Coalition Framework: An Assessment,”in Theories of the Policy Process, ed. Sabatier, pp. 117–66. Two causally relevant conditions affectingpolicy change are external shocks and core values. Their “advocacy coalition framework” parallelstheories of population dynamics in biology.60See James L. True, Frank R. Baumgartner, and Bryan D. Jones, “Punctuated-Equilibrium Theory:Explaining Stability and Change in American Policymaking,” in Theories of the Policy Process, ed.Sabatier, pp. 97–116; and Frank R. Baumgartner and Bryan D. Jones, Agendas and Instability inAmerican Politics (Chicago: University of Chicago Press, 1993).61Lindblom and Braybrooke, A Strategy of Decision.62See Sabatier and Jenkins-Smith, “The Advocacy Coalition Framework.”



ProblemStructuring

Forecasting


POLICYPROBLEMS

POLICYACTIONS

Evaluation

EXPECTEDOUTCOMES

OBSERVEDOUTCOMES

PracticalInference

POLICYPERFORMANCE

FIGURE 2The process of integrated policy analysis

POLICY ANALYSIS IN THE POLICY PROCESSThe purpose of policy analysis is to improve policy making. This is no simple taskwhen we consider that many of the most important policy changes are gradual,disjointed, and incremental. Large, discontinuous changes are relatively rare; andthey stem from shocks that are exogenous to the policy-making process, not fromthe relatively marginal influence of analyses conducted within the process.Nevertheless, multidisciplinary policy analysis is important because policy-relevantinformation has the potential to improve policy making (see Figure 2).

Potential Uses of AnalysisAmong policy analysts, it is an unspoken article of faith that good analysis potentiallyyields better policies. The reasons for this potential are evident in the purposes ofpolicy-analytic methods.

Problem Structuring. Problem-structuring methods supply policy-relevant infor-mation that can be used to challenge the assumptions underlying the definition ofproblems at the agenda-setting phase of policy making (see Table 1). Problem struc-



turing assists in discovering hidden assumptions, diagnosing causes, mapping possi-ble objectives, synthesizing conflicting views, and visualizing, discovering, and de-signing new policy options. For example, the problem of race and sex bias in thesome twenty million standardized tests administered annually in the United Stateswas placed on the legislative agendas of several states throughout the late 1980s.In Pennsylvania, analysts challenged the assumption that test bias is a problemrequiring legislative action—for example, the outright prohibition of standardizedtests—after synthesizing and evaluating research on test bias recommended bydifferent stakeholders. The large observed discrepancies in minority and whiteachievement test scores were not viewed as a problem of test bias, but as an indicator,or gauge, of inequality of educational opportunity. Analysts recommended thecontinued use of standardized tests to monitor and mitigate these inequalities.63

Forecasting. Methods for forecasting expected policy outcomes provide policy-relevant information about consequences that are likely to follow the adoption ofpreferred policies at the adoption phase of policy formulation. Forecasting helpsexamine plausible, potential, and normatively valued futures; estimate the conse-quences of existing and proposed policies; specify probable future constraints on theachievement of objectives; and estimate the political feasibility of different options.Analysts in the Health Care Finance Administration, for example, employed fore-casting methods to estimate the effects of revenue shortfalls on the Medicare trustfund. In the absence of new health care policy initiatives, the 1990 forecast was thatthe 40 million persons who had no health insurance were likely to increase innumber.64 In 2009, during debates about President Obama’s proposed AffordableHealth Choices Act, the earlier forecast proved to be generally accurate. Some46 million Americans are without health insurance today.65

Prescription. Methods for selecting preferred policy alternatives yield policy-relevant information about the benefits and costs—and more generally the valueor utility—of expected policy outcomes estimated through forecasting, thus aidingpolicy makers in the policy adoption phase. Through the process of prescribingpreferred policies, analysts estimate levels of risk and uncertainty, identify exter-nalities and spillovers, specify criteria for making choices, and assign administrativeresponsibility for implementing policies. For example, the debate over maximumspeed limits in the United States has focused on the costs per fatality averted underthe 55 mph and 65 mph options. In the 1980s, one recommendation, based on theconclusion that the 55 mph speed limit would continue to account for no morethan 2 percent to 3 percent of fatalities averted, was to shift expenditures

63William N. Dunn and Gary Roberts, The Role of Standardized Tests in Minority-Oriented CurricularReform, policy paper prepared for the Legislative Office for Research Liaison, Pennsylvania House ofRepresentatives, February 1987.64Sally T. Sonnefeld, Daniel R. Waldo, Jeffrey A. Lemieux, and David R. McKusick, “Projections ofNational Health Expenditures through the Year 2000.” Health Care Financing Review 13, no. 1 (fall1991): 1–27.65This number should be interpreted in light of the fact that citizens are entitled by law and in principleif not in fact to emergency care.



for maintaining the speed limit to the purchase of smoke detectors, which wouldsave many more lives.66 By 1987, some forty states were experimenting withhigher speed limits. Although the 55 mph speed limit was formally abandoned in1995, President Obama’s nominee to head the National Highway Traffic SafetyAdministration (NHTSA) has expressed his commitment to a return to the 55 mphspeed limit.

Monitoring. Methods for monitoring observed policy outcomes provide informa-tion about the consequences of adopting policies, thus assisting in the policy imple-mentation phase. Many agencies regularly monitor policy outcomes and impacts byemploying policy indicators in areas of health, education, housing, welfare, crime,and science and technology.67 Monitoring helps assess degrees of compliance,discover unintended consequences of policies and programs, identify implementa-tion obstacles and constraints, and promote administrative accountability. Forexample, economic and social welfare policies in the United States are monitored byanalysts in several agencies, including the Bureau of the Census and the Bureau ofLabor Statistics. A 2009 analysis concluded that the total percentage of personsbelow the poverty threshold fell from 13.5 to 12.3 percent from 1990 to 2009.The greatest reduction was among persons older than age 65.68 In the same period,the percentage of households in the bottom 20 percent increased from 3.4 to3.8 percent, and the percentage in the top 20 percent increased from 46.7 percentto 49 percent. In this case, policy monitoring reveals a marked increase in incomeinequality combined with a reduction in the poverty rate.

Evaluation. Methods for evaluating observed policy outcomes yield policy-relevantinformation about discrepancies between expected and actual policy performance,thus assisting in the policy assessment and adaptation phases. Evaluation not onlyresults in conclusions about the extent to which problems have been alleviated, but italso may contribute to the clarification and critique of values driving a policy, aid inthe adjustment or reformulation of policies, and establish a basis for restructuringproblems. A good example of evaluation is the type of analysis that contributes to theclarification and critique of values by proposing that ethical reasoning augment thetechnical reasoning that dominates environmental policy making in the EuropeanCommunity and other parts of the world.69

66See, for example, Charles A. Lave and Lester B. Lave, “Barriers to Increasing Highway Safety,” inChallenging the Old Order: Towards New Directions in Traffic Safety Theory, ed. J. Peter Rothe (NewBrunswick, NJ: Transaction Books, 1990), pp. 77–94.67A thorough and insightful source on the use of policy indicators is Duncan MacRae Jr., PolicyIndicators: Links between Social Science and Public Debate (Chapel Hill: University of North CarolinaPress, 1985).68Daniel R. Meyer and Geoffrey L. Wallace, “Poverty Levels and Trends in Comparative Perspective,”Focus 26, 2 (fall 2009): 9.69See, for example, Silvio O. Funtowicz and Jerome R. Ravetz, “Global Environmental Issues and theEmergence of Second Order Science” (Luxembourg: Commission of the European Communities,Directorate-General for Telecommunications, Information Industries, and Innovation, 1990). See alsoFuntowicz and Ravetz, “A New Scientific Methodology for Global Environmental Issues,” in EcologicalEconomics, ed. Robert Costanza (New York: Columbia University Press, 1991), pp. 137–52.



Uses of Analysis in PracticeIn principle, policy analysis has the potential for creating better policies. In practice,however, policy analysis and other fields within the social (and natural) scienceshave some or all of the following limitations.70

� Use is indirect, delayed, and general. Policy analysis is seldom used directlyand immediately as a basis for improving specific decisions involving theallocation of human and material resources. The use of analysis is typicallyindirect and general, and single analyses are rarely important unless they arepart of a larger body of knowledge. The indirect and general character of useis understandable when we recognize that policy making is a complex processcomposed of multiple cycles, ranging from termination and succession toadaptation and short-circuiting.

� Improvement is ethically controversial. When policy analysis is used, thequestion of what constitutes an improvement depends on the political,ideological, or ethical stance of observers. Those who believe that the publicinterest is served by assisting those who are worst off by taxing those who arebest off view “improvements” differently from those who believe that thepublic interest is served when individuals solve their problems withoutgovernment intervention. Although the goal of “efficiency improvement” issometimes seen as something everyone supports, it too is an ideological orethical matter, because it involves the selection of some values and normsover others.

� Being useful reflects personal, professional, and institutional interests. As socialscientists and as persons, policy analysts seek to enhance their personal statusand professional rewards as well as those of their institutions or agencies.Opportunities for involvement with persons of political power, privilege, oreconomic standing—as their advisers, consultants, expert witnesses, or staff—arepart of the motivation for being an analyst.

A good deal of the difficulty surrounding questions of the practical uses ofpolicy analysis stems from a failure to recognize that the process of using policyanalysis in policy making is just as complex as policy making itself. Consider thisdescription of making a decision by Chester Barnard, a successful CEO andmajor contributor to the field of public administration. Writing about an order,or policy, to move a telephone pole from one side of the street to the other,Barnard states,

It can, I think, be approximately demonstrated that carrying out that orderinvolves perhaps 10,000 decisions of 100 men located at 15 points, requiringsuccessive analyses of several environments, including social, moral, legal,economic, and physical facts of the environment, and requiring 9,000 redefinitionsand refinements of purpose. If inquiry be made of those responsible, probably

70The following discussion draws on Carol H. Weiss, “Introduction,” in Using Social Research inPublic Policymaking, ed. Weiss, pp. 1–22; and Carol H. Weiss with Michael J. Bucuvalas, Social ScienceResearch and Decision Making (New York: Columbia University Press, 1980).



not more than half-a-dozen decisions will be recalled or deemed worthy ofmention. . . . The others will be “taken for granted,” all of a part of the business ofknowing one’s business.71

Clearly, then, the use of policy analysis is a complex process with severaldimensions:72

� Composition of users. Individuals as well as collectives—for example,agencies, ministries, bureaus, courts, legislatures, parliaments—use policyanalysis. When the use of analysis involves gains (or losses) in the personalutility of information, the process of use is an aspect of individual decisions(individual use). By contrast, when the process of use involves manyindividuals, it is an aspect of collective decisions, or policies (collective use).

� Expected effects of use. The use of policy analysis has cognitive as well asbehavioral effects. Cognitive effects include the use of policy analysis tothink about problems and solutions (conceptual use) and to legitimize preferredformulations of problems and solutions (symbolic use). By contrast, behavioraleffects involve the use of policy analysis as a means or instrument for carryingout observable activities or functions (instrumental use). Conceptual andbehavioral uses of information occur among individual and collective users.73

� Scope of information used. The scope of information used by policy makersranges from the specific to the general. The use of “ideas in good currency” isgeneral in scope (general use), whereas the use of a particular recommendationis specific (specific use).74 Information that varies in scope is used by individualsand collectives, with effects that are conceptual and behavioral.

In practice, these three dimensions of information use overlap. The intersectionamong them provides a basis for assessing and improving the uses of policy analysis.

71Chester I. Barnard, The Functions of the Executive (Cambridge, MA: Harvard University Press, 1938,1962), p. 198. Quoted by Carol Weiss, “Knowledge Creep and Decision Accretion,” Knowledge:Creation, Diffusion, Utilization 1, no. 3 (March 1980): 403.72William N. Dunn, “Measuring Knowledge Use,” Knowledge: Creation, Diffusion, Utilization 5, no. 1(1983): 120–33. See also Carol H. Weiss and Michael I. Bucuvalas, “Truth Tests and Utility Tests: DecisionMakers’ Frames of Reference for Social Science Research,” American Sociological Review 45 (1980):302–13; and Jack Knott and Aaron Wildavsky, “If Dissemination Is the Solution, What Is the Problem?” inThe Knowledge Cycle, ed. Robert F. Rich (Beverly Hills, CA: Sage Publications, 1981), pp. 99–136.73On distinctions between conceptual, instrumental, and symbolic use, see Carol H. Weiss, “Researchfor Policy’s Sake: The Enlightenment Function of Social Science Research,” Policy Analysis 3 (1977):200–24; Weiss, “The Circuitry of Enlightenment,” Knowledge: Creation, Diffusion, Utilization 8, no. 2(1986): 274–81; Nathan Caplan, Andrea Morrison, and Roger Stambaugh, The Use of Social ScienceKnowledge in Policy Decisions at the National Level (Ann Arbor, MI: Institute for Social Research,Center for the Utilization of Scientific Knowledge, 1975); Robert F. Rich, “Uses of Social ScienceKnowledge by Federal Bureaucrats: Knowledge for Action versus Knowledge for Understanding,” inUsing Social Research in Public Policy Making, ed. Weiss, pp. 199–211; and Karin D. Knorr,“Policymakers’ Use of Social Science Knowledge: Symbolic or Instrumental?” in Using Social Researchin Public Policy Making, ed. Weiss, pp. 165–82.74The concept of “ideas in good currency” is discussed by Donald A. Schon, “Generative Metaphor:A Perspective on Problem Setting in Social Policy,” in Metaphors and Thought, ed. A. Ortony(Cambridge: Cambridge University Press, 1979), pp. 254–83.



REVIEW QUESTIONS1. What does it mean to say that policy analysis is

a process of producing knowledge about and inthe policy-making process?

2. What is the relation between the kinds of eco-nomic and political problems faced by societiesand the growth of what has been called an“analycentric” perspective?

3. Describe intellectual and social characteristicsof policy analysis. Give examples.

4. What is “organized anarchy”? How is it relatedto the “garbage can” model of policy making?

5. In your judgment, which model(s) of policychange are most useful? Why?

6. Considering the three dimensions of informa-tion use, how would you know when policymakers have used policy analysis? Explainyour answer.

7. What are the pros and cons of evidence-based policy making? Consider concepts of“technocratic guidance” and “technocraticcounsel” presented in the case study for thischapter.

DEMONSTRATION EXERCISES1. After reading Case 1 (Are Policy Analysts

Technocrats?), evaluate the following statementsabout “technocratic counsel” and “technocraticguidance” in policy making:� The technocratic guidance perspective pres-

ents an exaggerated assessment of the powerand influence of professional policy analysts.

� The technocratic counsel perspective over-estimates the symbolic importance of policyanalysts in legitimizing policies made forpurely political reasons.

� The remedy for “technocratic policy analysis”is not to abandon policy analysis, but tocounteract technocratic biases “by movingfrom phony neutrality to thoughtful partisan-ship, working disproportionately to assisthave-nots in understanding and making theircase . . . assisting all partisans in coping withuncertainties.”75

2. Obtain a published analysis of a contemporarypolicy problem. The analysis could be posted ona website or published in a professional journal,

75Edward J. Woodhouse and Dean A. Nieusma, “Democratic Expertise: Integrating Knowledge, Power,and Participation,” in Knowledge, Power, and Participation in Environmental Policy Analysis, vol. 12,Policy Studies Review Annual, ed. Matthijs Hisschemoller, Rob Hoppe, William N. Dunn, and Jerry R.Ravetz (New Brunswick, NJ: Transaction Books, 2001), p. 91.

Historically, the aim of policy analysis has beento provide policy makers with information thatcan be used to solve practical problems. As wehave seen, however, this practical role is limitedby the fact that policy analysis is an intellectualactivity embedded in a political process charac-terized by the exercise of power, privilege, anddomination. Although this process is sometimesseen as a set of temporally orderly phases, therelations among these phases are sufficientlycomplex to justify the use of metaphors suchas “garbage cans” and “organized anarchies.”

In response to this complexity, practitioners aswell as academics have developed models thatdescribe how and why policies change. Thesemodels, ranging from disjointed incrementalismto bounded rationality, critical convergence,and punctuated equilibrium, explain why theuses of policy analysis tend to be indirect,delayed, general, and ethically controversial.This is to be expected, considering that the aimsof policy analysis include the creation of know-ledge about as well as in the policy-makingprocess.

CHAPTER SUMMARY



newsletter, magazine, or newspaper. Use theprofile of research use found in Case 2 to write ashort report that answers the following ques-tions:� To what extent is the analysis of high

technical quality?� To what extent does the analysis conform to

what you expected?

� To what extent is the analysis actionable,that is, has an action orientation?

� To what extent is the analysis a challenge tothe status quo?

� What does your report suggest about the rea-sons why policy analysis is perceived as useful?

� What does your report suggest about thereasons why research is actually used.

BIBLIOGRAPHYBarber, Bernard. Effective Social Science: Eight

Cases in Economics, Political Science, andSociology. New York: Russell Sage Foundation,1987.

Council for Evidence Based Policy. 1301 K Street,NW, Suite 450 West, Washington, DC 20005.www.excelgov.org/evidence; www.evidencebased-programs.org/

De Leon, Peter. Advice and Consent: The Develop-ment of the Policy Sciences. New York: RussellSage Foundation, 1988.

Fischer, Frank. Technocracy and the Politics ofExpertise. Newbury Park, CA: Sage Publications,1990.

Freidson, Elliot. Professional Powers: A Study ofthe Institutionalization of Formal Knowledge.Chicago: University of Chicago Press, 1986.

Horowitz, Irving L., ed. The Use and Abuse ofSocial Science: Behavioral Science and PolicyMaking. 2d ed. New Brunswick, NJ: TransactionBooks, 1985.

Jasanoff, Sheila. The Fifth Branch: Science Advisorsas Policymakers. Cambridge, MA: HarvardUniversity Press, 1990.

Kingdon, John W. Agendas, Alternatives, andPublic Policies. Glenview, IL: Scott, Foresman,1984.

Lerner, Daniel, ed. The Human Meaning of the SocialSciences. Cleveland, OH: World PublishingCompany, 1959.

Lindblom, Charles E. Inquiry and Change: TheTroubled Attempt to Understand and ChangeSociety. New Haven, CT: Yale UniversityPress, 1990.

Lindblom, Charles E., and David K. Cohen.Usable Knowledge: Social Science and SocialProblem Solving. New Haven, CT: YaleUniversity Press, 1979.

Lindblom, Charles E., and Edward J. Woodhouse.The Policy-Making Process. 3d ed. EnglewoodCliffs, NJ: Prentice Hall, 1993.

Machlup, Fritz. Knowledge: Its Creation, Distri-bution, and Economic Significance. Vol. 1:Knowledge and Knowledge Production.Princeton, NJ: Princeton University Press, 1980.

Macrae, Duncan Jr. The Social Function of SocialScience. New Haven, CT: Yale UniversityPress, 1976.

Parsons, Wayne. Public Policy: An Introduction tothe Theory and Practice of Policy Analysis.Northampton MA: Edward Elgar PublishingInc., 1995.

Ravetz, Jerome. Science and Its Social Problems.Oxford: Oxford University Press, 1971.

Sabatier, Paul A. Theories of the Policy Process.Boulder, CO: Westview Press, 1999.

Schmandt, Jurgen, and James E. Katz. “TheScientific State: A Theory with Hypotheses.”Science, Technology, and Human Values 11(1986): 40–50.

United Kingdom, House of Commons, Science andTechnology Committee. Scientific Advice, Riskand Evidence Based Policy Making. SeventhReport of Session 2005–06, Volume I. London:HMO Printing House, 2006.

Weiss, Carol H. Social Science Research andDecision Making. New York: ColumbiaUniversity Press, 1980.


76Rowland Egger, “The Period of Crises: 1933 to 1945,” in American Public Administration: Past,Present, and Future, ed. Frederick C. Mosher (Tuscaloosa: University of Alabama Press, 1975), pp. 49–96.77Fritz Machlup, The Production and Distribution of Knowledge in the United States (Princeton, NJ:Princeton University Press, 1962).78Daniel Bell, The Coming of Post-Industrial Society: A Venture in Social Forecasting (New York: BasicBooks, 1976).79Jurgen Schmandt and James E. Katz, “The Scientific State: A Theory with Hypotheses,” Science,Technology, and Human Values 11 (1986): 40–50.80Bell, Coming of Post-Industrial Society, pp. xvi–xviii.

industrial society, is increasingly dominated by aneducated professional-technical class. Many of thecharacteristics of a postindustrial society areimportant to consider when we ponder thehistorical evolution and significance of policyanalysis:80

� Centrality of theoretical knowledge. Although theearliest civilizations relied on specializedinformation, only in the late twentieth andtwenty-first centuries are innovations in “soft”(social) and “hard” (physical) technologiesdirectly dependent on the codification oftheoretical knowledge provided by the social,physical, and biological sciences.

� Creation of new intellectual technologies. Theimprovement in mathematical and statisticaltechniques has made it possible to use advancedmodeling, simulation, and various forms of systemsanalysis to find more efficient and “rational”solutions to public problems.

� Spread of a knowledge class. The most rapidlyexpanding occupational group in the United Statesis composed of technicians and professionals. Thisgroup, including professional managers, represented28 percent of employed persons in 1975, 28.3percent in 1995, and 30.3 percent in 2000. Personsin nonfarm occupations grew from 62.5 to 98.4percent between 1900 and 1990.

� Shift from goods to services. In 1975, more than65 percent of the active labor force was engagedin the production and delivery of services, a figurethat exceeded 80 percent by 2000. Services are

In the first half of the twentieth century, theinvolvement of social and natural scientists in policymaking was largely a response to social, economic,and politico-military crises.76 Members of thesciences and professions were ad hoc and irregularparticipants in government in the period ofdepression, dislocation, and war that stretched fromthe early 1930s to the late 1940s. In succeedingyears, however, the social sciences and socialprofessions—political science, sociology, economics,public administration, business management, socialwork, planning—became one of the main sources offull-time governmental staff, advisers, andconsultants. For the first time, public agenciesrecruited on a regular basis specialists who hadreceived their training from and had been certifiedby their respective professional organizations.

For many observers, this new era signified theadvent of a new form of social and politicalorganization in which public policy making andsocietal guidance were critically dependent on thespecialized knowledge and technologies of thesciences and professions. Emphasizing its socialaspect, some referred to this new form oforganization as the “knowledge society”77 or the“postindustrial society.”78 Others, focusing on itspolitical aspect, spoke of the transition from the“administrative state” to the “scientific state.”79

Postindustrial Society and the Scientific State

Postindustrial society, an extension of patterns ofpolicy making and socioeconomic organization of

CASE 1 ARE POLICY ANALYSTS TECHNOCRATS?



81See Jeffrey D. Straussman, “Technocratic Counsel and Societal Guidance,” in Politics and the Futureof Industrial Society, ed. Leon N. Lindberg (New York: David McKay, 1976), pp. 126–66.82For example, Amitai Etzioni, The Active Society (New York: Free Press, 1968); and Donald T.Campbell, “The Experimenting Society,” in The Experimenting Society: Essays in Honor of Donald T.Campbell, ed. William N. Dunn (New Brunswick, NJ: Transaction Books, 1998), pp. 35–68.

(continued)

primarily human (e.g., health, education, socialwelfare) and professional-technical (e.g.,information services, computer programming,telecommunications).

� Instrumentalization of sciences. Although thenatural and social sciences have long been usedas instruments for controlling the human andmaterial environment, they have been largelyopen to internal criticism and protected by normsof scientific inquiry. In the present period, scienceand technology have become increasinglybureaucratized, subordinated to governmentgoals, and assessed in terms of their payoffs tosociety. The vast bulk of funded research isapplied research on practical problems ofbusiness, commerce, and government.

� Scarcity of scientific and technical

information. Information has increasingly becomeone of society’s most scarce resources. Informationis a collective, not a private, good, and cooperativerather than competitive strategies for its productionand use are required if optimal results are to beachieved. The rapid exponential growth of theInternet and World Wide Web are counteracting thescarcity of information demanded by governments,businesses, and nonprofit organizations.

The development of policy analysis and otherapplied sciences seems to be part of an emergingpostindustrial society and scientific state. Yet, whatdoes this development signify for issues ofdemocratic governance, civil liberties, and privacy?How much influence do producers of policy-relevantinformation have? Although theoretical knowledgeclearly seems to be more central than it has been inother historical periods, what does this mean for thedistribution of political power? Does theproliferation of new intellectual technologies mean

also that structures and processes of policy makinghave changed? Is the spread of a “knowledge class”commensurate with its power and influence?Considering that the aim of policy analysis and otherapplied sciences is to produce information forpractical purposes, whose purposes are beingserved? In short, how are we to interpret thehistorical transformation of policy-oriented inquiryfrom earlier times to the present-day “knowledgesociety” and “scientific state”?

Technocratic Guidance versus Technocratic Counsel

There are several contending perspectives of thesequestions, two of which we consider here.81 Oneperspective holds that the professionalization ofpolicy analysis and other applied social sciences is—or should be—shifting power and responsibilityfrom policy makers to policy analysts.82 Thisperspective, technocratic guidance, is associated with the“analycentric turn” described earlier in this chapter.The basic premise of the analycentric turn is that“the surest way to improve the quality of publicchoice is to have more analysts producing moreanalyses.”83 By contrast, a contending perspective,technocratic counsel, holds that the professionalizationof policy analysis and other applied social sciencessignifies new and more effective ways to enhance theinfluence of policy makers and other dominantgroups whose positions continue to rest on power,wealth, and privilege.84

Strictly speaking, neither of these perspectives iswholly accurate in its interpretation of eventssurrounding the movement toward postindustrialsociety; each contains a one-sided emphasis onparticular characteristics of contemporary society tothe exclusion of others. Yet just as the developmentof nineteenth-century policy analysis was a practicalresponse to problems of the day as viewed by

83Schick, “Beyond Analysis, p. 259.84See, for example, Guy Benveniste, The Politics of Expertise (Berkeley, CA: Glendessary Press, 1972); andFrank Fischer, Technocracy and the Politics of Expertise (Newbury Park, CA: Sage Publications, 1990).



dominant groups, so too is contemporary policyanalysis a consequence of changes in the structureand role of government as it has attempted tograpple with new problems.“The contemporaryboom in policy analysis, ” writes Schick,“has itsprimary source in the huge growth of Americangovernments, not in the intellectual development ofthe social sciences.”85 Throughout the twentiethcentury, the expansion of government was followedby demands for more policy-relevant information. Asgovernment has grown, so too has the market forpolicy analysis.86

The development of policy analysis in the pastcentury followed fits and starts in the growth offederal executive agencies.87 In 1900, there weresome ninety federal executive agencies. By 1940, thisfigure had grown to 196, and, by 1973, the totalnumber of federal executive agencies had jumped to394.The greatest percentage increase, 35 percent,occurred after 1960.The rapid growth of the federalgovernment was mainly a response to new problemsin areas of national defense, transportation, housing,health, education, welfare, crime, energy, and theenvironment. There were many new “challenges” topolicy making:88 participation overload, mobilizationof service sector workers, transformation of basicsocial values, realignment of interest groups, growingcleavages between urban and suburban citizens, andrecurrent fiscal crises. Some observers, notably policyscientist Yehezkel Dror, went so far as to formulatethe relationship between such problems and policyanalysis in the form of a general law: “While thedifficulties and dangers of problems tend to increaseat a geometric rate, the number of persons qualifiedto deal with these problems tends to increase at anarithmetic rate.”89 Thus, the growth of policy analysis

seems to be a consequence, not a cause, of changes inthe structure of government and the nature and scopeof social problems.

The technocratic guidance perspective assertsthat policy-relevant knowledge is an increasinglyscarce resource whose possession enhances thepower and influence of policy analysts. Thetechnocratic guidance perspective may besummarized in terms of five major propositions:90

� The growing interdependencies, complexity, andpace of change of contemporary society makeexisting knowledge obsolete, thus increasing thedemand for new forms of policy-relevant knowledge.

� Problems of contemporary society may beresolved with specialized knowledge produced bypolicy analysts.

� The technical complexity of policy choicesprompts higher levels of direct involvement ofpolicy analysts and other applied scientists.

� Higher levels of direct involvement enhance thepower of professional analysts to influence policymaking, making politicians increasinglydependent on them.

� The growing dependence of politicians onprofessional policy analysts erodes theirpolitical power.

A rival perspective, that of technocratic counsel,begins from the assumption that professional policyanalysts operate in settings in which policy makers,as the consumers of specialized knowledge,determine to a large extent the activities ofproducers. In this context, the primary role ofanalysts is to legitimize—that is, justify in scientificand technical terms—policy decisions made by thereal holders of power. The technocratic counsel

85Schick, “Beyond Analysis,” p. 258.86Schmandt and Katz, “Scientific State.”87Herbert Kaufman, Are Government Organizations Immortal? (Washington, DC: BrookingsInstitution, 1976), pp. 34–63.88Samuel P. Huntington, “Postindustrial Politics; How Benign Will It Be?” Comparative Politics 6(January 1974): 163–92; Ronald Inglehart, “The Silent Revolution in Europe: Intergenerational Changein Postindustrial Societies,” American Political Science Review, 65 (December 1971): 991–1017; andJames O’Connor, The Fiscal Crisis of the State (New York: St. Martin’s Press, 1973).89Dror, Ventures in Policy Sciences, p. 2.90Straussman, “Technocratic Counsel,” pp. 150–51.



perspective may also be summarized in terms ofseveral key propositions:91

� Major policy alternatives reflect conflicting valuesheld by different segments of the community.

� Value conflicts are associated with disparities ofpolitical power.

� The choice of a given policy alternativesymbolizes the victory of one segment of thecommunity over another.

� Policy makers use scientific and technicaljustifications produced by analysts to suppressconflicts and legitimize choices that are actuallymade on political grounds.

� The credibility of scientific and technicaljustifications requires that policy analysis andother applied sciences be presented as valueneutral, impartial, and apolitical.

� Professional analysts, although they are a sourceof scientific and technical justifications, areexpendable. They also serve as convenientscapegoats for policies that fail.

If we pick our cases carefully, we can show thatanalysts exercised a large measure of “technocraticguidance.” For example, the New York City–RANDCorporation’s analysis of factors influencing theresponse time of city firefighters to reported fires hasbeen credited with unusual success, even though otherRAND efforts have been less effective in shaping

91Ibid., pp. 151–52.92On this and other cases, see Greenberger, Crenson, and Crissey, Models in the Policy Process, pp. 231–318.93Nathan Caplan, “Factors Associated with Knowledge Use among Federal Executives,” Policy StudiesJournal 4, no. 3 (1976): 233. See also Robert F. Rich, Social Science Information and Public Policy Making(San Francisco, CA: Jossey-Bass, 1981); and David J. Webber, “The Production and Use of Knowledge inthe Policy Process,” in Advances in Policy Studies since 1950, ed. Dunn and Kelly, pp. 415–41.94Nathan Caplan et al., The Use of Social Science Knowledge in Policy Decisions at the National Level(Ann Arbor, MI: Institute for Social Research, Center for Research on the Utilization of ScientificKnowledge, 1975).

decisions.92 On the other hand, the use of specializedknowledge to make policy choices appears to be highlyuneven. For example, in a study of 204 federalexecutives conducted in 1973–74, social scienceresearch was relatively unimportant as a basis forchoosing among policy alternatives Moreover, theextent to which research was used depended onessentially nontechnical factors:“the level ofknowledge utilization is not so much the result of aslow flow of relevant and valid information fromknowledge producers to policy makers, but is due moreto factors involving values, ideology, and decision-making styles.”93,94 Similar findings are reported inrecent issues of journals such as Science Communication,Implementation Science, and the British Medical Journal.

The finding that policy analysis is used for politicalpurposes adds a measure of credibility to thetechnocratic counsel perspective. So does theapparent conservative character of many analyses. Inthis context, policy analysis has been characterized asa conservative and superficial kind of social sciencethat fails to pose radical questions about basic socialvalues and institutions, and neglects policyalternatives that depart significantly from existingpractices.95 Moreover, some observers see in policyanalysis an ideology in disguise that suppresses ethicaland value questions in the name of science.96 Undersuch conditions, it is understandable that analystsmight be used as instruments of everyday politics. �

95Charles E. Lindblom, “Integration of Economics and the Other Social Sciences through PolicyAnalysis,” in Integration of the Social Sciences through Policy Analysis, ed. James C. Charlesworth(Philadelphia: American Academy of Political and Social Sciences, 1972), p. 1. For an elaboration ofthis critique, see Charles E. Lindblom, Inquiry and Change: The Troubled Attempt to Understand andChange Society (New Haven, CT: Yale University Press, 1990).96Laurence H. Tribe, “Policy Science: Analysis or Ideology?” Philosophy and Public Affairs 2, no. 1(1972): 66–110.


j. supports your own position on the issue (CUE)k. provides evidence to back up

recommendations (TQ)l. contains politically feasible implications (AO)

m. agrees with previous knowledge about theissue (TQ)

n. can be implemented under the presentcircumstances (AO)

o. challenges existing assumptions andinstitutional arrangements (CSQ)

p. raises new issues or perspectives (CSQ)q. is inexpensive to implement (AO)r. was available at the time a decision had to

be made (AO)s. contains novel or unexpected findings

(negative coding)t. provides quantitative data (TQ)u. can be generalized to similar settings or

persons (TQ)v. addresses in a thorough manner different

factors that could explain outcomes (TQ)w. uses statistically sophisticated analyses (TQ)x. demonstrates high technical competence (TQ)y. presents internally consistent and

unambiguous findings (TQ)z. takes an objective and unbiased approach (TQ)

Using the following categories, sum up the scaledresponses (scale � numbers 1 through 5), and thenaverage them (divide by the number of questionsanswered).The capital letters after each question(e.g.,TQ) refer to the categories.

Technical Quality (TQ):

Conformity to User Expectations (CUE):

Action Orientation (AO):

Challenge Status Quo (CSQ): �

The following profile, drawn from the work of Carol H.Weiss at Harvard University, is helpful in learningabout the uses of research in policymaking.97 Pleaseuse the following scale for responses to questions abouta policy paper or report published by a governmentagency or a think tank.You may obtain the paper orreport on the Internet from a website such aswww.publicagenda.org, www.rand.org, or www.aei.org.

1 � very much2 � quite a bit3 � to some extent4 � not much5 � not at all

1. To what extent does the analysis contain infor-mation that is relevant to your work?

2. To what extent do you think the analysis isreliable (could be repeated with similarconclusions) and valid (plausibly explains whatit set out to explain)?

3. To what extent do the conclusions agree withyour own views about the issue?

4. Please indicate the extent to which the analysisa. deals with a high priority issue (AO)b. adds to theoretical knowledge in the field (TQ)c. adds to practical knowledge about policies

or programs (AO)d. agrees with your ideas or values (CUE)e. identifies outcomes that policy makers can

do something about (AO)f. suggests potential courses of action (AO)g. implies the need for major policy changes

(CSQ)h. focuses on specific policy outcomes (AO)i. contains explicit recommendations for

action (AO)

CASE 2 UNDERSTANDING THE USE OF POLICYANALYSIS

97These questions are drawn from Carol H. Weiss and Michael J. Bucuvalas, “The Challenge of Social Research toDecision Making,” in Using Social Research in Public Policy Making, ed. Weiss, Appendix 15A and 15B, pp. 231–33. I amgrateful to Carol Weiss for suggesting modifications in questions.


Structuring PolicyProblems



Structuring PolicyProblems

O B J E C T I V E S


� Distinguish problem structuringfrom problem solving.

� Understand the systemic, artificial,and interdependent nature of policyproblems.

� Contrast problem situations andproblems.

� Describe differences amongrelatively well-structured,

moderately structured, andill-structured problems.

� Compare and contrast types of policy models.

� Identify strengths and limitations of different methods of problemstructuring.

� Analyze cases of problemstructuring in policy analysis.

Inexperienced analysts suppose that problems are purely objective conditionsthat are determined by the “facts.” This methodologically innocent view fails torecognize that the same facts—for example, statistics that show that global

warming is on the upswing—are interpreted in varied ways by policy stakeholders.For this reason, the same policy-relevant information frequently results in conflictingdefinitions of a “problem.” These definitions are shaped by personal and institutionalinterests, assumptions about human nature, ideas about the proper role of government,and beliefs about the nature of knowledge itself.


Structuring Policy Problems

NATURE OF POLICY PROBLEMSPolicy problems are unrealized needs, values, or opportunities for improvement.1

Information about the nature, scope, and severity of a problem is produced byapplying the policy-analytic procedure of problem structuring. Problem structuring,which is a phase of policy inquiry in which analysts compare, contrast, and evaluaterival problem formulations, is among the most important procedures performed byanalysts. Problem structuring is a central guidance system or steering mechanismthat affects the success of all other phases of policy analysis. Analysts seem to failmore often because they solve the wrong problem than because they get the wrongsolution to the right problem.

Beyond Problem SolvingPolicy analysis is often described as a problem-solving methodology. Although thisis partly correct—and analysts do succeed in finding solutions for problems2—theproblem-solving image can be misleading. It wrongly suggests that analysts cansuccessfully identify, evaluate, and recommend solutions for a problem withoutinvesting a very considerable amount of time and effort in formulating that prob-lem. Policy analysis is a dynamic, multilevel process in which methods of problemstructuring are essential to the success of methods of problem solving (Figure 1).

Methods at one level are inappropriate and ineffective at the next, because thequestions are different at the two levels. For example, lower-level questions aboutthe net benefits of alternative solutions for problems of industrial pollution alreadyassume that industry is the problem. At the next higher level are questions aboutthe scope and severity of pollution, the conditions that may cause it, and potentialsolutions for its reduction or mitigation. Here we may find that the proper formu-lation of the problem is closely related to the driving habits of Americans for whompetroleum fuels are comparatively cheap, by world standards, and subsidized bythe government. This is a question of problem structuring; the former is a questionof problem solving. Similar distinctions include:

� Problem sensing versus problem structuring. The process of policy analysisdoes not begin with clearly articulated problems, but with a sense of diffuseworries and inchoate signs of stress.3 These are not problems but problemsituations. By contrast, policy problems “are products of thought acting onenvironments; they are elements of problem situations that are abstractedfrom these situations by analysis.”4

1See David Dery, Problem Definition in Policy Analysis (Lawrence: University Press of Kansas, 1984).2See, for example, Bernard Barber, Effective Social Science: Eight Cases in Economics, Political Science,and Sociology (New York: Russell Sage Foundation, 1987).3Martin Rein and Sheldon H. White, “Policy Research: Belief and Doubt,” Policy Analysis 3, no. 2(1977): 262. This is a paraphrase of John Dewey.4Russell A. Ackoff, Redesigning the Future: A Systems Approach to Societal Problems (New York:Wiley, 1974), p. 21.



PROBLEMSITUATION

POLICYPROBLEM

POLICYSOLUTION

RIGHTPROBLEM?

NO

NO

YES

YES RIGHTSOLUTION?

ProblemStructuring

ProblemSolving

ProblemSensing

ProblemDissolving

ProblemUnsolving

ProblemResolving

FIGURE 1Problem sensing versus problem structuring and problem sensing

5See Stephen H. Linder and B. Guy Peters, “From Social Theory to Policy Design,” Journal of PublicPolicy 4, no. 4 (1985): 237–59; John Dryzek, “Don’t Toss Coins into Garbage Cans: A Prologue toPolicy Design,” Journal of Public Policy 3, no. 3 (1983): 345–67; and Trudi C. Miller, “Conclusion:A Design Science Perspective,” in Public Sector Performance: A Turning Point, ed. T. C. Miller(Baltimore: Johns Hopkins University Press, 1985).

� Problem structuring versus problem solving. Policy analysis includeshigher-order methods of problem structuring as well as lower-order methodsof problem solving. These higher-order methods and the questions for whichthey are appropriate are what some call policy design, or design science.5



In effect, methods of problem structuring are metamethods—that is, theyare “about” and “come before” lower-order methods of problem solving.When analysts use lower-order methods to solve ill-structured problems,they run the risk of committing errors of the third kind: solving the wrongproblem.6

� Problem resolving versus problem unsolving and problem dissolving. Theterms problem resolving, problem unsolving, and problem dissolving refer tothree types of error correction.7 Although the three terms come from the sameroot (L. solvere, to solve or dissolve), the error-correcting processes to whichthey refer occur at distinct levels (see Figure 1). Problem resolving involves thereanalysis of a correctly structured problem to reduce calibrational errors, forexample, reducing the probability of type I or type II errors in testing the nullhypothesis that a policy has no effect. Problem unsolving, by contrast, involvesthe abandonment of a solution based on the wrong formulation of a prob-lem—for example, the policy of urban renewal implemented in central citiesduring the 1960s—and a return to problem structuring in an attempt to for-mulate the right problem. Finally, problem dissolving involves the abandon-ment of an incorrectly formulated problem and a return to problem structuringto start the process anew.

CHARACTERISTICS OF PROBLEMSThere are several important characteristics of policy problems:

1. Interdependency. Policy problems in one area (e.g., energy) frequently affectpolicy problems in other areas (e.g., health care and unemployment). In reality,policy problems are not independent entities; they are parts of whole systemsof problems best described as messes, that is, systems of external conditionsthat produce dissatisfaction among different segments of the community.8

Systems of problems (messes) are difficult or impossible to resolve by using anexclusively analytic approach—that is, one that decomposes problems intotheir component elements or parts—because rarely can problems be definedand resolved independently of one another. Sometimes it is easier “to solve teninterlocking problems simultaneously than to solve one by itself.”9 Systems ofinterdependent problems require a holistic approach, that is, one that viewsproblems as inseparable and unmeasurable apart from the system of whichthey are interlocking parts.10

6Howard Raiffa, Decision Analysis (Reading, MA: Addison-Wesley, 1968), p. 264.7Russell L. Ackoff, “Beyond Problem Solving,” General Systems 19 (1974): 237–39; and HerbertA. Simon, “The Structure of Ill Structured Problems,” Artificial Intelligence 4 (1973): 181–201.8Ackoff, Redesigning the Future, p. 21.9Harrison Brown, “Scenario for an American Renaissance,” Saturday Review, December 25, 1971, 18–19.10See Ian I. Mitroff and L. Vaughan Blankenship, “On the Methodology of the Holistic Experiment:An Approach to the Conceptualization of Large-Scale Social Experiments,” Technological Forecastingand Social Change 4 (1973): 339–53.



2. Subjectivity. The external conditions that give rise to a problem areselectively defined, classified, and evaluated. Although there is a sense in whichproblems are objective—for example, air pollution may be defined in terms oflevels of CO2 gases and particulates in the atmosphere—the same data aboutpollution are interpreted in markedly different ways. The reason is that policyproblems are abstracted from problem situations by analysts. Problems arepartly subjective constructs.11

3. Artificiality. Policy problems are possible only when human beings makejudgments about the desirability of altering some problem situation. Policyproblems are products of subjective human judgment; policy problems alsocome to be accepted as legitimate definitions of objective social conditions;policy problems are socially constructed, maintained, and changed.12 Problemshave no existence apart from the individuals and groups who define them,which means that there are no “natural” states of society that in and ofthemselves constitute policy problems.

4. Instability. There may be as many different solutions for a problem as there aredefinitions of that problem. “Problems and solutions are in constant flux; henceproblems do not stay solved. . . . Solutions to problems become obsolete even ifthe problems to which they are addressed do not [emphasis in original].”13

Problems are not discrete mechanical entities; they are purposeful (teleological)systems in which no two members are identical in all or even any of their propertiesor behaviors; the properties or behavior of each member has an effect on theproperties or behavior of the system as a whole; the properties and behavior of eachmember, and the way each affects the system as a whole, depend on the propertiesand behavior of at least one other member of the system; and all possible subgroupsof members have a nonindependent effect on the system as a whole.14 What thismeans is that systems of subjectively experienced problems—crime, poverty,unemployment, inflation, pollution, health, national security—cannot be decom-posed into independent subsets without running the risk of producing the rightsolution to the wrong problem.

A key characteristic of systems of problems is that the whole is greater—that is,qualitatively different—than the simple sum of its parts. A pile of stones may be definedas the sum of all individual stones but also as a pyramid. Similarly, a human being

can write or run, but none of its parts can. Furthermore, membership in thesystem either increases or decreases the capabilities of each element; it does notleave them unaffected. For example, a brain that is not part of a living body orsome substitute cannot function. An individual who is part of a nation or acorporation is thereby precluded from doing some things he could otherwise do,and he is enabled to do others he could not otherwise do.15

11Ackoff, Redesigning the Future, p. 21.12See Peter L. Berger and Thomas Luckmann, The Social Construction of Reality, 2d ed. (New York:Irvington, 1980).13Ackoff, Redesigning the Future, p. 21.14Mitroff and Blankenship, “On the Methodology of the Holistic Experiment,” pp. 341–42.15Ackoff, Redesigning the Future, p. 13.



Finally, recognition of the interdependency, subjectivity, artificiality, and instabilityof problems alerts us to the unanticipated consequences that may follow from poli-cies based on the right solution to the wrong problem. Consider, for example, theproblem situation confronted by Western European governments in the late 1970s.France and West Germany, seeking to expand the supply of available energy byconstructing nuclear power plants on the Rhine River, defined the energy problemin a way that assumed that the production of nuclear power is independent ofother problems. Consequently, the relation of energy to wider systems of problemsdid not enter into the formulation of the problem. One observer went so far as topredict that

malaria will arrive as a major epidemic in Europe within the next ten years,thanks to the decision in Germany and France to build atomic generators thatutilize river waters for their cooling systems and hence bring the water tempera-ture within the range in which anopheles (the malaria-carrying mosquito)breeds.16

Although this forecast was incorrect, the understanding that energy policy involvesa system of problems was not.

Problems versus IssuesIf problems are systems, it follows that policy issues are as well. Policy issues notonly involve disagreements about actual or potential courses of action, but they alsoreflect competing views of the nature of problems themselves. An apparentlyclear-cut policy issue—for example, whether the government should enforce airquality standards in industry—is typically the consequence of conflicting sets ofassumptions about the nature of pollution:17

1. Pollution is a natural consequence of capitalism, an economic system in whichthe owners of industry seek to maintain and increase profits from theirinvestments. Some damage to the environment is a necessary price to pay fora healthy capitalist economy.

2. Pollution is a result of the need for power and prestige among industrialmanagers who seek promotions in large career-oriented bureaucracies.Pollution has been just as severe in socialist systems in which there are noprofit-seeking private owners.

3. Pollution is a consequence of consumer preferences in high mass-consumptionsocieties. In order to ensure corporate survival, owners and managers must satisfyconsumer preferences for high-performance engines and automobile travel.

The ability to recognize differences between problem situations and policyproblems is crucial for understanding the different ways that common experiencesare translated into disagreements. The formulation of a problem is influenced by

16Ivan Illich in conversation with Sam Keen, reported in Psychology Today (May 1976).17See Ritchie P. Lowry, Social Problems: A Critical Analysis of Theories and Public Policy (Lexington,MA: D. C. Heath and Company, 1974), pp. 23–25.



the assumptions that different policy stakeholders bring to a problem situation.In turn, different formulations of the problem shape the ways that policy issues aredefined. In the example of environmental pollution, assumptions about the opera-tion of a healthy capitalist economy may result in a negative view of governmentenforcement of air quality standards in industry, whereas assumptions about corpo-rate managerial behavior may result in an affirmative position. By contrast,assumptions about consumer preferences and corporate survival may influence theview that government regulation is a nonissue, because government cannot legislateconsumer demand.

Policy problems are related to but different from policy issues. Policy issues maybe classified according to a hierarchy of types: major, secondary, functional, andminor (Figure 2). Major issues are those that are typically encountered at the highestlevels of government within and among national, regional, and local jurisdictions.Major issues typically involve questions of agency mission, that is, questions aboutthe nature and purposes of government organizations. The issue of whether theDepartment of Health and Human Services should seek to eliminate the conditionsthat give rise to poverty is a question of agency mission. Secondary issues are thoselocated at the level of agency programs at the federal, state, and local levels.Secondary issues may involve the setting of program priorities and the definition oftarget groups and beneficiaries. The issue of how to define poverty families is asecondary issue. Functional issues, by contrast, are those located at both the programand the project levels and which involve such questions as budgeting, finance, andprocurement. Finally, minor issues are those that are found most frequently at thelevel of specific projects. Minor issues involve personnel, staffing, employee benefits,vacation times, working hours, and standard operating procedures and rules.

STRATEGIC POLICIES

OPERATIONAL POLICIES

Major Issues

Secondary Issues

Functional Issues

Minor IssuesFIGURE 2Hierarchy of types of policy issues



Some issues call for policies that are strategic, whereas others require operationalpolicies. A strategic policy is one for which the consequences of decisions are rela-tively irreversible. Such issues as whether the United States should send troops to Iraqor Afghanistan, or whether there should be universal health care, call for strategicpolicies involving actions that cannot be reversed for many years. By contrast,operational policies—that is, policies for which the consequences of decisions arerelatively reversible—do not involve the level of risk and uncertainty present athigher levels. Although all types of policies are interdependent—for example, therealization of an agency’s mission depends in part on the adequacy of its personnelpractices—the complexity and irreversibility of policies increases as one moves up thehierarchy of types of policy issues.

Three Classes of Policy ProblemsThere are three classes of policy problems: well-structured, moderately structured,and ill-structured problems.18 The classification of these problems is determined bytheir relative complexity. The differences among well-structured, moderately struc-tured, and ill-structured problems may be illustrated by considering variations intheir common elements (Table 1).

Well-structured problems are those that involve one or a few decision makersand a small set of policy alternatives. Utilities (values) reflect goal consensus andare clearly ranked in order of decision makers’ preferences. The outcomes of eachalternative are known either with complete certainty (deterministically) or withinacceptable margins of probable error (risk). The prototype of the well-structuredproblem is the completely computerized decision problem, with which all conse-quences of all policy alternatives are programmed in advance. Relatively low-leveloperational problems in public agencies provide illustrations of well-structuredproblems. For example, problems of replacing agency vehicles are relatively simpleones that involve finding the optimum point at which an old vehicle should be

18See Ian I. Mitroff and Francisco Sagasti, “Epistemology as General Systems Theory: An Approach tothe Design of Complex Decision-Making Experiments,” Philosophy of the Social Sciences 3 (1973):117–34.

TABLE 1

Differences in the Structure of Three Classes of Policy Problems

Structure of Problem

Element Well Structured Moderately Structured Ill Structured

Decision maker(s) One Few ManyAlternatives Fixed Limited UnlimitedUtilities (values) Consensus Bargaining ConflictOutcomes Certain Uncertain RiskyProbabilities Deterministic Estimable Estimable



traded for a new one, taking into account average repair costs for older vehiclesand purchasing and depreciation costs for new ones.

Moderately structured problems are those involving a few decision makers and arelatively limited number of alternatives. Utilities (values) reflect manageabledisagreements subject to bargaining and mutual adjustment. Nevertheless, theoutcomes of alternatives are uncertain and estimable within margins of error. The pro-totype of the moderately structured problem is the policy simulation or game, forexample, the “prisoner’s dilemma.”19 In this game, two prisoners are held in separatecells; each is interrogated by the prosecuting attorney, who must obtain a confessionfrom one or both prisoners to obtain a conviction. The prosecutor, who has enoughevidence to convict each prisoner of a lesser crime, tells each that if neither confesses,they will both be tried for a lesser crime carrying a punishment; of one year each if bothconfess to the more serious crime, they will both receive a reduced sentence; of fiveyears each; but if only one confesses, he will receive probation, whereas the other willreceive a maximum sentence of ten years. The “optimal” choice for each prisoner,given that neither can predict the outcome of the other’s decision, is to confess. Yet it isprecisely this choice that will result in a five-year sentence for both prisoners, becauseboth are likely to try to minimize their sentences. This example not only illustratesthe difficulties of making decisions when outcomes are uncertain or unknown; theexample also shows that otherwise “rational” individual choices may contribute tocollective irrationality in small groups, government agencies, and corporations.

Ill-structured problems are those that involve many decision makers whoseutilities (values) are either unknown or impossible to rank in a consistent fashion.Whereas well-structured and moderately structured problems reflect processes ofconsensus and bargaining, a major characteristic of ill-structured problems isconflict among competing goals. Policy alternatives and their outcomes may also beunknown or subject to risks that may be estimated through the use of subjectiveprobabilities. The problem of choice is not to uncover known deterministic relationsbut rather to define the nature of the problem. The prototype of the ill-structuredproblem is the completely intransitive decision problem, that is, one for which it isimpossible to select a single policy alternative that is preferred to all others. Whereaswell-structured and moderately structured problems contain preference rankingsthat are transitive—that is, if alternative A1 is preferred to alternative A2, and alter-native A2 is preferred to alternative A3, then alternative A1 is preferred to alternativeA3—ill-structured problems have intransitive preference rankings.

Many of the most important policy problems are ill structured.20 For example,it is unrealistic to assume the existence of one or a few decision makers with consis-tent preferences, because public policies are sets of interrelated decisions made andinfluenced by many policy stakeholders over time. Consensus is rare, because policymaking typically involves conflicts among competing groups. Finally, it is seldompossible to identify the full range of alternative solutions for problems, in part

19Anatol Rapoport and Albert M. Chammah, Prisoner’s Dilemma (Ann Arbor: University of MichiganPress, 1965).20See, for example, David Braybrooke and Charles E. Lindblom, A Strategy of Decision (New York:Free Press, 1963); and Herbert A. Simon, “Theories of Decision-Making in Economic and BehavioralScience,” American Economic Review 1009 (1959): 255–57.



because of constraints on the acquisition of information, but also because it isdifficult to bound a problem and know when a solution has been reached. In effect,there is no “stopping rule” to determine when problems have been correctly definedor solved.

The reasons why ill-structured problems are critical for public policy analysishave been ably summarized by a number of social scientists.21 The contrastbetween ill-structured and “tame” problems shows why understanding the natureill-structured problems is critical to formulating problems, testing solutions, anddetermining whether we have formulated the “right” problem.22

PROBLEM STRUCTURING IN POLICY ANALYSISThe requirements for solving ill-structured problems are not the same as those forsolving well-structured problems. Whereas well-structured problems permit the useof conventional methods, ill-structured problems demand that the analyst take anactive and conscious part in defining the nature of the problem itself.23 In activelydefining the nature of the problem, analysts must not only impose part ofthemselves on the problem situation but also exercise judgment and insight.Problem solving is only one part of the work of policy analysis:

The problem-solving image holds that the work of policy begins with articulatedand self-evident problems. Supposedly, policy begins when recognizableproblems appear, problems about which one can hypothesize possible courses ofaction and in relation to which one can articulate goals. . . . It is not clearproblems, but diffuse worries, that appear. Political pressure groups becomeunusually active, or their activities become more telling; formal and informalsocial indicators give signs of unfavorable trends, or of trends that may beinterpreted as unfavorable. There are signals, then, of a problem, but no oneknows yet what the problem is. . . . In other words, the situation is such that theproblem itself is problematic. Policy analysis contains processes for finding andconstruing problems; it involves problem setting [structuring] in order tointerpret inchoate signs of stress in the system.24

Creativity in Problem StructuringThe criteria for determining the success of problem structuring are also differentfrom those used to judge the success of problem solving. Successful problem solving

21See, for example, Thomas R. Dye, Understanding Public Policy, 3d ed. (Englewood Cliffs, NJ:Prentice Hall, 1978), pp. 30–31; Richard O. Mason and Ian I. Mitroff, Creating a Dialectical SocialScience (Dordrecht, The Netherlands: D. Reidel, 1981); and James G. March and Johan P. Olsen,“The New Institutionalism: Organizational Factors in Political Life,” American Political Science Review78, no. 3 (1984): 739–49.22See Robert E, Horn, “Knowledge Mapping for Complex Social Messes.” A presentation to theFoundations in the Knowledge Economy at the David and Lucile Packard Foundation, July 16, 2001.http://www.stanford.edu/~rhorn/SpchPackard.htm23See John R. Hayes, Cognitive Psychology (Homewood, IL: Dorsey Press, 1978), pp. 210–13.24Rein and White, “Policy Research,” p. 262.



requires that analysts obtain relatively precise technical solutions for clearlyformulated problems. By contrast, successful problem structuring requires thatanalysts produce creative solutions for ambiguous and ill-defined problems. Criteriafor judging creative acts in general are also applicable to creativity in problemstructuring. Problem structuring is creative to the extent that25 (1) the product ofanalysis is sufficiently novel that most people could not or would not have arrived atthe same solution; (2) the process of analysis is sufficiently unconventional that itinvolves the modification or rejection of previously accepted ideas; (3) the process ofanalysis requires sufficiently high motivation and persistence that analysis takes placewith high intensity or over long periods of time; (4) the product of analysis isregarded as valuable because it provides an appropriate solution for the problem;and (5) the problem as initially posed is so ambiguous, vague, and ill defined that thechallenge is to formulate the problem itself.

Phases of Problem StructuringProblem structuring is a process with four related phases: problem search, problemdefinition, problem specification, and problem sensing (Figure 3). A prerequisite ofproblem structuring is the recognition or “felt existence” of a problem situation.In moving from problem situation, the analyst engages in problem search. At this

25See Alan Newell, J. C. Shaw, and Herbert A. Simon, “The Process of Creative Thinking,” inContemporary Approaches to Creative Thinking, ed. H. F. Gruber, G. Terrell, and M. Wertheimer(New York: Atherton Press, 1962), pp. 63–119; see also the excellent monograph by James L. Adams,Conceptual Blockbusting, 4th ed. (Cambridge, MA: Perseus, 2001).

ProblemSearch

ProblemDefinition

ProblemSensing

ProblemSpecification

METAPROBLEM

FORMALPROBLEM

SUBSTANTIVEPROBLEM

PROBLEMSITUATION

FIGURE 3Phases of problem structuring



stage, the goal is not the discovery of a single problem (e.g., that of the client or theanalyst); it is rather the discovery of many problem representations of multiple stake-holders. Practicing analysts normally face a large, tangled network of competing prob-lem formulations that are dynamic, socially constructed, and distributed throughoutthe policy-making process. In effect, analysts are faced with a metaproblem26—aproblem-of-problems that is ill structured because the domain of problem representa-tions held by diverse stakeholders seems unmanageably huge.27 The central task is todefine the boundaries of the metaproblem, that is, a second-order problem that maybe defined as the class of all first-order problems that are its members. Unless thesetwo levels are distinguished, analysts run the risk of formulating the wrong problemby confusing member and class. We violate the rule that “whatever involves all of acollection must not be one of the collection.”28

In moving from metaproblem to substantive problem, the analyst attempts todefine the problem in its most basic and general terms. For example, the analyst maydecide whether the problem is one of economics, sociology, political science, engi-neering, and so forth. If the substantive problem is conceptualized as an economicone, the analyst will treat it in terms of factors related to the production and distri-bution of goods and services—for example, market prices as a determinant of costsand benefits of public programs. Alternatively, if the problem is viewed as politicalor sociological, the analyst will approach it in terms of the distribution of power andinfluence among competing interest groups, elites, or classes. The choice of aconceptual framework is often similar to the choice of a worldview, ideology, orpopular myth and indicates a commitment to a particular view of reality.29

To illustrate the importance of worldviews, ideologies, and popular myths inconceptualizing substantive problems, consider the various ways to define poverty.Poverty may be defined as a consequence of accidents or inevitable states of society,of the actions of evil persons, or of imperfections in the poor themselves.30 Thesedefinitions of poverty contain elements of a worldview, myth, or ideology insofar aseach involves a selective perception of elements of a problem situation.

Worldviews, ideologies, and myths are partly true and partly false; they may beuseful and dangerous at the same time. In this example, the attribution of poverty tohistorical accidents or to inevitability represents a naturalistic perspective of socialproblems that distorts reality by claiming that questions about the distribution ofwealth are pointless, but this same myth may also sensitize analysts to the relativenature of definitions of poverty by pointing to the fact that no known society has

26See Yehezkel Dror, Design for Policy Sciences (New York: Elsevier, 1971).27Another distinguishing feature of an ill-structured problem is that its boundaries are unmanageablyhuge. See P. Harmon and D. King, Expert Systems: Artificial Intelligence in Business (New York:Wiley, 1985).28Alfred North Whitehead and Bertrand Russell, Principia Mathematica, 2d ed., vol. I (Cambridge:Cambridge University Press, 1910), p. 101. See also Paul Watzlawick, John Weakland, and Richard Fisch,Change: Principles of Problem Formation and Problem Resolution (New York: W. W. Norton), p. 6.29Ian Mitroff and Ralph H. Kilmann, Methodological Approaches to Social Science (San Francisco, CA:Jossey-Bass, 1978). See also Thomas Kuhn, The Structure of Scientific Revolutions, 2d ed. (Chicago:University of Chicago Press, 1971); and Ian G. Barbour, Myths, Models and Paradigms (New York:Harper & Row, 1976).30Lowry, Social Problems, pp. 19–46.



wholly resolved the problem. Similarly, the attribution of poverty to evil or morallycorrupt capitalists distorts their actual motivations. Yet this same moralisticperspective, which explains poverty in terms of presumed moral weaknesses, alsodirects attention to the ways in which private ownership promotes waste, exploita-tion, and social irresponsibility. Finally, to attribute poverty to imperfections in thepoor themselves not only results in blaming the victim rather than responsible socialforces but also points to the fact that some poor persons choose to live underconditions that the rest of society defines as “poverty.”

This environmentalist perspective, which attributes poverty and other socialproblems to characteristics of the immediate environment of victims, often resultsin a self-contradictory brand of humanitarianism—blaming the victim. Thehumanitarian can

concentrate his charitable interest on the defects of the victim, condemn thevague social and environmental stresses that produced the defect (some timeago), and ignore the continuing effect of victimizing social forces (right now).It is a brilliant ideology for justifying a perverse form of social action designedto change, not society, as one might expect, but rather society’s victim.31

Once a substantive problem has been defined, a more detailed and specificformal problem may be constructed. The process of moving from substantive toformal problem is carried out through problem specification, which involves thedevelopment of a formal symbolic or mathematical representation (model) of thesubstantive problem. At this point, difficulties may occur, because the relationshipbetween the substantive problem and a formal representation of that problem maybe tenuous.32 The specification of an ill-structured problem in formal mathematicalterms may be premature or inappropriate. The main task is not to obtain the correctmathematical solution but to define the nature of the problem itself.

Errors of the Third Type (EIII)A critical issue of problem structuring is how well substantive and formal problemsactually correspond to the original problem situation. If problem situations containcomplex systems of problems, then a central requirement of policy analysis is theformulation of substantive and formal problems that adequately represent thatcomplexity. The degree of correspondence between a given problem situation and asubstantive problem is determined at the problem definition phase. Here the analystcompares characteristics of the problem situation and the substantive problem. Thelatter may be based on implicit assumptions or beliefs about human nature, time,and the possibilities for social change through government action. Equally impor-tant, however, is the degree of correspondence between the problem situation andthe formal problem, which is often specified in the form of a mathematical formulaor set of equations.

31William Ryan, Blaming the Victim (New York: Pantheon Books, 1971), p. 7.32See Ralph E. Strauch, “A Critical Look at Quantitative Methodology,” Policy Analysis 2 (1976):121–44.



In the first instance (problem search), analysts who fail to engage in search, orstop searching prematurely, run the risk of defining the wrong boundaries of themetaproblem. Important aspects of the metaproblem—for example, the formula-tions of problems held by those who are or will be charged with implementing thepolicy—simply may be left outside the boundaries of the metaproblem. In thesecond instance (problem definition), analysts run the risk of choosing the wrongworldview, ideology, or myth to conceptualize a problem situation when theyshould have chosen the right one. In the third case (problem specification), the mainrisk is to choose the wrong formal representation (model) of the substantive prob-lem when the right one should have been chosen. In any case, analysts may commiterrors of the third type (EIII).

33 Type III error has been described by decision theoristHoward Raiffa in the following terms:

One of the most popular paradigms in . . . mathematics describes the case inwhich a researcher has either to accept or reject a so-called null hypothesis. In afirst course in statistics the student learns that he must constantly balancebetween making an error of the first kind (that is, rejecting the null hypothesiswhen it is true) and an error of the second kind (that is, accepting the nullhypothesis when it is false) . . . practitioners all too often make errors of a thirdkind: solving the wrong problem.34

The process of problem structuring raises a number of issues that are centralto the methodology of policy analysis and science in general. Each phase of prob-lem structuring requires different kinds of methodological skills and differentkinds of reasoning. For example, the kinds of skills most appropriate for discover-ing metaproblems are primarily observational; for substantive problems the re-quired skills are mainly conceptual. Mathematical and statistical subjects (econo-metrics or operations research) are primarily relevant for specifying formalproblems. Problem structuring also raises questions about different meanings ofrationality, because rationality is not simply a matter of finding a relatively preciseformal representation of an expected utility problem. This standard technical def-inition of rationality has been rightly criticized for its formal oversimplification ofcomplex processes.35 Rationality may be defined at more fundamental levels,where the unconscious or uncritical choice of a worldview or ideology may distortthe conceptualization of a substantive problem and its potential solutions. In thiscase, policy analysis may be viewed as an ideology in disguise.36

33See Ian I. Mitroff and Frederick Betz, “Dialectical Decision Theory: A Meta-Theory of Decision-Making,” Management Science, 19, no. 1 (1972): 11–24. Kimball defines type III error as “the errorcommitted by giving the right answer to the wrong problem.” See A. W. Kimball, “Errors of the ThirdKind in Statistical Consulting,” Journal of the American Statistical Association 52 (1957): 133–42.34Raiffa, Decision Analysis, p. 264.35See Jeffrey Friedman, ed. The Rational Choice Controversy (New Haven, CT: Yale University Press,1996); and Kristen R. Monroe, ed. Perestroika! The Raucous Rebellion in Political Science (NewHaven, CT: Yale University Press, 2006).36Laurence Tribe, “Policy Science: Analysis or Ideology?” Philosophy and Public Affairs 2, no. 1(1972): 66–110; and “Ways Not to Think about Plastic Trees,” in When Values Conflict: Essays onEnvironmental Analysis, Discourse, and Decision, ed. Laurence Tribe, Corinne S. Schelling, and JohnVoss (Cambridge, MA: Ballinger Publishing, 1976).



POLICY MODELS AND PROBLEM STRUCTURINGPolicy models are simplified representations of a problem situation.37 As such, thedevelopment of policy models is a form of problem structuring. Just as policy problemsare mental constructs based on the conceptualization and specification of elements of aproblem situation, policy models are constructions and reconstructions of reality.Policy models may be expressed as concepts, diagrams, graphs, or mathematical equa-tions and may be used not only to describe, explain, and predict elements of a problemsituation but also to improve it by recommending courses of action. Policy models arenever literal descriptions of a problem situation. Like policy problems, policy modelsare artificial devices for ordering and interpreting problem situations.

Policy models are useful and even necessary. They simplify systems of problems byhelping to reduce and make manageable the complexities encountered by analysts intheir work. Policy models may help distinguish essential from nonessential features of aproblem situation, highlight relationships among important factors or variables, andassist in explaining and predicting the consequences of policy choices. Policy modelsmay also play a self-critical and creative role in policy analysis by forcing analysts tomake their own assumptions explicit and to challenge conventional ideas and methodsof analysis. In any case, the use of policy models is not a matter of choice, becauseeveryone uses some kind of model. In the words of policy modeler Jay Forrester:

Each of us uses models constantly. Every person in his private life and in his busi-ness life instinctively uses models for decision-making. The mental image of theworld around you which you carry in your head is a model. One does not have acity or a government or a country in his head. He has only selected concepts andrelationships which he uses to represent the real system. A mental image is amodel. All of our decisions are taken on the basis of models. The question is notto use or ignore models. The question is only a choice among alternatives.38

By simplifying problem situations, models inevitably contribute to the selective dis-tortion of reality. Models themselves cannot tell us how to discriminate between essentialand nonessential questions; nor can they explain, predict, evaluate, or recommend,because these judgments are external to the model and not part of it. Whereas modelsmay help us undertake these analytic tasks, the key word is “us”, for it is we and notthe model who provide the beliefs and values necessary to interpret features of reality described by a model. Finally, policy models—particularly those expressed inmathematical form—are frequently difficult to communicate to policy makers and otherstakeholders for whom models are designed as an aid to better decision making.

Descriptive ModelsThe purpose of descriptive models is to explain and/or predict the causes and conse-quences of policy choices. Descriptive models are used to monitor the outcomes of

37See Saul I. Gass and Roger L. Sisson, ed., A Guide to Models in Governmental Planning andOperations (Washington, DC: Office of Research and Development, Environmental Protection Agency,1974); and Martin Greenberger, Mathew A. Crenson, and Brian L. Crissey, Models in the PolicyProcess (New York: Russell Sage Foundation, 1976).38Jay W. Forrester, “Counter-Intuitive Behavior of Social Systems,” Technological Review 73 (1971): 3.



policy actions, for example, outcomes included in the annual list of social indicatorspublished by the Office of Management and the Budget or by Eurostat. Descriptivemodels are also used to forecast economic performance. For example, in the UnitedStates the Council of Economic Advisers prepares an annual economic forecast forinclusion in the President’s Economic Report.

Normative ModelsBy contrast, the purpose of normative models is not only to explain and/or predictbut also to provide recommendations for optimizing the attainment of some utility(value). Among the many types of normative models used by policy analysts arethose that help determine optimum levels of service capacity (queuing models),the optimum timing of service and repairs (replacement models), the optimumvolume and timing of orders (inventory models), and the optimum return on publicinvestments (benefit-cost models). Normative decision models usually take thefollowing form: find the values of the manipulable (policy) variables that willproduce the greatest utility, as measured by the value of outcome variables thatpolicy makers wish to change. Often (but not always) this value is expressed as amonetary quantity such as dollars ($) or euros (€).

A simple and familiar normative model is compound interest. At one point oranother in their lives, many persons have used some variation of this model to findthe values of a “policy” variable (e.g., putting money in a bank versus a creditunion) that will produce the greatest interest income on savings, as measured by theamount of money that one may expect after a given number of years. This is thevalue of the outcome variable that a person wishes to change. The analytical modelfor compound interest is

where Sn is the amount to which savings will accumulate in a given (n) number of years,S0 is the initial savings, and (1 � r)n is the initial investment (1) plus the rate of interest (r)over the number of years of accumulation (n). If an individual (policy maker) knows theinterest rates of different savings institutions and wishes to optimize the return onsavings, this simple normative model should permit a straightforward choice of thatinstitution offering the highest rate of interest, provided there are no other importantconsiderations (e.g., the security of deposits or special privileges for patrons) that shouldbe taken into account. Note, however, that this normative model also predicts the accu-mulation of savings under different alternatives, thus pointing to a characteristic of allnormative models: They not only permit us to estimate future values of one or moreoutcome variables but also to optimize the attainment of values.

Verbal ModelsNormative and descriptive models may be expressed in three ways: verbally,symbolically, and procedurally.39 Verbal models are expressed in everyday language,rather than the language of symbolic logic (if p � q and q � r, then p � r) ormathematics (log1010,000 � 3). In using verbal models, analysts may make

Sn = S0(1 + r)n



reasoned judgments in the form of predictions and offer recommendations.Reasoned judgments are parts of policy arguments, rather than results presented inthe form of numbers. Verbal models are relatively easily communicated amongexperts and laypersons alike, and their costs are low. A potential limitation of verbalmodels is that the reasons offered for predictions and recommendations may beimplicit or hidden, making it difficult to reconstruct and critically examinearguments as a whole. The arguments for and against the blockade of the Sovietnavy during the Cuban Missile Crisis are good examples of competing verbal policymodels. President Kennedy’s own verbal model of the crisis argued, in effect, that ablockade was the United States’ only real option:

Above all, while defending our own vital interests, nuclear powers must avertthose confrontations which bring an adversary to a choice of either a humiliat-ing retreat or a nuclear war. To adopt that kind of course in the nuclear agewould be evidence only of the bankruptcy of our policy—of a collective deathwish for the world.40

Symbolic ModelsSymbolic models use mathematical symbols to describe relationships amongkey variables that are believed to characterize a problem. Predictions or optimalsolutions are obtained from symbolic models by employing methods of mathemat-ics, statistics, and logic. Symbolic models are difficult to communicate to laypersons,including policy makers; even among expert modelers, there are misunderstandingsabout basic elements of models.41 The costs of symbolic models are probably notmuch greater than those of verbal models, provided one takes into account the timeand effort expended on public debate, which is the main vehicle for expressingverbal models. A practical limitation of symbolic models is that their results may notbe easily interpretable, even among specialists, because the meanings of symbols andthe assumptions of symbolic models may not be adequately defined. Symbolicmodels may improve policy decisions, but only if

the premises on which models are constructed are made explicit. . . . All toofrequently what purports to be a model based on theory and evidence is nothingmore than a scholar’s preconception and prejudices cloaked in the guise ofscientific rigor and embellished with extensive computer simulations. Withoutempirical verification there is little assurance that the results of such exercisesare reliable, or that they should be applied for normative policy purposes.42

39Models may also be expressed physically, as when various materials are used to construct representa-tions of human organs, cities, or machines. The basic limitation of such models is that they cannotrepresent human action, which involves communication processes, social learning, and choice.40Quoted in Graham T. Allison, “Conceptual Models and the Cuban Missile Crisis,” American PoliticalScience Review 63, no. 3 (1969): 698.41See Greenberger and others, Models in the Policy Process, pp. 328–36.42Gary Fromm, “Policy Decisions and Econometric Models,” cited by Greenberger and others, Modelsin the Policy Process, p. 72.



Although we have already considered a simple symbolic model designed fornormative purposes (compound interest), the purpose of other symbolic models isdescriptive. A frequently used symbolic model is the simple linear equation

where Y is the value of the dependent variable the analyst seeks to predict and X isthe value of an independent policy variable that may be manipulated to achieve Y.The relation between X and Y is known as a linear function, which means that rela-tions between X and Y form a straight line when plotted on a graph (Figure 4). Inthis model, the symbol b denotes the amount of change in Y due to a change in X.This may be depicted by the slope of a straight line when plotted on a piece of paper(the steeper the slope, the greater the effect of X on Y). The symbol a (called anintercept constant) denotes the point where the straight line intercepts the vertical orY-axis when X is zero. In Figure 4, values of Y are one-half those of X along the bro-ken line (i.e., X), whereas along the solid line, each change in X pro-duces a 1.0-unit change in Y (i.e., X). This linear model predicts howmuch change in the policy variable (X) is required to produce a given value of theoutcome variable (Y).

Procedural ModelsProcedural models represent dynamic relationships among variables believed tocharacterize problems and solutions. Predictions and optimal solutions are obtainedby simulating and searching through sets of possible relationships—for example,economic growth, energy consumption, and food supplies in future years—thatcannot be adequately described because reliable data are unavailable. Simulation

Y = 0 + 1.0Y = 0 + 0.5

Y = a + bX

Y = 0.0 + 1.0X

Y = 0.0 + 0.5X

Y (Policy Outcome Variable)10

9

8

7

6

5

4

3

2

1

0 1 2 3 4 5 6 7 8 9 10

X (Policy Variable)FIGURE 4A symbolic model



and search procedures are generally performed with the aid of a computer that isprogrammed to yield alternative predictions under different sets of assumptions.

Procedural models also make use of symbolic modes of expression. The maindifference between symbolic and procedural models is that the former often useactual data to estimate relationships among policy and outcome variables, whereasprocedural models assume (simulate) such relationships. The costs of proceduralmodels are relatively high, as compared with verbal and symbolic models, largelybecause of the time required to develop and run computer programs. At the sametime, procedural models may be written in reasonably nontechnical language, thusfacilitating communication among laypersons. Although the strength of proceduralmodels is that they permit creative simulation and search, it is sometimes difficult tofind evidence to justify model assumptions.

A simple form of procedural model is the decision tree, which is created by pro-jecting several possible consequences of policies. Figure 5 illustrates a simple deci-sion tree that estimates the probability that each of several policy alternatives willreduce pollution.43 Decision trees are useful in comparing subjective estimates of thepossible consequences of various policy choices when it is difficult to calculate riskand uncertainty on the basis of existing data.

Models as Surrogates and PerspectivesA final important dimension of policy models relates to their assumptions. Policymodels, irrespective of their purpose or mode of expression, may be viewed as

43See Gass and Sisson, A Guide to Models in Governmental Planning and Operations, pp. 26–27.

RegulateIndustry

Pollution UpMore Regulation

Add Education

More Regulation

Add Education

Add Regulation

More Education

Add Regulation

More Education

(0.5)

Pollution Up

(0.8)

Pollution Down

(0.5)

Pollution Down

(0.2)

EducateConsumers

Key Decision Being Analyzed

Consequence Point

Probability of Consequence( )

Probable Future Decision Point

FIGURE 5Simulation model



surrogates or perspectives.44 A surrogate model is assumed to be a substitute for asubstantive problem. Surrogate models are based, consciously or unconsciously, onthe assumption that the formal problem is a valid representation of the substantiveproblem. By contrast, perspective models are viewed as one among several possibleways to structure a substantive problem. Perspective models assume that the formalproblem can never be a wholly valid representation of substantive problems.

The distinction between surrogate and perspective models is particularlyimportant in public policy analysis, where, as we have seen, many of the mostimportant problems are ill structured. The structure of most public policy problemsis sufficiently complex that the use of surrogate models significantly increases theprobability of errors of the third type (EIII)—that is, solving the wrong formulationof a problem when one should have solved the right one. This point may be clari-fied by considering two illustrations of formal modeling in policy analysis. The firstof these illustrations is concerned with the use of symbolic models, whereas thesecond deals with verbal models.45

Suppose that an analyst has constructed a simple symbolic model expressed inthe form of a linear equation such as that described earlier in Figure 4. Using theequation (later we shall write this equation as ). Theanalyst may plot values of X and Y on the scatterplot, as shown in Figure 6. Supposealso that the analyst assumes implicitly that the observed values of X and Y consti-tute a causal relationship, where the policy variable (X) is believed to produce a

Y = b0 + bx + eY = a + bX

44Strauch, “A Critical Look at Quantitative Methodology,” pp. 136–44.45These illustrations are adapted from Strauch, “A Critical Look at Quantitative Methodology,”pp. 131–33; and Watzlawick, Weakland, and Fisch, Change.

Y (Rainfall Rate)10

9

8

7

6

5

4

3

2

1

0 1 2 3 4 5 6 7 8 9 10

X (Depth of Reservoir)

Observed Y Valuesoff Regression Line

Predicted Y Valueson Regression Line

FIGURE 6Assumed effects of X on Y



significant change in an outcome variable (Y). At this point, the analyst will verylikely interpret the results of the formal symbolic model as a confirmation of thestructure of the substantive problem. For example, the analyst will interpret theslope of the line as a measure of the effects of X on Y, whereas the correlationbetween observed values of Y (data points) and those predicted by the equation(those lying on the straight line) will be taken as an estimate of the accuracy of theprediction. The probable conclusion is that a change in the value of the policyvariable will result in a corresponding change in the value of the outcome variable.

The point of this illustration is that the formal symbolic model provides noguidance in answering the question of whether X causes Y. If X happens to beunemployment and Y poverty, there may be a case for the predicted relation,provided the substantive problem has been defined in such a way as to provideplausible reasons for believing that unemployment is causally relevant to poverty. Yetthis causal information is not contained in the symbolic model; it comes from outsidethe model. The observed relation between X and Y might just as easily be interpretedas evidence of the effects of poverty on unemployment, provided the substantiveproblem has been defined in terms of the assumption that poverty is not an economicphenomenon but a cultural one. Poverty, for example, may be defined in terms of a“culture of poverty” that depresses the motivation to work.

The lesson of this illustration is that the conceptualization of substantiveproblems governs the specification of formal symbolic models. Formal representationsof substantive problems may conceal the fact that the formal model is a perspective of“reality” and not a surrogate. To clarify the point, consider the formal symbolic modelpreviously discussed:

Suppose, for example, that X is the mean annual depth of a reservoir and that Y is the annual rainfall in the area. . . . Because reservoir-management policiescan be changed, reservoir depth is a policy variable subject to policy manipulation.Annual rainfall is a variable we might be interested in controlling, and commonsense clearly suggests that the relationship between rainfall and reservoir depthis nonspecious. . . . In spite of this, however, the conclusion suggested by theanalysis—that we can decrease rainfall by draining water more rapidly from thereservoir . . . seems ludicrous. This is because the causal relationship assumed inthe analysis—reservoir depth causes rainfall—runs counter to our commonsense understanding that rainfall determines reservoir depth.46

Consider now a second illustration of the difficulties that arise when we confusesurrogate and perspective models. This time we will use an illustration that makesthe same point about verbal models, that is, models that are expressed in everydaylanguage. Suppose that we are confronted with the following problem situation: Thedirector of the state Department of Transportation has requested that a study beundertaken to recommend the least costly way to connect nine key transportationpoints in the central region of the state. As the agency’s policy analyst, you aredirected to show how all nine points may be connected by four sections of highway.You are also told that these four sections of highway must be straight (no curves will

46Strauch, “A Critical Look at Quantitative Methodology,” p. 132.



be permitted) and that each new section must begin at the point where the lastsection stopped (the construction team will not be permitted to retrace its steps).You are then shown a map of the region (Figure 7) and asked to make a recommen-dation that will solve the director’s problem.

Unless you were already familiar with this classic conceptual problem (calledsimply the “nine-dot problem”), it is unlikely that you were able to solve it. Fewpeople manage to find the solution by themselves, not because the problem istechnically complicated (in fact it is simple) but because they almost always commitan error of the third type (EIII), solving the wrong formulation of a problem whenthey should have solved the right one. This is because most people approach theproblem situation with implicit assumptions that make it impossible to solve theproblem. The central point, however, is that these assumptions are introduced byanalysts themselves; they are not part of the problem situation. In other words,analysts themselves create a substantive problem that cannot be solved.

The solution for the nine-dot problem is presented in Figure 8. The solution ap-pears surprisingly simple, novel, and unconventional; that is, it has several of thekey characteristics of creativity discussed earlier. In the process of analysis, itbecomes suddenly clear that we have been solving the wrong problem. So long as weassume that the solution must be found within the boundaries set by our verbalmodel—that is, the rectangle composed of nine dots—then a solution is not possible.Yet this condition is imposed not by the formal verbal model but by the implicitassumption of “squareness” that unconsciously shaped our definition of thesubstantive problem. Imagine what would have occurred if we had transformed the

AlexandriaMarysville

DithridgeMorewood

O'Hara

Elizabethtown

Bunker Hill

WillamsburgOakhurst

FIGURE 7Map of transportation points in central region

FIGURE 8Solution for nine-dot problem

2

1

4

3



verbal model into a symbolic one, for example, by using plane geometry to derivequantitative estimates of the distance between points. This would not only have ledus further and further away from the solution, it may have also created an aura ofscientific rigor and precision that would lend authority to the conclusion that theproblem is “insoluble.” Finally, this illustration helps convey a simple but importantpoint about the use of verbal, symbolic, and procedural models: Formal modelscannot themselves tell us whether we are solving the wrong formulation of aproblem. The map is not the territory! The menu is not the meal!

METHODS OF PROBLEM STRUCTURINGProblem structuring is the process of generating and testing alternative substantiveproblems. As we saw in Figure 3, problem structuring has four interrelated phases:problem sensing, problem search, problem definition, and problem specification.A number of methods and techniques are helpful in carrying out problem-structur-ing activities in each phase. These methods, along with their respective aims, pro-cedures, sources of knowledge, and criteria of performance, are displayed in Table2.

Boundary AnalysisAn important task of problem structuring is estimating whether the system of individualproblem formulations that we have called a metaproblem is relatively complete. Thistask is similar to the situation of the homesteaders described by Kline in his essays onthe myth of certainty in mathematics.47 The homesteaders, while clearing their land, areaware that enemies lurk in the wilderness that lies just beyond the clearing. To increasetheir security, the homesteaders clear a larger and larger area but never feel completelysafe. Frequently, they must decide whether to clear more land or attend to their cropsand domesticated animals within the boundaries of the clearing. They do their best topush back the wilderness but know full well that the enemies lurking beyond the clear-ing could surprise and destroy them. They hope that they will not choose to tend thecrops and livestock when, instead, they should have chosen to clear more land.

The analogy of the homesteaders accentuates a key problem of problem struc-turing in policy analysis. Policy analysts are rarely faced with a single, well-definedproblem; rather, they are faced with multiple problems that, distributed throughoutthe policy-making process, are defined in distinctly different ways by stakeholderswhose perspectives and actions are interdependent. Under these circumstances,analysts appear as homesteaders working within unmanageable boundaries, or asmodern counterparts of Diogenes, engaged in “a never-ending discourse with reality,to discover yet more facets, more dimensions of action, more opportunities forimprovement.”48

47Morris Kline, Mathematics: The Loss of Certainty (New York: Oxford University Press, 1980).See also the critical essays on uncertainty and creativity in modern mathematics by Michael Guillen,Bridges to Infinity (Ithaca, NY: Cornell University Press, 1988).48Dery, Problem Definition in Policy Analysis, pp. 6–7.


TAB

LE

2

Com

pari

son

of M

etho

ds o

f P

robl

em S

truc

turi

ng

Met

hod

Aim

Pro

cedu

reS

ourc

e of

Kno

wle

dge

Per

form

ance

Cri

teri

on

Bou

ndar

y A

naly

sis

Est

imat

e bo

unda

ries

of

met

apro

blem

Sat

urat

ion

sam

plin

g an

dpr

oble

m e

licit

atio

nS

take

hold

ers

Cor

rect

ness

-in-

limit

Cla

ssif

icat

ion

Ana

lysi

sC

lari

fica

tion

of

conc

epts

Log

ical

div

isio

n an

dcl

assi

fica

tion

of

conc

epts

Ana

lyst

Rul

es o

f lo

gic

Hie

rarc

hy A

naly

sis

Iden

tify

pos

sibl

e,pl

ausi

ble,

and

acti

onab

le c

ause

sL

ogic

al d

ivis

ion

and

clas

sifi

cati

on o

f ca

uses

Ana

lyst

Rul

es o

f lo

gic

Syn

ecti

csR

ecog

niti

on o

f si

mila

riti

esam

ong

prob

lem

sC

onst

ruct

ion

of p

erso

nal,

dire

ct,s

ymbo

lic,a

nd f

anta

syan

alog

ies

Gro

upP

laus

ibili

ty

Bra

inst

orm

ing

Gen

erat

ion

of id

eas,

goal

s,an

d st

rate

gies

Idea

gen

erat

ion

and

eval

uati

onG

roup

Con

sens

us

Mul

tipl

e P

ersp

ecti

veA

naly

sis

Gen

erat

ion

of in

sigh

tU

se o

f te

chni

cal,

orga

niza

tion

al,a

nd p

erso

nal

pers

pect

ives

Gro

upC

onse

nsus

Ass

umpt

ion

Ana

lysi

sC

reat

ive

synt

hesi

s of

conf

licti

ng a

ssum

ptio

nsS

take

hold

er id

enti

fica

tion

,as

sum

ptio

n su

rfac

ing,

chal

leng

ing,

and

synt

hesi

s

Gro

upC

onfli

ct

Arg

umen

t M

appi

ngC

riti

cal t

hink

ing

Pla

usib

ility

ana

lysi

s of

argu

men

tsG

roup

Pla

usib

ility



To make effective use of the methods and techniques of problem structuringdescribed in this chapter, it is important to conduct a boundary analysis. Methods ofproblem structuring just discussed, along with related methods that presuppose that theproblem has already been structured,49 do not themselves provide any way to knowwhether a set of problem formulations is relatively complete. In effect, there is no“stopping rule” that enables analysts to know when the search should be concluded.

The relative completeness of a set of problem formulations may be estimated bymeans of a three-step process:50

1. Saturation sampling. A saturation (or snowball) sample of stakeholders maybe obtained by a process that begins with a set of individuals and groupsknown to differ on a policy. Stakeholders in this initial set may be contacted,face-to-face or by telephone, and asked to name two additional stakeholderswho agree most and least with the arguments and claims discussed. Theprocess is continued until no new stakeholders are named. At this point, thereis no sampling variance, because all members of the working universe ofpolicy-relevant stakeholders in a specific area (e.g., a bill addressing healthcare reform) have been contacted.51

2. Elicitation of problem representations. This step is designed to elicit the alter-native problem representations that Heclo has described as the “ideas, basicparadigms, dominant metaphors, standard operating procedures, or whateverelse we choose to call the systems of interpretation by which we attach meaningto events.”52 The evidence required to characterize these problem representa-tions may be obtained from face-to-face interviews or, more realistically underthe time constraints facing most analysts, from telephone conversations anddocuments requested from stakeholders in the saturation sampling phase.

3. Boundary estimation. This step is to estimate the boundaries of themetaproblem. Here the analyst constructs a cumulative frequency distributionwhere stakeholders are arrayed on the horizontal axis and the number of newproblem elements—ideas, concepts, variables, assumptions, objectives,policies—are plotted on the vertical axis (Figure 9). As the new and nondu-plicative problem elements of each stakeholder are plotted, the slope of thecurve displays different rates of change. An initial rapid rate of change isfollowed by slow change and eventually stagnation, which is the point atwhich the curve becomes flat. After this point, the collection of additional

49These other methods include the analytic hierarchy process, interpretive structural modeling, policycapturing, and Q-methodology. See, respectively, Thomas L. Saaty, The Analytic Hierarchy Process(New York: McGraw-Hill, 1980); John N. Warfield, Societal Systems: Planning, Policy, and Complexity(New York: Wiley, 1976); Kenneth R. Hammond, Judgment and Decision in Public Policy Formation(Boulder, CO: Westview Press, 1977); and Stephen R. Brown, Political Subjectivity: Applications ofQ-Methodology in Political Science (New Haven, CT: Yale University Press, 1980). Computer softwareis available for applications of these methods.50See William N. Dunn, “Methods of the Second Type: Coping with the Wilderness of ConventionalPolicy Analysis,” Policy Studies Review 7, no. 4 (1988): 720–37.51On these points as they apply to sociometric and saturation sampling in general, see Seymour Sudman,Applied Sampling (New York: Academic Press, 1976).52Hugh Heclo, “Policy Dynamics,” in The Dynamics of Public Policy, ed. Richard Rose (Beverly Hills,CA: Sage Publications, 1976), pp. 253–54.



information about the nature of the problem is unlikely to improve thecompleteness of the problem representation, because the boundary of the metaproblem has been estimated.

The estimation procedures described satisfy requirements for sound inductiveestimates in general—character, coordination, cost-effectiveness, and correctness-in-the-limit.53 Applications of similar procedures in other areas—for example, esti-mates of boundaries of scientific literature, library holdings, languages, literaryworks, consumer preferences—suggest lawful regularities in the patterns and limitsof growth in knowledge systems.54 Boundary analysis, like other policy-analytic

53See Nicholas Rescher, Induction (Pittsburgh, PA: University of Pittsburgh Press, 1980), pp. 24–26. Forthe detailed argument relating these requirements to problem structuring in policy analysis, see Dunn,“Methods of the Second Type.”54For a brief but provocative essay on these regularities, see Herbert A. Simon, “The Sizes of Things,”in Statistics: A Guide to the Unknown, ed. Judith M. Tanur and others (San Francisco, CA: Holden-Day, 1972), pp. 195–202.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Stakeholder

Boundary

Percen

tC

um

ula

tive

Fre

qu

ency

16 17 18 19 20 21 22 23 24 25 26 27 28 29 300%

20%

40%

60%

80%

100%

0

20

40

60

80

100

FIGURE 9Pareto chart—estimated boundary of problem



procedures, yields results that are plausible and not certain. In conjunction withother problem structuring methods and techniques, these boundary estimationprocedures reduce the likelihood of type III errors in policy analysis.

Classification AnalysisClassification analysis is a technique for clarifying concepts used to define andclassify problem situations.55 In sensing a problem situation, analysts must some-how classify their experiences. Even the simplest descriptions of problem situationsare based on the classification of experience through inductive reasoning, a processwhereby general (abstract) concepts, such as poverty, crime, and pollution, areformed by experiencing particular (concrete) objects or situations. When we classifya problem situation in one way, we foreclose opportunities to classify it in anotherway, as the nine-dot problem illustrates so well.

Classification analysis is based on two main procedures: logical division andlogical classification. When we select a class and break it down into its componentparts, the process is called logical division; the reverse process, which involvescombining situations, objects, or persons into larger groups or classes, is calledlogical classification. The basis of any classification depends on the analyst’s pur-pose, which in turn depends on substantive knowledge about a problem situation.

Consider, for example, the analysis of poverty in the United States. All povertyhouseholds in the United States may be broken down into three age categories: lessthan 18 years, more than 65 years, and 18 to 65 years. We can also look at allpoverty households. If we stop at this point in the process of logical division, we willreach the conclusion that poverty in the United States is about the same in 2006 asit was in 1968, when President Lyndon Johnson’s War on Poverty was implemented.When the process of logical division is carried one step further, and poverty familiesare divided into the three classes on the basis of age, we reach an altogether differentconceptualization of the problem. Here we conclude that poverty has becomeslightly worse for adults ages 18 to 64 and for children under 18 years of age. Bycontrast, poverty has declined by more than 60 percent among those older than 64.The welfare reforms implemented under President Clinton have benefited the eld-erly, hurt children, and had virtually no effect on persons aged 18 to 64. It appearsthat poverty is not alleviated by the private enterprise system, because the number offamilies below the poverty line decreased only for elders, only 6 percent of whomearn employment income and almost all have the benefits of government support(Table 3 and footnote).

There is no way to know with certainty if the bases of a classification system arethe right ones. But several rules help ensure that a classification system is bothrelevant to a problem situation and logically consistent:

1. Substantive relevance. The basis of a classification should be developedaccording to the analyst’s purpose and the nature of the problem situation.This rule, deceptively simple in theory, means that classes and subclassesshould conform as closely as possible to the “realities” of the problem

55See John O’Shaughnessy, Inquiry and Decision (New York: Harper & Row, 1973), pp. 22–30.



situation. Yet as what we know about a situation is partly a function of theconcepts we use to experience it, there are no absolute guidelines that tell uswhen we have perceived a problem correctly. Poverty, for example, maybe classified as a problem of inadequate income, cultural deprivation, orpsychological motivation—it may be all of these and more.

2. Exhaustiveness. Categories in a classification system should be exhaustive.This means that all subjects or situations of interest must be “used up,” so tospeak. In the previous example of poverty, all persons in the United States mustfit into one or another of the various categories. If we discover that some haveno income, either because they make no effort to work or because they relyexclusively on government transfers, a new category might be created.

3. Disjointness. Categories must be mutually exclusive. Each subject or situa-tion must be assigned to one and only one category or subcategory. In classify-ing families, for example, they must fall into one or the other of the two mainsubcategories (income above and below the poverty line), which means that nofamily can be “double-counted.”

4. Consistency. Each category and subcategory should be based on a single clas-sificational principle. A violation of this rule leads to overlapping subclassesand is known as the fallacy of cross-division. For example, we commit the fal-lacy of cross-division if we classify families according to whether they areabove the poverty line or receive welfare payments, because many families fallinto both categories. This rule is actually an extension of rules of exhaustive-ness and disjointness.

5. Hierarchical distinctiveness. The meaning of levels in a classification system(categories, subcategories, sub-subcategories) must be carefully distinguished.The rule of hierarchical distinctiveness, which is a guideline for interpretingclassification systems, is derived from the principle discussed previously:Whatever involves all of a collection must not be one of the collection.

TABLE 3

Number of Households Living below Poverty Level,1968–2006*

Age Groups 1968 1990 2006

All 12.8% 13.5% 12.3%< 18 15.4 20.6 17.418–64 9.0 10.7 10.8> 64 25.0 12.1 9.4

*In 2011, the U.S. Bureau of the Census defined poverty as an individual earning lessthan $10,890, with an added $3,820 per household member ($22,350 for a family offour). Data exclude government transfers in the form of cash payments (SocialSecurity, public assistance), nutrition (food stamps), housing, health (Medicaid,Medicare), social services (OEO), employment and workforce, and education.

Source: Daniel R. Meyer and Geoffrey L.Wallace,“Poverty Levels and Trends inComparative Perspective,” Focus 26, 2 (2009). Focus is a publication of the University ofWisconsin Institute for Research on Poverty.



Humankind is the class of all individuals, but it is not itself an individual.Similarly, poverty is a characteristic of 45 million persons, but they cannot beunderstood in terms of the behavior of one person multiplied 45 million times.Thus, a population of 45 million persons is not just quantitatively differentfrom a single family, it is also qualitatively different because it involves a wholesystem.

The rule of hierarchical distinctiveness is central to problem structuring.In structuring policy problems, it frequently happens that analysts ignore thedistinction between member and class and that a class cannot be a member of itself.This point is best illustrated by returning to the nine-dot problem (Figure 7). Whena person first attempts to solve this problem, the

assumption is that the dots compose a square and that the solution must befound within that square, a self-imposed condition which the instructions donot contain. His failure, therefore, does not lie in the impossibility of the task,but in his attempted solution. Having now created the problem it does notmatter in the least which combination of four lines he now tries, and in whatorder; he always finishes with at least one unconnected dot. This means that hecan run through the totality of first-order change possibilities [i.e., those thatare confined to the level of members of the class defined as a square] . . . butwill never solve the task. The solution is a second-order change [i.e., one thatinvolves all of the class] which consists in leaving the field and which cannot becontained within itself because . . . it involves all of a collection and cannot,therefore, be part of it.56

A useful approach to classification analysis is set thinking.57 Set thinking involves thestudy of relations of sets to each other and to subsets, where a set is defined as aclearly delimited collection of objects or elements. Sets and subsets are the equiva-lents of classes and subclasses in a classification system and are expressed visuallywith aid of Venn diagrams.58 In the Venn diagram in Figure 10, the rectangle mightbe used to represent all families in the United States. In set language, it is known asthe universal set (U). Two of its component sets, shown as circles A and B in the

A � B � �

�

A B

56Watzlawick et al., Change, p. 25. See also Gregory Bateson, Steps to an Ecology of Mind (New York:Ballantine Books, 1972).57Set theory, created by the German mathematician-logician Georg Canter (1874–97), is the mathemati-cal theory of collections of aggregates of entities.58Venn diagrams, used extensively to illustrate set problems, are named after the English logician JohnVenn (1834–1923).

FIGURE 10Set union



rectangle, might be used to represent families above and below the poverty line. If weapply rules of exhaustiveness, disjointedness, and consistency, all families will bedivided into one or the other of the sets, A and B, so that the two sets reflect distinctlevels of income that do not overlap. In set language, the union of A and B is equal tothe universe (U) of all families. The symbol for union is �, read as “union” or “cup.”

Two sets intersect to form a subset, so that the properties of the original two setsoverlap, as shown in Figure 11. For example, the intersection (D) of nonpoverty (A)and poverty (B) families might be used to illustrate that some families in each groupreceive government transfer payments. In set language, the intersection of A and B isequal to D, which is expressed symbolically as A � B � D and read “A intersect (or cap) B equals D.” Union and intersection are the two most impor-tant set operations and may be used to construct classification schemes (Figure 12)and crossbreaks (Figure 13). Crossbreaks are a basic form of logical division used toorganize data in tables.

A B

D

A � B � D

� FIGURE 11Set intersection

Families

Non-Poverty Poverty

Do NotReceive

GovernmentTransfer

Payments

ReceiveGovernment

TransferPayments

Do NotReceive

GovernmentTransfer

Payments

ReceiveGovernment

TransferPayments

FIGURE 12Classification scheme

A1

A1 � non-poverty familiesA2 � poverty familiesB1 � do not receive transfer paymentsB2 � receive transfer payments

B1A1B1

B2 B2A1

A2

B1A2

B2A2

FIGURE 13Crossbreak



Venn diagrams, classification schemes, and crossbreaks are importanttechniques for structuring problems. Procedures of classification analysis, however,are employed mainly by individual analysts, rather than groups, and logical consis-tency is the criterion for assessing how well a problem has been conceptualized.Although logical consistency is an important aspect, there is no way to know withconfidence that the substantive basis of a category or subcategory is the right one.Even though classification analysis is a technique for improving the clarity ofconcepts and their relationships, it does not guarantee that concepts will havesubstantive relevance.

Hierarchy AnalysisHierarchy analysis is a technique for identifying possible causes of a problem.59

Formal logic and many social science theories provide little guidance in identifyingpossible causes. There is no acceptable way to formally deduce causes from effects,or effects from causes, and social science theories are frequently so general as to beof little help in specific situations. In order to identify possible causes, it is useful tohave conceptual frameworks that identify the many causes that may be operating ina given situation.

Hierarchy analysis helps identify three kinds of causes: possible causes, plausi-ble causes, and actionable causes. Possible causes are events or actions that,however remote, may contribute to the occurrence of a given problem situation. Forexample, resistance to work, unemployment, and the distribution of power andwealth among elites may all be taken as possible causes of poverty. By contrast,plausible causes are those that, on the basis of scientific research or direct experi-ence, are believed to exert an important influence on the occurrence of a situationjudged to be problematic. In the preceding example, resistance to work is unlikely tobe regarded as a plausible cause of poverty, at least among experienced observers,whereas unemployment is. Finally, the distribution of power and wealth amongelites is unlikely to be viewed as an actionable cause—that is, one that is subject tocontrol or manipulation by policy makers—because we cannot alter the socialstructure of an entire society. In this example unemployment is both a plausible andactionable cause of poverty.

Political scientists Stuart Nagel and Marian Neef provide a classic example ofthe uses of hierarchy analysis to structure policy problems. Many observers havebeen ready to accept the explanation that the major cause of overcrowded jails is thelarge number of persons arrested and detained in jail while awaiting trial. For thisreason, a policy of pretrial release—that is, a policy that provides for the release of acertain number of those arrested (usually for less serious offenses) prior to a formaltrial—has been favored by many reformers.

The difficulty with this policy, as well as the causal explanation on which it isbased, is that it overlooks plea bargaining as one among several plausible causes ofjail overcrowding. In plea bargaining, which is widely practiced in the U.S. judicialsystem, a defendant agrees to plead guilty in return for a prosecutor’s agreement to

59See O’Shaughnessy, Inquiry and Decision, pp. 69–80.



reduce the charge or the sentence. When plea bargaining is taken into account alongwith pretrial release, the following consequences may occur:

If the percentage of defendants released prior to trial goes up, then thepercentage of successful plea bargains will probably go down. . . . Now, ifguilty pleas go down as a result of increased pretrial release, then the numberof trials will probably go up. . . . And if the number of trials increases, then thedelay in going to trial will also increase, unless the system increases its quan-tity of prosecutors, judges, and public defenders. . . . If this delay in going totrial increases for cases in general, including defendants in jail, then the jailpopulation may increase, since its size depends on the length of time thatdefendants are kept in jail as well as on the quantity of defendants who gothere. Any decrease in the jail population as a result of increased pretrialrelease may be more than offset by the increased delay and length of pretrialdetention caused by the increased pretrial release and the consequentreduction in guilty pleas and increase in trials.60

This classic example not only illustrates the potentially creative role of hierarchyanalysis in structuring policy problems but also shows how hierarchy analysis cancontribute to the discovery of possible unanticipated consequences of public policieswhose effects seem self-evident. What could be more obvious than that pretrialrelease will result in a reduction of the jail population? The answer depends onhaving a satisfactory understanding of the plausible causes that contribute to theoriginal problem situation. Figure 14 provides a simple illustration of hierarchyanalysis applied to possible, plausible, and actionable causes of fires.

The rules for conducting a hierarchy analysis are the same as those used forclassification analysis: substantive relevance, exhaustiveness, disjointness, consistency,and hierarchical distinctiveness. Similarly, procedures of logical division and classifica-tion also apply to both types of analysis. The major difference between classificationanalysis and hierarchy analysis is that the former involves the division and classifica-tion of concepts in general, whereas hierarchy analysis builds particular concepts ofpossible, plausible, and actionable causes. Nevertheless, both forms of analysis focuson the individual analyst and use logical consistency as the primary criterion forassessing how well a problem has been conceptualized, and neither guarantees that thecorrect substantive basis for concepts will be found. Thus, hierarchy analysis may alsoforeclose opportunities to generate alternative causal explanations by relying on theindividual analyst, rather than on groups, as a source of knowledge.

SynecticsSynectics is a method designed to promote the recognition of analogous problems.61

Synectics, which refers broadly to the investigation of similarities, helps analysts

60Stuart S. Nagel and Marian G. Neef, “Two Examples from the Legal Process,” Policy Analysis 2, no. 2 (1976): 356–57.61See W. J. Gordon, Synectics (New York: Harper & Row, 1961); and Hayes, Cognitive Psychology,pp. 72, 241.



make creative use of analogies in structuring policy problems. Many studies showthat people frequently fail to recognize that what appears to be a new problem isreally an old one in disguise and that old problems may contain potential solutionsfor problems that appear to be new. Synectics is based on the assumption that anawareness of identical or similar relationships among problems will greatly increasethe problem-solving capacities of analysts.

In structuring policy problems, analysts may produce four types of analogies:

1. Personal analogies. In constructing personal analogies, analysts attempt toimagine themselves experiencing a problem situation in the same way as doessome other policy stakeholder, for example, a policy maker or client group.Personal analogies are especially important in uncovering political dimensions ofa problem situation, for “unless we are willing and able to think ‘politically’—ifonly as a matter of role playing—we will not be able to enter the phenomenolog-ical world of the policymaker and understand the policy process.”62

Human Action

Nonhuman Event

Direct

Indirect

Electrical

Radiant Heat

ChemicalReaction

LightingAccident

Heat ofMovingParts

AutoElectricSystem

Smoking

Oxidation

Other

Candle

Oil Lamp

Cigarette

Match

DriveShaft

Motor

Spark

Short

Heater

Radiator

Gas Heater

Solar Rays

Paint

Chemicals

Oily Rags

Short

FIGURE 14Hierarchy analysis of the causes of firesSource: John O’Shaughnessy, Inquiry and Decision (New York: Harper & Row, 1973), p. 76.

62Raymond A. Bauer, “The Study of Policy Formation: An Introduction,” in The Study of PolicyFormation, ed. R. A. Bauer and K. J. Gergen (New York: Free Press, 1968), p. 4.



2. Direct analogies. In making direct analogies, the analyst searches for similarrelationships among two or more problem situations. In structuring problemsof drug addiction, for example, analysts may construct direct analogies fromexperiences with the control of contagious diseases.63

3. Symbolic analogies. In making symbolic analogies, the analyst attemptsto discover similar relationships between a given problem situation andsome symbolic process. For example, symbolic analogies are often drawnbetween servomechanisms of various kinds (thermostats, automatic pilots)and policy processes. In each case, analogous processes of adaptation areviewed as consequences of continuous feedback from the environment.64

4. Fantasy analogies. In making fantasy analogies, analysts are completelyfree to explore similarities between a problem situation and some imaginarystate of affairs. Defense policy analysts, for example, have sometimesused fantasy analogies to structure problems of defense against nuclearattack.65

Synectics relies on individual analysts and groups to make appropriate analogies.The main criterion for assessing how well a problem has been conceptualized is theplausibility of comparisons, that is, the degree to which a given problem situation isactually similar to others taken as analogies.

BrainstormingBrainstorming is a method for generating ideas, goals, and strategies that helpidentify and conceptualize problem situations. Originally designed by Alex Osbornas a means to enhance creativity, brainstorming may be used to generate a largenumber of suggestions about potential solutions for problems.66 Brainstorminginvolves several simple procedures:

1. Brainstorming groups should be composed in accordance with the nature ofthe problem situation being investigated. This usually means the selection ofpersons who are particularly knowledgeable about the given situation, that is,experts.

2. Processes of idea generation and idea evaluation should be kept strictly apart,because intense group discussion may be inhibited by premature criticism anddebate.

3. The atmosphere of brainstorming activities should be kept as open andpermissive as possible in the idea-generating phase.

63See, for example, Mark H. Moore, “Anatomy of the Heroin Problem: An Exercise in ProblemDefinition,” Policy Analysis 2, no. 4 (1976): 639–62.64See, for example, David Easton, A Framework for Political Analysis (Englewood Cliffs, NJ: PrenticeHall, 1965).65See, for example, Herman Kahn, On Thermonuclear War (Princeton, NJ: Princeton University Press,1960). For a critique, see Philip Green, Deadly Logic: The Theory of Nuclear Deterrence (Columbus:Ohio State University Press, 1966).66Alex F. Osborn, Your Creative Power (New York: Charles Scribner, 1948).



4. The idea-evaluating phase should begin only after all ideas generated in thefirst phase have been exhausted.

5. At the end of the idea-evaluating phase, the group should prioritize ideas andincorporate them in a proposal that contains a conceptualization of theproblem and its potential solutions.

Brainstorming is a versatile procedure that may involve relatively structured orunstructured activities, depending on the analyst’s aims and the practical constraintsof the situation. Relatively unstructured brainstorming activities occur frequently ingovernment agencies and public and private “think tanks.” Here discussions of policyproblems are informal and spontaneous, involving the interaction of generalists andspecialists from several scientific disciplines or fields.67 Brainstorming activities mayalso be relatively structured, with various devices used to coordinate or focus groupdiscussions. These devices include the establishment of continuous decision semi-nars that, seeking to avoid the restrictive atmosphere of conventional committees,involve a team of highly motivated experts who meet with high frequency over anumber of years.68

Another device for coordinating and focusing brainstorming activities is theconstruction of scenarios, which are outlines of hypothetical future events that mayalter some problem situation. Scenario writing, which has been used to explorepotential military and political crises, involves the constructive use of the imagina-tion to describe some aspect of a future situation. There are two major types ofscenarios: operations-analytical and free form. In constructing a free-form scenario,the analyst is “an iconoclast, a model breaker, a questioner of assumptions, and—inrare instances—a fashioner of new criteria.”69 By contrast, an operations-analyticalscenario has limited aims:

Instead of building up a picture of unrestrained fiction or even of constructinga utopian invention that the author considers highly desirable, an operations-analytical scenario starts with the present state of the world and showshow, step by step, a future state might evolve in a plausible fashion out of thepresent one.70

A good example of a relatively structured brainstorming effort is the Year 2000Planning Program, a two-and-one-half-year project carried out in the U.S. Bureau ofthe Census.71 In this project, 120 self-selected participants from all levels and

67See Edgar F. Quade, Analysis for Public Decisions (New York: American Elsevier Publishing, 1975),pp. 186–88; and Olaf Helmer and Nicholas Rescher, On the Epistemology of the Inexact Sciences(Santa Monica, CA: RAND Corporation, February 1960).68Harold D. Lasswell, “Technique of Decision Seminars,” Midwest Journal of Political Science 4, no. 2(1960): 213–26; and Lasswell, The Future of Political Science (New York: Atherton Press, 1963).69Seyon H. Brown, “Scenarios in Systems Analysis,” in Systems Analysis and Policy Planning:Applications in Defense, ed. E. S. Quade and W. I. Boucher (New York: American Elsevier Publishing,1968), p. 305.70Olaf Helmer, Social Technology (New York: Basic Books, 1966), p. 10. Quoted in Quade, Analysisfor Public Decisions, p. 188.71See Ian I. Mitroff, Vincent P. Barabba, and Ralph II. Kilmann, “The Application of Behavioral andPhilosophical Technologies to Strategic Planning: A Case Study of a Large Federal Agency,”Management Science 24, no. 1 (1977): 44–58.



branches of the bureau, from secretaries to division heads and the director, wereasked to think as freely as possible about the future and to construct free-formscenarios, which to them indicated what the bureau should be like in the year 2000.Participants were asked to write group reports that were later integrated in a finalreport by an executive group composed of representatives of the individual groups.The final report was subsequently presented to the executive staff of the CensusBureau, as well as to the advisory committees of the American Statistical Association,the American Marketing Association, and the American Economic Association.

The Year 2000 Planning Program was successful on several counts. The reportwas received with moderate approval by all groups, and most members thoughtthat the program should be continued in some form, perhaps permanently. Twocreative products of the project were suggestions to establish an ombudsman (orombudswoman) to protect the interests of users of census data and to create aCensus University to develop and execute the bureau’s continuing education pro-grams. Those who were most positive about the report were oriented towardstrategic concerns involving ill-structured problems; less positive reactions camefrom persons with tactical or operational orientations toward well-structuredproblems. The program itself was made possible by a recognition among top-levelbureau staff, including the director, that the bureau was confronted by importantlong-range problems whose structure was complex and “messy.” Finally, althoughthe program involved significant resource allocations, there did not appear to bemajor risks in executing the program.

The main difference between brainstorming and other techniques for problemstructuring is that the focus is on groups of knowledgeable persons rather than individ-ual experts. Moreover, brainstorming activities are assessed not in terms of logical con-sistency or the plausibility of comparisons but according to consensus among membersof brainstorming groups. The major limitation of consensus as a criterion of perform-ance in problem structuring is that conflicts about the nature of problems may besuppressed, thus foreclosing opportunities to generate and evaluate potentially appro-priate ideas, goals, and strategies. Although the Year 2000 Planning Program sought tocreate an open and permissive atmosphere, the final evaluation of the program’ssuccess was based on consensus among authoritative decision makers (agency execu-tive staff) and experts (advisory committees of professional associations). In short, thisprogram and other relatively structured brainstorming activities provide no explicitprocedures to promote the creative use of conflict in structuring policy problems.

Multiple Perspective AnalysisMultiple perspective analysis is a method for obtaining greater insight intoproblems and potential solutions by systematically applying personal, organizational,and technical perspectives to problem situations.72 Seen as an alternative to the

72See Harold A. Linstone, Multiple Perspectives for Decision Making: Bridging the Gap betweenAnalysis and Action (New York: North-Holland Publishing, 1984); and Linstone and others, “TheMultiple Perspective Concept: With Applications to Technology Assessment and Other Decision Areas,”Technological Forecasting and Social Change 20 (1981): 275–325.



near-exclusive emphasis on so-called rational-technical approaches in planning,policy analysis, and other areas, multiple perspective analysis is expresslydesigned to address ill-structured policy problems. Although there are manycharacteristics of each of the three perspectives, their major features are asfollows:

1. Technical perspective. The technical (T) perspective views problems andsolutions in terms of optimization models and employs microeconomics,benefit-cost analysis, systems analysis, and decision analysis. The technicalperspective, said to be based on a scientific-technological worldview,emphasizes causal reasoning, objective analysis, prediction, and statisticalinference. An example of the T perspective, which notably minimizes orexcludes ethical considerations, is the decision to drop the atomic bomb onJapan. The problem was seen to be composed of five alternatives: bombingand blockade, invasion, atomic attack without warning, atomic attack afterwarning, and dropping the bomb on an uninhabited island. Given the goal ofunconditional surrender with a minimum loss of Allied lives and destructionof Japan, the T perspective resulted in the choice of the third alternative(atomic attack without warning).

2. Organizational perspective. The organizational (O) perspective viewsproblems and solutions as part of an orderly progression (with minor buttemporary crises) from one organizational state to another. Standard operat-ing procedures, rules, and institutional routines are major characteristics ofthe O perspective, which is often opposed to the T perspective and onlyminimally concerned with improving organizational performance. Thedecision to drop the atomic bomb shows how the O perspective differs fromthe T perspective. From an O perspective, a decision not to use the bombraised political fears, because the $2 billion in funding for the atomic bombwas expended without congressional approval. Dropping the bomb showedCongress that the funds were not wasted and inaugurated the Cold War withthe Soviet Union.

3. Personal perspective. The personal (P) perspective views problems andsolutions in terms of individual perceptions, needs, and values. A majorcharacteristic of the personal perspective is an emphasis on intuition,leadership, and self-interest as factors governing problem structuring. Theexample of the atomic bomb also shows how the P perspective suppliesinsights not available from either the T or the O perspective. In 1945 thenew president, Harry Truman, was an outsider to the FDR establishment,which had grown and solidified during Roosevelt’s three terms in office.Truman lacked the legitimacy and influence necessary to challenge theestablishment, including entrenched bureaucratic interests and policies, soearly in his presidency. A decision not to drop the atomic bomb would beperceived as a sign of weakness to contemporaries and to future historians.Truman, who had a strong sense of history, wanted to appear as a bold anddecisive leader.

Multiple perspective analysis is relevant to any sociotechnical problem found inareas of public policy making, corporate strategic planning, regional development,



and other domains. To employ multiple perspective analysis, Linstone andcolleagues have developed some of the following guidelines:

� Interparadigmatic mix. Form teams on the basis of an interparadigmatic ratherthan interdisciplinary mix. For example, a team composed of a businessperson, alawyer, and a writer is preferred to a team with an economist, a political scientist,and a psychologist. The interparadigmatic mix is preferable because it maximizesopportunities to find T, O, and P perspectives in the team.

� Balance among perspectives. In advance of planning and policy analysis, it isnot possible to decide how much emphasis to place on T, O, and P perspectives.As the team engages in its work, the discovery of the proper balance among the three perspectives will permit assignments to T, O, and P tasks. In themeantime, an equal distribution is preferable.

� Uneven replicability. The T perspective typically employs methods (e.g.,experimental design) that are replicable. The O and P perspectives are not soeasy to replicate. Much like a jury trial or a nonroutine executive decision, theprocess is not replicable.

� Appropriate communications. Adapt the medium of communication to themessage. Summaries, oral briefings, scenarios, and vignettes are appropriatefor communicating with those who hold O and P perspectives. Models, data,lists of variables, and analytical routines are appropriate for those with aT perspective.

� Deferred integration. Leave the integration of perspectives to the client orpolicy maker, but point out linkages among the T, O, and P perspectives andthe varied conclusions they yield.

Multiple perspective analysis has been employed extensively in the domain oftechnology assessment and other areas of public policy. Methods of multipleperspective analysis, developed on the basis of earlier work in foreign policy analysisand the design of knowledge systems,73 is a way to deal with ill-structured problemsthat originate in complex sociotechnical systems.

Assumption AnalysisAssumption analysis is a technique that aims at the creative synthesis of conflictingassumptions about policy problems.74 In many respects, assumption analysis isthe most comprehensive of all problem-structuring methods, because it includesprocedures used in conjunction with other techniques and may focus on groups,

73The antecedent works in foreign policy and knowledge systems design are, respectively, GrahamAllison, Essence of Decision: Conceptual Models and the Cuban Missile Crisis (Boston, MA: Little,Brown and Company, 1962); and C. West Churchman, The Design of Inquiring Systems (New York:Basic Books, 1971).74See Ian I. Mitroff and James R. Emshoff, “On Strategic Assumption-Making: A Dialectical Approachto Policy and Planning,” Academy of Management Review 4, no. 1 (1979): 1–12; Richard O. Masonand Ian I. Mitroff, Challenging Strategic Planning Assumptions: Theory, Cases, and Techniques (NewYork: Wiley, 1981); and Ian I. Mitroff, Richard O. Mason, and Vincent P. Barabba, The 1980 Census:Policymaking amid Turbulence (Lexington, MA: D. C. Heath, 1983).



individuals, or both. The most important feature of assumption analysis is that it isexplicitly designed to treat ill-structured problems, that is, problems for whichpolicy analysts, policy makers, and other stakeholders cannot agree on how toformulate a problem. The main criterion for assessing the adequacy of a givenformulation of a problem is whether conflicting assumptions about a problemsituation have been recognized, challenged, and creatively synthesized.

Assumption analysis is designed to overcome four major limitations ofpolicy analysis: (1) Policy analysis is often based on the questionable assumption of asingle decision maker with clearly ordered values that may be realized at a fixed pointin time, (2) policy analysis typically fails to consider in a systematic and explicit waystrongly differing views about the nature of problems and their potential solutions,(3) most policy analysis is carried out in organizations whose “self-sealing” charactermakes it difficult or impossible to challenge prevailing formulations of problems, and(4) criteria used to assess the adequacy of problems and their solutions often dealwith surface characteristics (e.g., consensus among leaders), rather than with basicassumptions underlying the conceptualization of problems.

Assumption analysis explicitly recognizes the positive as well as the negativefeatures of conflict and commitment.

Conflict is needed to permit the existence of maximally opposing policies toferret out and to challenge the underlying assumptions that each policy makes.Commitment on the other hand is also necessary if the proponents of eachpolicy are to make the strongest possible case (not necessarily the best) for theirrespective points of view.75

Assumption analysis involves the use of five procedures used in successive phases:

1. Stakeholder identification. In this phase, policy stakeholders are identified,ranked, and prioritized. The identification, ranking, and prioritization ofstakeholders are based on an assessment of the degree to which theyinfluence and are influenced by the policy process. This procedure resultsin the identification of stakeholders—for example, dissident groups ofadministrators or clients—who are usually excluded in the analysis of policyproblems.

2. Assumption surfacing. In this phase, analysts work backward from a recom-mended solution for a problem to the selective set(s) of data that support therecommendation and the underlying assumptions that, when coupled with thedata, allow one to deduce the recommendation as a consequence of the data.Each recommended solution put forth by policy stakeholders should contain alist of assumptions that explicitly and implicitly underlie the recommendation.By listing all assumptions—for example, that poverty is a consequence ofhistorical accidents, elite domination, unemployment, cultural deprivation, and so on—there is an explicit specification of the problem to which eachrecommendation is addressed.

3. Assumption challenging. In this phase, analysts compare and evaluate sets ofrecommendations and their underlying assumptions. This is done by

75Mitroff and Emshoff, “On Strategic Assumption-Making,” p. 5.



systematically comparing assumptions and counter assumptions that differmaximally from their counterparts. During this process, each assumptionpreviously identified is challenged by a counter assumption. If a counterassumption is implausible, it is eliminated from further consideration; if it isplausible, it is examined to determine whether it might serve as a basis for anentirely new conceptualization of the problem and its solution.

4. Assumption pooling. When the assumption-challenging phase has beencompleted, the diverse proposed solutions generated in previous phases arepooled. Here, assumptions (rather than recommendations) are negotiated byprioritizing assumptions in terms of their relative certainty and importance todifferent stakeholders. Only the most important and uncertain assumptionsare pooled. The ultimate aim is to create an acceptable list of assumptions onwhich as many stakeholders as possible agree.

5. Assumption synthesis. The final phase is the creation of a composite orsynthetic solution for the problem. The composite set of acceptable assump-tions can serve as a basis for the creation of a new conceptualization of theproblem. When issues surrounding the conceptualization of the problem andits potential solutions have reached this point, the activities of stakeholdersmay become cooperative and cumulatively productive.

The four phases of assumption analysis are illustrated in Figure 15, which helpsvisualize important features of the technique. First, the method begins with recom-mended solutions for problems rather than assumptions themselves because mostpolicy stakeholders are aware of proposed solutions for problems but seldom areconscious of underlying assumptions. By starting with recommended solutions, themethod builds on what is most familiar to stakeholders but then goes on to use fa-miliar solutions as a point of reference for forcing an explicit consideration ofunderlying assumptions. A second important feature of the technique is that itattempts, as far as possible, to focus on the same set of data or policy-relevant infor-mation. The reason for this is that conflicts surrounding the conceptualization ofpolicy problems are not so much matters of “fact” but matters involving conflicting

OriginalSolutions

CommonData

AssumptionSurfacing

AssumptionChallenging

AssumptionPooling

AssumptionSynthesis

CommonData

CommonData

CommonData

CounterSolutions

SolutionsPool

“Best”Solutions

FIGURE 15The process of assumption analysisSource: Adapted from Ian I. Mitroff and James R. Emshoff, “On Strategic Assumption-Making: A DialecticalApproach to Policy and Planning.” Academy of Management Review (1979).



interpretations of the same data. Although data, assumptions, and recommendedsolutions are interrelated, it is not so much the problem situation (data) that governsthe conceptualization of problems but the assumptions that analysts andother stakeholders bring to the problem situation. Finally, assumption analysissystematically addresses a major problem of policy analysis, which is that ofapplying systematic procedures to deal with conflict.

The aims and procedures of assumption analysis are intimately related to themodes of policy argument are beyond the scope of this chapter.76 Each mode ofpolicy argument contains characteristically different kinds of assumptions thatmay be used to produce alternative conceptualizations of problems. Assumptionanalysis is therefore a major vehicle for conducting reasoned debates about the na-ture of policy problems. Assumption analysis may be used with groups of policystakeholders who actually participate in structuring policy problems or by an indi-vidual analyst who simulates the assumptions of stakeholders in order to conducta reasoned debate with himself or herself. Assumption analysis can help reduceerrors of the third type (EIII).

Argument MappingAnother method of problem structuring is argument mapping. Graphic displaysare used to map the plausibility and importance of elements of a policy argument.The first step is to rate the elements—information, warrants, backings, objections,and rebuttals—on plausibility and importance scales. For example, recommenda-tions to abandon the 55 mph speed limit (National Maximum Speed Law of 1973)have been based on the warrant that the opportunity costs of time lost driving atslower speeds increases speeds and, in turn, promotes risky driving among mo-torists with higher incomes. Conversely, it could be argued that the frequency ofaccidents is likely to be less among younger motorists, a claim that is based on thequestionable warrant that because younger drivers earn less income, they havelower opportunity costs, take fewer risks, and therefore have fewer accidents. Thisdoubtful warrant may be rated by different stakeholders on nine-point plausibilityand importance scales (1 � low, 9 � high) and plotted on a graph such as thatshown in Figure 16.77

Figure 16 displays the plausibility and importance ratings for six stakehold-ers, represented by “X.” The graph shows that stakeholders are distributedacross all four quadrants, indicating broad disagreement about the plausibilityand importance of the warrant. This would probably generate objections andrebuttals. If stakeholders are members of a problem-structuring group, thedisagreements evident in the right half of the graph (i.e., high importance) can

77Computer software called Rationale 2 provides capabilities for entering, editing, saving, and evaluat-ing complex policy arguments. The program provides color codes, shading, and a three-point scale torate each element of an argument. See www.austhink.com.

76For an elaboration of assumption analysis using the structural model of argument presented in thechapter “Developing Policy Arguments,” see Mitroff, Mason, and Barabba, The 1980 Census, passim.



be discussed and perhaps resolved. The typical situation, however, is one in whichthe analyst must identify a range of stakeholders on the basis of telephoneinterviews, documents, and websites containing the stakeholders’ arguments andassumptions. Judgments are then made about the plausibility and importancethat stakeholders attach to a warrant, backing, objection, or rebuttal. For exam-ple, a review of documents on the 55 mph speed limit indicates that the stake-holder rating the warrant as highly plausible (P � 9) is an economist, whereasthe stakeholder who ranks the warrant as having low plausibility (P � 2) is anethnographer who specializes in the culture of young drivers and bases his orher analyses on interviews with young drivers, parents, and law enforcementpersonnel.78

On the basis of reasoned arguments and evidence provided by these twosources—along with information about the reasons provided by stakeholdersalong with statistics on accident rates by age group—we would conclude thatthis particular warrant has low plausibility. However, because the warrant hashigh importance it is relevant to the conclusions of the argument and must beexamined.

X

X

X

X

X

X9

8

7

6

4

3

2

1

1 2 3 4 6 7 8 9

Low Plausibility

High PlausibilityLo

w Im

por

tanc

e

High Im

portance

WARRANT: The opportunity costs ofdriving time are less for younger drivers,who earn less income. Younger driverstherefore may drive slower and take fewer risks.

FIGURE 16Distribution of warrant byplausibility and importance

78See Thomas H. Forrester, Robert F. McNown, and Larry D. Singell, “A Cost-Benefit Analysis of the55 mph Speed Limit,” Southern Economic Journal 50 (1984), reviewed by George M. Guess and PaulG. Farnham, Cases in Public Policy Analysis (New York: Longman, 1989), p. 199. For an ethnographicanalysis of younger drivers and the meanings they attach to accidents and fatalities—which are differentfrom those predicted by economic theory—see J. Peter Rothe, Challenging the Old Order (NewBrunswick, NJ: Transaction Books, 1990).



CHAPTER SUMMARYThis chapter provided an overview of the natureof policy problems, described the process ofstructuring these problems, examined relation-ships among policy models, and described

specific methods of problem structuring. One ofthe most important challenges facing policyanalysts is reducing the likelihood of a type IIIerror: defining the wrong problem.

REVIEW QUESTIONS1. “Our problem is not to do what is right,”

stated Lyndon Johnson during his years in theWhite House. “Our problem is to know whatis right.” Considering the major characteristicsand types of policy problems discussed in thischapter, to what extent can we know inadvance what is “right?”

2. A commonly accepted viewpoint among manypolicy analysts in government and in universi-ties is that policy analysis can be objective,neutral, and impartial. Given the characteris-tics of ill-structured problems, consider theextent to which this viewpoint is plausible.

3. Provide two or three examples from your ownexperience of ways that worldviews, ideolo-gies, and popular myths shape the formulationof policy problems.

4. There are several broad types of organizationalstructures in which policy formation occurs.One type is the “bureaucratic” structure,whose characteristics include centralization, hi-erarchical chain of command, specialization oftasks, and complete information. The bureau-cratic form of organization requires consensuson preferred policy outcomes, as well ascertainty that alternative courses of action willresult in certain preferred outcomes (J. D.Thompson, Organizations in Action [NewYork: McGraw-Hill, 1967], pp. 134–35). Ifmany of our most important policy problemsare ill-structured ones, what does this sayabout the appropriateness of the bureaucraticform of organization for formulating andresolving such problems?

5. If many of our most important problems areill-structured ones, to what extent is it possibleto hold individual policy makers, policyanalysts, and planners politically and morallyresponsible for their actions? (For the classicdiscussion of this point, see M. M. Webber and

H. W. J. Rittel, “Dilemmas in a General Theoryof Planning,” Policy Sciences 4, no. 2 [1973]:155–69.)

6. The ill-structured problems that follow aretaken from illustrations published in the jour-nal Policy Analysis (now the Journal of PolicyAnalysis and Management) under the title“Department of Unintended Consequences.”

For several thousand years, Egyptian agricul-ture depended on the fertilizing sedimentdeposited by the flood of the Nile. No longer,however. Due to expensive modern technologyintended to improve the age-old lot of the peas-ant, Egypt’s fields must be artificially fertilized.John Gall, writing in the New York TimesMagazine (December 26, 1976), reports thatthe Nile sediment is now deposited in theAswan Dam’s Lake Nasser. Much of the dam’selectrical output is used to supply enormousamounts of electricity to new fertilizer plantsmade necessary by the construction of the dam.

University of Illinois ecologists can explainhow certain harmful field mice spread fromtheir native regions into areas where they hadnever before been found. They are using thenew, limited-access, cross-country highways,which turn out to be easy escape routes withfew barriers. Older highways and roads, aswell as railroad rights-of-way, run into townsand villages every few miles and effectivelydeter mice migration. The Illinois group foundthat before interstate highways ran throughcentral Illinois, one type of mouse was limitedto a single county. But in six years of super-highways the four-inch-long creatures havespread sixty miles south through the center ofthe state. The ecologists are concerned lest themice, a species that loves to chew on trees,become a threat in central and southern



counties where apple orchards abound (WallStreet Journal, December 1, 1977).

Edward J. Moody . . . argues persuasively thatworship of Satan has the effect of normalizingabnormal people. Thus, to “keep secret” fromordinary people their satanic power and exis-tence, such persons are urged to behave asstraight as possible. The effect, of course, ismore effective social relations—the goal forwhich Satan’s name has been invoked in thefirst place! (P. E. Hammond, “Review ofReligious Movements in ContemporaryAmerica,” Science, May 2, 1975, p. 442).

Residents of San Francisco’s North Beachareas must now pay $10 for the privilege ofparking in their own neighborhood. A residen-tial parking plan was recently implemented toprevent commuters from using the area as adaytime parking lot. But according to a storyin the San Francisco Bay Guardian (March 14,1978), the plan has in no way improved theresidential parking situation. Numbers ofcommuters from outlying districts of the cityhave simply been changing their car registra-tions to North Beach addresses. A North

Beach resident—now $10 poorer—still spendsa lot of time driving around the block.

Choose one of these problems and write ashort essay on how classification analysis, hier-archy analysis, and synectics might be used tostructure this problem.

7. Construct a scenario on the state of one of thefollowing problem situations in the year 2030:

Availability of public mass transitArms control and national securityCrime prevention and public safetyQuality of public educationState of the world’s ecological system

8. Select two editorials on a current issue of publicpolicy from two newspapers (e.g., New YorkTimes, Washington Post, The Economist, LeMonde) or news magazine (e.g., Newsweek, TheNew Republic, National Review). After readingthe editorial:a. Use the procedures for argumentation

analysis to display contending positions andunderlying assumptions.

b. Rate the assumptions and plot them accord-ing to their plausibility and importance(Figure 16).

c. Which arguments are the most plausible?

� What conclusions can you draw about thepolicy problem(s) in the issue area?

Compare your work with Case Study 1 at theend of the chapter.

2. After reading Case 1, write an essay in whichyou compare and contrast the process ofboundary analysis and estimation in miningand transportation. In your comparison,address these questions:� What are the key differences in data collec-

tion, represented by the process of groupinterviewing and content analysis?

� Why do the cumulative frequency graphsflatten out the way they do?

� Evaluate the statement: “Boundary analysisis a reliable way to estimate the ‘universe ofproblem formulations’ in a given policyissue area.”

DEMONSTRATION EXERCISE1. Choose a policy issue area such as crime control,

national security, environmental protection, oreconomic development. Use the procedures forstakeholder analysis presented in ProceduralGuide 3 to generate a list of stakeholders whoaffect or are affected by problems in the issuearea you have chosen for analysis.After generating the list, create a cumulativefrequency distribution. Place stakeholders onthe horizontal axis, numbering them from1 . . . n. On the vertical axis, place the numberof new (nonduplicate) ideas generated by eachstakeholder (the ideas can be objectives, alter-natives, outcomes, causes, etc.). Connect thetotal new ideas of each stakeholder with a linegraph.� Does the line graph flatten out?� If so, after how many stakeholders?



BIBLIOGRAPHYAckoff, Russell L. The Art of Problem Solving:

Accompanied by Ackoff’s Fables. New York:John Wiley, 1978.

________________. Redesigning the Future.New York: Wiley, 1974.

Adams, James L. Conceptual Blockbusting.Stanford, CA: Stanford Alumni Association,1974.

Adelman, L., T. R. Stewart, and K. R. Hammond.“A Case History of the Application of SocialJudgment Theory to Policy Formulation.”Policy Sciences 6 (1975): 137–59.

Bobrow, Davis B., and John S. Dryzek. PolicyAnalysis by Design. Pittsburgh, PA: Universityof Pittsburgh Press, 1986.

Churchman, C. West. The Design of InquiringSystems. New York: Basic Books, 1971.

Dery, David. Problem Definition in PolicyAnalysis. Lawrence: University Press of Kansas,1984.

Dror, Yehezkel. Public Policy Making Reexamined.Rev Ed. New Brunswick. NJ: TransactionBooks, 1983.

Dryzek, John S. “Policy Analysis as aHermeneutic Activity.” Policy Sciences 14(1982): 309–29.

Dunn, William N. “Methods of the Second Type:Coping with the Wilderness of ConventionalPolicy Analysis.” Policy Studies Review 7, no. 4(1988): 720–37.

Dunn, William N., and Ari Ginsberg. “ASociocognitive Approach to OrganizationalAnalysis.” Human Relations 39, no. 11(1986): 955–75.

Fischhoff, Baruch. “Clinical Policy Analysis.” InPolicy Analysis: Perspectives, Concepts, andMethods. Edited by William N. Dunn.Greenwich, CT: JAI Press, 1986.

———. “Cost-Benefit Analysis and the Art ofMotorcycle Maintenance.” Policy Sciences 8(1977): 177–202.

George, Alexander. “Criteria for Evaluation ofForeign Policy Decision Making.” GlobalPerspectives 2 (1984): 58–69.

Hammond, Kenneth R. “Introduction toBrunswickian Theory and Methods.” NewDirections for Methodology of Social andBehavioral Science 3 (1980): 1–12.

Hofstadter, Richard. Godel, Escher, Bach. NewYork: Random House, 1979.

Hogwood, Brian W., and B. Guy Peters. ThePathology of Public Policy. Oxford: ClarendonPress, 1985.

Linder, Stephen H., and B. Guy Peters. “AMetatheoretic Analysis of Policy Design.” InAdvances in Policy Studies since 1950. Vol. 10of Policy Studies Review Annual. Edited byWilliam N. Dunn and Rita Mae Kelly. NewBrunswick, NJ: Transaction Books, 1992.

Linstone, Harold A. Multiple Perspectives forDecision Making. New York: North-Holland,1984.

Mason, Richard O., and Ian I. Mitroff.Challenging Strategic Planning Assumptions.New York: Wiley, 1981.

Meehan, Eugene J. The Thinking Game. Chatham,NJ: Chatham House, 1988.

Mitroff, Ian I., Richard O. Mason, and Vincent P.Barabba. The 1980 Census: Policymaking amidTurbulence. Lexington, MA: D. C. Heath, 1983.

Saaty, Thomas L. The Analytic Hierarchy Process.New York: McGraw-Hill, 1980.

Schon, Donald A. The Reflective Practitioner. NewYork: Basic Books, 1983.

Sieber, Sam. Fatal Remedies. New York: PlenumPress, 1981.

Warfield, John N. Societal Systems: Planning, Policy,and Complexity. New York: Wiley, 1976.

Watzlawick, Paul, John Weakland, and RichardFisch. Change: Principles of Problem Formationand Problem Resolution. New York: W. W.Norton, 1974.



BOX —CONDUCTING A STAKEHOLDER ANALYSIS

DefinitionA stakeholder is a person who speaks for orrepresents a group that is affected by or affects apolicy. Stakeholders include the president of alegislative assembly or parliament, a chairpersonof a legislative committee, or an executive directoror members of an organized interest or advocacygroup such as the National Rifle Association, theSierra Club, or Human Rights Watch. Policyanalysts and their employers are stakeholders, asare clients who commission a policy analysis.Persons or groups who do not have a stake in apolicy (e.g., an uninvolved college professor)are not stakeholders.

Assumptions• Stakeholders are best identified by policy issue

area. A policy issue area is a domain in whichstakeholders disagree or quarrel about policies.Housing, welfare, education, and internationalsecurity are policy issue areas.

• Stakeholders have specific names and titles—for example, State Senator Xanadi; Mr.Young,chairperson of the House Finance Committee;or Ms. Ziegler, a spokesperson for the NationalOrganization of Women (NOW).

• A sociometric or “snowball” sample such asthat described next is an effective way toestimate the “population” of stakeholders.

STEP 1: Using Google or a reference booksuch as The Encyclopedia of Associations,

identify and list about tenstakeholders who have taken apublic position on a policy. Makethe initial list as heterogeneous aspossible by sampling opponents aswell as supporters.

STEP 2: For each stakeholder, obtain apolicy document (e.g., a report,news article, e-mail, or telephone

interview) that describes theposition of each stakeholder.

STEP 3: Beginning with the first statement of the first stakeholder, list otherstakeholders mentioned as opponentsor proponents of the policy.

STEP 4: For each remaining statement, listthe new stakeholders mentioned.Do not repeat.

STEP 5: Draw a graph that displaysstatements 1, 2, . . . n on thehorizontal axis. On the vertical axis,display the cumulative frequency ofnew stakeholders mentioned in thestatements. The graph will graduallyflatten out, with no new stakeholdersmentioned. If this does not occurbefore reaching the last stakeholderon the initial list, repeat steps 2 to 4.Add to the graph the new statementsand the new stakeholders.

STEP 6: Add to the estimate stakeholderswho should be included because oftheir formal positions (organizationcharts show such positions) orbecause they are involved in one ormore policy-making activities:agenda setting, policy formulation,policy adoption, policyimplementation, policy evaluation,and policy adaptation, succession ortermination.

Retain the full list for further analysis. You nowhave an estimate of the “population” of keystakeholders who are affected by and affect thepolicy, along with a description of their positionson an issue. This is a good basis for structuringthe problem. �


79The architecture analogy is from Herbert Simon.

CASE 1 STRUCTURING PROBLEMS OF RISK INMINING AND TRANSPORTATION

Complex problems must be structured beforethey can be solved. The process of structuring apolicy problem is the search for and specification ofproblem elements and how they are the elements are

Policy stakeholders. Which stakeholders affect orare affected by a problem?

Policy alternatives. What alternative courses ofaction may be taken to solve the problem?

Policy actions. Which of these alternativesshould be acted on to solve the problem?

Policy outcomes. What are the probableoutcomes of action and are they part of thesolution to the problem?

Policy values (utilities). Are some outcomes morevaluable than others in solving the problem?

Most policy problems are messy or ill-structured.For this reason, one or more problem elements canbe incorrectly omitted from the definition of aproblem. Even when problem elements are correctlyspecified, relations among the elements may beunknown or obscure. This makes it difficult orimpossible to determine the strength and significance,practical as well as statistical, of causal relations.For example, many causal processes that are believedto govern relations among atmospheric pollution,global warming, and climate change are obscure. Theobscurity of these processes stems not only from thecomplexity of “nature” but also from the conflictingbeliefs of stakeholders who disagree, often intensely,about the definition of problems and their potentialsolutions. For this reason, the possible combinationsand permutations of problem elements—that is,stakeholders, alternatives, actions, outcomes,values—appear to be unmanageably huge.

Under these conditions, standard methods ofdecision theory (e.g., risk-benefit analysis), appliedeconomics (e.g., benefit-cost analysis), and politicalscience (e.g., policy implementation analysis) are of

limited value until the problem has beensatisfactorily defined. This is so because an adequatedefinition of the problem must be constructed beforethe problem can be solved with these and otherstandard methods. Standard methods are useful insolving relatively well-structured (deterministic)problems involving certainty, for example, problemsrepresented as fixed quantities in a spreadsheet.Standard methods are also useful in solvingmoderately structured (probabilistic) problemsinvolving uncertainty, for example, problemsrepresented as policy outcomes with differentprobabilities. However, ill-structured problems are ofa different order. Estimates of uncertainty, or risk,cannot be made because we do not even know theoutcomes to which we might attach probabilities.Here, the analyst is much like an architect who hasbeen commissioned to design a custom building forwhich there is no standard plan.79 The adoption of astandard plan, if such existed, would almost certainlyresult in a type III error: solving the wrong problem.

Public policies are deliberate attempts to changecomplex systems. The process of making andimplementing policies occurs in social systems inwhich many contingencies lie beyond the control ofpolicy makers. It is these unmanageable contingenciesthat are usually responsible for the success and failureof policies in achieving their objectives. Thecontingencies are rival hypotheses that can challengeclaims that a policy (the presumed cause) producedone or more policy outcomes (the presumed effects).In such cases, it is usually desirable to test, and whenpossible eliminate, these rival hypotheses through aprocess of eliminative induction. Eliminative inductiontakes this general form:“Repeated observations ofpolicy x and outcome y confirm that x is causallyrelevant to the occurrence of y. However, additionalobservations of x, z, and y confirm that unmanageablecontingency z and not policy x is responsible for the



occurrence of y.” By contrast, enumerative inductiontakes this general form:“Repeated observationsconfirm that the policy x is causally relevant to theoccurrence of policy outcome y.”

Eliminative induction permits a criticalexamination of contingencies that are beyond thecontrol of policy makers. Because the number ofthese contingencies is potentially unlimited, theprocess of identifying and testing rival explanationsis never complete. Yet, precisely for this reason, itseems impossible to identify and test anunmanageably huge number of potential rivalhypotheses. How is this to be done?

One answer is creativity and imagination. Butcreativity and imagination are impossible to teach,because there are no rules governing the replicationof creative or imaginative solutions. Another answeris an appeal to well-established theories. However,the bulk of theories in the social sciences aredisputed and controversial. “Well-established”theories are typically “well-defended” theories, andrival hypotheses are rarely considered seriously, letalone tested.

A more appropriate alternative is the use ofboundary analysis and estimation to structureproblems involving a large number of rivalhypotheses. Boundary analysis and estimation lookfor rival hypotheses in the naturally occurring policyquarrels that take place among stakeholders. Inaddition to the policy analyst, these stakeholdersinclude scientists, policy makers, and organizedcitizen groups. The aim of boundary estimation is toobtain a relatively complete set of rival hypotheses ina given policy context. Although boundary estimationstrives to be comprehensive, it does not attempt thehopeless task of identifying and testing all plausiblerival hypotheses. Although the range of rivalhypotheses is never complete, it is possible to estimate

the probable limit of this range.

Assessing the Impact of National Maximum Speed Limit

A boundary analysis was conducted with documentsprepared by thirty-eight state officials responsible

for reporting on the effects of the original 55 mphand later 65 mph speed limits in their states. Asexpected, there were sharp disagreements amongmany of the thirty-eight stakeholders. For example,some states were tenaciously committed to thehypothesis that speed limits are causally related tofatalities (e.g., Pennsylvania and New Jersey).Others were just as firmly opposed (e.g., Illinois,Washington, Idaho). Of direct importance toboundary estimation is that 718 plausible rivalhypotheses were used by thirty-eight stakeholders toaffirm or dispute the effectiveness of the 55 mphspeed limit in saving lives. Of this total, 109hypotheses were unique, in that they did notduplicate hypotheses advanced by any otherstakeholder.

Here, it is important to note that from thestandpoint of communications theory and language,the information content of a hypothesis is inverselyrelated to its relative frequency or probability ofoccurrence. Hypotheses that are mentioned morefrequently—those on which there is greaterconsensus—have less probative value than rarelymentioned hypotheses, because highly probable orpredictable hypotheses do not challenge acceptedknowledge claims.

The rival hypotheses were analyzed according to the cumulative frequency of unique (nonduplicate)causal hypotheses. As Figure C1 shows, thecumulative frequency curve of unique rivalhypotheses flattens out after the twenty-secondstakeholder. Although the total number of rivalhypotheses continues to increase without apparent

limit, the boundary of unique rival hypotheses isreached within a small and affordable number of observations. This indicates that a satisfactorydefinition of the problem has probably beenachieved. Indeed, of the 109 unique rivalhypotheses, several variables related to the state of the economy—unemployment, theinternational price of oil, industrial production—explain the rise and fall of traffic fatalities better than average highway speeds and the 55 mphspeed limit.

(continued)



Evaluating Research on Risk in Mine Safety and Health

In 1997, a branch of the U.S. Office of Mine Safetyand Health Research began a process of strategicplanning. The aim of the process was to reachconsensus, if possible, on the prioritization ofresearch projects that address different aspects ofrisk associated with the safety and health of miners.

Priority setting in this and other researchorganizations typically seeks to build consensus underconditions in which researchers, research managers,and external stakeholders use conflicting criteria toevaluate the relative merits of their own and otherresearch projects. Even when reliable and valid data areavailable—for example, quantitative data from large-sample studies of the probabilities of different kinds ofmine injuries and deaths—it is often unclear whether“objective” measures of risk, by themselves, provide asufficient basis for prioritizing research. The difficultyis that extra-scientific as well as scientific factors affectjudgments about the relative merits of research on risk.For example, the probabilities of the occurrence of

“black lung” disease and other high-risk conditionshave been intensively investigated. Despite theimportance of the black lung problem, additionalresearch is not a priority. Accordingly, data on highexpected severity (probability ¥ severity) do not aloneprovide a sufficient basis for prioritizing researchproblems. Because judgments about research prioritiesare based on multiple, hidden, and frequentlyconflicting criteria, it is important to uncover andevaluate these criteria as part of the process of problemstructuring. This is a classic ill-structured problem forwhich the fatal error is defining the wrong problem.

The problem-structuring process had three majorobjectives. The first was to uncover hidden sources ofagreement and disagreement, recognizing thatdisagreement is an opportunity for identifyingalternative approaches to priority setting. Second, theprocess was designed to generate from stakeholdersthe criteria they use to evaluate research on risksaffecting mine safety and health. Third, the processemployed graphs, matrices, and other visual displaysin order to externalize the criteria underlying individual

120

100

80

60

Riv

al C

ause

s

40

20

01 4 7 10 13

Stakeholders

16 23220%

25%

50%

75% Percent

100%

Boundary Estimate

FIGURE C1Pareto chart—cumulative frequency of rival causes of traffic fatalities



judgments. The priority-setting process enabled eachteam member to understand and debate the variedreasons underlying the priorities used to evaluateresearch on risk.

Stakeholders were prompted to state the criteriathey use to distinguish among twenty-five accidentand health problems. Each stakeholder waspresented with a form listing pre-randomized sets ofthree problems. One stakeholder was chosen atrandom to present criteria to the group. The firstpresented 14 criteria for evaluating research onrisk, which included two criteria—severity andcatastrophic potential of accidents and diseases—which were based on large-sample relativefrequency (probability) data. The second randomlychosen team member presented 17 additional (new)criteria. The third team member added 11, thefourth 12, the fifth 6, and so on, until no additional

criteria could be offered that were not mentionedbefore. In all, 84 criteria were generated, none afterthe eleventh presenter. The approximate boundaryof the problem was displayed with a Pareto chart toshow the cumulative frequency of constructs(Figure C2).

The process of interactive group priority settingnot only addressed the “objective” side of riskresearch but also captured “subjective” dimensionsthat go by such labels as “perceived risk,”“acceptable risk,” “researchable risk,” and“actionable risk.” The problem was successfullybounded by means of an open problem-structuringprocess of generating and discussing criteria thatwere otherwise tacit, concealed, or implicit. Themethod of boundary analysis and estimationprovides a “stop rule” that avoids the infinitesearch for problems. �

100

80

60

Ris

k C

rite

ria

40

20

01 2 3 4 5 6 7 8 9 10 11 12 13

0%

20%

40%

60%

Percent

80%

100%

120%Boundary Estimate

Stakeholder

FIGURE C2Pareto chart—cumulative frequency of criteria for evaluating research on risk


80Russell Ackoff, The Art of Problem Solving (New York: Wiley-Interscience, 1978), “Fable 3.2 AnUps-and-Downs Story,” pp. 53–4.81Another example of a (probable) type III error is the aphorism popularized by Aaron Wildavsky: “Nosolution, no problem.” See Wildavsky’s preface to Speaking Truth to Power: The Art and Craft ofPolicy Analysis (Boston, MA: Little Brown, 1979).

CASE 2 BRAINSTORMING ACROSS DISCIPLINES:THE ELEVATOR PROBLEM

This case is based on a story by Russell Ackoff, amaster at solving ill-structured problems, or what hecalled “messes.”80 The story, a classic of unknownorigin that has been in circulation for many years,shows why methods such as benefit-cost analysis,although appropriate for well-structured andmoderately structured problems, are not appropriatefor ill-structured problems.

The manager of a large office building had beenreceiving a large and increasing number ofcomplaints about the elevator service, particularlyduring rush hours. When tenants threatened tomove out unless the service was improved, themanager decided that a solution for the problemmust be found.

The manager first called on a group of consultingengineers who specialized in the design andconstruction of elevators. After looking into thesituation, the engineers identified three options: (1)add more elevators, (2) replace the existing elevatorswith faster ones, or (3) add a computerized controlsystem so that elevator service could be optimallyrouted for faster service.

In collaboration with an economist, the engineersconducted benefit-cost analyses of the three options.They found that investing in additional elevators,replacement elevators, or a computer control systemwould yield significant improvements of service.However, they found that the cost of any of the threealternatives was not justified by the monetary and

intangible benefits of renting the building. Becausenone of the alternatives was acceptable, the managerfaced a dilemma.

He was desperate. He called a meeting of hisstaff and presented the problem to them as part of abrainstorming session similar to the one describedearlier in this chapter. Many suggestions were madeand then discarded. The discussion slowed down andstalled.

During a break, a young new assistant in thehuman resources department, who had been silent tothis point, quietly made a suggestion that wasimmediately embraced by everyone present. A fewweeks later, the manager made a modest investmentand the problem disappeared.

Full-length mirrors had been installed on all the walls of the

elevator lobbies on each floor.

The young psychologist in the human resourcesdepartment observed that the waiting times wereactually quite short. He reasoned that thecomplaints came from the boredom of waiting forthe elevators. The time only seemed long. He gavepeople something to do: Look at themselves andothers (particularly those of the opposite sex)without appearing to do so. This kept themoccupied.

The consulting engineers and the economist hadcommitted a type III error:They defined the wrongproblem. Consequently, they falsely concluded thatthere was no solution.81 �


O B J E C T I V E S


Forecasting ExpectedPolicy Outcomes

� Distinguish projections, predictions,and conjectures.

� Understand the effects of temporal,historical, and institutional contextson forecast accuracy.

� Contrast potential, plausible, andnormative futures.

� Describe objects, bases, methods,and products of forecasts.

� Distinguish extrapolative,theoretical, and judgmentalforecasting methods.

� Use statistical software to makepoint and interval estimates ofexpected policy outcomes.

� Analyze a case in policy forecastinginvolving issues of environmentaljustice.

Forecasting provides a prospective vision of policy outcomes, thereby contri-buting to understanding, control, and policy guidance. However, forecasts of allkinds—whether based on expert judgment, on the extrapolation of historical

trends, or on technically sophisticated econometric models—are prone to errorsbased on faulty or implausible assumptions, on the effects of institutional incentivesystems, and on the accelerating complexity of policy issues in areas such as health,welfare, education, and the environment.

We begin this chapter with an overview of the forms, functions, and performanceof forecasting methods in policy analysis, stressing a range of criteria for assessingtheir strengths and limitations. We then compare and contrast three majorapproaches to forecasting: extrapolative forecasting, theoretical forecasting, andjudgmental forecasting. We conclude with a presentation of techniques employed inconjunction with these three approaches.



Forecasting Expected Policy Outcomes

FORECASTING IN POLICY ANALYSISForecasting yields factual information about expected policy outcomes on the basis ofprior information about policy problems. Forecasts take three principal forms: projec-tions, predictions, and conjectures.

1. A projection is a forecast that is based on the extrapolation of current andhistorical trends into the future. Projections put forth designative claims basedon arguments about methods and parallel cases, in which assumptions aboutthe validity of particular methods (e.g., time-series analysis) or similaritiesbetween cases (e.g., past and future policies) are used to establish the cogencyof claims. Projections may be supplemented by arguments from authority(e.g., the opinions of experts) and cause (e.g., field experiments).

2. A prediction is a forecast based on theoretical laws (e.g., the law ofdiminishing utility of money), theoretical propositions (e.g., the propositionthat civil disorders are caused by the gap between expectations and capabilities),or analogies (e.g., the analogy between the growth of government and thegrowth of biological organisms). The essential feature of a prediction is that itexplains the generative mechanisms (“causes”) and consequences (“effects”)believed to underlie a prediction. Predictions may be supplemented by argumentsbased on authority (e.g., informed judgment) and method (e.g., econometricmodeling).

3. A conjecture is a forecast based on informed judgments about future states ofsociety. These judgments may take the form of intuitive arguments, for whichassumptions about the insight, creative intellectual power, or tacit knowledgeof knowledgeables (e.g., “policy insiders”) are used to support claims aboutthe future. Judgments may also be expressed in the form of motivationalarguments whereby present or future goals, values, and intentions are used toestablish the plausibility of claims. For example, conjectures about futuresocial values (e.g., leisure) have been used to claim that the average workweekwill be reduced to thirty hours in the next twenty years. Conjectures may besupplemented by arguments based on authority, method, and cause.

Aims of ForecastingPolicy forecasts, whether based on extrapolation, theory, or informed judgment,have several important aims. First, and most important, forecasts provide informa-tion about future changes in policies and their consequences. The aims of forecastingare similar to those of much natural and social science research, insofar as both seekto understand and control the human and material environment. Nevertheless,efforts to forecast future societal states are “especially related to control—that is, tothe attempt to plan and to set policy so that the best possible course of action mightbe chosen among the possibilities which the future offers.”1

1Irene Taviss, “Futurology and the Problem of Values,” International Social Science Journal 21, no. 4(1969): 574.



Forecasting permits greater control through understanding past policies andtheir consequences, an aim that implies that the future is determined by the past.Yet forecasts also enable us to shape the future in an active manner, irrespective ofwhat has occurred in the past. In this respect, the future-oriented analyst must askwhat values can and should guide future action. This leads to a second and equallydifficult question: How can the analyst evaluate the future desirability of a givenstate of affairs? Even if the values underlying current actions could be predicted,would these values still be operative in the future? As Ikle has noted,

“Guiding predictions” are incomplete unless they evaluate the desirability of thepredicted aspects of alternative futures. If we assume that this desirability is tobe determined by our future rather than our present preferences . . . then wehave to predict our values before we can meaningfully predict our future.2

This concern with future values may complement traditional social sciencedisciplines that emphasize predictions based on past and present values. Although pastand present values may determine the future, this will hold true only if intellectualreflection does not lead to changes in their values and behavior, or if unpredictablefactors do not intervene to create profound social changes, including irreversibleprocesses of chaos and emergent order.3

Limitations of ForecastingIn the years since 1985, there have been a number of unexpected, surprising, andcounterintuitive events—for example, the formal abandonment of socialism in theSoviet Union, the dissolution of communist parties in Eastern Europe, the fall of theBerlin Wall, and the 2011 uprisings across the Middle East. These changes at oncecall attention to the importance and the difficulties of forecasting under conditionsof increasingly complex, rapid, and even chaotic changes. The growing difficulty offorecasting, however, should be seen in light of limitations and strengths of varioustypes of forecasts conducted over the past four decades and more:4

1. Forecast accuracy. The accuracy of relatively simple forecasts based on the extrapolation of values of a single variable, as well as relatively complexforecasts based on models incorporating hundreds of variables, has beenlimited. In the period leading up to the 2008 election, for example, the Bush administration changed its forecast of the U.S. budget deficit from$407 billion to $482 billion in a span of only six months. A similar recordof performance tends to characterize forecasts of the largest econometric

2Fred Charles Ikle, “Can Social Predictions Be Evaluated?” Daedalus 96 (summer 1967): 747.3See Alasdair MacIntyre, “Ideology, Social Science, and Revolution,” Comparative Politics 5, no. 3(1973); and Ilya Prigogine and Isabelle Stengers, Order Out of Chaos (New York: Bantam Books, 1984).4A seminal work on forecasting is William Ascher, Forecasting: An Appraisal for Policy Makersand Planners (Baltimore and London: Johns Hopkins University Press, 1978). Also see Ascher,“The Forecasting Potential of Complex Models,” Policy Sciences 13 (1981): 247–67; andRobert McNown, “On the Use of Econometric Models: A Guide for Policy Makers,” Policy Sciences 19 (1986): 360–80.



forecasting firms, which include Chase Econometrics, Wharton EconometricForecasting Associates, and Data Resources, Inc. Even in more stable and prosperous economic times, average forecasting error as a proportion of actual changes in GNP was as large as 50 percent in the 1970s and early 1980s.5

2. Comparative yield. The accuracy of predictions based on complextheoretical models of the economy and of the energy resource system hasbeen no greater than the accuracy of forecasting models based onextrapolation and informed judgment. If one of the advantages of suchmodels is sensitivity to surprising or counterintuitive events, simple modelsappear to have a comparative advantage over their technically complexcounterparts, because developers and users of complex models tend toemploy them mechanistically. Ascher’s question is to the point: “If theimplications of one’s assumptions and hypotheses are obvious when they are‘thought out’ without the aid of a model, why use one? If the implicationsare surprising, the strength of the complex model is in drawing from a set ofassumptions these implications..”6 Yet precisely these assumptions andimplications—for example, the implication that a predicted large increase ingasoline consumption may lead to changes in gasoline taxes, which aretreated as a constant in forecasting models—are overlooked or discardedbecause they are inconsistent with the assumptions of models.

3. Context. The assumptions of models and their results are sensitive to three kinds of contexts: institutional, temporal, and historical. Variations ininstitutional incentive systems are a key aspect of differences in institutionalcontexts, as represented by government agencies, businesses, and nonprofitresearch institutes. Forecasting accuracy tends to be greater in nonprofitresearch institutes than in businesses or government agencies. In turn, thetemporal context of a forecast, as represented by the length of time overwhich it is made (e.g., one quarter or year versus five years ahead), affectsforecast accuracy. The longer the time frame, the less accurate is theforecast. Finally, the historical context of forecasts affects accuracy. The relatively greater complexity of recent historical periods diminishesforecast accuracy, a pattern that is evident in the growth of forecastingerrors since the 1960s.

Forecasting accuracy and comparative yield are thus closely related to theinstitutional, temporal, and historical contexts in which forecasts are made.Accuracy and comparative yield are also affected by the assumptions that peoplebring to the process. As Ascher notes, one of the difficulties of assessing the per-formance of forecasts lies in identifying the assumptions of forecast developersand users. In many forecasts, there is a serious problem of “assumption drag,”7

5McNown, “On the Use of Econometric Models,” pp. 362–67.6Ascher, “The Forecasting Potential of Complex Models,” p. 255.7See Ascher, Forecasting.



that is, a tendency among developers and users of forecasting models to cling toquestionable or plainly implausible assumptions built into a model—for example,the assumption that the pricing policies of petroleum-producing countries willremain relatively stable. An important implication may be derived from thephenomenon of assumption drag—namely, that the task of structuring policyproblems is central to the performance of forecasters. Indeed, the correction oferrors by dissolving or unsolving problems is essential to forecasting accuracyand yield.

TYPES OF FUTURESPolicy forecasts, whether made in the form of projections, predictions, or conjectures,are used to estimate three types of future societal states: potential futures, plausiblefutures, and normative futures.8 Potential futures (sometimes called alternativefutures) are future societal states that may occur, as distinguished from societal statesthat eventually do occur. A future state is never certain until it actually occurs, andthere are many potential futures. Plausible futures are future states that, on the basisof assumptions about causation in nature and society, are believed to be likely ifpolicy makers do not intervene to redirect the course of events. By contrast,normative futures are potential and plausible futures that are consistent with ananalyst’s conception of future needs, values, and opportunities. The specification ofnormative futures narrows the range of potential and plausible futures, thus linkingforecasts to specific goals and objectives.

8See David C. Miller, “Methods for Estimating Societal Futures,” in Methodology of Social ImpactAssessment, ed. Kurt Finsterbusch and C. P. Wolf (Stroudsburg, PA: Dowden, Hutchinson & Ross,1977), pp. 202–10.

Past Present Future

NormativePasts

(What shouldhave happened)

NormativeFutures

(What shouldbe happen)

PotentialFutures(What

might be happen)

PotentialPasts

(What mighthave

happened)

PlausibleFutures(What

will be happen)

PlausiblePasts(What

happened)

FIGURE 1Three types of societal futures: potential, plausible, and normative



TABLE 1

Contrasts Between Goals and Objectives

Characteristic Goals Objectives

Specification of purposes

1. Broadly stated ( . . . to upgrade the quality ofhealth care . . . )

1. Concrete ( . . . to increasethe number of physiciansby 10 percent . . . )

Definition of terms

2.Verbal ( . . . the quality of health care refers to accessibility of medicalservices . . . )

2. Operational ( . . . the qualityof health care refers to thenumber of physicians per 100,000persons . . . )

Time period 3. Unspecified ( . . . in the future . . . )

3. Specified ( . . . in the period2000–2010 . . . )

Measurement procedure

4. Nonquantitative ( . . . adequate healthinsurance . . . )

4. Quantitative ( . . . the number of persons covered per 1,000 persons . . . )

Treatment of target groups

5. Broadly defined ( . . . persons in need of care . . . )

5. Specifically defined ( . . . familieswith annual incomes below$19,000 . . . )

Goals and Objectives of Normative FuturesAn important aspect of normative futures is the specification of goals and objectives.Today’s values are likely to change in the future, thus making it difficult to forecastnormative futures on the basis of existing preferences. The analyst must, therefore,be concerned with future changes in the ends as well as the means of policy. Inthinking about the ends of policy, it is useful to contrast goals and objectives.Although goals and objectives are both future oriented, goals express broadpurposes whereas objectives set forth specific aims. Goals are rarely expressed in theform of operational definitions—that is, definitions that specify the set of operationsnecessary to measure something—whereas objectives usually are. Goals are notquantifiable, but objectives may be and often are. Statements of goals usually donot specify the time period in which policies are expected to achieve desiredconsequences, whereas statements of objectives do. Finally, goals define targetpopulations in broad terms, whereas objectives define target populations specifically.Contrasts between goals and objectives are illustrated in Table 1.

The definition of normative futures not only requires that we clarify goals andobjectives, but also requires that we identify policies that might be relevant forthe achievement of these goals and objectives. Whose goals and objectives shouldthe analyst use as a focal point of forecasts? How does an analyst choose amonga large number of alternatives to achieve given goals and objectives? If analysts useexisting policies to specify goals, objectives, and alternatives, they run the risk ofapplying a conservative standard. If, on the other hand, they propose new goals,objectives, and alternatives, they may be charged with imposing their own beliefsand values or making choices that are closer to the positions of one stakeholder thananother.



Sources of Goals, Objectives, and AlternativesOne way to select goals, objectives, and alternatives is to consider their possiblesources. Alternatives imply goals and objectives, just as goals and objectives implypolicy alternatives. Sources of policy alternatives, goals, and objectives include thefollowing:

1. Authority. In searching for alternatives to resolve a problem, analysts mayappeal to experts. For example, the World Health Organization’s Expert Panelon Air Quality Guidelines 2005 report was a source of policy alternatives forimproving environmental equity among nations.9

2. Insight. The analyst may appeal to the intuition, judgment, or tacitknowledge of persons believed to be particularly insightful about a problem.These “knowledgeables,” who are not experts in the ordinary sense of theword, are an important source of policy alternatives. For example, workinggroups on child development in the U.S. Department of Education have beenused as a source of informed judgments about policy alternatives, goals, andobjectives in the area of child welfare.10

3. Method. The search for alternatives may benefit from innovative methodsof analysis. For example, new techniques of systems analysis may behelpful in identifying alternatives and rank-ordering multiple conflictingobjectives.11

4. Scientific theories. Explanations produced by the natural and socialsciences are another source of policy alternatives. For example, socialpsychological theories of learning have served as one source of earlychildhood education programs, such as Head Start.

5. Motivation. The beliefs, values, and needs of stakeholders may serve as asource of policy alternatives. Alternatives may be derived from the goals andobjectives of particular occupational groups, for example, workers whosechanging beliefs, values, and needs have created a new “work ethic” involvingdemands for leisure and flexible working hours.

6. Parallel case. Experiences with policy problems in other countries, states,and cities are an important source of policy alternatives. The experiences ofNew York and California with financial reforms have served as a source offinancial policies in other states.

7. Analogy. Similarities between different kinds of problems are a source ofpolicy alternatives. Legislation designed to increase equal employmentopportunities for women has been based on analogies with policies adoptedto protect the rights of minorities.

9Air Quality Guidelines: Global Update 2005 (Copenhagen: World Health Organization RegionalOffice for Europe, 2006).10See, for example, Ward Edwards, Marcia Guttentag, and Kurt Snapper, “A Decision-TheoreticApproach to Evaluation Research,” in Handbook of Evaluation Research, vol. 1, ed. Elmer L. Strueningand Marcia Guttentag (Beverly Hills, CA: Sage Publications, 1975), pp. 159–73.11An excellent example is Thomas L. Saaty and Paul C. Rogers, “Higher Education in the United States(1985–2000): Scenario Construction Using a Hierarchical Framework with Eigenvector Weighting,”Socio-Economic Planning Sciences, 10 (1976): 251–63. See also Thomas L. Saaty, The AnalyticHierarchy Process (New York: Wiley, 1980).



8. Ethical systems. Another important source of policy alternatives isethical systems. Theories of social justice put forward by philosophers andother social thinkers serve as a source of policy alternatives in a variety ofissue areas.12

APPROACHES TO FORECASTINGOnce goals, objectives, and alternatives have been identified, it is possible to selectan approach to forecasting. The analyst must (1) decide what to forecast, that is,determine what the object of the forecast is to be; (2) decide how to make theforecast, that is, select one or more bases for the forecast; and (3) choose techniquesthat are most appropriate for the object and base selected.

Objects of ForecastsThe object of a forecast is the point of reference of a projection, prediction, orconjecture. Forecasts have four objects:13

1. Consequences of existing policies. Forecasts may be used to estimate changesthat are likely to occur if no new government actions are taken. The statusquo, that is, doing nothing, is an existing policy. Examples are populationprojections of the U.S. Bureau of the Census and projections of female laborforce participation made by the U.S. Bureau of Labor Statistics.14

2. Consequences of new policies. Forecasts may be used to estimate changesin society that are likely to occur if new policies are adopted. For example,declining demand for petroleum may be projected on the basis of assumptionsabout the development of new and more efficient biofuels technologies.

3. Contents of new policies. Forecasts may be used to estimate changes in thecontents of new public policies. The Congressional Research Service, forexample, forecasts the possible adoption of a four-week annual paid vacationon the assumption that the government and labor unions will follow the leadof European countries, most of which have adopted four- or five-week annualpaid vacations for workers.15

4. Behavior of policy stakeholders. Forecasts may be used to estimate theprobable support (or opposition) to newly proposed policies. For example,techniques for assessing political feasibility may be used to estimate the

12John Rawls, A Theory of Justice (Cambridge, MA: Harvard University Press, 1971). On ethicalsystems as a source of policy alternatives, see Duncan MacRae Jr., The Social Function of Social Science(New Haven, CT: Yale University Press, 1976).13See William D. Coplin, Introduction to the Analysis of Public Policy Issues from a Problem-SolvingPerspective (New York: Learning Resources in International Studies, 1975), p. 21.14See, for example, Howard N. Fullerton Jr. and Paul O. Flaim, “New Labor Force Projections to1990,” Monthly Labor Review (December 1976).15See Everett M. Kassalow, “Some Labor Futures in the United States,” Congressional Research Service,Congressional Clearing House on the Future, Library of Congress (January 31, 1978).



probability that different stakeholders will support a policy at variousstages of the policy process, from adoption to implementation.16

Bases of ForecastsThe basis of a forecast is the set of assumptions or data used to estimate the conse-quences of existing or new policies, the content of new policies, or the behavior ofstakeholders. There are three major bases of forecasts: trend extrapolation, theory,and informed judgment. Each of these bases is associated with one of the threeforms of forecasts previously discussed.

Trend extrapolation is the extension into the future of trends observed in the past.Trend extrapolation assumes that what has occurred in the past will also occur in thefuture, provided that no new policies or unforeseen events intervene to change thecourse of events. Trend extrapolation is based on inductive logic, that is, the process ofreasoning from particular observations to general conclusions or claims. In trendextrapolation, we usually start with a set of time-series data, project past trends intothe future, and then invoke assumptions about regularity and persistence that justifythe projection. The logic of trend extrapolation is illustrated in Figure 2.

16See Michael K. O’Leary and William D. Coplin, Everyman’s “Prince” (North Scituate, MA: DuxburyPress, 1976).

By the year 2020 therewill be more than 1,000federal agencies.

CLAIM

Between 1865 and 1973 thenumber of federal agenciesgrew from 50 to 300, agrowth rate of more than5 percent annually.

INFORMATIONThe growth rate in1974–2020 will beabout the same asthe growth rate in1865–1973.

WARRANTProbably so...QUALIFIER

But the period 1965–1973 isunrepresentative of the period1974–2020. Opposition to thegrowth of government is nowstrong. The pattern of growth isnonlinear and unstable.

OBJECTIONopposes

FIGURE 2The logic of extrapolation: inductive reasoning



Policy analysts will have more power than policy makers incoming years.

CLAIM

We now live in a“post-industrial” or“knowledge” society.

INFORMATIONIn “post-industrial” or “knowledge” societies, thescarcity of policy expertisegives more power to policyanalysts.

WARRANTProbably so...QUALIFIER

support

Unless expertise is seenas a threat to policy makers,who will exercise strictcontrol over analysts.

OBJECTIONopposes

FIGURE 3The logic of theoretical prediction: deductive reasoning

Theories are systematically structured and empirically testable sets of laws andpropositions that make predictions about the occurrence of one event on the basis ofanother. Theories are causal in form, and their specific role is to explain and predict.The use of theories is based on deductive logic, that is, the process of reasoning fromgeneral statements, laws, or propositions to particular claims. For example, theproposition that in “postindustrial” society the knowledge of analysts is anincreasingly scarce resource that enhances their power may be used to move frominformation about the growth of professional policy analyst in government to thepredictive claim that policy analysts will have more power than policy makers incoming years (Figure 3).

Informed judgments refer to knowledge based on insight, or what the pragma-tist philosopher Charles Sanders Peirce called “hypothesis.” These judgments areusually expressed by experts or knowledgeables and are used when theory and/orempirical data are unavailable or inadequate. Informed judgments are often basedon abductive logic, that is, the process of reasoning that begins with claims aboutthe future and then works backward to the information and assumptions necessaryto support a claim. A good example of informed judgment as a basis of forecastsis the use of scientists or other knowledgeables to make conjectures about futurechanges in technology. Through abductive logic, experts may construct a scenariothat claims that there will be automated highways with adaptive automobile



autopilots by the year 2050. Experts then work backward to the information andassumptions needed to establish the plausibility of the claim (Figure 4).

Abductive reasoning is a creative way to explore ways in which potentialfutures may emerge out of the present. By contrast, inductive and deductive reason-ing are potentially conservative, because the use of information about past events orthe application of established scientific theories may restrict consideration ofpotential (as distinguished from plausible) futures. A good example of the restrictiveinfluence of past events and established scientific theories comes from the well-known astronomer William H. Pickering (1858–1938):

The popular mind pictures gigantic flying machines speeding across the Atlanticand carrying innumerable passengers in a way analogous to our modernsteamships. . . . It seems safe to say that such ideas must be wholly visionary,and even if a machine could get across with one or two passengers the expensewould be prohibitive.17

By the year 2050 there willbe automated highways withautomobile autopilots whichadapt speeds and fuelconsumption to drivingconditions.

CLAIM

Given that the researchers have participated in aDelphi study in which it isestimated that automatedhighways can be operationalin 2050, plus or minus 5 years.

INFORMATIONBecause researchers inwell-respected think tanksbelieve that highwayautomation is bothdesirable and feasiblewithin the next 20 years.

WARRANTWell, that seemsentirely possible...

QUALIFIER

But unforeseen eventssuch as war or recession may make this unfeasible by 2050.

OBJECTIONopposes

FIGURE 4The logic of conjecture: abductive reasoning

17Quoted in Brownlee Haydon, The Year 2000 (Santa Monica, CA: Rand Corporation, 1967).



Techniques of ForecastingAlthough the selection of an object and basis helps guide the analyst toward appro-priate techniques, there are literally hundreds of such techniques to choose from.18

A useful way to simplify the choice of these techniques is to group them according tothe bases of forecasts discussed earlier. Table 2 outlines the three approaches to fore-casting, their bases, appropriate techniques, and products. This table serves as anoverview of the rest of the chapter.

TABLE 2

Three Approaches to Forecasting with Their Bases,Appropriate Techniques and ProductsApproach Basis Appropriate Technique Product

Extrapolative forecasting Trend extrapolation Classical time-series analysis ProjectionsLinear trend estimationExponential weightingData transformationCatastrophe methodology

Theoretical forecasting Theory Theory mapping PredictionsCausal modelingRegression analysisPoint and interval estimationCorrelational analysis

Judgmental forecasting Informed judgment Conventional Delphi and Policy Delphi ConjecturesCross-impact analysisFeasibility assessment

18One of the most comprehensive sources on all types of forecasting is Forecasting Principles: Evidence-Based Forecasting at www.forecastingprinciples.com. Earlier useful works include Daniel P. Harrison,Social Forecasting Methodology (New York: Russell Sage Foundation, 1976); Denis Johnston,“Forecasting Methods in the Social Sciences,” Technological Forecasting and Social Change 2 (1970):120–43; Arnold Mitchell and others, Handbook of Forecasting Techniques (Fort Belvoir, VA: U.S. ArmyEngineer Institute for Water Resources, December 1975).

EXTRAPOLATIVE FORECASTINGMethods and techniques of extrapolative forecasting enable analysts to make projectionsof future societal states on the basis of current and historical data. Extrapolative forecast-ing is usually based on some form of time-series analysis, that is, on the analysis ofnumerical values collected at multiple points in time and presented sequentially. Time-series analysis provides summary measures (averages) of the amount and rate of changein past and future years. Extrapolative forecasting has been used to project economicgrowth, population decline, energy consumption, quality of life, and agency workloads.

When used to make projections, extrapolative forecasting rests on three basicassumptions:

1. Persistence. Patterns observed in the past will persist in the future. If energyconsumption has grown in the past, it will do so in the future.



19Fidelity to these and other methodological assumptions does not guarantee accuracy. In this author’sexperience, significantly inaccurate forecasts are frequently based on strict adherence to technicalassumptions. This is why problem structuring and the detection of “assumption drag” are so importantto all forms of forecasting, including econometric modeling.

2. Regularity. Past variations in observed trends will regularly recur in thefuture. If wars have occurred every twenty or thirty years in the past, thesecycles will repeat themselves in the future.

3. Reliability and validity of data. Measurements of trends are reliable (i.e., relatively precise or internally consistent) and valid (i.e., measure whatthey purport to be measuring). For example, crime statistics are relativelyimprecise measures of criminal offenses.

When these three assumptions are met, extrapolative forecasting may yieldinsights into the dynamics of change and greater understanding of potential futurestates of society. When any one of these assumptions is violated, extrapolativeforecasting techniques are likely to yield inaccurate or misleading results.19

Classical Time-Series AnalysisWhen making extrapolative forecasts, we may use classical time-series analysis, whichviews any time series as having four components: secular trend, seasonal variations,cyclical fluctuations, and irregular movements. Secular trend is a smooth long-termgrowth or decline in a time series. Figure 5 shows a secular trend in the growth ofcrimes per 1,000 persons in Chicago over a thirty-year period. By convention, the

19400

30

60

1945 1950 1955 1960 1965 1970

Years (X)

Arr

ests

per

1,0

00 p

opul

atio

n (Y

)

Straight-LineTrend

Time Series

FIGURE 5Demonstration of asecular trend; totalarrests per 1,000population in Chicago,1940–70

Source: Adapted from Ted R. Gurr,“The Comparative Analysis ofPublic Order,” in The Politics of

Crime and Conflict, ed. Ted R. Gurr,Peter N. Grabosky, and RichardC. Hula (Beverly Hills, CA: SagePublications, 1977), p. 647.



time-series variable is plotted on the Y-axis (called the ordinate), and years are plottedon the X-axis (called the abscissa). A straight-line trend has been used to summarizethe growth of total arrests per 1,000 persons between 1940 and 1970. In other cases(e.g., mortality), the straight-line trend shows a long-term decline; still other cases(e.g., consumption of coal oil) show a curvilinear trend, that is, a trend in which thenumerical values in a time-series display a convex or concave pattern.

Seasonal variation, as the term suggests, is the variation in a time series that recursperiodically within one year or less. The best examples of seasonal variations are theups and downs of production and sales that follow changes in weather conditions andholidays. The workloads of social welfare, health, and public utilities agencies alsofrequently display seasonal variations as a result of weather conditions and holidays.For example, the consumption of home heating fuels increases in the winter monthsand begins to decline in March of each year.

Cyclical fluctuations are also periodic but may extend unpredictably over a numberof years. Cycles are frequently difficult to explain, because each new cyclical fluctuationmay be a consequence of unknown factors. Total arrests per 1,000 population inChicago, displayed over a period of more than a hundred years, show at least three cyclicfluctuations (Figure 6). Each of these cycles is difficult to explain, although the third cycli-cal fluctuation coincides with the Prohibition Era (1919–33) and the rise of organizedcrime. This example points to the importance of carefully selecting an appropriate timeframe, because what may appear to be a secular trend may in fact be part of some largerlong-term pattern of cyclical fluctuations. Note also that total arrests per 1,000 personswere higher in the 1870s and 1890s than in most years from 1955 to 1970.

FirstCycle

SecondCycle

ThirdCycle

Appointment ofChief Wilson

1868 1883 1898 1912 1926 1941 1955 1970Year (X)

0

30

60

90

Arr

ests

per

1,0

00 p

opul

atio

n (Y

)

FIGURE 6Demonstration ofcyclical fluctuations:Total arrests per 1,000 population inChicago, 1868–1970Source: Ted R. Gurr, “TheComparative Analysis of PublicOrder,” in The Politics of Crime and

Conflict, ed. Ted R. Gurr, Peter N.Grabosky, and Richard C. Hula(Beverly Hills, CA: SagePublications, 1977), p. 647.



20See Donald T. Campbell, “Reforms As Experiments,” in Readings in Evaluation Research, ed. FrancisG. Caro (New York: Russell Sage Foundation, 1971), pp. 240–41.

The interpretation of cyclical fluctuations is frequently made more difficult bythe presence of irregular movements, that is, unpredictable variations in a time seriesthat appear to follow no regular pattern. Irregular movements may be the result ofmany factors (e.g., changes in government, strikes, natural disasters). As long as thesefactors are unaccounted for, they are treated as random error, that is, unknownsources of variation that cannot be explained in terms of secular trends, seasonalvariations, or cyclical fluctuations. For example, the irregular movement in totalarrests that occurred after 1957 (see Figure 6) might be regarded as an unpredictablevariation in the time series that follows no regular pattern. On closer inspection,however, the sharp temporary upswing in arrests may be explained by the changes inrecord keeping that came into effect when Orlando Wilson was appointed chief ofpolice.20 This example points to the importance of understanding the sociohistoricaland political events that may be responsible for changes in a time series.

Linear Trend EstimationA standard technique for extrapolating trends is linear trend estimation, a procedurethat uses regression analysis to obtain statistically reliable estimates of future societalstates on the basis of observed values in a time series. Linear regression is based onassumptions of persistence, regularity, and data reliability. When linear regressionis used to estimate trend, it is essential that observed values in a time series are notcurvilinear, because any significant departure from linearity will produce forecasts withsizable errors. Nevertheless, linear regression may also be used to remove the lineartrend component from a series that displays seasonal variations or cyclical fluctuations.

Regression analysis has two important properties:

1. Deviations cancel. The sum of the differences between observed values in atime series and values lying along a computed straight-line trend (called aregression line) will always equal zero. Thus, if the trend value (Yt) is sub-tracted from its corresponding observed value (Y) for all years in a time series,the total of these differences (called deviations) will equal zero. When theobserved value for a given year lies below the regression line, the deviation (Y � Yt) is always negative. By contrast, when the observed value is above theregression line, the deviation (Y � Yt) is always positive. These negative andpositive values cancel each other out, such that ∑(Y � Yt) � 0.

2. Squared deviations are a minimum. If we square each deviation (i.e., multiplyeach deviation value by itself) and add them all up, the sum of these squareddeviations will always be a minimum or least value. This means that linearregression minimizes the distances between the regression line and all observedvalues of Y in the series. In other words, it is the most efficient way to draw atrend line through a series of observed data points.

These two properties of regression analysis are illustrated with hypotheticaldata in Figure 7.



The computation of linear trend with regression analysis is illustrated in Table 3with data on energy consumption. In column (3) note that years in the series may becoded, for ease of calculation, as the numerical distance (x) of each year from themiddle of the entire period (i.e., 1973). The middle of this series is given a codedvalue of zero and treated as the origin. The values of x range from �3 to �3 and aredeviations from the origin. Observe also the computational procedures used incolumns (4) and (5) of Table 3. The value of xY is calculated by multiplying values ofenergy consumption (Y) by each coded time value (x), as shown in column (4). Thevalue of x2, shown in column (5), is calculated by multiplying each value of x by itself,that is, squaring all the x values in column (3). Finally, column (6) contains trendvalues of the time-series variable (Yt). These trend values, which form a straight line,are calculated according to the following equation:

where

Yt � the trend value for a given yeara � the value of Yt when x � 0b � the slope of the trend line representing the change in Yt for each unit

of timex � the coded time value for any year, as determined by its distance from

the origin

Yt = a + b1x2

Y�Yt �negative

Y�Yt �positive

Time-SeriesValues

(Y)

Trend Line(Yt)

Time Periods (X)

Time Series(Y)

Y

Y

Yt

Yt

X2X1

FIGURE 7Two properties of linear regression

Note: ∑(Y � Yt) � 0; ∑(Y � Yt)2 � a minimum (least) value.



TABLE 3

Time-Series Data on Total Energy Consumption Used in Linear Regression

YearsEnergy

Consumption

CodedTime Value

Columns (2) By (3)

Column (3)Squared

Trend Valuefor Energy

Consumption(X) (Y) (x) (xY) (x2) (Yt)(1) (2) (3) (4) (5) (6)

1970 66.9 -3 -200.7 9 68.351971 68.3 -2 -136.6 4 69.311972 71.6 -1 -71.6 1 70.271973 74.6 0 0 0 71.241974 72.7 1 72.7 1 72.201975 70.6 2 141.2 4 73.171976 74.0 3 222.0 9 74.13

n � 7 ΣY � 498.7 Σx � 0 Σ(xY) � 27.0 Σ(x2) � 28 ΣYt � 498.7

Yt = a + b1x2

a =

©Y

n=

498.7

7= 71.243

b =

©1xy2

©1x 22=

27.0

28.0= 0.964

Yt = 71.243 + 0.9641x2

Once the values of a and b have been calculated, estimates of total energyconsumption may be made for any year in the observed time series or for any futureyear. For example, Table 3 shows that the trend value for energy consumption in1972 is 70.27 quadrillion BTUs. To project total energy consumption for 1980, weset the value of x at 7 (i.e., seven time periods away from 1973, the origin) and solvethe equation Yt(1980) � a � b(x). The formula for computing the value of a is

where

∑Y � the sum of observed values in the seriesn � the number of years in the observed time series

The formula for computing the value of b is

where

(∑xY) � the sum of the products of the coded time values and the observedvalues in the series [see column (4) of Table 3]

∑(x2) � the sum of the squared coded time values [see column (5) ofTable 3]

b = ©1xY2

©1x22

a = ©Yn



TABLE 4

Linear Regression with an Even-Numbered Series

YearsEnergy

ConsumptionCoded Time

ValueColumns (2)

By (3)Column (3)

Squared

Trend Value for Energy

Consumption(X) (Y) (x) (xY) (x2) (Yt)(1) (2) (3) (4) (5) (6)

1969 64.4 -7 -450.8 49 66.14

1970 66.9 -5 -334.5 25 67.36

1971 68.3 -3 -204.9 9 68.57

1972 71.6 -1 -71.6 1 69.78

1973 74.6 1 74.6 1 71.00

1974 72.7 3 218.1 9 72.21

1975 70.6 5 353.0 25 73.43

1976 74.0 7 518.0 49 74.64

n � 8 ΣY � 563.1 Σx � 0 Σ(xY) � 101.9 Σ(x2) � 168 ΣYt � 563.1

Yt = a + b1x2

a =

©Y

n=

563.1

8= 70.39

b =

©1xY2

©1x22=

101.9

168= 0.607

Yt = 70.39 + 0.6071x2

Yt 119802 = 70.39 + 0.6071152 = 79.495 quadrillion BTUs

The calculations in Table 3 not only permit us to compute trend values forenergy consumption in the observed series, but also enable us to project totalenergy consumption for any given year in the future. Thus, the estimate of to-tal energy consumption in 1980 [Yt(1980)] is 77.99 quadrillion BTUs.

In illustrating the application of linear regression, we have used a time serieswith an odd number of years. We treated the middle of the series as the originand coded it as zero. Obviously, there are many time series that contain an evennumber of years, and a different procedure may be used for determining eachcoded time value (x), because there is no middle year. The procedure used witheven-numbered time series is to divide the series into two equal parts and codethe time values in intervals of two (rather than in intervals of one, as inodd-numbered series). Each year in Table 4 is exactly two units away from itsneighbor, and the highest and lowest coded time values are �7 and �7, ratherthan �3 and �3. Note that the size of the interval need not be 2; it could be 3, 4, 5, or any number provided that each year is equidistant from the next onein the series. The size of the interval does not affect the computation of results.Indeed, the standard way to code time periods (e.g., years, quarters, days, hours)is to give each period a number that is one greater than its predecessor, startingwith period 1.



Despite its precision in extrapolating secular trends, linear regression is limitedby several conditions. First, the time series must be linear, that is, display a constantincrease or decrease in values along the trend line. If the pattern of observations isnonlinear (i.e., the amount of change increases or decreases from one time period tothe next), other techniques must be used. Some of these techniques require fittingvarious types of curves to nonlinear time series. Second, plausible arguments mustbe offered to show that historical patterns will persist in the future, that is, continuein much the same form in subsequent years as they have in past ones. Third, patternsmust be regular, that is, display no cyclical fluctuations or sharp discontinuities.Unless all of these conditions are present, linear regression should not be used toextrapolate trends.

The SPSS output for the regression analyses computed by hand in Tables 3 and4 is displayed in Exhibit 1. As may be seen by comparing Table 3 in the text with theSPSS output (Table 3 in Exhibit 1), the results are identical. The SPSS output identi-fies the dependent variable (which was named “ENCONS” for energy consump-tion), the number of observations in the sample (n = 7), the simple correlation coef-ficient expressing the strength of the relationship between time and energyconsumption (r � 0.729), and the squared correlation coefficient (r2 � 0.531), whichexpresses the proportion of variance in the dependent variable (“ENCONS”) ex-plained by the independent variable (“TIMECODE”).

For present purposes, the most important parts of the SPSS output give thecoefficient for the constant, a, which is 71.243—this is the value of Y, energyconsumption, when X equals zero. The output also gives the coefficient for the slope ofthe regression line, b, which is 0.964—this is the amount of change in Y, energy con-sumption, for each unit change in X, the years. These two parts of the output enable us towrite the regression equation and, using the coefficients in this equation, forecast thevalue of energy consumption for any future year. The regression equation for Table 3 is

The regression equation for Table 4, which has eight rather than seven years in thetime series, is

Note how much of a difference there is in the slopes after adding only one year tothe series.

Many time-series data of concern to policy analysts—for example, data on crime,pollution, public expenditures, urbanization—are nonlinear. A variety of techniqueshave been developed to fit nonlinear curves to patterns of change that do not displaya constant increase or decrease in the values of an observed time series. Althoughthese techniques are not presented in detail, we discuss some of their properties andunderlying assumptions. Readers who wish to investigate these forecasting techniquesin greater depth should consult advanced texts on time-series analysis.21

Yt = 70.388 + 0.6071X2

Yt = 71.243 + 0.9641X2

21See G. E. P. Box and G. M. Jenkins, Time Series Analysis: Forecasting and Control (San Francisco,CA: Holden-Day, 1969); and S. C. Wheelwright and S. Makridakis, Forecasting Methods forManagement, 4th ed. (New York: Wiley, 1985).



EXHIBIT 1

SPSS Output for Tables 3 and 4

Model Summaryb

Model R R SquareAdjusted R

SquareStandard Errorof the Estimate Durbin Watson

1 .729a .531 .437 2.14575994 1.447

aPredictors: (Constant),TIMECODEbDependent Variable: ENCONS

ANOVAb

ModelSum ofSquares df Mean Square F Significance

1 Regression 26.036 1 26.036 5.655 .063a

Residual 23.021 5 4.604Total 49.057 6

aPredictors: (Constant),TIMECODEbDependent Variable: ENCONS

Coefficientsa

UnstandardizedCoefficients

StandardizedCoefficients

Mode B Std. Error Beta t Sig.

1 (Constant) 71.243 .811 87.843 .000TIMECODE .964 .406 .729 2.378 .063

aDependent Variable: ENCONS

Residuals Statisticsa

Minimum Maximum Mean Std. Deviation N

Predicted Value

68.349998 74.135712 71.242857 2.08309522 7

Residual �2.571429 3.3571429 8.120E-15 1.95880187 7

Std.Predicted Value

�1.389 1.389 .000 1.000 7

Std.Residual

–1.198 1.565 .000 .913 7

aDependent Variable: ENCONS



Nonlinear Time SeriesTime series that do not meet conditions of linearity, persistence, and regularity fallinto five main classes (Figure 8):

1. Oscillations. Here, there are departures from linearity, but only within years,quarters, months, or days. Oscillations may be persistent and regular (e.g.,most police arrests occur between 11 P.M. and 2 P.M. throughout the year) butnot show a constant increase or decrease within the period under examination(Figure 8[a]). Oscillations within years may occur in conjunction with long-term secular trends between years. Examples include seasonal variationsin unemployment, monthly variations in agency workloads, and dailyvariations in levels of pollutants.

(X )(a)

(X )

Osc

illat

ion

(Y)

Cyc

les

(Y)

Gro

wth

(Y)

Dec

line

(Y)

Cat

astr

ophe

(Y)

(X )(b)

(X )

(X ) (X )(c)

(X )

(X ) (X )(d)

(X )

(X ) (X )(e)

(X ) FIGURE 8Five classes of nonlinear time series



2. Cycles. Cycles are nonlinear fluctuations that occur between years or longerperiods of time. Cycles may be unpredictable or occur with persistence andregularity. Whereas the overall pattern of a cycle is always nonlinear, segments of agiven cycle may be linear or curvilinear (Figure 8[b]). Examples are business cyclesand the “life cycles” of academic fields, scientific publications, and civilizations.

3. Growth curves. Departures from linearity occur between years, decades, orsome other unit of time. Growth curves evidence cumulative increases in therate of growth in a time series, cumulative decreases in this rate of growth, orsome combination of the two (Figure 8[c]). In the latter case, growth curves areS-shaped and are called sigmoid or logistic curves. Growth curves, which de-veloped out of studies of biological organisms, have been used to forecast thegrowth of industry, urban areas, population, technology, and science. Althoughgrowth curves are not linear, they are nevertheless persistent and regular.

4. Decline curves. Here, departures from linearity again occur between years,decades, or longer periods. In effect, decline curves are the counterpart ofgrowth curves. Decline curves evidence either cumulative increases or decreasesin the rate of decline in a time series (Figure 8[d]). Increasing anddecreasing rates of decline may be combined to form curves with differentshapes. Patterns of decline are sometimes used as a basis for various dynamicor life-cycle perspectives of the decline of civilizations, societies, and urbanareas. Decline curves are nonlinear but regular and persistent.

5. Catastrophes. The main characteristic of time-series data that are catastrophic isthat they display sudden and sharp discontinuities. The analysis of catastrophicchange, a field of study founded by the French mathematician René Thom,involves not only nonlinear changes over time but also patterns of change that arediscontinuous (Figure 8[e]). Examples include sudden shifts in government policyduring war (surrender or withdrawal), the collapse of stock exchanges in times ofeconomic crisis, and the sudden change in the density of a liquid as it boils.22

The growth and decline curves illustrated in Figures 8(c) and (d) cannot bedescribed by a straight-line trend. Patterns of growth and decline display little or nocyclical fluctuation, and the trend is best described as exponential growth or decline,that is, growth or decline whereby the values of some quantity increase or decreaseat an increasing rate. The growth of federal government organizations between1789 and 1973 (Figure 9) is an example of exponential growth. After 1789, the to-tal number of organizations began to grow slowly, until about 1860 whenthe growth rate began to accelerate. Clearly, this growth trend is very different fromthe secular trends and cyclical variations examined so far.23

Techniques for fitting curves to processes of growth and decline are morecomplex than those used to estimate secular trend. Even though many ofthese techniques are based on linear regression, they require various transfor-mations of the time-series variable (Y). Some of these transformations

23The outstanding work on processes of growth in science and other areas is Derek de Sólla Price, LittleScience, Big Science (New York: Columbia University Press, 1963).

22See C. A. Isnard and E. C. Zeeman, “Some Models from Catastrophe Theory in the Social Sciences,” in TheUse of Models in the Social Sciences, ed. Lyndhurst Collins (Boulder, CO: Westview Press, 1976), pp. 44–100.



involve roots whereas others require logarithms (log Y) or exponents (ex).In each case, the aim is to express mathematically changes in a time series thatincrease by increasing or decreasing amounts (or, conversely, decrease byincreasing or decreasing amounts). These techniques will not be presented here,but their logic is sufficiently important to warrant further investigation.

A simple illustration will serve best to clarify techniques for curve fitting. Recallthe model of compound interest. This model states that

where

Sn � the amount to which a given investment will accumulate in a given (n)number of years

S0 � the initial amount of the investment(1 � r)n � a constant return on investment (1.0) plus the rate of interest (r) in a

given (n) number of years

Imagine that the manager of a small municipality of 10,000 persons isconfronted by the following situation. In order to meet immediate out-of-pocketexpenses for emergencies, it was decided in 2000 that a sum of $1,000 should be set

Sn = 11 + r2n S0

11nY2,

0

50

100

150

200

250

300

350

400

1780 1800 1820 1840 1860 1880 1900 1920 1940 1960 1973

Presidential Term (X)

Tota

l Org

aniz

atio

ns (

Y)

FIGURE 9Growth of federal government organizations in the United Statesby presidential term, 1789–1973

Source: Herbert Kaufman, Are Government Organizations Immortal? (Washington,DC: Brookings Institution, 1976), p. 62.



aside in a special checking account. The checking account is very handy but earns nointerest. In 2001 and subsequent years, because of rising inflation, the managerbegan to increase the sum in the special account by $100 per year. The increase infunds, illustrated in Figure 10(a), is a good example of the kind of linear trend wehave been examining. The time series increases by constant amounts ($100 per year)and by 2010, there was $2,000 in the special account. In this case, the time-seriesvalues (Y) are identical to the trend values (Yt � Y), because all values of Y are onthe trend line.

Now consider what would have occurred if the manager had placed the funds ina special interest-bearing account (we will assume that there are no legal restrictionsand that withdrawals can be made without penalties). Assume that the annual rate ofinterest is 10 percent compounded annually. Assume also that only the original$1,000 was left in the account for the ten-year period, that is, no further deposits weremade. The growth of city funds, illustrated in Figure 10(b), is a good example of thekind of growth trend we have been discussing. City funds increased by increasingamounts over the ten-year period (but at a constant interest rate compounded annu-ally), and the total funds available at the end of 2010 was $2,594, compared with$2,000 in the non-interest-bearing account (Figure 10[a]). Note that in the first case(no interest), it takes ten years to double the original amount. No accumulation above

2,600

1,000

1,200

1,400

1,600

1,800

2,000

2,200

2,400

1971 1973 1975 1977 1979

Years (X)

Linear Trend

(a)

Yt � Y

Yt

Yt � 1,550 � 47.27 (x)

2,600

1,000

1,200

1,400

1,600

1,800

2,000

2,200

2,400

1971 1973 1975 1977 1979

Fund

s ($

) In

Sp

ecia

lN

on-I

nter

est-

Bea

ring

Acc

ount

(Y)

Fund

s ($

) In

Sp

ecia

lIn

tere

st-B

earin

g A

ccou

nt (Y

)

Years (X)

(b)

Growth Trend

Y

Yt � 1,753.30 � 82.31 (x)

FIGURE 10Linear versus growth trends



the original $1,000 comes from interest. In the second case, it takes a little more thanseven years to double the original amount, and all of the additional amount ($1,594)comes from interest. The values used to calculate accumulation at the end of 2010 are

Note that accumulation for any given year (e.g., 2005 as the fifth year) may becalculated simply by substituting the appropriate values in the formula S5 � (1 � r)5

($1,000) � (1.6105)($1,000) � $1,610.51.Consider now the limitations of linear regression in estimating growth trends

such as that illustrated in Figure 10(b). The linear regression equation ofFigure 10(b) [Yt � 1,753.30 � 82.31(x)] will produce an inaccurate forecast. Forexample, a 1990 forecast using the linear regression equation yields a trendestimate of $3,399.50 [Yt(1990) � 1,753.30 � 82.31(20) � $3,399.50]. By contrast,the compound interest formula, which exactly represents the nonlinear growth inaccumulated funds, produces a 1990 estimate of $6,727.47 [S20 � (1.1)20($1,000)� 6.72747 ($1,000) � $6,727.47].24 The linear regression estimate, which isalmost half the amount calculated with the compound interest formula, is highlyinaccurate because it violates the key assumption that the series must be linear toemploy linear regression analysis is therefore highly inaccurate.

Fortunately, the linear regression technique may be adapted to estimate growthtrends. For regression to do this, however, it is necessary to change our originallinear equation [Yt � a � b(x)] to a nonlinear one. Although there are many waysto do this, two of the most common procedures are exponential weighting and datatransformation.

Exponential WeightingIn exponential weighting, we raise one of the terms in the regression equation to apower, for example, square the value of (x), or add another term, also raised to a power,to the equation [e.g., c(x2)]. The more pronounced the increase (or decrease) in theobserved growth pattern, the higher the power necessary to represent it. For example, ifwe have increasing amounts of change, such as those illustrated by the compoundinterest formula (Figure 10[b]), we might add to the original linear regression equation athird term [c(x2)] that has been raised to a power by squaring the coded time value (x).The equation (called a second-degree parabola) then reads

= 1753.30 + 82.311x2 + 1.81x22

Yt = a + b1x2 + c1x22

S10 = $2,594

= 11 + r2101$1,0002 = 11 + 0.102101$1,0002 = 2.59371$1,0002

Sn = 11 + r2n S0

24Note that for this example coded time values are one unit apart starting with 1971 equal to 1. This isconsistent with the compound interest calculation, which also treats 1971 as the first year in whichinterest is paid.



Note that the only part of this equation that is not linear is the squared timevalue (x2). This means that any value of x computed for a given year will increase bya power of 2 (e.g., a coded values of becomes 81). The higher the original x value,the greater the amount of change, because the squares of larger numbers producedisproportionately larger products than do the squares of smaller numbers. Whereasa coded time value (x) of 9 is three times the value of 3 in the linear regressionequation, in the second-degree parabola the same year is nine times the value of thecorresponding year

This is what it means to say that something increases at an increasing rate.

Data TransformationA second procedure used to adapt linear regression techniques to processesof growth and decline is data transformation. Whereas exponential weightinginvolves the writing of a new and explicitly nonlinear regression equation[e.g., Yt � a � b(x) � c(x2)], the transformation of data permits analysts to workwith the simple linear equation [Yt � a � b(x)], but only after values of the time-series variable (Y) have been appropriately transformed. One way to transformthe time-series variable is to take its square root Another way to transformdata is to take the common logarithm of values of the time-series variable. Thecommon (base 10) logarithm of a number is the power to which 10 must be raised toproduce that number. For example, 2 is the logarithm of 100, because 10 must beraised to the power of 2 to produce 100 (10 � 10 � 102 � 100). By contrast, thenatural (base e) logarithm of a number is the power to which the base e, whichequals 2.71828, must be raised to produce that number. For example, the naturallogarithm of 100 is 4.6052, because 2.71828 raised to the power 4.6052 equals 100.The abbreviation for base 10 logarithms is log, and the abbreviation for base e loga-rithms is ln. To grasp the nature of these transformations we can study a hypotheti-cal time series that exhibits extremely rapid growth (Table 5). We can then observethe consequences of taking roots and logarithms of the time-series variable beforesolving the simple linear regression equation.

Note that all computations used in Table 5 are identical to those already usedin applying linear regression to estimate trend. The only difference is that values ofY have been transformed, either by taking their square roots or by taking theircommon (base 10) logarithms. Observe that the linear regression procedure iswholly inappropriate for describing the sharp growth trend that progresses bymultiples of 10 in each period [column (2), Table 5]. The linear equation [Yt � a �b(x)] produces a 2011 forecast of 85,224, when the actual value of the series willbe 1,000,000! (We are assuming, of course, that the series will grow by a factor of10 in 2011 and subsequent years.) The linear equation based on the square roottransformation helps very little with this explosive time series.The transformation results in a 2011 estimate of 94,249, hardly much of animprovement.

[1Yt = a + b1x2]

11Y2.

93

= 3 and 92

32 = 9


TAB

LE

5S

quar

e R

oot

and

Log

arit

hm o

f a

Tim

e S

erie

s E

xhib

itin

g R

apid

Gro

wth

Year

Ori

gina

l Va

lue

ofTi

me-

S

erie

s Va

riab

le

Squ

are

Roo

t of

Tim

e-S

erie

s Va

riab

le

Log

arit

hm

of T

ime-

Ser

ies

Vari

able

Cod

ed

Tim

e Va

lue

Tim

e Va

lue

Squ

ared

Col

umns

(2

) by

(5)

Col

umns

(3

) by

(5)

Col

umns

(4)

by (

5)Tr

end

Est

imat

es(X

)(Y

)(

) t1Y

(log

Y)

(x)

(x2 )

(xY)

(x) t

1Y(x

log

Y)Y t

1Yt

log

Y t(1

)(2

)(3

)(4

)(5

)(6

)(7

)(8

)(9

)(1

0)(1

1)(1

2)

2006

103.

161.

0-

24

-20

-6.

32-

2-

19,7

18-

51.0

1.0

2007

100

10.0

02.

0-

11

-10

0-

10.0

-2

1,27

120

.62.

020

081,

000

31.6

23.

00

00

00

22,2

6092

.23.

020

0910

,000

100.

004.

01

110

,000

100.

004

43,2

4916

3.8

4.0

2010

100,

000

316.

235.

02

420

0,00

063

2.46

1064

,238

235.

45.

0

n �

511

1,30

046

1.01

15.0

010

209,

880

716.

1410

.011

,300

461.

0115

.0

=1,

000,

000

=an

tilo

g 6.

0=

94,2

49=

85,2

24

Yt1

20112=

ant

ilog

3.0

+ 1.

0132

Yt1

20112

= [9

2.2

+71

.6132]

2Y

t120

112

= 22

,260

+

20,9

88132

log

Y=

3.0

+1.

01x2

1Yt

=92

.2 +

71.61x2

Yt

= 2

2,26

0 +

20,9

881x2

b=

©1x

log

Y2

©1x

2 2

= 10 10

=1.

0b

=

©1x1Y2

©1x

2 2 =

716.

14

10

= 71

.6b

=

©1x

Y2

©1x

2 2=

209,

880

10 =

20,9

88

a=

© lo

g Y

n =

15 5 =

3.0

a=

©1Y n

=

461.

01

5

=92

.2a

=

©Y n

=

111,

300

5 =

22,2

60

log

Y=

a+

b1x2

1Yt

=a

+b1

x2Y

t=

a+

b1x2



Only the logarithmically transformed linear equation [log Y = a � b(x)] givesus the exact value for 2011. In effect, what we have done is “straighten out” thegrowth trend by taking the common logarithm of Y, thus permitting the use of lin-ear regression to extrapolate trend. This does not mean that the trend is actuallylinear (obviously, it is not). It only means that we can use linear regression to makea precise forecast of the value of the nonlinear trend in 2011 or any other year.Note, however, that to make a trend estimate for 2011, we must convert the loga-rithmic value (6.0) back into the original value of the Y variable. This is done(Table 5) by taking the antilogarithm of 6.0 (abbreviated “antilog 6.0”), which is1,000,000 (an antilogarithm is the number corresponding to the logarithm, e.g.,antilog 2 � 100, antilog 3 � 1,000, antilog 4 � 10,000, and so on). Note also thatroots must also be reconverted by raising the trend estimate by the appropriatepower (see Table 5).

Now that we understand some of the fundamentals of estimating growthtrends, let us return to our example of the manager of the small municipalityand the problem of compound interest. We know from the compound interestmodel that $1,000 invested at an interest rate of 10 percent (compoundedannually) will accumulate to $6,727.47 after 20 years, namely, by the end of1990. That is,

When earlier we used the linear regression equation to extrapolate to 1990, weobtained $3,399.50. This is $3,327.97 in error ($6,727.47 - 3,399.50 =

$3,327.97). But what can we do about this very large error, given that we wouldlike to use linear regression to forecast the growth trend? A simple way to dealwith this problem is to find the base 10 logarithms of all values of Y and usethese with the linear regression equation. This has been done in Table 6, whichshows that the 1990 trend estimate based on the logarithmic transformation is logYt(1990) = 3.828. When we take the antilogarithm of this value, we find that itequals approximately $6,730, an estimate that is very close to that produced bythe compound interest formula (the small error is due to rounding). Hence,through a logarithmic transformation of the time-series variable, we have appliedlinear regression to a nonlinear growth process. The same kinds of logarithmictransformations are regularly used in public policy analysis to extrapolate otherimportant growth trends, including national income, population, and governmentexpenditures.

Catastrophe MethodologyNo matter how successfully we adapt linear regression to problems of forecastinggrowth and decline, there is one condition that must be present: The processeswe seek to understand and extrapolate must be smooth and continuous. Yet as wehave seen (Figure 8[e]), many time series are discontinuous. It is here where

S20 = $6,727.50

= 11 + 0.102201$1,0002

Sn = 11 + r2n S0


TABLE 6

Linear Regression with Logarithmic Transformation of Time Series

YearsTime-Series

Variable

Logarithm ofTime-Series

VariableCoded Time

ValueSquared

Time ValueColumns (3)

By (4)Trend

Estimate(X) (Y) (log Y) (X) (X2) (X log Y) (log Yt)(1) (2) (3) (4) (5) (6) (7)

1981 1100 3.041 -9 81 -27.369 3.04131982 1210 3.083 -7 49 -21.581 3.08271983 1331 3.124 -5 25 -15.620 3.12411984 1464 3.165 -3 9 -9.495 3.16551985 1611 3.207 -1 1 -3.207 3.20691986 1772 3.248 1 1 3.248 3.24831987 1949 3.290 3 9 9.870 3.28971988 2144 3.331 5 25 16.655 3.33111989 2358 3.373 7 49 23.611 3.37251990 2594 3.414 9 81 30.726 3.4139

n �10 17533 32.276 0 330 6.838 32.2760

log Yt = a + b1x2

a =

© log Y

n =

32.276

10 = 3.2276

b =

©1x log Y2

©1x22 =

6.838

330 = 0.0207

log Yt = 3.2276 + 0.02071x2

log Yt119902 = 3.2276 + 0.02071292 = 3.828

Yt119902= antilog 3.828 = $6,745.27


25See Isnard and Zeeman, “Some Models from Catastrophe Theory in the Social Sciences.” RenéThom’s major work is Stabilite structurelle et morphogenese, trans. D. II. Fowler (New York:W. A. Benjamin, 1975). Related to catastrophe methodology is work on chaos theory, which offersexplanations of discontinuous change. See Prigogine and Stengers, Order Out of Chaos.

catastrophe methodology, a field within the special branch of mathematics calledtopography, provides useful insights. Catastrophe methodology, which involvesthe systematic study and mathematical representation of discontinuous processes,is specifically designed to forecast trends when small changes in one variable (e.g.,time) produce sudden large changes in another variable. The point at which asmall change in one variable results in a sudden and dramatic shift in anothervariable is called a catastrophe, a term that refers to a mathematical theoremthat classifies discontinuous processes into five major types rather than to anysense of impending doom or disaster. According to its founder, René Thom, it is amethodology (not a theory) for studying the elementary types of discontinuity innature and society.25



Catastrophe methodology is too complex to survey here. We should neverthe-less be aware of some of its major assumptions and applications to public policyanalysis:

1. Discontinuous processes. Many of the most important physical, biological,and social processes are not only curvilinear, but also abrupt anddiscontinuous. An example of discontinuous processes in the social realm isthe sudden shifts in public attitudes that sometimes follow smooth andgradually evolving changes in public opinion.

2. Systems as wholes. Social systems as a whole frequently exhibit changes thatare not the simple sum of their parts. Sudden shifts may occur in the structureof a social system as a whole, even though the system’s parts may havechanged gradually and smoothly. For example, public opinion on major policyissues may suddenly diverge, making for an intense debate or confrontation,while at the same time the opinions of individual citizens evolve gradually.Similarly, public policies may shift abruptly, even though the direction ofpublic opinion has changed gradually.

3. Incremental delay. In attempting to maintain or build public support, policymakers tend to choose policies that involve incremental changes of existingroutines and practices. Incremental choice involves “the continual successivecomparison of the adopted policy with all nearby alternatives.”26 Delay is aconsequence of many factors: incomplete information, the prevalence ofintuitive (ordinary common sense) forms of analysis, political loyalties andcommitments, institutional inertia, and historical precedent.

4. Catastrophic policy change. Incremental policy making delays catastrophicchange until the last possible moment, partly because smoothly evolvingchanges in the opinion of contending groups do not appear to requireabrupt changes of direction. At a given point in time, policy makers arecompelled to make sudden and discontinuous shifts in policy in order toretain popular support.

One application of catastrophe methodology in public policy analysis is in thearea of public opinion.27 A sudden and discontinuous shift in West German energypolicy, for example, has been explained by using catastrophe methodology to showhow gradual changes in public opinion, together with incremental delay, produced adecision to suddenly terminate plans for the construction of a nuclear power plantin Kaiserstuhl, an agricultural area adjacent to the Rhine River in the state of BadenWurttemberg.28 Following public hearings, construction was begun on the plant,only to be followed by sit-ins and the forcible removal of demonstrators from thesite. Throughout this period, public opinion gradually shifted in favor of the

26Isnard and Zeeman, “Some Models from Catastrophe Theory,” p. 52. Incremental delay is normallylabeled the “Delay Rule” (change policy in the direction that locally increases support). Other rules are“Maxwell’s Rule” (change policy to where support is maximum) and the “Voting Rule” (change policyto where the base, rather than the maximum, support is greatest).27See Isnard and Zeeman, “Some Models from Catastrophe Theory,” pp. 45–60.28See Rob Coppock, “Decision-Making When Public Opinion Matters,” Policy Sciences 8 (1977):135–46.



29For a similar case, although presented in terms of “speculative augmentation” rather thancatastrophe methodology, see Charles O. Jones, Clean Air (Pittsburgh, PA: University of PittsburghPress, 1975). Interestingly, Jones challenges the explanatory value of “disjointed incrementalism,”whereas Isnard and Zeeman see it as a major factor contributing to catastrophic change.30The classic discussion of relations between theory and empirical data in the social sciences is RobertK. Merton, Social Theory and Social Structure, rev. ed. (Glencoe, IL: Free Press, 1957), pp. 95–99.

farming population in the region. Finally, after a number of incremental policychanges and considerable cost to the government, the project was abruptlyabandoned.29

Catastrophe methodology provides concepts and techniques for understandingdiscontinuous policy processes. Yet it is a set of concepts and methods, rather than atheory. As such, it rests on the assumption that discontinuous processes observed inthe past will repeat themselves in the future. Although catastrophe methodologyattempts to provide sound reasons for the occurrence of future events (as contrastedwith simple beliefs that patterns observed in the past will repeat themselves inthe future), it is best viewed as a form of extrapolative forecasting. It is more away to think about discontinuous policy futures than to make predictions derivedfrom theory.

THEORETICAL FORECASTINGTheoretical forecasting methods help analysts make predictions of future societalstates on the basis of theoretical propositions and current and historical data. In con-trast to extrapolative forecasting, which uses assumptions about historical recurrenceto make projections, theoretical forecasting is based on propositions and laws aboutcause and effect. Whereas the logic of extrapolative forecasting is essentially induc-tive, the logic of theoretical forecasting is essentially deductive.

Deductive reasoning employs general statements (axioms, laws, propositions) toestablish the truth or falsity of other more specific statements, including predictions.In policy analysis, deductive reasoning is most frequently used in connection withcausal arguments that seek to establish that if one event (X) occurs, another event(Y) will follow. Although the distinguishing feature of theoretical forecasting is thatpredictions are deduced from propositions and laws, deduction and induction areinterrelated. The persuasiveness of a deductive argument is considerably increased iftheoretically deduced predictions are observed time after time through empiricalresearch. Similarly, an isolated empirical generalization (“The enemy chose towithdraw when threatened”) is much more persuasive if backed by one or moreassumptions contained in a theory (“The greater the costs of an alternative, the lesslikely that it will be chosen”).30

In this section, we consider several procedures that assist analysts in makingtheoretical forecasts: theory mapping, causal modeling, regression analysis, point andinterval estimation, and correlational analysis. Some of these procedures (e.g., theorymapping) are concerned with ways to identify and systemize theories, whereas others(e.g., regression analysis) provide better estimates of future societal states alreadypredicted from theory. Before we begin, however, it is essential to recognize that none



of these techniques actually makes predictions; only theory can make predictions.In the words of a key contributor to the area of theoretical modeling,

Owing to the inherent nature of the scientific method, there is a gap between thelanguages of theory and research. Causal inferences belong on the theoreticallevel, whereas actual research can only establish covariations and temporalsequences. . . . As a result, we can never actually demonstrate causal lawsempirically.31

Theory MappingTheory mapping is a technique that helps analysts identify and arrange keyassumptions within a theory or causal argument.32 Theory mapping can assist inuncovering four types of causal arguments: convergent, divergent, serial, andcyclic. Convergent arguments are those in which two or more assumptions aboutcausation are used to support a conclusion or claim. Divergent arguments are thosein which a single assumption supports more than one claim or conclusion. By con-trast, in serial arguments one conclusion or claim is used as an assumption tosupport a series of further conclusions or claims. Finally, cyclic arguments are serialarguments in which the last conclusion or claim in a series is connected with thefirst claim or conclusion in that series. The consequences of a cyclic argument maybe positively or negatively self-reinforcing. These four types of causal argumentsare illustrated in Figure 11.

A theory may contain a mixture of convergent, divergent, serial, and cyclicarguments. Several procedures may be used to uncover the overall structure of anargument or theory: (1) separate and number each assumption, which may be anaxiom, law, or proposition; (2) underline the words that indicate claims (therefore,thus, hence) or assumptions used to warrant claims (since, because, for); (3) whenspecific words (therefore, and so on) have been omitted, but are clearly implied,supply the appropriate logical indicators in brackets; and (4) arrange numberedassumptions and claims in an arrow diagram that illustrates the structure of thecausal argument or theory.

Let’s apply these procedures to an important theory about public policymaking. This theory, called “public choice,” is concerned with the institutionalconditions underlying efficient (and inefficient) government administration.33

Public choice theorists have attempted to show on theoretical grounds how variousinstitutional arrangements, particularly democratic forms of administration, willresult in greater efficiency and public accountability. Public choice theory is alsocontroversial, because it suggests that major problems of American government(inefficiency, inequity, unresponsiveness) are partly a consequence of the teachings

33See Vincent Ostrom, The Intellectual Crisis in American Public Administration, rev. ed. (University:University of Alabama Press, 1974).

31Hubert M. Blalock Jr., Causal Inferences in Nonexperimental Research (Chapel Hill: Universityof North Carolina Press, 1964), p. 172.32Adapted from John O’Shaughnessy, Inquiry and Decision (New York: Harper & Row, 1973),pp. 58–61; and M. C. Beardsley, Thinking Straight (Englewood Cliffs, NJ: Prentice Hall, 1950).



of public administration, a discipline that for fifty and more years has emphasizedcentralization, hierarchy, and the consolidation of administrative powers.

In an effort to explain the inefficiency of governmental institutions, one publicchoice theorist, Vincent Ostrom, offers the following argument, which has beenmapped by underlining key words and supplying logical indicators in brackets:

Gordon Tullock in The Politics of Bureaucracy (1965) analyzes the consequenceswhich follow when [IT IS ASSUMED THAT] rational, self-interested individualspursue maximizing strategies in very large public bureaucracies. Tullock’s“economic man” is [THEREFORE] an ambitious public employee who seeks toadvance his career opportunities for promotions within a bureaucracy.

Since career advancement depends upon favorable recommendations by hissuperiors, a career-oriented public servant will act so as to please his superiors.[THEREFORE] Favorable information will be forwarded; unfavorable infor-mation will be repressed. [SINCE] Distortion of information will diminishcontrol and create expectations which diverge from events generated by actions.Large-scale bureaucracies will thus become error-prone and cumbersome in

Increasing number ofstrikes by publicemployees

Declining output peremployee in publicorganizations

Increasingcentralization of publicmanagement

Convergent

(a)

Increasing alienationof public employees



Divergent

(b)

Increasing costs ofpublic services



Serial

(c)

Increasing alienationof public employees



Cyclic

(d)

FIGURE 11Four types of causal arguments



[SINCE] 1. Public employees working in very large public bureaucracies arerational, self-interested individuals who pursue maximizing strategies(“economic man”).

[AND] 2.The desire for career advancement is a consequence of rational self-interest.

[THEREFORE] 3. Public employees strive to advance their career opportunities.[SINCE] 4.The advancement of career opportunities is a consequence of

favorable recommendations from superiors.[AND] 5. Favorable recommendations from superiors are a consequence of

receiving favorable information from subordinates.[THEREFORE] 6. Subordinates striving to advance their careers will forward favorable

information and suppress unfavorable information.[SINCE] 7.The repression of unfavorable information creates errors by

management, reducing their flexibility in adapting to rapidly changingconditions.

[AND] 8.The repression of unfavorable information diminishes managerialcontrol.

[THEREFORE] 9. Managers compensate for their loss of control by attempting totighten control. Compensatory attempts to tighten control furtherencourage the repression of unfavorable information and (6) themagnification of management errors (7). Further (8) loss of controlproduces attempts to tighten control.

adapting to rapidly changing conditions. [SINCE] Efforts to correct the mal-functioning of bureaucracy by tightening control will simply magnify errors. Adecline in return to scale can [THEREFORE] be expected to result. [BECAUSE]The larger the organization becomes, the smaller the percent of its activities willrelate to output and the larger the proportion of its efforts will be expended onmanagement.34

In the argument just quoted, we have already used procedures (2) and (3) oftheory mapping. We have underlined words that indicate claims or assumptionsunderlying claims and supplied missing logical operators in brackets. Observe thatthe argument begins with an assumption about human nature (“economic man”),an assumption so fundamental that it is called an axiom. Axioms are regarded astrue and self-evident; that is, they are believed to require no proof. Note also thatthis is a complex argument whose overall structure is difficult to grasp simply byreading the passage.

The structure of the argument can be more fully described when we completesteps (1) and (4). Using procedure (1), we separate and number each claim andits warranting assumption, changing some of the words to improve the clarity ofthe original argument.

34Ibid., p. 60. Underlined words and bracketed insertions have been added for purposes of illustratingtheory-mapping procedures.



By separating and numbering each claim and warranting assumption, we havebegun to expose the logical structure of the argument. Note that there are differenttypes of causal arguments in this theory of governmental inefficiency. The first partof the quotation contains a serial argument that emphasizes the influence ofemployee motivation on information error and managerial control. The secondpart of the quotation contains another serial argument, but one that stresses theimportance of the size of public bureaucracies. In addition, there is one cyclic argu-ment (error magnification) and several divergent and convergent arguments.

The structure of the overall argument cannot be satisfactorily exposed until wecomplete procedure (4) and draw an arrow diagram that depicts the causal structureof the argument (Figure 12). Observe, first, that there are two serial arguments: (1,2, 3) and (5, 4, 3). The second of these (5, 4, 3) can stand by itself and does notrequire the first (1, 2, 3), which rests exclusively on deductions from the axiom of“economic man.” Second, there is one divergent argument (5, 4, 6) that indicatesthat the same factor (5) has multiple consequences. There are also three convergent

[AND] 10. Compensatory tightening of control is a consequence of the size ofpublic bureaucracies.

[SINCE] 11. Compensatory tightening of control requires the expenditure of alarger proportion of effort on management activities, and a smallerproportion of output activities.

[THEREFORE] 12.The larger the public bureaucracy, the smaller the proportion of effortexpanded on output activities in relation to size, that is, the less thereturn to scale.

123

4

5 6 7 8

1211910

1. Rational Self-Interest (RSA)2. Desire for Career Advancement (DCA)3. Striving to Advance Career Opportunities (SAC)4. Favorable Recommendations for Superiors (FRS)5. Favorable Information from Subordinates (FIS)6. Suppression of Unfavorable Information (SUI)7. Management Error (ME)8. Loss of Managerial Control (LMC)9. Compensatory Tightening of Control (CTC)10. Size of Public Bureaucracies (SPB)11. Expenditures on Management (EM)12. Return to Scale (RS)

FIGURE 12Arrow diagram illustrating the causal structure of an argument



arguments (4, 2, 3), (3, 5, 6), and (8, 10, 9), which suggest in each case that there aretwo factors that explain the occurrence of the same event.

The arrow diagram also exposes two central features of this theoretical argument.It makes explicit an important potential relationship, denoted by the broken linebetween (10) and (6), which suggests that the size of public bureaucracies mayindependently affect the tendency to forward or suppress information. At the sametime, the arrow diagram helps identify a cyclic argument (6, 7, 8, 9) that is crucial tothis theory. This part of the theory argues, in effect, that we may expect a cumulativeincrease in the amount of management error as a result of the self-reinforcing relation-ship among information suppression (6), management error (7), loss of managementcontrol (8), compensatory tightening of control (9), consequent further encourage-ment of information suppression (6), and so on. This cyclic argument, also known asa positive feedback loop (positive because the values of variables in the cycle continueto grow), suggests that any forecast based on public choice theory will predict acurvilinear pattern of growth in management error and government inefficiency.

If we had not used theory-mapping procedures, it is doubtful that we wouldhave uncovered the mixture of convergent, divergent, serial, and cyclic argumentscontained in the structure of the theory. Particular causal assumptions—forexample, the assumption that the size of public bureaucracies may combine withcompensatory tightening of control to repress unfavorable information—arenot clearly stated in the original passage. Because Theory mapping makes thestructure of claims and assumptions explicit.

Theoretical ModelingTheoretical modeling refers to a broad range of procedures for constructingsimplified representations (models) of theories. Modeling is an essential part oftheoretical forecasting, because analysts seldom make theoretical forecasts directlyfrom theory. Whereas analysts may begin with theories, they must develop modelsof these theories before they can actually forecast future events. Theoreticalmodeling is essential because theories are frequently so complex that they must besimplified before being applied to policy problems, and because the process ofanalyzing data to assess the plausibility of a theory involves constructing and testingmodels of theories, not the theories themselves.

Think of how you may compare and contrast models in terms of their aims (de-scriptive, normative), forms of expression (verbal, symbolic, procedural), andmethodological functions (surrogates, perspectives). The majority of theoreticalforecasting models are primarily descriptive, because they seek to predict ratherthan optimize some valued outcome. For the most part, these models are alsoexpressed symbolically, that is, in the form of mathematical symbols and equations.Note that we have already used several symbolic models in connection with extrap-olative forecasting, for example, the regression equation or model [Yt � a � b (x)].Although it is not a causal model (because no explicit causal arguments are offered),it is nevertheless expressed symbolically.

In public policy analysis, a number of standard forms of symbolic models assist inmaking theoretical forecasts: causal models, linear programming models, input-output



35For a discussion of these and other standard model forms, see Martin Greenberger and others, Modelsin the Policy Process (New York: Russell Sage Foundation, 1976), Chapter 4; and Saul I. Gass andRoger L. Sisson, eds., A Guide to Models in Governmental Planning and Operations (Washington, DC:U.S. Environmental Protection Agency, 1974). Whereas these models are here treated as descriptive andsymbolic, some (e.g., linear programming) are often treated as normative. Similarly, others (e.g., systemdynamics) are typically treated as procedural (or simulation) models. This should accentuate the pointthat the distinctions among types of models are relative and not absolute.36Sewall Wright, “The Method of Path Coefficients,” Annals of Mathematical Statistics 5 (1934): 193;quoted in Fred N. Kerlinger and Elazar J. Pedhazur, Multiple Regression in Behavioral Research(New York: Holt, Rinehart and Winston, 1973), p. 305.

models, econometric models, microeconomic models, and system dynamics models.35

As it is beyond the scope of this text to detail each of these model forms, we confine ourattention to causal models. In our review, we outline the major assumptions, strengths,limitations, and applications of causal modeling.

Causal ModelingCausal models are simplified representations of theories that attempt to explain andpredict the causes and consequences of public policies. The basic assumption ofcausal models is that covariations between two or more variables—for example,covariations that show that increases in per capita income occur in conjunction withincreases in welfare expenditures in American states—are a reflection of underlyinggenerative powers (causes) and their consequences (effects). The relation betweencause and effect is expressed by laws and propositions contained within a theoryand modeled by the analyst. Returning to our illustration from public choice theory,observe that the statement “The proportion of total effort invested in managementactivities is determined by the size of public organizations” is a theoretical proposi-tion. A model of that proposition might be Y � a � b (X), where Y is the ratio ofmanagement to nonmanagement personnel, a and b are constants, and X is the totalnumber of employees in public organizations of different sizes.

The strength of causal models is that they force analysts to make causal assump-tions explicit. The limitation of causal models lies in the tendency of analysts toconfuse covariations uncovered through statistical analysis with causality. Causalinferences always come from outside a model, that is, from laws, propositions, andbasic axioms within some theory. In the words of Sewall Wright, one of the earlypioneers in this type of modeling, causal modeling procedures are “not intended toaccomplish the impossible task of deducing causal relations from the values of thecorrelation coefficients.”36

Causal modeling has been used to identify the economic, social, and politicaldeterminants of public policies in issue areas ranging from transportation to health,education, and welfare.37 One of the major claims of research based on causal mode-ling is that differences in political structures (e.g., single versus multiparty polities)do not directly affect such policy outputs as education and welfare expenditures.

37For a review, see Thomas R. Dye and Virginia H. Gray, “Symposium on Determinants of PublicPolicy: Cities, States, and Nations,” Policy Studies Journal 7, no. 4 (summer 1979): 279–301.



On the contrary, differences in levels of socioeconomic development (income,industrialization, urbanization) determine differences in political structures, whichin turn affect expenditures for education and welfare. This conclusion is contro-versial because it appears to contradict the commonly shared assumption that thecontent of public policy is determined by structures and processes of politics,including elections, representative mechanisms, and party competition.

One of the main statistical procedures used in causal modeling is path analysis,a specialized approach to regression analysis that uses multiple (rather than single)independent variables. In using path analysis, we hope to identify those independentvariables (e.g., income) that singly and in combination with other variables (e.g.,political participation) determine changes in a dependent variable (e.g., welfareexpenditures). An independent variable is presumed to be the cause of a dependentvariable, which is presumed to be its effect. Estimates of cause and effect are calledpath coefficients, which express one-directional (recursive) causal relationshipsamong independent and dependent variables.

A standard way of depicting causal relationships is the path diagram. A pathdiagram looks very much like the arrow diagram used to map public choice theory,except that a path diagram contains estimates of the strength of the effects ofindependent variables on dependent variables. In Figure 13 a path diagram has beenused to model part of public choice theory. The advantage of path analysis andcausal modeling is that they permit forecasts that are based on explicit theoreticalassumptions about causes (the number of employees in individual public organiza-tions) and their consequences (the ratio of managerial to nonmanagerial staff andcosts in tax dollars per unit of public service). The limitation of these procedures, asalready noted, is that they are not designed for the impossible task of inferringcausation from estimates of relationships among variables. Although the absence ofa relationship may be sufficient to infer that causation is not present, only theorypermits us to make causal inferences and, hence, predictions.

1EPO

2RMN

3CUS

p32

p31

p21

1. Employees Per Public Organization (EPO)2. Ratio of Managers to Non-Managers (RMN)3. Costs in Tax Dollars Per Unit of Service (CUS)

e1

e2

FIGURE 13Path diagram illustrating a model ofpublic choice theory

Note: The symbol p designates a causal path and the sub-scripts specify the direction of causation: p31 means thatvariable 3 is caused by variable 1.Variables that have noantecedent cause are called exogenous variables (i.e.,their cause is external to the system), whereas all othersare called endogenous (i.e., their cause is internal to thesystem).The symbol e (sometimes designated as u) is anerror term, defined as the unexplained (residual) vari-ance in an endogenous variable that is left over aftertaking into account the effects of the endogenous vari-able that precedes it in the path diagram. Error termsshould be uncorrelated with each other and with othervariables.



38For a thorough and readable treatment of multiple regression and related techniques, see Kerlingerand Pedhazur, Multiple Regression in Behavioral Research; and Joshua D. Angrist and Jorn-SteffenPischke, Mostly Harmless Econometrics (Princeton, NJ: Princeton University Press, 2008).39If this point is not clear, you should return to the section on linear trend estimation and review Figure 7.40Consult the distinction between surrogate and perspective models in the chapter “Structuring PolicyProblems” and the example of the annual rainfall rate and reservoir depth. This example shows vividlythat regression analysis cannot answer questions about which variable predicts another.

Regression AnalysisA useful technique for estimating relationships among variables in theoretical fore-casting models is regression analysis. Regression analysis, which has already beenconsidered in our discussion of trend estimation, is a general statistical procedurethat yields estimates of the pattern and magnitude of a relationship between adependent variable and one or more independent variables. When regression analy-sis is performed with one independent variable, it is called simple regression; if thereare two or more independent variables, it is called multiple regression. Althoughmany theoretical forecasting problems require multiple regression, we limitourselves to simple regression in the remainder of this section.38

Regression analysis is useful in theoretical modeling. It provides summarymeasures of the pattern of a relationship between an independent variable and adependent variable. These summary measures include a regression line, which permitsus to estimate values of the dependent variable by knowing values of the independentvariable, and an overall measure of the vertical distances of observed values from theregression line. A summary measure of these distances, as we shall see, permits us tocalculate the amount of error contained in a forecast. Because regression analysis isbased on the principle of least squares, it has a special advantage already noted inconnection with linear trend estimation. It provides the one best “fit” between data andthe regression line by ensuring that the squared distances between observed andestimated values represent a minimum or least value.39

A second advantage of regression analysis is that it forces the analyst todecide which of two (or more) variables is the cause of the other, that is, to specify theindependent (cause) and dependent (effect) variable. To make decisions about causeand effect, however, analysts must have some theory as to why one variable shouldbe regarded as the cause of another. Although regression analysis is particularly wellsuited to problems of predicting effects from causes, the best that regression analysiscan do (and it does this well) is to provide estimates of relationships predicted by atheory. Yet it is the theory and its simplified representation (theoretical model), andnot regression analysis, that do the predicting. Regression analysis can only provideestimates of relationships between variables that, because of some theory, have beenstated in the form of predictions.40 For this reason, analysts should employ theory-mapping procedures before using regression analysis.

To illustrate the application of regression analysis to problems of theoreticalforecasting, let us suppose that municipal policy makers wish to determine thefuture maintenance costs of police patrol vehicles under two alternative policies.One policy involves regular police patrols for purposes of traffic and crime control.The total annual maintenance costs for ten patrol cars were $18,250, or $1,825 pervehicle. The total mileage for the ten vehicles was 535,000 miles, that is, 53,500



miles per vehicle. A new policy now being considered is one that would involve“high-impact” police patrols as a way to create greater police visibility, respondmore rapidly to citizens’ calls for assistance, and, ultimately, deter crime by increasingthe probability that would-be offenders are apprehended.41

Local policy makers are interested in any means that will reduce crime. Yet thereare increasing gaps between revenues and expenditures, and citizens’ groups havepressed for a cutback in fuel costs. Policy makers need some way to forecast, on thebasis of their own mileage and maintenance records, how much it will cost if severalof the ten vehicles are driven an additional 15,000 miles per year.

This forecasting problem may be effectively dealt with by using regressionanalysis. The relationship between cause and effect is reasonably clear, and theprimary determinant of maintenance costs is vehicle usage as measured by milesdriven.42 A municipal policy analyst might therefore plot the values of the independ-ent (X) and dependent (Y) variables on a scatter diagram (scatterplot), which willshow the pattern of the relationship (linear-nonlinear), the direction of the relation-ship (positive-negative), and the strength of the relationship (strong-moderate-weak)between annual mileage per vehicle and annual maintenance costs per vehicle (Figure 14). We assume that the pattern, direction, and strength of the relationshipare linear, positive, and strong, as shown in Figure 14(a).

Classical linear regression assumes that variables are related in a linear pattern.Although linear regression may also be used with negative linear relationships(Figure 14[b]), it will produce serious errors if applied to curvilinear patterns such asthose illustrated in Figures 14(c) and (d). In such cases, as we saw earlier in thediscussion of curve fitting, we must either use nonlinear regression or transformvalues of variables (e.g., by taking their logarithms) prior to applying conventionallinear regression techniques. Regression analysis will yield less reliable estimateswhen data are widely scattered, as in Figure 14(e). If data indicate no pattern orrelationship (Figure 14[f]), the best estimate of changes in Y due to X is the average(mean) of values of Y.

Recall from our discussion of trend estimation that the straight line describ-ing the relationship between X and Y variables is called a regression line. Theequation used to fit the regression line to observed data in the scatter diagram isidentical to that used to estimate linear trend, with several minor differences.The symbol Yt (the subscript t refers to a trend value) is replaced with Yc (thesubscript c refers to a computed, or estimated, value). These different subscriptsremind us that here, regression is applied to two substantive variables, whereas in

42The question of causation is never certain, as illustrated by this example. Under certain conditions,maintenance costs may affect miles driven, for example, if increasingly cost-conscious managers orpolice officers limit vehicle use in response to knowledge of high expenses. Similarly, annual mileage forcertain vehicles may mean larger patrol areas, which in turn may be situated where road conditions arebetter (or worse) for cars.

41Under the sponsorship of the Law Enforcement Assistance Administration, high-impact patrollinghas been used, with mixed success, in a number of large municipalities. In the current recession, highgasoline prices have prompted a return to foot patrolling. See “Foot Patrols: Crime Analysis andCommunity Engagement to Further the Commitment to Community Policing,” Community PolicingDispatch 2 no. 2 (February 2009).



linear and nonlinear trend estimation, one of the variables is time. The formulafor the linear regression equation is

where

a � the value of Yc when X � 0, called the Y intercept because it showswhere the computed regression line intercepts the Y-axis

b � the value of changes in Yc due to a change of one unit in X, called the slopeof the regression line because it indicates the steepness of the straight line

X � a given value of the independent variable

Yc = a + b1X2

(a)

Y

X

Linear/Positive/Strong

(b)

Y

X

Linear/Negative/Strong

(c)

Y

X

Curvilinear/Positive/Strong

(d)

Y

X

Curvilinear/Negative/Strong

(e)

Y

X

Linear/Positive/Weak

(f)

Y

X

No Pattern or Relationship

mean of Y(Y)

FIGURE 14Scatterplot illustrating differentpatterns and relationships betweenhypothetical annual maintenancecosts and annual mileage per vehicle



A second minor difference between the regression and trend equations is thecomputation of values of a and b. In regression analysis, we do not work withoriginal values of Y and coded time values of X but with mean deviations. A meandeviation is simply the difference between a given value of X or Y and the average(mean) of all values of X or Y. The formulas necessary to calculate the mean andmean deviations for X and Y are

The values of a and b in the regression equation are computed with the followingformulas:

Let us now return to the forecasting problem facing our municipal policyanalyst. All data needed to solve the regression equation are provided in Table 7.Observe that column (1) lists the ten vehicles in the municipal fleet. Annualmileage (X) and maintenance costs (Y) are expressed in thousands of miles anddollars [columns (2) and (3)] for ease of calculation. At the bottom of columns(1) and (2), we have also calculated the sums of X (X) and Y ( Y) and theirrespective means ( and ). Columns (4) and(5) contain the deviations of each X and Y value from its respective mean. Thesemean deviations and show us the spread of each Xand Y value about its own mean. Note that the values of the mean deviationsum to zero [bottom of columns (4) and (5)], because the means of X and Y areaverages, which guarantee that values above the mean will cancel those below the mean.

The mean deviations (x and y) exhibit the spread of each value around the Xand Y means. Nevertheless, they cannot be used as a summary measure of thisspread, because they sum to zero. For this reason, mean deviations are squared[columns (7) and (8)], thus eliminating negative signs, and summed. These sums,called the sums of squares (which is a shortened form of “sums of the squares ofmean deviations”), provide an overall summary of the spread of X and Y valuesaround their means. Finally, in column (6) we have multiplied the mean deviations(x and y), obtaining deviation cross-products (xy). These cross-products are anexpression of the pattern and strength of the relationship between X and Y, but onethat takes into account the spread of X and Y scores.

1y = Y - Y21x = X - X2

Y = © Y/n = 1.785X = © X/n = 53.6©

a = Y - b1X2

b = ©1xy2

©1x22

mean deviation of Y = y = Y - Y

mean deviation of X = x = X - X

mean of Y = Y =

©Yn

mean of X =X =

©Xn


TAB

LE

7

Wor

kshe

et f

or E

stim

atin

g F

utur

e M

aint

enan

ce C

osts

fro

m A

nnua

l Mile

age

per

Veh

icle

Vehi

cle

Mile

age

per

Vehi

cle

(Tho

usan

ds

of M

iles)

Mai

nten

ance

Cos

ts

per V

ehic

le

(Tho

usan

ds o

f Dol

lars

)

XM

ean

Dev

iati

on(X

-X

)

YM

ean

Dev

iati

on(Y

-Y

)D

evia

tion

C

ross

-Pro

duct

XM

ean

Dev

iati

onS

quar

ed

YM

ean

Dev

iati

onS

quar

edE

stim

ated

Valu

es o

fN

umbe

r(X

)(Y

)(x

)(y

)(x

y)(x

2 )(y

2 )(Y

c)(1

)(2

)(3

)(4

)(5

)(6

)(7

)(8

)(9

)

124

0.45

-29

.5-

1.33

539

.38

870.

251.

7820

0.83

82

231.

25-

30.5

-0.

585

17.8

493

0.25

0.34

200.

806

330

0.85

-23

.5-

0.98

523

.15

552.

250.

9700

1.03

04

401.

15-

13.5

-0.

685

9.25

182.

250.

4690

1.40

05

481.

95-

5.5

0.11

5-

0.63

330

.25

0.01

301.

606

665

1.85

11.5

0.01

50.

173

132.

250.

0002

2.15

07

652.

3511

.50.

515

5.92

313

2.25

0.26

502.

150

860

2.65

6.5

0.81

55.

298

42.2

50.

6640

1.99

09

902.

5536

.50.

715

26.0

9813

32.2

50.

5110

2.95

010

913.

2537

.51.

415

53.0

6314

06.2

52.

0020

2.98

2

n�

10Σ

X=

536

©Yq

=17

.85

0.0

0.0

Σ(x

y) =

179

.54

Σ(x

2 ) =

561

0.5

Σ(y

2 ) =

7.0

2Σ

Yc=

17.

85X

= 5

3.6

Y�

1.78

5 b=

©1x

y2

©1x

2 2

= 17

9.54

5610

.5

=0.

032

a=

Yq

- b1Xq2

= 1

.785

-0.

0321

53.62

=0.

0698

or

0.0

7

Yc

= a

+b1

X2

=0.

07

+ 0

.0321X2

Y15

0 =

0.0

7 +

0.0

3211

502

=4.

87=

$4,

870


To estimate future maintenance costs under the high-impact police patrolprogram, we first solve the regression equation

by substituting calculations from the worksheet into formulas for a and b. Hence

This regression equation means that the regression line intercepts the Y-axisat 0.07 thousand dollars (i.e., $70) and that maintenance costs increase by0.032 thousand dollars (i.e., $32) for every 1,000 miles driven. On the basis ofthis information, we can estimate maintenance costs on the basis of knowledge ofannual miles per vehicle. Computed values (Yc) for each original value of X[column (9)] may be plotted on a graph to create the regression line. Finally, ourforecast indicates that the city will incur an additional $4,870 in expenses byadopting high-impact patrolling, because an additional 150,000 miles will resultin Y150 � 0.07 � 0.032(150) � $4.87 thousand dollars.

Point and Interval EstimationRegression analysis, as we have seen, enables us to fit a regression line to data insuch a way that the squared distances between data points and the regression lineare a minimum or least value (see Figure 7). This means that regression analysis notonly uses (rather than loses) information about variations in costs and mileage, butalso permits us to make a sound estimate of the central tendency in a relationship andcompare this estimate with observed values in a scatter diagram. Because the dis-tances between the regression line and individual data points are minimized, the re-gression estimate contains less error than other kinds of estimates, for example,those calculated on the basis of average costs per mile driven. Regression analysisalso allows us to calculate the probable error in our estimate. This may be done inseveral ways (Table 8). One way is simply to subtract each value of Yc [column (3)]from its corresponding observed value [column (2)]. This tells us how much distancethere is between each pair of estimated and observed values. The error of a regres-sion estimate, expressed by the distance between values of Y and Yc, increases indirect proportion to the amount of scatter. The greater the scatter, the less accurateis the estimate.

Another way to calculate error is to sum and average these distances. Becauseone of the properties of linear regression is that these distances equal zero whenthey are added up [i.e., ∑(Y - Yc) = 0], we must square them before we divide bythe number of cases to find the average. Column (4) of Table 8 shows that the av-erage squared error (also called the variance of an estimate) is 1.304. Thisexpression of error, although useful for certain calculations to be described

Yc = 0.07 + 0.0321X2

a = Y - b1Y2 = 1.785 - 0.032153.62 = 0.07

b = ©1xy2

©1x22 =

179.545610.5

= 0.032

Yc = a + b1X2




43For readers who recall the normal curve, the standard error is the standard deviation of theaverage squared error. One standard error is one standard deviation unit to the left or right of themean of a normal distribution and therefore includes the middle 68.3 percent of all values in thedistribution. In the example given, we have assumed for purposes of illustration that data arenormally distributed.

TABLE 8

Calculation of Standard Error of Estimated Maintenance Costs

ObservedMaintenance

Costs

EstimatedMaintenance

Costs

Observed Minus Estimated

Costs

Observed MinusEstimated Costs

SquaredVehicle Number (Y) (Yc) (Y - Yc) (Y - Yc)2

(1) (2) (3) (4) (5)1 0.45 0.838 -0.388 0.1512 1.20 0.806 0.394 0.1553 0.80 1.030 -0.230 0.0534 1.10 1.400 -0.300 0.0905 1.90 1.606 0.294 0.0866 1.80 2.150 -0.350 0.1237 2.30 2.150 0.150 0.0238 2.60 1.990 0.610 0.3729 2.50 2.950 -0.450 0.203

10 3.20 2.982 0.218 0.048

n � 10 ΣY � 17.85 ΣYc � 17.85 Σ(Y-Yc) � 0 1.304

Sy # x = C

©1Y - Yc22

n = C

©1Y - Yc22

10 = C

1.304

10= 0.36111

Yi = Yc ; Z1Sy # x2

Yi11502 = 4.87 ; 210.36112 = 4.87 ; 0.72222

= 4.14778 to 5.59222

= $4,147.78 to $5,592.22

shortly, is difficult to interpret. Fortunately, the square root of this value is a goodapproximation of the error that will occur about two-thirds of the time in aregression estimate.43 The square root of the average squared error, called thestandard error of estimate, is calculated with the following formula:44

Sy # x = C

©1Y - Yc22

n

44Note that we divide by n, because the ten vehicles constitute the entire population of vehicles underanalysis. If we were working with a sample, we would divide by n � 1. This is a way of providing anunbiased estimate of the variance in the population sampled. Most packaged computer programs use n � 1, even though sampling may not have occurred.



where

(Y - Yc)2

= the squared difference between observed and estimated valuesof the dependent variable

n = the number of cases

In the previous example, the standard error of estimate is

What this means is that any actual value of maintenance costs is likely tobe $361.11 above or below an estimated value about two-thirds of the time. Thefigure of $361.11 is one standard error (1 × $361.11), two standard errors are$722.22 (2 × $361.11), and three standard errors are $1,083.33 (3 × $361.11). Thestandard error gives us a probability interpretation of these values, because onestandard error unit will occur about two-thirds (actually 68.3 percent) of the time,two standard error units will occur about 95 percent (actually 95.4 percent) of thetime, and three standard error units will occur about 99 percent (actually 99.7 per-cent) of the time.

The standard error of estimate allows us to make estimates that take errorinto account systematically. Rather than make simple point estimates—that is,estimates that produce a single value of Yc—we can make interval estimates thatyield values of Yc expressed in terms of one or more standard units of error. If wewant to be 95 percent confident that our estimate is accurate (this is usuallyregarded as a minimum standard of accuracy), we will want to express ourestimate in terms of two intervals above and below the original point estimate.We can use the following formula to find out how much error we might expect inour point estimate of $4,870 [Y150 � 0.07 � 0.032(150) � 4.87 thousand dollars]95 percent of the time:

where

Yc � the point estimate of YZ � a standard error unit, which takes the value of 2 (95% confidence) in

this caseSy·x � the value (0.3611) of one standard error unit

Yi � an interval estimate of Y ( 0.72222);

= $4,147.78 to $5,592.22

= 4.87 ; 0.72222 = 4.14778 to 5.59222

= 4.87 ; 210.36112

Yi = Yc + Z1Sy # x2

= $361.11 in annual maintenance costs

= 1.1304 = 0.3611 thousand

Sy # x = C

©1Y - Yc22

n = C

1.30410



Correlation AnalysisRegression analysis has an additional feature of importance to theoretical forecast-ing: It permits us to use correlation analysis to interpret relationships. Recall thatdifferent scatter diagrams not only exhibit the pattern of a relationship but also itsdirection and strength (see Figure 14). Yet it is desirable that we have measures ofthe direction and strength of these relationships, not just the visual images con-tained in scatter diagrams. Two measures that yield such information can be calcu-lated from the worksheet already prepared for estimating future maintenance costs(Table 7). The first of these measures is the coefficient of determination (r2), whichis a summary measure or index of the amount of variation in the dependent vari-able explained by the independent variable. The second is the coefficient ofcorrelation (r), which is the square root of the coefficient of determination. Thecoefficient of correlation, which varies between �1.0 and �1.0, tells us whetherthe direction of the relationship is positive or negative and how strong it is. If r is0, there is no relationship, whereas a value of � 1.0 (i.e., positive or negative) indi-cates a maximal relationship. Unlike the coefficient of correlation (r), the coeffi-cient of determination (r2) takes a positive sign only and varies between 0.0 and1.0. The formulas for both coefficients are applied hereto our data on maintenancecosts and mileage (see Table 7):

The analysis carried out by hand in Tables 7 and 8 has been done with SPSS(Exhibit 2). Identical results have been obtained, but with greater accuracy andefficiency. When we inspect the SPSS output, we see that the correlation coefficient(r), designated in the output as R, is 0.905. We also see that the coefficient ofdetermination (r2), designated as R SQUARE, is 0.818. All coefficients are inagreement with our hand calculations. The output also provides the value of theintercept (CONSTANT � 0.07), which conforms to the output in Table 7, as doesthe slope of the regression line (B � 0.032) for the variable annual miles per vehicle.Finally, the ANOVA (analysis of variance) table provides a breakdown of that partof the total sum of squares due to the effects of regressing the dependent variable onthe independent variable (regression � 5.746). The output also provides that part ofthe sum of squares due to error, that is, residual � 1.275. If we add the regressionand residual sum of squares and divide this total into the regression sum of squares,we obtain the coefficient of determination (R). Thus,

These coefficients tell us that 82 percent of the variance in annual maintenancecosts is explained by annual mileage (r2 � 0.82) and that the direction and strength

5.7465.746 + 1.275

= 0.818 = 0.82

r = 2r2= 10.818 = 0.90

r2=

b1©xy2

©1y22 =

0.0321179.542

7.02 = 0.818 or 0.82



of the relationship between these two variables are positive and strong (r � 0.90).Regression analysis, when supplemented by interval estimation and coefficients ofdetermination and correlation, provides more information of direct relevance topolicy makers than other forms of estimation. For example, simple estimates ofaverage costs per mile, apart from their relative inaccuracy, neither providesummary measures of the direction and strength of relationships nor anticipateforecasting errors in a systematic way. In this and many other cases, regressionanalysis can assist policy makers in dealing with the uncertainties that accompanyefforts to predict the future.

EXHIBIT 2

SPSS Output for Table 7

Variables Entered/Removedb

Model Variables Entered Variables Removed Method

1 Annual Miles Per Vehicle (000s)a EnteraAll requested variables enteredbDependent Variable: Annual Maintenance Costs Per Vehicle ($000)

Model Summary

Model R R Square Adjusted R Square Std. Error of the Estimate

1 .905a .818 .796 .39917721aPredictors: (Constant), Annual Miles Per Vehicle (000s)

ANOVAb

Model Sum of Squares df Mean Square F Sig.

1 Regression 5.746 1 5.746 36.058 .000a

Residual 1.275 8 .159Total 7.020 9

aPredictors: (Constant), Annual Miles Per Vehicle (000s)bDependent Variable: Annual Maintenance Costs Per Vehicle ($000)

Coefficientsa



Model B Std. Error Beta t Sig.1 (Constant)Annual Miles Per 6.973E-02 .312 .223 .829

Vehicle (000s) 3.200E-02 .005 .905 6.005 .000

aDependent Variable: Annual Maintenance Costs Per Vehicle ($000)



JUDGMENTAL FORECASTINGIn to extrapolative and theoretical forecasting, empirical data and/or theories play acentral role. By contrast, judgmental forecasting techniques attempts to elicit andsynthesize informed judgments. Judgmental forecasts are often based on argumentsfrom insight, because assumptions about the creative powers of persons making theforecast (and not their social positions per se) are used to warrant claims about thefuture. The logic of intuitive forecasting is essentially abductive, because analystsbegin with a conjectured state of affairs (e.g., a normative future such as worldpeace) and then work their way back to the data or assumptions necessary tosupport the conjecture. Nevertheless, inductive, deductive, and abductive reasoningare never completely separable in practice. Judgmental forecasting is therefore oftensupplemented by various extrapolative and theoretical forecasting procedures.45

In this section, we review three intuitive forecasting techniques: Delphitechnique, cross-impact analysis, and the feasibility assessment technique. These andother techniques, most of which have been widely used in government and industry,are particularly well suited to the kinds of problems that may be described as messyor ill structured. Because one of the characteristics of ill-structured problems is thatpolicy alternatives and their consequences are unknown, it follows that in such cir-cumstances there are no relevant theories and/or empirical data to make a forecast.Under these conditions, judgmental forecasting techniques are particularly useful.

The Delphi TechniqueThe Delphi technique is a judgmental forecasting procedure for obtaining,exchanging, and developing informed opinion about future events. The Delphitechnique (named after Apollo’s shrine at Delphi, where Greek oracles sought toforesee the future) was developed in 1948 by researchers at the RANDCorporation and has since been used in many hundreds of forecasting efforts in thepublic and private sectors. Originally, the technique was applied to problemsof military strategy, but it was later applied to forecasts in other contexts:education, technology, marketing, transportation, mass media, medicine, informa-tion processing, research and development, space exploration, housing, budgeting,and the quality of life.46 Whereas the technique originally emphasized the use ofexperts to verify forecasts based on empirical data, Delphi began to be applied toproblems of values forecasting in the 1960s.47 The Delphi technique has been usedby analysts in countries ranging from the United States, Canada, and the UnitedKingdom to Japan and Russia.

45In fact, even large-scale econometric models are dependent on judgment. The best sources on thispoint are Ascher, Forecasting; Ascher, “The Forecasting Potential of Complex Models”; and McNown,“On the Use of Econometric Models.”46Thorough accounts may be found in Harold Sackman, Delphi Critique (Lexington, MA: D. C. Heathand Company, 1975); and Juri Pill, “The Delphi Method: Substance, Contexts, a Critique and anAnnotated Bibliography,” Socio-Economic Planning Sciences 5 (1971): 57–71.47See, for example, Nicholas Rescher, Delphi and Values (Santa Monica, CA: RAND Corporation, 1969).



Early applications of Delphi were motivated by a concern with the apparentineffectiveness of committees, expert panels, and other group processes. Thetechnique was designed to avoid several sources of distorted communication foundin groups: domination of the group by one or several persons, pressures to conformto peer group opinion, personality differences and interpersonal conflict, and thedifficulty of publicly opposing persons in positions of authority. To avoid theseproblems, early applications of the Delphi technique emphasized five basic princi-ples: (1) anonymity—all experts or knowledgeables respond as physically separatedindividuals whose anonymity is strictly preserved; (2) iteration—the judgments ofindividuals are aggregated and communicated back to all participating experts in aseries of two or more rounds, thus permitting social learning and the modificationof prior judgments; (3) controlled feedback—the communication of aggregatedjudgments occurs in the form of summary measures of responses to questionnaires;(4) statistical group response—summaries of individual responses are presented inthe form of measures of central tendency (usually the median), dispersion (theinterquartile range), and frequency distributions (histograms and frequencypolygons); and (5) expert consensus—the central aim, with few exceptions, is tocreate conditions under which a consensus among experts is likely to emerge as thefinal and most important product.

These principles represent a characterization of conventional Delphi, whichdominated the field well into the late 1960s. It should be contrasted with policyDelphi, a constructive response to the limitations of conventional Delphi and anattempt to create new procedures that match the complexities of policy problems. Inthe words of one of its chief architects,

Delphi as it originally was introduced and practiced tended to deal withtechnical topics and seek a consensus among homogeneous groups of experts.The Policy Delphi, on the other hand, seeks to generate the strongest possibleopposing views on the potential resolution of a major policy issue . . . apolicy issue is one for which there are no experts, only informed advocatesand referees.48

Policy Delphi is based on two of the same principles as conventional Delphi(iteration and controlled feedback), but it also introduces several new ones:

1. Selective anonymity. Participants in a policy Delphi remain anonymous onlyduring the initial rounds of a forecasting exercise. After contending argumentsabout policy alternatives have surfaced, participants are asked to debate theirviews publicly.

2. Informed multiple advocacy. The process for selecting participants isbased on criteria of interest and knowledge, rather than “expertise” perse. In forming a Delphi group, investigators therefore attempt to select asrepresentative a group of informed advocates as may be possible in specificcircumstances.

48Murray Turoff, “The Design of a Policy Delphi,” Technological Forecasting and Social Change 2,no. 2 (1970): 149–71. See also Harold A. Linstone and Murray Turoff, eds., The Delphi Method:Techniques and Applications (New York: Addison-Wesley, 1975).



3. Polarized statistical response. In summarizing individual judgments,measures that purposefully accentuate disagreement and conflict are used.Although averages may also be used, policy Delphi supplements these withvarious measures of polarization (e.g., the range and inter-quartile range)among individuals and groups.

4. Structured conflict. Starting from the assumption that conflict is a normalfeature of policy issues, every attempt is made to use disagreement anddissension for creatively exploring alternatives and their consequences. Inaddition, efforts are made to surface and make explicit the assumptions andarguments that underlie contending positions. The outcomes of a policyDelphi are nevertheless completely open, which means that consensus as wellas a continuation of conflict might be results of the process.

5. Computer conferencing. When possible, computer consoles are used tostructure a continuous process of anonymous interaction among physicallyseparated individuals. Computer conferencing eliminates the need for a seriesof separate Delphi rounds.

A policy Delphi may be conducted in a number of ways, depending on thecontext and the skill and ingenuity of the persons using the technique. Becausepolicy Delphi is a major research undertaking, it involves a large number of techni-cal questions, including sampling, questionnaire design, reliability and validity,and data analysis and interpretation. Although these questions are beyond thescope of this chapter,49 it is important to obtain an overall understanding of theprocess of conducting a policy Delphi, which can best be visualized as a series ofinterrelated steps:50

Step 1: Issue specification. The analyst must decide which specific issuesshould be addressed by informed advocates. For example, if the area ofconcern is national drug abuse policy, one of the issues might be “Thepersonal use of marijuana should or should not be legalized.” One ofthe central problems of this step is deciding what proportion of issuesshould be generated by participants, and what proportion should begenerated by the analyst. If the analyst is thoroughly familiar with theissue area, it is possible to develop a list of issues prior to the first roundof the Delphi. These issues may be included in the first questionnaire,although respondents should be free to add or delete issues.

Step 2: Selection of advocates. The key stakeholders in an issue area shouldbe selected. To select a group of advocates who represent conflictingpositions, however, it is necessary to use explicit sampling procedures.One way to do this is to use “snowball” sampling. The analyst beginsby identifying one advocate, usually someone who is known to be

49Unfortunately, there are few shortcuts to developing methodologically sound questionnaires for use inpolicy Delphis. On questionnaire construction, see Delbert C. Miller and Neil Salkind, Handbook ofResearch Design and Social Measurement, 6th ed. (Thousand Oaks, CA: Sage Publications, 2002). Onreliability and validity, see Fred N. Kerlinger and Howard B. Lee, Foundations of Behavioral Research,4th ed. (Belmont, CA: Cenage Learning, 2000).50See Turoff, “The Design of a Policy Delphi,” pp. 88–94.



influential in the issue area, and asking that person to name two otherswho agree and disagree most with his or her own position. These twopersons are asked to do the same thing, which results in an additionaltwo persons who agree and disagree maximally, and so on (hence theterm snowball sample). Advocates should be as different as possible,not only in terms of positions attributed to them but also in terms oftheir relative influence, formal authority, and group affiliation. Thesize of the sample might range from ten to thirty persons, although thisdepends on the nature of the issue. The more complex the issue, andhence the more heterogeneous the participants, the larger the samplemust be to be representative of the range of advocates.

Step 3: Questionnaire design. Because a policy Delphi takes place in aseries of rounds, analysts must decide which specific items will go inquestionnaires to be used in the first and subsequent rounds. Thesecond-round questionnaire can be developed only after the results ofthe first round are analyzed; the third-round questionnaire mustawait results of the second round; and so on. For this reason, onlythe first-round questionnaire can be drafted in advance. Although thefirst-round questionnaires may be relatively unstructured (with manyopen-ended items), they may also be relatively structured, providedthe analyst has a good idea of the major issues. First-roundquestionnaires may include several types of questions: (1) forecastingitems requesting that respondents provide subjective estimates of theprobability of occurrence of particular events, (2) issue itemsrequesting respondents to rank issues in terms of their importance,(3) goal items that solicit judgments about the desirability and/orfeasibility of pursuing certain goals, and (4) options items requestingthat respondents identify alternative courses of action that maycontribute to the attainment of goals and objectives.

Several types of scales are available to measure responses to each ofthese four types of items. One procedure is to use different scales withdifferent types of items. For example, a certainty scale may be usedprimarily with forecast items, an importance scale with issue items,desirability and feasibility scales with goal items, and some combinationof these scales with options items. The best way to show what is involvedis to illustrate the way in which items and scales are presented in a policyDelphi questionnaire. This has been done in Table 9.

Observe that the scales in Table 9 do not permit neutral answers,although “No Judgment” responses are permitted when there is noresponse. This restriction on neutral responses is designed to bring outconflict and disagreement, an important aim of the policy Delphi. Animportant part of the construction of questionnaires is pretestingamong a sample of advocates and determining the reliability ofresponses.

Step 4: Analysis of first-round results. When the questionnaires are returnedafter the first round, analysts attempt to determine the initial positions


TAB

LE

9

Type

s of

Ite

ms

and

Sca

les

Use

d in

a P

olic

y D

elph

i Que

stio

nnai

re

Type

of I

tem

Item

Sca

le

For

ecas

tA

ccor

ding

to

a re

cent

pro

ject

ion

by r

esea

rche

rs a

t th

eN

atio

nal I

nsti

tute

of

Men

tal H

ealt

h,th

e nu

mbe

r of

mar

ijuan

a us

ers

per

1,00

0 pe

rson

s in

the

gen

eral

popu

lati

on w

ill d

oubl

e be

twee

n 19

80 a

nd 1

990.

How

cert

ain

are

you

that

thi

s pr

ojec

tion

is r

elia

ble?

Cer

tain

lyR

elia

ble

1 [ ]

Rel

iabl

e

2 [ ]

Ris

ky 3 [ ]

Unr

elia

ble

4 [ ]

No

Judg

men

t

0 [ ]

Issu

eT

he P

erso

nal u

se o

f m

ariju

ana

shou

ld/s

houl

d

not b

ele

galiz

ed.

Ver

yIm

port

ant

Impo

rtan

tS

light

lyIm

port

ant

Uni

mpo

rtan

tN

o Ju

dgm

ent

How

impo

rtan

t is

thi

s is

sue

rela

tive

to

othe

rs?

12

34

0[

][

][

][

][

]

Goa

lO

ne g

oal o

f na

tion

al p

olic

y m

ight

be

to in

crea

se p

ublic

awar

enes

s of

the

dif

fere

nce

betw

een

drug

use

(res

pons

ible

) an

d ab

use

(irr

espo

nsib

le).

How

des

irab

leis

this

obj

ecti

ve?

Ver

yD

esir

able

1 [ ]

Des

irab

le

2 [ ]

Und

esir

able

3 [ ]

Ver

y U

ndes

irab

le4 [ ]

No

Judg

men

t

0 [ ]

Opt

ions

It h

as b

een

sugg

este

d th

at d

rug

abus

e ed

ucat

ion

prog

ram

s co

ntri

bute

to

the

redu

ctio

n of

pot

enti

al u

sers

in t

he g

ener

al p

opul

atio

n.H

ow fe

asib

leis

thi

s po

licy

opti

on?

Def

init

ely

Fea

sibl

e1 [ ]

Pos

sibl

yF

easi

ble

2 [ ]

Pos

sibl

yU

nfea

sibl

e3 [ ]

Def

init

ely

Unf

easi

ble

4 [ ]

No

Judg

men

t

0 [ ]

Not

e:F

or a

ddit

iona

l inf

orm

atio

n,se

e Ir

ene

Ann

Jill

son,

“The

Nat

iona

l Dru

g A

buse

Pol

icy

Del

phi:

Pro

gres

s R

epor

t an

d F

indi

ngs

to D

ate,

”in

The

Del

phi M

etho

d: T

echn

ique

s an

d

App

licat

ions

,ed

.Har

old

A.L

inst

one

and

Mur

ray

Turo

ff (

New

Yor

k:A

ddis

on-W

esle

y,19

75),

pp.1

24–5

9.


on forecasts, issues, goals, and options. Typically, some items believedto be desirable or important are also believed to be unfeasible, and viceversa. Because there will be conflicting assessments among the variousadvocates, it is important to use summary measures that not onlyexpress the central tendency in the set of responses but also describethe extent of dispersion or polarization. These summary measures notonly eliminate items that are uniformly important, undesirable,unfeasible, and/or uncertain but also serve in the second-roundquestionnaire as a means to communi-cate to participants the results ofthe first round.

The calculation and subsequent presentation of these summarymeasures of central tendency, dispersion, and polarization are bestillustrated graphically. Assume for purposes of illustration that tenadvocates in the first round of a hypothetical policy Delphi provideddifferent assessments of the desirability and feasibility of two drug-control goals: to reduce the supply of illicit drugs and to increasepublic awareness of the difference between responsible andirresponsible drug use. Let us imagine that the responses were thosepresented in Table 10.

Observe that some respondents (advocates 2, 8, and 9) believethat the goal of reducing the supply of illicit drugs is very undesirable


TABLE 10

Hypothetical Responses in First-Round Policy Delphi: Desirability andFeasibility of Drug-Control Objectives

Goal 1 (Reduce Supply) Goal 2 (Public Awareness)

Advocate Desirability Feasibility Desirability Feasibility

1 1 4 1 12 4 1 2 23 3 3 2 14 4 2 1 25 1 4 2 16 2 3 2 17 1 4 1 18 4 2 1 29 4 1 2 2

10 1 4 1 2

Σ = 25 Σ = 28 Σ = 15 Σ = 15Md = 2.5 Md = 3.0 Md = 1.5 Md = 1.5Mn = 2.5 Mn = 2.8 Mn = 1.5 Mn = 1.5

Range = 3.0 Range = 3.0 Range = 1.0 Range = 1.0

Note: The median (Md) in a set of scores is the value of the score that falls in the middle when the scoresare arranged in order of magnitude. If the number of scores is even (as above), the median is the value ofthe score that is halfway between the two middle scores. The median is normally used in place of the mean(Mn) when we do not know if the intervals between measures (e.g., the intervals between 1 and 2 and 3and 4) are equidistant.



but possibly or definitely feasible, whereas others (advocates 1, 5, 7,and 10) believe that this goal is very desirable but definitelyunfeasible. When we compare these inconsistencies betweendesirability and feasibility with responses under goal 2 (publicawareness), we find much less inconsistency in the latter set of scores.All of this suggests that the responses to goal 1, although much lowerin average desirability and feasibility, also reflect the kinds ofimportant conflicts that policy Delphi is specially designed to address.In this case, analysts would not want to eliminate this item. Rather,they would want to report these conflicts as part of the second-roundinstrument, requesting that respondents provide the reasons,assumptions, or arguments that led them to positions that were sodifferent. Another way to bring out such disagreements is to constructand report an average polarization measure, which may be defined asthe absolute difference among scores for all combinations ofrespondents answering a particular question.51

Step 5: Development of subsequent questionnaires. Questionnaires must bedeveloped for second, third, fourth, or fifth rounds (most policyDelphis involve three to five rounds). As indicated previously, theresults of prior rounds are used as a basis for subsequent ones. One ofthe most important aspects of policy Delphi occurs in these rounds,because it is here that advocates have an opportunity to observe theresults of immediately preceding rounds and offer explicit reasons,assumptions, and arguments for their respective judgments. Note thatlater rounds do not simply include information about central tendency,dispersion, and polarization; they also include a summary of argumentsoffered for the most important conflicting judgments. In this way, thepolicy Delphi promotes a reasoned debate and maximizes theprobability that deviant and sometimes insightful judgments are notlost in the process. By the time the last round of questionnaires iscompleted, all advocates have had an opportunity to state their initialpositions on forecasts, issues, goals, and options; to examine andevaluate the reasons why their positions differ from those of others; andto reevaluate and change their positions.

Step 6: Organization of group meetings. One of the last tasks is to bringadvocates together for a face-to-face discussion of the reasons,assumptions, and arguments that underlie their various positions. Thisface-to-face meeting, because it occurs after all advocates have had achance to reflect on their positions and those of others, may create an

51Ibid., p. 92; and Jerry B. Schneider, “The Policy Delphi: A Regional Planning Application,” TechnologicalForecasting and Social Change 3, no. 4 (1972). The total number of combinations (C) is calculated by theformula C = k(k – 1)/2, where k is the number of responses to a particular item. The average difference isobtained by computing the numerical distance between all combinations, adding them up, and dividing by thenumber of respondents. This requires that we ignore the signs (plus or minus). Another procedure is to retainthe signs, square each difference (eliminating minus signs), sum, and average.



atmosphere of informed confidence that would not be possible in atypical committee setting. Face-to-face discussions also create conditionsin which advocates may argue their positions intensely and receiveimmediate feedback.

Step 7: Preparation of final report. There is no guarantee that respondentswill have reached consensus but much reason to hope that creativeideas about issues, goals, options, and their consequences will be themost important product of a policy Delphi. The report of final resultswill therefore include a review of the various issues and optionsavailable, taking care that all conflicting positions and underlyingarguments are presented fully. This report may then be passed on topolicy makers, who may use the results of the policy Delphi as onesource of information in arriving at decisions.

Cross-Impact AnalysisDelphi technique is closely related to another widely used judgmental forecasting tech-nique called cross-impact analysis. Developed by the same RAND Corporationresearchers responsible for early applications of conventional Delphi,52 cross-impactanalysis is a technique that elicits informed judgments about the probability of occur-rence of future events on the basis of the occurrence or nonoccurrence of related events.The aim of cross-impact analysis is to identify events that will facilitate or inhibit theoccurrence of other related events. Cross-impact analysis was expressly designed as asupplement to conventional Delphi. In the words of two of its early developers,

A shortcoming of [Delphi] and many other forecasting methods . . . is thatpotential relationships between the forecasted events may be ignored and theforecasts might well contain mutually reinforcing or mutually exclusive items.[Cross-impact analysis] is an attempt to develop a method by which theprobabilities of an item in a forecasted set can be adjusted in view of judgmentsrelating to the potential interactions of the forecasted items.53

The basic analytical tool used in cross-impact analysis is the cross-impactmatrix, a symmetrical table that lists potentially related events along row and col-umn headings (Table 11). The type of forecasting problem for which cross-impactanalysis is particularly appropriate is one involving a series of interdependentevents. Table 11, for example, expresses the interdependencies among events thatfollow the mass production of automobiles. Observe the string of direct positiveeffects (represented by “+” signs) directly above the blank cells in the main diagonal.These effects are first-, second-, third-, fourth-, fifth-, and sixth-order impacts ofmass automobile production. Note also the positive feedback effects of (E2 – E1),

52These researchers include Olaf Helmer, T. J. Gordon, and H. Hayward. Helmer is credited withcoining the term cross-impact. See T. J. Gordon and H. Hayward, “Initial Experiments with the Cross-Impact Matrix Method of Forecasting,” Futures 1, no. 2 (1968): 101–16.53Ibid., p. 100.



(E3 – E1), (E7 – E4), and (E7 – E5). These positive feedback effects suggest that easeof travel and patronization of large suburban stores may themselves affect the massproduction of automobiles, for example, by increasing demand. Similarly, variousforms of social deviance may intensify existing levels of alienation from neighborsand create even greater social-psychological dependence on family members.

This illustration purposefully oversimplifies linkages among events. In manyother situations, the linkage of one event with another is not unambiguously posi-tive; nor do events follow one another so neatly in time. Moreover, many events maybe negatively linked. For this reason, cross-impact analysis takes into account threeaspects of any linkage:

1. Mode (direction) of linkage. This indicates whether one event affects theoccurrence of another event and, if so, whether the direction of this effectis positive or negative. Positive effects occur in what is called the enhancingmode, whereas negative ones fall into a category called the inhibitingmode. A good example of linkages in the enhancing mode is increasedgasoline prices provoking research and development on synthetic fuels.The arms race and its effects on the availability of funds for urbanredevelopment are an illustration of linkages in the inhibiting mode.The unconnected mode refers to unconnected events.

2. Strength of linkage. This indicates how strongly events are linked,whether in the enhancing or inhibiting mode. Some events are stronglylinked, meaning that the occurrence of one event substantially changes thelikelihood of another’s occurring, whereas other events are weakly linked.

TABLE 11

Cross-Impact Matrix Illustrating Consequences of Mass Automobile Use

Events (E)

E1 E2 E3 E4 E5 E6 E7Events (E) E1 + 0 0 0 0 0

E2 � + 0 0 0 0E3 � 0 + 0 0 0E4 0 0 0 + 0 0E5 0 0 0 0 + 0E6 0 0 0 0 0 +E7 0 0 0 � � 0

E1 � mass production of automobilesE2 � ease of travelE3 � patronization of large suburban storesE4 � alienation from neighborsE5 � high social-psychological dependence on immediate family membersE6 � inability of family members to meet mutual social-psychological demandsE7 � social deviance in form of divorce, alcoholism, juvenile delinquencyNote: A plus (+) indicates direct one-way effects; a zero (0) indicates no effect; a circledplus sign indicates positive feedback effects.

Source: Adapted from Joseph Coates, “Technology Assessment: The Benefits, the Costs,the Consequences,” Futurist 5, no. 6 (December 1971).



In general, the weaker the linkage, the closer it comes to the unconnectedmode.

3. Elapsed time of linkage. This indicates the amount of time (weeks, years,decades) between the occurrence of linked events. Even though events may bestrongly linked, either in the enhancing or inhibiting modes, the impact ofone event on the other may require a considerable period of time. For example,the linkage between the mass production of automobiles and social deviancerequired an elapsed time of several decades.

Cross-impact analysis works on the principle of conditional probability.Conditional probability states that the probability of occurrence of one event isdependent on the occurrence of some other event, that is, the two events are notindependent. Conditional probabilities may be denoted by P(E1/E2), which is read“The probability of the first event (E1), given the second event (E2).” For example,the probability (P) of being elected president (E1) after a candidate has received theparty nomination (E2) may be 0.5; that is, there is a fifty-fifty chance of winning theelection [P(E1/E2) � 0.50]. However, the probability (P) of being elected (E1)without the party’s nomination (E3) is low, because party nomination is almost aprerequisite for the presidency [P(E1/E3) � 0.20].

This same logic is extended to cross-impact analysis. The construction of across-impact matrix begins with the question: “What is the probability that a cer-tain event (E) will occur prior to some specified point in time?” For example, anextrapolative forecast using time-series analysis may provide an interval estimatethat claims that there is a 90 percent (0.9) probability that energy consumptionwill exceed 100.0 quadrillion BTUs by 2020. The next question is, “What is theprobability that this event (E2) will occur, given that another event (E1) is certainto precede it?” For example, if presently unpredictable factors (new agreements,political turmoil, accidents) result in a doubling of oil prices (E1) by 1995, theprobability of energy consumption at the level of 100.0 quadrillion BTUs may bereduced to 0.5. In this case, note that “objective” data used to make the originalextrapolative forecast are combined with a “subjective” judgment about the con-ditional probability of the originally projected event, given the prior occurrenceof the first.

The construction of a cross-impact matrix for any reasonably complexproblem involves many thousands of calculations and requires a computer. Manyapplications of cross-impact analysis in areas of science and technology policy,environmental policy, transportation policy, and energy policy have involved morethan 1,000 separate iterations (called “games” or “plays”) to determine theconsistency of the cross-impact matrix, that is, to make sure that every sequence ofconditional probabilities has been taken into account before a final probability iscalculated for each event. Despite the technical complexity of cross-impact analy-sis, the basic logic of the technique may be readily grasped by considering a simpleillustration.

Suppose that a panel of experts assembled for a conventional Delphi providesestimates of the probability of occurrence of four events (E1 . . . E4) for futureyears. Suppose further that these four events are an increase in the price ofgasoline to $5 per gallon (E1), the “gentrification” of central city neighborhoods



TABLE 12

Hypothetical Illustration of the First Round (Play) in a Cross-Impact Matrix

Then the Changed Probability of Occurrence of These Events Is

If This Event Occurs (P = 1.0) E1 E2 E3 E4

E1 Gas to $5 per gallon 0.7 0.8 0.5

E2 Gentrification 0.4 0.7 0.4

E3 Crime doubles 0.5 0.4 0.1

E4 Electric autos 0.4 0.5 0.7

Events Original Probabilities (P)

E1 P1 � 0.5

E2 P2 � 0.5

E3 P3 � 0.6

E4 P4 � 0.2

by former suburbanites (E2), a doubling of reported crimes per capita (E3), andthe mass production of short-distance battery-powered autos (E4).54 The proba-bilities attached to these four events are, respectively, P1 � 0.5, P2 � 0.5, P3 �0.6, and P4 � 0.2. Given these subjective estimates, the forecasting problem isthis: Given that one of these events occurs (i.e., P � 1.0, or 100%), how will theprobabilities of other events change?55

Table 12 illustrates the first round (play) in the construction of a cross-impact ma-trix. Observe that the assumption that gasoline per gallon will go to $5 produces re-vised subjective probabilities for “gentrification” (an increase from 0.5 to 0.7),reported crimes per capita (an increase from 0.6 to 0.8), and the mass manufacture ofelectric autos (an increase from 0.2 to 0.5). These changes reflect the enhancing link-ages discussed earlier. By contrast, other linkages are inhibiting. For example, thegentrification of central cities makes it less likely that gasoline will increase to $5 pergallon (note the decrease from 0.5 to 0.4), on the assumption that oil companies willbecome more price competitive when former suburbanites drive less. Finally, there arealso unconnected linkages. Increased crime and mass-produced electric autos exert noinfluence on the probability of gas prices increasing to $5 per gallon or on the processof gentrification (note that original probabilities remain constant at 0.5).

54In 2011, the Chevrolet Volt, a short-distance battery-powered car, was named the Motor Trend Car ofthe Year.55Alternatively, we can ask the same question about the nonoccurrence of an event. Therefore, it isusually necessary to construct two matrices: one for occurrences and one for nonoccurrences.



The advantage of the cross-impact matrix is that it enables the analyst to discerninterdependencies that otherwise may have gone unnoticed. Cross-impact analysisalso permits the continuous revision of prior probabilities on the basis of newassumptions or evidence. If new empirical data become available for some of theevents (e.g., crime rates), the matrix may be recalculated. Alternatively, differentassumptions may be introduced—perhaps as a consequence of a policy Delphi thatyields conflicting estimates and arguments—to determine how sensitive certainevents are to changes in other events. Finally, information in a cross-impact matrixmay be readily summarized at any point in the process.

Cross-impact matrices may be used to uncover and analyze those complex inter-dependencies that we have described as ill-structured problems. The techniqueis also consistent with a variety of related approaches to intuitive forecasting,including technology assessment, social impact assessment, and technologicalforecasting.56 As already noted, cross-impact analysis is not only consistent withconventional Delphi but actually represents its adaptation and natural extension.For example, even though cross-impact analysis may be done by single analysts, theaccuracy of subjective judgments may be increased by using Delphi panels.

Cross-impact analysis, like the other forecasting techniques discussed in thischapter, has its limitations. First, the analyst can never be sure that all potentiallyinterdependent events have been included in the analysis, which calls attention to theimportance of problem structuring. There are other techniques to assist in identifyingthese events, including variations of theory mapping (see Table 9) and the construc-tion and graphic presentation of networks of causally related events called relevancetrees. Second, the construction and “playing” of a cross-impact matrix is a reason-ably costly and time-consuming process, even with the advent of packaged computerprograms and high-performance computer technology. Third, there are technical dif-ficulties associated with matrix calculations (e.g., nonoccurrences are not always an-alyzed), although many of these problems have been resolved.57 Finally, and most im-portant, existing applications of cross-impact analysis suffer from one of the sameweaknesses as conventional Delphi, namely, an unrealistic emphasis on consensusamong experts. Most of the forecasting problems for which cross-impact analysis isespecially well suited are precisely the kinds of problems for which conflict, and notconsensus, is widespread. Problem-structuring methods are needed to expose and de-bate the conflicting assumptions and arguments that underlie subjective conditionalprobabilities.

Feasibility AssessmentThe final judgmental forecasting procedure we consider in this chapter is one that isexpressly designed to produce conjectures about the future behavior of policy stake-holders. This procedure, most simply described as the feasibility assessment

56On these related approaches, see François Hetman, Society and the Assessment of Technology (Paris:Organization for Economic Cooperation and Development, 1973); and Finsterbusch and Wolf, eds.,Methodology of Social Impact Assessment.57See Dalby, “Practical Refinements to Cross-Impact Matrix Technique,” pp. 265–73.



technique, assists analysts in producing forecasts about the probable impact ofstakeholders in supporting or opposing the adoption and/or implementation ofdifferent policy alternatives.58 The feasibility assessment technique is particularlywell suited for problems requiring estimates of the probable consequences ofattempting to legitimize policy alternatives under conditions of political conflict andthe unequal distribution of power and other resources.

The feasibility assessment technique may be used to forecast the behavior ofstakeholders in any phase of the policy-making process, including policy adoptionand implementation. What makes this technique particularly useful is that itresponds to a key problem that we have already encountered in reviewing otherintuitive forecasting techniques: Typically, there is no relevant theory or availableempirical data that permit us to make predictions or projections about the behaviorof policy stakeholders. Although social scientists have proposed various theories ofpolicy-making behavior that are potentially available as a source of predictions,most of these theories are insufficiently concrete to apply in specific contexts.59

The feasibility assessment technique is one way to respond to a number of con-cerns about the lack of attention to questions of political feasibility and policyimplementation in policy analysis. Even though problems of policy implementationplay a large part in most policy problems, much of contemporary policy analysispays little attention to this question. What is needed is a systematic way to forecast“the capabilities, interests, and incentives of organizations to implement each alter-native.”60 In practical terms, this means that the behavior of relevant stakeholdersmust be forecasted along with the consequences of policies themselves. Only in thisway can the analyst ensure that organizational and political factors that may be criti-cal to the adoption and implementation of a policy are adequately accounted for.

The feasibility assessment technique, like other intuitive forecasting proce-dures, is based on subjective estimates. The feasibility assessment technique may beused by single analysts, or by a group of knowledgeables, much as in a Delphiexercise. Feasibility assessment focuses on several aspects of political and organiza-tional behavior:

1. Issue position. The analyst estimates the probability that various stakehold-ers will support, oppose, or be indifferent to each of two or more policyalternatives. Positions are coded as supporting (+1), opposing (-1), orindifferent (0). A subjective estimate is then made of the probability that eachstakeholder will adopt the coded position. This estimate (which ranges from 0 to 1.0) indicates the saliency or importance of the issue to each stakeholder.

58The following discussion draws from but modifies Michael K. O’Leary and William D. Coplin,“Teaching Political Strategy Skills with ‘The Prince,’ ” Policy Analysis 2, no. 1 (winter 1976): 145–60.See also O’Leary and Coplin, Everyman’s “Prince.”59These theories deal with elites, groups, coalitions, and aggregates of individuals and leaders. See, forexample, Raymond A. Bauer and Kenneth J. Gergen, eds., The Study of Policy Formation (New York:Free Press, 1968); and Daniel A. Mazmanian and Paul A. Sabatier, Implementation and Public Policy(Lanham, MD: University Press of America, 1989).60Graham T. Allison, “Implementation Analysis: ‘The Missing Chapter’ in Conventional Analysis:A Teaching Exercise,” in Benefit-Cost and Policy Analysis: 1974, ed. Richard Zeckhauser (Chicago:Aldine Publishing Company, 1975), p. 379.



2. Available resources. The analyst provides a subjective estimate of theresources available to each of the stakeholders in pursuing his or her respectiveposition. Available resources include prestige, legitimacy, budget, staff, andaccess to information and communications networks. Because stakeholdersnearly always have positions on other issues for which part of their resourcesare necessary, available resources should be stated as a fraction of totalresources held by the stakeholder. The resource availability scale, expressed asa fraction, varies from 0 to 1.0. Note that there may be a high probability thata given stakeholder will support a policy (e.g., 0.9), yet that same stakeholdermay have little capability to affect the policy’s adoption or implementation.This is typically the result of an overcommitment of resources (prestige,budget, staff) to other issue areas.

3. Relative resource rank. The analyst determines the relative rank of eachstakeholder with respect to his or her resources. Relative resource rank, onemeasure of the “power” or “influence” of stakeholders, provides informa-tion about the magnitude of political and organizational resources availableto each stakeholder. A stakeholder who commits a high fraction (say, 0.8) ofavailable resources to support a policy may nevertheless be unable to affectsignificantly a policy’s adoption or implementation because of inadequateresources.

Because the purpose of feasibility assessment is to forecast behavior underconditions of political conflict, it is essential to identify as representative and powerful agroup of stakeholders as possible. The analyst may identify representative stakeholdersfrom various organizations and organizational levels with different constituencies andwith varying levels of resources and roles in the policy-making process.

An illustration of the feasibility assessment technique appears in Table 13. Inthis example, a policy analyst in a large municipality has completed a study thatshows that local property taxes must be raised by an average of 1 percent in orderto cover expenditures for the next year. Alternatively, expenditures for municipalservices must be cut by a comparable amount, an action that will result in the firingof fifteen hundred employees. Because most public employees are unionized, andthere has been a threat of a strike for more than two years, the mayor is extremelyhesitant to press for a cutback in services. At the same time, local taxpayers’ groupsare known to be strongly opposed to yet another tax increase, even if it means theloss of some services. Under these conditions, the mayor has requested that theanalyst provide a feasibility assessment of the policy alternatives.

Table 13 shows that a tax increase is unfeasible. In fact, the negative sign of theindex at the bottom of the table indicates (not surprisingly) that there is more oppo-sition than support for a tax increase (F � 0.706). By contrast, the index for thebudget cut is positive (F � 0.412).

The feasibility assessment technique forces the analyst to make explicit subjec-tive judgments, rather than treat political and organizational questions in a loose orarbitrary fashion. Feasibility assessment also enables analysts to systematicallyconsider the sensitivity of issue positions and available resources to changes inpolicy alternatives. In the previous example, a smaller tax increase, combined withausterity measures and a plan to increase the productivity of municipal employees,would probably evoke a different level of feasibility.



TABLE 13

Feasibility Assessment of Two Fiscal Policy Alternatives

(a) Alternative 1 (Tax Increase)

Stakeholder (1)

Coded Position

(2)Probability

(3)

Fraction ofResourcesAvailable

(4)

Resource Rank (5)

Feasibility Score

(6)

Mayor +1 0.2 0.2 0.4 0.016Council -1 0.6 0.7 0.8 -0.336Taxpayers’

association-1 0.9 0.8 1.0 -0.720

Employees’ union +1 0.9 0.6 0.6 0.324Mass media +1 0.1 0.5 0.2 0.010

F = -0.706

(b) Alternative 2 (Budget Cut)

Stakeholder (1)

CodedPosition

(2)Probability

(3)

Fraction ofResources

Available (4)ResourceRank (5)

Feasibility Score(6)

Mayor +1 0.8 0.2 0.4 0.064Council +1 0.4 0.5 0.8 0.160Taxpayers’

association+1 0.9 0.7 1.0 0.630

Employees’ union -1 0.9 0.8 0.6 -0.432Mass media -1 0.1 0.5 0.2 -0.010

F � 0.412

The limitations of the feasibility assessment technique are similar to thosealready encountered with other judgmental forecasting techniques. The feasibilityassessment technique, like conventional Delphi and cross-impact analysis, providesno systematic way of surfacing the assumptions and arguments that underlie subjec-tive judgments. Perhaps the best way to resolve this difficulty is to adopt proceduresfrom policy Delphi or use assumption analysis. A second limitation of the techniqueis that it assumes that the positions of stakeholders are independent and that theyoccur at the same point in time. These assumptions are unrealistic, because they ig-nore processes of coalition formation over time and the fact that one stakeholder’sposition is frequently determined by changes in the position of another. To capturethe interdependencies of issue positions as they shift over time, we can employ someadaptation of cross-impact analysis. Finally, the feasibility assessment technique,like other judgmental forecasting techniques discussed in this chapter, is most usefulunder conditions in which the complexity of a problem cannot easily be grasped byusing available theories or empirical data. For this reason, any attempt to outline itslimitations should also consider its potential as a source of creative insight and sur-prising or counterintuitive findings.



In concluding this chapter, it is important to stress that different approaches toforecasting are complementary. Improvements in forecasting are therefore likely toresult from the creative combination of different approaches and techniques, thatis, from multi-method forecasting. Multi-method forecasting combines multipleforms of logical reasoning (inductive, deductive, retroductive), multiple bases(extrapolation, theory, judgment), and multiple objects (the content andconsequences of new and existing policies and the behavior of policy stakehold-ers). Multi-method forecasting recognizes that neither precision nor creativity isan end in itself. What appears to be a creative or insightful conjecture may lackplausibility and turn out to be pure speculation or quackery, whereas highlyprecise projections or predictions may simply answer the wrong question. In thewords of an accomplished policy analyst and former assistant secretary in theDepartment of Defense, “It is better to be roughly right than exactly wrong.”61

61Alain C. Enthoven, “Ten Practical Principles for Policy and Program Analysis,” in Benefit-Cost andPolicy Analysis: 1974, ed. Zeckhauser, p. 459.

CHAPTER SUMMARYThis chapter has provided an overview of theprocess of forecasting, highlighting the nature,types, and uses of forecasting in policy analysis.After comparing approaches based on inductive,deductive, and abductive reasoning, the chapterpresented specific methods and techniques.These included methods and techniques of

extrapolative, theoretical, and judgmental fore-casting. The ultimate justification for a forecastis whether it yields plausibly true beliefs aboutthe future, not whether it is based on a particu-lar type of method, quantitative or qualitative.In this and other respects, it is better to beapproximately right than exactly wrong.

REVIEW QUESTIONS1. What are the three forms of forecasting and

how are they related to bases of forecasts?2. In addition to promoting greater understanding

of the future, what other aims can be achievedthrough forecasting?

3. To what extent are econometric methods moreaccurate than methods of extrapolative andjudgmental forecasting? Explain.

4. How do the institutional, temporal, and histor-ical contexts of forecasts affect their accuracy?

5. Distinguish among potential, plausible, andnormative futures.

6. List the main differences between goals andobjectives. Provide examples.

7. Contrast inductive, deductive, and retroductivereasoning. Do the same for theoretical andjudgmental forecasting.

8. List and describe techniques used in the threemain types of forecasts. Provide examples.

9. Whether they are linear or nonlinear, mostforecasts are based on assumptions of persist-ence and regularity. To what extent are theseassumptions plausible?

10. Many forecasts employ linear regressionanalysis, also known as the classical linearregression (CLR) model. What are the mainassumptions of the CLR model when used tomake forecasts?

11. What corrective actions can be taken when assumptions of the CLR model areviolated?

12. If it is better to be approximately right thanexactly wrong, how does this affect the choiceof a forecasting method?



DEMONSTRATION EXERCISE1. After reading Case 1, answer these questions:

� Explain how the r, r2, and p-values can beidentical in Figure C1(a) through (d).

� Describe the patterns in scatterplot (a)through (d).

� Explain the importance of inspecting visualdisplays when interpreting statistical meas-ures and tests.

2. After reading Case 2, use SPSS or a similarstatistical program (Excel, Stata, or R) toperform a forecast in which you estimate thevalue of Metropolitan Atlanta Rapid TransitAuthority (MARTA) receipts in the year 2010.Assume you are conducting the analysis inJanuary 2000, as a consultant to MARTA. Theclient has hired you because MARTA needs anaccurate estimate of future receipts. Theestimate will be used to make decisions aboutfare increases that have become a focal point ofdebates about environmental justice. Althoughyou could not have known it at the time, theAtlanta City Council opposed a fare increase in2000 and 2001, amidst considerable politicalturmoil. MARTA’s operating budget in 2001was $307 million, which required a significantfare increase. This generated opposition fromcommunity groups, which contended that thefare increase violated principles of environ-mental justice.Perform the following SPSS analyses, which arebased on the procedures for extrapolativeforecasting covered in the chapter:� Enter the data from Table 14 (Payments to

MARTA, 1973–2010) into the SPSS DataEditor. Click on Variable View and completethe rows by entering the Names, Types,Labels, and so on, for the variables YEARand RECEIPTS. Be sure to indicate that thevalue of RECEIPTS is in thousands of dollars

and note that the value of receipts in 1996and 1997 have been interpolated.

� On the pull-down menu, click on Graphs andSequence to create a sequence (time-series)graph. Move RECEIPTS into the Variables:box. You do not need to fill in the Time AxisLabel box. Click OK and inspect and inter-pret the graph. Does the series meet assump-tions of the linear regression model? Click onthe Natural Log Transform button. Click OKand see what difference this makes.

� On the pull-down menu, click on Analyze,Regression, and Linear. Enter the dependent(RECEIPTS) and independent (YEAR) vari-ables. Click on Statistics and on Durbin-Watson. Also click on Save and then onUnstandardized Predicted Values, Residuals,and the 95 percent Confidence Interval forMean. Click OK to run the regression.Calculate by hand the point estimate for2010. Interpret the interval estimate printedon the SPSS output.

� Rerun the regression after using a logarithmictransformation of RECEIPTS (use the menufor Transform and Compute). Rerun theregression. Recalculate. Interpret the intervalestimate. You must take the antilog first.

� Interpret the point and interval estimatesprovided in the output.

3. How accurate is your estimate for 2010? Howabout estimates for the years 2001–09? Whatdo your estimates suggest about the need toraise fares? How sure are you?

4. Write a policy memo to MARTA in which youexplain and justify your forecast for the years2001–10. Include a policy recommendationabout options for increasing fares (includingno increase). Relate your memo to the issue ofenvironmental justice.

BIBLIOGRAPHYAllen, T. Harrell. New Methods in Social Science

Research: Policy Sciences and Futures Research.New York: Frederick A. Praeger, 1978.

Anscombe, F. J. “Graphs in Statistical Analysis.”American Statistician 27 (February 1973): 17–21.

Ascher, William. Forecasting: An Appraisal forPolicy Makers and Planners. Baltimore, MD:Johns Hopkins University Press, 1978.

——— “The Forecasting Potential of ComplexModels.” Policy Sciences 13 (1981): 247–67.

Bartlett, Robert V., ed. Policy through ImpactAssessment: Institutionalized Analysis as a PolicyStrategy. New York: Greenwood Press, 1989.

Box, G. E. P., and G. M. Jenkins. Time SeriesAnalysis: Forecasting and Control. SanFrancisco, CA: Holden-Day, 1969.



Dror, Yehezkel. Policymaking under Adversity.New Brunswick, NJ: Transaction Books, 1986.

Finsterbusch, Kurt, and C. P. Wolf, eds. Methodologyof Social Impact Assessment. Stroudsburg, PA:Dowden, Hutchinson & Ross, 1977.

Gass, Saul I., and Roger L. Sisson, eds. A Guide toModels in Governmental Planning andOperations. Washington, DC: U.S. EnvironmentalProtection Agency, 1974.

Guess, George M., and Paul G. Farnham. Cases inPublic Policy Analysis. New York: Longman, 1989,ch. 3, “Forecasting Policy Options,” pp. 49–67.

Harrison, Daniel P. Social Forecasting Methodology.New York: Russell Sage Foundation, 1976.

Liner, Charles D. “Projecting Local GovernmentRevenue.” In Budget Management: A Reader inLocal Government Financial Management.Edited by W. Bartley Hildreth and GeraldJ. Miller. Athens: University of Georgia Press,1983, pp. 83–92.

Linstone, Harold A., and Murray Turoff, eds. TheDelphi Method: Techniques and Applications.Reading, MA: Addison-Wesley, 1975.

Marien, Michael. Future Survey Annual: A Guideto the Recent Literature of Trends, Forecasts,and Policy Proposals. Bethesda, MD: WorldFuture Society, published annually.

——— “The Scope of Policy Studies: ReclaimingLasswell’s Lost Vision.” In Advances inPolicy Studies since 1950, Vol. 10 of

Policy Studies Review Annual. Edited byWilliam N. Dunn and Rita Mae Kelly.New Brunswick, NJ: Transaction Books,1992, pp. 445–88.

McNown, Robert. “On the Use of EconometricModels: A Guide for Policy Makers.” PolicySciences 19 (1986): 360–80.

O’Leary, Michael K., and William D. Coplin.Everyman’s “Prince.” North Scituate, MA:Duxbury Press, 1976.

Schroeder, Larry D., and Roy Bahl. “The Roleof Multi-year Forecasting in the AnnualBudgeting Process for Local Governments.”Public Budgeting and Finance 4, no. 1 (1984):3–14.

Thomopoulos, Nick T. Applied Forecasting Methods.Englewood Cliffs, NJ: Prentice Hall, 1980.

Toulmin, Llewellyn M., and Glendal E. Wright,“Expenditure Forecasting.” In Handbook onPublic Budgeting and Financial Management.Edited by Jack Rabin and Thomas D. Lynch.New York: Marcel Dekker, 1983, pp. 209–87.

Tufte, Edward R. The Visual Display of Quan-titative Information. Cheshire, CN: TheGraphics Press, 1983.

U.S. General Accounting Office. ProspectiveEvaluation Methods: The Prospective Evalua-tion Synthesis. Washington, DC: U.S. GeneralAccounting Office, Program Evaluation andMethodology Division, July 1989.


The four scatterplots in Figure C1 show differentpatterns among the observations on the graph. Eachset of observations of X and Y yields identical

correlation coefficients (r), coefficients ofdetermination (r2), and p-values. �

CASE 1 CONCEPTUAL ERRORS IN REGRESSIONANALYSIS:THE ANSCOMBE QUARTET

(a)

X

1614121086420

Y

109876543210

(b)

X1614121086420

Y

14

12

10

8

6

4

2

0

r = .82 r2

= .67p = .001

r = .82 r2

= .67p = .001

r = .82 r2

= .67p = .001

r = .82 r2

= .67p = .001

(c )

X

14121086420

Y

12

10

8

6

4

2

0

(d)

X

20181614121086420

Y

14

12

10

8

6

4

2

0

FIGURE C1The Anscombe Quartet


In the case that follows, policy makers responsiblefor the performance of the Metropolitan AtlantaRegional Planning Authority (MARTA) hireconsultants who use forecasting methods to projectfuture revenues generated by sales and use taxes.Even with the reasonably accurate estimates offuture revenues from sales tax receipts and userfees (a reasonably accurate forecast will lie within;10 percent of the observed value), MARTA policy

makers could not avoid making politicallyunpopular and controversial policy decisions thatinvolved raising fares and cutting back services.In the MARTA case, the accuracy of forecasts isdirectly related to issues of environmental justiceraised primarily by relatively poor anddisadvantaged minorities in the Atlanta region.Box 1 provides an overview of many of theseissues. �

CASE 2 REVENUE FORECASTING ANDENVIRONMENTAL JUSTICE

BOX 1 TRANSIT EQUITY: A LOOK AT MARTA*

Why Is TransportationImportant?

Other than housing, Americans spend more on transportation than on any other household expense. The average American householdspends one-fifth of its income on transportation.

What Counties Support MARTA? The Metropolitan Atlanta Rapid Transit Authority (MARTA) servesjust two counties, Fulton and DeKalb, in the ten-county Atlantaregion. In the 1960s, MARTA was hailed as the solution to theregion’s growing traffic and pollution problems. The first referendumto create a five-county rapid rail system failed in 1968. However, in1971, the city of Atlanta, Fulton County, and DeKalb Countyapproved a referendum for a 1 percent sales tax to support a rapidrail and feeder bus system. Cobb County and Gwinnett County votersrejected the MARTA system.

Who Pays for MARTA? MARTA’s operating budget comes from sales tax (46%), fares(34%), and the Federal Transit Administration and other sources(20%). Only Fulton and DeKalb County residents pay for the upkeepand expansion of the system with a one-cent MARTA sales tax.Revenues from bus fares generated $5 million more revenue thantaken in by rail in 1997. In 1999, the regular one-way fare onMARTA was $1.50, up from $1 in 1992.



Who Rides MARTA? A recent rider survey revealed that 78 percent of MARTA’s rail andbus riders are African American and other people of color.Whitesmake up 22 percent of MARTA riders.

Where Do MARTA Riders Live? More than 45 percent of MARTA riders live in the city of Atlanta,14 percent live in the remainder of Fulton County, 25 percent live inDeKalb County, and 16 percent of MARTA riders live outsideMARTA’s service area.

Where Are Weekday MARTARiders Headed?

The majority (58%) of MARTA’s weekday riders are on their way towork. The second highest use of MARTA was for getting to medicalcenters and other services (21%). Other MARTA riders use the systemfor attending special events (8%), shopping (7%), and school.

How Much Is MARTA’sProposed Fare Increase?

MARTA proposed raising one-way fares from $1.50 to $1.75,a 17 percent increase. The increase is proposed to offset a $10 millionshortfall associated with the openings of the Sandy Springs and NorthSprings stations. The proposal also calls for increasing the weekly transitpass from $12 to $13 and the monthly pass from $45 to $52.50.

Who Would Be Most Impactedby the Proposed MARTA FareIncrease?

Although the increase of $7.50 a month may not seem like a lot at firstglance, it could do irreparable harm to a $5.25 per hour minimum-wagetransit user. These fare increases would fall heaviest on the transit-dependent, low-income households and people of color who make up thegreatest percentage of MARTA users.

How Can the Public Comment onthe Proposed MARTA FareIncrease?

Because MARTA receives federal transportation dollars, it isrequired to hold public hearings before any fare increase takes effect.

How Has MARTA Grown? MARTA has grown from thirteen rail stations in 1979 to thirty-six railstations in 2000. Two additional stations (Sandy Springs and NorthSprings) along the North Line were under construction. These two newnorthern stations opened in early 2001.With its $270.4 million annualbudget, MARTA operates 700 buses and 240 rail cars. The systemhandles more than 534,000 passengers on an average weekday. MARTAoperates 154 bus routes that cover 1,531 miles and carry 275,000passengers on an average weekday. MARTA’s rail lines cover 46 mileswith rail cars carrying 259,000 passengers on an average weekday.

Who Uses MARTA’s ParkingSpaces?

MARTA provides nearly 21,000 parking spaces at twenty-three of itsthirty-six transit stations. Parking at MARTA lots is free except forthe overnight lots that cost $3 per day. MARTA provides 1,342 spacesin four overnight lots. All of the overnight lots are on MARTA’sNorth Line. It is becoming increasingly difficult to find a parkingspace in some MARTA lots. A recent license tag survey, “Who Parks-and-Rides,” covering the period 1988–97, revealed that 44percent of the cars parked at MARTA lots were from outside theMARTA Fulton/DeKalb County service area.


TABLE 14

MARTA Receipts, 1973–2010

Year Receipts1973 $43,820

1974 $50,5011975 $50,9461976 $52,8191977 $57,9331978 $66,1201979 $75,4721980 $88,3421981 $99,8361982 $104,6851983 $112,0081984 $123,4071985 $134,9021986 $147,1491987 $148,5821988 $158,5491989 $162,5431990 $165,7221991 $168,085

Following are statistics on MARTA receipts from sales tax and user fees from 1973 to 2010.


What Are the SimilaritiesBetween Atlanta and LosAngeles?

A similar transit proposal in Los Angles sparked a grassroots movement.In 1996, the Labor Community Strategy Center and the Bus RidersUnion (a grassroots group of transit users) sued the Los Angeles MTAover its plan to raise bus fares and build an expensive rail system at theexpense of bus riders, who made up 95 percent of transit users. TheMTA bus system, used largely by low-income persons and people ofcolor, received only 30 percent of the MTA’s transit dollars. Grassrootsorganizing and the Bus Riders Union’s legal victory resulted in $1.5billion for new clean-fuel buses, service improvements, lower fares, alandmark Civil Rights Consent Decree, and a vibrant multiracialgrassroots organization of more than 2,000 dues-paying members.

Where Can I Get MoreInformation on TransportationEquity?

Contact the Environmental Justice Resource Center at Clark AtlantaUniversity, 223 James P. Brawley Drive, Atlanta, GA 30314 (404)880–6911 (ph), (404) 880–6909 (fx), E-mail: [email protected]. Website:http://www.ejrc.cau.edu.�

*Prepared by the Environmental Justice Resource Center at Clark Atlanta University under its Atlanta TransportationEquity Project (ATEP).The ATEP is made possible by grants from the Turner Foundation and the Ford Foundation.See http://www.ejrc.cau.edu.



Year Receipts

1992 $167,0161993 $181,3451994 $198,4901995 $222,4751996 $229,5101997 $236,5451998 $243,5791999 $277,3092000 $299,1032001 $303,5622002 $284,4272003 $271,0062004 $283,3812005 $307,2272006 $334,4862007 $350,5262008 $349,6682009 $312,7042010 $307,525

Source: Guess and Farnham (2000), p. 196; and 2010 Comprehensive

Annual Financial Report, Metropolitan Atlanta Regional Planning Authority, 2010. Receipts for 1996 and 1997 are linear interpolations.



Prescribing PreferredPolicies



O B J E C T I V E S


Prescribing PreferredPolicies

� Distinguish policy prescriptionfrom other policy-analytic methods.

� Describe criteria employed to makereasoned choices among two ormore alternatives.

� Contrast the Pareto criterion, theKaldor–Hicks criterion, and theRawls criterion.

� Compare and contrast comprehensive rationalityand disjointed incrementalismas models of choice.

What should be done? Answers to this question call for the policy-analyticmethod of prescription. Although the method of forecasting assists inunderstanding expected policy outcomes, thereby enlarging capacities for

understanding and policy guidance, it does not provide explicit reasons for choosingone expected outcome over another. Hence, forecasting answers questions aboutwhat is likely to occur in future. But it does not permit answers to questions aboutthe benefits and costs of future courses of action.

This chapter begins with an overview of the policy-analytic method of prescrip-tion, focusing on policy choice as a fundamental problem of policy analysis. Centralto policy choice is the consideration of several bases for making rational choices,

� Describe the types of reasons under-lying different types of rational choice.

� Identify steps in conducting benefit-cost and cost-effectiveness analyses.

� Discuss the main sources of uncertainty in prescribing preferredpolicies.

� List and describe threats to theplausibility of policy prescriptions.

� Use case studies to perform acritical analysis of assumptions ofbenefit-cost analysis.


Prescribing Preferred Policies

especially choices based on normative economic reasoning and the analysis of theopportunity costs incurred when one policy is chosen over another. We then examineseveral approaches to prescription, principal among which are cost-benefit and cost-effectiveness analysis, along with a number of techniques associated with theseapproaches.

PRESCRIPTION IN POLICY ANALYSISThe method of prescription enables the transformation of information about ex-pected policy outcomes into information about preferred policies. To prescribe apreferred policy requires prior information about the expected outcomes of policies.However, making prescriptions requires that we also have information about whichoutcomes are more valuable than others. For this reason, the policy-analytic proce-dure of prescription is inherently normative. It is closely related to ethical and moralquestions.1

Prescription and Policy AdvocacyShould the United States increase its economic commitment to less developed coun-tries by raising levels of foreign aid and technical assistance? Should Congress passlegislation that will strictly curtail the pollution of the atmosphere and waterwaysby industry and drivers of vehicles used for work and recreation? Should stategovernments provide low-cost home-heating fuel to those who cannot afford it?Should the city council raise local taxes to build a community recreation center?Should the federal government provide a minimum annual income for all citizens orinvest in a cure for cancer?

These policy issues call for prescriptions that answer the question: What shouldbe done? As we have seen, answers to this question call for an approach that isnormative, rather than one that is merely empirical, because the question is one ofright action. Questions of right action demand choices among multiple advocativeclaims.2 Advocative claims have several characteristics:

1. Actionable. Advocative claims focus on actions that may be taken to resolvea policy problem. Although advocative claims require prior information aboutwhat will occur and what is valuable, they go beyond questions of “fact” and“value” because they include arguments about what should be done.

2. Prospective. Advocative claims are prospective, because they occur prior tothe time that actions are taken (ex ante). Whereas policy-analytic proceduresof monitoring and evaluation are retrospective, because they are applied after

1See Deborah Stone, “Policy Paradox,” Part I: Goals, pp. 35–130; William N. Dunn, “Values, Ethics,and Standards in Policy Analysis,” pp. 831–66 in Encyclopedia of Policy Studies, ed. Stuart S. Nagel(New York: Marcel Dekker, 1983).2Here, we are describing rational advocacy, not policy advocacy. “Policy advocacy” is often contrastedwith “rationality,” on the basis of the questionable assumption that advocacy cannot be rational. Onmultiple advocacy, see Alexander George, Presidential Decision Making in Foreign Policy: The EffectiveUse of Information and Advice (Boulder, CO: Westview Press, 1980).



actions are taken (ex post), forecasting and prescription are both appliedprospectively (ex ante).

3. Value laden. Advocative claims depend as much on facts as they do onvalues. To claim that a particular policy alternative should be adopted requiresnot only that the action being prescribed will have the predicted consequences,it also requires that the predicted consequences are valued by individuals andgroups and society as a whole.

4. Ethically complex. The values underlying advocative claims are ethicallycomplex. A given value may be intrinsic as well as extrinsic. Intrinsic valuesare valued as ends in themselves; extrinsic values are valued because theyproduce some other value. For example, health may be regarded as an endin itself and as a means for the attainment of other values, including security,freedom, self-actualization, and economic efficiency.

Multiple advocacy should be contrasted with the simplistic view of policyadvocacy, in which the aim of analysis is to support a predetermined policyposition by mustering information on behalf of that position. Multiple advocacycalls for the systematic comparison and critical assessment of multiple potentialsolutions; it is not a way to defend a single position at any cost. To be sure,analysts may arrive at a single prescription—but only after critically assessing thepros and cons of multiple prescriptions. Multiple advocacy is as much an ap-proach to problem structuring as it is to problem solving.3 When policy analystsfollow guidelines of multiple advocacy, they are less likely to fall into an over-ad-vocacy trap (Box 1), a trap that yields faulty solutions because analysts have de-fined the wrong problem.

A Simple Model of ChoiceAdvocative claims are called for when analysts face a choice between two or morealternatives. In some situations, the choice is between a new course of action and thestatus quo. In other situations, there may be several new alternatives.

In its simplest terms, choice is a process of reasoning involving three relatedcomponents: (1) the definition of a problem requiring action, (2) the comparison ofconsequences of two or more alternatives to resolve the problem, and (3) the prescrip-tion of the alternative that will result in a preferred outcome. For example, choice maybe described as a process of reasoning whereby the first alternative (A1) yields oneoutcome (O1), the second alternative (A2) yields another outcome (O2), and thevalue of O1 is greater than O2 (O1 � O2). Having this information, the analyst willprescribe A1 as the preferred alternative. The analyst reasons as follows: The firstalternative leads to one result, whereas the second leads to another. The first result is

3Ian I. Mitroff, Richard O. Mason, and Vincent P. Barabba, The 1980 Census: Policymaking amidTurbulence (Lexington, MA: D.C. Heath, 1983) see multiple advocacy as a methodology of problemstructuring with features similar to their dialectical approach to planning analysis.


more valuable than the second. Therefore A1, the first alternative, should be chosen(Figure 1).

This simple process of reasoning contains two essential elements of choice:factual and value premises. The first decision premise states that A1 will result in O1,whereas the second decision premise states that A2 will result in O2. These are factualpremises, that is, assumptions that may be shown to be true or false on the basisof factual knowledge. The third premise, however, is a value premise, that is, a prem-ise stating that outcome O1 is better the O2 on some scale of values. Such premisescannot be proved right or wrong by appealing to factual premises, because questionsof value require reasoned arguments about why the outcome in question is goodor right. All choices contain both factual and value premises.

This simple model has the advantage of pointing out that factual and valuepremises are present in all choice situations. The disadvantage of this simple modelis that it obscures the complexity of choice. Consider, for example, the conditionsthat must be present for this model of choice to be a valid one.4

4See, for example, Richard Zeckhauser and Elmer Schaefer, “Public Policy and Normative EconomicTheory,” in The Study of Policy Formation, ed. Raymond A. Bauer and Kenneth J. Gergen (New York:Free Press, 1968), p. 28.

A2 O2

O1 > O2

A1

∴ A1

O1

BOX 1

The Over-Advocacy TrapIn Presidential Decision Making and Foreign Policy: The

Effective Use of Information and Advice (1980),Alexander George has listed many shortcomingsand weaknesses of policy advice that he describesas the over-advocacy trap.* The over-advocacy trapoccurs when

• The client and the analyst agree too readily onthe nature of the problem and responses to it.

• Disagreements among policy advocatesincorporated in an analysis do not cover thefull range of policy alternatives.

• The analyst ignores advocates of unpopularpolicy alternatives.

• The analyst fails to communicate ideasrequiring that the client face up to a difficultor unpopular decision.

• The analyst is dependent on a single source ofinformation.

• The client is dependent on a single analyst.• Assumptions of a policy are evaluated only by

advocates of that policy.• The client dismisses the results of analysis

simply because they are perceived as negativeor counterintuitive.

• The client or the analyst uncritically acceptsconsensus findings without probing the basisfor the consensus and how it was achieved. �

*Alexander George, Presidential Decision Making in Foreign Policy

(Boulder, CO:Westview Press, 1980), pp. 23–24. I havechanged some of George’s language to show the applicabilityof his concepts to policy analysis (e.g., the term adviser hasbeen replaced with analyst).

FIGURE 1Simple model of choice.




1. Single decision maker. The choice must be confined to a single person.If choices are made by more than one person, conflicting factual and valuepremises are likely.

2. Certainty. The outcomes of choice must be known with certainty. Yet theseoutcomes are seldom if ever known with certainty. A part from the choice ofthe preferred alternative, many uncontrollable factors enhance or inhibit theoccurrence of an outcome.

3. Immediacy of consequences. Results of a course of action must occurimmediately. In most choice situations, however, outcomes of action are delayed. Because outcomes do not occur immediately, the values thatoriginally prompted the action may change over time.

Imagine now that our simple model of choice involves the issue of whether to pro-vide minimum wages to unskilled workers. Assume that part-time unskilled workersare now covered by a minimum wage of $7.25 per hour. On the basis of an analysis, weconclude that if we maintain the status quo ($7.25 per hour), the result will be anannual average part-time income of $4,000 for unskilled workers. We then forecastthat a minimum wage of $8.50 per hour will result in an annual average part-timeincome of $7,000 for a smaller group of workers. If we assume that more income isalways better than less income (a moment’s reflection should lead us to question thisassumption), we will have no difficulty in choosing as follows:

However, this case fails to satisfy the three conditions necessary for the simplemodel of choice. First, there are multiple decision makers, not simply one. Manystakeholders—for example, legislators, voters, administrators, employers—affect andare affected by the issue of minimum wages. Each of these stakeholders brings differentfactual and value premises to the choice situation, so there are likely to be disagreementsabout what should be done and why. For example, legislators may want to increase theincome of unskilled workers whereas constituents may want to reduce the tax burdenthat will accompany the enactment and implementation of minimum wage legislation.Second, the consequences of minimum wage legislation are uncertain. Many factorsother than the legislation—for example, the availability of college students as an alter-native source of unskilled labor—may determine whether or not employers will actuallypay the minimum wage, or fire some workers. Finally, the consequences of the legisla-tion will occur over time, which means that values may also change. If there is a deepeconomic recession such as that between 2008 and 2011, groups that formerlysupported an $8.50 per hour wage may conclude that this wage is likely to increaseunemployment. In short, the simple model of choice provides a distorted picture of theissues involved in setting a minimum wage for unskilled workers.

A Complex Model of ChoiceSuppose that a more thorough effort were made to gather relevant informationabout the policy issue of minimum wages. In addition to the original alternatives(minimum wage and the status quo), a third alternative is identified, for example,

∴ A2

O2 7 O1

A2 : O21$7,0002A1 : O11$4,0002


job training. This third alternative is based on the assumption that the problem isnot low wages, per se, but an absence of skills necessary to qualify workersfor higher-paying jobs. After adding this third alternative, we might forecast thefollowing consequences: The first alternative (A1 or $7.25 per hour) will produce by2011 an average annual income of $4,000 for some 12,000 unskilled workers; thesecond alternative (A2 or a $8.50 per hour) will produce by 2011 an average annualincome of $7,000 for 6,000 unskilled workers, with the remaining workers enteringthe ranks of the unemployed; the third alternative (A3 or job training) will produceby 2011 an average annual income of $4,000 for some 10,000 formerly unskilledworkers who will find jobs.

Each alternative will have different consequences. For example, legislators mustbe attentive to their prospects for reelection. A minimum wage of $7.25 will resultin the loss of seats in districts where labor and welfare rights organizations arepowerful. The costs of the program will be passed on to owners of small businesses,who have little influence in the legislature. Finally, training will result in a loss ofseats in districts with strong opposition to any new tax increase. In any case, thebenefits of training are not immediate and will not be felt for several years. The realearning capacity of trainees will increase gradually, reaching a level of $5,000 for12,000 workers by 2013. The real incomes of workers receiving the $7.50 hourlyminimum wage will decline as a result of inflation. Finally, the three alternativeshave different costs that will be borne unequally by various groups. The first alterna-tive involves no new costs, but the second will require that owners of smallbusinesses pay the additional costs. The third alternative will distribute costs amongtaxpayers who will foot the bill.

The simple model of choice has quickly become complicated as a result ofmultiple stakeholders, uncertainty about outcomes, and time. If we diagram thechoice situation in its new and complex form (Table 1), we find that the choice ismore difficult to make. If we are concerned only with immediate monetary results in2011, A1 is the preferred alternative, because O1 is greater than O4 and O7. But if

TABLE 1

Complex Model of Choice

A1 O1 O2 O3

($7.25) $48,000 income(2011)

$48,000 income (2013)

Retain 150 seats (2014)

A2 O4 O5 O6

($8.50) $42,000 income(2011)

$36,000 income (2013)


A3 O7 O8 O9

(Job Training) $40,000 income(2011)

$60,000 income (2013)


O1 7 O4 7 O7; ‹ A1

O8 7 O2 7 O5; ‹ A3

O6 7 O9 7 O3; ‹ A2




we consider future monetary results in 2013 and 2014, A3 is the preferred alterna-tive, because O8 is greater than O2, which in turn is greater than O5. Finally, if weare concerned with political consequences, A2 is the preferred alternative, becauseO6 is greater (i.e., produces more retained seats) than A9 and A3.

The problem with this complex choice is that we cannot arrive at a satisfactoryprescription that combines the values of all stakeholders at several points in time.A situation such as this, in which it is impossible to consistently rank alternatives,is termed intransitive. Intransitive choices typically involve multiple conflictingobjectives and should be contrasted with situations involving transitive choices.A transitive choice is one for which alternatives can be consistently ranked accord-ing to one or more attributes: If A1 is preferable to A2 in the choice set (A1, A2) andA2 is preferable to A3 in the choice set (A2, A3), then A1 is preferable to A3 inthe set (A1, A3). Transitive choices can be constructed by assigning values to eachalternative, so that if A1 is preferred to A2 and A3, it is assigned the highest value.The person making a choice is said to maximize utility by selecting the alternativethat yields the greatest value. Transitive and intransitive choice situations areillustrated in Table 2.

Forms of RationalityA rational choice is sometimes defined as a choice that maximizes individual utility.One problem with this view, as defined by rational choice theorists, is that it is notclearly grounded in a testable theory.5 A testable theory is one with hypotheses thatcan be confirmed or disconfirmed empirically. Rational choice theory tends to takechoice out of the picture because people are assumed to maximize their utility

5A good overview of the pros and cons of rational choice theory is Jeffrey Friedman, ed., The RationalChoice Controversy: Economic Models of Politics Reconsidered (New Haven, CT: Yale UniversityPress, 1996).

TABLE 2

Transitive and Intransitive Choices

(a) Transitive Outcome

Alternative 2011 Income 2013 Income 2014 Elections

A1 1st 1st 1stA2 2nd 2nd 2ndA3 3rd 3rd 3rd

(b) Intransitive Outcome

Alternative 2011 Income 2013 Income 2014 Elections

A1 1st 2nd 3rdA2 2nd 3rd 1stA3 3rd 1st 2nd


in accordance with utility maximization as the basic axiom of the model. There islittle or no room for choice, deliberation, or even error. Persons are said to berational if they maximize their utility. Why do they maximize their utility? Becausethey are rational. This is an unrealistic model of policy choice.

Another theory is that of reasoned choice.6 Analysts arrive at policy prescrip-tions because they have reasons for making choices. Rather than assuming peoplemake choices that maximize their utility, it is useful to think of choices based onreasons, that is, reasoned choices. Reasoned choices can be divided into five types:economic, technical, political, legal, and substantive. Each type is associated withidentifiable modes of reasoning.

In this context, any situation of choice can yield a course of action that ispreferred to all others because it will result in desired outcomes. Yet most choicesituations involve multiple stakeholders, uncertainty, and consequences that changeover time. In fact, conflict and disagreement are essential characteristics of mostpolicy issues:

Difficult choice problems in which attribute rankings conflict lie at the core ofthe decisions that must be made by public policy makers. These difficulties mayarise because their decisions affect many individuals. Though policy A might bebetter for one group in our society, policy B might be better for another. If timeis a crucial element we may find, for example, that policy B will be bettertwenty years from today. In a third context, policy A might be superior if someuncertain events turn out favorably but policy B might be a better hedge againstdisaster.7

For these and similar reasons, it may appear that the process of makingprescriptions is not and cannot be “rational.” Tempting as this conclusion might be,our inability to satisfy the conditions of the simple model of choice does not meanthat the process is not and cannot be rational. If by rationality we mean a reflectiveprocess of reasoning, most choices are based on multiple reasons. This means thatsimultaneous processes of reasoning underlie policy choices:8

1. Technical rationality. Technical rationality refers to reasoned choices basedon a comparison of alternatives according to their effectiveness. Choices basedon a comparison of the number of BTUs of energy produced by solar andnuclear energy technologies are an example of technical rationality.

2. Economic rationality. Economic rationality refers to reasoned choices basedon a comparison of alternatives according to their efficiency. Choices amongalternative health care systems in terms of their total costs and total benefitsexemplify economic rationality.

3. Legal rationality. Legal rationality refers to reasoned choices involving acomparison of alternatives according to their conformity to law. Choicesinvolving the award of public contracts according to whether or not companies

7Richard Zeckhauser and Elmer Schaefer, “Public Policy and Normative Economic Theory,” p. 30.8See Paul Diesing, Reason and Society (Urbana: University of Illinois Press, 1962).

6See Stephen Toulmin, Return to Reason (Cambridge, MA: Harvard University Press, 2001).




comply with laws against racial discrimination are an example of legalrationality.

4. Social rationality. Social rationality is a characteristic of reasoned choicesbased on a comparison of alternatives according to their maintenance ofvalued social institutions, that is, they promote institutionalization. Choicesinvolving the extension of rights to democratic participation at work are anexample of social rationality.

5. Substantive rationality. Substantive rationality is a characteristic of reasonedchoices that involves the comparison of multiple forms of rationality—techni-cal, economic, legal, social—in order to make the most appropriate choiceunder given circumstances. Many issues of government information policyinvolve questions about the usefulness of new computer technologies, theircosts and benefits to society, their legal implications for rights to privacy, andtheir consistency with democratic institutions. Debates about these issues maybe characterized in terms of substantive rationality.

Criteria for Policy PrescriptionThe several types of rationality may be viewed in terms of specific decision criteriafor making prescriptions. By decision criteria, we mean explicitly stated rules,principles, or standards used to justify policy choices. Decision criteria are ofsix main types: effectiveness, efficiency, adequacy, equity, responsiveness, andappropriateness.9

Effectiveness. Effectiveness refers to the principle that an alternative should promotethe achievement of a valued outcome of action. Effectiveness, which is synonymouswith technical rationality, is often measured in terms of units of products or services ortheir monetary value. If nuclear generators produce more energy than solar collectiondevices, the former are regarded as more effective, because nuclear generators producemore of the valued outcome. An effective health policy is one that provides more qual-ity health care to more people.

Efficiency. Efficiency refers to the amount of effort required to produce a givenlevel of effectiveness. Efficiency, which is synonymous with economic rationality, isthe relationship between effectiveness and effort, with the latter often measured interms of monetary benefits at a given level of costs. Efficiency may be determinedby calculating the costs per unit of product or service (e.g., dollars per gallon ofirrigation water or dollars per medical examination), or by calculating the totalmonetary benefits per gallon of irrigation water less the costs per gallon of thiswater. Policies that achieve the larger net benefits per unit cost are said to beefficient. One way to determine efficiency is to compare the opportunity costs of analternative against its rival.

9On decision criteria, see Theodore H. Poister, Public Program Analysis: Applied Research Methods.(Baltimore, MD: University Park Press, 1978), pp. 9–15.


TABLE 3

Criteria of Adequacy: Four Types of Problems

Costs

Effectiveness Equal Variable

Equal Type IV Type II

(equal-cost–equal-effectiveness)

(equal-effectiveness)

Variable Type I Type III(equal-cost) (variable-cost–variable-

effectiveness)

Adequacy. Adequacy refers to the extent to which any given level of effectivenesssatisfies a particular standard. The criterion of adequacy specifies expectationsabout the strength of a relationship between policy alternatives and a fixed orvariable value of a desired outcome. Criteria of adequacy and their relation toeffectiveness and efficiency are illustrated by the four types of problems displayedin Table 3 and discussed next:.

1. Type I problems. Problems of this type involve equal costs and variableeffectiveness. When maximum allowable budgetary expenditures result infixed costs, the aim is to maximize effectiveness within the limits of availableresources. For example, given a fixed budget of $1 million for each of twoprograms, a health policy analyst will prescribe the alternative that results inthe greater improvement in the quality of health care in the community. Theresponse to type I problems is called equal-cost analysis, because analystscompare alternatives that vary in effectiveness but whose costs are treated asequal. Here the most adequate policy is one that maximizes the attainment ofobjectives while remaining within the limits of fixed costs.

2. Type II problems. Problems of this type involve equal effectiveness and vari-able costs. When the level of valued outcomes is fixed, the aim is to minimizecosts. For example, if public transportation facilities must serve at least100,000 persons annually, the problem is to identify those alternatives—bus,monorail, subways—that will achieve this fixed level of effectiveness at leastcost. The response to type II problems is called equal-effectiveness analysis,because analysts compare alternatives that vary in costs but whose effective-ness is equal. Here the most adequate policy is one that minimizes costs whileachieving fixed levels of effectiveness.

3. Type III problems. Problems of this type involve variable costs and variableeffectiveness. For example, the choice of an optimal budget to maximize theattainment of agency objectives is a type III problem. The response to type IIIproblems is called variable-cost–variable-effectiveness analysis, because costsand effectiveness are free to vary. Here the most adequate policy is one thatmaximizes the ratio of effectiveness to costs.




4. Type IV problems. Problems of this type involve EQUAL costs as well as equal effectiveness. Type IV problems, which involve equal-cost–equal-effectiveness analysis, are especially difficult to solve. Analysts are not only limited by the requirement that costs not exceed a certain level but are also limited by the constraint that alternatives satisfy a predeterminedlevel of effectiveness. For example, if public transportation facilities mustserve a minimum of 100,000 persons annually—yet costs have been fixed at an unrealistic level—then any policy alternative must either satisfy bothconstraints or be rejected. In such circumstances, the only remainingalternative may be to do nothing.

The different definitions of adequacy contained in these four types of problemspoint to the complexity of relationships between costs and effectiveness. For exam-ple, two programs designed to provide municipal services (measured in units ofservice to citizens) may differ significantly in terms of both effectiveness and costs(Figure 2). Program I achieves a higher overall level of effectiveness than program II,but program II is less costly at lower levels of effectiveness. Should the analyst

4

6

8

10

12

0

2

10 20 30 40 50 600

Eff

ectiv

enes

s (U

nits

of S

ervi

ce in

Tho

usan

ds)

Costs ($ Thousands)

Program III

Program I

Program II

Equal-Cost(Type I Problem)

Equal-Effectiveness(Type II Problem)

Equal-Cost-Equal-Effectiveness(Type IV Problem)

Variable-Cost-Variable-Effectiveness(Type III Problem)

E1

C1 C2 C3 C4

E2

E3

FIGURE 2Cost-effectiveness comparisons using four criteria of adequacy.Source: Adapted from E. S. Quade, Analysis for Public Decisions (New York: American Elsevier,1975), p. 93.


prescribe the program that maximizes effectiveness (program I) or the program thatminimizes costs at the same level of effectiveness (program II)?

To answer this question, we must look at the relation between costs andeffectiveness, rather than view costs and effectiveness separately. Yet this is wherethe complications begin (see Figure 2): (1) If we are dealing with a type I (equal-cost) problem and costs are fixed at $20,000 (C2), program II is preferablebecause it achieves the higher level of effectiveness while remaining within thefixed-cost limitation. (2) If we are confronted with a type II (equal-effectiveness)problem and effectiveness is fixed at 6,000 units of service (E2), program I ispreferable. (3) If, on the other hand, we are dealing with a type III (variable-cost–variable-effectiveness) problem, for which costs and effectiveness are free tovary, program II is preferable, because the ratio of effectiveness to costs (called aneffectiveness–cost ratio) is greatest at the intersection of E1 and C1. Here programII produces 4,000 units of service for $10,000, that is, a ratio of 4,000 to 10,000or 0.4. By contrast, program I has an effectiveness–cost ratio of 0.32 (8,000 unitsof service for $25,000 � 8,000/25,000 � 0.32).10 Finally, (4) if we are dealingwith a type IV (equal-cost–equal-effectiveness) problem, for which both effective-ness and costs are fixed at E2 and C2, neither program is adequate. This dilemma,which permits no adequate solution, is known as criterion overspecification.

The lesson of this illustration is that it is seldom possible to choose between twoalternatives on the basis of either costs or effectiveness. Although it is sometimespossible to convert measures of effectiveness into dollar benefits, which permitsus to calculate net income benefits by subtracting monetary costs from monetarybenefits, it is frequently difficult to establish convincing dollar equivalents forimportant policy outcomes. What is the dollar equivalent of a life saved throughtraffic safety programs? What is the dollar value of international peace and securitypromoted by United Nations educational, scientific, and cultural activities? What isthe dollar value of natural beauty preserved through environmental protectionlegislation? Such questions will be examined further when we discuss cost-benefitanalysis; but it is important here to recognize that the measurement of effectivenessin dollar terms is a complex and difficult problem.11

Sometimes it is possible to identify an alternative that simultaneously satisfiesall criteria of adequacy. For example, the broken-line curve in Figure 2 (ProgramIII) adequately meets fixed-cost as well as fixed-effectiveness criteria and also hasthe highest ratio of effectiveness to costs. As this situation is rare, it is almost al-ways necessary to specify the level of effectiveness and costs that is regarded asadequate.

10A simple way to display effectiveness-cost ratios graphically is to draw a straight line from theorigin (i.e., lower left-hand corner) of a graph to the knee of the effectiveness-cost curve (i.e., thepoint of tangency at which the straight line touches the curve but does not pass through it). The morethat the line moves leftward toward the vertical effectiveness scale, the greater is the ratio betweeneffectiveness and costs.11See, for example, Richard Zeckhauser, “Procedures for Valuing Lives,” Public Policy (fall 1975):419–64; and Baruch Fischhoff, “Cost-Benefit Analysis and the Art of Motorcycle Maintenance,” PolicySciences 8 (1977): 177–202.




Questions of adequacy cannot be resolved by arbitrarily adopting a single criterion.For example, net income benefits (dollars of effectiveness minus dollar costs) are notan appropriate criterion when costs are fixed and a single program with the highestbenefit-cost ratio can be repeated many times. This is illustrated in Table 4, in whichprogram I can be repeated 10 times up to a fixed-cost limit of $40,000, with totalnet benefits of $360,000 (i.e., $36,000 � 10). Program I therefore has the highestbenefit-cost ratio. But if program I cannot be repeated—that is, if only one of thethree programs must be selected—program III is preferable because it yields thegreater net benefits, even though it has the lowest benefit-cost ratio.

Equity. The criterion of equity is closely related to legal and social rationalityand refers to the distribution of effects and effort among different groups in soci-ety. An equitable policy is one for which effects (e.g., units of service or monetarybenefits) or efforts (e.g., monetary costs) are fairly or justly distributed. Policiesdesigned to redistribute income, educational opportunity, or public services aresometimes prescribed on the basis of the criterion of equity. A given programmight be effective, efficient, and adequate—for example, the benefit-cost ratio andnet benefits may be superior to those of all other programs—yet it might still berejected on grounds that it will produce an inequitable distribution of costs andbenefits. This could happen under several conditions: Those most in need do notreceive services in proportion to their numbers, those who are least able to paybear a disproportionate share of costs, or those who receive most of the benefitsdo not pay the costs.

The criterion of equity is closely related to competing conceptions of justice or fair-ness and to ethical conflicts surrounding the appropriate basis for distributing resourcesin society. Such problems of “distributive justice,” which have been widely discussedsince the time of the ancient Greeks, may occur each time a policy analyst prescribes acourse of action that affects two or more persons in society. Although we may seek away to measure social welfare, that is, the aggregate satisfaction experienced bymembers of a community. Yet individuals and groups within any community aremotivated by different values, norms, and institutional rules. What satisfies one personor group often fails to satisfy another. Under these circumstances, the analyst mustconsider a fundamental question: How can a policy maximize the welfare of society, and

TABLE 4

Comparison of Benefit-Cost Ratio and Net Benefits as Criteriaof Adequacy (in thousands of dollars)

Program Benefits Costs Benefit-cost Ratio Net Benefits

I 40 4 10 36

II 64 8 8 56III 100 40 2.5 60

Source: Adapted from Richard Zeckhauser and Elmer Schaefer, “Public Policy andNormative Economic Theory,” in The Study of Policy Formation, ed. R. A. Bauer and K. J. Gergen (New York: The Free Press, 1968), p. 73.


not just the welfare of particular individuals or groups? The answer to this question maybe pursued in several different ways:

1. Maximize individual welfare. The analyst can attempt to maximize thewelfare of all individuals simultaneously. This requires that a single transitivepreference ranking be constructed on the basis of all individual values. Arrow’simpossibility theorem, as we have seen, demonstrates that this is impossibleeven when there are only two persons and three alternatives.

2. Protect minimum welfare. The analyst can attempt to increase the welfare ofsome persons while still protecting the positions of persons who are worse off.This approach is based on the Pareto criterion, which states that one socialstate is better than another if at least one person is better off, and no one isworse off. A Pareto optimum is a social state in which it is not possible tomake any person better off without also making another person worse off.

3. Maximize net welfare. The analyst can attempt to increase net welfare (e.g.,total benefits less total costs) but assumes that the resulting gains could be usedto compensate losers. This approach is based on the Kaldor-Hicks criterion:One social state is better than another if there is a net gain in efficiency (totalbenefits minus total costs) and if those who gain can compensate losers. For allpractical purposes, this criterion, which does not require that losers actually becompensated, avoids the issue of equity.

4. Maximize redistributive welfare. The analyst can attempt to maximizeredistributional benefits to selected groups in society, for example, theracially oppressed, poor, or sick. One redistributive criterion has been putforth by philosopher John Rawls: One social state is better than another if itresults in a gain in welfare for members of society who are worst off.12

12John Rawls, A Theory of Justice (Cambridge, MA: Harvard University Press, 1971).

Rawls formulation attempts to provide an ethical foundation for the conceptof justice. It does so by requesting that we imagine ourselves in an “original” state,where there is a “veil of ignorance” about the future distribution of positions, statuses,and resources in a civil society yet to be established. In this original state, individualswill choose a social order on the basis of the redistributive criterion just described,because it is in everyone’s individual interest to establish a society in which they willnot be worst off.

By postulating this original condition, it becomes possible to reach consensus ona just social order. This original condition should be contrasted with the conditionsof present societies, in which vested interests make it impossible to reach consensuson the meaning of justice. The weakness of Rawls’s formulation is its oversimplifica-tion or avoidance of conflict. The redistributive criterion is appropriate forwell-structured problems and not for the types of problems typically encountered bypublic policy analysts. This does not mean that the redistributive criterion cannot beused to make choices, but it does mean that we have still not reached a single basisfor defining social welfare.

None of these criteria of equity is fully satisfactory. The reason is that conflict-ing views about the rationality of society as a whole (social rationality) or of theappropriateness of legal norms guaranteeing rights to property (legal rationality)




cannot be resolved simply by appealing to formal economic rules (e.g., the Pareto orKaldor–Hicks criteria) or to formal philosophical principles (e.g., Rawls’s redistributivecriterion). Questions of equity, fairness, and justice are political ones; that is, they areinfluenced by processes involving the distribution and legitimation of power in society.Even though economic theory and moral philosophy can improve our capacity tocritically assess competing criteria of equity, they cannot replace the political process.

Responsiveness. Responsiveness refers to the extent that a policy satisfies theneeds, preferences, or values of particular groups. The criterion of responsiveness isimportant because an analyst can satisfy all the previously discussed criteria—effec-tiveness, efficiency, adequacy, equity—yet still fail to respond to the stated needs ofa group that is supposed to benefit from a policy. A recreation program might resultin an equitable distribution of facilities but be unresponsive to the needs of particu-lar groups (e.g., the elderly). In effect, the responsiveness criterion asks a practicalquestion: Do criteria of effectiveness, efficiency, adequacy, and equity actually reflectthe needs, preferences, and values of particular groups?

Appropriateness. The final criterion to be discussed here is appropriateness, whichis closely related to substantive rationality, because questions about theappropriateness of a policy are not concerned with individual criteria but with twoor more criteria taken together. Appropriateness refers to the value or worth of aprogram’s objectives and to the tenability of assumptions underlying these objec-tives. Although all other criteria take objectives for granted—for example, the valueof efficiency is seldom questioned—the criterion of appropriateness asks whetherthese objectives are proper ones for society. To answer this question, analysts mayconsider all criteria together—that is, reflect on the relations among multiple formsof rationality—and apply higher-order criteria (metacriteria) that are logically priorto those of effectiveness, efficiency, adequacy, equity, and responsiveness.13

The criterion of appropriateness is necessarily open ended, because by definitionit is intended to go beyond any set of existing criteria. For this reason, there is not andcannot be a standard definition of appropriateness. The best we can do is considerseveral examples:

1. Equity and efficiency. Is equity as redistributional welfare (Rawls) an appro-priate criterion when programs designed to redistribute income to the poor areso inefficient that only a small portion of redistributional benefits actuallyreaches them? When such programs are viewed as a “leaky bucket” used tocarry benefits to the poor, analysts may question whether equity is anappropriate criterion.14

2. Equity and entitlement. Is equity as minimum welfare an appropriatecriterion when those who receive additional benefits have not earned themthrough socially legitimate means? Analysts may question the appropriatenessof the Pareto criterion when those who gain (even though none lose) have doneso through corruption, fraud, discrimination, and unearned inheritance.15

13On similar metacriteria, including generality, consistency, and clarity, see Duncan MacRae Jr.,The Social Function of Social Science (New Haven, CT: Yale University Press, 1976).14See Arthur M. Okun, Equality and Efficiency (Washington, DC: Brookings Institution, 1975).


3. Efficiency, equity, and humanistic values. Are efficiency and equity appro-priate criteria when the means required to achieve an efficient or just societyconflict with democratic processes? Analysts may challenge the appropriate-ness of efficiency or equity when efforts to rationalize decision making forthese purposes subvert conditions required for emancipation, individuation,or self-actualization.16 Efficiency, equity, and humanism are not necessarilyequivalent. Alienation, as Marx and other social thinkers have recognized, isnot automatically eliminated by creating an abundant society of equals.

4. Equity and reasoned ethical debate. Is equity as redistributive welfare(Rawls) an appropriate criterion when it subverts opportunities for reasonedethical debate? This criterion of equity can be challenged on grounds that itpresupposes an individualistic conception of human nature with which ethicalclaims are no longer derived from reasoned arguments but are

the contractual composite of arbitrary (even if comprehensible) valuesindividually held and either biologically or socially shaped. The structure ofthe Rawlsian argument thus corresponds closely to that of instrumentalrationality; ends are exogenous, and the exclusive office of thought in theworld is to ensure their maximum realization. [Rawls’s premises] reduce allthought to the combined operations of formal reason and instrumentalprudence in the service of desire.17

APPROACHES TO PRESCRIPTIONIn making policy prescriptions, the analyst typically addresses a number of interre-lated questions: Whose needs, values, and opportunities are at issue, and whatalternatives are available for their satisfaction? What goals and objectives should beattained, and how should they be measured? How much will it cost to attain objec-tives and what kinds of constraints—budgetary, legal, administrative, political—may impede their attainment? Are there side effects, spillovers, and other antici-pated and unanticipated consequences that should be counted as costs or benefits?How will the value of costs and benefits change over time? How certain is it thatforecasted outcomes will occur? What should be done?

Public versus Private ChoiceAnswers to these questions are designed to enhance our capabilities for makingpersuasive policy prescriptions. Although these questions are just as relevant forpolicy making in the private as in the public sector, there are several importantdifferences between these sectors that should be kept in mind:18

17See Laurence H. Tribe, “Ways Not to Think about Plastic Trees,” in Tribe and others, When ValuesConflict (Cambridge MA: Ballinger, 1976), p. 77.18See H. H. Hinrichs and G. M. Taylor, Systematic Analysis: A Primer on Benefit-Cost Analysis andProgram Evaluation (Pacific Palisades, CA: Goodyear Publishing, 1972), pp. 4–5.

16See, for example, Jurgen Habermas, Toward a Rational Society (Boston, MA: Beacon Books, 1970);and Legitimation Crisis (Boston, MA: Beacon Books, 1975).

15See, for example, Peter Brown, “Ethics and Policy Research,” Policy Analysis 2 (1976): 325–40.




1. Nature of public policy processes. Policy making in the public sector involvesbargaining, compromise, and conflict among citizens’ groups, legislativebodies, executive departments, regulatory commissions, businesses, and manyother stakeholders. There is no single producer or consumer of goods andservices whose profits or welfare is to be maximized. The presence of numer-ous stakeholders with competing or conflicting values makes problems ofchoice more complex in the public than in the private sector.

2. Collective nature of public policy goals. Policy goals in the public sector arecollective ones that are supposed to reflect society’s preferences or some broad“public interest.” The specification of these collective goals, as we have seen,often involves multiple conflicting criteria, from effectiveness, efficiency, andadequacy to equity, responsiveness, and appropriateness.

3. Nature of public goods. Public and private goods may be divided into threegroups: specific goods, collective goods, and quasi-collective goods. Specificgoods are exclusive, because the person who owns them has the legal right toexclude others from their benefits. The allocation of specific goods (e.g., cars,houses, or physicians’ services) is often made on the basis of market prices,which are determined by supply and demand. Collective goods are nonexclusive,because they may be consumed by everyone. No person can be excluded fromthe consumption of clean air, water, and roads provided by the government. It isfrequently impossible to allocate collective goods on the basis of market prices,because normal relationships between supply and demand do not operate in thepublic sector. Quasi-collective goods are specific goods whose production hassignificant spillover effects for society. Although elementary education can beprovided by the private sector, the spillovers are regarded as so important thatthe government produces great quantities at a cost affordable by all.

Organizations in the public and private sectors produce each of these threetypes of goods. Nevertheless, the public sector is primarily occupied with theprovision of such collective and quasi-collective goods as defense, education, socialwelfare, public safety, transportation, environmental protection, recreation, andenergy conservation. By contrast, the private sector is chiefly concerned with theproduction of such specific goods as food, appliances, machinery, and housing.Contrasts between the production of these goods in the public and private sectorsare illustrated in Figure 3.

Because the nature of these three types of goods differs, so do the procedures forestimating their value to producers and consumers. The primary aim of a privatefirm producing specific goods for the market is to make profits, that is, to maximizethe difference between the total revenues earned by selling a product and the totalcosts required for its production. When a private firm is faced with a choice betweentwo or more products that differ in the revenues they will earn and the costsrequired for their production, the firm will choose that product that maximizesprofits, defined as total revenue minus total costs. If the firm should decide to investin a product that yields lower profits, either in the short or in the long run, we canestimate the opportunity cost of the decision. Opportunity cost refers to the benefitssacrificed by investing resources to produce one product when another moreprofitable alternative might have been chosen.


PRIVATE SECTOR

SpecificGoods

Quasi-Collective

Goods

PUBLIC SECTOR

CollectiveGoods

Supply and DemandOpportunity costs in the private sector can be estimated by using market prices asa measure of costs and benefits. Market prices of specific goods are determined bysupply and demand. If we look at various combinations of the price and quantity ofa specific good, we observe that (1) consumers will demand greater quantities (Q)of a product as the price (P) for that product decreases, and (2) producers willsupply greater quantities (Q) of a product as the price (P) of that product increases(Figure 4). Finally, (3) the combination of price and quantity that yields a singlelevel of consumer demand and producer supply—that is, the point at which supplyand demand intersect (PeQe in Figure 4)—indicates the price and quantity ofspecific goods that will be sold on the market. Graphic representations of thevarious combinations of price and quantity at which consumers and producers arewilling to buy and sell a product are called a demand curve and a supply curve,respectively. The point at which demand and supply curves intersect is called theequilibrium price–quantity combination.

The equilibrium price–quantity combination represents that point at whichconsumers and producers, if they are completely free to choose what they will buy andsell, will produce and consume equal quantities of a good at a given price. If we knowthe equilibrium price–quantity combination for a specific good (e.g., electric tooth-brushes), we can determine the profit of a given investment by subtracting the totalcosts required to produce a specific good from the total revenues earned by selling theproduct at the equilibrium price (Pe) and equilibrium quantity (Qe). By knowingthe profits that can be earned from alternative investments, it is possible to estimatethe opportunity costs of investing resources to produce one product when anothermore profitable investment might have been made. For example, if a firm producingelectric toothbrushes (a specific good) and earning an annual profit of $1 million findsthat it could have earned a profit of $2 million by producing electric drills at the same

FIGURE 3Three types of goods in thepublic and private sectors.Source: H. H. Hinrichs and G. M.Taylor,Systematic Analysis: A Primer on Benefit-Cost and

Program Evaluation (Pacific Palisades, CA:Goodyear Publishing, 1972), p. 5.




Qe

Quantity (Q)

Q1

P1

Price–Quantity Combination (P1 Q1)

Price–QuantityCombination (P2 Q2)

Equilibrium Price–QuantityCombination (Pe Qe)

Q2

PePric

e (P

)

P2

DEMANDCURVE

SUPPLYCURVE

cost (including costs for any additional labor, technology, or marketing facilities), theopportunity cost of the investment in electric toothbrush production is $1 million.

The logic of profit maximization in the private sector can be extended topublic policy. We can view public programs as if they were private firms attempt-ing to maximize profits on investments. Instead of using profits (total revenueminus total cost) as a criterion for prescribing alternatives, we can use net benefits(total benefits minus total costs). We can also apply the concept of opportunitycosts by viewing public programs as investments in the production of goods thatmight have been made by private firms. For example, if government invests$50 million in the construction of a dam that will yield $10 million in net benefitsto farmers and other beneficiaries, it must extract these investment resources fromprivate citizens who could have invested the $50 million elsewhere. If privateinvestment would have yielded $20 million in net benefits (profits) to privatecitizens, the opportunity cost of building the dam is $10 million.

Public ChoiceThis logic begins to break down when we consider differences between public andprivate choice, including contrasts among specific, quasi-collective, and collectivegoods. Although the logic of profit maximization can be applied to certain kinds ofpublic goods (e.g., the production of hydroelectric power), there are reasons why con-cepts of profit and opportunity costs are difficult to apply to problems of public choice:

1. Multiple legitimate stakeholders. Public policy making involves multiplestakeholders whose claims on public investments are frequently guaranteed bylaw. Although there are also multiple stakeholders in private policy making,

FIGURE 4Supply and demand curves and the equilibrium price–quantitycombination.


none except owners and stockholders can legitimately make claims oninvestments. In the public sector, with its many legitimate stakeholders, it isdifficult to know whose benefits should be maximized and who should bearthe costs of public investments.

2. Collective and quasi-collective goods. Because most public goods are collec-tive (e.g., clean air) or quasi-collective (e.g., education), it is often difficult orimpossible to sell them on the market, where transactions are made on thebasis of ability to pay. For this reason, market prices are frequently unavailableas a measure of net benefits or opportunity costs. Even when prices are basedon interview surveys among citizens, some persons may falsely indicate nowillingness to pay for a public good but later use the good at a price lowerthan that they are actually willing to pay. This is called the free-rider problem.

3. Limited comparability of income measures. Even when the net benefits ofpublic and private investments can be expressed in dollars as a common unitof measure, private investments with higher net benefits are not always preferableto public ones with lower (or even zero or negative) net benefits. For example,a private investment in a new office building that yields $1 million in net benefitsis unlikely to be preferred over a public investment in a cure for cancer that yieldszero or negative net benefits. Even when comparisons are confined to publicinvestments, the use of income measures to prescribe alternatives implies that thegoal of increasing aggregate income is more important than good health or bettereducation or the elimination of poverty. These other goals are wrongly viewed asillegitimate if they fail to increase future income.19

4. Public responsibility for social costs and benefits. Private firms are responsiblefor their own private costs and private benefits but not for social ones, exceptas prescribed by moral convention or law (e.g., antipollution legislation). Bycontrast, the costs and benefits of public programs are socialized, thus becomingsocial costs and social benefits that go beyond the need to satisfy private stock-holders. Social costs and social benefits (e.g., the costs of damaging the naturalenvironment by building highways) are difficult to quantify and often have nomarket price. Such social costs and social benefits are called intangibles.

Contrasts between public and private choice do not mean that the logic ofprofit maximization is wholly inapplicable to public problems. Contrasts domean, however, that there are limitations to such logic when applied to publicproblems. These strengths and limitations are evident when we consider two ofthe most important approaches to prescription in policy analysis: cost-benefit andcost-effectiveness analysis.

Cost-Benefit AnalysisCost-benefit analysis is an approach to policy prescription that permits analysts tocompare and advocate policies by quantifying their total monetary costs and totalmonetary benefits. Cost-benefit analysis may be used to prescribe policy actions, inwhich case it is applied prospectively (ex ante), and it may also be used to evaluate

19Alice Rivlin, Systematic Thinking for Social Action (Washington, DC: Brookings Institution, 1971), p. 56.




policy performance. In this case, it is applied retrospectively (ex post). Much ofmodern cost-benefit analysis is based on that field of economics that treats problemsof how to maximize social welfare, that is, the aggregate economic satisfactionexperienced by members of a community.20 This field is called welfare economics,because it is especially concerned with the ways that public investments maycontribute to the maximization of net income as a measure of aggregate satisfaction(welfare) in society.

Cost-benefit analysis has been applied to many different kinds of public programsand projects. The earliest applications were in the area of dam construction and theprovision of water resources, including efforts to analyze the costs and benefits ofhydroelectric power, flood control, irrigation, and recreation. Other more recentapplications include transportation, health, personnel training, and urban renewal.

When used to make prescriptions in the public sector, cost-benefit analysis hasseveral distinctive characteristics:

1. Cost-benefit analysis seeks to measure all costs and benefits to society that mayresult from a public program, including various intangibles that cannot beeasily measured in terms of monetary costs and benefits.

2. Traditional cost-benefit analysis epitomizes economic rationality, because thecriterion most frequently employed is global economic efficiency. A policy orprogram is said to be efficient if its net benefits (i.e., total benefits minus totalcosts) are greater than zero and higher than those net benefits that would haveresulted from an alternative public or private investment.

3. Traditional cost-benefit analysis uses the private marketplace as a point ofdeparture in prescribing public programs. The opportunity costs of a publicinvestment are often calculated on the basis of what net benefits might havebeen gained by investing in the private sector.

4. Contemporary cost-benefit analysis, sometimes called social cost-benefit analysis,can also be used to measure redistributional benefits. As social cost-benefit analy-sis is concerned with criteria of equity, it is consistent with social rationality.

Cost-benefit analysis has a number of strengths. First, both costs and benefitsare measured in dollars as a common unit of value. This permits analysts to subtractcosts from benefits, a task that is not possible with cost-effectiveness analysis.Second, cost-benefit analysis permits us to go beyond the confines of a single policyor program and link benefits to the income of society as a whole. This is possiblebecause the results of individual policies and programs can be expressed inmonetary terms, at least in principle. Finally, cost-benefit analysis allows analyststo compare programs in widely differing areas (e.g., health and transportation),because net efficiency benefits are expressed in terms of dollars. This is not possiblewhen effectiveness is measured in terms of units of service, because the number ofpersons treated by physicians cannot be directly compared with the number of milesof roads built.

20For comprehensive treatments of cost-benefit analysis, see, for example, Edward J. Mishan, Cost-Benefit Analysis (New York: Frederick A. Praeger, 1976); and Edward M. Gramlich, Benefit-CostAnalysis of Public Programs, 2d ed. (Englewood Cliffs, NJ: Prentice Hall, 1990).


Cost-benefit analysis, in both its traditional and its contemporary forms, alsohas several limitations. First, an exclusive emphasis on economic efficiency can meanthat criteria of equity are meaningless or inapplicable. In practice, the Kaldor–Hickscriterion simply ignores problems of redistributional benefits, whereas the Paretocriterion seldom resolves conflicts between efficiency and equity. Second, monetaryvalue is an inadequate measure of responsiveness, because the actual value of incomevaries from person to person. For example, an extra $100 of income is far moresignificant to the head of a poverty household than to a millionaire. This problem oflimited interpersonal comparisons often means that income is an inappropriatemeasure of individual satisfaction and social welfare. Third, when market prices areunavailable for important goods (e.g., clean air or health services), analysts are oftenforced to estimate shadow prices; that is, they make subjective estimates of the pricethat citizens might be willing to pay for goods and services. These subjective judg-ments may simply be arbitrary expressions of the values of analysts.

Cost-benefit analysis, even when it takes into account problems of redistribu-tion and social equity, is closely tied to income as a measure of satisfaction. For thisreason, it is difficult to discuss the appropriateness of any objective that cannot beexpressed in monetary terms. The exclusive concentration on net income or redistri-butional benefits often inhibits reasoned debates about the ethical or moral bases ofalternative policies. Cost-benefit analysis is therefore limited in its capacity toconsider relations among alternative forms of reason (technical, economic, social,legal) as part of an overall effort to establish the substantive rationality of policiesand programs. Hence, cost-benefit analysis may come very close to providing uswith “the price of everything and the value of nothing.”

Types of Costs and BenefitsIn using cost-benefit analysis, it is essential to consider all costs and benefits that mayresult from a policy or program. Although such a comprehensive inventory of costsand benefits is quite difficult to achieve in practice, it can help reduce errors that occurwhen we omit some costs and benefits from our analysis. One of the best ways toguard against such errors is to classify costs and benefits as inside (“internal”) versusoutside (“external”), directly measurable (“tangible”) versus indirectly measurable(“intangible”), primary (“direct”) versus secondary (“indirect”), and net efficiency(“real”) versus distributional (“pecuniary”). These types of costs and benefits, and thequestions they are based on, are illustrated in Figure 5.

Inside versus outside costs and benefits. Here the question is whether a given costor benefit is internal or external to a given target group or jurisdiction. Inside orinternal costs and benefits are called internalities; outside or external ones are calledexternalities. What is an inside cost or benefit (internality) in one case will be anoutside one (externality) in another. The difference depends on how the analystdraws boundaries around the target group or jurisdiction. If the boundary is societyas a whole, there may be no externalities. If, however, the boundary is a particulartarget group or jurisdiction, there will be internalities as well as externalities.Externalities are the positive and negative spillovers outside the boundaries of thejurisdiction or target group. For example, the construction of high-rise apartments




TYPE OFCOST ORBENEFIT

Outside

DirectlyMeasurable

DirectlyMeasurable

Is the cost orbenefit“internal” or“external” to agiven targetgroup orjurisdiction?

Is the cost orbenefit a“direct” or“indirect”result of theprogram?

Do combinedcosts andbenefits createa “real”increase inaggregateincome or“pecuniary”shifts from onegroup toanother?

Is the cost orbenefit a“tangible” or“intangible”one?

PrimaryNet Efficiency

Redistributional

Net Efficiency

RedistributionalSecondary

IndirectlyMeasurable

IndirectlyMeasurable

Inside

Primary

Secondary

Primary

Secondary

Primary

Secondary

Net Efficiency

Redistributional

Net Efficiency

Redistributional

Net Efficiency

Redistributional

Net Efficiency

Redistributional

Net Efficiency

Redistributional

Net Efficiency

Redistributional

FIGURE 5Classification of costs and benefits according to four types of questions.


in a central-city area as part of an urban renewal program has certain costs andbenefits within the urban jurisdiction, including expenditures on construction andincome derived from rents. The same urban renewal program also has external coststo suburban jurisdictions that must provide additional fire and police services inareas where vagrants or criminals have resettled because they can no longer affordto live in the central-city area.

Directly measurable versus indirectly measurable costs and benefits. The questionhere is whether the cost or benefit is a tangible or intangible one. Tangibles are costsand benefits that are directly measurable in terms of known market prices for goodsand services; intangibles are costs and benefits that are indirectly measurable interms of estimates of such market prices. When dealing with intangibles, such as theprice of clean air, the analyst may attempt to estimate shadow prices by making asubjective judgment about the dollar value of costs and benefits. In the urbanrenewal example, the analyst may try to estimate the opportunity costs that variousgroups might be willing to pay for the destruction of the sense of communityresulting from urban renewal.

Primary and secondary costs and benefits. Here the question is whether the cost orbenefit is a direct or indirect result of a program. A primary cost or benefit is onethat is related to the most highly valued program objectives; a secondary cost orbenefit is one that is related to objectives that are less valued. For example, an urbanrenewal program may have as its most important objective the provision of low-costhousing to the poor, in which case the primary costs and benefits would includeexpenditures for construction and income from rents. Secondary costs and benefitswould involve lesser objectives, including intangible costs, such as the destruction ofa sense of community, or tangible benefits, such as reduced costs of police and fireservices because of better street lighting and fire-resistant construction.

Net efficiency versus redistributional benefits. Here the question is whethercombined costs and benefits create an increase in aggregate income or result merelyin shifts in income or other resources among different groups. Net efficiency benefitsrepresent a “real” increase in net income (total benefits minus total costs);redistributional benefits result in a “pecuniary” shift in the incomes of one group atthe expense of another but without increasing net efficiency benefits. Such changesare called real benefits and pecuniary benefits, respectively. For example, an urbanrenewal project may produce $1 million in net efficiency benefits. If it also results inincreased sales in small grocery stores in the immediate area—and decreased sales instores that are more distant from new high-rise apartments—the benefits and thecosts of income gained and lost are pecuniary. They cancel each other out withoutproducing any change in net efficiency benefits.

To call one type of benefit real and the other merely pecuniary can introduceconsiderable bias: Is an increase in the disposable income of the poor less real thanan improvement in the net benefits of the community? For this reason, it is best todrop the categories of real versus pecuniary altogether, provided that we do not addany increase in redistributional benefits to net efficiency benefits without alsosubtracting the costs of redistribution to those whose income or other benefits were




reduced. This error, called double counting, can produce invalid estimates of netefficiency benefits.

Observe that answers to these four questions can result in many combinations ofcosts and benefits. A given inside cost or benefit may be directly measurable (tangible)or indirectly measurable (intangible). An intangible cost or benefit may in turn beprimary or secondary, depending on the relative importance of different objectives.A primary or secondary objective may itself be defined in terms of net efficiency orredistribution. Typically, either net efficiency or redistribution must be a primaryobjective, because it is seldom possible to create higher net efficiency benefits andredistributional benefits at the same time. In other words, these two types of benefitsconflict. They represent a situation requiring trade-offs, that is, conscious efforts todetermine how much of one objective should be sacrificed to obtain another.

Tasks in Cost-Benefit AnalysisIn conducting a cost-benefit analysis, the following tasks (Table 5) are important formaking maximally plausible prescriptions:

Problem structuring. The search for alternative formulations of the problem andthe definition of the boundaries of the metaproblem are essential tasks of cost-bene-fit analysis. Problem structuring does not occur once, at the beginning of the cost-benefit analysis, but occurs at many points throughout the analysis. Problem struc-turing yields information about the potentially relevant goals, objectives,alternatives, criteria, target groups, costs, and benefits to guide the analysis.Problem structuring also may result in dissolving, resolving, and unsolving theproblem many times during the cost-benefit analysis.

Specification of objectives. Analysts usually begin with general aims or goals, forexample, controlling cocaine addiction. Goals, as we have seen, must be convertedinto objectives that are temporally specific and measurable. The goal of controllingcocaine addiction may be converted into a number of specific objectives, for exam-ple, a 50 percent reduction in the supply of cocaine within five years. Objectivesusually imply policy alternatives. The reduction in supply implies a policy of druginterdiction, whereas another objective—for example, a 50 percent reduction in thedemand for cocaine—not only implies a policy of drug interdiction to raise the priceof cocaine and lower demand but also suggests drug rehabilitation as a way to de-crease the number of addicts and, in turn, lower demand. In this and other cases, therelation between objectives and policy alternatives rests on causal assumptions thatmay prove to be questionable or plainly mistaken.

Identification of alternative solutions. Once objectives have been specified, theanalyst’s assumptions about the causes of a problem and its potential solutions arealmost inevitably transformed into alternative policies to achieve these objectives.The way a problem has been structured—for example, the problem of cocaine may beformulated as a demand problem requiring the interdiction of the flow of drugs fromSouth America—thus governs the policies perceived to be appropriate and effective.


TABLE 5

Ten Tasks in Conducting a Cost-Benefit Analysis

Task Description

Problem structuring Formulation of metaproblem by defining boundaries ofgoals, objectives, alternatives, criteria, target groups,costs, and benefits

Specification of objectives Conversion of general aims (goals) into temporallyspecific and measurable aims (objectives)

Identification of alternatives Identification of most promising policy alternatives froma larger set of potential solutions defined at theproblem-structuring phase

Information search, analysis,and interpretation

Location, analysis, and interpretation of informationneeded to forecast the outcomes of specified policyalternatives

Identification of target groups and beneficiaries

Listing of all groups (stakeholders) that are a target ofaction (e.g., regulation) or inaction (i.e., status quo) orwill benefit from action or inaction

Estimation of costs and benefits

Estimation in units of monetary value the specificbenefits and costs of each alternative in all classes(internal and external, directly and indirectlymeasurable, primary and secondary, efficiency andredistributional)

Discounting of costs and benefits

Conversion of monetary costs and benefits into theirpresent value on the basis of a specified discount factor

Estimation of risk and uncertainty

Use of sensitivity and a fortiori analysis to estimateprobabilities that benefits and costs will occur in future

Choice of decision criterion Choice of criterion for selecting among alternatives:Pareto improvement, net efficiency improvement,distributional improvement, internal rate of return

Prescription Selection of alternative that is most plausible,considering rival ethical and causal hypotheses

If problem search has been severely restricted, inadvertently or for explicit politicaland ideological reasons, the policy metaproblem will exclude otherwise promisingpolicy alternatives. Eventually, problem unsolving may be necessary.

Information search, analysis, and interpretation. The task here is to locate, analyze,and interpret information relevant for forecasting the outcomes of policy alterna-tives. At this point, the principal objects of a forecast are the costs and benefits ofpolicy alternatives already identified at the previous stage of analysis. Informationmay be acquired from available data on costs and benefits of similar existingprograms. For example, prior to the establishment of the Drug EnforcementAdministration, published budgetary data on the costs of drug enforcement activi-ties of the U.S. Customs Service were used to estimate some of the costs of new drug




interdiction policies. The benefits of interdiction in the form of decreased supply anddemand were based on economic and administrative assumptions that eventuallyproved to be highly questionable or simply wrong.21

Identification of target groups and beneficiaries. Here the task is to conduct a stake-holder analysis that lists all groups that have a stake in the policy issue because theywill be affected, negatively or positively, by the adoption and implementation ofpolicy prescriptions. Target groups are the objects of new regulations or restrictionsthat typically involve the loss of freedom or resources—for example, regulationsrestricting the choice of women to terminate pregnancies or new tax schedules thatincrease the tax burden on the middle class. By contrast, beneficiary groups will gainfrom the adoption and implementation of prescriptions, for example, commercialtruckers who receive increased net income as a consequence of some forty statesmoving from the 55 mph to the 65 mph speed limit in the late 1980s.

Estimation of costs and benefits. This task, requires the estimation in monetaryterms of all benefits and costs that are likely to be experienced by target groupsand beneficiaries. As we have seen (Figure 5), there are several types of benefitsand costs: internal and external, directly and indirectly measurable, primary andsecondary, and net efficiency and redistributional. In many areas of public policy,it is difficult and even practically impossible to estimate costs or benefits. For ex-ample, the monetary benefits of a fatality averted through mandatory seat beltlaws, obligatory state vehicle inspections, or breast cancer screening programsare subject to broad disagreement, especially when the costs of these fatality-averting policies are subtracted from the benefits of a human life. The validity,reliability, and appropriateness of such measures are often disputed.

Discounting of costs and benefits. If a certain level of costs and benefits is projectedfor a future time period, estimates must adjust for the decreasing real value ofmoney due to inflation and future changes in interest rates. The real value of costsand benefits is usually based on the technique of discounting, a procedure thatexpresses future costs and benefits in terms of their present value. For example, thevalue of human life is sometimes expressed as an hourly wage rate multiplied by lifeexpectancy. If the human life being valued is that of a twenty-year-old with lowearning capacity—and if the discount factor used is 4 percent—the present value ofthat person’s life can amount to no more than a few thousand dollars. This raisesethical questions about the comparability of values attached to the lives of youngand old, poor and rich, rural and urban populations.

Estimation of risk and uncertainty. The task here is to employ sensitivity analysis,a generic term that refers to procedures that test the sensitivity of conclusions toalternative assumptions about the probability of occurrence of different costs

21See George M. Guess and Paul G. Farnham, Cases in Public Policy Analysis, Chapter 5, “Copingwith Cocaine” (New York: Longman, 1989), pp. 7–48; and Constance Holden, “Street-Wise CrackResearch.” Science 246 (December 15, 1989): 1376–81. Neither the Bush nor the Obama administrationmade an effort to unsolve this problem.


and benefits or to different discount factors. It is difficult to develop reliableprobabilistic estimates because different forecasts of the same future outcome oftenhave widely differing levels of accuracy that stem from differences in institutional,temporal, and historical contexts.

Choice of a decision criterion. Here the task is to specify a decision criterion or rulefor choosing between two or more alternatives with different mixes of costs andbenefits. As we have seen, these criteria are of six types: efficiency, effectiveness,adequacy, equity, responsiveness, and appropriateness. Among efficiency criteria arenet efficiency improvement (the net present value after discounting costs and bene-fits to their present value must exceed zero) and internal rate of return (the rate ofreturn on a public investment must be greater than that which could be earned at theactual rate of interest paid). Effectiveness criteria include the marginal effectivenessof graduated levels of public investment in producing the greatest volume of somevalued good or service (e.g., access to medical treatment per dollar invested in healthmaintenance organizations versus traditional health care providers). Distributionaland redistributional criteria include Pareto improvement (at least one group gainsand none loses) and Rawlsian improvement (those worst off become better off),respectively. The choice of a decision criterion has important ethical implications,because decision criteria are grounded in very different conceptions of moralobligation and the just society.

Prescription. The final task in cost-benefit analysis is to make a prescription bychoosing among two or more policy alternatives. The choice of alternatives isseldom clear and unequivocal, which calls for a critical analysis of the plausibility ofprescriptions, taking into account rival causal and ethical hypotheses that mayweaken or invalidate a prescription. For example, a prescription based on Paretoimprovement may be weakened by establishing that the rich will benefit at theexpense of the middle and lower classes. In retrospect, this has been the actualhistory of social and economic policy in the United States over the past thirty years.

Cost-Effectiveness AnalysisCost-effectiveness analysis is an approach to policy prescription that permitsanalysts to compare and advocate policies by quantifying their total costs andeffects. In contrast to cost-benefit analysis, which attempts to measure all relevantfactors in a common unit of value, cost-effectiveness analysis uses two units ofvalue. Costs are measured in monetary units, whereas effectiveness is typicallymeasured in units of goods, services, or some other valued effect. In the absence of acommon unit of value, cost-effectiveness analysis does not permit net effectivenessor net benefit measures, because it makes no sense to subtract total costs from totalunits of goods or services. It is possible, however, to produce cost-effectiveness andeffectiveness-cost ratios, for example, ratios of costs to units of health service orunits of health service to costs.

These ratios have an altogether different meaning than cost-benefit ratios.Whereas effectiveness-cost and cost-effectiveness ratios tell us how much of a goodor service is produced per dollar expended—or, alternatively, how many dollars are




expended per unit produced—benefit-cost ratios tell us how many times morebenefits than costs are produced in a given instance. Benefit-cost ratios mustbe greater than 1 if there are any net benefits at all. For example, if total benefitsare $4 million and total costs are $6 million, then the benefit-cost ratio is 4 to 6,or 0.66, and total net benefits are minus $2 million. If net benefits are zero($4 million minus $4 million equals zero), then the benefit-cost ratio is always1 (4/4 � 1). By definition, then, a benefit-cost ratio must exceed 1 for thereto be any net benefits. This is not true of effectiveness-cost ratios, which have adifferent meaning from one case to the next.

Cost-effectiveness analysis, like its cost-benefit counterpart, may be appliedprospectively (ex ante) as well as retrospectively (ex post). In contrast to cost-benefitanalysis, cost-effectiveness analysis grew out of work done for the DefenseDepartment in the early 1950s, and not from work in the field of welfare economics.Much of the early development of cost-effectiveness analysis was carried out by theRAND Corporation in projects designed to evaluate alternative military strategiesand weapons systems. In the same period, cost-effectiveness analysis was applied toproblems of program budgeting in the Department of Defense and was extended inthe 1960s to other government agencies.22

Cost-effectiveness analysis is particularly appropriate for questions involvingthe most efficient way to use resources to attain objectives that cannot be expressedin terms of income. Cost-effectiveness analysis has been used to prescribe alternativepolicies and programs in criminal justice, personnel training, transportation, health,defense, and other areas.

Cost-effectiveness several distinguishing characteristics:

1. Cost-effectiveness analysis, because it avoids problems of measuring benefits inmonetary terms, is more easily applied than cost-benefit analysis.

2. Cost-effectiveness analysis epitomizes technical rationality, because itattempts to determine the utility of policy alternatives without relating theirconsequences to global economic efficiency or aggregate social welfare.

3. Cost-effectiveness analysis, because it relies minimally on market prices, isless dependent on the logic of profit maximization in the private sector.Cost-effectiveness analysis, for example, frequently makes no attempt todetermine whether benefits exceed costs or whether alternative investmentsin the private sector would have been more profitable.

4. Cost-effectiveness analysis is well suited to the analysis of externalities andintangibles, because these types of effects are difficult to express in dollars asa common unit of measure.

5. Cost-effectiveness analysis typically addresses fixed-cost (type I) or fixed-effectiveness (type II) problems, whereas cost-benefit analysis typicallyaddresses variable-cost–variable-effectiveness (type III) problems.

22See, for example, David Novick, Efficiency and Economy in Government through New Budgetingand Accounting Procedures (Santa Monica, CA: RAND Corporation, February 1954); Roland J.McKean, Efficiency in Government through Systems Analysis (New York: Wiley, 1958); Charles J.Hitch and Roland J. McKean, The Economics of Defense in the Nuclear Age (New York: Atheneum,1965); and T. A. Goldman, ed., Cost-Effectiveness Analysis (New York: Frederick A. Praeger, 1967).


The strengths of cost-effectiveness analysis are its comparative ease of application,its capacity to treat collective and quasi-collective goods whose values cannot beestimated on the basis of market prices, and its appropriateness for analyzingexternalities and intangibles. The main limitation of cost-effectiveness analysis isthat prescriptions are less easily related to questions of aggregate social welfare. Unlikecost-benefit analysis, efforts to measure costs and effectiveness are confined to givenprograms, jurisdictions, or target groups and cannot be used to calculate net incomebenefits as a measure of the aggregate satisfaction experienced by members of acommunity.

Cost-effectiveness analysis also attempts to consider all costs and benefits ofpolicies and programs, with the exception that benefits are not measured in monetaryterms. Many of the same types of costs and benefits as those discussed previously areimportant in cost-effectiveness analysis: inside versus outside, directly measurableversus indirectly measurable, and primary versus secondary. Although it is notpossible to calculate net efficiency benefits, redistributional effects may be analyzed.

The tasks in conducting a cost-effectiveness analysis are similar to thoserequired in cost-benefit analysis (see Table 5), with two exceptions. Only costs arediscounted to their present value and criteria of adequacy differ from those normallyused in cost-benefit analysis. In cost-effectiveness analysis, several criteria ofadequacy may be employed:

1. Least-cost criterion. After establishing a desired level of effectiveness, thecosts of programs with equal effectiveness are compared. Programs with lessthan the fixed level of effectiveness are dropped; the program that meets thefixed level of effectiveness at least cost is prescribed.

2. Maximum-effectiveness criterion. After establishing an upper level ofpermissible costs (usually a budgetary constraint), programs with equal costsare compared. Programs that exceed the upper cost limit are dropped; theprogram that meets the fixed-cost level with the maximum effectiveness isprescribed.

3. Marginal effectiveness criterion. If units of some service or good as well asthe costs of that service or good can be expressed on two continuous scales,the marginal effectiveness of two or more alternatives can be calculated. Forexample, the costs of providing security services by municipal police and byprivate security firms contracted by municipalities can be expressed on twocontinuous scales: a graduated scale of costs associated with each hour ofpatrols and the number of crimes against property reported, deterred, investi-gated, and cleared. A continuous cost-effectiveness function may be establishedfor each type of service provider. The provider with the highest effectiveness-cost ratio at any point along the function beyond some minimum level of effec-tiveness has the greater marginal effectiveness—that is, the highest effectivenessachieved at the last dollar expended at the margin between the last dollar andthe subsequent dollar cost (see point E1 – C1 in Figure 2).

4. Cost-effectiveness criterion. The cost-effectiveness of two or more alterna-tives is the cost per unit of goods or services. For example, the 55 mphand 65 mph speed limits involve different costs per fatality averted, withthe latter ranking slightly lower in cost-effectiveness. If the 55 mph and




65 mph speed limits have approximately equal costs (the denominator),but the latter averts slightly fewer fatalities (the numerator), then theaverage cost per fatality averted will be greater for the 65 mph speed limit.Effectiveness-cost ratios, in contrast to the benefit-cost ratios used in cost-benefit analysis, may not take into account all costs (e.g., the valueof lives lost driving at the higher speed limit). For this reason, simpleeffectiveness-cost ratios may yield biased estimates of the actual costs of agiven service or good, although the exclusion of costs is by no means aninherent limitation of cost-effectiveness analysis.

METHODS AND TECHNIQUES FOR PRESCRIPTIONA range of methods and techniques is available for making prescriptions with cost-benefit and cost-effectiveness analysis (Table 6). These methods and techniques aremost easily understood if they are viewed as tools for carrying out the tasks of cost-benefit analysis described earlier (see Table 5). Cost-benefit analysis and cost-effec-tiveness analysis, as we have already seen, differ in certain important respects.These differences, however, are mainly related to the choice of decision criteria—for example, net efficiency improvement cannot be estimated with cost-effective-ness analysis because benefits are not expressed in monetary terms. With few ex-ceptions (e.g., calculating the present value of future benefits), methodologies andtechniques of cost-benefit analysis are also appropriate for cost-effectivenessanalysis.

Objectives MappingA frequent difficulty experienced in making policy prescriptions is knowing whatobjectives to analyze. Objectives mapping is a technique used to array goals andobjectives and their relationship to policy alternatives. Goals, objectives, andalternatives that have been identified by one or more methodologies of problemstructuring may be depicted in the form of an objectives tree, which is a pictorial dis-play of the overall structure of objectives and their relationships.23 The objectives,when mapped in the form of a tree diagram, often form a hierarchy in which certainobjectives that are necessary for the attainment of other objectives are arranged ver-tically. An objectives tree for the development of a national energy policy is illus-trated in Figure 6.

In constructing an objectives tree, the analyst may start with existing objectivesor may generate new ones. If it is necessary to generate new objectives, severalproblem-structuring techniques are available for this purpose, including hierarchyanalysis, classification analysis, brainstorming, and argumentation analysis. Once aset of objectives has been obtained, it is possible to arrange them hierarchically in atree such as that provided in Figure 6. Objectives trees are more general at the topand progressively more detailed as one moves lower in the hierarchy. Typically, the

23See T. Harrell Allen, New Methodologies in Social Science Research: Policy Sciences and FuturesResearch (New York: Frederick A. Praeger, 1978), pp. 95–106.


TABLE 6

Methods and Techniques for Prescription

Task Method/Technique

Problem structuring(a) Boundary analysis

Classificational analysisHierarchy analysisMultiple perspective analysisArgumentation analysisArgumentation mapping

Specification of objectives Objectives mappingValue clarificationValue critique

Information search, analysis, and interpretation Boundary analysis(b)

Identification of target groups and beneficiaries Boundary analysis

Estimation of costs and benefits Cost element structuringCost estimationShadow pricing

Discounting of costs and benefits Discounting

Estimation of risk and uncertainty Feasibility assessment(c)

Constraints mappingSensitivity analysisA fortiori analysis

Choice of decision criterion Value clarificationValue critique

Prescription Plausibility analysis(a)See the chapter “Structuring Policy Problems” for these methodologies and techniques.(b)See the chapter “Structuring Policy Problems.”(c)See the chapter “Forecasting Expected Policy Outcomes.”

uppermost part of an objectives tree contains broad purposes, while lower levelsrepresent goals, prime objectives, and sub-objectives. Observe that when we readdownward, we answer the question: How should we accomplish this objective?When we read upward, we answer the question: Why should we pursue this objec-tive? Objectives mapping is therefore useful, not only for purposes of mapping thecomplexities of policy implementation (i.e., “how” questions) but also for clarifyingthe ends of action (i.e., “why” questions). Finally, most objectives can be regardedas both ends and means.

Value ClarificationValue clarification is a procedure for identifying and classifying value premisesthat underlie the selection of policy objectives. The need for value clarification inmaking policy prescriptions is most evident when we consider competing criteria




DEVELOPNATIONALENERGYPOLICY

DoNothing

UseCoal

UseSolarPower

UseNuclearPower

Shift10 Percent

to Poor

Cut GasConsumption

RaisePrices

InsulateHomes

ProduceMore

Energy

RedistributeEnergy

ConserveEnergy

Goals

PrimeObjectives

Sub-Objectives

for prescription (effectiveness, efficiency, adequacy, responsiveness, equity,appropriateness) and the multiple forms of rationality from which these criteriaare derived. Value clarification helps answer such questions as the following:What value premises underlie the choice of policy objectives? Are these valuepremises those of policy analysts, of policy makers, of particular social groups, orof society as a whole? What circumstances explain the fact that certain groups arecommitted to these value premises and objectives, whereas others are opposed?Finally, what reasons are offered to justify particular value premises andobjectives?

There are several major steps in value clarification:24

1. Identify objectives of a policy or program. Objectives may simply be listed ordisplayed in the form of an objectives tree (see Figure 6).

2. Identify stakeholders who affect and are affected by the attainment ornonattainment of the objectives. Be sure to include yourself, as policyanalyst, in the list of stakeholders.

3. List the value premises that underlie each stakeholder’s commitment toobjectives. For example, environmentalists may value energy conservationbecause it preserves the esthetic qualities of the natural environment, whereas

FIGURE 6Objectives tree for national energy policy.


suburban homeowners may value energy conservation because it provides fora more secure material existence.

4. Classify value premises into those that are simply expressions of personal tasteor desire (value expressions), those that are statements about the beliefs ofparticular groups (value statements), and those that are judgments about theuniversal goodness or badness of the actions or conditions implied by theobjective (value judgments). For example, you as policy analyst may express apersonal desire for more energy consumption, environmental groups may statetheir belief in energy conservation, and oil producers may offer judgmentsabout increased energy production that are based on beliefs in universal rightsto use private property as owners see fit.

5. Further classify value premises into those that provide a basis for explainingobjectives (e.g., environmentalists seek to conserve energy because this isconsistent with their belief in the inviolability of nature) and those thatprovide a ground for justifying objectives (e.g., energy conservation is anappropriate objective because nature and humanity alike are entitled to rightsof self-protection).

The advantage of value clarification is that it enables us to go beyond theanalysis of objectives as if they were merely expressions of personal desire or taste.Value clarification also takes us one step beyond the explanation of circumstancesthat explain objectives. Although the bases of values are important—for example,it is important to know that the poor favor greater social equity—ethical andmoral disputes cannot be resolved without examining the grounds for justifyingobjectives.

Value CritiqueValue critique is a procedure for examining the persuasiveness of conflictingarguments offered in the course of a debate about policy objectives. Whereas valueclarification enables us to classify values according to their form, context, andfunction, value critique allows us to examine the role of values in policy argumentsand debates. Value clarification focuses on the objectives and underlying valuesof individual stakeholders. By contrast, value critique focuses on conflicts amongthe objectives and underlying values of different stakeholders. Value clarificationalso has a static quality; value critique looks toward changes in values that may re-sult from reasoned debates.

Procedures for value critique are an extension of the ethical mode of policyargument. In conducting a value critique, it is necessary to

1. Identify one or more advocative claims that set forth a prescription for action.2. List stakeholders who will affect and be affected by the implementation of the

prescription.

24For background material, see the chapter “Evaluating Policy Performance.”




3. Describe each stakeholder’s argument for and against the prescription.4. Identify each element in the debate: information (I), claim (C), qualifier (Q),

warrant (W), backing (B), objection (O), and rebuttal (R).5. Assess the ethical persuasiveness of each argument and determine whether to

retain, alter, or reject the prescription.

To illustrate the process of value critique, imagine an analyst who has completeda cost-benefit analysis of low-cost home insulation. The prescription or claim (C) isthat the government should adopt the low-cost home insulation program.Information (I) indicates that the net benefits of the program are $50 million. Thewarrant (W) is that programs which result in efficiency improvements are Paretooptimal, whereas the backing (B) states that in principle the winners can compensatelosers (Kaldor–Hicks criterion). The initial qualifier (Q1) is that the program seemsto be the right thing to do. After considering the strong objection about social equityand a weak rebuttal, the revised qualifier (Q2) states that the program is not asbeneficial as it first seemed. After the arguments of another each another stakeholderhave been considered, the prescription seems less persuasive (Figure 7).

This debate is simultaneously economic and ethical, because it involves questionsof efficiency and justice at the same time. One result of the debate might be to modifythe prescription in such a way that the government would provide low-cost homeinsulation on a graduated-cost basis. Those who cannot afford it might pay nothing;those in higher income brackets would pay the most. Yet this would make the programrelatively less attractive to large consumers of heating fuel. If equal costs weremaintained across the board (with the exception of the poor and the elderly, whowould pay nothing), the total net benefits may well be negative. Although valuecritique cannot finally answer the question of how much efficiency should be sacrificedfor a given increase in social equity, it does make possible a reasoned ethical debateabout such questions, rather than expecting cost-benefit analysis to answer questionsfor which it is ill suited. Economics is not ethics any more than price is value.25

Cost Element StructuringThe concept of cost is central to the process of making policy prescriptions, becausethe pursuit of one objective almost always requires that we sacrifice another.Opportunity costs, as we have seen, are the benefits that could otherwise be obtainedby investing resources in the attainment of some other objective. In other words, costsare benefits forgone or lost, including net benefits (total benefits minus total costs).

Cost element structuring is a procedure for classifying and describing all coststhat will be incurred by establishing and operating a program.26 The cost elementstructure is a list of functions, equipment, and supplies that require the expenditureof resources. The purpose of cost element structuring is to create a list of functions,equipment, and supplies that is exhaustive and mutually exclusive. If the list isexhaustive, no important costs will be overlooked. If the list contains mutuallyexclusive items, no costs will be double counted.

25This is true because prices are only one kind of value.26See H. G. Massey, David Novick, and R. E. Peterson, Cost Measurement: Tools and Methodology forCost-Effectiveness Analysis (Santa Monica, CA: RAND Corporation, February 1972).


A cost element structure contains two main divisions: primary (direct) andsecondary (indirect) costs. Primary costs are subdivided into three categories:one-time fixed costs, investment costs, and recurrent (operating and maintenance)costs.27 A typical cost element structure is illustrated in Table 7.

INFORMATIONGiven that the low-cost home insulation program will result in efficiency improvements of $50 million.

WARRANTBecause net efficiency improvement is synonymous with a social state in which at least one person is better off and no one is worse off (Pareto Criterion).

BACKINGSince in principle thewinners can compensate the losers (Kaldor–Hicks Criterion).

OBJECTIONBut the program does not produce a socialstate in which those who are worst off aremade better off (Rawls Criterion)..

REBUTTALHowever, the program still savesenergy, and this is essential to reduce global warming.

QUALIFIER IThis is probably the right thing to do.

QUALIFIER IIBut consideringthe strong objectionand strong rebuttal,the program is notas beneficial as itfirst seemed.

CLAIMThe government should adopt the low-cost home insulation program.

27Poister, Public Program Analysis, pp. 386–87; and H. P. Hatry, R. E. Winnie, and D. M. Fisk, PracticalProgram Evaluation for State and Local Government (Washington, DC: Urban Institute, 1973).

FIGURE 7Value-critical debate.




TABLE 7

Cost Element Structure

I. Primary (direct) costs1. One-time fixed costs

ResearchPlanningDevelopment, testing, and evaluation

2. Investment costsLandBuilding and facilitiesEquipment and vehiclesInitial training

3. Recurring (operating and maintenance) costsSalaries, wages, and fringe benefitsMaintenance of grounds, vehicles, and equipmentRecurrent trainingDirect payments to clientsPayments for extended support servicesMiscellaneous materials, supplies, and services

II. Secondary (indirect) costs1. Costs to other agencies and third parties2. Environmental degradation3. Disruption of social institutions4. Other

Cost EstimationCost estimation provides information about the dollar value of items in a costelement structure. The cost element structure indicates what elements should bemeasured; cost estimation actually measures them. A cost-estimating relationship isan explicit measure of the relationship between the quantity of functions, materials,or personnel and their costs. The simplest type of cost-estimating relationship is costper unit, for example, the cost per acre of land, per square foot of facilities, or perworker in a given job category. More complex cost-estimating relationships involvethe use of linear regression and interval estimation. For example, linear regressionmay be used to provide an interval estimate of vehicle maintenance costs, usingmiles driven as the independent variable.

Cost-estimating relationships are central to the process of making prescriptions,because explicit measures of the relationship between the quantity of functions,materials, or personnel and their costs permits analysts to compare two or morepolicy alternatives in a consistent manner. Cost-estimating relationships are usedto construct cost models that represent some or all of the costs that will be requiredto initiate and maintain a program. A simplified cost model for initial investment isillustrated in Figure 8.


Shadow PricingShadow pricing is a procedure for making subjective judgments about the monetary valueof benefits and costs when market prices are unreliable or unavailable. Market pricesmight be distorted (i.e., not represent the actual social value of costs and benefits) for anumber of reasons. These reasons include unfair competition, monopolistic or oligopolis-tic practices, and government price support programs. In such cases, upward (or down-ward) adjustments can be made in the actual market price of a good or service. In othercases, market prices are simply unavailable. This is often the case with certain collectivegoods (e.g., clean air or a sense of community) that we discussed earlier as intangibles.Analysts can use several procedures to estimate the price of such intangibles.28

Number ofAcres

of Land

UnitCost

InitialLandCosts

� �

Number ofSquare Feetof Buildings

UnitCost

InitialBuildingCosts

� �

Number ofVehicles

UnitCost

InitialVehicleCosts

� �

Number ofPieces of

Equipment

UnitCost

InitialEquipment

Costs

� �

Number ofTrainees

PerCapitaCost

InitialTrainingCosts

� �

�

�

�

�

TotalInitial

Investment

FIGURE 8Simplified partial cost model for total initial investment.

28Poister, Public Program Analysis, pp. 417–19.




1. Comparable prices. The analyst uses prices for comparable or similar itemsin the market. For example, in estimating the economic benefits of time savedby the construction of a mass rapid transit system, the analyst might use areawage rates as a measure of the monetary value of time saved by using the newtransportation facilities. This equation of work time and leisure time is basedon the questionable assumption that time saved in travel could have been usedto earn income.

2. Consumer choice. The analyst estimates the value of intangibles (e.g., traveltime) by observing consumer behavior when individuals are forced to choosebetween a given intangible and money. For example, estimates of the valueof time can be made by observing actual choices between low-cost andtime-consuming versus high-cost and time-reducing modes of travel. This mayhelp determine how much consumers are willing to pay for reduced traveltime, although it assumes that consumers have complete information aboutthe relation between time and costs and that both alternatives are equal in allrespects, except time and cost.

3. Derived demand. The value of intangibles for which there are no marketprices (e.g., satisfaction with government parks and recreation areas that chargeno fees) can be estimated on the basis of the indirect costs paid by visitors.A demand curve (i.e., a curve that displays the various price-quantity combina-tions at which consumers will use a service) can be derived from the costs paidby consumers in traveling to the park. These indirect costs are taken to beactual prices paid for the use of parks and recreation areas. This procedureassumes that the sole purpose of the travel is to use the park or recreation area.

4. Survey analysis. The analyst surveys citizens by interviewing them or askingthem to complete mailed questionnaires. Respondents indicate the level ofcosts at which they would be willing to pay for a given service (e.g., bustransportation). One weakness of this procedure is the so-called free-rider prob-lem; that is, consumers will claim that they are not willing to pay for a servicein the hope that they will benefit from the service while others pay for it.

5. Cost of compensation. The analyst estimates the value of intangibles—particularly those that occur in the form of unwanted outside costs (negativeexternalities)—by obtaining prices for actions required to correct them. Forexample, the costs of environmental damage might be estimated by basingdollar values on the prices of programs for water purification, noise abatement,or reforestation. Similarly, the benefits of an antipollution enforcement programmight be estimated on the basis of medical costs that will be avoided becausepeople contract less lung cancer, emphysema, and other chronic diseases.

Constraint MappingConstraint mapping is a procedure for identifying and classifying limitations andobstacles that stand in the way of achieving policy and program objectives.Generally, constraints fall into six categories:

1. Physical constraints. The attainment of objectives may be limited by thestate of knowledge or technology development. For example, the reduction of


pollution through the use of solar energy is constrained by the present lowlevel of solar technology development.

2. Legal constraints. Public law, property rights, and agency regulations oftenlimit attempts to achieve objectives. For example, social programs designedto redistribute resources to the poor are frequently constrained by reportingrequirements.

3. Organizational constraints. The organizational structure and processesavailable to implement programs may limit efforts to achieve objectives. Forexample, excessive centralization, poor management, and low morale limit theeffectiveness and efficiency of public programs.29

4. Political constraints. Political opposition may impose severe limitationson the implementation as well as the initial acceptance of programs. Suchopposition is reflected in organizational inertia and tendencies to avoidproblems by practicing incremental decision making. For example, problemssuch as universal health care or take years to be placed on the formal agendaof governments.30

5. Distributional constraints. Public programs designed to provide socialservices efficiently are often limited by the need to ensure that benefits andcosts are equitably distributed among different groups. Programs that achievethe highest net efficiency benefits, as we have seen, are frequently those thatproduce the least social equity, and vice versa.

6. Budgetary constraints. Government budgets are limited, thus requiring thatobjectives be considered in light of scarce resources. Fixed budgets create type Iproblems, forcing analysts to consider alternatives that maximize effectivenesswithin the limits of available resources.

An effective way to identify and classify constraints is to construct a constraintstree, which is a graphic display of limitations and obstacles that stand in the way ofachieving objectives. A constraints tree involves the superimposition of constraintson an objectives tree (see Figure 6). A simplified constraints tree for national energypolicy is illustrated in Figure 9.

Cost InternalizationCost internalization is a procedure for incorporating all relevant outside costs(externalities) into an internal cost element structure. Costs as well as benefitscan be internalized by considering significant positive and negative spillovers ofpublic policies and programs. When costs are fully internalized, there are noexternalities because, by definition, all costs are internal ones. This means that avariety of important external costs—for example, costs of pollution, environmen-tal degradation, or social dislocation—are explicitly built into the process ofprescription.

29For a general treatment of these organizational constraints, see Vincent Ostrom, The IntellectualCrisis in American Public Administration (University: University of Alabama Press, 1974).30See, for example, Roger W. Cobb and Charles D. Elder, Participation in American Politics: TheDynamics of Agenda-Building (Boston, MA: Allyn and Bacon, 1972).




31Hinrichs and Taylor, Systematic Analysis, pp. 18–19.

DEVELOPNATIONALENERGYPOLICY

DoNothing

UseCoal

ProduceMore

Energy

RedistributeEnergy

ConserveEnergy

PollutionStandards

UseSolarPower

InefficientTechnology

UseNuclearPower

PoliticalOpposition

Shift10 Percent

to Poor

Costs ofAdministration

RaisePrices

ConsumerOpposition

InsulateHomes

LimitedBudget

Cut GasConsumption

Lose Next Election

FIGURE 9Constraints map for national energy policy.

In general, policy analysts should pay particular attention to four kinds ofspillovers or externalities:31

1. Production-to-production spillovers. The products of a program servingone target group or jurisdiction may affect (positively or negatively) theproducts of another. For example, successful alcoholism and drug-treatment


programs may result in a decrease of law enforcement activities in thesurrounding community.

2. Production-to-consumption spillovers. The products of a particular programmay affect the quality and quantity of goods consumed by members withinanother target group or jurisdiction. For example, publicly funded highwayprojects may displace residents from their homes or change the demand forgoods and services in neighboring areas.

3. Consumption-to-consumption spillovers. The consumption activities ofpublic programs in one area may affect consumption patterns within adja-cent target groups or jurisdictions. For example, the construction of a largegovernment office facility may make it impossible for local citizens to findadequate parking.

4. Consumption-to-production spillovers. The consumption activities of publicprograms in one area may affect the production activities of public and privateprograms in adjacent areas. For example, the construction of a public housingproject may improve the market for local businesses.

The importance of cost internalization is evident in Table 8, which shows total,outside, and inside costs for three programs designed to provide maternal care. If allthree programs are equally effective, an analyst at the state level who has not inter-nalized federal cost sharing will prescribe program I, because it satisfies the least-cost criterion. If federal costs are internalized, program III will be prescribed, be-cause its total costs are the least. This example calls attention to the relative natureof internal and external costs, because what is an externality to one party is not toanother.

DiscountingDiscounting, as we saw above in the description of steps in conducting a cost-benefit analysis, is a procedure for estimating the present value of costs and benefits

TABLE 8

Internal, External, and Total Costs of Maternal Care

Program

Type of CostMaternal and Infant

Care Project INeighborhood Health

Center II Private Physician III

Cost to state(internal)

$31 10% $96 48.3% $85 48.3%

Cost to federalgovernment (external)

282 90 103 51.7 90 51.7

Total cost $313 100% $199 100% $175 100%

Source: Adapted from M. R. Burt, Policy Analysis: Introduction and Applications to Health Programs (Washington, DC:Information Resources Press, 1974), p. 35.




that will be realized in the future. Discounting is a way to take into account theeffects of time when making policy prescriptions. Time is important because futurecosts and benefits have less value than present ones. A dollar spent today is morevaluable than a dollar spent one or two years from now, since today’s dollar canbe invested so that it might generate $1.05 one year from now (at 5% interest).Apart from interest earnings, many people prefer to consume now rather than later,since the future is uncertain and it is hard to delay the consumption that producesimmediate satisfaction.

Many policies and programs generate different levels of costs and benefitsthat flow forward into time. These flows of costs and benefits, called cost (orbenefit) streams, are unevenly distributed in time. For this reason, each cost andbenefit stream must be discounted to its present value. For example, two healthprograms with equal effectiveness and identical total costs over a five-year periodwould appear to be equally prescribable. Yet in one of these programs (programI), costs are heavily concentrated in the first year; in the other (program II),they are concentrated in the last year. If we assume that the dollar is losing valueat the rate of 10 percent per year, then program II achieves the same level of effectiveness at less cost. If we do not take time and the changing value of money into account, there is no basis for distinguishing the two programs(Figure 10).

The present value of a future benefit or cost is found by using a discount factor,which expresses the amount by which the value of future benefits and costs shouldbe decreased to reflect the fact that present dollars have more value than futureones. Once we know the discount factor, we simply multiply this factor by theknown dollar value of a future benefit or cost. This was done in Figure 8. Given acertain discount rate, that is, the rate at which the value of future benefits and costsshould be reduced, the discount factor may be calculated by knowing the number of

40

30

20

10

00 1 2 3 4 5

Program I

Program II

Year

(a) Undiscounted

Cos

ts (i

n $

mill

ions

)

40

30

20

10

00 1 2 3 4 5

Program I

Program II

Year

(b) Discounted

Cos

ts (i

n $

mill

ions

)

FIGURE 10Comparison of discounted and undiscounted costs cumulated for twoprograms with equal effectiveness.


years over which future benefits and costs must be discounted. The formula for thediscount factor is

where r is the discount rate and n is the number of years over which benefitsor costs are discounted.32 For example, to calculate the discount factor (DF)for a discount rate of 10 percent at five years, we make the followingcomputations:

If we want to discount the value of $100 in costs incurred in the fifth year,we simply multiply this cost by the discount factor, that is, $100 � 0.621 � $62.10.In other words, the present value of $100 in costs incurred in the fifth year at adiscount rate of 10 percent is $62.10. Present value is the dollar value of future costsor benefits that have been multiplied by the appropriate discount factor. Theformula for the present value of a stream of costs or benefits is

where FV designates the future value of costs or benefits, DF is the discount factor,and the symbol ∑ (sigma) tells us to sum or add all the products of costs or benefitstimes the discount factor for all years in the cost (or benefit) stream.33

The calculation of the present value of a cost stream is illustrated in Table 9.Observe that the future value of all undiscounted costs is $180 million, whereas thepresent value of the discounted cost stream is $152.75 million. Assume that thepresent value of the benefit stream is $175 million. If we were using the net benefitcriterion to make a prescription (i.e., net benefits must be greater than zero), the

PV = ©1FV # DF2

= 0.621

= 1

11.612

= 1

11 + 0.1025 =

1

11.125

DF = 1

11 + r2n

DF = 1

11 + r2n

33Another formula for present value, which gives identical results, is PV � FV/(1 � r)n, where FV isfuture costs or benefits, r is the discount rate, and n is the number of years over which costs or benefitsmust be discounted.

32The discount factor assumes that future dollar values in a cost (or benefit) stream are unequal(e.g., heavy costs at the beginning of a period decrease over time). If the future dollar values in a cost (orbenefit) stream are identical from year to year (e.g., $1 million in costs every year for five years), it is cus-tomary to use a shortcut, called an annuity factor (annuity means annually, or regularly over a number

of years). The formula for the annuity factor is Stated more concisely, AF � 1 � DF/r.

AF = 1 - a 1

1 + r bn/r.




undiscounted costs would result in a rejection of the program ($175 � 180 � �$5).By contrast, when we discount the cost stream, our prescription would be positive,because net benefits are $22.5 million ($175 � 152.75 � $22.5).

Once we know the appropriate discount rate, it is a simple matter to calculatethe discount factor and the present value of a cost (or benefit) stream. The problemis that the choice of a discount rate depends on judgments that reflect the ethicalvalues of analysts, policy makers, or particular social groups. For example, thehigher the rate at which we discount future benefits, the more difficult it is to obtainnet benefits and benefit-cost ratios that justify public programs. Uniformly highdiscount rates lead to a minimal role for government investments and a maximalone for private ones. On the other hand, uniformly low discount rates encourage anexpanded role for government in society. For this reason, it is important to recognizecompeting bases for selecting a discount rate:

1. Private discount rates. The selection of a discount rate is based onjudgments about the appropriate rate of interest charged for borrowingmoney in the private sector. The argument for using private rates is thatpublic investments are made with the tax monies of citizens who could haveinvested in the private sector. The private rate of discount thereforemeasures the benefits that would have been obtained if funds had been leftin the private sector. These are the opportunity costs of public (versusprivate) investment. The arguments against private rates are that privaterates differ widely (from 3% to 20%) because of market distortions,private rates do not reflect the external social costs of private investment(e.g., pollution), and private rates reflect narrow individual and grouppreferences and not those of society as a whole.

2. Social discount rates. The selection of a discount rate is based on judgmentsabout the social time preference of society as a whole. The social time preferencerefers to the collective value that society attaches to benefits or costs realized insome future time period. This collective value is not simply the sum of individualpreferences, because it reflects a collective sense of what is valuable for thecommunity as a whole. The social rate of discount, that is, the rate at which

TABLE 9

Calculation of Present Value of Cost Stream at 10 Percent Discount Rateover Five Years (in millions of dollars)

YearFuture Value

(FV)DiscountRate (r)

Number ofYears (n)

Discount Factor(DF) [1/(1 � r)n]

Present Value(PV) [FV.DF]

2001 $100 0.10 1 0.909 $90.902002 50 0.10 2 0.826 41.302003 10 0.10 3 0.751 7.512004 10 0.10 4 0.683 6.832005 10 0.10 5 0.621 6.21

$180.0 $152.75


future collective benefits and costs should be discounted, is generally lower thanthe private rate. The arguments for a social rate of discount are that the socialrate compensates for the narrowness and short-sightedness of individualpreferences; it takes into account external social costs of private investment(e.g., the depletion of finite natural resources); and it reflects a concern with thesecurity, health, and welfare of future generations. Arguments against the socialrate of discount are that it is economically inefficient or “irrational” because itshould be higher than the private rate. Taking a number of these considerationsinto account, Moore and colleagues propose a social rate of discount between3.0 and 4.0 percent.34 In the period following the recession of 2008, theappropriate rate lies near the lower bound of this interval.

3. Government discount rates. A discount rate is selected on the basis of thecurrent costs of government borrowing. The problem is that the rate at whichfederal, state, and local governments may borrow money varies considerably.The Office of Management and Budget (OMB) has advocated varying discountrates for government programs, but the standard rate does not reflect theopportunity cost of investments when lower interest rates are available withina given state, region, or community.

Conflicts surrounding the choice of a discount rate cannot be resolved bypretending that market prices will produce value consensus. The choice of a discountrate is closely connected to competing views of the proper role of government insociety and to the different forms of rationality discussed earlier. Under these circum-stances, the analyst must examine the sensitivity of net benefits or benefit-cost ratiosto alternative discount rates and the assumptions that justify their selection.Alternatively, the analyst can calculate the internal rate of return on investment, aprocedure used to prescribe an alternative when the rate of undiscounted benefitsreturned for each discounted dollar of costs exceeds the actual rate at whichinvestment funds may be borrowed. Here, as elsewhere, it is essential to examine theunderlying assumptions and ethical implications of choice.

Sensitivity AnalysisSensitivity analysis is a procedure for examining measures of cost and benefits underdifferent assumptions. In comparing two or more alternatives, considerableuncertainty almost always exists about the expected outcomes of a policy, eventhough a single overall measure of costs and benefits (e.g., net benefits) may havebeen calculated. In such circumstances, which are often characterized as a determin-istic (as distinguished from probabilistic) choice, we can introduce alternativeassumptions about future costs—for example, assumptions about the likelihood ofhigh, medium, and low costs—and compute net benefits under each of these assump-tions. The point is to see how “sensitive” the results are to different assumptions.

34Mark A. Moore, Anthony E. Boardman, Aidan R. Vining, David L. Weimer, and David H. Greenberg,“‘Just Give Me a Number!’ Practical Values for the Social Discount Rate,” Journal of Policy Analysisand Management 23, 4 (2004): 789–812.




For example, in comparing two personnel training programs, we may examinethe sensitivity of transportation costs to potential changes in the price of gasoline.Here we may introduce assumptions that the price of gasoline will increase by10 percent, 20 percent, and 30 percent over the life of the program. As program I issituated in a semirural area, trainees must drive a longer distance to reach thetraining site than trainees in program II, which is located in the city. A markedincrease in gasoline prices could result in prescribing the more costly of the twoprograms. An example of this kind of sensitivity analysis is provided in Table 10.Another type of sensitivity analysis involves the use of regression analysis to makeinterval estimates of the values of costs or benefits—in effect, “best” case and“worst” case scenarios.

A Fortiori AnalysisA fortiori analysis is a procedure used to compare two or more alternatives byresolving uncertainties in favor of an intuitively favored alternative that appearsafter preliminary analysis to be weaker than a competing alternative. If the strongeralternative still dominates the intuitively favored one, there is an even stronger casefor its selection. Note that a fortiori means “with even greater strength.”

Plausibility AnalysisPlausibility analysis is a procedure for testing a prescription against rival claims.Plausibility analysis—for which plausibility is understood as a subjective probabilitybased on prior experience—may be presented in the form of a policy debate ordiscourse. For example, there are several classes of rival claims, each of which is athreat to the plausibility of prescriptions based on cost-benefit or cost-effectivenessanalysis. These threats to the plausibility of policy prescriptions—which areanalogous to rival hypotheses or “threats to the validity” of designative claims

TABLE 10

Sensitivity Analysis of Gasoline Prices on Costs of Training Programs

Price Increase

Program 10% 20% 30%

I. Semirural

Total costs $4400 $4800 $5200Trainees $5000 $5000 $5000Costs per trainee $ 0.88 $ 0.96 $ 1.04

II. UrbanTotal costs $4500 $4800 $5100Trainees $5000 $5000 $5000Costs per trainee $ 0.90 $ 0.96 $ 1.02


about the causal relevance of policies to specified outcomes—are illustrated inFigure 11 and discussed next.

� Invalidity. A policy prescription is based on an invalid assumption about thecausal relation between a policy and its outcomes. For example, debates aboutthe efficacy of the 55 mph speed limit (National Maximum Speed Law of 1973)have focused on the extent to which enforcing a maximum speed on all intercityhighways is responsible for the observed decline in traffic fatalities after 1973.The claim that the new maximum speed (55 mph) was responsible for the declineof 9,800 fatalities between 1973 and 1975 has been challenged on grounds thatthe decline was due to the new compressed range or standard deviation of speeds;improvements in automobile and highway safety; the interaction of maximum

The 55 MPH Speed Limitshould be retained.

CLAIM

And between 1973 and 1975there was a decline of 9,800fatalities at a cost of $21,200per fatality averted. Netbenefits are $2.3 billion.

INFORMATIONSo, certainlyQUALIFIER I

Because the speed limit wasassociated with and precededthe decline in fatalities.

BACKINGBut probably not,considering thethreats to plausibility.

QUALIFIER II

- Invalidity - Inefficiency- Ineffectiveness - Exclusion- Unresponsiveness - Illegality- Infeasibility - Inequity- Inappropriateness - Misformulation

THREATS TO PLAUSIBILITY

However, these threatsare only possible and notplausible.

REBUTTALS

Since the 55 MPH speedlimit caused the declinein fatalities.

WARRANT

FIGURE 11Threats to the plausibility of claims about the benefits of the 55 mph speed limit.




speed with population density; and the effects of recession, unemployment, anddeclining gasoline prices on miles driven and, consequently, fatality rates.35

� Inefficiency. Estimates of the net efficiency benefits of the 55 mph speed limitvary markedly, depending on the value attached to human lives and the costsof time lost by driving at 55 mph rather than 65 mph. One estimate of netefficiency benefits is $2.3 billion; another is minus $3.4 billion.36

� Ineffectiveness. Estimates of the cost-effectiveness of the 55 mph speed limitalso vary markedly. The costs per fatality averted range from approximately$1,300 to $21,000, depending on assumptions about what should be includedas costs and benefits.37

� Exclusion. The exclusion of legitimate costs and benefits will produceimplausibly high or low net efficiency benefits. For example, costs of time lostby driving at 55 mph or benefits in the form of the monetary value of humanlives may be excluded.

� Unresponsiveness. The costs of time and other resources are often based onassumptions that individuals would be willing to pay the average wage rateto engage in a time-saving activity. The average wage rate is typicallyunresponsive to the price individuals are actually willing to pay. For example,approximately half of drivers are engaged in recreational activities, notcommerce or business. The average wage rate, which markedly increases theestimated costs of the 55 mph speed limit, is therefore unresponsive to theperceived costs of driving at 55 mph.

� Illegality. In some cases, the legality of a prescription may be challenged. It isnot appropriate to count as costs (or benefits) income earned through illegality,fraud, or unlawful discrimination. In cases involving the comparable worth ofjobs held by men, women, and minorities, illegality is a major factor. In thecase of the 55 mph speed limit, it is not.

� Unfeasibility. Policy prescriptions can be challenged on grounds that they arenot feasible because of political, budgetary, and administrative constraints.The implementation of the 55 mph speed limit varies across states because ofdiffering capacities for implementing or enforcing speed laws. Implementationcapacity is generally greater in states with high population density.

� Inequity. Prescriptions may be challenged on ethical grounds involvingalternative conceptions of social equity and justice. The group that pays adisproportionately high cost in terms of lives lost is younger drivers. The medianage of victims of traffic fatalities is approximately thirty years, which means that

35See Duncan MacRae Jr. and James Wilde, Policy Analysis for Public Decisions (North Scituate, MA:Duxbury Press, 1979), pp. 133–52; Edward R. Tufte, Data Analysis for Politics and Policy (EnglewoodCliffs, NJ: Prentice Hall, 1974), pp. 5–18; Charles T. Clotfelter and John C. Hahn, “Assessing theNational 55 mph Speed Limit,” Policy Sciences 9 (1978): 281–94; and Charles A. Lave and Lester B.Lave, “Barriers to Increasing Highway Safety,” in Challenging the Old Order: Toward New Directionsin Traffic Safety Theory, ed. J. Peter Rothe (New Brunswick, NJ: Transaction Books, 1990), pp. 77–94.The author has tested the effects of unemployment and gasoline prices on mileage and fatality rates,explaining more than 90 percent of the variance in mileage death rates.36See Grover Starling, Strategies for Policy Making (Chicago: Dorsey Press, 1988), pp. 407–10; andGuess and Farnham, Cases in Public Policy Analysis, pp. 177–203.37Starling, Strategies for Policy Making, pp. 409–11; see also references in footnote 33.


half of traffic deaths occur in those below thirty. The Rawlsian conception offairness—that is, those worst off should benefit—may be used to challengepolicies designed to reduce fatalities through speeding laws rather than througheducational and driver training programs that target younger drivers.38

� Inappropriateness. Prescriptions based on estimates of the value of humanlives sometimes employ discounted lifetime earnings as a measure of value.Efforts to establish the cost (discounted or undiscounted) of a human life maybe challenged on grounds that it is inappropriate to calculate a price forhuman lives, which are not commodities on an open market.39

� Misformulation. A standing challenge to prescriptions based on cost-benefitanalysis is that the problem has been misformulated. In the case of the 55 mphspeed limit, saving fuel during the oil crisis of 1970–73 was the original prob-lem for which the National Maximum Speed Law was a prescribed solution.The definition of the problem shifted to averting fatalities after the oil crisispassed. The existing formulation of the problem (averting traffic fatalities) maybe challenged on grounds that the problem should be formulated as one ofsaving a nonrenewable resource; mitigating the emission of pollutants; andaverting fatalities by a significant increase in taxes on gasoline, which inconstant dollars costs less per gallon in 1990 than in 1974.

These threats to the plausibility of policy prescriptions do not apply solely tocost-benefit and cost-effectiveness analyses. Because all policy prescriptions arebased on causal as well as ethical premises, these threats to plausibility arerelevant to almost any policy that seeks to achieve reforms through the regulation,allocation, or reallocation of scarce resources.40

The policy-analytic method of prescription involves many uncertainties. Oneof these has to do with the role of values and ethics in policy analysis. The purposeof a policy prescription is not simply to forecast or predict some future outcome orto monitor its past or present effects—but to advocate a course of action whoselikely consequences are valuable because they are right in and of themselves, orbecause they promote justice, equality, efficiency, democracy, liberty, or security.However, there are practically significant difficulties in using economic theory andthe tools of economic analysis to justify such claims about what is valuable formembers of a community. For this reason, uncertainties about values may be treatedas a matter for reasoned ethical argument and debate.

A closely related source of uncertainty stems from incomplete knowledge aboutthe effects of policies. Even if there were a consensus on important social, economic,and political values, we still would not know which policies work best under differentconditions. Some of this uncertainty is a result of incomplete information about therange of costs and benefits that should be considered in a reasonably thoroughanalysis. Another important source of uncertainty stems from measures of varyingreliability and validity and fallible judgments about shadow prices and discount rates.

38On this question, see, especially, Rothe, Challenging the Old Order.39See the discussion in Guess and Farnham, Cases in Public Policy Analysis.40See Dunn, “Policies as Arguments”; and Frank Fischer, Politics, Values, and Public Policy: TheProblem of Methodology (Boulder, CO: Westview Press, 1980).




CHAPTER SUMMARY

REVIEW QUESTIONS1. The late Milton Friedman, Nobel Laureate

in economics from the University of Chicago,argued for a positive (rather than normative)economics that focuses on the study of the factual premises underlying policychoices:

Differences about economic policy amongdisinterested citizens derive predominantlyfrom different predictions about the eco-nomic consequences of taking action—differences that in principle can be eliminatedby the progress of positive economics—rather than from fundamental differences inbasic values, differences about which mencan only fight. [Essays in Positive Economics(Chicago: University of Chicago Press,1953), p. 5]

Write a short essay on the strengths and weak-nesses of Friedman’s position. In your answer,refer to contrasts between descriptive andnormative analysis.

2. “If individuals can order their preferences in atransitive fashion, it should be possible to usemajority rule to obtain a collective preferenceranking that is also transitive.” Discuss.

3. Is rational choice possible? Write an essay onthis question. In your answer, distinguishdifferent types of rationality.

4. How are different types of rationality relatedto different criteria for prescription?

5. Indicate how you would measure the effective-ness of efforts to achieve the following goals:

Close the gap between municipal revenuesand expenditures

Deter the commission of serious crimes inurban areas

Enhance the national security of the country

Win the war on terror(ism)

Improve the quality of life of citizens

Reduce national unemploymentIncrease the income of families living inpoverty

This chapter has provided an overview of the na-ture and role of prescription in policy analysis,compared and contrasted two major approachesto prescription, and described techniques usedin conjunction with these approaches. As wehave seen, policy prescriptions answer thequestion: What should be done? For this reason,policy prescriptions require an approachthat is normative, and not one that is merelyempirical or merely evaluative. All policyprescriptions involve claims about right action,rather than claims that are simply designative orevaluative.

Two major approaches to prescription inpolicy analysis are cost-benefit analysis andcost-effectiveness analysis. Although bothapproaches seek to measure costs and benefits

to society, only cost-benefit analysis measuresmonetary benefits as well as costs in a commonunit of value. Costs and benefits are of severaltypes: inside versus outside, tangible versusintangible, primary versus secondary, and realversus pecuniary. There are several tasksin cost-benefit analysis: specification of objec-tives; identification of alternatives; collection,analysis, and interpretation of information;specification of target groups and beneficiaries;identification of types of costs and benefits; dis-counting of costs and benefits; and specificationof criteria for prescription. The criterion mostfrequently employed in traditional cost-benefitanalysis is net efficiency improvement; contem-porary cost-benefit analysis supplements thiscriterion with that of net redistribution benefits.


6. Return to Question 5. Indicate how you wouldmeasure the efficiency of efforts to achievethese goals.

7. Many economists argue that equality and effi-ciency are competing objectives. For this reason,the relation between equality and efficiency isone that requires a trade-off whereby gains inequality are sacrificed for gains in efficiency, andvice versa [see Arthur M. Okun, Equality andEfficiency: The Big Trade-Off (Washington,DC: Brookings Institution, 1975)]. Other econ-omists [see Jaroslav Vanek, The ParticipatoryEconomy (Ithaca, NY: Cornell University Press,1970)] and political scientists [see CarolePateman, Participation and Democratic Theory(Cambridge, MA: Cambridge University Press,1970)] argue that the trade-off between equalityand efficiency is a consequence of the limitedforms of democratic participation in politics andwork. Equality and efficiency, they argue, donot have to be “traded” when politics andeconomics are fully democratic. Why would thisbe true? What does it suggest about the appropri-ateness of economic concepts such as opportunitycosts and net efficiency versus redistributionalbenefits?

8. Return to Figure 2 and consider the followingcriteria for prescription:a. Maximize effectiveness at least cost [Note:

Be careful—this is a tricky question].b. Maximize effectiveness at a fixed cost of

$10,000.c. Minimize costs at a fixed-effectiveness level

of 4,000 units of service.d. Achieve a fixed-effectiveness level of 6,000

units of service at a fixed cost of $20,000.e. Assuming that each unit of service has a

market price of $10, maximize net benefits.

f. Again assuming that each unit of service hasa market price of $10, maximize the ratio ofbenefits to costs.

Indicate which of the two main programs(program I and program II) should be selectedunder each of these criteria, and describe theconditions under which each criterion may bean adequate measure of the achievement ofobjectives.

9. The estimation of risk and uncertainty is anessential aspect of the policy-analytic method-ology of prescription. Although it seems obvi-ous that the best way to obtain valid estimatesof uncertainty is to use the most highly quali-fied experts, there is evidence that “people whoknow the most about various topics are notconsistently the best at expressing the likeli-hood that they are correct” [Baruch Fischoff,“Cost-Benefit Analysis and the Art ofMotorcycle Maintenance,” Policy Sciences 8(1977): 184]. Discuss.

10. Use procedures of value clarification and valuecritique to analyze value premises that underliethe following claim: Increased energy produc-tion is essential to the continued growth of theeconomy. The government should thereforemake massive investments in research anddevelopment on new energy technologies.Treat the first sentence as information (I) andthe second as the claim (C).

12. Calculate the discount factor (DF) at ten yearsfor a 3, 4, and 10 percent discount rate.

13. Calculate the present value (PV) of the follow-ing cost and benefit streams (in millions ofdollars) for two health programs. Use adiscount rate of 4 percent. Which programhas the higher present net efficiencybenefits? Why?

Year

Program 2000 2005 2010 2015 2020 Total

I Costs $60 $10 $10 $10 $10 $100Benefits 10 20 30 40 50 150

II Costs 20 20 20 20 20 100Benefits 10 20 30 40 50 150




DEMONSTRATION EXERCISERead Case 1 (Opportunity Costs of Saving Lives—The 55 mph Speed Limit) below. Prepare a shortanalysis in which you answer these questions:� What assumptions govern estimates of the value

of time lost driving? Are some assumptions moretenable than others? Why?

� What is the best way to estimate the value oftime? Justify your answer.

� What is the best way to estimate the cost of agallon of gasoline? Justify your answer.

� Driving speeds and miles per gallon estimatesmay be based on official statistics on highway

traffic from the Environmental ProtectionAgency and the Department of Energy or onengineering studies of the efficiency of gasolineengines. Which is the more reliable? Why? Whatare the consequences of using one source ratherthan another?

� What is the value of a life saved? Explain.� Which policy is preferable, the 55 mph speed

limit or the 65 mph limit that was abandoned in1994?

41Adapted from Starling, Strategies for Policy Making, pp. 407–10.

BIBLIOGRAPHYBoardman, Anthony E., David H. Greenberg, Aidan

R. Vining, and David L. Weimer. Cost-BenefitAnalysis: Concepts and Practice. 4th ed. UpperSaddle River, NJ: Pearson Prentice Hall, 2011.

Fischhoff, Baruch. “Cost-Benefit Analysis and theArt of Motorcycle Maintenance,” PolicySciences 8, (1977): 177–202.

Gramlich, Edward M. Benefit-Cost Analysis ofPublic Programs. 2d ed. Englewood Cliffs, NJ:Prentice Hall, 1990.

Haveman, Robert H., and Julius Margolis, eds.Public Expenditure and Policy Analysis. 3d ed.Boston: Houghton Mifflin, 1988.

Miller, Roger Le Roy, Daniel K. Benjamin, andDouglass C. North. The Economics of PublicIssues. 16th ed. New York: Addison-Wesley,2010.

Mishan, Edward J. Cost-Benefit Analysis. NewYork: Praeger, 1976.

Moore, Mark A., Anthony E. Boardman, Aidan R.Vining, David L. Weimer, and David H.Greenberg “ ‘Just Give Me a Number!’ PracticalValues for the Social Discount Rate.” Journal ofPolicy Analysis and Management, 23, 4 (2004):789–812.

Conducting a benefit-cost analysis is not only a technical matter of economic analysis. It is also a matter of identifying, and if necessarychallenging, the assumptions on which benefit-costanalysis is based. This can be seen if we examinethe case of the National Maximum Speed Limit of 1974.

Table 11 describes steps in conducting a benefit-cost analysis and a critique of the assumptionsunderlying the analysis. The case shows, among otherthings, that all steps in conducting a benefit-cost aresensitive to these assumptions. �

CASE 1 OPPORTUNITY COSTS OF SAVING LIVES—THE 55 MPH SPEED LIMIT41


(continued)


TABLE 11

Measuring the Costs and Benefits of the 55 mph Speed Limit:A Critical Appraisal

Steps Critique

Costs

1.The major cost of the National Maximum SpeedLaw (NMSL) was the additional time spentdriving as a result of slower speeds. To calculatethe number of hours spent driving in 1973,divide the total number of vehicle miles traveledon interstate highways by the average highwayspeed (65 mph) and then multiply by theaverage occupancy rate per vehicle, which isapproximately 1.77 persons.

Why use 1973 mileage without any adjustment? The average

growth rate in travel before 1973 was 4 percent. Therefore, the

formula should be

Next, find the number of hours spent driving in1974 by dividing total vehicle miles traveled oninterstate highways by the average highwayspeed in 1974 (58 mph).The NMSL causedsome people to cancel trips and others to findalternative modes of transportation; as a result,time calculations based on 1974 mileage wouldbe an underestimate. Therefore, we should usethe 1973 mileage of 525 million miles.

H = a VM1973

S1974 -

VM1973

S1973 b * R

Using the following formula, where VM is vehiclemiles, S is average speed, R is average occupancyrate, and H is the number of hours lost,

H = a 1.04VM1973

S1974 -

VM1973

S1973 b * R

The number of hours lost driving in 1974, basedon this equation, is estimated to be 1.72 billion.

Using the above formula, the estimated number of hours lost

should be 1.95 billion—not 1.72 billion.

2.To estimate the value of this time, begin with theaverage wage rate for all members of the laborforce in 1974—$5.05.The value of one hour’stravel is not $5.05 per hour because very fewpersons would pay this sum to avoid an hour oftravel. We estimate that the people will pay upto 33 percent of their average hourly wage rateto avoid an hour of commuting. The value oftime spent traveling is therefore about $1.68per hour.

Why take a percentage of the $5.05 figure based on what

commuters would pay to avoid an hour of travel? We should

avoid reducing the value of people’s time for two reasons. First,

the value of time in cost to society is equal to what society will

pay for productive use of that time. Time’s value is not what a

commuter will pay to avoid commuting because commuting has

other benefits, such as solitude for thinking or the advantages of

suburban living. Second, the value of time spent driving for a

trucker is many times the industrial wage rate. Discounting would

greatly underestimate the value of commercial drivers.

3. Application of the cost figure ($1.68) to thetime lost figure (1.72 billion hours) results in anestimated travel cost of $2.89 billion.

Applying the value of one hour’s time to the hours lost as

calculated above (1.95 billion) results in an estimated travel

cost of $9.85 billion.



4.The NMSL also has some enforcement costs.Total enforcement costs for signs, advertising,and patrolling are about $810,000.

Total enforcement cost should be about $12 million—not

$810,000.

a. New signs were posted. Cost estimates from25 states for modification of speed limit signstotaled $707,000; for 50 states, this resultsin an estimated $1.23 million. Spread outover the three-year life of traffic signs, we getan estimate of $410,000.

OK.

b. The federal government engaged in anadvertising campaign encouraging compliance.The Federal Highway Administration’sadvertising budget for 1974 was $2 million.About 10 percent of this, or $200,000, wasspent to encourage compliance with theNMSL. Assume that an additional amountof public service advertising time was donated,for a total of $400,000.

Not OK. The Federal Highway Administration does other

advertising. Public service advertising estimate also seems low.

c. Compliance costs are difficult to estimate.The cost of highway patrols cannot be usedbecause these persons were patrollinghighways before the NMSL. Assume thatstates did not hire additional personnel solelyfor enforcement of the NMSL. Therefore, weassume that enforcement of the NMSL willnot entail any additional costs aboveenforcement of previous speed limits.

Compliance costs pose some problems, but they can be

estimated. In 1973, some 5,711,617 traffic citations jumped by

1,713,636 to over 7.4 million. Each additional traffic citation

includes an opportunity cost to society. If a law enforcement

officer were not issuing traffic tickets, he or she could be solving

other crimes. Assuming that it requires 15 minutes for a law

enforcement officer to issue a speeding ticket, the total cost of law

enforcement is $2.9 million. This figure is based on the average

cost of placing a law enforcement officer on the streets at $6.75

per hour. This figure is clearly an underestimate because it does

not count time lost waiting to catch speeders.

Approximately 10 percent of all speeders will demand a court

hearing. Estimating an average of 30 minutes for each hearing

and an hourly court cost of $45 results in an additional cost to

society of $3.8 million for 171,000 cases. Given the overloaded

court dockets, this opportunity cost may be even higher.

Benefits

1.The most apparent benefit of the NMSL is theamount of gasoline saved. The average gasolineeconomy improves from 14.9 miles per gallon at65 miles per hour to 16.1 at 58 miles per hour.Use this information to estimate the number ofgallons of gasoline saved by traveling at lowerspeeds. Gallons saved will be calculated by thefollowing formula, where VMT is vehicle milestraveled on interstate highways (not allhighways) and MPG is miles per gallon.

Why estimate gasoline saved by comparing 1973 and 1974

miles-per-gallon figures in relation to vehicle miles traveled?

The federal figures for average miles per hour are estimates

based on several assumptions. Given the conflict between

industry estimates, Environmental Protection Agency

estimates, and Energy Department estimates, any miles-

per-hour estimate must be considered unreliable. The number

of vehicle miles traveled is also based on gallons of fuel sold

multiplied by average miles per hour. Hence, this figure is also

subject to error.


= 3.487 billion gallons

G = VM1973

MPG1973 -

VM1973

MPG1974

Studies of the efficiency of gasoline engines show that theeffect of reducing the average speed of free-flow interstatehighways would be to save 2.57 percent of the normal gas used.In 1979, American motorists consumed 106.3 billion gallons ofgasoline. Saving 2.57 percent would total 2.73 billion gallons.

In 1974, the average price of gasoline was 52.8cents per gallon. This market price, however, doesnot reflect the social cost of gasoline, due togovernment price controls on domestic oil. Themarginal (or replacement) cost of crude oil isthe price of foreign oil. Therefore, the price ofgasoline must reflect the higher cost of foreignoil. Use the market price of gasoline in theabsence of price controls, which is about 71.8cents per gallon. This figure yields an estimate of$2.50 billion in benefits through gasoline saved.

Why use the market price? There is no way to determine

whether a marginal gallon of gasoline will be imported or come

from domestic reserves. In addition, the costs and benefits of

the NMSL should not be distorted simply because the U.S.

government does not have a market-oriented energy policy.

In 1974, gasoline cost 52.8 cents per gallon, and therefore, a

gallon of gasoline saved was worth 52.8 cents.

2. A major second-order benefit of the 55 mphlimit was a large drop in traffic fatalities, from55,087 in 1973 to 46,049 in 1974. Part ofthe gain must be attributable to reduction intraffic speeds. Studies by the National SafetyCouncil estimate that up to 59 percent of thedecline in fatalities was the result of the speedlimit. Applying this proportion to the decline infatalities provides an estimated 5,332 livessaved. The consensus of several studies is that a traffic fatality costs $240,000 in 1974dollars. Using this figure, the value of lives saved in 1974 is estimated at $1,279.7 million.

OK.

3.The NMSL also resulted in a reduction ofnonfatal injuries. Use the 59 percent figurefound in the fatality studies. Between 1973 and1974, nonfatal traffic injuries declined by182,626. Applying the estimated percentagesresults in 107,749 injuries avoided. Generally,three levels of injuries are indentified: (1)permanent total disability, (2) permanent partialdisability and permanent disfigurement, and (3)nonpermanent injury. In 1971, the proportion oftraffic injuries that accounted for injuries ineach category was 0.2 percent, 6.5 percent, and93.3 percent, respectively. The NationalHighway Traffic Safety Administrationestimated that in

OK.

(continued)




1971, the average cost of each type of injurywas $260,300, $67,100, and $2,465,respectively. The average injury, therefore, cost$8,745 in 1974 dollars. Applying this figure toour injury estimate results in $942.3 million asthe social benefit of injury reduction.

4.The final benefit of the reduction in propertydamage fell from 25.8 million to 23.1 million.About 50 percent of this reduction was theresult of lower speeds. The NMSL saved 1.3million cases of property damage at an averagecost of $363.Therefore, the total benefit fromproperty damage prevented is $472 million.

OK.

Conclusion

The first estimate of the costs and benefits of theNMSL results in the following figures (in millions):

Using different assumptions, the second estimate of the

costs and benefits of the NMSL is as follows (in millions):

Costs Costs

Time spent traveling $2,890.0 Time spent traveling $9,848.0

Enforcement .8 Enforcement 12.0

$2,890.8 $9,860.0

Benefits Benefits

Gasoline saved $2,500.0 Gasoline save $1,442.0

Lives saved 1.297.7 Lives saved 998.0

Injuries prevented 942.3 Injuries prevented 722.0

Property damage averted 472.0 Property damage adverted 236.0

$5,212.0 $3,398.0

Net benefits: $2,321.2 million Net benefits: –$6,462 million

Benefits to costs ratio: 1.8 Benefits to costs ratio: .345

Source: The steps and data were suggested by Charles T. Clotfelter and John C. Hahn, “Assessing the National 55 m.p.h. SpeedLimit,” Policy Sciences 9 (1978): 281–94.The critical comments are based on Charles A. Lave, “The Costs of Going 55,” Car and

Driver. May 1978, p. 12.


O B J E C T I V E S


Monitoring ObservedPolicy Outcomes

� Distinguish monitoring from otherpolicy-analytic methods.

� Describe the main functions ofmonitoring.

� Distinguish policy outcomes,impacts, processes, and inputs.

� Compare and contrast socialsystems accounting, social

experimentation, social auditing,and research and practice synthesis.

� Define and illustrate threats tointernal and external validity.

� Perform interrupted time-series andcontrol-series analyses to explainpolicy outcomes.

The consequences of policies are never fully known in advance. For this reason,much of the work required to assess the consequences of policies is carried outafter they have been adopted. Hence, it is not that there are too many good ex

ante analyses of potential solutions to problems, but that ex ante analyses have notbeen performed along with ex post analyses of what happens later. In effect, ex anteanalyses are hypotheses about the likely effects of policies: If policy p is taken at timet0, expected outcome O will result at time t � 1. Usually, forecasting suppliesevidence that such policies will work; otherwise they would be blind speculation orpure guesswork. Nevertheless, policy hypotheses must be tested against subsequentexperience.1

1A useful practical framework for implementation analysis is R. Kent Weaver, “But Will It Work?Implementation Analysis to Improve Government Performance,” Issues in Governance Studies 32(2010): 1–7.



Monitoring Observed Policy Outcomes

MONITORING IN POLICY ANALYSISMonitoring permits the production of information about the causes and consequencesof policies. Because monitoring investigates relations between policy-programoperations and their observed outcomes, it is the primary source of information aboutthe success of efforts to implement policies.2 Monitoring goes beyond the normativeprescriptions of economic analysis by establishing factual premises about policies.Although all policies rest on factual and value premises, only monitoring establishesfactual claims retrospectively (ex post).

Monitoring performs several functions including compliance, auditing, accounting,and explanation.

� Compliance. Monitoring helps determine whether the actions of programmanagers are in compliance with norms, values, and standards mandated bylegislatures, regulatory agencies, and professional associations. For example,the Environmental Protection Agency’s Continuous Air Monitoring Program(CAMP) produces information about pollution levels to determine whetherindustries are complying with federal air quality standards.

� Auditing. Monitoring helps discover whether resources and services intendedfor target groups and beneficiaries actually reach them, for example, recipientsof social services or municipalities that qualify for federal and state grants.By monitoring federal revenue sharing, we can determine the extent to whichfunds are reaching local governments.3

� Accounting. Monitoring produces information that is helpful inaccounting for social and economic changes that follow the implementationof policies. For example, the World Bank, the European Union, the UnitedNations, the United States, and other national and international bodiesemploy social and economic indicators to monitor changes in quality of life and well-being as measured by indicators such as average education,percentage of the population living below the poverty line, and averageannual paid vacations.4

� Explanation. Monitoring also yields information that helps explain whythe outcomes of public policies and programs differ. Whereas social indica-tors represent reasoning based on signs or symptoms, field experiments aredesigned to investigate whether policy-related and extra-policy factors arecausally relevant to policy outcomes. The Campbell Collaboration, whichhas spread to more than 40 countries, is an archive of field experiments ineducation, labor, criminal justice, housing, social work, and other areas.

2Monitoring is like a field experiment, where the principal aim is to make valid causal inferences aboutthe effects of policy interventions.3Richard P. Nathan and others, Monitoring Revenue Sharing (Washington, DC: Brookings Institution,1975).4See World Bank, World Development Indicators at http://data.worldbank.org/indicator; AnthonyBarnes Atkinson, Social Indicators: The EU and Social Inclusion (New York: Oxford University Press,2003); Duncan MacRae Jr. Policy Indicators: Links Between Social Science and Public Debate (ChapelHill: University of North Carolina Press, 1985; Office of Management and the Budget, SocialIndicators, 1973 (Washington, DC: U.S. Government Printing Office, 1974).



The Campbell Collaboration and the parallel organization in medicine andpublic health, the Cochrane Collaboration, aim at evidence-based policy andevidence-based medicine based on causally relevant factors that explainwhat works.5

Sources of InformationTo monitor policies, information must be relevant, reliable, and valid. If we want toknow about the consequences of programs designed to provide educational oppor-tunity for disadvantaged children, we need information that not only documentsobstacles to achieving educational opportunity in general, but information thatyields answers about specific factors that policy makers may manipulate to inducegreater opportunity. The first type of information is appropriately termedmacronegative, whereas the latter is best described as micropositive. Informationacquired through monitoring must also be reliable, which means that observationsshould be reasonably precise and dependable. For example, we have known formany years that information on crime is unreliable by a factor of at least 2:5 to 1,that is, some two to three times more crimes are actually committed than arereported to police.6 Finally, we also want to know whether information about policyoutcomes is actually measuring what we think it is, that is, whether it is valid infor-mation. If we are interested in violent crimes, information on crimes in general(which includes auto theft and white-collar crimes) will not be a valid measure ofthe kinds of policy outcomes in which we are interested.7

Information on policy outcomes is regularly collected at various points in time atconsiderable cost to federal, state, and local governments, private research institutes,and universities. Some of this information is general—for example, social, economic,and demographic characteristics of the population as a whole—and some is morespecific because it deals with regions, states, municipalities, and other subpopulations.Information about policy outcomes is published by the U.S. Bureau of the Census, theU.S. Bureau of Labor Statistics, offices and clearinghouses attached to federal agenciesand institutes, nonprofit research centers, and agencies of the European Union and theUnited Nations.

In addition to these sources, federal, state, and municipal agencies produce reportson special programs and projects in education, health, welfare, labor, employment,crime, energy, pollution, foreign affairs, and other areas. Research centers, institutes,

5The preeminent source is Will Shadish, Thomas D. Cook, and Donald T. Campbell, Experimental and Quasi-Experimental Designs for Generalized Causal Inference (Chicago: Rand McNally, 2002). See also William N. Dunn, ed. and contributor, The Experimenting Society: Essays in Honor of Donald T. Campbell (New Brunswick, NJ: Transaction, 1998). Early sources in this tradition are Alice Rivlin, Systematic Thinking for Social Action (Washington, DC: Brookings Institution, 1971); and George W. Fairweather and Louis G. Tornatzky, Experimental Methods for Social Policy Research(Oxford and New York: Pergamon Press, 1977).6The original estimates are in R. H. Beattie, “Criminal Statistics in the United States,” Journal ofCriminal Law, Criminology, and Police Science 51 (May 1960): 49–51.7See Wesley G. Skogan, “The Validity of Official Crime Statistics: An Empirical Investigation,” SocialScience Quarterly 55 (June 1974): 25–38.



and universities also maintain data archives that contain a wide variety of political,social, and economic data, and there is a large stock of books, monographs, articles,and reports written by applied researchers throughout the country. Much of thisinformation can now be accessed through many online information systems, includingJSTOR for academic publications and PolicyFile, a unique resource for U.S. publicpolicy research. PolicyFile contains timely, updated information from more than350 public policy think tanks, nongovernmental organizations, research institutes,university centers, advocacy groups, and other entities. More than 75 public policytopics are covered, from foreign policy to domestic policy.8

When information is not available from existing sources, monitoring maybe carried out by some combination of mailed and online questionnaires, face-to-face interviews, field observations, agency records, and field experiments.9

Because policy analysts rarely conduct original research—given that the fundingand time required for such research is scarce—existing data archives and publishedand unpublished reports are the major sources of information.

Types of Policy OutcomesIn monitoring policy outcomes, we must distinguish between two kinds of conse-quences: outputs and impacts. Policy outputs are the goods, services, and monetaryresources received by target groups and beneficiaries. Per capita welfare expendituresand units of home food service received by the elderly are examples of policy outputs.Policy impacts, by contrast, are changes in behavior or attitudes that result frompolicy outputs. For example, although units of home food service (meals) provided tothe elderly are an appropriate output indicator, the average daily protein intake ofelderly persons is a measure of impact. Similarly, the number of hospital beds per1,000 persons is an indicator of policy output. To monitor policy impact, however,we would have to determine how many members of a particular target group useavailable hospital beds when they are ill.

In monitoring policy outputs and impacts, it is important to distinguish betweentarget groups and beneficiaries. Target groups are individuals, organizations, orcommunities on whom policies and programs are designed to have an effect;beneficiaries are groups for whom the effects of policies are beneficial or valuable.For example, industrial and manufacturing firms are the targets of federal programsadministered by the Occupational Safety and Health Administration (OSHA), butworkers and their families are the beneficiaries. However, today’s target groups andbeneficiaries are not necessarily tomorrow’s, because future generations are affectedin different ways by present-day policies and programs. A good example is the effortto minimize the risk of cancer in adult life by protecting children from harmfulchemicals and pollutants.

8The URL for PolicyFile is http://www.policyfile.com.9A review of applied research methods is beyond the scope of this text. For reference materials, see Miller and Salkind, Handbook of Research Design and Social Measurement 6th ed (2002); ClairSelltiz and others, Research Methods in Social Relations (New York: Holt, Rinehart and Winston,1976); and David Nachmias, Public Policy Evaluation: Approaches and Methods (New York: St.Martin’s Press, 1979).


Types of Policy ActionsTo account satisfactorily for variations in policy outputs and impacts, we must themback to prior policy actions. Generally, policy actions have two major purposes:regulation and allocation. Regulative actions are those designed to ensurecompliance with certain standards or procedures, for example, those of theEnvironmental Protection Agency or the Federal Aeronautics Administration.By contrast, allocative actions are those that require inputs of money, time,personnel, and equipment. Regulative and allocative actions may have consequencesthat are distributive as well as redistributive.10 The regulation of liquor licenses bystate liquor control boards affects the distribution of business opportunities; theallocation of resources to the Social Security Administration results in the distribu-tion of benefits to retired persons. By contrast, the allocation of resources to federalrevenue-sharing programs is designed in part to redistribute revenues from states tomunicipalities. Note, however, that all regulative actions require resource inputs.For example, federal occupational health and safety regulations require largeresource allocations for effective enforcement. Regulative and allocative actions areimplemented by federal, state, and municipal agencies in the form of programs andprojects (Figure 1).

Policy actions may also be further subdivided into policy inputs and policyprocesses. Policy inputs are the resources—time, money, personnel, equipment,supplies—used to produce outputs and impacts. One of the best examples of policyinputs is the program budget, which contains a systematic account of resourcesallocated for program activities and tasks. By contrast, policy processes are theadministrative, organizational, and political activities and attitudes that shape thetransformation of policy inputs into policy outputs and impacts. For example, conflicts


10Theodore Lowi categorizes the impacts of public policies as regulatory, distributive, and redistributive.These categories may be treated as attributes of either actions or outcomes, depending on how stakeholdersperceive policy problems. See Theodore J. Lowi, “American Business, Public Policy Case Studies, andPolitical Theory,” World Politics 16 (1964): 689–90; and Charles O. Jones, An Introduction to the Study ofPublic Policy, 2d ed. (North Scituate, MA: Duxbury Press, 1977), pp. 223–25. In the scheme used here, allregulative action is also distributive or redistributive, no matter what the intent of the regulative action.

Regulative

Allocative Agencies Programs Projects

PREFERREDPOLICIES

FIGURE 1Regulative and allocative actions and their implementation through agencies,programs, and projects



among agency staff and management, dissatisfaction with working conditions, orinflexible decision-making procedures may explain why programs that have the sameresource inputs produce lower levels of outputs and impacts. The important point is todistinguish between inputs and processes, on one hand, and outputs and impacts, onthe other. To fail to do so is like measuring “the number of times a bird flaps its wingswithout any attempt to determine how far the bird has flown.”11 Table 1 illustrates in-puts, processes, outcomes, and impacts in the areas of criminal justice, municipal serv-ices, and social welfare.

Definitions and IndicatorsOur success in obtaining, analyzing, and interpreting data on policy outcomesdepends on our capacity to construct reliable and valid measures. One way toconstruct measures is to specify the variables we are interested in monitoring.A variable is any characteristic of a person, event, or object that takes on differentnumerical values; a constant is a characteristic that does not vary. For example, inthe Congressional Caucus for Women’s Issues, gender is a constant, whereas it is avariable in the U.S. Congress as a whole. Policy variables include level ofeducational opportunity, degree of public safety, and air quality. One difficulty withpublic policy analysis is that frequently we do not have precise definitions of these

11Edward A. Suchman, Evaluative Research: Principles and Practice in Public Service and Social ActionPrograms (New York: Russell Sage Foundation, 1969), p. 61.

TABLE 1

Types of Policy Actions and Policy Outcomes: Inputs, Processes, Outputs, and Impacts inThree Issue Areas

Policy Actions Policy Outcomes

Issue Area Inputs Processes Outputs Impacts

Criminaljustice

Dollar expendituresfor salaries,equipment,maintenance

Illegal arrests aspercentages of totalarrests

Criminals arrestedper 100,000known crimes

Criminals convicted per100,000 knowncrimes

Municipalservices

Dollar expendituresfor sanitationworkers andequipment

Morale amongworkers

Total residencesserved

Cleanliness ofstreets

Socialwelfare

Number of socialworkers

Rapport withwelfare recipients

Welfare cases persocial worker

Standard of living of dependentchildren



and other variables. For this reason, it is useful to think about two kinds ofdefinitions of variables: constitutive and operational. Constitutive definitions givemeaning to variables by using other synonymous words. For example, educationalopportunity may be constitutively defined as “the freedom to choose learningenvironments consistent with one’s abilities.” Such definitions, although essential,provide us with no concrete rules or guidelines for actually monitoring changes ineducational opportunity. In fact, it is impossible to measure educational opportunitydirectly, because constitutive or “dictionary” definitions provide only general linkswith the “real world” of policy making.

We experience policy actions and outcomes indirectly, by using operationaldefinitions of variables. An operational definition gives meaning to a variable byspecifying the operations required to observe and measure it. For example, we cango much beyond our constitutive definition of educational opportunity byspecifying that educational opportunity is “the number of children from familieswith less than $6,000 annual income who attend colleges and universities, asdocumented by census data.” In this case, our definition is operational because itspecifies the operation required to observe and measure educational opportunity.Here we are directed to locate and read those sections of the census on familyincome and educational levels of children. In so doing, we observe educationalopportunity indirectly. With this information in hand, we can proceed to monitorthe impact of public policies.

Operational definitions not only specify procedures required to observe andmeasure something, but they also help specify indicators of input, process, output, andimpact variables. Indicators of variables—for example, average school enrollment, thenumber of drug addicts, or the amount of sulfur dioxide in the air—are observablecharacteristics that are substituted for indirectly observable or unobservable charac-teristics. We do not directly observe job satisfaction, quality of life, or economicprogress.

Many alternative indicators may be used to operationally define the samevariable. This creates problems of interpretation. For example, it may be quitedifficult to know whether one, several, or all of the following indicators areadequate measures of the impact of crime-control policies: arrests per officer,arrests per known crime, the ratio of false to total arrests, number of crimescleared, number of convictions, number of citizens reporting victimization bycriminals, and citizens’ feelings of personal safety. Because the relationship betweenvariables and indicators is complex, it is often desirable to use multiple indicatorsof the same action or outcome variable.12 Sometimes it is possible to construct anindex, that is, a combination of two or more indicators that provides a bettermeasure of actions and outcomes than any one indicator does by itself. Among themany types of indices used in policy analysis are those of cost of living, pollution,

12The general case for multiple indicators may be found in Eugene Webb and others, UnobtrusiveMeasures: Nonreactive Research in the Behavioral Sciences, Rev Ed (Newbury Park, CA: SagePublications, 2000). For applications to criminal justice and other areas, see Roger B. Parks,“Complementary Measures of Police Performance,” in Public Policy Evaluation, ed. Kenneth M. Dolbeare (Beverly Hills, CA: Sage Publications, 1975), pp. 185–218.



crime severity, energy consumption, health care, administrative centralization, andthe quality of life.13

APPROACHES TO MONITORINGMonitoring is central to policy analysis. Yet there are so many ways to monitoroutputs and impacts that it is sometimes difficult to distinguish monitoring fromsocial research in general. Fortunately, monitoring can be broken down into severalidentifiable approaches: social systems accounting, social experimentation, socialauditing, and research and practice synthesis. These approaches may be contrastedin terms of two major properties (Table 2):

1. Types of controls. Approaches to monitoring differ in terms of the ways theyexercise control over variations in policy actions. Only one of the approaches(social experimentation) involves direct controls over policy inputs and policyprocesses. The other three approaches “control” inputs and processes bydetermining after the fact how much of an observed variation in outcomes isdue to inputs and processes, as compared to extraneous factors that are notdirectly connected with policy actions.

2. Types of information required. Approaches to monitoring differ according totheir respective information requirements. Some approaches (socialexperimentation and social auditing) require the collection of new information.Social systems accounting may or may not require new information; researchand practice synthesis relies exclusively on available information.

13On various kinds of indices, see references in Miller and Salkind, Handbook of Research Design andSocial Measurement; James L. Price, Handbook of Organizational Measurement (Lexington, MA: D. C. Heath and Company, 1972); Dale G. Lake and others, Measuring Human Behavior (New York:Columbia University Teachers College Press, 1973); Paul N. Cheremisinoff, ed., Industrial PollutionControl: Measurement and Instrumentation (Westport, CT: Technomic Press, 1976); Leo G. Reader andothers, Handbook of Scales and Indices of Health Behavior (Pacific Palisades, CA: Goodyear Publishing,1976); Lou E. Davis and Albert B. Cherns, The Quality of Working Life, vol. 1 (New York: Free Press,1975); and Albert D. Biderman and Thomas F. Drury, eds., Measuring Work Quality for Social Reporting(New York: Wiley, 1976).

TABLE 2

Four Approaches to Monitoring

Approach Types of Control Type of Information Required

Social systemsaccounting

Quantitative Available and/or newinformation

Social experimentation Direct manipulation andquantitative

New information

Social auditing Quantitative and/or qualitative New information

Research and practice synthesis

Quantitative and/or qualitative Available information



Each of these four approaches has some common features. First, each isconcerned with monitoring policy-relevant outcomes. Therefore, each approachdeals with variables that are relevant to policy makers because they are indicators ofpolicy outputs and/or impacts. Some policy-relevant variables can be manipulated bypolicy makers (e.g., resource inputs or new processes for delivering services); otherscannot. These nonmanipulable variables include preconditions, which are presentbefore actions are taken (e.g., the average age or the cultural values of a targetgroup), as well as unforeseen events that occur in the course of policy implementation(e.g., sudden staff turnover, strikes, or natural disasters).

A second common feature of these approaches is that they are goal focused.This means that policy outcomes are monitored because they are believed to enhance thesatisfaction of some need, value, or opportunity—that is, outcomes are seen as ways toresolve a policy problem. At the same time, some policy outcomes are monitoredbecause they may inhibit the satisfaction of some need, value, or opportunity.14

Nonmanipulable variables, that is, policy preconditions and unforeseen events, are alsogoal focused to the extent that they are known to affect policy outcomes.

A third feature common to these approaches is that they are change oriented.Each approach seeks to monitor change by analyzing changes in outcomes over time;by comparing such changes across two or more programs, projects, or localities; orby using some combination of the two. Some approaches (e.g., social systemsaccounting) are oriented toward macro-level changes in societies, states, regions, andcommunities; other approaches (e.g., social auditing and social experimentation) areprimarily oriented toward micro-level changes in programs and projects.

A fourth common feature of these approaches is that they permit the cross-classification of outputs and impacts by other variables, including variables used tomonitor policy inputs and policy processes.15 For example, such outputs as per pupileducational expenditures may be cross-classified with input variables (e.g., teachers’salaries) and those intended to measure processes (e.g., class sizes). Outputs andimpacts may also be cross-classified by types of preconditions (e.g., the averageincome of community residents) and unforeseen events (e.g., frequency of strikes).

Finally, each approach is concerned with objective as well as subjectivemeasures of policy actions and outcomes. For example, all approaches providemeasures of such objective outcomes as units of health care received as well as suchsubjective outcomes as satisfaction with medical and health services. Objectiveindicators are frequently based on available data (e.g., census materials), whereassubjective ones are based on new data acquired through sample surveys or fieldstudies. In some instances, objective and subjective measures are based onboth available and new information. For example, past studies may be repeated

14There is also the special case in which outputs and impacts are neither enhancing nor inhibiting, butneutral. In this context, one of the objections to an exclusive concern with social indicators that have“direct normative interest” is that what is relevant to today’s goals may not be so in later years. SeeEleanor B. Sheldon and Howard E. Freeman, “Notes on Social Indicators: Promises and Potential,”Policy Sciences 1 (1970): 97–98. Note that if a goal-focused outcome is defined in terms of its potentialeffect on a policy problem—and if policy problems are artificial, dynamic, and interdependent—then alloutcomes, including “neutral” ones, may someday be of direct normative interest.15Ibid., p. 97.



(replicated) to obtain new information that may then be compared with old infor-mation in an effort to monitor the direction and pace of social change.16

Each of these common features contributes to a general definition of monitoringas the process of obtaining policy-relevant information to measure objective andsubjective changes in goal-focused social conditions among various target groups andbeneficiaries.17 Social conditions include policy actions and outcomes as well as policypreconditions and unforeseen events that affect actions and outcomes in the course ofimplementation. Impacts may be immediate (first-order impacts) or secondary (second- ,third- , and nth-order impacts), as when policy actions produce “side effects” and“spillovers” beyond their intended effects. Side effects and spillovers may enhance orinhibit the satisfaction of needs, values, and opportunities. Elements of this generaldefinition of monitoring are diagrammed in Figure 2, which represents a generalframework for monitoring in policy analysis.

16This combination of old and new information is called replication of baseline studies. See Otis DudleyDuncan, Toward Social Reporting: Next Steps (New York: Russell Sage Foundation, 1969).17Compare the definition of social indicators offered by Kenneth C. Land, “Theories, Models, andIndicators of Social Change,” International Social Science Journal 25, no. 1 (1975): 14.

POLICYINPUTS

POLICYOUTPUTS

POLICYIMPACTS

POLICYPROCESSES

In1

Ini

In2

P1

Pi

P2

PC1

PCx

PC2

E1

Eq

E2

Q1

Qn

Q2

Im1

Imn

Im2

SES1

SESx

SES2

PRECONDITIONS UNFORESEENEVENTS

SIDE-EFFECTSAND

SPILLOVERS

UNMANIPULATED CAUSES UNCONTROLLED EFFECTS

MANIPULABLE ACTIONS CONTROLLED OUTCOMES

FIGURE 2General framework for monitoring

Note: The solid lines indicate effects on manipulable actions and controlled outcomes.The broken lines indicate unforeseen events, side effects and spillovers. Side effects andspillovers are uncontrollable secondary impacts that may enhance or inhibit the satisfactionof needs, values, and opportunities.



Social Systems AccountingWith this general framework in mind, we can proceed to contrast the four approachesto monitoring. Social systems accounting is an approach and set of methods thatpermit analysts to monitor changes in objective and subjective social conditions overtime.18 The term social systems accounting comes from a report published by theNational Commission on Technology, Automation, and Economic Progress, a bodyestablished in 1964 to examine the social consequences of technological developmentand economic growth. The commission’s report recommended that the federalgovernment establish a “system of social accounts” that would serve as a counterpartto national economic accounts.19 Several years earlier, a project undertaken by theAmerican Academy of Arts and Sciences explored the second-order impacts of thespace program. One result of this project was the development of methods formonitoring social and political, as well as economic, trends, and a major book waspublished in 1966 with the title Social Indicators.20 Although these efforts to developsocial systems accounting occurred in the 1960s, they actually represent a continuationof work begun in 1933 under the auspices of the President’s Research Committee onSocial Trends. Under the direction of sociologist William F. Ogburn, the committeeproduced a two-volume report, Recent Social Trends, before its work was abandonedduring the Great Depression.21

Work on social systems accounting continued throughout the 1960s and 1970s.Some of the major products of this period were produced by social scientists interestedin objective and subjective indicators of social change.22 Other major initiatives camefrom federal agencies, including the Department of Health, Education and Welfare;the Office of Management and Budget; and the Department of Labor.23 In addition,several international organizations (the United Nations and the Organization for

18Much of the following account is based on Kenneth C. Land and Seymour Spilerman, eds., SocialIndicator Models (New York: Russell Sage Foundation, 1974); and Land, “Theories, Models, andIndicators of Social Change,” pp. 7–20.19National Commission on Technology, Automation, and Economic Progress, Technology and theAmerican Economy (Washington, DC: U.S. Government Printing Office, 1966). A primary influence onthe commission’s recommendations was Daniel Bell, whose major work (Toward Post-IndustrialSociety: A Venture in Social Forecasting) was described in Case 2.1: Are Policy Analysts Technocrats?20Raymond A. Bauer, ed., Social Indicators (Cambridge, MA: MIT Press, 1966). Chapter 3, by BertramM. Gross, is titled “The State of the Nation: Social Systems Accounting.”21President’s Research Committee on Social Trends, Recent Social Trends (New York: McGraw-Hill,1933). Ogburn’s major work was Social Change: With Respect to Culture and Original Nature (NewYork: B. W. Huebsch, 1922).22Two companion volumes were devoted, respectively, to objective and subjective dimensions of socialchange. See Eleanor B. Sheldon and Wilbert E. Moore, eds., Indicators of Social Change: Concepts andMeasurements (New York: Russell Sage Foundation, 1968); and Angus Campbell and Philip E. Converse, eds., The Human Meaning of Social Change (New York: Russell Sage Foundation, 1972).Also see “America in the Seventies: Some Social Indicators,” Annals of the American Academy ofPolitical and Social Science 435 (January 1978).23Relevant publications are U.S. Department of Health, Education, and Welfare, Toward a Social Report(Washington, DC: U.S. Government Printing Office, 1969); Executive Office of the President, Office ofManagement and Budget, Social Indicators, 1973; and U.S. Department of Labor, State Economic andSocial Indicators (Washington, DC: U.S. Government Printing Office, 1973). In 1974, the NationalScience Foundation funded a Center for the Coordination of Research on Social Indicators, administeredunder the Social Science Research Council. The center publishes Social Indicators Newsletter.



Economic Cooperation and Development) have undertaken work on social indicators,as have the governments of France, the United Kingdom, West Germany, Canada,Norway, Sweden, and Japan.24

The main analytic measure in social systems accounting is the social indicator. Thereare various definitions of this term, but the most useful is “statistics which measuresocial conditions and changes therein over time for various segments of a population.By social conditions, we mean both the external (social and physical) and the internal(subjective and perceptional) contexts of human existence in a given society.”25

Social indicators are available to monitor special aspects or contexts of socialchange such as community indicators and indicators of pollution, health care, andthe quality of working life.26

Table 3 presents a list of representative social indicators. These indicators, drawnfrom several major sources, are grouped by area. Note, first, that many indicators aredesigned to monitor objective social states, for example, persons in state mental hos-pitals. Other indicators depend on subjective responses, for example, persons afraidto walk alone at night. Second, most indicators are expressed in units of time and arecross-classified by various segments of the population, for example, persons afraid towalk alone at night are arranged in a time series and cross-classified by race, age, ed-ucation, and income. Third, some indicators are expressed in terms of present expe-rience, whereas others are stated as future goals. Finally, many indicators are relatedto outputs and impacts of public policies. For example, the labor force participationrate for women ages 35 to 64 may be taken as an indicator of impacts of equal em-ployment opportunity–affirmative action policies, and indices of air pollution may beused to monitor the impact of programs implemented by the EnvironmentalProtection Agency. In short, most social indicators are policy relevant, goal focused,change oriented, cross-classified, and expressive of objective and subjective socialconditions.

In using social indicators for purposes of monitoring, it is frequently necessary tomake assumptions about the reasons for change in a social indicator. For example, ifreported crime is used to monitor the output of crime-control policies, officialstatistics from the Federal Bureau of Investigation (Uniform Crime Reports) indicatethat total reported crimes per 100,000 inhabitants declined by 16 percent between2000 and 2009 Violent crimes and property crimes declined by about the sameamount. To attribute the decrease to criminal justice policies, it is necessary to makethe assumption that changes in crime are a consequence of policy actions undertakenby law enforcement authorities. This assumption is questionable, even when we haveaccurate data on resource inputs (expenditures for personnel and equipment),because we ignore other causally relevant factors, for example, an aging population.If we ignore this and other causally relevant factors, we are forced to treat the

24For a comprehensive annotated bibliography, see Leslie D. Wilcox and others, Social Indicators andSocietal Monitoring: An Annotated Bibliography (New York: American Elsevier Publishing, 1974). The major international journal is Social Indicators Research: An International and InterdisciplinaryJournal for Quality of Life Measurement, published since 1974.25Land, “Theories, Models, and Indicators of Social Change,” p. 15.26See footnote 12. The most complete source on policy-relevant social indicators is MacRae, PolicyIndicators.



relation between inputs and outputs as a kind of “black box,” that is, an unknownarea that symbolizes what we do not know about the relation between inputsinto outputs.

The use of social indicators has several advantages. First, efforts to developindicators appropriate for monitoring policy outcomes may alert us to areas inwhich there is insufficient information. For example, although much informationexists about municipal services, much of it deals with outputs, that is, the numberand types of services provided per capita. Information about the impacts of municipalservice policies on citizens—for example, impacts measured in terms of satisfactionwith transportation, sanitation, and recreational facilities—is often inadequate.When social indicators provide reliable information about the impacts of policies ontarget groups, it becomes possible to modify policies and programs. Social indica-tors also provide information that helps structure policy problems and modifyexisting policy alternatives. We may find, for example, that the number of highschool graduates ages 25 and over has increased, but that the social mobility of theeducated population has not changed. In this case, we may wish to restructure the

TABLE 3

Some Representative Social Indicators

Area Indicator

Health and illness Persons in state mental hospitals(a)

Public safety Persons afraid to walk alone at night(b)

Education High school graduates ages 25 and older(a)

Employment Labor force participation rate for women(a)

Income Percentage of population below poverty line(a)

Housing Households living in substandard units(b)

Leisure and recreation Average annual paid vacation in manufacturing(a)

Population Actual and projected population(b)

Government and politics Quality of public administration(c)

Social values and attitudes Overall life satisfaction and alienation(d)

Social mobility Change from father’s occupation(a)

Physical environment Air pollution index(e)

Science and technology Scientific discoveries(f)

Source:(a)U.S. Department of Health, Education and Welfare, Toward a Social Report (Ann Arbor: Universityof Michigan Press, 1970).(b)Office of Management and Budget, Social Indicators, 1973 (Washington, DC: U.S. GovernmentPrinting Office, 1974).(c)U.S. Department of Labor, State Economic and Social Indicators (Washington, DC: U.S. GovernmentPrinting Office, 1973).(d)Angus Campbell, Philip Converse, and Willard Rodgers, The Quality of American Life (New York:Russell Sage Foundation, 1976).(e)Otis Dudley Duncan, Toward Social Reporting: The Next Steps (New York: Russell Sage Foundation, 1969).(f)National Science Board, Science and Engineering Indicators (Washington, DC: National Science Foundation.



problem of social inequality in a way that places less emphasis on educationalpolicies and more on the family as an explanation for educational performance andsocial mobility.27

Social indicators also have limitations. The choice of some indicators—forexample, the percentage of households below the poverty line or the number ofbusiness executives charged with ethics violations compared with governmentemployees at the same rank—reflects some social and political values rather thanothers and may convey the political biases of analysts. Given that policy problemsare artificial and subjective, it is doubtful that any social indicator can be totally freeof the values of those who develop and apply it for purposes of monitoring. Second,social indicators may not be directly useful to policy makers faced with practicalchoices. One study of federal policy makers, for example, found that social indicatorswere not seen as having much instrumental value.28 Hence, even though socialindicators may help conceptualize or structure problems, they may be so generalthat they cannot be used to find specific solutions for particular problems.

Second, most social indicators are based on available data about objective socialconditions. It is easier to use available data on objective conditions than it is tocollect new information about subjective conditions, but it may be just as importantto monitor subjective conditions as it is to monitor objective ones.29 For example,reported crimes per 100,000 inhabitants may decrease, yet citizens’ sense ofpersonal insecurity may remain the same or even increase. Finally, social indicatorsprovide little information about the various ways that policy inputs are transformedinto policy outcomes. Claims about variations in policy outputs and impacts arebased on observed correlations between policy inputs and outcomes, rather than onknowledge about the processes by which resource inputs are transformed intooutputs and impacts. Policy inputs are measured, policy outputs are measured, andthe two are related by establishing the degree of their association. Concurrently,efforts are made to determine whether preconditions and unforeseen events—that is,factors apart from the original policy inputs—enhance or inhibit the production ofsome output. Yet the process of accounting for these nonpolicy effects takes placeafter policy outputs have been produced and is based on the use of statisticalcontrols, that is, techniques that permit analysts to observe the effects of inputs onoutputs under carefully specified conditions. For example, the effects of educationalexpenditures on school achievement will be analyzed for poor and middle-classfamilies separately.

27See David Grissmer, Ann Flanagan, and Stephanie Williamson, Improving Student Achievement(Santa Monica, CA: RAND Publishing, 2000). A major study published earlier is Christopher Jencksand others, Inequality: A Reassessment of the Effect of Family and Schooling in America (New York:Basic Books, 1972).28The journals Science Communication and Public Understanding of Science address similar questionsabout the use of scientific and technical information. An early seminal study is Nathan Caplan andothers, The Use of Social Science Information by Policy-Makers at the Federal Level (Ann Arbor:Institute for Social Research. Center for Research on the Utilization of Scientific Knowledge, Universityof Michigan, 1975).29The journal Social Indicators Research addresses these issues in many articles published since it wasfounded in 1974.



Social ExperimentationOne of the consequences of using social indicators is that it may take a very largenumber of successes and failures to find out what works best and why. Such anapproach has been called random innovation and contrasted with systematicexperimentation.30 Random innovation is the process of executing a large numberof alternative policies and programs whose inputs are neither standardized norsystematically manipulated. Because there is no direct control over policy actions,the outcomes of policies cannot easily be traced back to known sources. By contrast,social experimentation is the process of systematically manipulating policy actionsin a way that permits more or less precise answers to questions about the sources ofchange in policy outcomes. Social experimentation, of which there are manyvarieties—for example, laboratory experimentation, field experimentation, quasi-experimentation—is advocated as a way to find solutions for social problems bydeliberately maximizing the differences between types of policy actions in a smalland carefully selected group of programs and assessing their consequences prior tomaking large-scale investments in untested programs.31

Social experimentation is based on adaptations of procedures used in classicallaboratory experiments in the physical sciences:32

� Direct control over experimental treatments (stimuli). Analysts who usesocial experimentation directly control experimental treatments (policyactions) and attempt to maximize differences among them in order to produceeffects that differ as much as possible.

� Comparison (control) groups. Two or more groups are used in social experi-ments. One group (called the experimental group) receives the experimentaltreatment, and other groups (called control groups) receive no treatment or asignificantly different treatment than that received by the experimental group.

� Random assignment. Potential sources of variation in policy outcomesother than those produced by the experimental treatment are eliminated byrandomly selecting members of the policy experiment, by randomly dividingthem into experimental and control groups, and by randomly assigning the“treatment” (policy or program) to these groups. Random assignmentminimizes biases in the selection of members and groups who may responddifferently to the experimental treatment. For example, children frommiddle-class families may respond to a special education program in a morepositive way than children from poor families. This would mean that factorsother than the program itself are responsible for such outputs as higherreading scores. These selection biases are reduced or eliminated throughrandomization.

30Rivlin, Systematic Thinking for Social Action; and Rivlin, “How Can Experiments Be More Useful?”American Economic Review 64 (1974): 346–54.31See, for example, Fairweather and Tornatzky, Experimental Methods for Social Policy Research.32See Donald T. Campbell and Julian C. Stanley, Experimental and Quasi-experimental Designs forResearch (Chicago: Rand McNally, 1966); and Campbell, “Reforms as Experiments,” in Handbook ofEvaluation Research, vol. 1, ed. Elmer I. Struening and Marcia Guttentag (Beverly Hills, CA: SagePublications, 1965), pp. 71–100.



Social experiments and quasi-experiments have been advocated as a way tomonitor the outcomes of public policy since the New Deal years of the 1930s.33 In thepost–World War II period, social experiments have been conducted in many areas ofpublic policy: public health, compensatory education, welfare, criminal justice, drugand alcohol abuse, population control, nutrition, highway safety, and housing.34 Oneof the best known social experiments is the New Jersey–Pennsylvania GraduatedWork Incentives Experiment, funded by the Office of Economic Opportunity toanswer questions surrounding the reform of the social welfare system in the 1960s.A random sample of able-bodied men ages 15 to 58 from low-income families wasselected from three sites in New Jersey (Trenton, Paterson-Passaic, and Jersey City)and one in Pennsylvania (Scranton). In each city, some families received various levelsof guaranteed income and tax breaks, whereas others received none. Altogether, some1,350 families participated in the experiment.

Critics of welfare reform expected that income supplements and tax breaks (calleda negative income tax) would induce men from low-income families to work less.This expectation was not substantiated by the original experiment, which indicatedthat the experimental (income maintenance) and control (no income maintenance)groups did not differ significantly in their employment behavior, as reflected bychanges in earnings before and after the experiment. In fact, the earnings of the exper-imental groups showed a slightly higher increase than those of the control group.

Social experimentation has the potential of showing in precise terms whethercertain policy actions (e.g., the provision of income maintenance) result in certainoutcomes (e.g., family earnings). The capacity of experiments and quasi-experimentsto produce valid causal inferences about the effects of actions on outcomes is calledinternal validity. The greater the internal validity, the more confidence we have thatpolicy outputs are a consequence of policy inputs. Threats to validity are specialconditions of persons, policies, or contexts or methodological features of theexperiment itself that compromise or diminish the validity of claims about policyoutcomes. In fact, procedures for social experimentation are a way to reduce threatsto the internal validity of claims about policy outcomes.35 Among these threats tointernal validity, the most important are the following:

1. History. A number of unforeseen events can occur between the time a policyis implemented and the point at which outcomes are measured. For example,public disturbances, strikes, or sudden shifts in public opinion may represent aplausible rival explanation of the causes of variations in policy outcomes.

2. Maturation. Changes within members of groups, whether these be individuals,families, or larger social units, may exert an independent effect on policyoutcomes. For example, with the passage of time, individual attitudes may

33A. Stephen “Prospects and Possibilities: The New Deal and New Social Research,” Social Forces13 (1935): 515–21. See Robert S. Weiss and Martin Rein, “The Evaluation of Broad-Aim Programs:A Cautionary Case and a Moral,” in Readings in Evaluation Research, ed. Francis G. Caro (New York:Russell Sage Foundation, 1971), pp. 37–42.34See Carl A. Bennett and Arthur A. Lumsdaine, eds., Evaluation and Experiment (New York: AcademicPress, 1975); and Fairweather and Tornatzky, Experimental Methods for Social Policy Research.35For a fuller discussion of threats to internal validity, see Campbell and Stanley, Experimental andQuasi-experimental Designs for Research, pp. 5–6.



change, learning may occur, or geographical units may grow or decline. Suchmaturation may make it difficult to explain policy outcomes.

3. Instability. Fluctuations in a time series may produce unstable variations inoutcomes that are a consequence of random error or procedures for obtaininginformation and not a result of policy actions. Because most time series areunstable, causal inferences about the effects of actions on outcomes are oftenproblematic.

4. Testing. The very fact of conducting an experiment and measuring outcomesmay sensitize members of experimental and control groups to the aims andexpectations of the experiment. When this occurs, policy outcomes may be aresult of the meaning attributed to policy actions by participants, and not theactions (e.g., income guarantees) themselves.36

5. Instrumentation. Changes in the system or procedures for measuring anoutcome, rather than variations in the outcome itself, may be the reason forthe success (or failure) of a policy.

6. Mortality. Some of the experimental or control groups, or their members,may drop out of the experiment before it is completed. This makes it more dif-ficult to make causal inferences. For example, in the New Jersey–Pennsylvaniaexperiment, some families broke up before the experiment was over.

7. Selection. In many situations, random sampling is not possible, thus requiring aquasi-experimental design that does not fully eliminate biases in selecting respon-dents. For example, if one group of children in a quasi-experimental programdesigned to improve reading skills is drawn primarily from upper-class families inwhich regular reading is more prevalent, the difference between reading scoresbefore and after the program will be less pronounced than those of childrendrawn from lower-class families. This may make the program look less effectivethan it really is and, in some cases, can even make the experiment look harmful.37

8. Regression artifacts. When members of experimental and control groups areselected on the basis of extreme characteristics (e.g., poor children selected for anexperimental reading program may be high achievers), artificial changes inreading scores may occur as a result of what is known as regression toward themean. Regression toward the mean is the statistical phenomenon wherebyextreme values of some characteristic of a population automatically regressestoward or “goes back to” the average of the population as a whole. For example,the height of children of very tall or very short parents tends to regress toward thearithmetic mean of all parents. Children of very tall and very short parents tend,respectively, to be shorter and taller than their fathers and mothers.

36The effects of interpretation and testing are sometimes known as the “Hawthorne effect,” named aftera series of experiments conducted in the Western Electric Company’s Hawthorne plant in Chicago,Illinois, in 1927–31. The output of workers increased after working conditions and other factors(inputs) were changed. Later, output continued to increase even after working conditions (rest pauses,lighting, and so on) were changed back to their original state, largely because the experiment itselfprovided opportunities for workers’ participation. See Paul Blumberg, Industrial Democracy: The Sociology of Participation (London: Constable, 1968).37See Donald T. Campbell and A. E. Erlebacher, “How Regression Artifacts in Quasi-experimentalEvaluations Can Mistakenly Make Compensatory Education Look Harmful,” in CompensatoryEducation: A National Debate, vol. III, ed. J. Hellmuth (New York: Brunner-Mazel, 1970), pp. 185–225.



There are many ways to increase the internal validity of social experiments andquasi-experiments by carefully designing research. Generally, such research involvesrandom selection, repeated measures of outcome variables over time, and measure-ment of the preprogram outcome measures for some members of the experimentaland control groups but not others. The latter procedure eliminates the possibility oftesting effects for some groups, whose outcome measures after the program can becompared with those of equivalent groups whose outcomes have been measuredboth before and after receiving inputs.38

Social experimentation is weakest in the area of external validity, which refersto the generalizability of causal inferences outside the particular setting in which anexperiment is conducted. Important threats to the external validity of claims aboutpolicy outputs are similar to those that threaten internal validity, including inter-pretation, testing, and selection. In addition, other threats are important to thegeneralizability of claims. The most important of these is artificiality. Conditionsunder which a social experiment is carried out may be atypical or unrepresentativeof conditions elsewhere. For example, conditions in cities in New Jersey andPennsylvania are different from those in San Francisco, Seattle, and Anchorage,making generalizations problematic.

Social experimentation is frequently unsuccessful in monitoring policy processes,including patterns of interaction among staff and clients and their changing attitudesand values. Many of the most important policies and programs are so complex thatsocial experimentation results in the oversimplification of policy processes. For example,social experimentation is not well suited to broad-aim programs that involve high levels of conflict among stakeholders or contexts in which the same inputs of personnel,resources, and equipment are perceived in altogether different ways by differentgroups.39 For these and other reasons, various qualitative methods for obtaining thesubjective judgments of stakeholders have been used to uncover otherwise neglectedaspects of policy processes. These methods—which range from participant observationand the use of logs or diaries to group sessions that resemble a policy Delphi—may beviewed as a way to complement or replace social experimentation.40

Social AuditingOne of the limitations of social systems accounting and social experimentation—namely, that both approaches neglect or oversimplify policy processes—is partlyovercome in social auditing. Social auditing explicitly monitors relations among

38The design described here is the “Solomon 4-group design.” The best overview of alternative researchdesigns and their role in reducing threats to internal validity is Fred N. Kerlinger, Foundations ofBehavioral Research, 2d ed. (New York: Holt, Rinehart and Winston, 1973), pp. 300–377.39See Weiss and Rein, “The Evaluation of Broad-Aim Programs,” in Readings in Evaluation Research,ed. Caro, pp. 287–96; and Ward Edwards, Marcia Guttentag, and Kurt Snapper, “A Decision-TheoreticApproach to Evaluation Research,” in Handbook of Evaluation Research, vol. 1, ed. Stuening andGuttentag, pp. 139–82.40See M. G. Trend, “On the Reconciliation of Qualitative and Quantitative Analysis: A Case Study,”Human Organization 37, no. 4 (1978): 345–54; and M. Q. Patton, Alternative Evaluation ResearchParadigm (Grand Forks: University of North Dakota Study Group on Evaluation, 1976).



inputs, processes, outputs, and impacts in an effort to trace policy inputs “from thepoint at which they are disbursed to the point at which they are experienced by theultimate intended recipient of those resources.”41 Social auditing, which has beenused in areas of educational and youth policy by analysts at the RAND Corporationand the National Institute of Education, helps determine whether policy outcomesare a consequence of inadequate policy inputs or a result of processes that divertresources or services from intended target groups and beneficiaries.

The essential differences between social auditing and other approaches tomonitoring are most easily seen if we compare social auditing with social systemsaccounting and social experimentation. Variations of the latter approaches havebeen used to monitor U.S. educational policies in two important studies, Equality ofEducational Opportunity and the Westinghouse Learning Corporation’s evaluationof the Head Start program.42 More recently, in the period since 2005, indicators ofcorporate social responsibility and environmental sustainability have been publishedby the International Organization for Standardization (ISO), a nongovernmentalorganization based in Geneva, Switzerland. In the case of Equality of EducationalOpportunity, which was a variation of social systems accounting, the effectivenessof various levels of school resources was monitored by measuring the relationbetween resource inputs (teachers, textbooks, school facilities) and outcomes,measured in terms of the achievement levels of students. The ISO reports are alsovariations of social systems accounting.

In the Head Start study, which had features similar to a field experiment, therelations between special reading and skill-development activities (inputs) andcognitive and affective skills (outputs) of children exposed to the program werestudied in a number of cities. The essential feature of these two approaches is that

the policy inputs are measured, policy outcomes are measured, and the two arerelated (with, of course, the use of experimental or statistical controls toneutralize the effect of situational variables). Whatever institutional structureintervenes is taken in effect as a black box into which the input resources go andout of which the desired outcomes and side effects come. In the first researchmentioned above [Equality of Educational Opportunity], the school was theblack box. Inputs to it were teacher salaries, pupil-teacher ratio, age of text-books, size of library, and a variety of other traditional measures of schoolresources. Outputs studied were achievement of students in verbal and mathe-matical skills. In the second study, the resource inputs were the additionalresources provided by Head Start; the outputs were cognitive and affectivemeasures on the children.43

41James S. Coleman, Policy Research in the Social Sciences (Morristown, NJ: General Learning Press,1972), p. 18. See also Michael Q. Patton, Utilization-Focused Evaluation (Beverly Hills, CA: SagePublications, 1978).42See James S. Coleman and others, Equality of Educational Opportunity (Washington, DC: U.S.Government Printing Office, 1966): and Victor Cicarelli and others, The Impact of Head Start: An Evaluation of the Effects of Head Start on Children’s Cognitive and Affective Development(Washington, DC: U.S. Government Printing Office, 1969).43Coleman, Policy Research in the Social Sciences, p. 18.



In monitoring policy processes, social auditing provides important informationabout what goes on inside the “black box.” The processes monitored in a socialaudit are of two main types: resource diversion and transformation.44 In resourcediversion, original inputs are diverted from intended target groups and beneficiaries asa result of the passage of resources through the administrative system. For example, thetotal expenditures for two personnel training programs may be equal, yet oneprogram may expend a higher percentage of funds on salaries and other employeecosts, thus resulting in fewer staff members per dollar and a diversion of services frombeneficiaries. Far more important is the process of resource transformation. Hereresources and their actual receipt by target groups may be identical, yet the meaning ofthese resources to program staff and target groups may be altogether different. If themeaning of resources is in fact different, then resources may have been transformed insuch a way as to enhance (or retard) their impact on beneficiaries. For this reason,quantitative methods may be supplemented by qualitative methods designed toprovide information about the subjective interpretations of policy actions by stake-holders who affect and are affected by the implementation of policies.45

A good example of the supplementary use of qualitative methods to monitorpolicy processes is that provided by a housing experiment conducted by the U.S.Department of Housing and Urban Development (HUD). This particular experi-ment was designed to find out whether direct cash payments for housing would helplow-income families obtain adequate housing on the open market.46

In this case, solely quantitative measures of inputs and outputs meant thatanalysts treated policy-program processes as a “black box,” preventing them fromseeing that certain factors—for example, spontaneous assistance by housingcounselors—accounted for program impacts. The supplementary use of qualitativemethods yielded strikingly different results than those that might have been producedby focusing exclusively on quantitative measures of inputs and outputs. This casealso shows that qualitative and quantitative approaches to monitoring are comple-mentary. Social experimentation and quantitative measurement were successfullycombined with social auditing and the qualitative research on policy processes.

Research and Practice SynthesisSocial auditing and social experimentation require the collection of new informationabout policy actions and outcomes. Social systems accounting, although basedchiefly on available information, also requires new data insofar as information

44The following account of social auditing differs from that of Coleman, which is confined to “resourceloss.” Coleman’s account does not include resource transformation and is based on the restrictiveassumption that social auditing is appropriate only for policies involving resource distribution.45A thorough general exposition of qualitative research methods is John W. Cresswell and Raymond C. Maietta, “Part 4: Qualitative Research,” pp. 143–197 in Miller and Neil Salkind, Handbook ofResearch Design and Social Measurement. The best expositions of qualitative methodology are BarneyG. Glaser and Anselm L. Strauss, The Discovery of Grounded Theory: Strategies for QualitativeResearch (Chicago: Aldine Publishing, 1967); and Norman K. Denzin, The Research Act (Chicago:Aldine Publishing, 1970).46See Trend, “On the Reconciliation of Qualitative and Quantitative Analyses.”



about subjective social conditions is out of date or unavailable. In contrast to eachof these approaches, research and practice synthesis is an approach to monitoringthat involves the systematic compilation, comparison, and assessment of the resultsof past efforts to implement public policies. It has been used to synthesize informa-tion in a number of policy issue areas that range from social welfare, agriculture,and education to municipal services and science and technology policy.47 It has alsobeen employed to assess the quality of policy research conducted on policy processesand policy outcomes.48

The two primary sources of available information relevant to research andpractice syntheses are case studies of policy formulation and implementation andresearch reports that address relations among policy actions and outcomes. Whenresearch and practice synthesis is applied to case studies, it may be based on the casesurvey method, a set of procedures used to identify and analyze factors that accountfor variations in the adoption and implementation of policies.49 The case surveymethod requires that the analyst first develop a case-coding scheme, a set of cate-gories that captures key aspects of policy inputs, processes, outputs, and impacts.In one such case survey, analysts sought to determine the influence of politicalparticipation and other process variables on the delivery of municipal services, anoutput variable.50 Other case surveys, focusing on public agencies and privatecorporations, have attempted to determine what factors account for successfuland unsuccessful efforts to implement and utilize a variety of management innova-tions.51 Table 4 illustrates representative indicators and coding categories from atypical case-coding scheme.

When research and practice synthesis is applied to available research reports, itis based on the research survey, research synthesis, or evaluation synthesis, a set ofprocedures used to compare and appraise results of past research on policy actionsand outcomes.52 Some of the more important applications of the research survey

47See, for example, Jack Rothman, Planning and Organizing for Social Change: Action Principles fromSocial Science Research (New York: Columbia University Press, 1974); Gerald Zaltman, RobertDuncan, and James Holbek, Innovations and Organizations (New York: Wiley-Interscience, 1973);Everett Rogers and F. Floyd Shoemaker, The Communication of Innovations (New York: Free Press,1971); Robert K. Yin and Douglas Yates, Street-Level Governments: Assessing Decentralization andUrban Services (Santa Monica, CA: RAND Corporation, October, 1974); U.S. General AccountingOffice, Designing Evaluations (Washington, DC: U.S. GA Office, July 1984); and Richard J. Light andDavid B. Pillemer, Summing Up: The Science of Reviewing Research (Cambridge, MA: HarvardUniversity Press, 1984).48See, for example, Ilene N. Bernstein and Howard E. Freeman, Academic and EntrepreneurialResearch: The Consequences of Diversity in Federal Evaluation Programs (New York: Russell SageFoundation, 1975).49See Robert K. Yin and Kenneth Heald, “Using the Case Survey Method to Analyze Policy Studies,”Administrative Science Quarterly 20, no. 3 (1975): 371–81; William A. Lucas, The Case SurveyMethod: Aggregating Case Experience (Santa Monica, CA: Rand Corporation, October 1974); and Yin,Case Study Analysis (Beverly Hills, CA: Sage Publications, 1985).50Yin and Yates, Street-Level Governments.51William N. Dunn and Frederic W. Swierczek, “Planned Organizational Change: Toward GroundedTheory,” Journal of Applied Behavioral Science 13, no. 2 (1977): 135–58.52See Rothman, Planning and Organizing for Social Change; U.S. General Accounting Office, DesigningEvaluations; and Light and Pillemer, Summing Up.



method have yielded insights into factors associated with the diffusion and commu-nication of innovations, planned social change, and the outputs and impacts ofpublic policies and programs.53 The research survey method provides several kindsof information: empirical generalizations about sources of variation in policyoutcomes, summary assessments of the confidence researchers have in these general-izations, and policy alternatives or action guidelines that are implied by thesegeneralizations. Table 5 illustrates sample results of one application of the researchsurvey method.

The research survey method, like the case survey method, requires the construc-tion of a format for extracting information about research reports. The typicalresearch report form includes a number of items that help analysts summarizethe research and appraise its quality. These items include variables measured, thetype of research design and methods used, the issue area in which the research was

53Rogers and Shoemaker, The Communication of Innovations; Gerald Zaltman, Toward a Theory ofPlanned Social Change: A Theory-in-Use Approach, prepared for the Second Meeting of the Networkof Consultants on Knowledge Transfer, Denver, Colorado, December 11–13, 1977; and Rothman,Planning and Organizing for Social Change, ch. 6, pp. 195–278.

TABLE 4

Case-Coding Scheme

Type of Indicator Indicator Coding Categories

Input Adequacy of resources [ ] Totally adequate

[ ] Mostly adequate

[ ] Not adequate

[ ] Insufficient information

Process Involvement of analytic staff in defining problem

[ ] Makes decisions

[ ] Influences decisions

[ ] No influence


Output Utilization of results of policyresearch

[ ] High

[ ] Moderate

[ ] Low

[ ] None


Impact Perceived resolution ofproblem(s)

[ ] Complete

[ ] Partial

[ ] No resolution




carried out, and an overall assessment of the reliability and validity of the researchfindings. Figure 3 presents a sample research survey form .

Research and practice synthesis has several major advantages as an approach tomonitoring. First, the case survey and research survey methods are comparativelyefficient ways to compile and appraise an increasingly large body of cases andresearch reports on policy implementation.54 Whether the focus is upon cases orresearch reports, research and practice synthesis allows analysts to systematically

TABLE 5

Sample Results of the Research Survey Method: Empirical Generalizations,ActionGuidelines, and Levels of Confidence in Generalizations

Generalization(a) Action Guidelines(b) Confidence in Generalization(c)

1. Current resourceexpenditures are stronglyassociated with pastresource expenditures.

In determining the receptivityto new policies and programs,analysts should seek outsituations in which substantialresource expenditures haveoccurred in the past.

3

2. Policy outcomes areassociated with the levelof economic developmentof the municipality, state,or region where policyactions are undertaken.

In determining the receptivityto new policies and programs,analysts should focus attentionon wealthy, urban, andindustrial areas.

4

3. Outputs of educationalpolicies in municipalitieswith professional citymanagers (reformgovernments) are greaterthan in municipalitieswithout professionalmanagers (nonreformgovernments).

Analysts should adjustrecommendations according to information about the typeof municipality where policyactions are intended to takeeffect.

2

Notes:(a)Empirical generalizations are based on available research reports, including scholarly books, articles, and government reports.(bAction guidelines are derived from empirical generalizations. The same generalization frequently implies multiple action guidelines.(c)The numbers used to express the degree of confidence in a generalization are based on the number of reliable and validstudies that support generalization.

Source: Adapted from Jack Rothman, Planning and Organizing for Social Change: Action Principles from Social Science Research (New York:Columbia University Press, 1974), pp. 254–65.

54On case materials, see the Electronic Hallway, a repository of many hundreds of cases in publicadministration and policy. This case repository, housed and managed at the Evans School at the Universityof Washington, is the product of a consortium of public policy schools.



1. Title of Report

2. Authors of Report

3. Abstract of Report (conceptual framework, hypotheses, research methodology, major findings)

4. Issue Area (for example, health, labor, criminal justice):

5. Major Variables Measured (Describe): (a) Input (b) Process (c) Output (d) Impact

6. Empirical Generalizations (List):

7. Research Design (for example, case study, social experiment, cross-sectional study)

8. Research Quality (reliability of data, internal validity, external validity)

FIGURE 3Sample research survey form

and critically examine different empirical generalizations and their implications.Second, the case survey method is one among several ways to uncover differentdimensions of policy processes that affect policy outcomes. Generalizationsabout policy processes may be used to support arguments from parallel cases andanalogies, for example, by showing that policies and programs carried out undersimilar circumstances have similar outcomes. Finally, the case survey method is agood way to examine objective as well as subjective social conditions. It is an inex-pensive and effective way to obtain information about the subjective perceptions ofpolicy processes held by different stakeholders.55

The main limitations of research and practice synthesis are related to the reliabilityand validity of information. Cases and research reports not only vary in quality anddepth of coverage but are often self-confirming.

For example, most available cases and research reports contain no explicitdiscussion of the limitations and weaknesses of the study, and frequently they putforth only one point of view. Similarly, available cases often report only “successful”efforts to implement policies and programs. Despite these limitations, research andpractice synthesis is a systematic way to accumulate knowledge about policy actionsand outcomes in many issue areas. Because it relies exclusively on available informa-tion, it is less expensive than social auditing, social experimentation, and social

55See Zaltman, Toward a Theory of Planned Social Change; and Dunn and Swierczek, “PlannedOrganizational Change.”



systems accounting. At the same time, these other approaches to monitoring havetheir own advantages and limitations. The strengths of one approach are often theweaknesses of another. For this reason, the most persuasive generalizations aboutpolicy outcomes are those that have multiple foci (inputs, processes, outputs,impacts), use different types of controls (direct manipulation, quantitative analysis,qualitative analysis), and are based on a combination of available and new informationabout objective and subjective conditions.

TECHNIQUES FOR MONITORINGMonitoring, unlike other policy-analytic methods, does not involve procedures thatare clearly limited to one approach rather than another. Many of the same techniquesare therefore appropriate for each of the four approaches: social systems accounting,social experimentation, social auditing, and research and practice synthesis. Table 6shows techniques appropriate for the four approaches to monitoring.

Graphic DisplaysMuch information about policy outcomes is presented in the form of graphicdisplays, which are pictorial representations of the values of one or more action oroutcome variables. A graphic display can be used to depict a single variable at oneor more points in time or to summarize the relationship between two variables.In each case, a graph displays a set of data points, each of which is defined by thecoordinates of two numerical scales. The horizontal scale of a graph is called theabscissa and the vertical scale is called the ordinate. When a graph is used to displaya causal relationship, the horizontal axis is reserved for the independent variable (X)and is called the X-axis; the vertical axis is used to display the dependent variable(Y) and is called the Y-axis. Because one of the main purposes of monitoring is to

TABLE 6

Techniques Appropriate to Four Approaches to Monitoring

ApproachGraphicDisplays

TabularDisplays

IndexNumbers

InterruptedTime-Series

AnalysisControl-Series

Analysis

Regression-Discontinuity

Analysis

Social systemsaccounting

× × × × × O

Social auditing × × × × × O

Socialexperimentation

× × × × × ×

Research andpracticesynthesis

× × O O O O

Note: ×, technique appropriate to approach; O, technique not appropriate to approach.



explain how policy actions affect policy outcomes, we usually place input andprocess variables on the X-axis and output and impact variables on the Y-axis.

One of the simplest and most useful kinds of graph is the time-series graph,which displays an outcome variable on the vertical axis and time on the horizontalaxis. If we wish to monitor the effects of police traffic control activities on motorvehicle deaths, we might use time-series graphs such as the ones presented inFigures 4 (a) and (b). Note that graph (a) has a vertical scale marked off in units of1,000 motor vehicle deaths and an origin of 44,000 (the origin is the point where thehorizontal and vertical axes intersect). By contrast, graph (b) is marked off in units of5,000 deaths and has an origin of zero. Although both graphs display the same data,the decline in deaths is far more dramatic in graph (a) than in graph (b). This exampleshows how the stretching or compression of values on the vertical axis can distort thesignificance of data. Although there is no clear rule to follow in selecting the size ofintervals, it is sometimes useful to use Tufte’s Lie Factor (LF), which is equal to theratio of the size of an effect shown in a graphic divided by the size of the effect in thedata.56 An LF equal to one indicates minimal distortion, whereas an LF greater than1.05 or less than 0.95 indicates substantial distortion.57 In Figure 4 (b), the LF for the

5857565554535251504948474645

1970 1972 1974 1976

(a) Stretched axis

(Y) M

otor

veh

icle

dea

ths

(100

0s)

(x) Years

7065605550454035302520151050

1970 1972 1974 1976

(b) Compressed axis

(Y) M

otor

veh

icle

dea

ths

(100

0s)

(x) Years

FIGURE 4Two graphic displays of motor vehicle deaths, 1970–77Note: Deaths include those involving collisions between motor vehicles and collisions between motor vehicles and trains,cycles, and fixed objects.

Source: National Safety Council.

56Edward R. Tufte, The Visual Display of Quantitative Information (Cheshire, CT: Cheshire Press,1983), p. 57.57An index with better norming properties may be created by taking the logarithm of the values 1.05 and 0.95. In this case, substantial distortions are greater than zero (> 0) or less than zero (< 0).



line segment between 1973 and 1974 is approximately 4.7, which suggests a verystrong effect of the 55 mph speed limit when the effect was actually much weaker.

Graphs may also be used to depict relationships between two or more variablesat one point in time. Graphs of this kind are called scatter diagrams. We may use ascatter diagram to find out if one variable changes in the same direction as another,that is, whether two variables are correlated. If one of the variables precedes theother in time (e.g., a new program precedes changes in the target population), or ifthere is a persuasive theory that explains the correlated variables (e.g., economictheory posits that greater income results in a higher propensity to save), then wemay be warranted in claiming that there is a causal relationship between the vari-ables. Otherwise, variables are simply correlated—that is, the values of two vari-ables covary or “go with” each other—and one variable cannot be assumed to bethe cause of the other.

A common problem in using graphs is spurious interpretation, a situation inwhich each of two variables that appear to be correlated is actually correlated withsome other variable. A good example of spurious interpretation is that of the analystwho observes from an examination of data on municipal firefighting activities that thenumber of fire engines present at the scene of a fire (input variable) is positively corre-lated with the amount of fire damage (output variable). This observed correlationmight be used to claim that the number of fire engines deployed does not reduce firedamage, because no matter how many fire engines are present, the amount of firedamage continues to rise at the same rate. This interpretation is spurious (false)because a third variable—the size of fires—is not taken into account. Once we includethis variable in our analysis, we find that the size of fires is correlated with both thenumber of fire trucks deployed and fire damage. The original correlation disappearswhen we include this third variable in our analysis, and our interpretation is plausible,rather than spurious (Figure 5). Spurious interpretation is a serious problem becausewe can never be completely certain that we have identified all relevant variables thatmay be affecting the original two-variable relationship.

Number of Fire Engines

Amount of Fire Damage

Amount of Fire Damage

Number of Fire Engines

Size of Fire

(a) Spurious Interpretation

(b) Plausible Interpretation

FIGURE 5Spurious and plausible interpretations of data on municipal firefightingactivitiesSource: Example adapted from H.W. Smith, Strategies of Social Research: The Methodological Imagination

(Englewood Cliffs, NJ: Prentice Hall, 1975), pp. 325–26.



Another means for displaying data on policy outcomes is the bar graph, apictorial representation that presents data in the form of separated parallelbars arranged along the horizontal (or vertical) axis. The bar graph presented inFigure 6 was used to display data on policy inputs (total municipal personnel costsper capita) as part of an analysis of the fiscal crisis of U.S. cities. Note that citieswith growing population have the lowest per capita personnel costs, whereas citiesthat are losing population have higher costs. The bar graph also shows the some-what special position of New York City in 1973.

Information about policy outcomes is often available in the form of groupedfrequency distributions, in which the numbers of persons or target groupsin particular categories (e.g., age or income) are presented graphically. For example, if we want to monitor the number of persons below the poverty line in various age groups, we might begin with the data in Table 7. These data canthen be converted into two kinds of graphic displays: histograms and frequencypolygons.

A histogram is a kind of bar graph that organizes information about agrouped frequency distribution of an action or outcome variable at one point intime. In a histogram, the width of the bars along the horizontal axis is equal,and there is no space between the bars. The height of the bars shows thefrequency of occurrence in each group (called a class interval). The histogram inFigure 7 (a) shows the numbers of persons below the poverty threshold in severalage categories in 1977. The histogram can be easily converted into a frequencypolygon (Figure 7[b]) by using the midpoints of each class interval as data pointsand connecting these points with a line. The difference between a histogramand a frequency polygon is that the latter uses lines to represent the frequencydistribution.

Another way to display information about policy outcomes is the cumulativefrequency polygon, a graph that shows the cumulative frequencies of a distributionalong the vertical axis. As one moves from left to right along the horizontal axis,the frequency of persons (or families, cities, states) in the first group is plotted on

700

600

500

400

300

200

100

0

$140

GrowingCities

$240

DecliningCities

$637

New YorkCity

Tota

l Mun

icip

al P

erso

nnel

Cos

tsP

er C

apita

l (d

olla

rs)

FIGURE 6Bar graph showing total municipalpersonnel costs per capita for citieswith growing and decliningpopulations and for New York CityNote: Total personnel costs are for 1973.The growthand decline of population is based on a 1960–73average.Source: Thomas Muller, Growing and Declining Urban Areas:

A Fiscal Comparison (Washington, DC: Urban Institute,1975).



TABLE 7

Grouped Frequency Distribution: Number of Persons Below the Poverty Threshold by Age Group in 1977

Age Groups Number of Persons below the Poverty Threshold

Under 14 7,856,000

14 through 21 4,346,000

22 through 44 5,780,000

45 through 54 1,672,000

55 through 59 944,000

60 through 64 946,000

65 and over 3,177,000

Total 24,721,000

Note: The poverty threshold was based on an index that took into account consumption requirements based on sex, age, and residencein farm or nonfarm areas. Poverty thresholds are updated to reflect changes in the Consumer Price Index. In 1977 the poverty thresh-old for all persons, irrespective of age and place of residence, was $3,067 annually. See Figure 9 for more recent poverty data.

Source: U.S. Department of Commerce, Bureau of the Census.

8.0

7.0

6.0

5.0

4.0

3.0

2.0

1.0

0.0

Per

sons

Bel

ow P

over

tyTh

resh

old

(mill

ions

)

Under14

14�21

22�44

45�54

55�59

60�64

65 andOver

Age Groups

(a) Histogram

8.0

7.0

6.0

5.0

4.0

3.0

2.0

1.0

0.0

Per

sons

Bel

ow P

over

tyTh

resh

old

(mill

ions

)

Under14

14�21

22�44

45�54

55�59

60�64

65 andOver

Age Groups

(b) Frequency Polygon

FIGURE 7Histogram and frequency polygon:Number of persons below the povertythreshold by age group in 1977Source: U.S. Department of Commerce, Bureau of the Census.



Line ofEquality

19891975

Lowest Second Third Fourth Highest

Quintiles

100

90

80

70

60

50

40

30

20

10

0

Cum

ulat

ive

Per

cent

age

of T

otal

Inco

me

Ear

ned

FIGURE 8

Lorenz curve showing distribution of familypersonal income in the United States in1989 and 1975Source: U.S. Department of Commerce, Bureau of the Census.

the vertical axis; the frequency of persons in the first and second groups is thenplotted; and so on until we reach the end of the horizontal scale, which is the sumof all frequencies. In monitoring the effects of policies designed to alleviate poverty,a useful type of cumulative frequency polygon is the Lorenz curve, which can beused to display the distribution of income, population, or residential segregation ina given population.58 For example, a Lorenz curve enables us to compare theshare of total income earned by each successive percentage group in the popula-tion. These percentage groups are called quintiles or deciles, which are groupscomposed of one-fifth or one-tenth of the population, respectively. Figure 8 showsthe distribution of family personal income in the United States in 1947 and 1975.As the Lorenz curve moves closer to the diagonal (which represents perfect equality),family incomes become more equitably distributed. The curve shows that thedistribution of family personal income was more equitable in 1975 than it was in1947.

The Gini IndexThe Lorenz curve illustrated in Figure 8 displays the distribution of income amongfamilies at two points in time. Lorenz curves may also be used to display the distri-bution of population or certain types of activities (e.g., crime or racial segregation)among spatially organized units such as cities.59 Lorenz curves can also be expressedin the form of the Gini concentration ratio (sometimes called simply the Gini index),

58The original exposition of the Lorenz curve may be found in M. O. Lorenz, “Methods of Measuringthe Concentration of Wealth,” Quarterly Publications of the American Statistical Association 9, no. 70(1905): 209–19.59For other indices, see Jack P. Gibbs, ed., Urban Research Methods (Princeton, NJ: Van Nostrand, 1961).


which measures the proportion of the total area under the diagonal that lies in thearea between the diagonal and the Lorenz curve. The formula for calculating thisproportion is

where

Xi � the cumulative percentage distribution of the number of areas (e.g., cities)Yi � the cumulative percentage distribution of the population or of some activity(e.g., crime)

To illustrate the calculation of the Gini index, let us consider an example fromthe area of criminal justice policy. Suppose that we wish to depict the concentrationof violent crimes in cities of various sizes in 1976. We would use the followingprocedures, illustrated in Table 8:

1. Place the number of cities and the total violent crimes reported for each classsize of cities in columns (1) and (2).

2. Compute the proportionate distribution of cities from column (1) and place incolumn (3). For example, 59 � 7361 = 0.008.

3. Compute the proportionate distribution of violent crimes from column (2) andplace in column (4). For example, 1095 � 2892 � 0.379.

4. Cumulate the proportions of column (3) downward and place the results incolumn (5). For example, 0 � 0.008 � 0.008, 0.008 � 0.015 � 0.23, and so on.

5. Cumulate the proportions of column (4) downward and place the results incolumn (6). For example, 0 � 0.379 � 0.379, 0.379 � 0.198 � 0.577, and so on.

6. Multiply the first line in column (6) by the second line in column (5), thesecond line by the third line, and so on. For example, 0.379 � 0.023 �0.0087, 0.577 � 0.059 � 0.0340, and so on. Place the results in column (7).

7. Multiply the first line in column (5) by the second line in column (6), and soon. For example, 0.008 � 0.577 � 0.0046, 0.023 � 0.721 � 0.0166, and soon. Place the results in column (8).

8. Sum column (7) and record the total (i.e., 1.3477). Sum column (8) and recordthe total (i.e., 0.5322).

9. Subtract the total of column (8) from the total of column (7) and then divideby the total of column (7). That is, 1.3477 � 0.5322 � 1.3477 � 0.8155 �1.3477 � 0.605 � 0.61. This result is the Gini index.

The Gini index shows the proportion of the area under the diagonal that liesbetween the diagonal and the Lorenz curve. It can be used to depict the concentrationof wealth, income, population, ethnic groups, urban residents, crime, and other socialconditions. The advantage of the Gini index is that it is a precise measure of concentra-tion that supplies much more information than the pictorial representation provided bythe Lorenz curve. Furthermore, the Gini index ranges from zero (no concentration atall) to 1.0 (maximum concentration). In the example of the concentration of violentcrimes, note that approximately 2 percent of the largest cities (100,000 and up) have

GI =

c aaXiYi+1b - aaXi+1Yib d

aXiYi+1




almost 58 percent of the violent crimes. If violent crimes were more or less equallydistributed among cities of all class sizes, the Gini index would be approximately zero.In fact, however, violent crimes are heavily concentrated in large cities. The Gini indexvalue of 0.61 shows that 61 percent of the total area is between the diagonal and theLorenz curve. This is a substantial concentration with important implications forcriminal justice policy in the United States.

Tabular DisplaysAnother useful way to monitor policy outcomes is to construct tabular displays.A tabular display is a rectangular array used to summarize the key features of one ormore variables. The simplest form of tabular display is the one-dimensional table,which presents information about policy outcomes in terms of a single dimension ofinterest, for example, age, income, region, or time. In monitoring changes in energydemand in the United States , an analyst might use a one-dimensional tabular display.

Information may also be arranged in tables with two dimensions, for example,changes in poverty rates by age and race over time. An analyst concerned with theimpact of welfare reform may wish to monitor changes in the percentage of personsbelow the poverty level in the United States in 1968, 1990, and 2006. Table 9

TABLE 8

Computation of Gini Concentration Ratio for Violent Crimes Known to Police per100,000 Population in 1976

ProportionCumulativeProportion

Size of City

Numberof Cities

(1)

ViolentCrimes

(2)Cities(3)

Crimes(4)

Cities(Yi) (5)

Crimes(Xi) (6)

Xi Yi+1

(7)Xi+1Yi

(8)

250,000 and over 59 1,095 0.008 0.379 0.008 0.379 0.0087 0.0046

100,000–249,999 110 573 0.015 0.198 0.023 0.577 0.0340 0.0166

50,000–99,999 265 416 0.036 0.144 0.059 0.721 0.1016 0.0494

25,000–49,999 604 338 0.082 0.117 0.141 0.838 0.2774 0.1306

10,000–24,999 1,398 254 0.190 0.088 0.331 0.926 0.9260 0.331

Under 10,000 4,925 216 0.669 0.074 1.000 1.000

Totals 7,361 2,892 1.000 1.000 � 1.3477g � 0.5322g

Gini concentration ration (GI) =

c agXiYi+1b - agXi+1Yib d

gXiYi+1

= 1.3477 - 0.5322

1.3477 =

0.8155

1.3477

= 0.605 = 0.61 (rounded)

Source: Federal Bureau of Investigation, Uniform Crime Reports for the United States, 1976.


provides information showing that the poverty rate among children increased since1968, whereas the rate among elders decreased sharply. The overall poverty ratewas virtually the same in 2006 as it was in 1968.

Another type of multidimensional table involves the analysis of two or more groupsby levels of some outcome variable, which might be employment, services received, orearnings. For example, social experiments are based on comparisons of groups that havereceived a particular type of treatment (experimental group) and groups that have not(control group). Table 9 assessed whether groups participating in a publicly fundedalcoholism treatment program differed in the number of arrests following treatment(Table 10). The results indicate that a court-ordered policy of forced short-term referralsto alcoholism clinics or to Alcoholics Anonymous for treatment (experimental groups)appears to be less effective than no treatment (control group).


TABLE 9

Poverty Rates in 1968, 1990, and 2006 by Age and Race

Poverty Rates (in percentages)

1968 1990 2006

All 12.8 13.5 12.3

Children 15.4 20.6 17.4

Ages 18–64 9.0 10.7 10.8

Elders 25.0 12.1 9.4

All Less Than Age 65 11.5 13.7 12.7

White 7.5 8.7 8.4

Black 32.8 31.6 24.2

Hispanic 23.8 28.3 20.7

Source: Daniel R. Meyer and Geoffrey L.Wallace,“Poverty Levels and Trends in Comparative Perspective,” Focus 26, 2 (Fall 2009): 9.

TABLE 10

Number of Rearrests among 241 Offenders in Three Treatment Groups

Treatment Group

Rearrests Alcoholism Clinic Alcoholics Anonymous No Treatment Total

None 26 (32.0) 27 (31.0) 32 (44.0) 85

One 23 (28.0) 19 (22.0) 14 (19.0) 56

Two or more 33 (40.0) 40 (47.0) 27 (37.0) 100

Total 82 (100.0) 86 (100.0) 73 (100.0) 241

Source: John P. Gilbert, Richard J. Light, and Frederick Mosteller, “Assessing Social Innovations: An Empirical Base for Policy,”in Evaluation and Experiment: Some Critical Issues in Assessing Social Programs, ed. Carl A. Bennett and Arthur A. Lumsdaine (New York:Academic Press, 1975), pp. 94–95. Numbers in parentheses refer to percentages, which total 100 percent in the last row.



Index NumbersAnother useful way to monitor changes in outcome variables over time is indexnumbers. Index numbers are measures of how much the value of an indicator or setof indicators changes over time relative to a base period. Base periods are arbitrarilydefined as having a value of 100, which serves as the standard for comparingsubsequent changes in the indicators of interest. Many index numbers are used inpublic policy analysis. These include index numbers used to monitor changes inconsumer prices, industrial production, crime severity, pollution, health care,quality of life, and other important policy outcomes.

Index numbers may focus on changes in prices, quantities, or values. For example,changes in the price of consumer items are summarized in the form of the ConsumerPrice Index (CPI); changes in the value of goods produced by industrial firms aresummarized with the Index of Industrial Production. Index numbers may be simpleor composite. Simple index numbers have only one indicator (e.g., the number ofcrimes per 100,000 persons); composite index numbers include several differentindicators. Good examples of composite index numbers are the two CPIs used tomeasure the costs of some four hundred goods and services. One index includes allurban consumers; the other includes urban wage earners and clerical workers.60

Index numbers may be implicitly weighted or explicitly weighted. In the formercase, indicators are combined with no explicit procedure for establishing their value.For example, an air pollution index may simply aggregate various kinds of pollu-tants (e.g., carbon monoxide, oxidants, sulfur oxides) without taking into accountthe different costs in impaired health that result from some of these pollutants(e.g., carbon monoxide). An explicitly weighted index takes such factors intoaccount by establishing the relative value of each indicator.

There are two general procedures for constructing index numbers: aggregationand averaging. Aggregative indices are constructed by summing the values of indica-tors (e.g., consumer prices) for given periods. Averaging procedures (or the so-calledaverage of relatives method) require the calculation of average changes in the value ofan indicator over time, whereas aggregative procedures do not. A useful and simpleaggregative price index is the purchasing power index (PPI), which measures the realvalue of earnings in successive periods. It may be used to monitor the impact of wageagreements on the real income of employees as part of public sector collective-bargaining activities. A purchasing power index is constructed by using values of theCPI. This is done by locating the year (e.g., 2001) for which we wish to determine thepurchasing power of wages and converting this value into a price relative, that is, avalue that expresses the price of goods and services in that year relative to a base year(2001). If we then divide this price relative into one (this is called a reciprocal), wewill obtain the purchasing power of wages in 2001 dollars.

In Table 11, the values of the purchasing power index for given years indicatehow much a dollar is worth in real terms, relative to the CPI base year of 2001. Forexample, in 2005 every dollar in wages buys about 91 cents of goods and serviceswhen compared with the purchasing power of one dollar in 2001. The purchasing

60In January 1978, the Consumer Price Index was revised to include all urban consumers, irrespective ofwhether they are employed, and covers 80 percent of the noninstitutional population.



TABLE 11

Use of Consumer Price Index (1982–84 �� 100) to Compute Purchasing Power Index

ConsumerPrice Index (1) Price Relative (2) Reciprocal (3)

Purchasing Power Index (4)

2001 177.1 177.1 � 177.1 � 1.0 1.0 � 1.0 � 100 � 100.0

2002 179.9 179.9 � 177.1 � 1.016 1.0 � 1.016 � 100 � 98.4

2003 184.0 184.0 � 177.1 � 1.039 1.0 � 1.039 � 100 � 96.2

2004 188.9 188.9 � 177.1 � 1.039 1.0 � 1.067 � 100 � 93.8

2005 195.3 195.3 � 177.1 � 1.103 1.0 � 1.103 � 100 � 90.6

2006 201.6 201.6 � 177.1 � 1.138 1.0 �1.138 � 100 � 87.9

2007 207.3 207.3 � 177.1 � 1.171 1.0 �1.171 � 100 � 85.4

2008 215.3 215.3 � 177.1 � 1.216 1.0 � 1.216 � 100 � 82.3

2009 214.5 1.216 214.5 � 177.1 � 1.211 1.0 � 1.211 � 100 � 82.6

2010 218.1 218.1 � 177.1 � 1.232 1.0 � 1.232 � 100 � 81.2

Source: U.S. Department of Labor, Bureau of Labor Statistics.

power index can be directly converted into real wages by taking the nominal wagesfor given years and multiplying by the purchasing power index. This has been donein Table 12, which shows the real value of weekly earnings in the years 2001–10.Real weekly earnings declined from $596 to $594 from 2001 to 2008. Real weeklyearnings increased to more than $610 in 2009 and more than $604 in 2010.

TABLE 12

Real Weekly Wages in the United States, 2001–10

Year

NominalEarnings

(1)

Purchasing PowerIndex (2)

Reciprocal (3)

Real Wages (4)

$596 1.0 1.000 � $596 � $596

2002 608 98.4 .984 � 608 � 598.27

2003 620 96.2 .962 � 620 � 596.44

2004 638 93.8 .938 � 638 � 598.44

2005 651 90.6 .906 � 651 � 589.81

2006 671 87.9 .879 � 671 � 589.80

2007 695 85.4 .854 � 695 � 593.53

2008 722 82.3 .823 � 722 � 594.21

2009 739 82.6 .826 � 739 � 610.41

2010 747 81.2 .812 � 747 � 606.56

Source: U.S. Department of Labor, Bureau of Labor Statistics.


Another useful aggregative index is the Index of Pollutant Concentration devel-oped under the CAMP of the Environmental Protection Agency. The concentrationof various kinds of pollutants, measured in parts per million (ppm) or microgramsper cubic meter of air (mg/m3), is punched into a recording tape every five minutesin various locations across the country. Pollutants fall into two main categories:gases and suspended particulates. The maximum concentrations of pollutantsreported in Chicago, San Francisco, and Philadelphia for two averaging times (fiveminutes and one year) in 1964 are displayed in Table 13. Between 2000 and 2009,there was substantial progress in reducing pollutants in these cities, with pollutantsdeclining from 10 percent to 25 percent in Chicago and Philadelphia and as much as50 percent in San Francisco.61

An averaging time of one year is the total quantity of pollutants reported in allfive-minute intervals over a year’s period divided by the total number of five-minuteintervals for the year. An averaging time of five minutes is simply the maximumconcentration of pollutants registered in any five-minute period. An implicitlyweighted aggregative-quantity index can be developed to monitor the concentrationof pollutants over time. The formula for this index is

where

QIn � the quantity index for gaseous pollutants in a given time period nqn � the quantity of gaseous pollutants in period n of a time seriesq0 � the quantity of gaseous pollutants in the base period 0 of a time series

On the basis of this formula, we can calculate index numbers to monitor changes inlevels of pollutants (measured in millions of tons) between 1970 and 1975.We would begin by selecting 1970 as the base period (q0) and calculate values of thequantity index for 1973, 1974, and 1975. These values are provided in Table 14,which shows that total pollutant emissions have declined since 1970.

This implicitly weighted aggregative-quantity index of pollution does not takeinto account variations in the concentrations of pollutants or their relative damageto health. By contrast, the index developed as part of the Environmental ProtectionAgency’s CAMP provides explicit weights, because it permits analysts to examinethe maximum concentration of pollutants in given periods (averaging times) andrelate these concentrations to known health hazards. For example, even though theoverall quantity of pollutants declined between 1970 and 2009, some cities haveregistered very high levels of pollution in particular periods. Data on these periodsare not reflected in the implicitly weighted quantity index illustrated in Table 14.Most important, the concentration of different types of pollutants per time periodcreates different kinds of health hazards. An explicitly weighted quantity indextakes into account both the concentration of pollutants per time period (averaging

QIn = gqnq0

# 100


61Air Quality Monitoring Trends, U.S. Environmental Protection Agency, 2010,http://www.epa.gov/airtrends/factbook.html.


TAB

LE

13

Max

imum

Con

cent

rati

on o

f P

ollu

tant

s R

epor

ted

in C

hica

go,P

hila

delp

hia,

and

San

Fra

ncis

co f

or A

vera

ging

Tim

es o

fFi

ve M

inut

es a

nd O

ne Y

ear

Type

of P

ollu

tant

Gase

s(a)

Susp

ende

d P

artic

ulat

es(b

)

Cit

y an

dA

vera

ging

Tim

eC

arbo

nM

onox

ide

Hyd

ro-

carb

ons

Nit

ric

Oxi

deN

itro

gen

Dio

xide

Nit

roge

nO

xide

sO

xide

ntS

ulfu

rD

ioxi

deL

ead

Org

anic

(Ben

zene

Sol

uble

)Su

lfate

Chi

cago

5 m

inut

es64

.020

.00.

970.

791.

120.

171.

62N

AN

AN

A

1 Y

ear

12.0

3.0

0.10

0.05

0.15

0.02

0.18

0.6

15.1

16.6

Phi

lade

lphi

a

5 m

inut

es43

.017

.01.

350.

371.

500.

251.

00N

AN

AN

A

1 Y

ear

7.0

2.0

0.04

0.04

0.08

0.02

0.09

0.8

12.2

19.8

San

Fra

ncis

co

5 m

inut

es47

.016

.00.

680.

380.

910.

290.

26N

AN

AN

A

1 Y

ear

7.0

3.0

0.09

0.05

0.14

0.02

0.02

NA

9.2

6.2

Not

es:

(a)

Gas

eous

pol

luta

nts

mea

sure

d in

par

ts p

er m

illio

n (p

pm)

of a

ir in

196

4.(b

)S

uspe

nded

par

ticu

late

s m

easu

red

in m

icro

gram

s pe

r cu

bic

met

er (

mg/

m3)

of

air

in 1

964.

NA

�no

t av

aila

ble.

Sou

rce:

N.D

.Sin

gpur

wal

la,

“Mod

els

in A

ir P

ollu

tion

,”in

A G

uide

to M

odel

s in

Gov

ernm

enta

l Pla

nnin

g an

d O

pera

tions

,ed

.Sau

l I.G

ass

and

Rog

er L

.Sis

son

(Was

hing

ton,

DC

:U

.S.E

nvir

onm

enta

l Pro

tect

ion

Age

ncy,

1975

),pp

.69–

71.



TABLE 14

Implicitly Weighted Aggregative Quantity Index to Measure Changes inPollutants in the United States, 1970–75

Quantitya

Pollutant 1970 1973 1974 1975

113.7 111.5 104.2 96.2

Sulfur oxides 32.3 32.5 31.7 32.9

Hydrocarbons 33.9 34.0 32.5 30.9

Particulates 26.8 21.9 20.3 18.0

Nitrogen oxides 22.7 25.7 25.0 24.2

229.4 225.6 213.7 202.2Quantity index (1970 � 100) 100.0 98.34 93.16 88.14

QIn = gqn

gq0 # 100

QI1970 = g113.7 + 32.3 + 33.9 + 26.8 + 22.7

g113.7 + 32.3 + 33.9 + 26.8 + 22.7 # 100

= 229.4

229.4 # 100 = 100.0

QI1973 = g111.5 + 32.5 + 34.0 + 21.9 + 25.7

g113.5 + 32.3 + 33.9 + 26.8 + 22.7 a

œ 100

= 225.6

229.4 # 100 = 98.34

aIn millions of tons of pollutants (gases and particulates).

Source: U.S. Environmental Protection Agency.

times) and potential health hazards. Although we will not construct an explicitlyweighted aggregative-quantity index on the basis of averaging times reported by theCAMP, it is important to consider relationships between the duration of exposure topollutants and damage to health and the environment (Table 15).

Index numbers have several limitations. First, explicit weighting proceduresoften lack precision. For example, estimates of the damage caused by different kindsof pollutants are partly subjective, and it is difficult to attach dollar costs to resultanthealth problems. Second, it is difficult to obtain sample data for indexes that areequally meaningful for all groups in society. The CPI, for example, is based on asample of more than four hundred consumer goods and services purchased by urbanresidents. The sample is selected monthly from fifty-six urban areas, and data thusobtained are used to construct a composite national index. Although the develop-ment of the two-part CPI in 1978 makes index numbers more meaningful to retired,elderly, and unemployed persons, as well as to wage earners and clerical workers, theCPI does not reflect price differences in many urban and rural areas. Finally, index



TABLE 15

Duration of Exposure to Pollutants and Consequent Damage to Health and theEnvironment

Pollutant

Duration of Exposure Consequence

CarbonMonoxide Oxidants Particulates

SulfurOxides

1 second Sensation

Odor � � � �

Taste � � � �

Eye irritation � � � �

1 hour Athletic performanceimpaired � � � �

Visibility reduced � � � �

8 hours Judgment impaired � � � �

Stress to heart patients � � � �

1 day Health impaired � � � �

4 days Health impaired � � � �

1 year Vegetation damaged � � � �

Corrosion � � � �

Note: A minus (�) indicates no damaging consequences. A plus (�) indicates damaging consequence.

Source: N. D. Singpurwalla, “Models in Air Pollution,” in A Guide to Models in Governmental Planning and Operations, ed. Saul I.Gass and Roger L. Sisson (Washington, DC: U.S. Environmental Protection Agency, 1975), p. 66.

numbers do not always reflect qualitative changes in the meaning or significanceof index items over time. For example, as we have learned more about the effects ofvarious kinds of pollutants (e.g., lead particulates and asbestos), their significancechanges. The simple quantity of pollutants may not reflect the changing social value(or disvalue) attached to them in different time periods.

Index numbers may be used to monitor a wide variety of changes. Yet thesetechniques provide no systematic way of relating changes in policy outcomes to priorpolicy actions; that is, index numbers are signs, not causes or effects. We now considerthree sets of techniques that permit analysts to make systematic judgments about theeffects of policy actions on policy outcomes: interrupted time-series analysis, control-series analysis, and regression-discontinuity analysis. These procedures are based oncorrelation and regression techniques discussed in the chapter “Forecasting ExpectedPolicy Outcomes.”

Interrupted Time-Series Analysis62

Interrupted time-series analysis is a set of procedures for displaying in graphic andstatistical form the effects of policy actions on policy outcomes. Interrupted time-series

62The following discussion is based on Campbell, “Reforms as Experiments.”



analysis is appropriate when an agency initiates some action that is put into effectacross an entire jurisdiction or target group, for example, in a particular state or amongall families living below the poverty threshold. Because policy actions are limited topersons in the jurisdiction or target group, there is no opportunity for comparing policy outcomes across different jurisdictions or among different categories of thetarget group. In this situation, the only basis of comparison is the record of outcomesfor previous years.

The graphing of interrupted time series is a powerful tool for assessing theeffects of a policy intervention on a measure of some policy outcome (e.g., highwaydeath rates, persons receiving medical assistance, job trainees employed, hospitalmortality rates). As Figure 9 shows, some policy interventions have genuine effectson policy outcomes, whereas others do not. Interventions (a) and (b) indicate thatthe policy intervention is plausibly related to the increase in some valued outcome,because there is a practically significant jump in the value of the measured policyoutcome after the intervention. Intervention (b), however, shows that the valuedoutcome was not durable—it began to “fade” after the fifth time period.

Intervention (c) also lacks durability, but it was also preceded by an extremehigh in the measured policy outcome before the intervention. This suggests that thereprobably was “regression” toward the average value of the time series after the inter-vention—that is, the increase between the fourth and fifth periods probably wouldhave occurred anyway (see later discussion). Intervention (d) shows that the inter-vention had no effect because there is no difference in the rate of change in the valueof the measured outcome variable. Finally, intervention (e) shows a slight increase inthe measured policy outcome but one that was preceded by an increase already inprogress. This weak effect is not sufficient to conclude that the intervention wasresponsible for the increase between the fourth and fifth periods. If data on subse-quent time periods were available, the post-intervention increase might prove to lackdurability or might increase at an increasing rate. Here it is not possible to know.

Interrupted time-series analysis is one among many approaches to socialexperimentation that are called quasi-experimental. Quasi-experimental refers toexperiments that have many but not all characteristics of a genuine experiment: randomselection of subjects, random assignment of subjects to experimental and controlgroups, random assignment of a “treatment” (a policy or program) to experimental andcontrol groups, and measurement of outcomes before and after the experimental treat-ment (policy or program). The only feature that interrupted time-series analysis doesnot have is random selection of participants or subjects. The other three characteristicsare present in many quasi-experiments designed to “test” policies and programs todetermine whether they make a difference in producing some outcome of interest.

The best way to visualize interrupted time-series analysis is to consider a well-known illustration.63 In 1956, after a record high number of traffic fatalities in 1955,the governor of Connecticut (Abraham Ribicoff) implemented a severe crackdown onviolators of the state’s speeding laws. At the end of 1956, there were 284 traffic deaths,

63See Campbell, “Reforms as Experiments,” pp. 75–81; and Donald T. Campbell and H. LawrenceRoss, “The Connecticut Crackdown on Speeding: Time-Series Data in Quasi-experimental Analysis,”Law and Society Review 3, no. 1 (1968): 33–53.



compared with 324 deaths the year before. Governor Ribicoff interpreted the out-comes of the crackdown on speeding in the following way: “With the saving of 40 livesin 1956, a reduction of 12.3 percent from the 1955 motor vehicle death toll, we can saythat the program is definitely worthwhile.” The apparent results of the speeding crack-down are displayed graphically in Figure 10. The values on the vertical scale have beendeliberately stretched to exaggerate the magnitude of effects.

Measure of Policy Outcome

15

12

9

6

3

01 2 3 4 5 6 7 8

Time Period

(a)

(b)

(c)

(d)

(e)

Policy Intervention

FIGURE 9Interrupted time series showing effects and no effectsSource: Donald T. Campbell and Julian C. Stanley, Experimental and Quasi-experimental Designs for Research (Chicago: RandMcNally, 1967).



If we wish to monitor the outcomes of the Connecticut crackdown on speedingusing interrupted time-series analysis, we will follow several steps:

1. Compress the values along the vertical axis so that the apparent observed effect isnot so dramatic. Here we will use intervals of twenty-five rather than ten deaths.

2. Obtain values of traffic deaths in multiple years before and after the implemen-tation of the policy and plot them on the graph. This creates an extended timeseries from 1951 through 1959.

3. Draw a vertical line that interrupts the time series at the beginning of the yearwhen the policy was implemented, that is, in 1956 (hence, an “interruptedtime series”). Note that intervals used to display years in a time series denotethe middle of each year (July 1), so that any point midway between twointervals is the value of the time series on January 1.

The results of following these procedures are displayed in Figure 11, which iscalled an interrupted time-series graph. Observe that this graph creates an altogetherdifferent visual image of policy outcomes than that presented in Figure 10.

The main advantage of interrupted time-series analysis is that it enables us toconsider systematically various threats to the validity of causal inferences aboutpolicy outcomes. The importance of these threats to validity can be easily seen whenwe compare policy outcomes (traffic deaths) displayed in the interrupted time-seriesgraph and Governor Ribicoff’s causal inference that the speeding crackdowndefinitely resulted in lowered traffic fatalities. Let us now examine several threats tothe validity of this claim by considering the data displayed in Figures 10 and 11.

1. Maturation. Changes within the members of groups (in this case, drivers)may exert effects on policy outcomes apart from those of policy actions. Forexample, death rates in Connecticut may have been declining for a number of

280

1955(Before Policy

Action)

Years

1956(After Policy

Action)

290

300

Traf

fic D

eath

s

310

320

FIGURE 10Connecticut traffic deaths before andafter the 1956 crackdown on speedingSource: Donald T. Campbell, “Reforms as Experiments,” inHandbook of Evaluation Research, vol. 1, ed. Elmer L. Strueningand Marcia Guttentag (Beverly Hills, CA: SagePublications, 1975), p. 76.



200

225

250

275

300

325

1951 1952 1953 1954 1955 1956 1957 1958 1959

Traf

fic D

eath

s

Pol

icy

Act

ion

Years

FIGURE 11Interrupted time-series graph displaying Connecticut trafficfatalities before and after the 1956 crackdown on speedingSource: Adapted from Donald T. Campbell, “Reforms as Experiments,” in Handbook

of Evaluation Research, vol. 1, ed. Elmer L. Struening and Marcia Guttentag (BeverlyHills, CA: Sage Publications, 1975), p. 76.

years as a result of increased social learning among drivers who have beenexposed to drivers’ education programs or to public safety campaigns. Thesimple pre-policy–post-policy graph (Figure 10) does not allow us to examinesuch possible independent effects. The interrupted time-series graph (Figure11) enables us to rule out maturation as a plausible rival explanation, becausethe time series does not show a long-term decline in traffic fatalities.

2. Instability. All time series are to some degree unstable. The interrupted time-series graph permits us to display this instability, which is rather pronounced.The decline between 1955 and 1956 is only slightly greater than declines in1951–52 and 1953–54, thus suggesting that the policy outcome may be aconsequence of instability, and not the policy action. On the other hand, thereare no increases in traffic fatalities after 1956. Although this lends persuasive-ness to the governor’s claim, this argument would not have been possiblewithout the interrupted time-series graph of Figure 11.

3. Regression artifacts. In monitoring policy outcomes, it is often difficult toseparate the effects of policy actions from effects exerted by the persons orgroups selected for a program. If traffic fatalities for 1955 represent anextreme—as might be the case if we have only Figure 10 to go on—then thedecline in traffic deaths in 1956 may simply reflect the tendency of extremescores to regress toward the mean or general trend of a time series. Fatalities in



1955 are in fact the most extreme ones in the series. For this reason, part ofthe decline from 1955 to 1956 may be attributed to regression artifacts,together with the instability of the time series. Regression artifacts are aparticularly important threat to the validity of claims about policy outcomesfor one important reason: policy makers tend to take action when problemsare most acute or extreme.

Control-Series AnalysisControl-series analysis involves the addition of one or more control groups to an inter-rupted time-series design in order to determine whether characteristics of differentgroups exert an independent effect on policy outcomes, apart from the original policyaction. The logic of control-series analysis is the same as that of interrupted time-seriesanalysis. The only difference is that a group or groups that were not exposed to thepolicy actions in question are added to the graph. Figure 12 shows the original datafrom Connecticut along with data on traffic fatalities from control states.

Control-series analysis helps further scrutinize the validity of claims about theeffects of policy actions on policy outcomes. For example, rival claims based on history

17

16

15

14

13

12

11

10

9

8

7

1951 1952 1953 1954 1955 1956 1957 1958 1959Years

Fata

lity

Rat

e

Control States

Connect Circuit

FIGURE 12Control-series graph displaying traffic fatalities inConnecticut and control states, 1951–59

Note: The fatality rate is a population-based measure that represents deaths per100,000 persons in the population of states.Source: Donald T. Campbell, “Reforms as Experiments,” in Handbook of Evaluation

Research, vol. 1, ed. Elmer L. Struening and Marcia Guttentag (Beverly Hills, CA:Sage Publications, 1975), p. 84.



(sudden changes in weather conditions) and maturation (safety habits learned bydrivers) would appear to be partly supported by the control-series data. HenceConnecticut and the control states show a similar general pattern of successiveincreases and decreases in traffic deaths between 1952 and 1955. Yet it is alsoevident that Connecticut traffic fatalities declined much more rapidly than those incontrol states after 1955. The control-series graph not only enables us to assessmore critically the governor’s claim about the success of his policy, but also in thiscase allows us to rule out plausible rival interpretations. This has the effect ofproviding much firmer grounds for policy claims.

The use of interrupted time-series and control-series analyses, together with thevarious threats to internal and external validity, may be viewed as a policy argu-ment. The original data on the decline in traffic fatalities between 1955 and 1956represent information (I), whereas the policy claim (C) is “the program is definitelyworthwhile.” The qualifier (Q) expresses maximum certainty, as indicated by theterm definitely, and the implicit warrant (W) necessary to carry information to claimis that the policy action caused the reduction in traffic fatalities. What is importantin the present case is the series of objections (O) that may be advanced in the formof threats to internal and external validity: history, maturation, instability, testing,instrumentation, mortality, selection, and regression artifacts. These threats to thevalidity of claims have been put in the form of a policy argument in Figure 13.

The final procedure to be discussed in this chapter is an extension of regressionand correlational procedures. Because linear regression and correlation have alreadybeen described in some detail, we will not repeat the discussion of these techniqueshere, except to note that they are applicable to problems of monitoring as well asforecasting. Regression analysis may be used to explain past outcomes, rather thanpredict future ones, and the results of analysis are sometimes called postdictions toemphasize that we are predicting past events. Even when regression analysis is usedto predict future events, the analysis is necessarily based on information acquired bymonitoring past policy outcomes.

Imagine that we have obtained information about dollar investments in fivepersonnel training programs, together with information about the subsequentemployment of trainees who have completed these programs. The problem can bestated in the following way: How does the level of investment in personnel trainingprograms affect the subsequent employment of trainees? A worksheet necessary tocompute the regression equation [Yc � a � b(x)], the coefficient of determination(r2), the simple correlation coefficient (r), the standard error of estimate (Sy.x), andan interval estimate (Yi) is provided in Table 16. The SPSS results, shown in Exhibit1, should be self-explanatory. If they are not, review information on correlation andregression.

Regression-Discontinuity AnalysisHaving reviewed computational procedures for correlation and regression analysis,we are now in a position to consider regression-discontinuity analysis. Regression-discontinuity analysis is a set of graphic and statistical procedures used to computeand compare estimates of the outcomes of policy actions undertaken among twoor more groups, one of which is exposed to some policy treatment while the other



The crackdown onspeeding was worthwhile.

CLAIM

There was a 12.3 percent declinein traffic fatalities after thecrackdown on speeding in 1955.

INFORMATIONDefinitelyQUALIFIER 1

The crackdown causedthe decline.

WARRANT

The crackdown preceded the declinein time and it is not plausible thatother factors caused the decline.

BACKINGRegression towardthe mean makes itunlikely that thecrackdown wasworthwhile.

QUALIFIER 2

But weather conditions were severe in 1956 and caused more accidents(HISTORY). Driver education resulted in safer driving (MATURATION). Theselection of the year 1956 is biased because fatalities were at a low point(SELECTION). Fatalities in 1955 were extreme; they regressed toward themean in 1956 (REGRESSION ARTIFACT).

OBJECTION

However, several OBJECTIONS are not plausible: Meteorologists report thatthe weather was mild in 1956. The effect of driver education on fatalities isunknown. Seven of the years have lower fatalities than 1956.

REBUTTALS

But the graph clearly shows regression toward to mean. This, not the crackdown, is the reason for the decline.

OBJECTION

FIGURE 13Threats to validity as objections

is not. Regression-discontinuity analysis is the only procedure discussed so far thatis appropriate solely for social experimentation. In fact, regression-discontinuityanalysis is designed for a particularly important type of social experiment, namely,one in which some resource input is in such scarce supply that only a portion ofsome target population can receive needed resources. Such experiments involve“social ameliorations that are in short supply, and that therefore cannot be given to


TAB

LE

16

Wor

kshe

et f

or R

egre

ssio

n an

d C

orre

lati

on:I

nves

tmen

t in

Tra

inin

g P

rogr

ams

and

Sub

sequ

ent

Em

ploy

men

t of

Tra

inee

s

Pro

gram

Per

cent

age

of T

rain

ees

Em

ploy

ed (

Y)

Pro

gram

Inve

stm

ent

(mill

ions

of

dolla

rs)(

X)Y

– (y

)y

X–

(x)

xxy

y2x2

Y cY

�Y c

(Y�

Y c)2

A10

2�

20�

240

400

412

�2

4

B30

30

�1

00

121

981

C20

4�

100

010

00

30�

1010

0

D50

620

240

400

448

24

E40

510

110

100

139

11

150

200

090

4000

1015

00

190

Y=

30X

=4

S yx

=

Dg(Y

-Y

c)2

n-

2=

C190 3

=7.

96b

= g

(xy)

g(x

)2

= 90 10

=

9

Yi(

6)(9

5%)

=Y

c(6)

;2Z

(sy,

x)=

48;

1.96

(7.9

6)a

=Y

-b(

X)

=30

-9(

4)=

-6

=48

;15

.6=

63.6

to

32.4

Yc

=a

+b(

X)

=-

6+

9(X

)

r2=

b1g

xy2

gy2

=

9(90

)

1000

=

0.81

r=

2r2

=0.

90



all individuals.”64 At the same time, randomization may be politically unfeasibleand morally unjustifiable, because some of those who are most in need (or mostcapable) will be eliminated through random selection. For example, limitedresources for job training cannot be allocated randomly among a sample ofunskilled persons, because job training is intended for the most needy. Similarly, alimited number of scholarships cannot be distributed randomly among a sample ofapplicants of varying abilities, because it is the most meritorious student (and notthe most needy one) who is believed to deserve scholarship funds. Althoughrandomization would help determine the effects of personnel training or of scholar-ships on the subsequent behavior of typical or representative members of a targetpopulation, randomization frequently violates principles of need or merit and isthus politically and morally unacceptable.

The advantage of regression-discontinuity analysis is that it allows us tomonitor the effects of providing a scarce resource to the most needy or most

64Campbell, “Reforms as Experiments,” pp. 86–87.

EXHIBIT 1

SPSS Output for Table 17

Model Summary

Model R R SquareAdjustedR Square

Std. Error of theEstimate

1 .900a .810 .747 7.95822426

a. Predictors: (Constant), PROGINV

ANOVAb


1 Regression 810.000 1 810.000 12.789 .037a

Residual 190.000 3 63.333

Total 1000.000 4

a. Predictors: (Constant), PROGINV

b. Dependent Variable: PERCEMP

Coefficientsa



Model B Std. Error Beta t Sig.

1 (Constant) �6.000 10.677 �.562 .613

PROGINV 9.000 2.517 .900 3.576 .037

a. Dependent Variable: PERCEMP



deserving members of a target population that is larger than a given program canaccommodate. Imagine a situation in which large numbers of deserving persons(e.g., those who obtain high scores on a law school entrance examination) are to beselected to receive some scarce resource (e.g., a scholarship to law school). One ofthe aims of the admission policy is to select those persons who will succeed later inlife, with “success” defined in terms of subsequent income. Only the most deservingapplicants are awarded a scholarship, thus satisfying the principle of merit. But themost deserving students would probably be successful in later life even if theyreceived no scholarship. Under these circumstances, it is difficult to determinewhether late success in life is a consequence of receiving a scholarship or of otherfactors, including the family backgrounds of applicants or their social position.Do scholarships affect subsequent success in life?

One way to answer this question is to conduct an experiment (called a “tie-breaking” experiment) in which scarce resources are randomly allocated to a smallnumber of persons who are identical or “tied” in their abilities or level of need. Theeasiest way to visualize a tie-breaking experiment is to imagine five individuals, allof whom have scored 100 on an examination. The problem here is to give awards totwo of the most deserving students. Because all students are equally deserving, someprocedure must be used to break the tie. One such procedure is randomization.

This same logic is extended to social experimentation through regression-discontinuity analysis. In a tie-breaking experiment, we take a narrow band of meritor need (e.g., as determined by entrance examination scores or family income) abovesome point that marks the cutoff between those who qualify for some resource andthose who do not. To illustrate, imagine that this narrow band of ability is between90 and 95 percent correct on a law school entrance examination. Persons with scoresof 89 percent or less will not be given a scholarship, and persons with scores of96 percent or more will be given a scholarship. But persons falling into the 90–95interval (i.e., the narrow band of ability) will be randomly divided into two groups.One group will receive a scholarship and the other will not. Note that this procedurecan be justified only when there are more deserving or needy persons seeking avalued resource than can be accommodated within existing resource constraints.

Without this tie-breaking equipment, we would be forced to choose only the topstudents. Under these conditions, we would expect to find entrance examinationscores to be strongly and positively correlated with subsequent success in life, asshown by the broken line in Figure 14. But by randomly selecting persons from thenarrow band of ability (i.e., the 90–95 interval), admitting some and rejecting others(the exact number depends on how many students can be accommodated in anygiven year), we can find out whether scholarships have an effect on later achieve-ment over and above that of family background and social position. If entrance examination scores do have this augmenting effect, they will be displayedin the form of a discontinuous solid line that separates those who received scholar-ships from those who did not (Figure 14).

Regression-discontinuity analysis is based on principles of correlation andregression. The main difference is that regression-discontinuity analysis requiresthat we compute a regression equation [Yc � a � b(x)] for each of two groups, onethat receives some valued resource while the other does not. The standard error ofestimate (Sy.x), the coefficient of determination (r2), and the simple correlation



coefficient (r) may also be computed for each group. Note that the standard errorsof regression estimates for the experimental and control groups can be convertedinto interval estimates that establish the significance of any discontinuity, for exam-ple, that displayed by the solid line in Figure 14.

Let us now use another hypothetical example to illustrate the application ofregression-discontinuity analysis. Imagine that we want to monitor the effects ofenforcement policies of the Environmental Protection Agency on the reduction of airpollution levels in various cities. Because enforcement is costly, it is not possible tomount programs in every city where pollution is a significant problem. A randomsample of cities is not politically feasible because those areas with the greatest needfor enforcement efforts would be eliminated simply by chance. At the same time, wewant to find out whether enforcement activities, as compared with no enforcementat all, reduce pollution.

Assume that twenty cities with moderate to high pollution levels have beenselected for the enforcement experiment. These cities are ranked according to dataon pollutants provided by the CAMP. The distribution of these cities by an index ofpollution severity (100 � highest) is illustrated in Table 17.

Six cities in the interval between 81 and 85 have been identified as the tie-breakinggroup for purposes of the enforcement experiment. Three of these cities have beenselected at random to receive the enforcement program; the remaining three citiesreceive no program. Because these cities were chosen solely on the basis of chance, it

40

30

20

10

080 85 90 95 100

Examination Scores on WhichScholarship Award Decided

DiscontinuityLa

ter

Ach

ieve

men

t (a

nnua

l inc

ome)

AwardNo Award

Tied Scores inAchievement Band

Tie-Breaking Experiment

Normal Award ProcedureFIGURE 14Tie-breaking experiment and regression-discontinuity analysisSource: Adapted from Donald T. Campbell, “Reforms as Experiments,” in Handbook of Evaluation

Research, vol. 1, ed. Elmer L. Struening and Marcia Guttentag (Beverly Hills, CA: SagePublications, 1975), p. 87.



is unlikely that charges of favoritism will be advanced against policy makers, eventhough pollution severity scores for two of the control cities are slightly above thoseof the experimental cities. Imagine that air pollution levels in all twenty cities (teneach in the experimental and control groups) were monitored by CAMP one yearbefore and one year after the conclusion of the enforcement experiment.Hypothetical results, displayed in Table 18, provide all data necessary to computeregression equations, standard errors of estimates, coefficients of determination, andsimple correlation coefficients. Note that these calculations are made separately forthe experimental and control groups. The SPSS output is displayed in Exhibit 2.

The results of the regression-discontinuity analysis help answer several ques-tions about the outcomes of the enforcement experiment. First, observe the pre- andpost-enforcement means for the experimental and control cities. The pollution levelsfor both groups are lower in the post-enforcement period, even though the controlgroup received no enforcement program. Nevertheless, the difference between thepre- and post-enforcement means is larger for the experimental cities (88.6 � 84.7 � 3.9)than for the control cities (77.5 � 76.7 � 0.8), suggesting that the enforcement 294

TABLE 17

Distribution of Cities by Scores on a HypotheticalIndex of Pollution Severity

Pollution Severity Number of Cities

97 1

95 192 190 189 187 186 1

85 2

84 183 Tie-breaking group 182 1

81 1

80 179 178 176 174 171 167 1

Total 20

Source: Fictitious data.


TAB

LE

18

Res

ults

of

Reg

ress

ion-

Dis

cont

inui

ty A

naly

sis

for

Hyp

othe

tica

l Exp

erim

enta

l and

Con

trol

Cit

ies

Exp

erim

enta

l Cit

ies

City

Pre

-enf

orce

men

tP

ollu

tion

Leve

l(X

)

Pos

t-enf

orce

men

tP

ollu

tion

Leve

l(Y

)x

yx2

y2xy

Y c(Y

�Y c

)2

197

938.

48.

370

.56

68.8

969

.72

92.5

60.

19

295

916.

46.

340

.96

39.6

940

.32

90.6

90.

10

392

873.

42.

311

.56

5.29

7.82

87.8

80.

77

490

871.

42.

31.

965.

293.

2286

.01

0.98

589

840.

4�

0.7

0.16

0.49

�0.

2885

.07

1.15

687

82�

1.6

�2.

72.

567.

294.

3283

.20

1.44

786

83�

2.6

�1.

76.

762.

894.

4282

.27

0.53

885

82�

3.6

�2.

712

.96

7.29

9.72

81.3

30.

45

984

80�

4.6

�4.

721

.16

22.0

921

.62

80.3

90.

15

1081

78�

7.6

�6.

757

.76

44.8

950

.92

77.5

90.

17

� 8

8.6

X�

84.

7Y

00

226.

4020

4.10

211.

8084

7.00

5.74

b=

g(x

y)

g(x

2 ) =

211.

80

226.

40

=0.

936

a

=Y

-b(

X)

=84

.7-

0.93

6(88

.6)

=1.

8

Yc

=a

+b(

X)

=1.

8+

0.93

6(X

)

S y #

x=

C1Y-

Yc2

2

n-

2=

C5.74 8

=0.

86

r2=

bag

xyb

g(y

2 )

= 93

6(21

1.80

)

204.

10

=0.

97

r=

2r2

=0.

98


Con

trol

Cit

ies

City

Pre

-enf

orce

men

tP

ollu

tion

Leve

l(X

)

Pos

t-enf

orce

men

tP

ollu

tion

Leve

l(Y

)x

yx2

y2xy

Y c(Y

�Y c

)2

1185

887.

511

.356

.25

127.

6984

.75

87.2

4.84

1283

845.

57.

330

.25

53.2

940

.15

84.4

1.96

1382

864.

59.

320

.25

86.4

941

.85

83.0

1.00

1480

782.

51.

36.

251.

693.

2580

.20.

04

1579

751.

5�

1.7

2.25

2.89

�2.

5578

.80.

04

1678

770.

50.

30.

250.

090.

1577

.40.

36

1776

75�

1.5

�1.

72.

252.

892.

5574

.61.

96

1874

73�

3.5

�3.

712

.25

13.6

912

.95

71.8

4.84

1971

71�

6.5

�5.

742

.25

32.4

937

.05

67.6

11.5

6

2067

60�

10.5

�16

.711

0.25

278.

8917

5.35

62.0

25.0

0

�77

.5X

�76

.7Y

00

282.

5060

0.10

395.

5076

7.0

51.6

0

b=

g(x

y)

g(x

2 ) =

395.

5

282.

5

=1.

4

a=

Yq-

b(Xq

)=

76.7

-1.

4(77

.5)

=-

31.8

Yc

=a

+b(

X)

=-

31.8

+1.

4(X

)

S y# x

=

C1Y-

Yc2

2

n-

2=

C51.6

0

8=

2.41

r2=

b1g

xy2

g(y

2 )

= 1.

4(39

5.5)

600.

10

=0.

923

r=

2r2

=0.

961



EXHIBIT 2


Model Summary


Std. Error of the Estimate

1 .985a .971 .967 .86302379

a. Predictors: (Constant), PREENF

ANOVAb


1 Regression 198.142 1 198.142 266.030 .000a

Residual 5.958 8 .745

Total 204.100 9

a. Predictors: (Constant), PREENF

b. Dependent Variable: POSTENF

Coefficientsa




1 (Constant) 1.814 5.089 .356 .731

PREENF .936 .057 .985 16.310 .000

a. Dependent Variable: POSTENFModel Summary


Std. Error ofthe Estimate

1 .961a .923 .913 2.40831892

a. Predictors: (Constant), PREENFC

ANOVAb


1 Regression 553.700 1 553.700 95.466 .000a

Residual 46.400 8 5.800

Total 600.100 9

a. Predictors: (Constant), PREENFCb. Dependent Variable: POSTENFC



program contributed to the decrease in pollution. Second, experimental cities have astronger coefficient of determination (r2 � 0.971) and simple correlation coefficient(r � 0.986) than those for control cities (r2 � 0.932, r � 0.961). The standard errorof estimate for the experimental cities (Sy·x � 0.76) is also lower than that for thecontrol cities (Sy·x � 2.27). Hence, there is less error in the estimated relationshipbetween enforcement and pollution levels for the experimental cities.

Finally, there is a partial discontinuity between the regression lines for the twogroups. If we calculate interval estimates for the two tie-breaking cities withidentical pre-enforcement scores of 85, we obtain the following results:

These interval estimates mean that the highest pollution level that is likely to occur95 percent of the time in experimental city 8 is 82.8, and the lowest that is likely tooccur 95 percent of the time in control city 11 is also 82.8. In other words, when wetake the highest and lowest pollution levels that are likely to occur 95 percent of thetime, the experimental city equals the control city. At the same time, the point estimatesare closer for experimental city 10 (77.59) and control city 14 (80.2), indicating thatthe discontinuity is not as great for some cities.

Note that this kind of comparison would not have been possible without regres-sion-discontinuity analysis. By randomly assigning tied scores to experimental and

= 82.8 to 91.7

= 87.2 ; 4.45

Yi(control city 11) = Yc(85) ; 1.96(2.27)

= 79.8 to 82.8

= 81.33 ; 1.49

Yi(experimental city 8) = Yc(85) ; 1.96(0.76)

Coefficientsa




1 (Constant) �31.800 11.131 �2.857 .021

PREENFC 1.400 .143 .961 9.771 .000

a. Dependent Variable: POSTENFCYi(control city 11) = Yc(85) ; 1.96(2.27)

= 87.2 ; 4.45

= 82.8 to 91.7

EXHIBIT 2



control groups, we are able to determine the effects of the enforcement program oncities that are identical or very similar in their pre-enforcement pollution levels.The results of the regression-discontinuity analysis are more readily visualized if wedisplay them in graphic form. In Figure 15, a discontinuity between the two groupsis evident.

Regression-discontinuity analysis is a useful technique for monitoring theoutcomes of social experiments that involve the distribution of some scarceresource. While our hypothetical example illustrates some of the calculationsrequired to apply the technique, it also avoids some of its complications. Theapplication of the technique normally requires a much larger number of cases thanthat used here. For example, as many as two hundred cases might have beenrequired to do the kind of analysis just described. The need for a relatively largesample of cases is one of several conditions that must be satisfied in order to obtainvalid results. These conditions, which include variables that are normally distributed,are discussed in available statistics texts.65


65See, for example, William Mendenhall and James E. Reinmuth, Statistics for Management andEconomics, 3d ed. (North Scituate, MA: Duxbury Press, 1978), ch. 11.

85

90

95

100

80

70

75

60

65

60 65 70 90 95858075 100Pre-enforcement Pollution Scores

� Experimental Cities

Pos

t-en

forc

emen

t P

ollu

tion

Sco

res

(Y)

� Control Cities

FIGURE 15Graphic display of results of regression-discontinuity



CHAPTER SUMMARYThis chapter has provided an overview of thenature and functions of monitoring in policyanalysis, compared and contrasted fourapproaches to monitoring—social systemsaccounting, social auditing, social experimenta-

tion, and research and practice synthesis—anddescribed and illustrated the application oftechniques used in conjunction with these fourapproaches.

REVIEW QUESTIONS1. How is monitoring related to forecasting?2. What is the relationship between ill-structured

problems and approaches to monitoring?3. What is the relationship between monitoring

and the four strategies of policy analysis?

4. Construct constitutive and operational defini-tions for any five of the following action andoutcome variables:

5. The following table lists several policy prob-lems. For five of these problems, provide anindicator or index that would help determine

whether these problems are being solvedthrough government action.

6. The following table reports the number of crimi-nal offenses known to the police per 100,000population. Known offenses are broken downinto two categories—violent crimes againstpersons and crimes against property—from

1985 to 2009. Construct two curved-line graphsthat display trends in crime rates over the period.Label the two graphs appropriately. What dothese graphs suggest about crime as a policyproblem?

7. In the following table are data on the percent-age distribution of family income by quintiles in

1975, 1989, and 2006. Use these data to con-struct three Lorenz curves that depict changes

Program expenditure Equality of educational opportunity

Personnel turnover National security

Health services Work incentives

Quality of life Pollution

Satisfaction with municipal services Energy consumption

Income distribution Rapport with clients

Work alienation School dropouts

Crime Poverty

Energy crisis Fiscal crisis

Inflation Racial discrimination

Crime Rates in the United States: Offenses Known to Police per 100,000Population, 1985–2009

1985 1990 1995 2000 2005 2009

Violent 556 730 685 507 469 430Property 4,651 5,079 4,591 3,618 3,432 3,036

Source: U.S. Department of Justice, Federal Bureau of Investigation.



in the distribution of income in 1975, 1989,and 2006. Label the curves and the two axes.

8. Calculate the Gini concentration ratio for1975, 1989, and 2006 data presented in studysuggestion 7. What do the Lorenz curves andGini coefficients suggest about poverty as apolicy problem? If poverty and other problems

are “artificial” and “subjective,” how valid isthe information displayed by the Lorenzcurves? Why?

9. Policy issue: Should average monthly benefitspaid to mothers under the Aid to Familieswith Dependent Children (AFDC) program beincreased?

Prepare a policy memo that answers this question.Before writing the memo:a. Prepare a purchasing power index for all

years in the series, using 1970 as the base year.b. Convert the average monthly benefits into

real benefits for these years.c. Repeat the same procedures, using 1980 as

the base year.10. Imagine that you are examining the effects of

ten scholarship awards on the subsequent per-formance of disadvantaged youths in college.There are fewer awards than applicants, andawards must be based on merit. Therefore, it isnot possible to provide all disadvantaged stu-dents with awards, nor is it politically feasible toselect students randomly because this conflicts

with the principle of merit. You decide to allo-cate the ten awards according to the followingrules: No student will receive an award withoutscoring at least 86 on the examination; fiveawards will be given automatically to the topstudents in the 92–100 interval; and the remain-ing five awards will be given to a randomsample of students in the 86–91 interval.One year later, you obtain information on thegrade point averages of the ten award studentsand ten other disadvantaged students who didnot receive an award. You have collegeentrance examination scores for all twentystudents. Does the provision of scholarshipawards to disadvantaged students improvesubsequent achievement in college?

Year Average Monthly Benefit Consumer Price Index (1982–84 = 100)1970 $183.13 38.81975 219.44 53.81980 280.03 82.41985 342.15 107.61988 374.07 118.3

Award Group Non-award Group

Examination Scores Grade Point Average Examination Scores Grade Point Average

99 3.5 88 3.297 3.9 89 3.2

Percentage Distribution of Family Personal Income in the United States by Quintiles, 1975, 1989, and 2006

Quintiles 1975 1989 2006

Highest 41.0 46.7 52.1Second 24.0 24.0 20.3Third 17.6 15.9 14.3Fourth 12.0 9.6 9.5Lowest 5.4 3.8 4.7Total 100.0 100.0 100.0

Source: U.S. Bureau of the Census, Current Population Reports, Series P-60.



a. Construct a regression-discontinuity graphthat displays examination scores and gradepoint averages for the award and non-award groups. Use Xs and Os to displaydata points for the experimental (X) andcontrol (O) groups.

b. Construct a worksheet and compute foreach group the values of a and b in theequation Yc � a � b(X).

c. For each group, write the regression equa-tion that describes the relation betweenmerit (examination scores) and subsequentachievement (grade point averages).

d. Compute the standard error of estimate atthe 95 percent estimation interval (i.e., twostandard errors) for each group.

e. Compute r2 and r.f. Interpret information contained in (a)

through (e) and answer the question: Doesthe provision of scholarship awards to dis-advantaged students improve subsequentachievement in college? Justify your answer.

b. Does the slope of the regression line changefor the control group? Why?

c. Does the standard error of estimate changefor the control group? Why?

d. Does r2 and r change for the controlgroup? Why?

e. What do your answers show about theappropriateness of correlation and regres-sion analysis for problems for which pretestand posttest scores are highly correlated?

f. What, then, are the special advantages ofregression-discontinuity analysis?

13. Using at least four threats to validity, constructrebuttals to the following argument: (B) Thegreater the cost of an alternative, the less likelyit is that the alternative will be pursued. (W)The enforcement of the maximum speed limitof 55 mph increases the costs of exceeding thespeed limit. (I) The mileage death rate fell from4.3 to 3.6 deaths per 100 million miles afterthe implementation of the 55 mph speed limit.(C) The 55 mph speed limit (NationalMaximum Speed Law of 1973) has been defi-nitely successful in saving lives. Study Figure13 before you begin.

DEMONSTRATION EXERCISEA simple (bivariate) linear regression equation iswritten as

OR

(1)

When we use this equation to estimate thevalue of a variable in a time series, the equation iswritten as

y = b0 + b1x

Y = a + b(x)

(2)

In Equation 1, the values of the variables y and xare not ordered in time. For example, the priceand age of automobiles would be unrelated to thetime at which price and age were measured.By contrast, Equation 2 expresses price as a func-tion of time, for example, the year in which pricewas measured. The data would be arrayed likethis:

yt = b0 + b1x1

Examination Scores Grade Point Average Examination Scores Grade Point Average

94 3.4 90 3.392 3.0 86 3.092 3.3 91 3.586 3.1 85 3.190 3.3 81 3.089 3.2 79 2.888 3.2 84 3.091 3.4 80 2.6

12. Subtract 0.5 from each student’s grade pointaverage in the non-award (control) group inthe previous problem.a. Does the Y intercept change for the control

group? Why?



Equation 2 is a time-series regression equation. It isfrequently used to forecast the value of a variablein future years, for example, the price of Nissans inthe year 2000.Another type of time-series regression equation hastwo or more independent (predictor) variables, X1t . . . xkt, which are presumed to be causes of a dependent (response) variable, yt, which ispresumed to be the effect (because regressionanalysis has nothing to do with causality, per se,the term presumed is used here). The equation iswritten as

(3)

When using time-series regression analysis, it isimportant to remember the following points:

1. We often want to estimate the effects of a policyintervention on a policy outcome. We do this bycreating a so-called dummy (categorical)variable, which takes the values of 0 before thepolicy intervention and 1 after the policy inter-vention. For example, a dummy variable may besymbolized as x2t when the dummy variable isthe second predictor variable. The first variable istime, x1t, measured in years. The policy interven-tion regression equation would be written:

�

(4)2. In linear regression, we must satisfy assump-

tions of linearity and homoskedasticity. Anadditional assumption must be satisfied in time-series analysis: The observations of y in the timeseries must be independent (uncorrelated). Thisis called the non-autocorrelation assumption.

b2x2t(policy intervention)

yt(policy outcome) = b0 + b1x1t(time)

yt = b0 + b1x1t + b2x2t + bkxkt

Note that, just as there are tests for linearity(e.g., plotting the y values against normal scoresin a normal probability plot), there are tests forautocorrelation. One of these is the Durbin–Watson (D–W) test. We can apply this test withSPSS and other statistical packages.

3. If the autocorrelation coefficient, r, is statisti-cally significant at a specified level of α (usu-ally p = 0.05), we reject the null hypothesis thatadjacent observations in a time series areuncorrelated—and we thereby accept the alter-native hypothesis that adjacent observationsare autocorrelated. When there is statisticallysignificant autocorrelation, we often can elimi-nate most of its effects by regressing the valuesof yt on their lagged values, yt�1. This is a lagof one time period (e.g., one year). The laggedvalues (one or more time periods) of a variablecan be easily computed with SPSS and otherstatistical packages.

The regression equation with a lagged dependentvariable (in this case, one year) is written as follows:

(5)

Equations 4 and 5 can be combined to express theeffects of a lagged policy outcome variable (yt�1),time (xt1), and a policy intervention (xt2) on apolicy outcome variable (yt). Here is the combinedequation:

(6)

In this demonstration exercise, we will be using adata file from the Fatal Accident Reporting System of

+ b3x3t(policy intervention)

+ b2x2t(time)

yt = b0 + b1yt-1(lag 1 period)

yt = b0 + b1yt-1

CASE PRICE (Yt) YEAR (Xt)

12

$10,00010,200

19851986

Note: Years may be coded as 1, 2, . . . ,T,or as �1, �2, . . . , 0, 1, 2, . . . ,T.

3 10,300 19874 10,600 19885 10,500 19896 11,100 19907 11,100 19918 11,200 19929 11,500 1993



BIBLIOGRAPHYBartlett, Robert V., ed. Policy through Impact

Assessment: Institutionalized Analysis as a PolicyStrategy. New York: Greenwood Press, 1989.

Campbell, Donald T. Methodology andEpistemology for Social Science: Selected Papers,ed. E. Samuel Overman. Chicago: University ofChicago Press, 1989.

Cook, Thomas D., and Charles S. Reichardy, eds.Qualitative and Quantitative Methods inEvaluation Research. Beverly Hills, CA: SagePublications, 1979.

Light, Richard J., and David B. Pillemer. SummingUp: The Science of Reviewing Research.Cambridge, MA: Harvard University Press,1984.

MacRae, Duncan Jr. Policy Indicators: Linksbetween Social Science and Public Debate.Chapel Hill: University of North CarolinaPress, 1985.

Miles, Matthew B., and A. Michael Huberman.Qualitative Data Analysis: A Sourcebook ofNew Methods. Beverly Hills, CA: SagePublications, 1984.

Shadish, William, Thomas D. Cook, and DonaldT. Campbell. Experimental and Quasi-Experimental Design for External Validity.Boston, MA: Houghton Mifflin, 2002.

Trochim, William M. K. Research Design forProgram Evaluation: The Regression-Discontinuity Approach. Beverly Hills, CA: SagePublications, 1984.

U.S. General Accounting Office. DesigningEvaluations. Methodology Transfer Paper 4.Washington, DC: Program Evaluation andMethodology Division, U.S. GeneralAccounting Office, July 1984.

Yin, Robert K. Case Study Analysis. Beverly Hills,CA: Sage Publications, 1985.

the National Highway Traffic Safety Administrationto perform the tasks described next:1. Estimate the effect of time on fatalities before

(1966–73) and after (1974–2000) the adoptionof the 55 mph speed limit. Run two separateregressions. Interpret the two regression esti-mates. In this part of the exercise, do not usedata after 2000.

2. Estimate the effect of employment on fatalities.On the basis of the 1966–90 series, forecasttraffic fatalities for 2009. The actual value offatalities in 2009 is 33,808. Interpret the com-puter printout and compare the forecast withactual fatalities for 2009.

3. Estimate the effect of time (years) and the policyintervention (55 mph speed limit) on fatalities,using a dummy variable for the intervention.Assess whether the effect of the policy interven-tion is statistically significant and interpret thecomputer output.

4. Reestimate the equation in part 3. This time,control for the effect of autocorrelation byadding Yt–1 (the lagged value of the dependentvariable) as a third predictor. Interpret the outputand compare it to output obtained in part 3.

5. Estimate the effect of the policy intervention onfatalities after including miles traveled andemployment as additional predictor variables. Inthis regression, do not include time as a predictorvariable. Interpret your computer output.

6. Use the data in Exhibit 4 to create interruptedtime series and control series graphs for (a) theUnited States (US); (b) for European countriesthat adopted 48 and 54 mph (80 and 90 kph)speed limits (EUREXP); and (c) for Europeancountries that did not adopt a speed limit(EURCON). Interpret the graphs and indicatehow your analysis answers the question: Didthe speed limit cause the decline in fatalities?


EXHIBIT 3

CASE 1 UNDERSTANDING POLICY OUTCOMES:THE POLITICAL ECONOMY OF TRAFFICFATALITIES IN EUROPE AND THE UNITED STATES

YEAR FAT BIMI UEMR EMP POP IIP

1966 50894 926 3.2 72895000 195578000 32.501967 50724 964 3.1 74372000 197457000 34.301968 52725 1016 2.9 75920000 199399000 35.401969 53543 1062 2.8 77902000 201385000 37.301970 52627 1110 4.4 78627000 203984000 36.901971 52542 1179 5.3 79120000 206827000 36.501972 54589 1260 5.0 81702000 209284000 39.001973 54052 1313 4.2 84409000 211357000 43.00

The 55 mph speed limit was adopted in 1974 as ameans to reduce gasoline consumption during the1973–74 OPEC oil embargo. Unexpectedly, thepolicy was followed by a sharp decline in trafficfatalities. Between January 1, 1974, and December31, 1975, there was a decline of 9,100 fatalities, a16.8 percent drop. Despite continuing oppositionfrom rural Western states and the trucking industry,there was broad political support for the policy. And,clearly, the problem is not unimportant or trivial.Traffic fatalities represent the leading cause of deathamong persons age 35 and younger. Average annualhighway deaths since 1974 are equivalent to a fullyloaded 767 aircraft crashing with no survivors everythird day of the week.

On April 2, 1987, Congress enacted the SurfaceTransportation and Uniform Relocation AssistanceAct of 1987, overriding President Reagan’s veto.Provisions of this bill permitted individual states toexperiment with speed limits up to 65 mph on ruralinterstate highways. By July 1988, forty states hadraised the speed limit to 60 or 65 mph on 89 percentof rural interstate roads. Senator John C. Danforth,an influential supporter of the 55 mph speed limit,argued against the new policy. The 65 mph speedlimit, said Danforth, would save an average of oneminute per day per driver but result in an annual

increase of 600 to 1,000 deaths. The Washington Post

joined the opponents: “The equation is in minutesversus lives. It’s not even close. . . . A hundred milesat 55 mph take about 17 minutes longer than at 65.That’s the price of those lives.It ought to be the easiest vote the House takes thisyear.” Eight years later, in a November 1995 pressrelease, U.S. Secretary of Transportation FedericoPena reaffirmed the Clinton administration’sopposition to the higher speed limits (see Chapter 1,Case 1).The National Maximum Speed Limit wasofficially abandoned the same year.

Many analysts who have evaluated the effects ofthe intervention, along with elected officials from theten northeastern states that retained the 55 mphspeed limit until its repeal, affirm that the 1974 lawwas responsible for the decline in traffic fatalities.However, the evidence suggests that they may havefailed to consider rival hypotheses—which had theybeen identified and tested would have resulted in adifferent explanation of traffic deaths and,consequently, a different policy recommendation.

Here are data on U.S. traffic fatalities and othervariables for 1966 to 2010 (Exhibit 3). In addition,fatality rates in the United States and Europeancountries are provided in a separate listing (Exhibit 4).Use the data for the demonstration exercise.



YEAR FAT BIMI UEMR EMP POP IIP

1974 45196 1281 4.9 85936000 213342000 44.401975 44525 1328 7.9 84783000 215465000 40.601976 45523 1402 7.1 87485000 217561000 42.20

1977 47878 1467 6.3 90546000 219758000 44.701978 50331 1545 5.3 94373000 222093000 47.001979 51093 1529 5.1 96945000 224569000 50.501980 51091 1527 6.9 97722000 227255000 51.101981 49301 1553 7.4 98878000 229637000 50.501982 43945 1595 9.9 98047000 231996000 48.701983 42589 1653 9.9 99347000 234284000 47.701984 44257 1720 7.4 103413000 236477000 53.001985 43825 1774 7.0 105613000 237924000 54.401986 46087 1835 6.9 107860000 240133000 55.701987 46390 1921 6.2 110634000 242289000 55.901988 47087 2026 5.5 113153000 244499000 60.101989 45582 2096 5.2 115453000 246819000 61.701990 44599 2144 5.6 116660000 249403000 61.201991 41508 2172 7.0 115657000 252138000 60.701992 39250 2247 7.8 116395000 255039000 60.901993 40150 2297 7.2 118149000 257800000 63.901994 40716 2360 6.1 121999000 260350000 66.101995 41817 2423 5.6 124766000 262755000 70.401996 41907 2469 5.4 125311000 265284000 71.601997 42013 2560 4.9 129558000 267744000 76.501998 41471 2619 4.5 131463000 270299000 82.901999 41611 2691 4.2 133488000 272691000 86.102000 41945 2750 4.0 135208000 282172000 90.202001 42196 2797 4.7 136933000 285082000 90.702002 43005 2856 5.8 136485000 287804000 86.802003 42884 2890 6.0 137736000 290326000 89.502004 42836 2965 5.5 139252000 293046000 90.402005 43510 2989 5.1 141730000 295753000 94.102006 42708 3014 4.6 144427000 298593000 95.502007 41059 3030 4.6 146047000 301580000 97.502008 37423 2926 5.8 145362000 304375000 99.802009 33808 2991 9.3 139877000 307007000 87.402010 31000 2980 9.6 132000000 308400408 87.60

Note: FAT � fatalities; BIMI � billions of miles traveled; UEMR � unemployment rate;EMP � employment; POP � population; IIP � Index of Industrial Production.



YEAR US EUREXP EURC UK FIN FR DEN

1970 25.8 21.9 24.3 14.3 24.8 23.5 251971 25.4 22.8 24.7 14.8 25.9 25.4 251972 26.1 22.8 25.0 14.6 25.2 27.4 241973 25.6 22.1 23.5 14.7 23.8 25.6 241974 21.2 17.4 21.3 13.0 18.0 22.5 161975 20.1 17.6 20.2 11.9 19.0 24.0 171976 20.9 18.3 18.8 12.0 20.0 26.0 17

Note: EUREXP � European “experimental” states that adopted a 90 kph (54 mph) speed limit in 1972;EURCON � European “control” states. Other names refer to countries.

EXHIBIT 4


O B J E C T I V E S


Evaluating PolicyPerformance

� Distinguish methods of monitoringand evaluation.

� List the main characteristics ofpolicy evaluation.

� Identify and illustrate criteria forevaluating policy performance.

� Contrast evaluation and meta-evaluation.

� Compare values, ethics, and meta-ethics.

� Analyze ethical and economic issuesrelated to social equity and justice.

The primary aim of monitoring is to make sound claims about the conse-quences of policies. Monitoring is about facts; evaluation is about facts andvalues. Monitoring answers the question: Did the policy produce its intended

outcome? Evaluation answers a related but distinct question: Of what value is theoutcome?

In this chapter, we consider the policy-analytic method of evaluation by firstreviewing several ways in which ethics and values are important for public policy.Frameworks for thinking about values, ethics, and meta-ethics (the philosophicalstudy of ethical reasoning) are then introduced. We go on to examine the nature,aims, and functions of evaluation in policy analysis, showing how evaluation as aform of ethical appraisal helps produce information about policy performance. Bycontrast, monitoring as a type of causal appraisal helps produce informationabout policy outcomes. We then compare and contrast three approaches to evalu-ation and examine methods and techniques employed in conjunction with theseapproaches.



Evaluating Policy Performance

ETHICS AND VALUES IN POLICY ANALYSISSome analysts believe that policy evaluation should concentrate on explanation andprediction, treating ethics and values as an afterthought. Policy disagreementswould diminish or disappear, argue “predictivists,” if analysts could explain andpredict with greater certainty. There is no point spending much time on differencesin basic values, because they are subject to dispute and difficult to resolve.1

This view of policy analysis, which is associated with the philosophy of sciencecalled logical positivism, makes a sharp separation between facts and values. Theacquisition of facts is seen as positive, because facts affirm what is, as distinguishedfrom speculations about what ought to be. Logical positivism2 was developed inEurope in the 1920s by a group of physicists and philosophers of science called theVienna Circle. The original Vienna Circle positivists, and later their followers inthe social sciences, believed that questions of ethics and morality were not propersubjects of science. Because observation, explanation, and prediction are deemedto be the only legitimate aims of science, ethical and value judgments are seen asunscientific, extrascientific, or antiscientific. Science and values should be keptstrictly apart. In the social sciences, analysts should confine their work to explainingand predicting policy outcomes, limiting themselves to the observation and analysisof what is rather than what ought to be.

Thinking About ValuesPositivism was abandoned by philosophers in the mid-1950s. It is no longerregarded as a tenable model of science. The consensus today is that science, which issubject to a great many social, cultural, economic, political, and psychologicalinfluences, does not work the way positivists said it does. In this context, analystsneed to look critically at the relation (not separation) between facts and values. Forthis reason, it is important to understand what we mean when we use the termsvalues and ethics.

Values may be defined in a comprehensive way, so that it includes any wants,needs, interests, or preferences.3 Here values represent “objects” along a broadspectrum of behaviors involving choice, for example, “He prefers guns to butter” or“She wants guns more than butter.” In this view, which refers to values-as-objects(e.g., the preference for efficiency), values are anything of interest to those whomake evaluations. Given this broad conception, policy analysts can be seen toaddress a sweeping array of values. It is in this broad sense that policy choicesinvolve “the authoritative allocation of values.”4

A second, more specific conception of values refers to values-as-criteria. Herethe concern is not only with values as objects but also with the criteria usedto choose values. This second meaning of values incorporates decision rules or

1See Milton Friedman, Essays in Positive Economics (Chicago: University of Chicago Press, 1953), p. 5.Friedman is speaking about “positive economics.”2Logical positivism is also called logical empiricism, and sometimes scientific empiricism.3Stephen C. Pepper, The Sources of Value (Berkeley: University of California Press, 1958).4This definition, well known to political scientists, is that of David Easton in The Political System(Chicago: Alfred Knopf, 1953).



criteria employed to choose between two or more values. Whereas a broaddefinition of values leads to statements such as “The majority of citizens wantprograms to be more efficient,” a narrower definition specifies the criterion orrule: “The majority of citizens want programs according to the rule: Select theprogram that is more efficient because it produces the greatest net benefit (i.e.,benefits minus costs).”

These two usages of the term value are related to the different contexts— personal, standard, and ideal—in which we understand values.5 In the personalcontext, values are expressed in the form of preferences, wants, and needs. A valueexpression is “I prefer public to private schools.” By contrast, the standard contextinvolves statements about an individual or group that holds certain values, forexample, “School busing for racial integration is a good policy in the eyes of middle-class citizens.” Finally, the ideal context involves value judgments that are notreducible to value expressions or value statements.

Value judgments are distinctive because they justify values such as efficiencyexpressed with the rule: Select the policy that maximizes net benefits. Accor-dingly, it is the role of value judgments to justify such decision rules, placing themin their proper context as instances of ethical argumentation. For example,“Policy A is preferable to policy B, because it produces the greatest net benefit,thereby increasing the aggregate satisfaction of members of the community—that is, the greatest good for the greatest number.” This value judgment goesbeyond the description of personal preferences and the values of a particulargroup, for example, rational choice theorists who may state uncritically that“Maximize net benefits!” is a self-evident moral imperative, when in fact it is arule requiring ethical justification. Although we can describe values, we can andshould supply grounds for their justification—this is the special role of valuejudgments.6

In addition to differences in the contexts of values, the following distinctions arealso important:

� Value versus values. Because people express values that we can describe andexplain, it does not follow that such value expressions refer to things thatare of value. The value of something cannot be established on the basis ofobservation alone, because it is always possible to ask: Are these values good?Consider the quip attributed to Oscar Wilde: “An economist is a man whoknows the price of everything and the value of nothing.”

� Values versus needs. Although values may be a source of needs, as when thevalue of health justifies attempts to satisfy unmet needs for minimumnutritional standards, values are not the same as needs. A perceived deficiencythat gives rise to a need (e.g., nutrition as a material need) may be relativelyindependent of such values as security, order, justice, and equality. Complexrelationships between needs and values create conceptual and methodologicalissues for analysts who conduct “needs assessments,” because needs are notthe same as values.

5Abraham Kaplan, The Conduct of Inquiry (San Francisco, CA: Chandler, 1964), pp. 387–97.6Ibid., pp. 387–89.



� Values versus norms. Whereas norms are rules of conduct applicable inspecific contexts, values are standards of achievement applicable in many or allcontexts. Values of power, respect, rectitude, affection, well-being, wealth, andenlightenment7 may each result in the establishment of separate sets of normsintended to regulate relationships between analyst and politician, analyst andanalyst, analyst and journalist, analyst and client, or analyst and the public atlarge. Conversely, the norm of public accountability may involve power,respect, well-being, and other values. Although values provide grounds forjustifying norms, the more general the norm, the more difficult it is todistinguish it from a value. Codes of conduct for regulating the behavior ofplanners, policy analysts, and public managers contain norms that aresufficiently general to be virtually indistinguishable from values.8

� Valuation versus evaluation. Another important distinction is betweenvaluation and evaluation.9 Evaluation as a general social process involvingmany forms of appraisal should be distinguished from valuation, a specialprocess involving efforts to establish the grounds on which appraisals are made.Whereas evaluation requires that two or more alternatives be appraised—forexample, according to the criterion of efficiency improvement—valuationrequires that the criterion itself be justified, for example, in terms of theutilitarian philosophy of the greatest good for the greatest number. Disputessurrounding the aims and nature of policy and program evaluation often turnon the distinction between valuation and evaluation.10

� Valuation versus prescription. Value judgments, which are products ofvaluation, should not be confused with nonrational emotional appeals,ideological exhortations, or doctrinaire forms of policy advocacy. Thisconfusion is related to the notion that policy analysis must be valueneutral,11 for so long as valuation is mistakenly believed to be subjectiveand nonrational, value neutrality is the only protection against analysesthat are seen as unscientific and arbitrary. Many analysts fail to recognizethat value judgments are not “commands, prescriptions, or other forms oftelling people to do things . . . telling people what they should do is nottelling or exhorting them to do it . . . it is purporting to give them soundsolutions to their practical problems.”12

7See Harold D. Lasswell and Abraham Kaplan, Power and Society (New Haven, CT: Yale UniversityPress, 1950).8For illustrations, see Elizabeth Howe and Jerome Kaufman, “Ethics and Professional Practice inPlanning and Related Policy Professions,” Policy Studies Journal 9, no. 4 (1981): 585–94.9See John Dewey, “Theory of Valuation,” International Encyclopedia of Unified Science 11, no. 4(1939).10See, for example, Ernest House, Evaluating with Validity (Beverly Hills, CA: Sage Publications, 1980);and Deborah Stone, “Conclusion: Political Reason,” in her Policy Paradox: The Art of PoliticalDecision Making (New York: W. W. Norton, 2002), pp. 384–415.11Edith Stokey and Richard Zeckhauser, A Primer for Policy Analysis (New York: W.W. Norton, 1978,pp. 4–5.12Kurt Baier, “What Is Value? An Analysis of the Concept,” in Values and the Future, ed. Kurt Baierand Nicholas Rescher (New York: Free Press, 1969), p. 53.



Ethics and Meta-ethicsEthics refers to the reflective study of moral choice, whereas meta-ethics refers tothe reflective study of ethics itself. The term ethics is often applied to habitual orcustomary behavior, as in the “ethics of policy makers.” This everyday usagederives etymologically from the Greek and Latin ethos and mores, which refer tohabits or customs. Although it is primarily this everyday usage that has guided thedevelopment of ethical codes for public managers, planners, and policy analysts,13

these efforts at professional standard setting are based on implicit ethical and moraljudgments:

Even when “customary morality” is spoken of, the reference of the term is notmerely to the customs as such—in the sense of regular, repeated sequences ofbehavior—but also to the view, at least implicitly held by the participants, thatwhat they regularly do is in some way right; it is not merely what is done, it isalso what is to be done.14

As can be seen, the term has two different but related meanings. Descriptive ethics,which involves the analysis of descriptions of customary morality, should becontrasted with normative ethics, which is concerned with the analysis, evaluation,and development of normative statements that provide guidance to those trying tosolve practical problems.

Normative ethics is concerned with the appraisal of criteria for justifyingnormative statements. The central questions of normative ethics are of the form:What criteria justify normative claims about the rightness, goodness, or justice ofpublic actions? By contrast, meta-ethics is concerned with the nature and meaningof normative claims in general. Questions of meta-ethics are of the form: What(meta-ethical) criteria warrant the choice of (normative ethical) criteria employed tojustify claims about the rightness, goodness, or justice of public policies? In this andother cases, the term meta signifies something that is “about” or “of ” somethingelse. “Meta-policy,” for example, is making policy about policies,15 and “meta-evaluation” is the evaluation of evaluations, because it develops and applies criteriato evaluate program and policy evaluations.16

There has been a growing concern with the role of normative ethics and meta-ethics in public policy analysis. An emphasis on the reflective evaluation of criteriaemployed to justify normative claims may be found in the general literature ofpolicy analysis.17 In addition, there have been specialized attempts to examine

13For example, Herman Mertins, Professional Standards and Ethics: A Workbook for PublicAdministrators (Washington, DC: American Society for Public Administration, 1979).14Louis A. Gewirth, Reason and Morality (Chicago: University of Chicago Press, 1978), p. 67.15Yehezkel Dror, Public Policymaking Reexamined (New York: Elsevier, 1968).16Thomas D. Cook and Charles Gruder, “Metaevaluation Research,” Evaluation Quarterly 2 (1978):5–51. MacRae has developed an important original application of meta-ethics to the normative ethics ofpolicy analysis. Duncan MacRae Jr., The Social Function of Social Science (New Haven, CT: YaleUniversity Press, 1976).17For example, MacRae, The Social Function of Social Science; Frank Fischer, Evaluating Public Policy(Chicago: Nelson-Hall, 1995).



meta-ethical questions that are ontological (Does ethical knowledge exist?), episte-mological (What criteria govern the truth or falsity of ethical claims?), and practi-cal (What is the relation between ethics and moral action?). Michalos, forexample, argues that widely accepted but mistaken assumptions about the exis-tence of two mutually exclusive realms of experience (“facts” and “values”) leadanalysts to the false conclusion that ethical claims are “scientificallyincorrigible.”18 Drawing on critical theory and the work of Jurgen Habermas andother members of the Frankfurt School, Dallmayr argues that “policy evaluationrequires a critically reflective ‘practical discourse’ open not only to experts orpolicy analysts but to the public at large . . . recovery of a fully non-instrumental‘practical’ judgment presupposes an evaluation not only of concrete policies but ofthe status of ‘policy’ itself.”19

Distinctions among descriptive ethics, normative ethics, and meta-ethics are par-ticularly applicable to issues of social equity and justice. Issues of distributive justice—for example, disagreements about the appropriate rule for distributing income—areoften addressed by describing the consequences of adopting one or more of thefollowing neo-utilitarian decision criteria:

� Net efficiency improvement: Maximize total benefits minus total costs.� Pareto improvement: Maximize total benefits minus total costs up to that

point at which it is not possible to make any person better off without alsomaking another person worse off.

� Kaldor–Hicks improvement: Also called virtual Pareto improvement, maximizetotal benefits minus total costs, provided that winners may in principlecompensate losers.

At the level of descriptive ethics, the satisfaction of these criteria can beaddressed by calculating a measure of inequality such as the Gini index, whichmeasures differences in the distribution of household income. By contrast, at thelevel of normative ethics, the justifiability of any given distribution of income orwealth may be addressed by normative theories of justice, for example, John Rawls’stheory of justice-as-fairness.20 In a pre–civil society in which discrepancies of power,wealth, and privilege are as yet unknown—a society assumed to be under aRawlsian “veil of ignorance”—citizens would accept a decision criterion differentfrom those mentioned earlier. The Rawlsian decision criterion is to maximize thewelfare of members of society who are worst off. The justifiability of this criterioncan be challenged at a more basic, meta-ethical level. In this context, the Rawlsiantheory has been challenged on meta-ethical grounds—that it presupposes an individ-ualistic conception of human nature with which normative commitments are no

18Alex C. Michalos, “Facts, Values, and Rational Decision Making,” Policy Studies Journal 9, no. 4(1981): 544–51.19Fred R. Dallmayr, “Critical Theory and Public Policy,” Policy Studies Journal 9, no. 4 (1981): 523.Daneke makes a general case for the introduction of meta-ethics into the curricula of schools of publicpolicy and management. Gregory Daneke, “Beyond Ethical Reductionism in Public Policy Education,”in Ethics, Values, and the Practice of Policy Analysis, ed. William N. Dunn (Lexington, MA: D. C.Heath, 1982).20John Rawls, A Theory of Justice (Cambridge, MA: Harvard University Press, 1968).



longer products of reasoned social discourse. For Rawls, normative commitmentsare

the contractual composite of arbitrary (even if comprehensible) values individu-ally held and either biologically or socially shaped. . . . The structure of theRawlsian argument thus corresponds closely to that of instrumental rationality;ends are exogenous, and the exclusive office of thought in the world is to ensuretheir maximum realization. . . . [Rawls’s arguments] reduce all thought tothe combined operations of formal reason and instrumental prudence in theservice of desire.21

Standards of ConductThe standards of conduct or norms of “customary morality” that guide the behav-ior of analysts are variable. In part, this variability may be attributed to the diverseapproaches to teaching ethics and values in schools of public policy and manage-ment.22 Government policy analysts and planners are themselves divided alongmethodological as well as ethical lines.23 Efforts to reduce this variability havedrawn on several sources of standards: social values and norms, including equity,honesty, and fairness, which may guide the actions of policy analysts as well ascitizens at large; scientific values and norms, including objectivity, neutrality, andinstitutionalized self-criticism; professional codes of conduct, including formallystated obligations, duties, and prescriptions; and legal and administrative proce-dures, including mechanisms for securing informed consent by seeking the formalapproval of institutional review boards.

The communication of social values and norms is central to the research andteaching activities of individual scholars, whereas the communication of scientificand professional norms has occurred largely through the curriculum developmentefforts of the National Association of Schools of Public Affairs and Administration(NASPAA) and through the traditional conference and publications activities of thePolicy Studies Organization (PSO), the Association for Public Policy andManagement (APPAM), and the American Society for Public Administration(ASPA). In addressing the need for a code of professional conduct, but stoppingshort of the formal codification of standards, ASPA has published and disseminateda pamphlet titled Professional Standards and Ethics: A Workbook for PublicAdministrators. By contrast, legal and administrative procedures to regulate theconduct of research involving human subjects and research integrity generally areproducts of government initiatives rather than those of professional associations oruniversities.

21Laurence H. Tribe, “Ways Not to Think about Plastic Trees,” in When Values Conflict, ed. L. H. Tribe, C. S. Schelling, and J. Voss (Cambridge, MA: Ballinger, 1976), p. 77.22Joel L. Fleishman and Bruce L. Payne, Ethical Dilemmas and the Education of Policymakers(Hastings-on-Hudson, NY: Hastings Center, 1980).23Elizabeth Howe and Jerome Kaufman, “The Ethics of Contemporary American Planners,” Journalof the American Planning Association 45 (1979): 243–55; and “Ethics and Professional Practice inPlanning and Related Planning Professions.”



DESCRIPTIVE ETHICS, NORMATIVE ETHICS,AND META-ETHICSTheories of ethics and values may be conveniently classified according to theirfunctions: the description, classification, and measurement of ethics and values(descriptive theories); the development and application of criteria to assess ethicalbehaviors (normative theories); and the development and application of additionalcriteria to assess normative ethical theories themselves (meta-ethical theories). Thisclassification has important subdivisions—for example, teleological versus deonto-logical normative theories, or cognitivist versus non-cognitivist meta-ethical theories.

Descriptive Value TypologiesThe work of Milton Rokeach24 represents a major synthesis of descriptive theoriesof values developed since the 1930s. The scope and depth of his concern is evidentin the claim that

the concept of values, more than any other, is the core concept across all thesocial sciences. It is the main dependent variable in the study of culture, soci-ety, and personality, and the main independent variable in the study of socialattitudes and behavior. It is difficult for me to conceive of any problem socialscientists might be interested in that would not deeply implicate humanvalues.25

One of Rokeach’s contributions is the development of a basic typology for develop-ing and testing descriptive theories of values. The basic typology has two major dimensions: terminal and instrumental values. Terminal values, which are both per-sonal and social, are beliefs about desirable end-states of existence. Instrumentalvalues are beliefs about desirable modes of conduct (Table 1).

Howe and Kaufman employed a similar typology in their study of the ethicsof professional planners.26 Distinguishing between “ends-oriented” and “means-oriented” ethics regarding the importance of a large number of values reported inexisting literature, Howe and Kaufman draw two classes of ethical principlesfrom the 1962 Code of Professional Responsibility and Rules of Procedure of theAmerican Institute of Planners. Among the major findings of the study are thatprofessional and personal ethical norms are often inconsistent and that a commit-ment to certain ends (e.g., social equity) affects ethical judgments about the use ofparticular means (e.g., leaking information to low-income groups). Although thisstudy is essentially exploratory, it addresses important competing principles:good ends justify the selection of means27 versus means should be justified intheir own terms.28

24Milton Rokeach, The Nature of Human Values (New York: Free Press, 1973).25Ibid., p. ix.26Howe and Kaufman, “The Ethics of Contemporary American Planners.”27See Saul Alinsky, Rules for Radicals (New York: Random House, 1971).28See Sissela Bok, Lying: Moral Choice in Public and Private Life (New York: Pantheon, 1978).



TABLE 1

Basic Value Typology:Terminal and Instrumental Values

Terminal Value Instrumental Value

A comfortable life AmbitiousAn exciting life BroadmindedA sense of accomplishment CapableA world at peace CheerfulA world of beauty CleanEquality CourageousFamily security ForgivingFreedom HelpfulHappiness HonestInner harmony ImaginativeMature love IndependentNational security IntellectualPleasure LogicalSalvation LovingSelf-respect ObedientSocial recognition PoliteTrue friendship ResponsibleWisdom Self-controlled

Source: Milton Rokeach, The Nature of Human Values (New York: Free Press,1973),Table 2.1, p. 28. Reliability coefficients and parenthetical qualifiershave been omitted.

Developmental Value TypologiesBasic typologies can serve as a basis for examining theories of individual andcollective value change and development. For example, Rokeach tests a theory ofcognitive inconsistency with data organized according to the typology of terminaland instrumental values.29 He also identifies a variety of developmental patternsbased on age differences, concluding that values change not only during adolescencebut also throughout life.30 These patterns are related to the stage theory of moraldevelopment created and tested by Kohlberg.31 Kohlberg’s developmental typologydistinguishes three levels in the development of moral reasoning: preconventional,conventional, and postconventional. Because each level has two stages, the typologyyields six stages in all: stage 1 (punishment and obedience orientation), stage 2(instrumental relativist orientation), stage 3 (interpersonal concordance orientation),stage 4 (law-and-order orientation), stage 5 (social-contract legalistic orientation),and stage 6 (universal ethical principle orientation).

29Rokeach, Nature of Human Values, pp. 215–34.30Ibid., p. 81.31Lawrence Kohlberg, Stages in the Development of Moral Thought and Action (New York: Holt,Rinehart, and Winston, 1961).



Kohlberg’s developmental typology is of great potential importance for assess-ing the role of ethical norms and principles in policy analysis. The six stages areclaimed to represent an invariant sequence whereby individuals progress in theircapacity for moral judgment. Because the sequence is invariant and sequential,developmental transitions can occur only between adjacent stages. Accordingly,individuals cannot be expected to move directly from an egocentric market society(stage 2) to a society based on universal ethical principles (stage 6), for example,from a concern with an ethic of income maximization to an ethic of the typeoutlined by Rawls. Because Kohlberg’s theory is based on relatively firm empiricalgrounds, it has an important bearing on questions surrounding possibilities fornormative ethical discourses in contemporary society: “Any conception of whatmoral judgment ought to be must rest on an adequate conception of what is.The fact that our conception of the moral ‘works’ empirically is important for itsphilosophic adequacy.”32

Normative TheoriesNormative theories of value, or normative ethics, may be divided into three maintypes: deontological, teleological, and practical. Deontological normative theoriesclaim that certain kinds of actions are inherently right or obligatory (the Greekdeontos means “of the obligatory”), or right because they conform to some formalprinciple. Teleological normative theories, by contrast, hold that certain actions areright because they result in good or valuable ends (the Greek teleios means “broughtto its end or purpose”). Finally, practical normative theories hold that certain actionsare right because they conform to principles, or result in consequences, whose right-ness or goodness has been established through reasoned discourses or transactions(the Greek praktikos means “to experience, negotiate, or transact”) among thosewho affect and are affected by the creation and application of moral rules.

Deontological normative theories rest on “deontic” concepts and principles, asdistinguished from concepts and principles based on the goodness of consequences.Theories of justice such as that offered by Rawls are mainly deontological becausethey employ formal principles of obligation (a pre-civil social contract executedunder a “veil of ignorance”) to justify arguments about distributive justice (“justiceas fairness”) and a just society (a society that maximizes the welfare of those whoare worst off). By contrast, teleological theories evaluate actions according to thegoodness of their consequences. A prominent form of teleological theory, one thathas deeply affected policy analysis through modern welfare economics, is utilitarian-ism. The classical utilitarian theories of Jeremy Bentham and John Stuart Millheld that right actions are those that promote the maximum or greatest good foreveryone. Policy analysts who use cost-benefit analysis are utilitarians to the degreethat they base policy recommendations on the criterion of maximizing net income

32Lawrence Kohlberg, “From Is to Ought: How to Commit the Naturalistic Fallacy and Get Away withIt in the Study of Moral Development,” in Cognitive Development and Epistemology, ed. T. Mischel(New York: Academic Press, 1971), pp. 151–235. The “naturalistic fallacy” is the fallacy of deriving an“ought” from an “is.”



benefits, a criterion presumed to reflect the aggregate satisfaction experienced bymembers of society.

The main properties of teleological theories, including modern neo-utilitariantheories, are most easily visualized by comparing them with deontological theories:33

� Teleological theories justify actions because of their consequences (e.g.,maximization of net income benefits), whereas deontological theories justifyactions because they conform to some principle (e.g., procedural fairness) orbecause they are deemed to be inherently right (e.g., truth telling).

� Teleological theories set forth conditional obligations (e.g., freedom can becompromised to achieve greater national security), whereas deontologicaltheories set forth absolute obligations (e.g., freedom is a generic right thatcannot be compromised.).

� Teleological theories advance material or substantive criteria (e.g., pleasure,happiness, or satisfaction), whereas deontological theories advance formal orrelational criteria (e.g., social equity or administrative impartiality).

� Teleological theories supply aggregate criteria (e.g., social welfare as the totalsatisfaction experienced by members of a community), whereas deontologicaltheories supply distributive criteria regulating the allocation of goods (e.g.,educational opportunities or a living wage) or ills (e.g., exposure to toxicwastes and carcinogens).

Practical normative theories stress reflective moral action as a process fordiscovering the criteria on which values and ethics may be known. Practical norma-tive theories have stressed reasoned moral discourse as a process for creating andevaluating ethical as well as scientific knowledge.34 Sometimes called “good reasons”theories, these theories are “practical” to the extent that actions are deemed right orvaluable because they conform to principles, or result in consequences, that havebeen established on the basis of reasoned transactions (among persons who affect andare affected by the development and application of moral rules).

Meta-ethical TheoriesThe function of normative ethical theories in policy analysis is to answer the ques-tion: According to which criteria can we determine whether public actions are rightor wrong? Answers to this question, as we saw earlier, may be based on one or moretypes of normative ethical criteria: teleological, deontological, and practical. Thefunction of meta-ethical theories, by contrast, is to answer questions about norma-tive ethical criteria themselves: Can we determine the truth and falsity of normativeethical claims? Does normative ethics produce a kind of knowledge and, if so, what

33See Louis A. Gewirth, Reason and Morality (Chicago: University of Chicago Press, 1978).34See P. W. Taylor, Normative Discourse (Englewood Cliffs, NJ: Prentice Hall, 1971); Stephen Toulmin,The Place of Reason in Ethics (Oxford: Oxford University Press, 1950); and The Uses of Argument(Cambridge: Cambridge University Press, 1958); Kurt Baier, The Moral Point of View (Ithaca, NY:Cornell University Press, 1965); Frank Fischer, Politics, Values, and Public Policy (Boulder, CO:Westview Press, 1980).



kind of knowledge is it? If normative ethics is not capable of being true and false,what kinds of non-cognitive results are produced?

Meta-ethical theories differ in terms of their assumptions about the epistemo-logical status of normative ethical theories—for example, so-called “cognitivist”meta-ethics affirms that normative ethical theories are capable of being true or falseand, therefore, that normative ethics constitutes a kind of knowledge. This isdenied by “non-cognitivist” meta-ethical theories. Meta-ethical theories areassociated with normative ethical theories.

Thus, for example, the growth of logical positivism in the social sciences hascontributed to non-cognitivist meta-ethical doctrines that, in turn, have resulted inthe devaluation of normative ethical discourse and an inability to recognize thatputative empirical theories (welfare economics) and analytic routines (cost-benefitanalysis) are based on controversial ethical premises. In this case, a particular meta-ethical doctrine (non-cognitivism) has exerted a direct and logically constrainingeffect on normative ethics by preempting opportunities for reflective normativediscourse and the development of ethical knowledge.

EVALUATION IN POLICY ANALYSISDescriptive, normative, and meta-ethical theories provide a foundation for evalua-tion in policy analysis, whereby evaluation refers to the production of informationabout the value or worth of policy outcomes. When policy outcomes do in fact havevalue, it is because they contribute to goals and objectives. In this case, we say thata policy or program has attained some significant level of performance.

The Nature of EvaluationThe main feature of evaluation is that it results in claims that are evaluative incharacter. Here the main question is not one of facts (Does something exist?) or ofaction (What should be done?) but one of values (Of what worth is it?). Evaluationtherefore has several characteristics that distinguish it from other policy-analyticmethods:

1. Value focus. Evaluation, as contrasted with monitoring, focuses on judg-ments regarding the desirability or value of policies and programs. Evaluationis primarily an effort to determine the worth or utility of a policy or program,and not simply an effort to collect information about the anticipated andunanticipated outcomes of policy actions. As the appropriateness of policygoals and objectives can always be questioned, evaluation includes theevaluation of goals and objectives themselves.

2. Fact-value interdependence. Evaluation depends as much on “facts” as itdoes on “values.” To claim that a particular policy or program has attaineda high (or low) level of performance requires not only that policy outcomesare valuable to some individual, group, or society as a whole, but it alsorequires that policy outcomes are actually a consequence of actions under-taken to resolve a particular problem. Hence, monitoring is a prerequisiteof evaluation.



3. Present and past orientation. Evaluative claims, as contrasted with advoca-tive claims produced through recommendation, are oriented toward presentand past outcomes, rather than future ones. Evaluation is retrospective andoccurs after actions have been taken (ex post). Recommendation, while alsoinvolving value premises, is prospective and occurs before actions have beentaken (ex ante).

4. Value duality. The values underlying evaluative claims have a dual quality,because they may be regarded as ends and means. Evaluation is similar torecommendation insofar as a given value (e.g., health) may be regarded asintrinsic (valuable in itself) as well as extrinsic (desirable because it leads tosome other end). Values are often arranged in a hierarchy that reflects therelative importance and interdependence of goals and objectives.

Functions of EvaluationEvaluation performs several main functions in policy analysis. First, and mostimportant, evaluation provides reliable and valid information about policy perform-ance, that is, the extent to which needs, values, and opportunities have been realizedthrough public policies. In this respect, evaluation reveals the extent to which partic-ular goals (e.g., improved health) and objectives (e.g., a 20 percent reduction ofchronic diseases by 2020) have been attained.

Second, evaluation contributes to the clarification and critique of values thatunderlie the selection of goals and objectives. Values are clarified by defining andoperationalizing goals and objectives. Values are also critiqued by systematicallyquestioning the appropriateness of goals and objectives in relation to the problembeing addressed. In questioning the appropriateness of goals and objectives, analystsmay examine alternative sources of values (e.g., public officials, vested interests,client groups) as well as their grounds in different forms of rationality (technical,economic, legal, social, substantive).

Third, evaluation may contribute to the application of other policy-analyticmethods, including problem structuring and prescription. Information about inade-quate policy performance may contribute to the restructuring of policy problems,for example, by showing that goals and objectives should be redefined. Evaluationcan also contribute to the definition of new or revised policy alternatives by showingthat a previously favored policy alternative should be abandoned and replaced withanother one.

Criteria for Policy EvaluationIn producing information about policy performance, analysts use different types ofcriteria to evaluate policy outcomes. These types of criteria have already been dis-cussed in relation to policy prescription. The main difference between criteria forevaluation and criteria for prescription is the time at which criteria are applied.Criteria for evaluation are applied retrospectively (ex post), whereas criteria for pre-scription are applied prospectively (ex ante). Evaluation criteria are summarized inTable 2.



TABLE 2

Criteria for Evaluation

Type of Criterion Question Illustrative Criteria

Effectiveness Has a valued outcome beenachieved?

Units of service

Efficiency How much effort was requiredto achieve a valued outcome?

Unit costNet benefits

Benefit-cost ratio

Adequacy To what extent does theachievement of a valuedoutcome resolve the problem?

Fixed costs (type I problem)Fixed effectiveness (type II problem)

Equity Are costs and benefitsdistributed equitably amongdifferent groups?

Pareto criterionKaldor–Hicks criterionRawls’s criterion

Responsiveness Do policy outcomes satisfy theneeds, preferences, or valuesof particular groups?

Consistency with citizen surveys

Appropriateness Are desired outcomes(objectives) actually worthyor valuable?

Public programs should beequitable as well as efficient

Note: See chapter on “Prescribed Prescripton Policies” for extended descriptions of criteria.

APPROACHES TO EVALUATIONEvaluation, as we saw earlier, has two interrelated aspects: the use of variousmethods to monitor the outcomes of public policies and programs and the applica-tion of some set of values to determine the worth of these outcomes to some per-son, group, or society as a whole. Observe that these two interrelated aspects pointto the presence of factual and value premises in any evaluative claim. Yet manyactivities described as “evaluation” in policy analysis are essentiallynonevaluative—that is, they are primarily concerned with the production of desig-native (factual) claims rather than evaluative ones. In fact, each of the fourapproaches to monitoring is frequently mislabeled as an approach to “evaluationresearch” or “policy evaluation.”35

Given the present lack of clarity about the meaning of evaluation in policyanalysis, it is essential to distinguish among several different approaches topolicy evaluation: pseudo-evaluation, formal evaluation, and decision-theoreticevaluation. These approaches and their aims, assumptions, and major forms areillustrated in Table 3.

35See, for example, the description of “evaluation” research in Marcia Guttentag and Elmer Struening,ed., Handbook of Evaluation Research, 2 vols. (Beverly Hills, CA: Sage Publications, 1975); andLeonard Rutman, ed., Evaluation Research Methods: A Basic Guide (Beverly Hills, CA: SagePublications, 1977).



Pseudo-evaluationPseudo-evaluation is an approach that uses descriptive methods to produce reliableand valid information about policy outcomes, without attempting to questionthe worth or value of these outcomes to persons, groups, or society as a whole. The major assumption of pseudo-evaluation is that measures of worth or value areself-evident or uncontroversial.

In pseudo-evaluation, the analyst typically uses a variety of methods (quasi-experimental design, questionnaires, random sampling, statistical techniques) toexplain variations in policy outcomes in terms of policy input and process variables.Yet any given policy outcome (e.g., number of employed trainees, units of medicalservices delivered, net income benefits produced) is taken for granted as an appro-priate objective. Major forms of pseudo-evaluation include the approaches to mon-itoring: social experimentation, social systems accounting, social auditing, and re-search and practice synthesis.

Formal EvaluationFormal evaluation is an approach that uses descriptive methods to produce reliableand valid information about policy outcomes but evaluates such outcomes on the

TABLE 3

Three Approaches to Evaluation

Approach Aims Assumptions Major Forms

Pseudo-evaluation

Use descriptive methods to produce reliable and valid information aboutpolicy outcomes

Measures of worth orvalue are self-evident or uncontroversial

Social experimentationSocial systemsaccountingSocial auditingResearch and practice synthesis

Formal evaluation Use descriptive methods toproduce reliable and validinformation about policyoutcomes that have beenformally announced aspolicy-program objectives

Formally announcedgoals and objectives ofpolicy makers andadministrators areappropriate measures of worth or value

DevelopmentalevaluationExperimentalevaluationRetrospective process evaluationRetrospective outcome evaluation

Decision-theoreticevaluation

Use descriptive methods toproduce reliable and validinformation about policyoutcomes that are explicitlyvalued by multiplestakeholders

Formally announced aswell as latent goals andobjectives of stakeholdersare appropriate measuresof worth or value

Evaluability assessmentMultiattributeutility analysis



basis of policy-program objectives that have been formally announced by policymakers and program administrators. The major assumption of formal evaluation isthat formally announced goals and objectives are appropriate measures of the worthor value of policies and programs.

In formal evaluation, the analyst uses the same kinds of methods as thoseemployed in pseudo-evaluation and the aim is identical: to produce reliable andvalid information about variations in policy outputs and impacts that may be tracedto policy inputs and processes. The difference, however, is that formal evaluationsuse legislation, program documents, and interviews with policy makers and admin-istrators to identify, define, and specify formal goals and objectives. The appropri-ateness of these formally announced goals and objectives is not questioned. Informal evaluations, the types of evaluative criteria most frequently used are those ofeffectiveness and efficiency.

One of the major types of formal evaluation is the summative evaluation,which involves an effort to monitor the accomplishment of formal goals andobjectives after a policy or program has been in place for some period of time.Summative evaluations are designed to appraise products of stable and well-established public policies and programs. By contrast, a formative evaluationinvolves efforts to continuously monitor the accomplishment of formal goals andobjectives. The differences between summative and formative evaluation shouldnot be overemphasized, however, because the main distinguishing characteristic offormative evaluation is the number of points in time at which policy outcomes aremonitored. Hence, the difference between summative and formative evaluation ismainly one of degree.

Formal evaluations may be summative or formative, but they may alsoinvolve direct or indirect controls over policy inputs and processes. In the formercase, evaluators can directly manipulate expenditure levels, the mix of programs,or the characteristics of target groups—that is, the evaluation may have one ormore characteristics of social experimentation as an approach to monitoring. Inthe case of indirect controls, policy inputs and processes cannot be directly ma-nipulated; rather, they must be analyzed retrospectively on the basis of actionsthat have already occurred. Four types of formal evaluation—each based on adifferent orientation toward the policy process (summative versus formative) andtype of control over action (direct versus indirect)—are illustrated in Table 4.

TABLE 4

Types of Formal Evaluation

Control over Policy Actions

Orientation Toward Policy Process

Formative Summative

Direct Developmental evaluation Experimental evaluationIndirect Retrospective process evaluation Retrospective outcome evaluation



Varieties of Formal EvaluationDevelopmental evaluation refers to evaluation activities that are explicitly designedto serve the day-to-day needs of program staff. Developmental evaluation is useful“for alerting the staff to incipient weaknesses or unintended failures of a programand for insuring proper operation by those responsible for its operation.”36

Developmental evaluation, which involves some measure of direct control over pol-icy actions, has been used in a wide variety of situations in the public and privatesectors. Thus, for example, businesses frequently use developmental evaluations todistribute, test, and recall new products. In the public sector, developmental evalua-tions have been used to test new teaching methods and materials in public educationprograms, such as Sesame Street and Electric Company. Such programs are system-atically monitored and evaluated by showing them to audiences composed of chil-dren within specified age limits. Subsequently, they are “revised many times on thebasis of systematic observations of which program features achieved attentionand on the basis of interviews with the children after viewing the program.”37

Developmental evaluations, as they are both formative and involve direct controls,can be used to adapt immediately to new experience acquired through systematicmanipulations of input and process variables.

Retrospective process evaluation involves the monitoring and evaluation of pro-grams after they have been in place for some time. Retrospective process evaluation,which often focuses on problems and bottlenecks encountered in the implementa-tion of policies and programs, does not permit the direct manipulation of inputs(e.g., expenditures) and processes (e.g., alternative delivery systems). Rather it relieson ex post facto (retrospective) descriptions of ongoing program activities, whichare subsequently related to outputs and impacts. Retrospective process evaluationrequires a well-established internal reporting system that permits the continuousgeneration of program-related information (e.g., the number of target groupsserved, the types of services provided, and the characteristics of personnel employedto staff programs). Management information systems in public agencies sometimespermit retrospective process evaluations, provided they contain information onprocesses as well as outcomes.

Title I of the Elementary and Secondary Education Act (1965), which laterbecame President Bush’s No Child Left Behind in 2001, was subjected to a formof retrospective process evaluation by the U.S. Office of Education, but with disap-pointing results. Title I provided funds to local school systems in proportion to thenumber of pupils from poor or deprived families. However, local school districtssubmitted inadequate and marginally useful information, thus making it impossibleto evaluate programs. In this context, retrospective process evaluations presupposea reliable and valid information system, which is often difficult to establish.

Experimental evaluation involves the monitoring and evaluation of outcomesunder conditions of direct controls over policy inputs and processes. The ideal ofexperimental evaluation has generally been the “controlled scientific experiment,”

36Peter H. Rossi and Sonia R. Wright, “Evaluation Research: An Assessment of Theory, Practice, andPolitics,” Evaluation Quarterly 1, no. 1 (1977): 21.37Ibid., p. 22.



in which all factors that might influence policy outcomes except one—that is, a par-ticular input or process variable—are controlled, held constant, or treated as plausi-ble rival hypotheses. Experimental and quasi-experimental evaluations include theNew Jersey–Pennsylvania Income Maintenance Experiment, the California GroupTherapy–Criminal Recidivism Experiment, the Kansas City Preventive PatrolExperiment, Project Follow Through, the Supported Work Demonstration Project,and various experiments in educational performance contracting. Many experimen-tal evaluations from around the world are included in the online CampbellCollaboration (www.campbellcollaboration.com).

Experimental evaluations must meet rather strict requirements before they canbe carried out:38 (1) a clearly defined and directly manipulable set of “treatment”variables that are specified in operational terms; (2) an evaluation strategy that per-mits maximum generalizability of conclusions about performance to many similartarget groups or settings (external validity); (3) an evaluation strategy that permitsminimum error in interpreting policy performance as the actual result of manipu-lated policy inputs and processes (internal validity); and (4) a monitoring systemthat produces reliable data on complex interrelationships among preconditions,unforeseen events, inputs, processes, outputs, impacts, and side effects andspillovers. As these demanding methodological requirements are rarely met, experi-mental evaluations typically fall short of the “true” controlled experiment and arereferred to as “quasi-experimental.”

Retrospective outcome evaluations also involve the monitoring and evaluationof outcomes but with no direct control over manipulable policy inputs andprocesses.39 At best, controls are indirect or statistical—that is, the evaluatorattempts to isolate the effects of many different factors by using quantitative meth-ods. In general, there are two main variants of retrospective process evaluation:cross-sectional and longitudinal studies. Longitudinal studies are those that evaluatechanges in the outcomes of one, several, or many programs at two or more points intime. Many longitudinal studies have been carried out in the area of family plan-ning, in which fertility rates and changes in the acceptance of contraceptive devicesare monitored and evaluated over reasonably long periods of time (five to twentyyears). Cross-sectional studies, by contrast, seek to monitor and evaluate multipleprograms at one point in time. The goal of the cross-sectional study is to discoverwhether the outputs and impacts of various programs are significantly differentfrom one another, and if so, what particular actions, preconditions, or unforeseenevents might explain the difference.

Two prominent examples of retrospective outcome evaluations that are cross-sectional in nature are Project Head Start, a program designed to provide compensa-tory education to preschool children, and the Coleman Report. Indeed,

almost every evaluation of the national compensatory education programsstarted during the middle 1960s has been based on cross-sectional data. Pupils

38Walter Williams, Social Policy Research and Analysis (New York: Elsevier, 1971), p. 93.39Note that three of the approaches to monitoring discussed in the chapter “Developing PolicyArguments”—that is, social systems accounting, social auditing, and research and practice synthesis—may also be considered as forms of retrospective outcome evaluation. Similarly, cost-benefit and cost-effectiveness analysis may be viewed as particular forms of retrospective outcome evaluation.



enrolled in compensatory education programs were contrasted to those whowere not, holding constant statistically such sources of competing explanationsas family backgrounds, ethnicity, region, city size, and so on.40

Decision-Theoretic EvaluationDecision-theoretic evaluation is an approach that uses descriptive methods to producereliable and valid information about policy outcomes that are explicitly valued bymultiple stakeholders. The key difference between decision-theoretic evaluation, onone hand, and pseudo- and formal evaluation on the other, is that decision-theoreticevaluation attempts to surface and make explicit the latent as well as manifest goalsand objectives of stakeholders. This means that formally announced goals andobjectives of policy makers and administrators are but one source of values, because allparties who have a stake in the formulation and implementation of a policy (e.g.,middle- and lower-level staff, personnel in other agencies, client groups) are involved ingenerating the goals and objectives against which performance is measured.

Decision-theoretic evaluation is a way to overcome several deficiencies ofpseudo-evaluation and formal evaluation:41

1. Underutilization and nonutilization of performance information. Much ofthe information generated through evaluations is underutilized or never usedto improve policy making. In part, this is because evaluations are not suffi-ciently responsive to the goals and objectives of parties who have a stake informulating and implementing policies and programs.

2. Ambiguity of performance goals. Many goals of public policies and programsare vague. This means that the same general goal—for example, to improvehealth or encourage better conservation of energy—can and do result inspecific objectives that conflict with one another. This is evident when weconsider that the same goal (e.g., improved health) may be operationalizedin terms of at least six types of evaluation criteria: effectiveness, efficiency,adequacy, equity, responsiveness, and appropriateness. One of the purposes ofdecision-theoretic evaluation is to reduce the ambiguity of goals and makeconflicting objectives explicit.

3. Multiple conflicting objectives. The goals and objectives of public policiesand programs cannot be satisfactorily established by focusing on the values ofone or several parties (e.g., Congress, a dominant client group, or a headadministrator). In fact, multiple stakeholders with conflicting goals and objec-tives are present in most situations requiring evaluation. Decision-theoreticevaluation attempts to identify these multiple stakeholders and surface theirgoals and objectives.

40Rossi and Wright, “Evaluation Research,” p. 27.41For accounts of these and other deficiencies, see Carol H. Weiss, Evaluation Research (EnglewoodCliffs, NJ: Prentice Hall, 1972); Edwards, Guttentag, and Snapper, “A Decision-Theoretic Approach toEvaluation Research,” in Handbook of Evaluation Research, ed. Guttentag and Struening, pp. 139–81;and Martin Rein and Sheldon H. White, “Policy Research: Belief and Doubt,” Policy Analysis 3, no. 2(1977): 239–72.



One of the main purposes of decision-theoretic evaluation is to link infor-mation about policy outcomes with the values of multiple stakeholders. Theassumption of decision-theoretic evaluation is that formally announced as well aslatent goals and objectives of stakeholders are appropriate measures of the worthor value of policies and programs. The two major forms of decision-theoreticevaluation are evaluability assessment and multiattribute utility analysis, both ofwhich attempt to link information about policy outcomes with the values ofmultiple stakeholders.

Evaluability assessment is a set of procedures designed to analyze the deci-sion-making system that is supposed to benefit from performance information andto clarify the goals, objectives, and assumptions against which performance is tobe measured.42 The basic question in evaluability assessment is whether a policyor program can be evaluated at all. For a policy or program to be evaluable, atleast three conditions must be present: a clearly articulated policy or program,clearly specified goals and/or consequences, and a set of explicit assumptions thatlink policy actions to goals and/or consequences.43 In conducting an evaluabilityassessment, analysts follow a series of steps that clarify a policy or program fromthe standpoint of the intended users of performance information and theevaluators themselves:44

1. Policy-program specification. Which federal, state, or local activities andwhich goals and objectives constitute the program?

2. Collection of policy-program information. What information must be collectedto define policy-program objectives, activities, and underlying assumptions?

3. Policy-program modeling. What model best describes the program andits related objectives and activities from the point of view of intendedusers of performance information? What causal assumptions link actions tooutcomes?

4. Policy-program evaluability assessment. Is the policy-program model suffi-ciently unambiguous to make the evaluation useful? Which types of evaluationstudies would be most useful?

5. Feedback of evaluability assessment to users. After presenting conclusionsabout policy-program evaluability to intended users, what appear to be thenext steps that should (or should not) be taken to evaluate policy performance?

A second form of decision-theoretic evaluation is multiattribute utilityanalysis.45 Multiattribute utility analysis is a set of procedures designed to elicitfrom multiple stakeholders subjective judgments about the probability of occur-rence and value of policy outcomes. The strengths of multiattribute utility analysisare that it explicitly surfaces the value judgments of multiple stakeholders; it

42On evaluability assessment, see Joseph S. Wholey and others, “Evaluation: When Is It ReallyNeeded?” Evaluation 2, no. 2 (1975): 89–94; and Wholey, “Evaluability Assessment,” in EvaluationResearch Methods, ed. Rutman, pp. 41–56.43See Rutman, Evaluation Research Methods, p. 18.44Wholey, “Evaluability Assessment,” pp. 43–55.45See Edwards, Guttentag, and Snapper, “A Decision-Theoretic Approach,” pp. 148–59.



recognizes the presence of multiple conflicting objectives in policy-programevaluation; and it produces performance information that is more usable from thestandpoint of intended users. The steps in conducting a multiattribute utilityanalysis are the following:

1. Stakeholder identification. Identify the parties who affect and are affected bya policy or program. Each of these stakeholders will have goals and objectivesthat they wish to maximize.

2. Specification of relevant decision issues. Specify the courses of action or inac-tion about which there is a disagreement among stakeholders. In the simplestcase, there will be two courses of action: the status quo and some new initiative.

3. Specification of policy outcomes. Specify the range of consequences that mayfollow each course of action. Outcomes may be arranged in a hierarchy inwhich one action has several consequences, each of which itself has furtherconsequences. A hierarchy of outcomes is similar to an objectives tree, exceptthat outcomes are not objectives until they have been explicitly valued.

4. Identification of attributes of outcomes. Here the task is to identify all rele-vant attributes that make outcomes worthy or valuable. For example, eachoutcome may have different types of benefits and costs to different targetgroups and beneficiaries.

5. Attribute ranking. Rank each value attribute in order of importance. For example, if increased family income is an outcome of a poverty program,this outcome may have several value attributes: sense of family well-being,greater nutritional intake, and more disposable income for health care.These attributes should be ranked in order of their relative importance toone another.

6. Attribute scaling. Scale attributes that have been ranked in order of impor-tance. To do so, arbitrarily assign the least important attribute a value of ten.Proceed to the next most important attribute, answering the question: Howmany times more important is this attribute than the next-least-important one?Continue this scaling procedure until the most important attribute has beencompared with all others. Note that the most important attribute may have ascale value 10, 20, 30, or more times that of the least important attribute.

7. Scale standardization. The attributes that have been scaled will have differ-ent maximum values for different stakeholders. For example, one stakeholdermay give attribute A a value of 60; attribute B, a value of 30; and attribute C,a value of 10. Another stakeholder, however, may give these same attributesvalues of 120, 60, and 10. To standardize these scales, sum all original valuesfor each scale, divide each original value by its respective sum, and multiplyby 100. This results in separate scales whose component values sum to 100.

8. Outcome measurement. Measure the degree to which each outcome is likelyto result in the attainment of each attribute. The maximum probability shouldbe given a value of 100; the minimum probability should be given a value of 0(i.e., there is no chance that the outcome will result in the attainment of theattribute).



9. Utility calculation. Calculate the utility (value) of each outcome by usingthe formula

where

Ui � the aggregate utility (value) of the ith outcomewi � the standardized scale value of the ith attributeuij � the probability of occurrence of the ith outcome on the jth attribute

10. Evaluation and presentation. Specify the policy outcome with the greatestoverall performance, and present this information to relevant decision makers.

The strength of multiattribute utility analysis is that it enables analysts todeal systematically with conflicting objectives of multiple stakeholders. This ispossible, however, only when the steps just described are carried out as part of agroup process involving relevant stakeholders. Hence, the essential requirementof multiattribute utility analysis is that stakeholders who affect and are affectedby a policy or program are active participants in the evaluation of policyperformance.

METHODS FOR EVALUATIONA number of methods and techniques can assist analysts in evaluating policyperformance. Nearly all of these techniques, however, may also be used inconjunction with other policy-analytic methods, including problem structuring,forecasting, recommendation, and monitoring. Thus, for example, argumenta-tion analysis may be used to surface assumptions about expected relationshipsbetween policy actions and objectives. Cross-impact analysis may prove useful inidentifying unanticipated policy outcomes that work against the achievement ofpolicy-program objectives. Similarly, discounting may be as relevant to policy-program evaluation as it is to recommendation, given that cost-benefit and cost-effectiveness analysis may be used retrospectively (ex post) as well as prospec-tively (ex ante). Finally, techniques that range from graphic displays and indexnumbers to control-series analysis may be essential for monitoring policy out-comes as a prelude to their evaluation.

The fact that various techniques may be used with more than one policy-analytic method points to the interdependence of problem structuring, forecasting,recommendation, monitoring, and evaluation in policy analysis. Many methods andtechniques are relevant to pseudo-evaluation, formal evaluation, and decision-theoretic evaluation (Table 5).

Only one of the techniques listed in Table 5 has not already been described.User-survey analysis is a set of procedures for collecting information about theevaluability of a policy or program from intended users and other stakeholders.46

Ui = ©wiuij

46See Wholey, “Evaluability Assessment,” pp. 44–49.



User surveys are central to the conduct of evaluability assessments and other formsof decision-theoretic evaluation. The major instrument for collecting information isan interview protocol with a series of open-ended questions. Responses to thesequestions provide the information required to complete the several steps in anevaluability assessment previously described: policy-program specification, policy-program modeling, policy-program assessment, and presentation of evaluabilityassessment to users. A sample interview protocol for a user survey analysis ispresented in Table 6.

TABLE 5

Techniques for Evaluation by Three ApproachesApproach Technique

Pseudo-evaluation Graphic displaysTabular displaysIndex numbersInterrupted time-series analysisControl-series analysisRegression-discontinuity analysis

Formal evaluation Objectives mappingValue clarificationValue critiqueConstraint mappingCross-impact analysisDiscounting

Decision-theoretic evaluation BrainstormingArgumentation analysisPolicy DelphiUser-survey analysis

TABLE 6

Interview Protocol for User-Survey Analysis

Step in Evaluability Assessment Questions

Policy-program specification 1.What are the objectives of the policy orprogram?

2.What would be acceptable evidence of theachievement of policy-program objectives?(a)

Policy-program modeling 3.What policy actions (e.g., resources, guidelines,staff activities) are available to achieveobjectives?(b)

4.Why will action A lead to objective O?(b)

(continued)



CHAPTER SUMMARYThis chapter has provided an overview of theprocess of evaluation, contrasted three appro-aches to evaluation, and presented specificmethods and techniques used in conjunctionwith these approaches. The process of valuation

was then distinguished from evaluation, andalternative ethical and meta-ethical theorieswere examined. Normative ethics and meta-ethics provide rationales for selecting criteria toevaluate policy performance.

REVIEW QUESTIONS1. Compare and contrast methods of evaluation

and prescription in terms of time and the typesof claims produced by each.

2. Many policy-program evaluations fail to rec-ognize the latent purposes of evaluation,including a desire to (a) make programs lookgood by focusing on their surface characteris-tics (“eyewash”), (b) cover up program failures(“white-wash”), (c) destroy a program (“sub-marine”), (d) engage in evaluation merely as aritual that must be practiced to receive funding(“posture”), and (e) postpone attempts to resolveproblems (“postponement”). See Edward A.Suchman, “Action for What? A Critique ofEvaluative Research,” in Evaluating ActionPrograms, ed. Carol H. Weiss (Boston: Allynand Bacon, 1972), p. 81. What problems does

this raise for defining objectives against whichperformance is evaluated?

3. Compare and contrast formative and summativeevaluation. Which of these two types of evalua-tion provides performance information that islikely to be most useful to policy makers? Why?

4. What are the strengths and limitations of cost-benefit analysis as an approach to evaluation?In your answer, refer to contrasts amongpseudo-evaluation, formal evaluation, and de-cision-theoretic evaluation.

5. Select a policy or program that you would liketo evaluate. Compare and contrast evaluabilityassessment and multiattribute utility analysisas techniques of evaluation, indicating whichof the two procedures is likely to yield themore reliable, valid, and useful results.

Step in Evaluability Assessment Questions

Policy-program evaluabilityassessment

5.What do various stakeholders (e.g.,Congress,OMB, state auditor general, mayor’s office) expect of the program in terms of performance? Are these expectations consistent?

6.What is the most serious obstacle to achieving objectives?

Feedback of evaluability assessment to users

7.What performance information do you need on the job? Why?

8. Are present sources of performance information adequate? Why? Why not?

9.What is the most important source ofperformance information you will need inthe next year?

10.What key issues should any evaluation address?

Notes:(a)Answers to this question yield operational measures of objectives.(b)Answers to these questions yield causal assumptions about the relation between actions and objectives.

Source: Adapted from Joseph S.Wholey,“Evaluability Assessment,” in Evaluation Research Methods: A Basic Guide, ed.Leonard Rutman (Beverly Hills, CA: Sage Publications, 1977), Fig. 3, p. 48.



DEMONSTRATION EXERCISEIssues of a “living wage” turn on questions ofpoverty, inequality, income distribution, and eco-nomic efficiency. Inequalities of many kinds haveincreased in the past thirty-five and more years,irrespective of whether Democrats or Republicanswere in power in the United States, although con-servatives continue to refer to American society asan “opportunity society” and liberals as an“inequality society.”Attempts to define inequality often use the analogyof a cake that can be cut into slices according todifferent rules.47 These rules, along with theirunderlying justifications, deal with the distributionof income, wealth, educational opportunity, andhealth care. Some of these rules are the basis oftheories of justice put forth by well-known politicaland ethical theorists including John Rawls48 andRobert Nozick.49 Rawls has given us a theory ofjustice that is based primarily on what he sees as afair distribution of social goods. For Rawls, thefundamental rule of fairness is “allocate socialgoods so that those worse off are made better off.”Nozick, by contrast, has a theory of justice basedon what he sees as a fair process for achievingsocial goods. Here the rule is “allocate social goodsin accordance with the rule that everyone has anequal opportunity to acquire them.” Both theoristshave had a significant influence on the way manypeople think about values, ethics, and public policy.

1. Divide into three groups. Each group will usedifferent rules for distributing a cake amongmembers of the class. Group A will use threerules for distributing a social good in accor-dance with who benefits. By contrast, group Bwill use two rules based on what is distributed.In turn, group C will use two rules based onthe process for distributing the good.

Set A: Who Benefits?

� Divide the cake equally among classmembers.

� Divide the cake by the rank of classmembers. Define ranks according to

the number of years of education ofmembers.

� Divide the cake equally among twohypothetical groups: a group composedof 20 percent of the class memberswho have experienced historical dis-crimination in wages and the remain-der of the group. Each group mustdivide its equal portion in accordancewith rank, as determined by years ofeducation.

Set B: What is Being Distributed?

� Divide the cake so that persons withcaloric intake below some standardreceive larger pieces to compensate fortheir caloric deficit.

� Divide the cake so that persons withpreferences for three pieces receive alarger piece than those with a prefer-ence for two pieces, who in turnreceive a larger piece than those whoprefer one piece or none.

Set C: What is the Process for Distributingthe Good

� Divide the cake according to acompetitive process in which membersin the first row of the room have a100 percent chance of getting a pieceof cake, members in the second rowhave an 80 percent chance, members inthe third row have a 60 percentchance, members in the fourth rowhave a 40 percent chance, and mem-bers sitting farther away have a 20 percent chance. Some members inthe last row have no chance at all.

47Deborah Stone, Policy Paradox: The Art of Political Decision Making, Chapter 2: Equity, pp. 39–60,and Chapter 3: Efficiency, pp. 61–85.48Rawls, A Theory of Justice.49Robert Nozick, Anarchy, State, and Utopia (New York: Basic Books, 1974).



� Divide the cake according to arandom process (lottery) in whicheach person has an equal probabilityof being selected to receive a pieceof the cake. Repeat the process30 times.50

� Divide the cake according to an elec-toral process in which a simple majoritygets two-thirds of the cake. Choosetwo candidates and vote for them.

2. Groups A, B, and C will apply their rules tothe issue of whether or not the “living wage”policies described in Case 1 are justifiable.Justify each of your rules by drawing on nor-mative ethical and meta-ethical theoriesdescribed in this chapter, including those ofNozick and Rawls. Then construct a 10-minutebriefing in which you present a recommenda-tion(s) to the mayor of a city considering aliving wage.

50This is sampling with replacement, whereby each person selected is returned to the pool and can beselected again in each of the 30 draws.

BIBLIOGRAPHYBok, Sissela. Lying: Moral Choice in Public and

Private Life. New York: Pantheon Books, 1978.Dasgupta, Partha. “Valuation and Evaluation:

Measuring the Quality of Life and EvaluatingPolicy.” Background paper prepared for theWorld Bank’s annual World DevelopmentReport (2000).

Mac Rae, Duncan, Jr. The Social Function ofSocial Science. New Haven, CT: Yale UniversityPress, 1976.

Nozick, Robert. Anarchy, State, and Utopia.New York: Basic Books, 1974.

Rawls, John. A Theory of Justice. Cambridge, MA:Harvard University Press, 1971.

Sen, Amartya. Inequality Reexamined. Oxford:Clarendon Press, 1992.

Stone, Deborah. Policy Paradox: The Art ofPolitical Decision Making. Rev ed. New York:W. W. Norton, 2002.


CASE 1 THE ECONOMICS OF MORAL JUDGEMENT:EVALUATING LIVING WAGE POLICIES

The living wage movement is considered by many tobe the most important grassroots politicalmobilization since the U.S. civil rights movement.Issues surrounding the adoption of a living wage areeconomic as well as moral and ethical. Economistsare divided on the effects of the living wage onpoverty, inequality, and employment; the majority ofreligious and community leaders have supported theliving wage for more than 100 years.

The following calculations have been extractedfrom a website (www.livingwage.geog.psu.edu)developed by Dr. Amy K. Glasmeier of the

Massachusetts Institute of Technology. The livingwage shown is the hourly rate in 2011 that a full-time employed person (2,080 hours per year)in New York City (Queens County) must earn tosupport families of different sizes. In New YorkCity, the minimum wage of $7.25 per hour is thesame for all persons, regardless of the number ofdependents. Although the poverty rate is typicallyquoted as gross annual income, for purposes ofcomparison it has been converted into an hourlywage. Wages that are less than the living wage areitalicized.

Municipalities such as San Francisco, Detroit,and Chicago set the minimum wage higher than thefederally mandated 2011 minimum wage of $7.25.The minimum living wage for an adult with one

child and larger families also differs. InPittsburgh, the living wage for an adult with onechild is $34,244 per year, whereas in Los Angeles it is $45,235.

Hourly WagesOne Adult

One Adult,One Child

Two Adults

Two Adults,One Child

Two Adults,Two Children

Living Wage $11.86 $19.66 $16.29 $24.10 $30.30Poverty Wage $5.04 $6.68 $6.49 $7.81 $9.83Minimum Wage $7.25 $7.25 $7.25 $7.25 $7.25

Typical Expenses

These figures were used to calculate the 2011 livingwage in New York City. Numbers vary by family size.

For typical expenses, living wages, and salaries fordifferent occupations across the United States,see www.livingwage.geog.psu.ed. �



Monthly Expenses

One Adult

One Adult,One Child

Two Adults

Two Adults,One Child

Two Adults,Two Children

Food $232 $378 $448 $594 $740Child Care $0 $572 $0 $572 $1,012

Medical $76 $151 $152 $227 $302

Housing $1,185 $1,318 $1,185 $1,318 $1,318

Transportation $232 $397 $464 $629 $794

Other $188 $369 $376 $557 $738

Monthly After-Tax IncomeThat’s Required

$1,913 $3,185 $2,625 $3,897 $4,904

Annual After-Tax IncomeThat’s Required

$22,956 $38,220 $31,500 $46,764 $58,853

Annual Taxes $1,706 $2,678 $2,387 $3,373 $4,161Annual BeforeTax IncomeThat’s Required

$24,662 $40,898 $33,887 $50,137 $63,014

Source: © 2011 Dr. Amy K. Glasmeier and The Pennsylvania State University Site created by West Arete Computing


Developing PolicyArguments




Developing PolicyArguments

� Describe the structure of a policyargument.

� Compare and contrast four types ofpolicy claims.

� Describe relations among differentelements of an argument.

� Explain how objections andrebuttals affect the strength of anargument.

� Demonstrate how qualifiers changein response to objections andrebuttals.

� Distinguish and illustrate differentmodes of policy argumentation.

� Explain why formal and informalfallacies diminish the strength ofclaims.

� Use argument mapping techniquesto represent cases of foreign policyargumentation.

What policy makers understand and analysts often forget is that policyargumentation is central to policy making.1 Policy arguments are amongthe major vehicles for communicating policy-relevant information and an

important source of knowledge about the ways policies are made and put into effect.Because policy arguments are carriers of policy-relevant information, they are alsoimportant for understanding the use and misuse of policy analysis by policy makers.

1One of the landmark books on policy argumentation was Giandomenico Majone, Evidence,Argument, and Persuasion in the Policy Process (New Haven, CT: Yale University Press, 1992).

O B J E C T I V E S


Developing Policy Arguments

The ability to organize, structure, and evaluate a policy argument is central tocritical analytical thinking. In contexts of practice, argumentation is not limited tothe kinds of reasoning employed in formal logic or in the social sciences, forexample, reasoning based on quantitative models designed to explain politicalbehavior or on formal mathematical models of rational choice used in economics.Many other modes of argumentation—from ethical and political reasoning toreasoning by analogy and metaphor—coexist and compete for the attention ofpolicy makers. Competent analysts should be able to compare and contrast differentmodes of reasoning and express technical arguments in a language that iscomprehensible to policy makers.

THE STRUCTURE OF POLICY ARGUMENTSA policy argument is the product of argumentation, which is the process.In real-life policy settings, arguments are complex and prone to misunderstanding.For this reason, conceptual models or maps are useful in identifying and relatingthe elements of policy arguments. One such model is the structural model ofargument originated by the English-born philosopher Stephen Toulmin andextended by others.2 Another structural model of argument called Rationale,developed by Tim van Gelder and associates, is what we use here to map policyarguments (see Figure 1).3

Structural models are designed to investigate the organization of practicalreasoning. Arguments based on practical reasoning, as distinguished from thosebased on the formal reasoning of mathematics and deductive logic, lack thecertainty of formally valid logical arguments, for example, if A is preferred toB, and B is preferred to C, then A is preferred to C. Practical arguments are alwaysuncertain, as are the reasons and evidence employed to justify conclusions.Reasons are not always stated explicitly, and in some cases they are not stated atall. Even when they are stated, they are seldom complete or conclusive.In Toulmin’s words, practical reasoning yields conclusions “about which we arenot entirely confident by relating them back to other information about which wehave greater assurance.”4 Policy arguments rarely if ever have the certainty ofdeductive logic.

2Stephen Toulmin, The Uses of Argument (Cambridge: Cambridge University Press, 1958); andToulmin, Robert Rieke, and Alan Janik, An Introduction to Reasoning, 2d ed. (New York: Macmillan,1984). Other models of reasoning and argument are Hayward Alker Jr., “The Dialectical Logic ofThucydides’ Melian Dialogue,” American Political Science Review 82, no. 3 (1988): 805–20; MichaelScriven, Reasoning (New York: McGraw Hill, 1977); D. R. Des Gasper, “Structures and Meanings:A Way to Introduce Argumentation Analysis in Policy Studies Education,” Africanus (University ofSouth Africa) 30, no. 1 (2000): 49–72; and D. R. Des Gasper, “Analyzing Policy Arguments,” EuropeanJournal of Development Research 8, no. 1 (1996): 36–62.3See Tim van Gelder, “The Rationale for Rationale.” Law, Probability, and Risk 6 (2007): 23-42.The computer application called Rationale is available at www.austhink.com.4Toulmin, The Uses of Argument, p. 127.


A policy claim is the conclusion of apolicy argument. There are four typesof claims: definitional, designative,evaluative, and advocative. Argumentsmay include subordinate claims andarguments.

CLAIM

Policy-relevant informationprovides the grounds forsupporting a policy claim.Policy-relevant informationmay be statistical data,experimental findings, experttestimony, common sense,or political judgments. Theinformation begins with thewords “Given that ...”

INFORMATIONA qualifier expresses thedegree of confidence in aclaim. Qualifiers may bestated statistically(p = 0.01) or in everydaylanguage (“probably” or“not at all”). Qualifiersanswer the question “Isthe claim approximatelytrue?”

QUALIFIERA warrant is a reason tosupport a claim. Warrantsmay be economic theories,ethical principles, politicalideas, religious authority,and so forth. Warrantsbegin with the word“Because...”

WARRANT

A rebuttal is an objection to an objection.Rebuttals challenge objections byidentifying special conditions orexceptions that reduce confidence inthe objection. Rebuttals begin with theword “However...”

REBUTTAL

A backing is a reasonto support the warrant.Backings begin with theword “Since...”

An objection challengesinformation by identifyingspecial conditions orexceptions that reduceconfidence in theinformation. Objectionsbegin with the word“But...”

BACKINGOBJECTION

An objection challengesa warrant or backingby identifying specialconditions or exceptionsthat reduce confidencein the warrant or backing.Objections begin with“But...”

OBJECTION

The qualifier maychange as a resultof objectionsand rebuttals. Asecond qualifiermay replacethe first.

Five types of policy-relevant informationare created andtransformed byemploying policy-analytic methods(Fig 1.1).


FIGURE 1Structure of a policy argument


Types of Knowledge ClaimsA knowledge claim or contention is the conclusion of an argument. There are fourtypes of knowledge claims: definitive, designative, evaluative, and advocative.

� Definitive. If a claim asserts that a policy has a particular definition, then theclaim is definitive. Look for words such as is, is not, constituted by, representedby, similar to, and different from. A definitive claim states what somethingis or is not, what it is like or not like, or whether it belongs in one class orcategory rather than another. One type of definitive claim with particularrelevance to policy is the metaphor. Policies have been defined metaphorically in terms of “war,” “contagion,” “epidemic,” and “quarantine.”

� Designative. If a claim asserts that some aspect of a policy has been or canbe observed, or if observations are used to infer causes, then the claim is designa-tive. Look for words that describe observed or observable characteristics: became,originated, linked, caused, effected, consequence, prediction. Some terms thatseem to be designative are really evaluative, for example, “He is a good liberal”or “The evil empire (or axis of evil) is the problem.” Claims using these and other“appraising” descriptions are not questions of fact; they are evaluative. At thesame time, it is also possible to study values empirically by making observationsrelevant to appraising descriptions. Good examples are measures of democracy,ethical behavior, conservatism, and liberalism included in questionnaires andinterview schedules.5

� Evaluative. If a claim asserts that some aspect of a policy has or does nothave value or worth, it is evaluative. Look for words such as good, bad, right,wrong, beneficial, costly, efficient, responsive, equitable, just, fair, and secure.Examples: “The policy will bring about a living wage, thus moving societytoward equity and responsiveness.” “The program is less efficient thanexpected.” “The policy will build a healthy economy.” Some evaluative claimsrefer to states or conditions (e.g., a just society) and some to proceduralprocesses (e.g., a fair trial). Evaluative claims rest on warrants that involvevalues, ethics, and meta-ethics.

� Advocative. If a claim asserts that a government or division within it shouldtake action, the claim is advocative. Look for words such as should, needs to,and must. Examples: “The World Bank should terminate its structuraladjustment program.” “Congress should pass the Equal Rights Amendment.”“The United States should sign the Kyoto Treaty.” “Auto manufacturersshould produce more fuel efficient vehicles.” Advocative claims rest onwarrants that simultaneously involve facts and values.

Policy MapsPolicy maps are useful in representing complex arguments. Arguments have sevenelements: claim, information, warrant, backing, objection, rebuttal, and qualifier,


5See Delbert C. Miller and Neil J. Salkind, Handbook of Research Design and Social Measurement,6th ed. (Newbury Park, CA: Sage Publications, 2002).



(see Figure 1). The claim is the conclusion or contention of an argument. It issupported by policy-relevant information, which is usually the starting point of anargument. The warrant is a reason for making the claim on the basis of the informa-tion supplied. The qualifier expresses the approximate truth, plausibility, or confi-dence in the claim. Consider the following example:

The senator supports the privatization of the federal highway system, which willbring gains in efficiency and reduced taxes. Considering that the privatization ofpublic services has been successful in other areas, this is definitely a “no brainer.”This same conclusion was reached by a panel of experts on privatization.

The senator’s argument may be broken down into its basic elements (Figure 2):“The senator supports the privatization of the federal highway system (INFORMA-TION), which will bring significant gains in efficiency and a reduction in taxes(CLAIM). Considering the WARRANT that the privatization of public services has

Privatization will bringsignificant gains in efficiencyand a reduction in taxes.

CLAIM

The Senator supportsthe privatization of thefederal highway system.

INFORMATIONThis is definitely a“no brainer.”

QUALIFIER IThe Senator believes that theprivatization of public services hasbeen successful in other areas.

WARRANT

The Senator’s beliefs aboutprivatization are based on theconclusions of an expert panel.

But these areas involvedmass rapid transit, whichis different from thefederal highway system.

BACKING OBJECTIONConsideringthe objections,the claim seemsdoubtful.

QUALIFIER II

But the experts are consultants tothe Senator’s political party. Theconclusions are biased, or at leasthave the appearance of being so.

OBJECTION

However, the experts aredistinguished economists.

REBUTTAL

FIGURE 2Argument map—privatizing transportation



been successful in other areas, QUALIFIER 1 states that “this is definitely a ‘nobrainer.’” Additional elements are then introduced to strengthen the argument. Forinstance, a BACKING has been added (“This is the conclusion of an expert panel onprivatization.”). Finally, an OBJECTION to the backing states that the experts areconsultants to the senator’s party. A weak REBUTTAL then states that the expertsare distinguished economists. The rebuttal is weak because it is not a plausible chal-lenge to the objection, which is about political bias and not academic standing. Asecond OBJECTION states that the management of urban mass transit is differentfrom managing the federal highway system. This appears to be a plausible challenge.Finally, QUALIFIER 1 was that the claim is a “no brainer.” In QUALIFIER 2, thisis reduced to “perhaps,” considering the strength of the objections and rebuttal. Thisconsiderably weakens the CLAIM that “Privatization will bring significant gains inefficiency and a reduction in taxes.”

BOX 1

Mapping a Policy Argument 6. Repeat the same procedure with the backing.

If there is a question whether a statement isa backing or a warrant, look for the one thatis more general. This is the backing.

7. Remember that a warrant or backing may beimplicit and unstated—do not expectarguments to be entirely transparent.

8. Look to the arguments of other stakeholdersto find objections and rebuttals. If possible,obtain objections and rebuttals fromsomeone who actually believes them.

9. Remember that elements may contain rules,principles, or entire arguments.

10. An uncontested argument is static;argumentation, which involves at leasttwo parties, is dynamic and usuallycontested.

11. The initial qualifier usually changes whenobjections and rebuttals are advanced tochallenge the claim.

12. Most qualifiers become weaker, althoughsome stay the same. Some can grow stronger(a fortiori) by withstanding challenges.

13. Argumentation produces “trees” and“chains” involving dynamic processesof argumentation that change over time.

Policy arguments have seven elements: information,claim, qualifier, warrant, backing, objection, andrebuttal. The following guidelines are useful inidentifying and arranging these elements:

1. If possible, identify arguments by performinga stakeholder analysis. Stakeholders are themain source of policy arguments.

2. Start by locating the claim, which is theendpoint or output of the argument. A claimis always more general than the informationon which it is based. Claims involve an“inferential leap” beyond information.

3. Look for language that indicates the degreeof credibility the arguer attaches to theclaim—this is the qualifier.

4. Look for the information that supports theclaim. The information answers twoquestions:What does the arguer have to goon? Is it relevant to the case at hand?

5. Look for the warrant, which in conjunctionwith the information supports the claim.The warrant answers the question:Why is thearguer justified in making the claim on thebasis of the information?



MODES OF POLICY ARGUMENTATIONDistinct modes of argumentation are used to justify policy claims. Modes of argumen-tation, which are specific patterns of reasoning include reasoning from authority,method, generalization, classification, cause, sign, motivation, intuition, analogy,parallel case, and ethics.6 Each of these modes of argumentation and its characteristicreasoning pattern is described in Table 1. Note that more than one mode may be usedin a policy argument.

6Some of the modes presented here draw from Wayne Brockriede and Douglas Ehninger, “Toulmin orArgument: An Interpretation and Application,” Quarterly Journal of Speech 1006 (1960): 45–53; andToulmin, Rieke, and Janik, An Introduction to Reasoning, pp. 213–37. I have added several additional modes.

TABLE 1

Modes of Policy Argumentation with Reasoning Patterns

Mode Reasoning Pattern

Authority Reasoning from authority is based on warrants having to do with the achieved orascribed statuses of producers of policy-relevant information, for example, experts,insiders, scientists, specialists, gurus, power brokers. Footnotes and references aredisguised authoritative arguments.

Method Reasoning from method is based on warrants about the approved status of methodsor techniques used to produce information. The focus is on the achieved or ascribedstatus or “power” of procedures. Examples include approved statistical,econometric, qualitative, ethnographic, and hermeneutic methods.

Generalization Reasoning from generalization is based on similarities between samples andpopulations from which samples are selected. Although samples can be random,generalizations can also be based on qualitative comparisons. In either case,the assumption is that what is true of members of a sample is also trueof members of the population not included in the sample. For example, randomsamples of n � 30 are taken to be representative of the (unobserved and oftenunobservable) population of elements from which the sample is drawn.

Classification Reasoning from classification has to do with membership in a defined class.The reasoning is that what is true of the class of persons or events described in the warrant is also true of individuals or groups described in the information.An example is the untenable ideological argument that because a country hasa socialist economy it must be undemocratic, because all socialist systems areundemocratic.

Cause Reasoning from cause is about generative powers (“causes”) and theirconsequences (“effects”). A claim may be made based on general propositions,or laws, that state invariant relations between cause and effect for example, thelaw of diminishing utility of money. Other kinds of causal claims are based onobserving the effects of some policy intervention on one or more policy outcomes.Almost all argumentation in the social and natural sciences is based on reasoningfrom cause.



Mode Reasoning Pattern

Sign Reasoning from sign is based on signs, or indicators, and their referents.The presence of a sign or indicator is believed to justify the expectation that someother sign or indicator will occur as well. Examples are indicators of institutionalperformance such as “organizational report cards” and “benchmarks” orindicators of economic performance such as “leading economic indicators.” Signsare not causes, because causality must satisfy temporal precedence and otherrequirements not expected of signs.

Motivation Reasoning from motivation is based on the motivating power of goals, values, andintentions in shaping individual and collective behavior. For example, a claim thatcitizens will support the strict enforcement of pollution standards might be based onreasoning that since citizens are motivated by the desire to achieve the goal of cleanair and water, they will support strict enforcement.

Intuition Reasoning from intuition is based on the conscious or preconscious cognitive,emotional, or spiritual states of producers of policy-relevant information. Forexample, the belief that an advisor has some special insight, feeling, or “tacitknowledge” may serve as a reason to accept his or her judgment.

Analogy Reasoning from analogies is based on similarities between relations found in a givencase and relations characteristic of a metaphor or analogy. For example, the claimthat government should “quarantine” a country by interdicting illegal drugs—withthe illegal drugs seen as an “infectious disease”—is based on reasoning that sincequarantine has been effective in cases of infectious diseases, interdiction will beeffective in the case of illegal drugs.

Parallel Case Reasoning from parallel case is based on similarities among two or more casesof policy making. For example, the claim that a local government will be successfulin enforcing pollution standards is based on information that a parallel policy wassuccessfully implemented in a similar local government elsewhere.

Ethics Reasoning from ethics is based on judgments about the rightness or wrongness,goodness or badness, of policies or their consequences. For example, policy claimsare frequently based on moral principles stating the conditions of a “just” or“good” society, or on ethical norms prohibiting lying in public life. Moral principlesand ethical norms go beyond the values and norms of particular individuals orgroups. In public policy, many arguments about economic benefits and costs involveunstated or implicit moral and ethical reasoning.

Argumentation from AuthorityHere claims are based on authority. Whereas information consists of factual reportsor expressions of opinion, the warrant affirms the reliability or trustworthinessof the source of the information. Depending on the social context, authorities maybe kings, magicians, or religious leaders, or they may be scientists, professors, ornews reporters.

In an authoritative argument, the claim reiterates the policy-relevant infor-mation that has been provided by the authority, whose reliability, status, or


sagacity has been underwritten by the warrant. To illustrate (Figure 3), letus imagine that a policy analyst advising the National Security Council at theheight of the 1999 U.S.-NATO attack on Yugoslavia made the designative claim,


Though Western officials continue to deny it, therecan be little doubt that the bombing campaign hasprovided both motive and opportunity for a widerand more savage Serbian operation than whatwas first envisioned.

CLAIM

Carnes Lord, a professor at theFletcher School of Law andDiplomacy and former BushAdministration national securityadvisor, states that “ThoughWestern officials continue todeny it, there can be little doubtthat the bombing campaign hasprovided both motive andopportunity for a wider and moresavage Serbian operation thanwhat was first envisioned.”

INFORMATIONThe claim isalmost certainlytrue.

QUALIFIERMr. Lord has had a distinguishedcareer in government andacademia and wide practicalexperience. He is considered bymany to be an honest, reliable,and perceptive analyst,

WARRANT

Lord’s judgment is consistent with thatof US-NATO Commander Wesley Clark,who stated that the outcome of thebombing was “entirely predictable.”

BACKING

Another reliable source, State Department spokesmanJames Rubin, evades the question by stating that “theUnited States is extremely alarmed by reports of anescalating pattern of Serbian attacks on KosovarAlbanian citizens.”

OBJECTION

Rubin’s alarm merely reflects the ClintonAdministration’s effort to deny responsibilityfor the consequences of the US-led bombingcampaign.

REBUTTAL

FIGURE 3Argumentation from authority—unintended consequences of the U.S.-NATO attackon Yugoslavia


C: “Though Western leaders continue to deny it, there can be little doubt that thebombing campaign has provided both motive and opportunity for a wider andmore savage Serbian operation than what was first envisioned.”7 The informa-tion, I, is from a statement by Carnes Lord, a professor at the Fletcher School ofLaw and Diplomacy and former Bush administration national security advisor.The warrant, W, affirms Lord’s reliability and is backed, B, by an additionalauthoritative argument intended to add to the persuasiveness of the argument.The objection, O, challenges the initial warrant, but without weakening theclaim’s credibility, which is stated in the qualifier, Q. The rebuttal, R, hasvirtually no effect on Q, which does not change in response to the challenge by arival authority, State Department spokesman James Rubin.

MethodArgumentation from method is based on warrants about the approved status ofmethods used to produce information. Policy-relevant information may consist offactual statements or reports. The role of the warrant is to provide a reason foraccepting the claim by associating the information with the use of an approvedmethod, rule, or principle. Usually, the claim is that the condition described in theinformation should be regarded as valuable (or worthless), because of the methodused to produce it. Consider the following public investment problem: An analysthas information I, that the production of energy per dollar is greater in nuclearpower plants than in hydroelectric plants, which in turn produce more energy thensolar plants. The claim, C, is that

The government should invest in nuclear energy. The warrant, W, associates theinformation, I, with claim, C, by invoking the transitivity rule of mathematicaleconomics.8 The warrant is backed, B, by the presumption that transitivity is a“universal selection rule” that guarantees the rationality of choice.

The rebuttal, R, challenges the presumption about the universal validity oftransitivity by pointing to the presence of cyclical preferences. The original qualifierQ1 is reduced from “very likely” to the new qualifier Q2, which is now stated as“quite uncertain” (Figure 4).

In argumentation from method, claims are assessed in terms of the achieved orascribed status of methods or the rules guiding their use. Those who accept theauthority of analytic methods such as econometrics, benefit-cost analysis, a ordecision analysis falsely believe that the use of such methods actually “sets thepolicy agenda and its directions, that useful analysis will be used analysis.”9

The authority of methods need not be derived from rules of formal logic ormathematics, as the history of qualitative methods shows. Many qualitative methodsoriginate in the hermeneutic tradition, which evolved from the interpretation of


7Boston Globe, April 4, 1999. Quoted in Noam Chomsky, Rogue States: The Rule of Force in WorldAffairs (Cambridge, MA: South End Press, 2000), p. 35.8On rules expressing transitive and cyclical preferences, see, for example, Norman Frohlich and Joe A.Oppenheimer, Modern Political Economy (Englewood Cliffs, NJ: Prentice Hall, 1978), pp. 6–13.9Allen Schick, “Beyond Analysis,” Public Administration Review 37, no. 3 (1977): 259.



biblical texts. The authority of qualitative methods, like any other, stems from theprofessional and scientific communities that create definitions of the purpose, scope,and proper application of approved methods.10 These communities are extra-scien-tific sources of authority.

The transitivity rule and other axioms of mathematics have also arisen fromextra-scientific sources:

Consider the axiom which asserts the transitivity of preference: if A ispreferred to B, and B to C, then A is (or rationally must be) preferred to C.The intuitive appeal of this assertion is so great that few if any economistsfeel the urge to build formal economic systems in which the axiom fails.Geometry . . . is the classic example; the intuitive strength of Euclid’s

The government shouldinvest in nuclear energy.

CLAIM

The production of energy per dollaris greater in nuclear (N) power plantsthan hydroelectric (H) plants, whichin turn produce more energy thansolar (S) plants.

INFORMATIONAbsolutely. Thisis a universalselection rule.

QUALIFIER IThe transitivity rule states thatif N is preferred to H, andH is preferred to S, then N ispreferred to S.

WARRANT

But the dangers ofnuclear power outweighthe monetary benefits.

OBJECTION BACKING

But universality cannot be guaranteed.When groups make choices, preferencesare cyclical, so that S is actuallypreferred to N.

OBJECTIONHowever, the dangers ofnuclear power are minimalin new plants ascompared with old ones.

REBUTTAL

Transitivity is a universalselection rule necessary formaking rational choices. Therule is supported by rationalchoice theorists.

Probably not,considering thestrength of theobjections.

QUALIFIER II

FIGURE 4Argumentation from method—intransitivity of preferences for nuclear power

10Approved methods are part of Kuhn’s disciplinary matrix. Thomas Kuhn, The Structure of ScientificRevolutions, 2d ed. (Chicago: University of Chicago Press, 1971), p. 103.


“postulates” was so great that for two thousand years geometers played theirgames strictly within the domain . . . which Euclid had laid down. Even whennon-Euclidean geometries were discovered in the early nineteenth century,most mathematicians never thought of them as valid.11

Adherence to approved methods is wrongly believed to make policy decisionsmore “rational.” A rational choice is thought to be possible if the analyst canorder all consequences associated with action, if the ordering of consequencesis transitive, and if the analyst can consistently and in a transitive fashion choosethe alternative that will bring the greatest benefit in relation to cost.12 Challengesto this argument may be made on empirical, authoritative, intuitive, and ethicalgrounds.13 New schools of analysis may serve as a source of approved methods,new axioms providing for nontransitive preferences may come to be acceptedon intuitive grounds, and rules that run counter to moral principles and ethicalnorms may be replaced with new ones.14 Challenges may also be made on prag-matic grounds, for example, by arguing that the transitivity rule does not actuallypromote better decisions in policy settings characterized by incompleteinformation, value conflicts, multiple competing objectives, partisan mutualadjustment, and “organized anarchy.”15 In the last analysis, a successful argumentfrom method is a pragmatic matter. An argument from method must demonstrateonly that the results of using particular rules are superior to those that occurwithout them and that the observed improvement is a consequence of using therule or procedure.16

GeneralizationArguments from generalization involve samples. Policy-relevant informationconsists of sampled elements—events, persons, groups, organizations, countries—that are taken to be representative of a larger population of the same elements. Thefunction of the warrant is to affirm that what is true of the sampled elements is alsotrue of the unobserved (and often unobservable) elements in the population. Thepolicy claim rests on the argument that the sample is a satisfactory representation ofthe population.


11C. West Churchman, The Design of Inquiring Systems: Basic Concepts of Systems and Organization(New York: Basic Books, 1971), p. 25.12Joseph L. Bower, “Descriptive Decision Theory from the ‘Administrative’ Viewpoint,” in The Study of Policy Formation, ed. Raymond A. Bauer and Kenneth J. Gergen (New York: Free Press, 1968), pp. 104–6.13Good examples of these challenges may be found in Jeffrey Friedman, ed., The Rational ChoiceControversy: Economic Models of Politics Reconsidered (New Haven, CT: Yale University Press, 1996).14See the discussion of the evolution of welfare economics in public policy analysis by Duncan MacRaeJr., The Social Function of Social Science (New Haven, CT: Yale University Press, 1976), pp. 107–57.15See the chapter “Policy Analysis in the Policy Process.”16Bower, “Descriptive Decision Theory from the ‘Administrative’ Viewpoint,” p. 106. See NicholasRescher’s essays on methodological pragmatism, for example, Induction (Pittsburgh, PA: Universityof Pittsburgh Press, 1980).



To illustrate, consider the director of a community food bank who wantsto know whether persons receiving food are getting an adequate daily allowanceof calcium, one of the most important minerals in the body (Figure 5). Thedirector, who is attentive to reasons that might justify additional funding forthe food bank, makes the claim, C, that it is “pretty likely” that food bank clientsare receiving sufficient calcium—specifically, given that the sampled amountof 755 milligrams (mg) of calcium could have occurred by chance 9 timesout of 100, it is not reliably different from the recommended daily allowance(RDA) of 800 mg, an amount prescribed by the Food and Nutrition Board of theNational Academy of Sciences.

In this case, the information, I, describes the average daily intake of calcium(755 mg) measured in a random sample (sometimes inappropriately called a“scientific sample” in policy circles) of fifty clients. The information, I, indicatesthat the difference between 755 mg and 800 mg could occur by chance 9 timesout of 100, a conclusion reached on the basis of a statistical test (a “two-sampletest”). The probability value (p � 0.09) is included in the qualifier, Q, with theordinary words “probably true.” The warrant, W, which provides the reason formoving from I to C, has two parts: The rule that a random sample of at least 30 isadequate in such cases to generalize to the population from which the sampleis selected; and the practice of accepting a level of statistical significance of p � 0.05 (5 times out of 100) as an “acceptable” level of risk. When pressed foradditional justification, the director checks her statistics text to find theappropriate theoretical backing, B, for the rule n � 30. The B is the Central LimitTheorem of probability theory.

A member of the director’s staff believes that clients served by the food bankmay have a calcium deficiency. He therefore challenges the claim with severalrebuttals: the data on calcium intake are unreliable because of the way they werecollected; and the sample is biased because it is not a random sample. The rebuttalsare plausible, and the director’s qualifier, Q1, changed from “probably true” to Q2“not likely” that clients are receiving their minimum RDA of calcium. The director’soriginal claim—that calcium intake is probably adequate—is diminished by theobjections of the well-meaning and critical staff member.

Arguments from generalization are not always statistical, in the specific sensethat statistics are estimates of population values (called parameters). Nonrandomsamples—for example, purposive samples, theoretical samples, and sociometric(snowball) samples—do not permit statistical estimation. They are neverthelessuseful in making claims about populations.17 Even case studies (where n � 1) maybe used to generalize to wider populations by means of various pattern-matchingmethods.18

17Generalizability (external validity) in experimental research is essentially nonstatistical. Among otherreasons this is why Campbell called it “proximal similarity.” See William R. Shadish, Thomas D. Cook,and Donald T. Campbell, Experimental and Quasi-Experimental Designs for Generalized CausalInference (Boston, MA: Houghton Mifflin, 2002).18See William N. Dunn, “Pattern Matching: Methodology,” International Encyclopedia of the Socialand Behavioral Sciences (New York: Elsevier, 2002).



Food Bank clients are receiving sufficient calcium intheir diet. There is no statistically significant differencebetween their daily intake of 800 mg of calcium andthe recommended daily allowance (RDA) of 755 mgapproved by the Food and Drug Administration.

CLAIM

The average daily intakeof calcium in a sampleof 50 Food Bank clientsis 755 mg. The recommended dailyallowance (RDA) is800 mg.

INFORMATIONThe claim isprobably true.

QUALIFIER I

No, it is probablyuntrue

QUALIFIER II

A random sample of 30 or more persons isusually acceptable as a basis for generalizingfrom the sample to the population. Given asample of 30, a test was performed todetermine the statistical significance of adifference in calcium intake of 45 mg(800–755). The test indicates thatthe difference is not statistically significantat the p = 0.05 level.

WARRANT

The Central Limit Theoremjustifies a sample size of 30and a statistical test basedon the normal distribution,also known as the bell-shaped curve.

BACKING

The sample was not drawn randomly, but was basedon clients who visited the Food Bank on the week-ends, when dieticians seemed to be available tointerview clients. Tests of significance are notvalid without a random sample.

OBJECTION

An evaluation of records showingthe intake of calcium indicatedthat some clients were asked toestimate their calcium intake,while other were required toprovide evidence of what foodsthey ate. Data on calcium intakeare unreliable and should notbe used for statistical tests.

OBJECTION

However, the weekend visitors were selected randomly bymeans of an interval sample of every 5th client who entered theFood Bank.

REBUTTAL

FIGURE 5Argumentation from generalization—statistical inference as a basis for success in community nutrition



ClassificationArgumentation from classification focuses on membership in a defined class. Thereasoning is that what is true of the class of persons or events described in the warrant is also true of individuals or events that are members of the class, as theyare described in the information. To illustrate, consider the following argumentabout the relationship between regime type and the control of terrorism (Figure 6).The information, I, is that Iran, Iraq, and the Palestinian Authority are authoritariandictatorships. The claim, C, is that Middle Eastern states can control terrorismwithin their borders. The warrant, W, is that authoritarian dictatorships exercise firmcontrol of terrorists and other armed groups within their territories. The backing, B,has two parts. The first is that firm control exercised by authoritarian dictatorshipsis possible because such regimes do not permit the private use of weapons withoutthe express approval of the government. The objections, O, are that one or moreof the Middle Eastern regimes lack such control because they permit the privateuse of weapons; and that the definition of “authoritarian regime” may be inappro-priate. The original qualifier Q1 (“virtually certain”) is reduced to Q2 (“not . . . certain at all”).

Middle Eastern statesare authoritariandictatorship.

INFORMATIONThis is virtuallycertain.

QUALIFIER IA key characteristic of authoritarian dictator-ships is that they exercise firm control overarmed groups within their borders.

WARRANT

Firm control is possible becausethe private use of weapons isprohibited without the expressapproval of state authorities.

BACKINGConsidering theobjections, theclaim does notseem certainat all.

QUALIFIER IIUnless the definitionof authoritariandictatorship, which wasdeveloped tocharacterize communistregimes, is not usefulas a way to understandMiddle Eastern states.

OBJECTION

But some states lack such control becausethey do not prohibit the private use ofweapons. This is not be an importantcharacteristic of authoritarian dictatorshipsafter all.

OBJECTION

Middle Eastern states can controlterrorist activities within their bordersif they wish to do so.

CLAIM

FIGURE 6Argumentation from classification—challenging claims about authoritarian rule andterrorism



The plausibility of classification arguments depends on the completeness andinternal consistency of the definition of the class. Various classes of politicalregimes—authoritarian dictatorships, totalitarian democracies, socialist democra-cies, capitalist democracies—are less homogeneous and internally consistent thanpopular classifications suggest. The same is true for classes of policies (e.g.,“privatization”), organizations (e.g., “bureaucracy”), political doctrines (e.g.,“liberal” and “conservative”), and groups (e.g., “lower class,” “middle class,”“upper class”). Many apparently simple classifications may turn out to be ideologiesin disguise.

CauseArgumentation from cause focuses on the causes and effects of policies.19 In causalarguments, information consists of one or more evidently factual statements orreports about a policy. The warrant transforms these statements or reports by relat-ing them to generative powers (causes) and their results (effects). The claim relatesthese causes and effects back to the information supplied.

The role of causal arguments in transforming policy-relevant informationinto policy claims may be illustrated by Allison’s well-known causal explana-tions of foreign policy behavior during the Cuban missile crisis in October1962.20 Showing how different models yield alternative explanations of foreignpolicy, Allison argues that (1) government policy analysts think about problemsof foreign policy in terms of implicit conceptual models that shape their thought;(2) most analysts explain the behavior of governments in terms of one basicmodel, one that assumes the rationality of political choices (rational policymodel); and (3) alternative models, including those that emphasize organiza-tional processes (organizational process model) and bureaucratic politics(bureaucratic politics model), provide bases for improved explanations offoreign policy behavior.

19Causal argumentation dominates efforts of political scientists to explain policy making. Recentexamples include contributors to Sabatier, Theories of the Policy Process, 2nd ed.(Boulder, CO:Westview Press, 2007). Other examples are James E. Anderson, Public Policy-Making (New York:Praeger Publishers, 1975); Thomas R. Dye, Understanding Public Policy, 3d ed. (Englewood Cliffs, NJ:Prentice Hall, 1978); Robert Eyestone, The Threads of Public Policy: A Study in Policy Leadership(Indianapolis, IN: Bobbs-Merrill, 1971); Jerald Hage and J. Rogers Hollingsworth, “The First Stepstoward the Integration of Social Theory and Public Policy,” Annals of the American Academy ofPolitical and Social Science, 434 (1977): 1–23; Richard I. Hofferbert, The Study of Public Policy(Indianapolis, IN: Bobbs-Merrill, 1974); Charles O. Jones, An Introduction to the Study of PublicPolicy, 2d ed. (North Scituate, MA: Duxbury Press, 1977); Robert L. Lineberry, American PublicPolicy: What Government Does and What Difference It Makes (New York: Harper & Row, 1977);Austin Ranney, ed., Political Science and Public Policy (Chicago: Markham, 1968); Richard Rose, ed.,The Dynamics of Public Policy: A Comparative Analysis (Beverly Hills, CA: Sage Publications, 1976);Ira Sharkansky, ed., Policy Analysis in Political Science (Cambridge, MA: Markham, 1970); and PeterWoll, Public Policy (Cambridge, MA: Winthrop Publishers, 1974).20Graham T. Allison, “Conceptual Models and the Cuban Missile Crisis,” American Political ScienceReview 3002, no. 3 (1969): 689–718.



In contrasting alternative models, Allison examines explanatory argumentsderived from the three conceptual models. In 1962, the policy alternativesopen to the United States ranged from no action and diplomatic pressures tosecret negotiations, invasion, surgical air strike, and blockade. The alternativesare assessed in terms of rival explanations of foreign policy behavior. The expla-nations conform to a type of causal explanation that philosophers call deductive-nomological (D-N), which holds that valid explanations are possible only whengeneral theoretical propositions or laws link prior circumstances with subsequentevents.21

Among the several advocative claims made at the time of the Cuban missile crisis,let us consider the policy recommendation actually adopted by the United States:“The United States should blockade Cuba.” In this case, the policy-relevant informa-tion, I, is “The Soviet Union is placing offensive missiles in Cuba.” The warrant, W, issince “the blockade will force the withdrawal of missiles by showing the Russians thatthe United States is determined to use force.” In providing additional reasons to acceptthe claim, the backing, B, states because “[a]n increase in the cost of an alternativereduces the likelihood of that alternative being chosen.”22 B represents a general theo-retical proposition, or law, within the rational policy model (Figure 7). In this case,Q1 (probably) changes to Q2 (probably not) after the objection, O, has successfullychallenged the backing B.

The purpose of Allison’s account is not to demonstrate the inherent superiorityof one or another of the three explanatory models. It is rather to show that the useof multiple competing models can result in improved explanations of foreign policybehavior. The use of multiple models moves analysis from a self-contained and staticsingle argument about the relation between information and claim to a new stage ofdynamic debate. In this context, the organizational, process model provides anobjection, O, with its own backing, B, in the form of a competing causal argument.The objection, O, states that Soviet leaders may be unable to force their own orga-nizational units to depart from assigned tasks and routines. This can be expectedto occur because “[m]ajor lines of organizational behavior are straight, that is,behavior at one time is marginally different from that same behavior at t � 1.”23

The backing, B, for the objection, O, is again a general proposition or law withinthe organizational process model.

The case of the Cuban missile crisis illustrates some of the limitations ofcausal argumentation based on D-N explanation. First, several competing causalarguments are equally compatible as general propositions or laws. These argu-ments, each backed by social science theories, cannot be confirmed or refutedsolely on the basis of this or any other information or data.24 Second, a givencausal argument, however persuasive, cannot directly lead to an advocative claim

21See Carl G. Hempel, Aspects of Scientific Explanation (New York: Free Press, 1965).22Allison, “Conceptual Models,” p. 694.23Ibid. p. 702.24Georg H. von Wright, Explanation and Understanding (Ithaca, NY: Comell University Press, 1970),p. 145; and Kuhn, Structure of Scientific Revolutions.



or recommendation, because traditional causal explanations do not themselvescontain value premises.25 Indeed, there is a suppressed value premise, which isthat U.S. leaders are motivated by the value of security from the Soviet militarypresence in the Western Hemisphere. If some other value had motivated policymakers, either of the two competing causal explanations would have supportedaltogether different claims, for example, that the United States should invade andoccupy Cuba.


CLAIM

Reliable intelligencereports confirm that theSoviet Union is placingoffensive missiles in Cuba.

INFORMATIONA blockade will show Soviet leadersthat the United States means businessbecause it is prepared to use force.

WARRANTProbablyQUALIFIER I

An increase in the cost of an action reduces the likelihood that it will be taken.

BACKING

But Soviet leaders may not convince their naval units to depart from standard operating procedures which permit only marginal changes in behavior.

OBJECTION

Research on organizations shows that major lines ofbehavior are straight: Behavior at time t+1 is almost alwaysmarginally different from behavior at time t.

WARRANT

Consideringthe objections,probably not.

QUALIFIER II

FIGURE 7Argumentation from theoretical cause—competing deductive-nomologicalexplanations of the cuban missile crisis

25This is not to say that they do not imply values, because empirical theories in the social and naturalsciences rest on unstated value premises. See, for example, M. Gunther and K. Reshaur, “Science andValues in Political ‘Science,’ ” Philosophy of Social Sciences 1 (1971): 113–21; J. W. Sutherland,“Axiological Predicates in Scientific Enterprise,” General Systems 19 (1974): 3–14; and Ian I. Mitroff,The Subjective Side of Science: A Philosophical Inquiry into the Psychology of the Apollo MoonScientists (New York: American Elsevier Publishing, 1974).



Causal argumentation based on the D-N model of scientific explanationattempts to develop and test general propositions about causes and effects.26 CarlHempel has elaborated D-N explanation as follows:

We divide explanation into two major constituents, the explanandum andthe explanans. By the explanandum, we understand the sentence describingthe phenomenon to be explained (not that phenomenon itself); by theexplanans the class of those sentences which are adduced to account for thephenomenon. . . . [Scientific explanation] answers the question, “Why didthe explanandum-phenomenon occur?” by showing that the phenomenonresulted from particular circumstances, specified in C1, C2, . . . , Ck, inaccordance with laws L1, L2, . . . , Lr. By pointing this out, the argumentshows that, given the particular circumstances and the laws in question, theoccurrence of the phenomenon was to be expected; and it is in this sense thatthe explanation enables us to understand why the phenomenon occurred.27

A simple example illustrates traditional causal (D-N) explanation.28 If I leave mycar outside overnight and the temperature drops below freezing, my full radiator(without antifreeze) will burst. Why will this happen? “My radiator burst”(explanandum). “My radiator was full of water, the cap was tightly fastened, and thetemperature outside dropped below freezing” (circumstances, or Ck, in theexplanans). And, “the volume of water expands when it freezes” (general propositionor law, Lr, in the explanans).29 In this example, knowledge of prior circumstances andthe appropriate law permits a prediction of the resultant event.

Questions have been raised about the suitability of D-N explanation in history andthe social sciences.30 These questions arise, among other reasons, because policy analysisand other social sciences are partly evaluative and advocative (normative) in character.Every advocative claim contains both factual and value premises, whereas in traditionalcausal explanations, we evidently find only factual premises. Traditional causal explana-tions also require that the explanans precede (or accompany) the explanandum. Yetmany advocative claims reverse this sequence, insofar as circumstances that explainaction are situated in the future. Future circumstances, including intentions, goals, anddesires, explain present actions to the extent that actions cannot occur without themotivation provided by such intentions, goals, and desires.31 Finally, any correspon-dence between the results of acting on an advocative claim and the conclusions of acausal argument may be purely coincidental. In policy making, predictions based onD-N explanations will fail if policy actors employ intelligent reflection to change theirbehavior or if unpredictable factors deriving from creative thought intervene.32

26Allison, “Conceptual Models,” p. 690 (note 4) tells us that Hempel’s D-N (deductive-nomological)explanation is the basis (backing) for his three models.27Hempel, Aspects of Scientific Explanation, pp. 247–58.28See von Wright, Explanation and Understanding, p. 12.29Ibid., p. 12.30Ibid., p. 11.31Ibid., pp. 74–124; G. E. M. Anscombe, Intention (London: Basil Blackwell, 1957); and W. H. Dray,Laws and Explanation in History (London: Oxford University Press, 1957).32Alasdair Maclntyre, “Ideology, Social Science, and Revolution,” Comparative Politics 5, no. 3 (1973): 334.



D-N explanation is not the sole legitimate form of causal argumentation in publicpolicy. Another form is hypothetico-deductive (H-D) explanation, which involves thededuction of hypotheses from theories that do not involve propositions about invariantcausal relations or laws. Often the hypotheses of interest are those dealing with policyor program actions designed to achieve some practical outcome.33 The relationbetween action and outcome, however, is not certain. If it were, it would conform tothe following requirements, usually called the “essentialist” view of causation:

� The policy, x, must precede the outcome, y, in time.� The occurrence of the policy, x, must be necessary for the occurrence of y, the

outcome, which must not occur in the absence of the policy, x.� The occurrence of the policy, x, must be sufficient for the occurrence of y, the

outcome, which must occur when the policy, x, is present.

If these requirements were met, the relation between a policy action and an outcomewould be certain. These requirements are virtually never satisfied in real-life policysettings. Instead, what occurs is that other conditions—uncontrolled contingenciesthat lie beyond the control of policy makers—make it impossible to know definitelywhether a policy is necessary or sufficient for the occurrence of an outcome. Theuncontrolled contingencies are plausible rival hypotheses that must be taken intoaccount and, when possible, eliminated as competing explanations of a policyoutcome. Here the best that may be expected is an optimally plausible claim, that is,an approximately valid causal inference.34 Figure 8 displays causal argumentationin the quasi-experimental tradition founded by Donald T. Campbell, a traditionbased on the fundamental premise that causal argumentation in real-life policysettings requires the formulation, testing, and elimination of rival hypotheses.35

Figure 8 shows that rival hypotheses used to challenge the warrant change the initialqualifier from “definitely worthwhile” (QI) to “perhaps” (QII).

SignReasoning from sign is based on indicators and their referents. The presence of asign indicates the presence of an event, condition, or process, because the sign andwhat it refers to occur together. Examples are indicators of institutional perform-ance such as “organizational report cards,” “benchmarks,” and “best practices.”36

Another example is the widely used set of indicators (actually indices) of economicperformance—“leading,” “lagging,” and “coincident” economic indicators—publishedperiodically by the Conference Board. Signs are not causes. As we saw earlier,

33See the discussion of the “activity theory of causation” in Thomas D. Cook and Donald T. Campbell,Quasi-Experimentation: Design and Analysis Issues for Field Settings (Boston, MA: Houghton Mifflin,1979), ch. 1.34The preeminent source on validity questions in the social and behavioral sciences is Shadish, Cook,and Campbell, Experimental and Quasi-Experimental Designs for Generalized Causal Inference.35Rival hypotheses are also known as “threats to validity.” See Donald T. Campbell, Methodology andEpistemology for Social Science: Collected Papers, ed. E. S. Overman (Chicago: University of ChicagoPress, 1988). The example is from Campbell’s “Reforms as Experiments,” in the Overman edition.36For example, William T. Gormley Jr. and David L. Weimer, Organizational Report Cards (Cambridge,MA: Harvard University Press, 1999).



causality must satisfy additional requirements not expected of signs. Figure 9displays argumentation from sign based on a correlation coefficient and probabilityvalue (p-value).

Figure 9 makes an important distinction between signs and causes. In modernstatistical analysis, measures of correlation, regression, and statistical

Moreover, the statistical phenomenon of regression toward the mean is often found in time series of this kind.

WARRANT

Connecticut Governor Abraham Ribicoff contended that the crackdown on speeding is responsible for saving 40 lives in 1956, a reduction of 12.3 percent from the 1955 death toll. The program is “definitely worthwhile,” said Ribicoff.

CLAIM

Data on Connecticuttraffic fatalities between1955 and 1956 show a12.3 percent decline.

INFORMATION“definitelyworthwhile...”

QUALIFIER I

Considering theobjection, a better word is “perhaps.”

QUALIFIER II

We reduced fatalities “by enforcing thelaw, something the safety experts saidcouldn’t be done because the peoplewouldn’t be behind it,” said Ribicoff.

WARRANT

But fatalities in 1955were at an all time high.In 1956, fatalitiesregressed toward theaverage (mean) numberof fatalities in the timeseries. This and not thepolicy was the realcause of the decline.

OBJECTION

FIGURE 8Argumentation from practical cause—rival explanations of the effects of theconnecticut crackdown on speeding



significance (e.g., chi-square, t, and F values) are signs that refer to covariation.Covariation is a necessary but not sufficient condition of causation. In Figure 9,the rebuttal, R, states a causal relation between participation in Head Startprograms and increased graduation rates. It is a causal relation because studentswho participated had higher graduation rates; those who did not had lower grad-uation rates; the program came before the graduation in time; and there was,in addition, a positive correlation between participation and graduation. It ispositive, moderate in strength (r � 0.61), and statistically significant (p � 0.05).The objection, O, successfully challenges the argument from sign, which mistak-enly infers that Head Start is ineffective. The initial qualifier QI (“probably”)becomes QII (“probably not”).

Arguments from sign—whether presented as organizational report cards,benchmarks, or correlation coefficients—are at best statements of covariationor coincidence. Although covariation must be present for a causal relation toexist, causation requires that we fulfill certain conditions. John Stuart Mill,

Head Start programs are not effective in preparing disadvantaged students for graduation.

CLAIM

According to the expert, the larger the number of students enrolled in Head Start programs, the smaller the percentage of graduates. The correlation between number of students enrolled and number of graduates is negative (r = – 0.14) and statistically significant (p < 0.01).

INFORMATIONThis is probably true.

QUALIFIER IAn expert in a cable newsshow reported that participationin Head Start programs causesstudents to fail. The larger thenumber of participants in theseprograms, the lower thegraduation rate.

WARRANT

But participants should be compared to non-participants. A peer-reviewed study compared participants and non-participants in Head Start programs. Participants were 3 timesmore likely to graduate than non-participants (r = 0.57, p < 0.01).

OBJECTIONNo, it is probablyuntrue, considering the strong objection and weak rebuttal.

QUALIFIER II

However, this does not “prove” causation.REBUTTAL

Causation is more or less plausible, not perfect.OBJECTION

FIGURE 9Argumentation from sign—quantitative indicators such as correlation coefficients andP-values do not “prove” causation



the nineteenth-century English philosopher, presented a set of methods (some-times called “canons”) of inductive inference designed to discover causalrelations. Mills’s methods are broadly employed today in the social and behav-ioral sciences, policy analysis, and program evaluation.37 The methods are thoseof agreement, difference, agreement and difference (the so-called “jointmethod”), concomitant variation, and residues. The basic idea of the first threemethods is as follows:

� If on two or more occasions a presumed effect has only one antecedentcondition in common, then that condition is probably the cause of thepresumed effect. If on the first occasion, presumed effect Y is preceded by conditions X1, X3, and X5, and on the second occasion the presumedeffect Y is preceded by conditions X2, X3, and X6, then X3 is probablythe cause of Y.

� If a presumed effect and a presumed noneffect share every antecedentcondition except one, which occurs along with the presumed effect, thenthat condition is probably the cause of the presumed effect. If the presumedeffect Y and the presumed noneffect ~ Y share antecedent conditions X1,X2, X5, and X6, but do not share condition X3, which occurs along withpresumed effect Y, then X3 is probably the cause of Y.

� If two or more occasions when a presumed effect occurs have only oneantecedent in common, whereas two or more occasions when a presumed effectdoes not occur have nothing in common except the absence of that antecedentcondition, then the antecedent condition in which the presumed effects andpresumed noneffects differ is probably the cause. If on two or more occasionswhen presumed effect Y occurs it is accompanied solely by antecedent conditionX3, whereas two or more occasions when presumed effect Y does not occurhave nothing in common except the absence of antecedent condition X3, thenX3 is probably the cause of Y.

The method of concomitant variation, which we know today as correlation(covariation, association), does not require additional comment here, except to saythat it is a necessary condition of causation. The method of residues is similarto what we now call the analysis of residual (error) variance in econometrics.The logic is that what is “left over” when we have explained the effects of all the(presumed) causes of a phenomenon are the “residues” of other possible (andusually unknown) causes. To know whether we are analyzing causation or correla-tion, however, requires that we first employ the first three methods, because nostatistical analyses, however advanced, are sufficient to establish causation.Statistics provides signs, not causes.

37Quasi-experimental design in program evaluation is based very largely on Mills’s methods. The bestexample is Cook and Campbell, Quasi-Experimentation, ch. 1, who critique and go beyond Mills’smethods. For another critique and reformulation, see William N. Dunn, “Pragmatic EliminativeInduction,” Philosophica 60, no. 2 (1997), Special issue honoring Donald T. Campbell. Comparativepolitical science, comparative sociology, comparative public policy, and experimental psychology havealso drawn on Mills’s methods.



MotivationIn motivational arguments, claims assert that an action should be adopted becauseof the motivating power of intentions, goals, or values. Motivational argumentsseek to demonstrate that the intentions, goals, or values underlying a recommendedcourse of action are such as to warrant its acceptance, adoption, or performance. Itis often sufficient to know that large or important groups actually desire to followthe course of action stated in the claim.

Argumentation from motivation represents a form of reasoning that philoso-phers since Aristotle have called the practical syllogism, or practical inference.In practical inference, the major premise or warrant, W, describes some desired stateor end of action, whereas the minor premise or information, I, relates a course ofaction to this desired state as a means to an end. The conclusion or claim, C, consists of a recommendation to act in a certain way to secure the desired state orend. Whereas in theoretical inference acceptance of a general proposition or lawleads to the conclusion or claim, in practical inference acceptance of a premise aboutgoals, values, or intentions leads to a conclusion or claim about actions that are inaccordance with the premises.38 Claims in practical inference are usually designed tounderstand actions, whereas claims in theoretical inference seek to explain events.39

Practical reasoning is of great importance to policy analysis, a field in which oneof the chief problems is to explain actions in terms of goals, values, and intentions:

The practical syllogism provides the sciences of man with something long missingfrom their methodology: an explanation model in its own right which is a definitealternative to the subsumption-theoretic covering law model [i.e., deductive-nomological explanation—author]. Broadly speaking, what the subsumption-theoretical model is to causal explanation and explanation in the natural sciences,the practical syllogism is to teleological explanation and explanation in historyand the social sciences.40

Motivational argumentation not only provides an alternative explanatory modelfor policy analysis but also compels us to conceptualize policy making as a politi-cal process. Arguments from motivation force analysts to think in terms of thegoals, values, and intentions of policy actors and to “enter the phenomenologicalworld of the policy maker.”41 Motivational arguments also bring us closer toquestions of values and ethics, which in other modes of policy argumentation arefrequently and regrettably separated from “factual” matters. Figure 10 presents anargument from motivation that is challenged by another argument frommotivation and an argument from classification. In this example, QI, (“definitely”)becomes QII (“probably”) after the rebuttal.

38von Wright, Explanation and Understanding, pp. 22–27.39Ibid., pp. 22–24. See also Fred R. Dallmayr and Thomas A. McCarthy, eds., Understanding andSocial Inquiry (Notre Dame, IN: University of Notre Dame Press, 1977); and Rein, Social Science andPublic Policy (Baltimore, MD: Penguin 1976), pp. 14–15, 139–70.40von Wright, Explanation and Understanding, p. 27.41R. A. Bauer, Study of Policy Formation (New York: Free Press, 1971), p. 4.



IntuitionIn reasoning from intuition, policy claims are based on assumptions about theinsight of participants in the policy-making process. Policy-relevant informationconsists of factual reports or expressions of opinion. The function of the warrantis to affirm that inner mental states (insight, judgment, understanding) of producersof information make them specially qualified to offer opinions or advice. The policyclaim may simply reiterate the report or opinion supplied in the information.Consider this example of early military policy:

When in 1334 the Duchess of Tyrol, Margareta Maultasch, encircled thecastle of Hochosterwitz in the province of Carinthia, she knew only too wellthat the fortress, situated on an incredibly steep rock rising high above thevalley floor, was impregnable to direct attack and would yield only to a longsiege. In due course, the situation of the defenders became critical: they weredown to their last ox and had only two bags of barley corn left. Margareta’ssituation was becoming equally pressing, albeit for different reasons: hertroops were beginning to be unruly, there seemed to be no end to the siege insight, and she had similarly urgent military business elsewhere. At this pointthe commandant of the castle decided on a desperate course of action, whichto his men must have seemed sheer folly; he had the last ox slaughtered, had itsabdominal cavity filled with the remaining barley, and ordered the carcass

Congress should pass the Equal Rights Amendment (ERA).

CLAIM

A recent survey shows that the majorityof citizens are motivated by the desire toprevent discrimination against women.

INFORMATIONDefinitelyQUALIFIER I

If not definitely,probably, given the weak but importantobjection.

QUALIFIER II

The ERA is needed to prevent discrimination against women.

WARRANT

However, gender equality is a precondition of all other values including representative government.

REBUTTAL

But our system of government is a republic, where representatives of the people make policy.Most members of Congress are against the ERA.

OBJECTION

FIGURE 10Argumentation from motivation—support for the equal rights amendment



thrown down the steep cliff onto a meadow in front of the enemy camp. Uponreceiving this scornful message from above, the discouraged duchess abandonedthe siege and moved on.42

Figure 11 serves to emphasize some of the unique advantages of insight, judg-ment, and tacit knowledge in developing creative solutions to problems. However,it also points to difficulties. Although policy scholars urge that intuition, judgment,and tacit knowledge be incorporated into policy analysis,43 it is seldom possible

42Quoted in Paul Watzlawick, John Weakland, and Richard Fisch, Change: Principles of ProblemFormation and Problem Resolution (New York: W. W. Norton & Company, 1974), p. xi.

The men should throw downthe last ox.

CLAIM

The commandant believes that throwingdown the last slaughtered ox will causethe duchess to abandon the siege.

INFORMATIONProbablyQUALIFIER

The commandant hasproved to be capable ofsound intuitive judgments.

WARRANT

However, the Commandant has nevercommunicated with the Duchess, directly orthrough an intermediary.

REBUTTAL

The commandant understands (tacitly knows)that the duchess will abandon the siege ifthere is no longer any opposition.

BACKING

But the commandant has asecret deal with the Duchess.

OBJECTION

FIGURE 11Argumentation from intuition—a counterintuitive solution avoids certain death

43For example, Yehezkel Dror, Ventures in Policy Sciences (New York: American Elsevier Publishing,1971), p. 52; Sir Geoffrey Vickers, The Art of Judgment: A Study of Policy Making (New York: BasicBooks, 1965); and Edgar S. Quade, Analysis for Public Decisions (New York: American Elsevier, 1975),pp. 4–5. Some observers have also commented favorably on the possibility of drug-induced changes inthe mental states of policy makers. See Kenneth B. Clark, “The Pathos of Power: A PsychologicalPerspective,” American Psychologist 26, no. 12 (1971): 1047–57.



to identify in advance the methods of reasoning that are likely to yield insight orcreativity. A creative act, observes Churchman, “is an act that cannot be designedbeforehand, although it may be analyzable in retrospect. If this is the correct meaningof creativity, then no intelligent technician can be creative.”44

AnalogyReasoning from analogies and metaphors is based on similarities betweenrelationships found in a given case and relationships found in a metaphor, analogy,or allegory (Figure 12). For example, the claim that government should“quarantine” a country by interdicting illegal drugs—with the illegal drugspresented as an “infectious disease”—is based on reasoning that because quarantinehas been effective in cases of infectious diseases, interdiction will be effective in thecase of illegal drugs. In arguments from analogy, claims are based on assumptionsthat relationships among two or more cases (not the cases themselves) are essentially

Congress should pass the EqualRights Amendment (ERA).

CLAIM

The ERA is analogous to theprevention of discriminationagainst women.

WARRANTPublic opinion polls show thatmost citizens are againstdiscrimination against women.

INFORMATIONCertainlyQUALIFIER I

Because both types of legislationdemonstrates the general relationship between legislation andthe prevention of discrimination.

BACKING

But this is a false analogy. Legislationagainst racial discrimination wasaccomplished through an ExecutiveOrder, not an amendment to theconstitution. The latter is not politicallyfeasible at present.

OBJECTION

Considering the strongobjection, this is notfeasible.

QUALIFIER II

FIGURE 12Argumentation from analogy—the equal rights amendment and false analogy

44Churchman, The Design of Inquiring Systems, p. 17.



similar. For example, claims about ways to reduce employment discriminationagainst women are sometimes based on assumptions about the success of policiesdesigned to reduce discrimination against ethnic minorities.45

Parallel CaseReasoning from parallel case focuses on similarities among two or more cases ofpolicy making (Figure 13). For example, a reason that a local government shouldstrictly enforce pollution standards is that a parallel policy was successfully imple-mented in a similar local government elsewhere. Here, the claim is based on theassumption that the results of policies adopted in similar circumstances are worth-while or successful. Government agencies in the United States and abroad often facesimilar problems, and policy claims may be based on their common experiences.The British experiences with comprehensive medical care and city planning

45Lineberry, American Public Policy, p. 28.

The United States should adopt theDutch model and decriminalize theuse of illegal drugs.

CLAIM

The Netherlands hasbeen successful indecriminalizingthe use of illegal drugs.

INFORMATIONThis seemsplausible.

QUALIFIER I

The proposallacks merit,considering thestrength of theobjection.

QUALIFIER II

Because the United States and the Netherlands have similar problems with drug addicts in urban centers. Both countries are industrial democracies in which there is broad public support for policies to deal with the use of illegal drugs.

WARRANT

But this is a false parallel. Dutch politicalculture is tolerant and oriented towardrehabilitation, whereas American politicalculture is less tolerant and oriented towardpunishment. Two-thirds of inmates in United States federal prisons are there for offenses which include the use of illegal drugs.

OBJECTION

FIGURE 13Argumentation from parallel case—the dutch model and false parallel



(“new towns”), and the Dutch and Swiss approaches to the decriminalization ofillegal drugs, have influenced debates about urban policy and drug policy in theUnited States. The experience of some states in adopting taxation, open housing,and equal employment opportunity policies has been used as a basis for policy at thefederal level.46 A variation of argumentation from parallel case is an argumentbased on the experience of the same agency over time. Past policies in the sameagency are used to support claims that the agency should adopt particular courses ofaction, usually those that are marginally different from the status quo. Claims aboutfederal and state budgetary policies are typically based on assumptions about simi-larities with past policies adopted in the same agency.47

EthicsReasoning from ethics is based on the rightness or wrongness, goodness orbadness, of policies or their consequences (Figure 14). For example, policy claimsare frequently based on moral principles stating the conditions of a “just” or“good” society, or on ethical norms prohibiting lying in public life. Moralprinciples and ethical norms go beyond the values and norms of particular individ-uals or groups. In public policy, many arguments about economic benefits andcosts involve unstated or implicit moral and ethical reasoning. The warrant in anargument from ethics provides reasons for accepting a claim by associating it withsome moral principle or ethical rule. The claim is that the person, situation,or condition referred to in the information should be regarded as valuable orworthless, or that a policy described in the information should or should notbe adopted.

To illustrate ethical argumentation, consider Figure 14. Here, the evaluativeclaim, C, is that “the existing distribution of income in the United States is unjust.”The information, I, is the following:

In 1975, the top 20 percent of American families received 41 percent of allincome, whereas the bottom 20 percent received 5.4 percent. In 1989, the topand bottom percentages were 46.0 and 3.8 percent; in 2007 they were 49.0and 3.4 percent. From 1975 to 2007, average incomes increased by about 3 percent, and winners compensated losers through social service and welfareprograms. But the United States has the greatest income equality of any indus-trialized state.

The warrant, W, is the Pareto rule, named after the Italian economist and sociologistVilfredo Pareto (1848–1923). The Pareto rule is a simple ethical principle supported bymany policy economists.48 The rule states that “an optimum distribution of income insociety is one in which some individuals benefit without others losing.” The backing, B,

46Ibid.47See, for example, Aaron Wildavsky’s now classic treatment of incremental policy making inThe Politics of the Budgetary Process (Boston: Little, Brown and Company, 1964).48See Peter G. Brown, “Ethics and Policy Research,” Policy Analysis 2 (1976): 332–35.



for the warrant is “Since income is earned through ability, hard work, and one’s ownlabor.” The objection, O, is “But many of the winners receive income through fraud,discrimination, and unearned inheritance.” The partial rebuttal, R, is “However,persons have a right to their own family’s property” (Figure 14).

Figure 14 shows how an important contending moral principle, John Rawls’sprinciple of “justice as fairness,” can also serve as an objection to the Paretoprinciple. The example of Pareto optimality demonstrates that a widely acceptedethical rule, although it supports claims about a just society, does not apply to

The existing distribution of incomein the United States is just.

CLAIM

In 1975, the top 20 percent Americanhouseholds received 41 percent of all income, while the bottom 20 percent received 5.4 percent. In 1989, the top and bottom percentages were 46.7 percent and 3.8 percent, while in 2007 they were 49 percent and 3.4 percent. But from 1975to 2007, average incomes increased while losers were compensated through social service and welfare programs. However,the U.S has the greatest income inequalityof any industrialized country.

INFORMATIONEvidently, thedistributionis just.

QUALIFIER IBecause a just distributionof income is one whereat least one person benefitswithout others losing(Pareto Optimality). If thereare losers, they can becompensated by winners,creating a state of “VirtualPareto Optimality” (alsoknown as the Kaldor–Hickscorollary.)

WARRANT

Since income is receivedthrough ability, hard work,and one’s own labor.

But the losers are becoming worse off without adequate compensation. Between 1975 and 2007, the income of the bottom 20 percent declined from 5.4 percent to 3.4 percent, adecline of nearly 60 percent.This is not just.

BACKING OBJECTIONThis is not evident, considering thestrength of theobjections.

QUALIFIER II

But the winners receive income through fraud,discrimination, andfamily inheritance.

OBJECTION

However, families havea right to their own property.

REBUTTAL

Because a just distribution of income isone where those worse off are madebetter off. This is John Rawls’s principleof justice as fairness.

WARRANT

FIGURE 14Argumentation from ethics—income distribution and justice as fairness



situations involving fraud, discrimination, and inheritance. By engaging in thesystematic analysis of underlying ethical and moral reasoning, parties to a debateare also compelled to clarify the meaning of key concepts such as entitlement, whichare more complex than may be apparent at first glance. Parties making a claimalso may be compelled to consider whether particular ethical rules, such as Paretoand Rawlsian optimality, violate their own moral convictions. Proponents of thePareto rule may see that its application violates moral convictions about the ne-cessity of basing principles of entitlement on ability and work. If this is done forwelfare recipients, why not for heirs to estates? In short, the analysis of ethical ar-gumentation can help us probe ethical rules and moral principles to determinewhether they are general in their applicability and internally consistent.49 Ethicalargumentation, it should be emphasized, differs from each of the other modes ofreasoning in one essential respect: Whereas each of the other modes takes valuesas “given”—for example, values that are described in public opinion surveys area basis for arguments from motivation (Figure 10)—the process of ethical argu-mentation attempts to discover whether there are good reasons to make ethicalclaims.

EVALUATING POLICY ARGUMENTSThe evaluation of policy arguments facilitates critical thinking in public policyanalysis. So far, we have looked at the structure of arguments, the process of argu-mentation, and modes of reasoning employed in making policy claims. This enablesthe identification of hidden or tacit assumptions and shows how the plausibility of aclaim, as expressed in its qualifier, changes as a result of objections and rebuttalsintroduced in the course of policy argumentation.

We now turn to specific criteria for evaluating arguments. Some of these criteriacome from propositional logic, a discipline that offers criteria for determining—without qualification and with deductive certainty—the formal validity ofarguments.50 Other criteria originate in a continuously evolving body of standardsfor assessing the informal validity of arguments.51 Still other criteria come fromprocedures employed by methodologists who work in the hermeneutics and inter-pretationist (Verstehende) traditions. These traditions have long been concernedwith discovering and accurately representing the meaning of human actions,whether expressed in the form of written texts or as spoken language.52 Finally,some criteria originate in philosophical pragmatism, an epistemology and method-ology that is useful for evaluating entire systems of argumentation.53

49MacRae, Social Function of Social Science, pp. 92–94.50Classic sources on formal logic include Irving M. Copi, An Introduction to Logic (New York:Macmillan, 1953).51See Toulmin, Rieke, and Janik, An Introduction to Reasoning, Part V: Fallacies: How ArgumentsGo Wrong, pp. 129–97.52The classic treatise on hermeneutics, originally published in 1960, is Hans Georg Gadamer, Truthand Method (New York: Seabury, 1975).53See Rescher, Induction.



Some Hermeneutic GuidelinesHermeneutics investigates the meanings of human texts. It is perhaps the most com-prehensive and systematic of the qualitative methodologies. The term qualitative,contrary to a common misunderstanding in the social sciences,54 is not simply thelogical complement of “quantitative,” in the sense that qualitative refers to thatwhich is not quantitative or statistical. For this reason, it is inappropriate to employthe term qualitative to small-n case study research or the discrete (versus continu-ous) level of measurement. Rather, qualitative methods—also known as hermeneu-tic, interpretive, ethnographic, or constructivist methods—are expressly designed toinvestigate the meanings of individual and collective action.55 Statistical and otherforms of quantitative analysis are not appropriate for this purpose.

Human texts refer not only to written documents that are products of humanaction, for example, legislative transcripts and laws that originate as policy debates;they also refer to the actions themselves, whether or not they have been expressed inwritten form. Among various ways to demonstrate the importance of hermeneuticguidelines to the evaluation of policy arguments, a focus on ethical argumentation ismost revealing. Consider alternative ethical arguments about a prisoner captured bysoldiers from the opposing army:

� Argument A. The prisoner is not a member of a regular fighting force. He isa common criminal, a terrorist who lies when interrogated about terrorist plotsand other military secrets. Because he does not qualify formally as a prisonerof war, he should not be protected against “inhumane treatment” and torture.

� Argument B. The prisoner is a freedom fighter who is combating the terrorand oppression inflicted by the enemy. He is a loyal soldier who has a moralduty to mislead interrogators about military secrets. He should be protected asa prisoner of war.

� Argument C. The prisoner is a member of a fighting force constituted by astanding army. He is a prisoner of war whose loyalty demands that he misleadinterrogators about military secrets. As such, he should receive the protectionsagainst inhumane treatment afforded any prisoner of war.

The evaluation of these arguments can benefit from guidelines for the interpretationof written or oral argumentation (Procedural Guide 2).

The main function of arguments A and B is rhetorical, not dialectical or logical-empirical. Although the use of A and B as rebuttals to each other would contribute tothe dialectical function, this is not the main purpose of A and B. A and B do presentevidently factual statements, some of which appear to be empirically sound and uncon-troversial. A prisoner was captured, he was involved as a combatant, and he was nottelling the truth. What is disputed are two main issues: Is it is morally and legally rightto lie under the circumstances? Does the soldier qualify as a prisoner of war?

54See, for example, a widely used text by Gary King, Robert Keohane, and Sidney Verba, DesigningSocial Inquiry: The Logic of Inference in Qualitative Research (Princeton, NJ: Princeton UniversityPress, 1994).55For an extended critical essay on the meaning of qualitative see John Searle, The Construction ofSocial Reality (New York: The Free Press, 1995).



Other guidelines also apply. The quotation marks around some of the words(“prisoner of war” and “inhumane treatment”) conceal or obscure meanings. Thecontexts of A and B are also important for understanding the arguments. Given thehistory of many conflicts, it is not surprising that both sides regard opponents asterrorists. The principle of hermeneutic charity encourages giving the benefit of thedoubt to each party, for example, by trying to understand that both seek tobe treated equally. The one side contests the moral and legal acceptability of attackson their troops (and on civilians) by out-of-uniform combatants, arguing that suchpractices are unfair. The other side affirms the moral and legal acceptability of suchacts, on grounds that such acts are fair when one side has a preponderance of mod-ern weaponry. Finally, both arguments make liberal use of pejoratives—“extremist,”“criminal,” “terrorist,” “propagandists,” “freedom fighter,” “oppression”—thatobscure rather than clarify moral and legal issues.

If we were to rephrase the arguments, introducing rebuttals and replacing absolutewith qualified claims, it would look something like argument C. Argument C has the

BOX 2

Guidelines for Interpreting Arguments

� Policy argumentation has three majorfunctions: to generate debate that improves thevalidity, soundness, and efficacy of policies(dialectical function); to present optimallyvalid and empirically sound conclusions(logical-empirical function); and to persuadeothers to accept policy arguments (rhetoricalfunction), apart from the validity, soundness, orusefulness of the arguments.

� Look for concealed meanings in words,sentences, and entire arguments. A word orsentence may not mean what it says on thesurface. Example:“He is a good socialist” maynot mean that the person described performswell as a socialist; it rather may mean that theperson’s identity as a socialist is associatedwith some kind of weakness or flaw.

� Distinguish between the surface meaning of aword, sentence, or argument and its meaning inthe context of the arguer. Try to identify anydifferences in your understanding from that ofthe arguer. Example: “The mayor should nothave publicly acquiesced to the demonstrators’demands.” Several potential misinterpretations

are possible: Someone other than the mayorshould have acquiesced; the mayor should nothave acquiesced in public; the mayor shouldnot have acquiesced at all.

� Observe the principle of hermeneutic charity,which requires that disagreements aboutmeaning be resolved by accepting, or trying tounderstand, what the arguer is trying to say.Example: Critics of arguments presented inquantitative language often label sucharguments (and the arguers) as “logicalpositivists,” notwithstanding the fact thatquantification, per se, has little to do withlogical positivism. A charitable effort tounderstand what the arguer actually believescan solve this problem.

� Look for terms that are used pejoratively todiscredit a person or policy. On the surface,these terms can be neutral; but in context, theyare often pejorative. Examples: “This is justanother new bureaucracy.”“These are thearguments of typical ‘tree-huggers.’ ”“Thereport, written by a bunch of ‘qualitative’researchers, is unacceptable.”



advantage of isolating the contested issue, which is one of conflicting obligation:In the ethics of war, there has been a widespread understanding that prisoners whomislead interrogators about military secrets are displaying courage, honor, patriotism,and other virtues. Here, the same understanding would presumably apply to both sides.

Guidelines from Informal and Formal LogicHermeneutic guidelines are designed to enhance the understanding of meanings.Questions concerning the soundness, credibility, or plausibility of arguments do notarise, because the aim is to achieve an accurate interpretation of what arguers mean.By contrast, the fields of informal and formal logic provide guidelines for recognizingand assessing the significance of informal fallacies.56 Here as elsewhere the termguideline is used in place of “rule,” because there is no way to determine withcertainty whether an argument is fallacious. Hence the analysis of informal fallaciesdoes not permit all or none conclusions.

As we saw in the first part of this chapter, numerous modes of argumentationare generally recognized as appropriate to policy discourse. Arguments of thefollowing kind are formally valid:

� Hypothetical syllogism. If p implies q, and q implies r, then p implies r. Or:p)q, q ); r, ∴ p ) r () � implies). Example: A transitive preference orderingis one form of the hypothetical syllogism. Given three projects, A, B, and C, ifA p B, and B p C, then A p C (p � preferred to). This formally valid argumentcan be empirically unsound.

� Modus ponens. Modus ponens (method of affirming) asserts that if p impliesq, and p occurs, then q will occur. If p ) q, and p, then q. Example: If invest-ment I in a project produces outcome O, and investment I is made, thenoutcome O will result. Although this argument is formally valid, the conclusionassumes that no causally relevant factor other than I is present, a situation that almost never exists. This formally valid argument can beempirically unsound.

� Modus tollens. Modus tollens (method of denying) asserts that if p implies q,and q does not occur, then p is not a cause . If p ) q, and q does not occur (~ q),then p is not a cause (~ p). Example: If investment I in a program producesoutcome O, and O does not occur, then I is not a cause. This formally validargument can be empirically unsound.

We now turn to modes of argumentation that are generally recognized asformally invalid, inappropriate, or unsound—however persuasive they may appear onfirst glance. These modes of argumentation are called fallacies. A fallacy is an argument that is weakened or seriously flawed because it uses irrelevant orinadequate information, erroneous or unsound reasoning, or inappropriate andmisleading language. Table 2 provides a listing of fallacies and guidelines that arehelpful in recognizing them.

56The terms informal logic and informal fallacy are used in the discipline of logic, in which formal logicand formal fallacy are also distinguished.



TABLE 2

Guidelines for Identifying Invalid Arguments and Fallacies

Fallacy Guideline

Affirming theConsequent

A matter of formal (propositional) logic. A logically invalid argument is thefollowing: If p then q, and q, then p (p ) q, q, therefore p).“The paradigm of ‘proofthrough prediction,’ ” notes Merton, “is, of course, logically fallacious: If A(hypothesis), then B (prediction). B is observed. Therefore, A is true.”* If thestrict enforcement of speeding causes traffic accidents to decline, and trafficaccidents decline, then strict enforcement was the cause. In scientific research, thisform of argument, although formally invalid, can be useful because a hypothesismay be improved by testing conditions other than B, that is, rival hypotheses.

Denying theAntecedent

Again, a matter of formal logical validity. If p then q, and not-p, then not-q (p )

q, ~ p, therefore ~ q) is fallacious. Example: Because market economies aredemocracies, and country X is not a market economy, it is not a democracy.

False Analogy The comparison of two relationships believed to be similar disregardsimportant differences that make the comparison relatively unsound. Example:Because drug addiction is like an infectious disease, quarantining addicts isthe only policy that will work.

False Parallel The comparison of two cases believed to be similar disregards importantdifferences that make the comparison unsound. The acquiescence of theUnited States in World War II led to genocide and ethnic cleansing.The United States cannot acquiesce in ethnic cleansing in the Balkans.

HastyGeneralization

In making a generalization from particular instances of a case, a failure torecognize that there are too few instances of the case, or that the instances areexceptional rather than typical. In conducting opinion surveys, an inadequatesample size will yield too few instances, and a failure to use random sampling—with which every element or instance has an equal chance of being selected—islikely to yield exceptional conclusions rather than those typical of the population.Example: “Focus group” interviews with fifteen typical voters conducted beforethe election show that there is greater support for candidate A than candidate B.

False Cause In making a claim about cause and effect, arguing that a single cause isresponsible for an effect, but without examining other plausible causes. Falsecauses also stem from confusing statistical correlation or covariance withcausality and inferring cause from temporal sequence alone (post hoc fallacy).Examples: Excessive government spending is responsible for the slow growth ofGDP (single false cause).That economic conditions affect social well-being isevident from the statistically significant positive correlation (r � 0.74, p � 0.05)between suicide and unemployment (false cause based on correlation). Afterthe Reagan (or Clinton) administration took office, we had the highestunemployment (or government spending) in twenty years (post hoc fallacy).

Fallacy ofComposition

Concluding that something is true of the whole because it is true of its parts.The fallacy of composition (also called the aggregative or holistic fallacy)involves all parts, not just a sample, so it differs from the fallacy of hastygeneralization (see earlier entry). Example: Vehicle safety studies of the severity



Fallacy Guideline

of damage suffered by test robots riding at different speeds in automobiles showthat speed and severity of damage are strongly and positively correlated. This isstriking evidence that “speed kills!” But studies of fatal accidents show thatapproximately 20 percent of fatal accidents are related to speeding.

Fallacy of Division Concluding that something is true of the parts because it is true of the whole(also called the individualistic fallacy). Example: Because the per capitaincome of a country has increased, everyone is better off. In many countries,however, this is false. Persons who are better off become even better off,whereas those worse off become even worse off. Another example is using thearithmetic mean and other averages to describe a group, without examiningdifferences among the group’s members (e.g., outliers in a scatter plot).

Fallacy of theSlippery Slope

Concluding on the basis of insufficient or inadequate evidence that if oneevent occurs, then others will follow in an inevitable or uncontrollablesequence. Example: If the legislature passes a new law requiring stricterregistration of handguns, it will lead to government confiscation of all guns.

Begging theQuestion

A claim is assumed as a reason or evidence. Example: With a force of500,000 troops we will be able to invade Iraq and topple President SaddamHussein. It will take this many troops to do the job.

Ad Hominem An individual’s personal characteristics are used as part of an argument, whensuch characteristics are irrelevant to an issue. Examples: (1) An eminent naturalscientist concludes that welfare reform is unsound. (2) The argument of theenvironmentalists is deeply flawed. After all, these “tree huggers” are just socialistsin disguise. and (3) Theories of economic development are products of Westernthinking. Obviously, they are inapplicable to the non-Western world. Note that whenpersonal characteristics are relevant, no ad hominem fallacy is involved. Forexample, “expert witnesses” in court cases should have appropriate expertise.

Ad Populum The characteristics or beliefs of a group or community are used as part of anargument, when the characteristics or beliefs are irrelevant to the issue.Example: The majority of the community believes that fluoride causes cancer.

Appeal to Tradition A claim is based on conformity to tradition, when tradition is largely orentirely irrelevant to the issue. Examples: (1) We have always done it thisway. (2) The Founding Fathers would be appalled by the senator’s proposal.and (3) The successful disciplines have succeeded because they have emulatedphysics. This should be the model the social sciences, if they want to succeed.

Accent A misplaced emphasis on a word, phrase, or portion of an argument results inmisunderstanding or misinterpretation. The use of italics, boldface print,variable fonts, photos, clip art, and colors can accentuate the importance ofrelatively sound or plausible, as well as relatively unsound or implausible,arguments or parts of arguments. A leading example of the fallacy of accentis quoting or extracting information, reasons, or arguments out of context.

* Robert K. Merton Social Theory and Social Structure, rev. ed. (Glencoe, IL: Free Press, 1957), p. 99n.The same point is made byDonald T. Campbell, Epistemology and Methodology for Social Science: Selected Essays (Chicago: University of Chicago Press, 1988), p.168.



CHAPTER SUMMARYThis chapter has examined in detail the structureand process of policy argumentation, focusingon contrasts among types of claims, the identifi-cation and arrangement of elements of policyarguments, and the effects of objections andrebuttals on the dynamics of argumentation.

The chapter also contrasted different modes ofpolicy reasoning and offered guidelines for theidentification and assessment of commonfallacies that weaken or seriously flaw policyarguments. Policy argumentation is central topolicy analysis and the policy-making process.

REVIEW QUESTIONS1. Use three (3) of the following terms to con-

struct definitive, designative, evaluative, andadvocative claims. There should be twelve (12)claims in all.

crime fiscal crisispollution human rightsterrorism ethnic cleansingquality of life unemploymentglobal warming poverty

2. Develop a policy argument on the basis of oneor more of the terms in question 1.

3. Convert the argument in question 2 into a policydebate by providing an objection and a rebuttal.

4. Explain why the qualifier changed (if it did)after introducing the objection and rebuttal. Ifthe qualifier did not change, why not?

5. Define the term fallacy. Does the commissionof a formal fallacy such as affirming the conse-quent invalidate an argument? Is the same truefor informal fallacies, for example, false analogyor ad populum?

DEMONSTRATION EXERCISES1. Obtain an online or hard copy of the interna-

tional affairs section of a newspaper. Identifyand describe as many modes of argument asyou can. Would you expect to find differentmodes of argument in academic journals thanin newspapers? Explain.

2. Read the letters to the editor in a newspaper,magazine, or online bulletin board or blog. Findas many examples of formal and informal falla-cies as you can. A variation of this exercise is tobreak into groups to complete the assignment.

3. Read Case 1 (Pros and Cons of BalkanIntervention), which is drawn from an editorialin the Los Angeles Times. Use the argument-mapping procedures presented in this chapterto analyze the pros and cons (or strengths andweaknesses) of the recommendation that the

United States should not intervene in theBalkans. In doing this exercise, either displaythe elements of argument with Microsoft Drawor use Rationale, the special computer programfor mapping the structure of policy arguments.

4. Write a one-page analysis in which you assessthe overall plausibility of the claim “The con-flict in Bosnia is somebody else’s trouble. TheUnited States should not intervene militarily.”Prepare an argument map and hand it in withyour one-page analysis.

5. Following is an argument map in which thewarrants, backings, objections, rebuttals, andqualifiers have been scrambled.57 Rearrangethe elements to make a persuasive argumentand counterargument. Study Case 2 as an ex-ample.

57This was done with a sub-program in Rationale.



BIBLIOGRAPHYAlker, H. R., Jr. “The Dialectical Logic of Thucydides’

Melian Dialogue.” American Political ScienceReview 82, no. 3 (1988): 805–20.

Anderson, C. W. “Political Philosophy, PracticalReason, and Policy Analysis.” In ConfrontingValues in Policy Analysis. Edited by F. Fischerand J. Forester. Newbury Park, CA: SagePublications, 1987.

Apthorpe, R., and D. Gasper, eds. ArguingDevelopment Policy: Frames and Discourses.London: Frank Cass, 1996.

Dunn, W. N. “Policy Reforms as Arguments.” InThe Argumentative Turn in Policy Analysis andPlanning. Edited by F. Fischer and J. Forester.Durham, NC: Duke University Press, 1993.

Fischer, D. H. Historians’ Fallacies: Toward a Logicof Historical Thought. New York: RandomHouse, 1970.

Fischer, F. Evaluating Public Policy. Chicago:Nelson Hall, 1995.

Fischer, F., and J. Forester, eds. The ArgumentativeTurn in Policy Analysis and Planning. Durham,NC: Duke University Press, 1993.

Gasper, D., and R. V. George. “Analyzing Argumen-tation in Planning and Public Policy: Improvingand Transcending the Toulmin- Dunn Model.”Environment and Planning B: Planning andDesign 25 (1998): 367–90.

Majone, G. Evidence, Argument, and Persuasion inthe Policy Process. New Haven, CT: YaleUniversity Press, 1989.

McCloskey, D. N. The Rhetoric of Economics.Madison: University of Wisconsin Press, 1988.

Mitroff, I. I., R. O. Mason, and V. Barabba. The1980 Census: Policy Making Amid Turbulence.Lexington, MA: D. C. Heath, 1985.

The Senate shouldnot authorize the useof force in Libya.

Risky anduncertain

The U.S. hasan obligationto intervene.

Without adoubt.

The leader is a brutaldictator who has nointerest except hisown

The Minister ofForeign Affairs isreported to be asupporter of terrorism.

Economic sanctions are a betteralternative. They have worked inthe past in Asia.

An intervention is unwarranted. It isan artificial creation of the Presidentand no vital interests are at stake.

The actions of the Libyan Army area clear violation of the UniversalDeclaration of Human Rights.

There is no assurancethe dissident groupswill not oppose theU.S. later.

The museum holdsancient treasures andhistorical artifacts.

Diplomatic optionshave not beenexhausted.

This is a civil war andthe U.S. should notintervene in the internalaffairs of a state.

The war may not bewinnable unless theU.S. puts “boots onthe ground.”

Dissident groups are engagedin armed conflict with theLibyan Army, which is winning.



Roe, E. Narrative Policy Analysis. Durham, NC:Duke University Press, 1994.

Scriven, M. Reasoning. New York: McGraw-Hill, 1976.Stone, D. Policy Paradox: The Art of Political

Decision Making. Rev Ed. New York: W. W.Norton, 2002.

Toulmin, S., R. Rieke, and A. Janik. An Introductionto Reasoning. 2d ed. New York: Macmillan, 1984.

Van Gelder, Tim. “The Rationale for Rationale.” Law,Probability & Risk 6, nos. 1–4 (2007): 23–42.

CASE 1 PROS AND CONS OF BALKANINTERVENTION58

“Must the agony of Bosnia-Herzegovina beregarded, with whatever regrets, as somebody else’strouble? We don’t think so, but the arguments onbehalf of that view deserve an answer. Among themare the following:

� The Balkan conflict is a civil war and unlikely tospread beyond the borders of the formerYugoslavia.Wrong. Belgrade has missiles trainedon Vienna. Tito’s Yugoslavia claimed, by way ofMacedonia, that northern Greece as far south asThessaloniki belonged under its sovereignty.Those claims may return. “Civil” war pitting non-Slavic Albanians against Serbs could spreadto Albania,Turkey, Bulgaria, and Greece.

� The United States has no strategic interest in theBalkans. Wrong. No peace, no peace dividend.Unless the West can impose the view that ethnicpurity can no longer be the basis for nationalsovereignty, then endless national wars willreplace the Cold War. This threat has appearedin genocidal form in Bosnia. If it cannot becontained here, it will erupt elsewhere, and theClinton administration’s domestic agenda will bean early casualty.

� If the West intervenes on behalf of the Bosnians,the Russians will do so on behalf of the Serbs,

and the Cold War will be reborn. Wrong. TheRussians have more to fear from “ethniccleansing” than any people on Earth. Nothingwould reassure them better than a new, post-ColdWar Western policy of massive, early responseagainst the persecution of national minorities,including the Russian minorities found in everypost-Soviet republic. The Russian right may favorthe Serbs, but Russian self-interest lies elsewhere.

� The Serbs also have their grievances. Wrong.They do, but their way of responding to thesegrievances, according to the State Department’sannual human rights report, issued this pastweek,“dwarfs anything seen in Europe since Nazitimes.” Via the Genocide Convention, armedintervention is legal as well as justified.

� The UN peace plan is the only alternative.Wrong. Incredibly, the plan proposes thereorganization of Bosnia-Herzegovina followedby a cease-fire. A better first step would be aUN declaration that any nation or ethnic groupproceeding to statehood on the principle of ethnicpurity is an outlaw state and will be treated assuch. As now drafted, the UN peace plan, with amap of provinces that not one party to theconflict accepts, is really a plan for continued‘ethnic cleansing.’” �

58From the Los Angeles Times, January 24, 1993, p. A7.


CASE 2 IMAGES,ARGUMENTS,AND THE SECONDPERSIAN GULF CRISIS, 1990–91

The analysis of policy arguments can be employed toinvestigate the ways that policy makers represent orstructure problems.We can thereby identify theimages, or problem representations, that shapeprocesses of making and justifying decisions. Forexample, during times of crisis, the images that U.S.policy makers have of another country affectdeliberations about the use of peacekeeping andnegotiation, the imposition of economic sanctions, orthe use of deadly force. This case looks at thedeliberations surrounding the U.S. decision to usemilitary force to produce an Iraqi withdrawal fromKuwait during the Second Persian Gulf Crisis of1990–91.

It is important to recognize that there have beenthree Persian Gulf crises since 1980.The FirstPersian Gulf Crisis, which involved eight years ofwar between Iraq and Iran, lasted from September1980 to August1988. An estimated half-millioncivilians and military combatants died in the conflict.

The Third Gulf Crisis began in 2003 andappeared to be coming to a conclusion in 2011, asoccupying U.S. and coalition troops continued asteady withdrawal from Iraq. Between 2003 and2010, it is estimated that between 700,000 and1.5 million Iraqi civilians died as a directand indirect result of the war, and some11,800 members of the Iraqi security forces andpolice were killed.59 There were nearly 4,500 U.S.

military deaths and another 220 among coalitionforces, about one-half of whom were British.

The Second Persian Gulf Crisis may be convenientlydated to August 2, 1990, when Iraq invaded Kuwait.Seven months later, on March 3, 1991, a cease-fire wassigned.60 When Iraq invaded Kuwait, the United Statesbegan sending troops and Patriot missiles to defendSaudi Arabia and Israel. The purpose was to defendagainst Iraqi SCUD missile attacks on Riyadh, Tel Aviv,and Haifa. An economic embargo was also put in placealong with a U.S.-UN Security Council demand for animmediate withdrawal from Kuwait. Over the next fivemonths, Saddam Hussein failed to comply. At thatpoint, a Security Council resolution was passed givingIraq until January 15, 1991, to withdraw. Theresolution authorized intervention by a U.S.-ledcoalition after that date.

Prior to the deadline for the withdrawal fromKuwait, the U.S. Senate debated the merits of U.S.intervention. Two policy alternatives werepresented. One of these, the Dole-Warnerresolution, supported the Bush administration andthe United Nations by authorizing the use ofmilitary force to compel a withdrawal. The otherresolution, the Mitchell-Nunn resolution, called forthe continued use of sanctions. The vote in favor ofthe Dole-Warner resolution was 52–47.

Political scientist Richard Cottam and cognitivepolitical psychologist James Voss show how the images

59Hannah Fischer, “Iraq Casualties: U.S. Military Forces and Iraqi Civilians, Police, and SecurityForces.” U.S. Congressional Research Service Report (June 11, 2010).60Earlier, on February 15, the Shiite cleric Muqtada al-Sadr led a revolt against Saddam Hussein’sregime, partly in response to President George H. W. Bush’s earlier call on February 2 for Iraqis to over-throw Hussein’s government. In the same month, the Kurds in Northern Iraq also rose up in an attemptto overthrow Saddam’s regime.



of another state can be investigated by focusing onindicators of the perceived motivation, capability,culture, and decision processes of that state:61

� Ally Image. The image of the ally is known byfinding in foreign policy discourse indicatorsreferring to the mutually beneficial goals of theally, to the ally’s adequate but often less thanpossible military and economic capabilities, theally’s comparably civilized culture, and the ally’swell-managed and popularly supported process ofgovernmental decision making.

� Enemy Image. The image of the enemy stateis known by finding in foreign policy discourseindicators referring to the aggressive, evil, andexpansionistic motivations of the enemy; to theenemy’s comparable but penetrable military andeconomic capabilities; to the enemy’s comparablycivilized culture; and to the enemy’s monolithicand undemocratic process of governmentaldecision making.

� Radical Image. The image of the radicalstate is known by finding in foreign policydiscourse indicators referring to fanatic orextremist motivations, to the radical state’s useof terror to compensate for its inferior militaryand economic capabilities, to the radical state’sless than civilized culture, and to a decisionprocess that is well organized and clever.

� Degenerate Image. The image of thedegenerate state is known by finding in foreignpolicy discourse indicators referring to leadersmotivated by the accumulation and preservationof personal power, to the degenerate state’sinferior and declining military and economiccapabilities, to the degenerate state’s less thancivilized culture, and to a confused anddisorganized decision process.

� Imperial Image. The image of the imperialstate is known by finding in foreign policydiscourse indicators referring to motivationsbased on nationalism and modernization in theinterests of the people, to the imperial state’sneed for external help because of inferior militaryand economic capabilities, to the imperial state’sless than civilized culture, and to poorly manageddecision processes that require military andeconomic assistance.

In the Second Persian Gulf Crisis, Senate debateswere held on January 10, 11, and 12 of 1991.Thearguments put forth in the debates contain indicators ofthe images just described.62 For example, one supporterof the Dole-Warner resolution, whose position wasnearly identical to that of the White House, wasSenator Bryan (D-Nebraska). He argued in termsindicating a traditional enemy image of Iraq: “Husseinnow has under arms more men than Hitler when theGerman Army marched into the Rhineland . . . moretanks than when the Panzer Divisions crushedFrance . . . and most chilling of all, much closer tohaving a nuclear weapon than Adolf Hitler ever was.”

This is not the image of a radical or terroriststate, but that of a traditional aggressive andexpansionistic state. Senator Roth (D-Delaware)reinforced this image: “Hussein has demonstratedthat with the Cold War fading, the real threat tofreedom-loving nations is the proliferation of arms inthe hands of despotic dictators. Intercontinentalmissiles, chemical, biological, and nuclear arms canturn unstable Third World nations into first-ratemilitary powers.” By contrast, Senator Cohen(R-Maine), another supporter of Dole-Warner, usedan argument suggesting the image of a radical orterroristic state: “Not one of us . . . is safe from theviolence currently being inflicted in Kuwait . . . Our

61The images are based on Richard W. Cottam, Foreign Policy Motivation: A General Theory and aCase Study (Pittsburgh, PA: Pittsburgh University Press, 1977). The descriptions of images and use ofquotations are based on Voss et al., “Representations of the Gulf Crisis as Derived from the U.S. SenateDebate,” pp. 279–302 in Donald A. Sylvan and James F. Voss, eds. Problem Representation in ForeignPolicy Decision Making (Cambridge: Cambridge University Press, 1998).62All statements were extracted from The Congressional Record (January 9, 10,11) by Voss et al.,“Representations of the Gulf Crisis as Derived from the U.S. Senate Debate.”



The Senate should authorizethe use of force under the Dole-Warner resolution.

CLAIM (52)

The Dole-Warner resolution supportsU.N. Security Council Resolution 678of November 29, 1990 authorizingU.S. military intervention to force anIraqi withdrawal from Kuwait.

INFORMATION (98)

Since Iraq has a large military (37), regionalpolitical aspirations (32), and petroleumreserves that give economic control (16).

Because Iraq is a threatto the security of theregion and the world.

WARRANT (51)Because the U.S.has an obligation tointervene.

WARRANT (16)Immediately andwithout hesitation.

BACKING (16-37)Since the U.S. is a member of the U.N.,committed to Israel and moderate Arabstates, and opposed to human rightsviolations.

BACKING (16)

QUALIFIER (30)

But the authorization is unwarranted.It is an artificial creation of thePresident and no vital interests areat stake.

OBJECTION (37)

But sanctions are working. GNP iscut in half and imports and exportshave been reduced by 90−97%.

OBJECTION (47)But availablediplomatic optionshave not been pursued.

OBJECTION (33)

However, sanctions have not produceda withdrawal. They will cause economicproblems for allies and cause thesuffering of innocent persons.

REBUTTAL (34)However, diplomacyhas not worked in thepast and will notwork now.

REBUTTAL (9)

FIGURE C2Senate arguments supporting and opposing U.S. involvement under security council resolution 678 authorizing military intervention to force the withdrawal of Iraq from Kuwait (November 29, 1990)

security is only a Pan Am 103 away at any moment.”Senator Hatch (R-Utah) used similar language:“We have a major political interest in preventingHussein from radicalizing the Arab world.”

Among the supporters of the Mitchell-Nunnresolution were Senator Joseph Biden (D-Delaware),who argued: “Yes, we have interests in the MiddleEast. We wish to support the free flow of oil. Wewish to promote stability, including the securing ofIsrael. But we have not heard one cogent argumentthat any vital American interest is at stake in a waythat impels us to war.” As for military capability,Senator Patrick Moynihan (D-New York) arguedthat “The Iraqis do not (even) have the technologyto print their own paper money.” In support of thecontinued use of economic sanctions, Senator Frank

Lautenberg (D-New Jersey) argued:“Historicalanalysis of the use of economic sanctions suggeststhat they can be effective over time in forcing thewithdrawal of Iraqi troops from Kuwait.”

Figure C1 maps the arguments supporting theresolution sponsored by Bob Dole (R-Kansas) and JohnWarner (R-Virginia).The figure also maps theobjections of senators favoring the counter-resolutionsponsored by George Mitchell (D-Maine) and SamNunn (D-Georgia). Rebuttals to objections are alsodisplayed. The numbers in parentheses are the numbersof votes for the different parts of the argumentsupporting military intervention. The votes were alongparty lines, except for two Democrats who voted for theDole-Warner Resolution. Because two senators werenot present for the vote, the total number is 98 votes.


The Senate debate includes arguments thatappear to be the consequence of at least threeimages of Iraq. Arguments supporting the Dole-Warner Resolution authorizing a military attackappear to be a consequence of images of Iraq as a radical-terrorist state, or a degenerate state.In turn, arguments offered in support of the

continuation of economic sanctions appearto originate in an image of Iraq as a conventionalenemy state such as Germany during World War II.The main point is that images drive problemrepresentations, which in turn drive processesof argumentation surrounding critical issues offoreign and domestic policy. �



O B J E C T I V E S


Communicating PolicyAnalysis

� Describe stages in the processof policy communication.

� Contrast policy analysis, materialsdevelopment, interactivecommunication, and knowledgeutilization.

� Explain the relations amongprocesses of policy analysis, policycommunication, and policy making.

� Describe the main elements of policyissue papers and policy memos.

� Identify factors that explain the use, misuse, and abuse of policyanalysis.

� Translate policy argument into astructured written narrative.

� Plan, present, and evaluate an oralbriefing.

� Communicate a complex statisticalanalysis to multiple audiences.

Policy analysis is the beginning, not the end, of efforts to improve policies. Thisis why we define policy analysis not only as a process of creating and criticallyassessing policy-relevant information but also of communicating this informa-

tion with the expectation that it will be used. To be sure, the quality of analysis isimportant, but quality analysis is not necessarily used analysis. There is a large di-vide between the production of analysis and its use by policy makers (see Box 1).



Communicating Policy Analysis

work properly.The problem is that someonehas forgotten to build the mill to turn the logsinto lumber in all its usable forms.The logscontinue to pile up at one end of the systemwhile the construction companies continue tomake due at the other end. . . .There has beengovernmental and foundation support for thelogging operation.There has also been somesupport for the construction companies.There has been almost nothing, however, forthe planning and running of the mill. �

Source: Jack Rothman, Social R&D: Research and Development

in the Human Services (Englewood Cliffs, NJ: Prentice Hall,1980), p. 16.

BOX 1 PRODUCING POLICY ANALYSIS—A POORLY MANAGED LUMBER MILL

The social science researchers have gone intothe forest of knowledge, felled a good andsturdy tree, and displayed the fruits of theirgood work to one another. A few enterprising,application-minded lumberjacks have draggedsome logs to the river and shoved them offdownstream (“diffusion” they call it).Somewhere down the river the practitionersare manning the construction companies.They manage somehow to piece together a fewmake-shift buildings with what they can findthat has drifted down the stream, but on thewhole they are sorely lacking lumber in thevarious sizes and forms they need to do their

THE PROCESS OF POLICY COMMUNICATIONThe communication of policy-relevant knowledge may be viewed as a four-stageprocess involving policy analysis, materials development, interactive communication,and knowledge utilization. As Figure 1 shows, policy analysis is initiated on the basisof requests for information or advice from stakeholders situated at various points inthe policy-making process. In responding to requests, analysts create and criticallyassess information about policy problems, expected policy outcomes, preferred poli-cies, observed policy outcomes, and policy performance. To communicate such infor-mation, analysts must develop documents of different kinds: policy memoranda, pol-icy issue papers, executive summaries, appendices containing qualitative andquantitative information, news releases, and letters to the editor. In turn, the sub-stance of these documents is communicated through interactive presentations of dif-ferent kinds: conversations, conferences, meetings, briefings, and hearings. The pur-pose of developing policy-relevant documents and making presentations is to enhanceprospects for the utilization of knowledge.

The broken line in Figure 1 indicates that analysts influence the utilization ofknowledge only indirectly. By contrast, the solid lines indicate that policy analystscan directly affect the quality of conclusions and recommendations reached throughanalysis. Analysts can also directly affect the form, content, and appropriatenessof policy-relevant documents and oral presentations.

Tasks in Policy DocumentationThe knowledge and skills needed to conduct policy analysis are different from thoseneeded to develop policy documents. The development of documents that conveyusable knowledge requires skills in synthesizing, organizing, translating, simplifying,displaying, and summarizing information.



Synthesis. Analysts typically work with hundreds of pages of previously publishedreports, newspaper and journal articles, summaries of interviews with stakeholdersand other key informants, copies of existing and model legislation, tables of statisticalseries, and more. Information must be synthesized into documents ranging in lengthfrom one or two pages (policy memoranda) to more than several dozen pages inlength (policy issue papers and reports). Information also must be synthesized in theform of executive summaries and news releases.

Organization. Analysts must be able to organize information in a coherent, logicallyconsistent, and economical manner. Although documents vary in style, content, andlength, they typically have certain common elements:

� Overview or summary of contents of paper� Background of previous efforts to solve the problem� Diagnosis of the scope, severity, and causes of the problem� Identification and evaluation of alternative solutions� Recommendations of actions that may solve the problem� Policy issue papers, as contrasted with policy memos, usually include tables

and graphs, technical appendices that explain results of data analysis,summaries of existing and proposed legislation, explanations of formulasand equations, and other supporting material

MaterialsDevelopmentPolicy Analysis

InteractiveCommunication

KnowledgeUtilization

KNOWLEDGE

Policy ProblemsPolicy FuturesPolicy ActionsPolicy OutcomesPolicy Performance

POLICYANALYST

DOCUMENTS

Policy MemorandaPolicy Issue PapersExecutive SummariesAppendicesNews Releases

STAKEHOLDERS

Agenda SettingPolicy FormulationPolicy AdoptionPolicy ImplementationPolicy Assessment

PRESENTATIONS

ConversationsConferencesMeetingsBriefingsHearings

FIGURE 1The process of policy communication



Translation. The specialized terminology and techniques of policy analysis must betranslated into the languages of policy stakeholders. In many cases, this requires theconversion of abstract theoretical concepts and complex analytical and statisticalroutines into the ordinary language and arguments of nonexperts. When the audienceincludes experts on the problem (e.g., other analysts and staff specialists), a detailedexposition of theoretical concepts and analytical and statistical routines can beincorporated in appendices.

Simplification. Potential solutions to a problem are often complex. The combina-tions and permutations of policy alternatives and outcomes can easily exceed thehundreds. In such cases, a smaller set of options and outcomes can be displayedin the form of a matrix or “scorecard.”1 Another way to simplify complexity isto use a decision tree or a strategy table as part of a decision tree. The simplificationof complex quantitative relationships also can be accomplished by presenting in or-dinary language cases that typify quantitative profiles.2

Visual displays. The availability of advanced, user-friendly computer graphics hasdramatically increased capabilities of effective visual communication. The visualdisplay of quantitative information—bar charts, histograms, pie charts, line graphs,maps generated from geographic information systems (GIS)—is essential for effec-tive policy communication.3

Summaries. Policy makers with crowded agendas operate under severe time con-straints that limit their reading to no more than a few minutes a day. Membersof the U.S. Congress spend approximately fifteen minutes per day reading, and mostof this time is spent on local and national newspapers.4 The same is true for higher-level policy makers in agencies such as the U.S. Department of State. Under thesecircumstances, staff analysts perform an essential role in briefing policy makers,who are far more likely to read an executive summary or condensed memorandumthan a full policy issue paper.

The multiplicity of policy documents draws attention to the fact that there aremany ways to develop appropriate written materials on the basis of the same policy

1The scorecard or matrix displays developed by Bruce F. Goeller of the RAND Corporation areuseful as means for simplifying a large set of interdependent alternatives. For an overview of the Goellerscorecard or matrix, see Bruce F. Goeller, “A Framework for Evaluating Success in Systems Analysis”(Santa Monica, CA: RAND Corporation, 1988).2A good example is Ronald D. Brunner, “Case-Wise Policy Information Systems: Redefining Poverty,” Policy Sciences 19 (1986): 201–23.3The outstanding source on the methodology of graphic displays is Edward R. Tufte’s trilogy,The Visual Display of Quantitative Information (Cheshire, CT: Graphics Press, 1983);Envisioning Information (Cheshire, CT: Graphics Press, 1990); Visual Explanations (Cheshire, CT:Graphics Press, 1997).4Personal communications with staff of the U.S. General Accountability Office.



analysis. There are also multiple audiences for policy-relevant information.Immediate “clients” are often only one audience, and effective communication maydemand that analysts develop different documents for different audiences, thus thinkingstrategically about opportunities for policy improvement: “Thinking strategicallyabout the composition of the audience is essential for effective communication. Theselection of the audience is not something to be left solely to the immediate client. . . .There are many clients to be reached; some may be peripheral, some remote, some inthe future, yet all are part of a potential audience.”5 For example, when the prepara-tion of news releases is permitted under standard operating procedures, the newsrelease is an appropriate vehicle for reaching the general public through the massmedia.6 A policy issue paper and policy memo are not. If the aim is to communicatewith the immediate client, however, it is the executive summary or policy memoran-dum that is likely to be most effective.

Tasks in Oral Presentations and BriefingsJust as procedures for conducting analysis are different from procedures fordeveloping policy documents, so are procedures for developing these documentsdifferent from procedures for making oral presentations and briefings. A commonmedium of communication is the mailed document, an impersonal means ofreaching clients and other policy stakeholders. The major limitation of thismedium is the probability that a document will reach intended beneficiaries—butthen sit on the shelf. The probability of utilization is enhanced when ideas arecommunicated through policy presentations. Policy presentations, which includeconversations, conferences, briefings, meetings, and hearings, constitute an inter-active mode of communication that is positively associated with the utilizationof policy-relevant knowledge.7

Although there is no codified body of rules for making oral presentations,experience has shown that a number of general guidelines are important for effectivepolicy communication. These guidelines offer multiple communications strategies

5Arnold Meltsner, “Don’t Slight Communication,” Policy Analysis 2, 2 (1978): 221–31.6On the role of the mass media in communicating social science knowledge, see Carol H. Weiss andEleanor Singer, with the assistance of Phyllis Endreny, Reporting of Social Science in the National Media(New York: Russell Sage Foundation, 1987). A much-neglected area, the role of academic and commercialpublishing in communicating ideas emanating from the sciences and humanities, is addressed in IrvingLouis Horowitz, Communicating Ideas: The Crisis of Publishing in a Post-Industrial Society (New York:Oxford University Press, 1986).7The efficacy of the “interactive” model of communicating and using information has beenreported in literature for at least thirty years. See Ronald G. Havelock, Planning for Innovation:Through Dissemination and Utilization of Knowledge (Ann Arbor: Institute for Social Research,Center for the Utilization of Scientific Knowledge University of Michigan, 1969); Carol H. Weiss,“Introduction,” in Using Social Research in Public Policy Making (Lexington, MA: D. C. Heath,1977), pp. 1–22; Charles E. Lindblom and David Cohen, Usable Knowledge: Social Science andSocial Problem Solving (New Haven, CT: Yale University Press, 1979); and Michael Huberman,“Steps toward an Integrated Model of Research Utilization,” Knowledge: Creation, Diffusion,Utilization 8, no. 4 (June 1987): 586–611.



appropriate for the various contingencies found in complex practice settings. Thefollowing are among these contingencies:

� The size of the group that comprises the audience� The number of specialists in the problem area addressed� The familiarity of group members with methods of analysis� The credibility of the analyst to the group� The extent to which the presentation is critical to policies under active

consideration

In such contexts, multiple communications strategies are essential: There is no“universal client” who uses the same standards to evaluate the plausibility, relevance,and usability of policy analysis. Effective policy presentations are contingent onmatching communications strategies to characteristics of the audience for policyanalysis (Box 2).

THE POLICY ISSUE PAPERA policy issue paper should provide answers to a number of questions:

� What actual or potential courses of action are objects of disagreement amongstakeholders?

� In what different ways can the problem be defined?� What is the scope and severity of the problem?� How is the problem likely to change in the future?� What goals and objectives should be pursued to solve the problem?� How can the degree of success in achieving objectives be measured?

• Pinpoint reasons for your lack ofcredibility, choosing a strategy to overcomethe problem—for example, arrange to beintroduced by a credible associate or presentas part of a team.

• Be sensitive to time constraints and thepossibility that the group is already committedto a course of action.

• Position your supporters next to people withanticipated negative reactions.

• Prioritize your points so that you present thosethat are most critical to the group’s preferreddecision. �

*Adapted from Version 2.0 of Presentation Planner, a softwarepackage developed by Eastman Technology, Inc.

BOX 2 CONTINGENT COMMUNICATION

Policy analyses are frequently presented to groupsthat have few experts in the problem area,minimum familiarity with analytic methods,limited confidence in analysts, and little patiencefor meetings that take up precious time.Whatcommunications strategies are likely to beeffective under these conditions?*

• Make sure that the presentation addresses theneeds of key decision makers and recognizesaudience diversity.

• Avoid giving too much background information.• Focus on conclusions.• Use simple graphics to display data.• Discuss methods only if necessary to support

conclusions.



8The terms basic and applied are used for convenience only. It is widely acknowledged that thesetwo orientations toward science overlap in practice. Many of the most important advances in the basicsocial sciences originated in applied work on practical problems and conflicts. See, especially, KarlW. Deutsch, Andrei S. Markovits, and John Platt, Advances in the Social Sciences, 1900–1980: What,Who, Where, How? (Lanham, MD: University Press of America and Abt Books, 1986).

TABLE 1

Two Kinds of Policy DisciplinesCharacteristic Basic Applied

Origin of Problems • University Colleagues • Governmental Clients and CitizensTypical Methods • Quantitative Modeling • Development of Sound ArgumentsType of Research • Original Data Collection • Synthesis and Evaluation

of Existing DataPrimary Aim • Improve Theory • Improve PracticeCommunications Media • Article or Book • Policy Memo or Issue PaperSource of Incentives • University Departments • Government Departments

and Citizen Groups

� What activities are now under way to resolve the problem?� What new or adapted policy alternatives should be considered as ways

to resolve the problem?� Which alternatives are preferable, given the goals and objectives?

In answering these questions, the analyst needs knowledge and skills in writingpolicy issue papers. Only recently have such knowledge and skills become integral partsof instructional programs in university departments and professional schools of publicpolicy, public management, and public administration. Policy analysis, although itdraws from and builds on social science disciplines, differs from those disciplinesbecause it seeks to improve as well as understand policy-making processes. In thisrespect, policy analysis has characteristics that make it an applied rather than a basicpolicy discipline.8 Analysts need to acquire knowledge and skills provided in basicpolicy disciplines. But the purpose of policy analysis remains applied rather than basic.Characteristics of basic and applied policy disciplines are displayed in Table 1.

Issues Addressed in the Policy Issue PaperThe policy issue paper may address issues in almost any area: health, education,welfare, crime, labor, energy, foreign aid, national security, human rights, and soon. Papers in any one of these issue areas may focus on problems at one or morelevels of government. Air pollution, global warming, and terrorism, for example,are international, national, and local in scope. Issue papers may be presented as“staff reports,” “briefing papers,” “options papers,” or so-called “white papers.”An illustrative list of issues that may be the focus of a policy issue paper follows:

� Which of several contracts should a union bargaining team accept?� Should the mayor increase expenditures on road maintenance?



9E. S. Quade, Analysis for Public Decisions (New York: American Elsevier Publishing, 1975), p. 69.10Compare Eugene Bardach, The Eight-Step Path to Policy Analysis: A Handbook for Practice(Berkeley: University of California Press, 1996); Quade, Analysis for Public Decisions, pp. 68–82; andHarry Hatry and others, Program Analysis for State and Local Governments (Washington, DC: UrbanInstitute, 1976), appendix B, pp. 139–43.

� Should the city manager install a computerized management information system?� Which public transportation plan should the mayor submit for federal funding?� Should a state agency establish a special office to recruit minorities and women

for civil service positions?� Should a citizens’ group support environmental protection legislation now

before Congress?� Should the governor veto a tax bill passed by the state legislature?� Should an agency director support a plan for flexible working hours (flextime)?� Should a legislator support a bill restricting the sale of handguns?� Should the president withhold foreign aid from countries that violate human

rights?� Should the United Nations General Assembly condemn the violation of human

rights in a particular country?� Should the United States withdraw from the International Labor

Organization?� Should taxes on foreign investments of multinational corporations registered in

the United States be increased?

Elements of the Policy Issue PaperA policy issue paper should

explore the problem at a depth sufficient to give the reader a good idea of itsdimensions and the possible scope of the solution, so that it might be possiblefor a decision maker to conclude either to do nothing further or to commissiona definitive study looking toward some action recommendation.9

In this author’s experience, most issue papers deal primarily with the formulationof a problem and possible solutions. Rarely does the issue paper reach definitiveconclusions or recommendations. Although an issue paper may contain recommenda-tions and outline plans for monitoring and evaluating policy outcomes, it is essentiallythe first phase of an in-depth policy analysis that may be undertaken at a later time.

In preparing an issue paper, the analyst should be reasonably sure that all majorquestions have been addressed. Although issue papers vary with the nature of theproblem being investigated, most contain a number of standard elements.10 Theseelements have been organized around the framework for policy analysis presented inthis text (see Table 1).

Observe that each element of the issue paper requires different policy-analyticmethods to produce and transform information. A policy issue paper, however, isessentially a prospective (ex ante) investigation. It is based on limited informationabout past actions, outcomes, and performance, and in this respect it differs fromprogram evaluations and other retrospective (ex post) studies. Appendix 1 providesa checklist for preparing a policy issue paper.



The Policy MemorandumIt is widely but mistakenly believed that policy analysts spend most of their timedeveloping policy issue papers, studies, and reports. In fact, the primary activity ofmost analysts is the preparation of policy memoranda. Whereas the issue paper is along-term activity involving the conduct of policy research and analysis over manymonths, the policy memorandum is prepared over a short period of time—usuallyno more than one month, but often a matter of days. The differences between theissue paper, study, or report and the policy memorandum reflect some of the differ-ences between basic and applied policy analysis summarized in Table 1.

The policy memorandum should be concise, focused, and well organized. Itreports background information and presents conclusions or recommendations,usually in response to a request from a client. The policy memo often summarizesand evaluates one or more issue papers or reports, along with accompanying backupdocuments and statistical data.

The format for a memo, which varies from agency to agency, is designed for quickand efficient reading. Most memos display the name of the recipient, the name of theanalyst submitting the memo, the date, and the subject on separate lines at the top ofthe page. Most agencies have preprinted forms for memos. Word-processing programshave templates for memos and other standard documents. Writers of policy memoshould take every opportunity to communicate clearly and effectively the main pointsof the analysis:

� The subject line should state in concise terms the main conclusion,recommendation, or purpose of the memo.

� The body of the memo (usually no more than two pages) should containheadings that describe major subdivisions.

� Bulleted lists should be used to highlight a small set (no more than five) of important items such as goals, objectives, or options.

� The introductory paragraph should review the request for informationand analysis, restate the major question(s) asked by the client, and describethe objectives of the memo.

TABLE 2

Elements of an Issue Paper and Methods for Creating Information Relevant to Each Element

Element Method

Letter of Transmittal

Executive Summary

I. BACKGROUND OF THE PROBLEMA. Describe client’s inquiry MONITORINGB. Overview problem situationC. Describe prior efforts to solve problem

(continued)


The Executive SummaryThe executive summary is a synopsis of the major elements of a policy issue paper orreport. The executive summary typically has the following elements:

� Purpose of the issue paper or study being summarized� Background of the problem or question addressed� Major findings or conclusions� Approach to analysis and methodology (when appropriate)� Conclusions and recommendations (when requested)

The Letter of TransmittalA letter of transmittal will accompany a policy issue paper or study. Although theletter of transmittal has many elements of a policy memo, the purpose of the letter isto introduce the recipient to the larger paper or study. Common elements of a letterof transmittal are as follows:

� The letterhead or address of the analyst transmitting the issue paper or study� The name, formal title, and address of the client(s) who requested the issue

paper or study


TABLE 2

Element Method

II. SIGNIFICANCE OF THE PROBLEMA. Evaluate past policy performance EVALUATIONB. Assess the scope and severity of problemC. Determine the need for analysis

III. PROBLEM STATEMENTA. Diagnose problem PROBLEMB. Describe major stakeholders STRUCTURINGC. Define goals and objectives

IV. ANALYSIS OF ALTERNATIVESA. Describe alternatives FORECASTINGB. Forecast consequences of alternativesC. Describe any spillovers and externalitiesD. Assess constraints and political feasibility

V. CONCLUSIONS AND RECOMMENDATIONSA. Select criteria or decision rules PRESCRIPTIONB. State conclusions and recommendationsC. Describe preferred alternative(s)D. Outline implementation strategyE. Summarize plan for monitoring and evaluationF. List limitations and unanticipated consequences

ReferencesAppendices



� A short paragraph stating the question or problem that the client(s) expects the analyst to address

� A brief summary of the most important conclusions or recommendations� A review of arrangements for further communication or work with the client,

for example, a scheduled briefing on the issue paper or study� A concluding statement indicating where and how the analyst can be reached

to answer any questions� A signature with the name and title of the analyst or the analyst’s supervisor

THE ORAL BRIEFING AND VISUAL DISPLAYSWritten documents are only one means for communicating the results of policy analysis.The other is the oral briefing. Although the oral briefing or presentation has many of thesame elements as the issue paper, the approach to planning and delivering an oral presen-tation is quite different. The elements of an oral briefing typically include the following:

� Opening and greeting to participants� Background of the briefing� Major findings of the issue paper or report� Approach and methods� Data used as basis for the analysis� Conclusions or recommendations� Questions from participants� Closing

Knowing the audience is one of the most important aspects of planning oralbriefings. In this context, a series of important questions should be answered prior tothe briefing:

� How large is the audience?� How many members of the audience are experts on the problem you are

addressing?� What percentage of the audience understands your research methods?� Are you credible to members of the audience?� Does the audience prefer detailed or general information?� Where does the briefing fall in the policy-making process? At the beginning or

agenda setting phase? In the middle or implementation and adaptation phases? At the end or policy termination phase?

Answers to these questions govern the strategy and tactics of communicationappropriate for the particular audience. For example, if the audience is a medium-size group with twenty to twenty-five persons, the potential diversity of the audiencecalls for specific communications strategies and tactics:

� Circulate background materials before the meeting so that everyone will startfrom a common base.

� Tell the group that the strategy selected is designed to meet the objectivesof the group as a whole but that it may not satisfy everyone’s needs.

� Focus on the agendas of key policy makers, even if you lose the rest ofthe audience.



Another common problem in giving oral briefings is that policy makers may bringtheir own staff experts to the briefing. If a high percentage of the audience has expert-ise in the area you are addressing, the following strategies and tactics are appropriate:

� Avoid engaging in a “knowledge contest” with subject experts. Focus insteadon the purpose of the presentation.

� Capitalize on the group’s expertise by focusing on findings and recommendations.� Avoid a lengthy presentation of background material and description of

methods.� Attempt to generate dialogue and a productive debate on policy options.

An equally important problem in oral briefings is knowing individual membersof the group. For example, it is important to know if the immediate clients for policy analysis are

� Experts in the problem area� Familiar with your analytic methods� Influential participants in the policy process� Oriented toward detailed versus broad information� Have a high, medium, or low stake in the outcome� Politically important to the success of your recommendations

Finally, the effectiveness of oral briefings can be significantly enhanced byvarious graphic displays. The most common of these are transparencies used withoverhead projectors, and slides created by computer graphics and presentationprograms such as Microsoft PowerPoint®. In using overhead transparencies andslides, it is important to recognize that they are not appropriate when you want aninformal style of presentation—overheads tend to make briefings very and makeopen dialogue difficult. It is also important to observe the following guidelines:

� Keep transparencies and slides simple, with condensed and highlighted text.� Use clear, bold, uncluttered letters and lines of text.� Avoid complicated color schemes, pop art, and fancy templates.� Highlight important points and use different colors. A dark background with

superimposed white or yellow letters is effective.� Limit the text on any page to a maximum of ten lines.� Be sure the width of the screen is at least one-sixth of the distance between

the screen and the farthest viewer.� Make sure that the nearest viewer is a minimum of two screen widths from

the screen.� Remain on the same transparency or slide for a minimum of two minutes.

This gives viewers time to read, plus an additional thirty seconds to study thecontent.

� Leave the room lights on—transparencies and slides are designed for lightedrooms. It is not a movie theater.

� Turn off the overhead projector, or blacken the computer screen, when youwant to redirect attention back to you and reestablish eye contact.



These are but a few of the strategies and tactics that are appropriate for differentaudiences and communications media. Given the goal of selecting communicationsmedia and products that match the characteristics of the audience, many types ofvisual displays are available to the analyst who wishes to effectively communicateideas. These same graphics can be created with Microsoft Excel and PowerPoint.

� Options-impact matrices (“Goeller scorecards”)� Spreadsheets� Bar charts� Pie charts� Frequency histograms� Ogives (cumulative frequency curves)� Scatter plots� Interrupted time-series and control-series graphs� Influence diagrams� Decision trees� Strategy tables

POLICY ANALYSIS IN THE POLICY-MAKING PROCESSWe define policy analysis as a process of multidisciplinary inquiry aiming at thecreation, critical assessment, and communication of policy-relevant information.As a problem-solving discipline, it draws on social science methods, theories, andsubstantive findings to solve practical problems. We emphasized that policy analy-sis is a series of intellectual activities embedded in a political process known aspolicy making. The distinction between these two dimensions—the intellectualand the political—is important for understanding the use, nonuse, and abuse ofanalysis by policy makers.

Consider the three interrelated dimensions of information use:

� Composition of users. Analysis is used by individuals as well as collectives(e.g., agencies, bureaus, legislatures). When the use of analysis involves individualgains (or losses) in the perceived future value of information, the process of useis an aspect of individual decisions (individual use). By contrast, when theprocess of use involves public enlightenment or collective learning, the use ofinformation is an aspect of collective decisions (collective use).

� Effects of use. The use of policy analysis has different kinds of effects. Policyanalysis is used to think about problems and solutions (conceptual use). It isalso used to legitimize preferred formulations of problems and solutionsby invoking the authority of experts and methods (symbolic use). By contrast,behavioral effects involve the use of policy analysis as a means or instrumentfor carrying out observable policy-making activities (instrumental use).



Conceptual, symbolic, and instrumental uses occur at individual andcollective levels.

� Scope of information used. The scope of use ranges from the specific to thegeneral. The use of “ideas in good currency” is general in scope (general use),whereas the use of a policy recommendation is specific (specific use).Information that varies in scope is used by individuals and collectives, witheffects that are conceptual, symbolic, and behavioral.

With these distinctions in mind, the use of information in policy making isshaped by at least five types of factors.11 These factors include differences in thecharacteristics of information, the modes of inquiry used to produce informa-tion, the structure of policy problems, political and bureaucratic structures,and the nature of interactions among policy analysts, policy makers, and otherstakeholders.

Characteristics of InformationThe characteristics of information affect its use by policy makers. Information thatconforms to the product specifications of policy makers is more likely to be usedthan information that does not, because product specifications reflect policy makerneeds, values, and perceived opportunities. Policy makers tend to value informationthat is communicated in personal verbal reports, rather than formal written docu-ments, and expressed in a language that reflects concrete policy contexts rather thanthe more abstract vocabularies of the natural and social sciences.12 Policy makersalso attach greater value to information that is accurate and generalizable to similarsettings.13

Modes of InquiryThe use of information by policy makers is shaped by the process of inquiry usedby analysts to produce that information. Information that conforms to standardsof quality research and analysis is more likely to be used. Yet there are widelydiffering views about the meaning of “quality.” For many analysts, quality isdefined in terms of the use of causal modeling, field experimentation, randomsampling, and quantitative measurement.14 The assumption is that information

11For case study research on these factors, see William N. Dunn. “The Two-Communities Metaphorand Models of Knowledge Use: An Exploratory Case Survey,” Knowledge: Creation, Diffusion,Utilization, no. 4 (June 1980).12See Mark van de Vall, Cheryl Bolas, and Thomas Kang, “Applied Social Research in IndustrialOrganizations: An Evaluation of Functions, Theory, and Methods,” Journal of Applied BehavioralScience 12, no. 2 (1976): 158–77.13See Nathan Caplan, Andrea Morrison, and Roger J. Stambaugh, The Use of Social Science Knowledgein Policy Decisions at the National Level (Ann Arbor, MI: Institute for Social Research, 1975).14See I. N. Bernstein and H. E. Freeman, Academic and Entrepreneurial Research (New York: RussellSage Foundation, 1975).


use is a function of the degree to which policy research and analysis conformsto accepted scientific methods.

By contrast, other analysts define quality in different terms. Quality may bedefined in terms that emphasize nonquantitative procedures designed to uncoversubjective understandings about problems and potential solutions held by policymakers and other stakeholders.15

Structure of ProblemsThe utilization of information by policy makers is also influenced by the goodness-of-fit between modes of inquiry and types of problems. Relatively well-structuredproblems involving consensus on goals, objectives, alternatives, and their conse-quences require different methodologies than relatively ill-structured problems.Because the essential characteristic of ill-structured problems is conflict, notconsensus, they require methodologies that are holistic and that bring to bearmultiple perspectives of the same problem situation in defining the nature of theproblem itself.16

The distinction between well-structured and ill-structured problems is associ-ated with the distinction between lower-level (micro) and higher-level (meta)problems. Meta-level problems involve issues of how to structure a problem, how toassist policy makers in knowing what should be known, and how to decide whichaspects of a problem should be resolved on the basis of “conceptual” knowledge.17

Most policy research and analysis is oriented toward the production of “instrumental”knowledge, that is, knowledge about the most appropriate means for attainingtaken for granted ends. Instrumental knowledge, since it is appropriate for well-structured problems at the microlevel, is less likely to be used by policy makers whoface ill-structured problems involving disagreements about the nature of problemsand their potential solutions.

Political and Bureaucratic StructuresThe use of information is also shaped by differences in the formal structures, proce-dures, and incentive systems of organizations. The influence of policy-making elites,the bureaucratization of roles, the formalization of procedures, and the operation


15Martin Rein and Sheldon H. White, “Policy Research: Belief and Doubt,” Policy Analysis 3, 2 (1977);Michael Q. Patton, Alternative Evaluation Research Paradigm (Grand Forks: University of NorthDakota, 1975); and H. Aeland, “Are Randomized Experiments the Cadillacs of Design?” PolicyAnalysis 5, no. 2 (1979): 223–42.16Russell Ackoff, Redesigning the Future: A Systems Approach to Societal Problems (New York: Wiley,1974); Ian I. Mitroff, The Subjective Side of Science (New York: American Elsevier Publishing, 1974);and Ian I. Mitroff and L. Vaughan Blankenship, “On the Methodology of the Holistic Experiment: AnApproach to the Conceptualization of Large-Scale Social Experiments,” Technological Forecasting andSocial Change 4 (1973): 339–53.17See Nathan Caplan, “The Two-Communities Theory and Knowledge Utilization,” AmericanBehavioral Scientist 22, no. 3 (1979): 459–70.



CHAPTER SUMMARYskills in developing policy documents andgiving oral briefings. To be effective, analystsneed to master and apply a broad range ofcommunications skills, thereby narrowingthe present divide between intellectual andpolitical dimensions of policy making.

This chapter has provided an overview of theprocess of policy communication and its im-portance to the use of analysis by policymakers. Policy analysis is the beginning, notthe end, of efforts to improve policy making.Knowledge and skills in conducting policyanalysis are different from knowledge and

18Robert F. Rich, The Power of Social Science Information and Public Policymaking: The Caseof the Continuous National Survey (San Francisco, CA: Jossey-Bass, 1981).19See Weiss, “Introduction,” Using Social Research in Public Policy Making, pp. 13–15.

of incentive systems that reward conservatism and punish innovation contribute to the underuse and nonuse of information. These and other factors, although theyare external to the methodology of policy analysis, create a political and bureaucraticcontext for information use.18

Interactions Among StakeholdersThe nature and types of interaction among stakeholders in the various phases ofpolicy making also influence the use of information by policy makers.19 Policyanalysis is not simply a scientific and technical process; it is also a social processwhereby the structure, scope, and intensity of interaction among stakeholdersgovern the creation and use of information.

The interactive nature of policy analysis makes questions of information use com-plex. Analysts rarely produce information that is or can be used merely for problemsolving in the narrow sense of discovering the most appropriate means to well-definedends. Many of the most important policy problems are sufficiently ill structured thatthe narrow problem-solving model of policy analysis is often inappropriate or inappli-cable. Instead, we have described policy analysis as a process of inquiry wherebymultiple methods—problem structuring, forecasting, prescription, monitoring, andevaluation—are used continuously to create, critically assess, and communicate infor-mation that is useful in improving public policy making.



REVIEW QUESTIONS1. Why are communication and use of informa-

tion central to the aims of policy analysis?2. If policy analysis is an intellectual activity carried

out within a political process, what are the impli-cations for the use of analysis by policy makers?

3. “Policy analysis is the beginning, not the end, ofefforts to improve the policy-making process.”Comment.

4. Before intended beneficiaries can use policy-relevant information, it must be converted intopolicy-relevant documents and communicatedin presentations of different kinds. Does this

guarantee that intended beneficiaries will useinformation? Explain.

5. Describe stages of the process of policycommunication.

6. Why are skills needed to develop policydocuments and give oral presentations differ-ent from skills needed to conduct policyanalysis?

7. Discuss factors that affect the use of policy analy-sis by policy makers and other stakeholders.

8. Why is the production of policy analysisdescribed as a poorly managed lumber mill?

DEMONSTRATION EXERCISES1. After reading Case 1, create argument maps for

the four types of documents described in thecase:

� Position paper� Issue paper or memo� Policy brief� Letter to the editor

After creating the maps, develop written docu-ments that include the reasoning used in eachof the four maps.

2. The purpose of this exercise is to sharpenknowledge and skills in planning and conduct-ing effective oral briefings. This is a groupexercise, although different individuals couldalso complete it. Divide the class into fourgroups. Each group should perform the follow-ing tasks:a. Respond to one of the following requests (I,

II, III, or IV) to give an oral briefing on thereduction of harmful levels of lead in gaso-line. Assume that you are the authors of theanalysis in Case 2. It is an analysis preparedby David Weimer and Aidan Vining, twoexperienced policy analysts. Your group hasbeen asked to make one of the four briefingsbelow.

b. Each group will have fifteen minutes togive its briefing and respond to ten minutesof questioning. The main policy questionyou must answer is the following: Shouldthe government regulate lead additives ingasoline?

c. Other class members, who will be using theattached rating scale, will evaluate presenta-tions. The evaluations will be used as a basisfor a critique and discussion of the briefings.

� Group I. Prepare an oral briefing for acommunity environmental group composedof about sixty citizens, most of whom havelittle or no knowledge of statistical methodsand biomedical research on lead. The groupdoes not trust you, in part because yourteam is composed of “outsiders” fromWashington. Among members of theenvironmental group is a disruptive personwho likes to get attention by engagingin a “knowledge contest.” This person isrunning for public office in the nextelection. The environmental group wantsto see new legislation on the public agenda.Your answer to the policy question statedearlier is supposed to help them.

� Group II. Prepare an oral briefing for agroup of ten experts in biostatistics, econo-metrics, and environmental economics. Allhave extensive knowledge of the healtheffects of leaded gasoline. Most of them are colleagues with whom you have workedpreviously. One member of the workinggroup, a respected biostatistician, is a paidconsultant working for the InternationalLead and Zinc Union (ILZU), an organiza-tion that markets lead and zinc products.



The ILZU takes the position that“statistical correlations do not provecausation.” The working group is designinga research proposal to investigate the healtheffects of lead. Your answer to the policyquestion is supposed to help its members.

� Group III. Prepare an oral briefing for agroup of thirty business executives from theoil refining industry. They are opposed togovernment regulation of leaded gasoline.They believe that regulation will reduceprofits and destroy the industry. Althoughthey have been briefed by their staff experts,they are not familiar with statistics andbiomedical research on lead. The executivesare preparing to fight a congressional bill,which, if adopted next month, wouldregulate and eventually ban leaded gasoline.Your answer to the policy question issupposed to help them.

� Group IV. Prepare an oral briefing for agroup of six experts from the OccupationalSafety and Health Administration. They

have done extensive research on thebiomedical effects of lead exposure andare convinced that low levels of exposureto lead are extremely harmful, especially toinner-city children who are exposedto unusually high amounts of automotiveexhaust. The experts are evaluating youranalysis in preparation for an annualconference. Your answer to the policyquestion is supposed to help them.

In preparing your briefing, use argument maps toorganize your presentation, state your argu-ment, and anticipate challenges.

Remember that oral briefing should be based on acommunications strategy appropriate to thecharacteristics of your audience. To developan appropriate communications strategy, usethe guidelines described in this chapter.The briefings will be evaluated on the basis ofthe following checklist and rating scales. Youor your instructor should make four (4) copiesof the checklist—three to evaluate the otherpresentations and one for your records.

Box 3 Checklist for Evaluating an Oral Briefing

Group evaluated (circle): I II III IV

Use the following rating scale for your evaluation. Provide a brief written

justification in the space provided next to each item.

1 � Very good2 � Good3 � Adequate4 � Poor5 � Very poor



BIBLIOGRAPHYBarber, Bernard. Effective Social Science: Eight Cases

in Economics, Political Science, and Sociology.New York: Russell Sage Foundation, 1987.

Freidson, Elliot. Professional Powers: A Study ofthe Institutionalization of Formal Knowledge.Chicago: University of Chicago Press, 1986.

Hacker, Diana. A Writer’s Reference. 4th ed.New York: St. Martin’s Press, 1999.

Horowitz, Irving L., ed. The Use and Abuse ofSocial Science: Behavioral Science and PolicyMaking. 2d ed. New Brunswick, NJ: TransactionBooks, 1985.

Jasanoff, Sheila. The Fifth Branch: ScienceAdvisors as Policymakers. Cambridge, MA:Harvard University Press, 1990.

Lindblom, Charles E., and David K. Cohen. UsableKnowledge: Social Science and Social ProblemSolving. New Haven, CT: Yale University Press,1979.

Machlup, Fritz. Knowledge: Its Creation,Distribution, and Economic Significance,Vol. 1, Knowledge and Knowledge Production.Princeton, NJ: Princeton University Press, 1980.

Schmandt, Jurgen, and James E. Katz. “TheScientific State: A Theory with Hypotheses.”Science, Technology, and Human Values, 11(1986): 40–50.

Tufte, Edward R. The Visual Display of QuantitativeInformation. Cheshire, CT: Graphics Press, 1983.

———. Envisioning Information. Cheshire, CT:Graphics Press, 1990.

———. Visual Explanations. Cheshire, CT:Graphics Press, 1997.

Weiss, Carol H. with Michael Bucuvalas. SocialScience Research and Decision Making. NewYork: Columbia University Press, 1980.

1. Rate the effectiveness of the following elements of the briefing:(a) opening _________(b) background _________(c) findings _________(d) methods _________(e) evidence/data _________(f) recommendations _________(g) questions and answers _________(h) closing _________

2. Appropriateness of briefing to:(a) size of the audience _________(b) expertise of audience _________(c) familiarity of audience with methods _________(d) audience information preferences _________(e) phase of policy-making process _________(f) perceptions of presenter’s credibility _________

3. Logic, organization, and flow of briefing: _________

4. Use of visual displays: _________

5. Overall, this oral briefing was _________


In the chapter “Developing Policy Arguments,” thecase of the Second Persian Gulf War was presented.The argument map showing the pros and cons of U.S.intervention was shown in Figure C8.1

This argument map can be translated into fourkinds of policy documents:

� The position paper. The position paper states onlythe reasons for making a policy claim. No

An important skill is the translation of policyarguments into structured written narratives.Thesewritten narratives go by different names: whitepapers, issue papers, memos, press releases, andletters. All are used to communicate the results ofanalysis in hopes of maximizing the likelihood thatthey will actually be used to make and influencedecisions.

CASE 1 TRANSLATING POLICYARGUMENTS INTO ISSUE PAPERS, POSITIONPAPERS,AND LETTERS TO THE EDITOR—THECASE OF THE SECOND PERSIAN GULF WAR

INFORMATION (98)The Dole-Warner resolution/November 29,1990 supports UN Security CouncilResolution 678 authorizing U.S. militaryintervention to force an Iraqi withdrawalfrom Kuwait.

WARRANT (51)Because Iraq is a threatto the security of theregion and the world.

BACKING (16-37)Since Iraq has a large military (37), regionalpolitical aspirations (32), and petroleumreserves that give economic control (16).

OBJECTION (37)But the authorization is unwarranted. Itis an artificial creation of the presidentand no vital interests are at stake.

WARRANT (16)Because the U.S.has an obligation tointervene.

BACKING (16)Since the U.S. is a member of the UN,committed to Israel and moderateArab states, and opposed to humanrights violations.

OBJECTION (47)But sanctions are working. GNP iscut in half and imports and exportshave been reduced by 90% to 97%.

REBUTTAL (34)However, sanctions have not produced a withdrawal.They will cause economic problems for allies andcause the suffering of innocent persons.

OBJECTION (33)But availablediplomatic optionshave not beenpursued.

REBUTTAL (9)However, diplomacy hasnot worked in the past andwill not work now.

QUALIFIER (30)Immediately andwithout hesitation.

CLAIM (52)The Senate should authorizethe use of force under the Dole-Warner resolution.

This case draws on the subprogram in Rationale for creating structured narratives from argument maps.



opposing points of view, objections, or rebuttalsare included. Position papers state one side of anargument as persuasively as possible.

� The policy brief. The policy brief is similar to alegal brief. It states the reasons for a claim,followed by objections, which are rebutted. Issuepapers and memos also state opposing argumentsand then show they are not plausible or valid.

� The policy issue paper or policy memo. The policyissue paper or policy memo (a shortened versionof the policy issue paper) presents both sides ofan issue equally, without taking sides. Reasonsand objections are provided to dispute issues.

� The letter to the editor. The letter to the editor beginsby considering an objection to the claim, rebuttingit, and then providing multiple reasons to supportthe claim.The letter to the editor is an example of afortiori analysis.

In each of the following written documents theargument map on the Dole-Warner resolution hasbeen translated into a draft version of a structuredwritten narrative.The statements in bold letters aretaken directly from the argument map.The documentswould require editing before release.

Argument Map for a Position Paper

Position Paper on the U.S. Invasion of IRAQ

There are three reasons to believe that the Senate

should authorize the use of force under the Dole-Warner

resolution.

First, the Dole-Warner resolution is justified by UN Security

Council Resolution 678 of November 29, 1990. This is evidentgiven that Resolution 678 authorizes U.S. military intervention

in Kuwait. Further evidence is that Resolution 678 will

force an Iraqi withdrawal from Kuwait.

Second, Iraq is a security threat to the region and

the world.This is supported by the reason thatIraq has a large military and regional political aspirations.Furthermore, Iraq has petroleum reserves that give it

economic control.Finally, the U.S. has an obligation to intervene.

Evidence for this is that the U.S. is a member of the UN

and governed by the UN Charter. In addition, The U.S. is

committed to the defense of Israel and moderate Arab states,which also shows that the U.S. has an obligation to

intervene.Given these considerations, it is clear that the

Senate should authorize the use of force under the Dole-Warner

resolution.

because

because because

because

because because

because

because because

CLAIMThe Senate shouldauthorize the use offorce under the Dole-Warner resolution.

WARRANTThe Dole-WarnerResolution is justifiedby UN SecurityCouncil Resolution 678of November 29, 1990.

WARRANTThe U.S. has anobligation to intervene.

WARRANTIraq is a securitythreat to the regionand the world.

BACKINGResolution 678authorizes U.S.military interventionin Kuwait.

BACKINGResolution 678 willforce an Iraqiwithdrawal fromKuwait.

BACKINGIraq has a largemilitary andregional politicalaspirations.

BACKINGIraq has petroleumreserves that give iteconomic control.

BACKINGThe U.S. is amember of the UNand governed bythe UN Charter.

BACKINGThe U.S. iscommitted to thedefense of Israeland moderateArab states.



Argument Map for a Policy Brief

Issue Brief on the U.S. Invasion of IRAQ

The Senate should authorize the use of force under the Dole-

Warner resolution.There are two main reasonssupporting this contention, and the main objection toit can be rebutted.

The first reason is that the Dole-Warner resolution is

justified by UN Security Council Resolution 678 of November 29,

1990. This reason is based on the claim that Resolution

678 authorizes U.S. military intervention in Kuwait.

The second reason that the Senate should authorize the

use of force under the Dole-Warner resolution is that the U.S.

has an obligation to intervene.This is because the U.S. is

committed to the defense of Israel and moderate Arab states.

On the other hand, a consideration against theidea that the Senate should authorize the use of force under the

Dole-Warner tesolution is that fiplomatic options are available

and have not been pursued. This objection is notconvincing, however, because diplomacy has not worked in

the past with Iraq and will not work now.

Based on this reasoning, it is clear that the Senate


resolution.

Argument Map for a Policy Issue Paper or Policy Memo

Policy Memo on the U.S. Invasion of IRAQ

The question has been raised whether the Senate


resolution.This analysis considers a number ofperspectives to determine whether to supportor reject this claim.

One reason to accept that the Senate should authorize

the use of force under the Dole-Warner resolution is thesuggestion that the Dole-Warner resolution is justified by UN

Security Council Resolution 678 of November 29, 1990.Thisreason is supported by the fact that Resolution 678

authorizes U.S. military intervention in Kuwait. An objectionto the claim that the Dole-Warner resolution is justified by

UN Security Council Resolution 678 of November 29, 1990, isthat the resolution is an artificial creation of the president and no

vital interests are at stake.

Another reason to believe that the Senate should

authorize the use of force under the Dole-Warner resolution isthat Iraq is a security threat to the region and the world. This isbased on the idea that Iraq has a large military and regional

political aspirations.

The Senate should authorizethe use of force under theDole-Warner resolution.

The Dole-Warnerresolution is justified byUN Security Council resolution 678/November29, 1990.

But diplomaticoptions areavailable andhave not beenpursued.

The U.S. hasan obligation tointervene.

The U.S. iscommitted to thedefense of Israel and moderateArab states.

However,diplomacy has notworked in the pastwith Iraq and willnot work now.

CLAIM

WARRANT OBJECTION WARRANT

BACKING REBUTTAL Resolution 678authorizes U.S.military interventionin Kuwait.

BACKING



Nonetheless, the opposing view that sanctions

are working and military action is not necessary providesan interesting perspective. On balance, however,it seems more reasonable to reject this argument.

On the other hand, a consideration against theclaim that the Senate should authorize the use of force under

the Dole-Warner resolution is that the authorization is

unwarranted. This objection is supported by the claimthat the U.S. is a sovereign nation and should not be subservient

to the UN. Nevertheless, the rebuttal that the resolution is

supported by a 51–47 vote of the U.S. Senate must beconsidered to determine whether the objection isindeed acceptable.

Finally, there is another significant objection tothe claim that the Senate should authorize the use of force

under the Dole-Warner resolution. Diplomatic options are

available and have not been pursued. To support this view isthe reason that diplomacy should be given a chance. Inrebuttal, it is suggested that diplomacy has not worked in

the past with Iraq and will not work now.

In summary, it would appear more reasonable toaccept the claim that the Senate should authorize the use of

force under the Dole-Warner resolution.The strongest line ofreasoning for this view is that Iraq is a security threat to

the region and the world. Iraq has a large military and regional

political aspirations.

Argument Map for a Letter to the Editor

Letter to the Editor on the U.S. Invasion of IRAQ

Dear Sir / Madam,I am writing to contend that the Senate should

authorize the use of force under the Dole-Warner resolution.

This is contrary to the view voiced by the Senateminority, which has argued that sanctions are working

and military action is not necessary. I reject this claim becausediplomacy has not worked in the past with Iraq and it will not

work now.

Furthermore, there are three strong reasons tosupport my view that the Senate should authorize the

use of force under the Dole-Warner resolution. The firstreason is that the Dole-Warner resolution is justified by

UN Security Council Resolution 678 of November 29, 1990,

since Resolution 678 authorizes U.S. military intervention in

Kuwait. In addition, this reason is supported by thefact that Britain and other key allies support

Resolution 678.

The second reason is that Iraq is a security threat to the

region and the world.

WARRANTThe Dole-Warner Resolution is justifiedby UN SecurityCouncil resolution 678of November 29, 1990.


BACKING Resolution678 authorizesU.S. militaryinterventionin Kuwait.

OBJECTION The resolutionis an artificialcreation of thepresidentand no vitalinterests areat stake.

BACKING Iraq has alarge militaryand regionalpoliticalaspirations.

OBJECTION Sanctionsare workingand militaryaction is notnecessary.

WARRANTThe U.S. isa sovereignnation andshould not besubservient tothe UN.

REBUTTALThe resolutionis supportedby a 51--47vote of theU.S. Senate.

WARRANTDiplomacyshould begiven achance.

REBUTTALHowever,diplomacyhas notworked inthe past withIraq and willnot work now.

OBJECTION The authorizationis unwarranted.

WARRANT Diplomatic optionsare available andhave not beenpursued.

CLAIM The Senate shouldauthorize the use offorce under the Dole-Warner resolution.



Finally, it is clear that the Senate should authorize the

use of force under the Dole-Warner resolution, given that Iraq

has large military and regional political aspirations. This reasonis supported by the contention that Saddam Hussein has

threatened to remain in Kuwait.

I restate my contention that the Senate should

authorize the use of force under the Dole-Warner resolution. Iurge you to also adopt this view.

Yours sincerely,[Your name]

Policy analysts must deal with many kindsof empirical evidence. Often the constraintsof time and resources, as well as the natureof the policy problem under consideration, forceanalysts to rely on qualitative and fragmentarydata. Sometimes, however, analysts can find datathat enable them to estimate the magnitudes

because

because because

because because

because

CLAIMThe Senate shouldauthorize the use offorce under the Dole-Warner resolution.

WARRANTThe Dole-WarnerResolution is justifiedby UN SecurityCouncil Resolution 678of November 29, 1990.

but

OBJECTION Sanctions are workingand military action isnot necessary.

because

REBUTTALDiplomacy has notworked in the pastwith Iraq and it will not work now.

WARRANTIraq has a largemilitary and regional political aspirations.


BACKINGResolution 678authorizes U.S.military interventionin Kuwait.

BACKINGBritain and otherallies supportresolution 678.

BACKINGSaddam Husseinhas threatened toremain in Kuwait.

CASE 2 COMMUNICATING STATISTICALANALYSES TO MULTIPLE AUDIENCES:REGULATING LEADED GASOLINEWhen Statistics Count: Revising the EPA Lead Standard20

of the effects of policy interventions on the social,economic, or political conditions.The estimatesmay permit the analysts to calculate the likelynet social benefits of proposed policies or evenapply formal optimization techniques to findbetter alternatives.

20From David Weimer and Aidan Vining, Policy Analysis: Concepts and Practice, 2d ed.(Englewood Cliffs, NJ: Prentice Hall 1992), Ch. 13, pp. 382–406. Footnotes 21 through 56 beloware from Chapter 13 of Weimer and Vining.



The effective use of quantitative data requires anunderstanding of the basic issues of research designand a facility with the techniques of statisticalinference. Even when analysts do not have theresources available to do primary data analysis, theyoften must confront quantitative evidence producedby other participants in the policy process orextracted from the academic literature. If they lackthe requisite skills for critical evaluation, they risklosing their influence in the face of quantitativeevidence that may appear more objective andscientific to decision makers.Well-trained analysts,therefore, need some facility with the basic conceptsof statistical inference for reasons of self-defense, iffor no others.

The basics of research design and statisticalinference cannot be adequately covered in anintroductory course in policy analysis. Similarly, wecannot provide adequate coverage in this text.Yet wecan provide an example of a policy change wherequantitative analysis was instrumental: the decisionby the U.S. Environmental Protection Agency in1985 to reduce dramatically the amount of leadpermitted in gasoline.The story of the new leadstandard has a number of elements frequentlyencountered when doing quantitative analysis inorganizational settings: replacement of an initial“quick and dirty” analysis with more sophisticatedversions as more time and data become available;repeated reanalysis to rule out alternativeexplanations (hypotheses) offered by opponents ofthe proposed policy; and serendipitous events thatinfluence the analytical strategy. Although ourprimary purpose is to tell what we believe to be anintrinsically interesting and instructive story aboutthe practice of policy analysis, we hope to offer afew basic lessons on quantitative analysis alongthe way.

Background:The EPA Lead Standards

The Clean Air Amendments of 1970 give theadministrator of the Environmental ProtectionAgency authority to regulate any motor fuel

component or additive that produces emissionproducts dangerous to public health or welfare.21

Before exercising this authority, however, theadministrator must consider “all relevant medicaland scientific evidence available to him” as well asalternative methods that set standards for emissionsrather than for fuel components.22

In 1971 the EPA administrator announced thathe was considering possible controls on lead additivesin gasoline.23 One reason given was the possibleadverse health effects of emissions from engines thatburned leaded gasoline, the standard fuel at the time.Available evidence suggested that lead is toxic in thehuman body, that lead can be absorbed into the bodyfrom ambient air, and that gasoline engines accountfor a large fraction of airborne lead.The otherreason for controlling the lead content of gasolinewas the incompatibility of leaded fuel with thecatalytic converter, a device seen as having potentialfor reducing hydrocarbon emissions fromautomobiles.The first reason suggests consideringwhether reductions in the lead content of all gasolinemight be desirable; the second argues for the totalremoval of lead from gasoline for use by newautomobiles equipped with catalytic converters.

Without going any further one might ask:Whyshould the government be concerned with the level oflead in gasoline? And why had the federalgovernment already decided to require theinstallation of catalytic converters? The followinganalysis does not explicitly deal with the desirabilityof catalytic converters. As we have argued, however,it is always important to make sure that there is aconvincing rationale for public action.Theinterventions at hand, the catalytic converter and therelated lead restrictions, deal with market failures—viewed as problems of either negative externalities orpublic goods (ambient public goods). In addition,another market failure may come into play withrespect to lead—imperfect consumer informationabout the impacts of lead on health and themaintenance cost of vehicles. Because the health andmaintenance impacts may not manifest themselves

21The Clean Air Act Amendments of 1970, Public Law 91–604, December 31, 1970.22Section 211(c)(2)(A), 42 U.S.C. @ 1857f-6c(c)(2)(A).2336 Fed. Reg. 1468 (January 31, 1971).



for many years (leaded gasoline is a post-experiencegood), markets may be inefficient because ofinformation asymmetry.These apparent marketfailures make a case for considering governmentintervention.

But intervention itself may be costly. It is quiteconceivable that the costs of intervention couldexceed the benefits. Several questions must beanswered before the comparison of costs and benefitscan be made:What are the impacts in the EconomicAnalysis Division [that] had worked on the 1982regulations. Schwartz used his accumulatedknowledge from previous analyses to complete in twoor three days a quick and dirty benefit-cost analysisof a total ban on lead additives.

Because a total ban was one of the optionsconsidered in 1982, it was fairly easy for Schwartzto come up with a reasonable estimate of cost. As isoften the case in evaluating health and safetyregulations, the more difficult problem wasestimating the benefits of a total ban. Schwartzlooked at two benefit categories: increased IQ scoresof children due to lower levels of blood lead and theavoided damage to catalytic converters.

Schwartz used estimates from a variety ofsources to piece together a relationship between leademissions and the present value of lifetime earningsof children.The first step involved using estimatesfrom the 1982 analyses of the relationship betweenlead emissions and blood lead levels in children. Henext turned to epidemiological studies that reporteda relationship between blood lead levels and IQs.Finally, he found econometric studies that estimatedthe contribution of IQ points to the present value offuture earnings.

As a first cut at quantifying the benefits resultingfrom the more effective control of other emissions,Schwartz estimated the cost of catalytic convertersbeing contaminated under the current standards thatwould not be contaminated under a total ban. Heused the number of converters saved multiplied bythe price per converter as the benefit measure.Assuming that converters themselves have benefit-cost ratios greater than one, this benefit measurewould be conservative.

These “back-of-the-envelope” calculationssuggested that the benefits of a total ban on lead

additives would be more than twice the costs.Schwartz discussed the results with his branch chief,G. Martin Wagner, who sent word to the office of theadministrator that further analysis of a lead banseemed worthwhile. A few weeks later Schwartz andfellow analyst Jane Leggett began a two-montheffort to move from the back of an envelope to apreliminary report.

Pulling the Pieces Together

The most urgent task facing Schwartz and Leggettwas to develop better measures of the benefits of alead ban.The key to improving the measure ofbenefits from avoided converter poisonings was amore sophisticated accounting of the future agecomposition of the U.S. vehicle fleet.The key toimproving the measure of benefits from reduced leademissions was a better quantitative estimate of therelationship between gasoline lead and blood lead.Modeling and statistical analysis would be animportant part of their work.

The age composition of the vehicle fleetcontinually changes as old vehicles, some of whichhave contaminated or partially contaminatedconverters, are retired and new ones, with freshconverters, are added. A ban on lead would havedifferent effects on different vintage vehicles. Forinstance, it would be irrelevant for vehicles thatalready have contaminated converters, but veryimportant for vehicles that would otherwise becontaminated and remain in service for a long time.

The analysts developed an inventory model thattracked cohorts of vehicles over time. Each year anew cohort would enter the fleet. Each successiveyear a fraction of the vehicles would be retiredbecause of accidents and mechanical failures. Inaddition, a fraction would suffer converter poisoningfrom misfueling. By following the various cohortsover time, it was possible to predict the total numberof converters that would be saved in each future yearfrom specified reductions in the lead content ofleaded gasoline today and in the future. Avoided lossof the depreciated value of the catalytic converters(later avoided costs of the health and propertydamage from pollutants other than lead) could thenbe calculated for each future year.With appropriatediscounting, the yearly benefits could then be



summed to yield the present value of the leadreduction schedule being considered.

Two important considerations came to lightduring work on the vehicle fleet model. One was abody of literature suggesting that lead increases theroutine maintenance costs of vehicles. Subsequently,a new benefit category, avoided maintenance costs,was estimated using the vehicle fleet model.The otherconsideration was the possibility that some enginesmight suffer premature valve-seat wear if fueled withtotally lead-free gasoline. Although the problem wasfairly limited (primarily automobile enginesmanufactured prior to 1971 and some newer trucks,motorcycles, and off-road vehicles), it suggested theneed to consider alternatives to a total ban.

Efforts to quantify better the link betweengasoline lead and blood lead focused on analysis ofdata from the second National Health and NutritionExamination Survey (NHANES II).The NHANES IIsurvey was designed by the National Center forHealth Statistics to provide a representative nationalsample of the population between ages 6 months and74 years. It included 27,801 persons sampled at

sixty-four representative sites from 1976 to 1980.Of the 16,563 persons who were asked to provideblood samples, 61 percent complied.The leadconcentrations in the blood samples were measured,providing data that could be used to track averageblood levels over the four-year survey period.Theamount of lead in gasoline sold over the period couldthen be correlated with the blood leadconcentrations.

A positive relationship between blood leadconcentrations from the NHANES II data andgasoline lead had already been found by researchers.The positive correlation is apparent in [Figure 2],which was prepared by James Pirkle of the Centersfor Disease Control to show the close tracking ofblood lead concentrations and total gasoline lead.Much more than the apparent correlation, however,was needed for the benefit-cost analysis. Schwartzused multiple regression techniques . . . to estimatethe increase in average micrograms of lead perdeciliter of blood (µg/dl) due to each additional 100metric tons per day of lead in consumed gasoline.He also employed models that allowed him to

1976 1977 1978 1979 1980Year

50

60

70

80

90

100

110

Tota

l lea

d u

sed

6-m

onth

per

iod

(1,0

00 t

ons)

10

9

11

12

13

14

15

16

Ave

rage

blo

od le

ad le

vels

(�g/

dl)

Averagebloodlead levels

Lead used ingasoline

FIGURE 2Lead use in gasoline production and average NHANES II bloodlead levels

Source: J. Schwartz et al., Costs and Benefits of Reducing Lead in Gasoline: Final Regulatory Impact Analysis,

Publication No. EPA-230-05-85-006 (Washington, D.C.: EPA, 1985), p. E-5.



estimate the probabilities that children withspecified characteristics will have toxic levels ofblood lead (then defined by the Centers for DiseaseControl as greater than 30 µg/dl) and gasoline lead.These probabilities could then be used to predict thenumber of children in the population who wouldavoid lead toxicity if a ban on gasoline lead wereimposed.

In early November of 1983, Schwartz andLeggett pulled together the components of theiranalyses and found a ratio of benefits to costs of atotal ban to be greater than Schwartz’s originalback-of-the-envelope calculation.Together with theirbranch chief, they presented their results to DeputyAdministrator Alm, who found them encouraging. Hegave the green light for the preparation of a refinedversion that could be presented to the administratoras the basis for a new regulation. Alm also wantedthe various components of the refined study to besubjected to peer review by experts outside the EPA.At the same time, he urged speed in order to reducethe chances that word would get out to refiners andmanufacturers of lead additives, the primaryopponents of a ban, before the EPA had anopportunity to review all the evidence.

The branch chief expanded the analytical team inorder to hasten the preparation of the refined reportthat would be sent to the administrator. JoiningSchwartz and Leggett were Ronnie Levin; HughPitcher, an econometrician; and Bart Ostro, an experton the benefits of ozone reduction. In little more thanone month the team was ready to send a draft reportout for peer review.

The effort involved several changes in analyticalapproach. Because of the valve-head problem, theanalysis focused on a major reduction in grams oflead per leaded gallon (from 1.1 gplg to 0.1 gplg) aswell as a total ban.The available evidence suggestedthat the 0.1 gplg level would be adequate to avoidexcessive valve-head wear in the small number ofengines designed to be fueled only with leadedgasoline. At the same time, considerable effort wasput into quantifying the maintenance costs that

would be avoided by owners of other vehicles if thelead concentration were reduced. It soon appearedthat the maintenance benefits consumers would enjoywould more than offset the higher prices they wouldhave to pay for gasoline. Finally, the team decidedthat the benefits based on the blood lead (throughIQ) to future earnings relationship would be toocontroversial. Instead, they turned their attention tothe costs of compensatory education for children whosuffered IQ losses from high blood lead levels.

In late December, sections of the report were sentto experts outside the EPA for comments.The listincluded automotive engineers, economists,biostatisticians, toxicologists, clinical researchers,transportation experts, and a psychologist. DuringJanuary 1984, the team refined [its] analysis andincorporated, or at least responded to, the commentsmade by the external reviewers.

Finally, in early February, the team was ready topresent its results to Administrator Ruckelshaus. Heagreed that their analysis supported a new standardof 0.1 gplg. He told the team to finalize [its] reportwithout a proposed rule and release it for publiccomment. Ruckelshaus also directed the Office of theAssistant Administrator for Air and Radiation todraft a proposed rule.

The team’s Draft Final Report was printed andeventually released to the public on March 26,1984.24 The team continued to refine the analysis inthe following months. It also had to devoteconsiderable time to external relations. ExecutiveOrder 12291 requires regulatory agencies to submitproposed regulations that would have annual costs ofmore than $100 million to the Office of Managementand Budget for review.The team met with OMBanalysts several times before securing theiracceptance of the benefit-cost analysis of the tighterstandard.

The political environment was taking shapepretty much as expected. Opposition would comefrom refiners and manufacturers of lead additives.The refiners, however, generally seemed resigned tothe eventual elimination of lead from gasoline.

24Joel Schwartz, Jan Leggett, Bart Ostro, Hugh Pitcher, and Ronnie Levin, Costs and Benefits ofReducing Lead in Gasoline: Draft Final Report (Washington, DC: Office of Policy Analysis, EPA,March 26, 1984).



Their primary concern was the speed of imple-mentation. Some refiners seemed particularlyconcerned about the first few years of a tighterstandard, when they would have difficulty making therequired reductions with their existing configurationsof capital equipment. In response, the team beganexploring the costs and benefits of less stringentcompliance schedules.

The manufacturers of lead additives were ready tofight tighter standards. In May Schwartz attended aconference at the Centers for Disease Control inAtlanta, Georgia, on the proposed revision of theblood lead toxicity standard for children from 30µg/dl to 25 µg/dl. Representatives of the leadmanufacturers were also there.They openly talkedabout their strategy for challenging tighter standards.They planned to argue that refiners would blend morebenzene, a suspected carcinogen, into gasoline toboost octane if lead were restricted. Schwartzinvestigated this possibility following the conference.He found that, even if more benzene were added togasoline, the total emissions of benzene would declinebecause of the reduction in the contamination ofcatalytic converters, which oxidize the benzene if notcontaminated.The day that the proposed rule waspublished Schwartz put a memorandum on the docketcovering the benzene issue, thus preempting themanufacturers’ main attack.

The EPA published the proposed rule on August2, 1984.25 It would require that the permitted levelof gasoline lead be reduced to 0.1 gplg on January 1,1986.The proposal stated the EPA assumption thatthe new standard could be met with existing refiningequipment, but indicated that alternative phase-inschedules involving more gradual reductions alsowere being considered in case this assumption provedfalse. Finally, the proposal raised the possibility of acomplete ban on gasoline lead by 1995.

A Closer Look at the Link Between Gasoline Leadand Blood Lead

Calculation of the direct health benefits of tighteningthe lead standard requires quantitative estimates ofthe contribution of gasoline lead to blood lead.

The NHANES II data, combined with information ongasoline lead levels, enabled the study team to makethe necessary estimates.Their efforts provide anexcellent illustration of how statistical inference canbe used effectively in policy analysis.

The Need for Multivariate Analysis

A casual inspection of [Figure 2] suggests a strongpositive relationship between gasoline lead and bloodlead.Why is it necessary to go any further? Onereason is the difficulty of answering, directly from[Figure 2], the central empirical question: How muchdoes the average blood lead level in the UnitedStates decline for each 1,000-ton reduction in thetotal gasoline lead used over the previous month?[Figure 2] indicates a positive correlation betweengasoline lead and blood lead—changes in blood leadtrack changes in gasoline lead quite closely. But forthe same correlation in the data, we could have verydifferent answers to our central question.

Figure 3 illustrates the difference betweencorrelation and magnitude of effect with stylizeddata. If our data were represented by the diamonds,we might “fit” line one as our best guess at therelationship between blood lead and gasoline lead.The effect of reducing gasoline lead from 500 tonsper day to 400 tons per day would be to reduceaverage blood lead from 10.1 µg/dl to 10.0 µg/dl, or0.1 µg/dl per 100 tons per day. An alternative sampleof data is represented by the dots, which are plottedto have approximately the same correlation betweengasoline lead and blood lead as the data representedby triangles.The slope of line two, the best fit to thedots, is 1.0 µg/dl per 100 tons per day—ten timesgreater than the slope of line one.That is, althoughthe two data sets have the same correlation, thesecond implies a much greater contribution ofgasoline lead to blood lead than the first.

Even after we display the data in [Figure 2] as aplot between blood lead and gasoline lead, ouranalysis is far from complete because the apparentrelationship (the slopes of lines one and two in ourexample) may be spurious. Blood lead and gasolinelead may not be directly related to each other but to

2549 Fed. Reg. 31031 (August 2, 1984).



2 3 4 5Gasoline lead (100 metric tons/day)

Ave

rage

blo

od le

ad (�

g/d

l)

9.0

10.0

11.0

12.0

13.0

14.0

Line two

Line one

some third variable that causes them to changetogether.The classic illustration of this problem is thecorrelation sometimes found between the density ofstork nests and the human birth rate. If one were toplot birth rates against the density of stork nests fordistricts in some region, one might very well find apositive relationship and perhaps conclude that theremight be something to the myth about storksbringing babies.

Of course, there is a more plausible explanation.A variable measuring the degree to which a district isrural “intervenes” between birth rate and nestdensity—rural areas have farmers who want a lot ofchildren to help with chores as well as lot of openland that provides nesting grounds for storks; moreurbanized areas tend to have people who wantsmaller families as well as less hospitable nestinggrounds. Looking across a sample that included bothrural and urban districts could yield the positivecorrelation between birth rate and nest density. If wewere to “control” statistically for the interveningvariable by looking at either rural or urban districtsseparately rather than pooled together, we would

expect the correlation between birth rate and nestdensity to become negligible.

Returning to our analysis of lead, we must beconcerned that one or more intervening variablesmight explain the positive correlation betweengasoline lead and blood lead. For example, there issome evidence that cigarette smokers have higherblood lead levels than nonsmokers. It may be thatover the period that the NHANES II data werecollected the proportion of smokers in the sampledeclined, so that much of the downward trend inaverage blood lead levels should be attributed toreductions in smoking rather than reductions ingasoline lead.

To determine whether smoking is an intervening,or confounding, variable, we might construct separatediagrams like [Figure 3] for smokers andnonsmokers. In this way, we could control for thepossibility that changes in the proportion of smokersin the sample over time were responsible for changesin blood lead levels. If we fit lines with similarpositive slopes to each of the samples, we wouldconclude that smoking behavior was not an

FIGURE 3Data samples with identical correlations but differentregression lines



intervening variable for the relationship betweengasoline lead and blood lead.26

With unlimited quantities of data, we could alwayscontrol for possible intervening variables in this way.Unfortunately, with a fixed sample size, we usuallystretch our data too thin if we try to form subsets ofdata that hold all variables constant (e.g., everyone isa nonsmoker) except the dependent variable (bloodlead levels) we are trying to explain and theindependent variable (gasoline lead) we areconsidering as a possible explanation. For example,occupational exposure, alcohol consumption, region,and age are only a few of the other variables thatmight be causing the relationship in our data betweengasoline lead and blood lead. If we selected from oursample only adult males living in small cities in theSouth who are moderate drinkers and nonsmokers andhave no occupational exposure to lead, we might havetoo few data to reliably fit a line as in [Figure 3]. Evenif we had adequate data, we would end up withestimates of the relationship between gasoline leadand blood lead from all of our subsets. (The number ofsubsets would equal the product of the number ofcategories making up our control variables.) We mightthen have difficulty combining these subset estimatesinto an overall estimate.27

The Basic Linear Regression Model

Linear regression provides a manageable way tocontrol statistically for the effects of severalindependent variables.28 Its use requires us toassume that the effects of the various independentvariables are additive.That is, the marginal effect onthe dependent variable of a unit change in any one ofthe independent variables remains the same nomatter what the values of the other independentvariables [are].29 We can express a linear regressionmodel in mathematical form:

y � b0 � b1x1 � b2x2 � b2x2 � . . . � bkxk � e

where y is the dependent variable, x1, x2, . . . , xk are thek independent variables, b0, b1, . . . , bk are parameters(coefficients) to be estimated, and e is an error termthat incorporates the cumulative effect on y of all thefactors not explicitly included in the model. If we wereto increase x1 by one unit while holding the values ofthe other independent variables constant, y wouldchange by an amount b1. Similarly, each of thecoefficients measures the marginal effect of a unitchange in its variable on the dependent variable.

Imagine that we set the values of all the independentvariables except x1 equal to zero.We could then plot y

26If the two lines coincided, we would conclude that smoking did not contribute to blood lead. If thelines were parallel but not identical, the difference between their intercepts with the vertical axis wouldrepresent the average effect of smoking on blood lead. If the lines were not parallel, we might suspectthat smoking interacted with exposure to gasoline lead so that their effects are not additive. That is,smokers were either more or less susceptible to exposure to gasoline lead than nonsmokers.27If the relationship truly varies across our subsets, then it would generally not make sense to make anoverall estimate. As we will explain later, however, we never really observe the true relationship—italways contains some unknown error. Therefore, we may not be sure if it is reasonable to combine oursubsample data. Because the variance of the error will be larger, the smaller the size of our subsamples,the more we divide our data, the more difficult it is to determine whether observed differences reflecttrue differences.28For clear introductions, see Eric A. Hanushek and John E. Jackson, Statistical Methods for the SocialSciences (New York: Academic Press, 1977); and Christopher H. Achen, Intrepreting and UsingRegression (Beverly Hills, Calif.: Sage Publications, 1982).29This assumption is not as restrictive as it might at first seem. We can create new variables that arefunctions of the original independent variables to capture nonlinearities. For example, if we thoughtsmokers were likely to absorb environmental lead faster than nonsmokers, we might include in ourlinear regression model a new variable that is the product of the number of cigarettes smoked per dayand the level of gasoline lead. This new variable, capturing the interaction of smoking and gasoline lead,would have an effect that would be additive with the other independent variables in the model, includ-ing the smoking and gasoline lead variables. The marginal effect of gasoline lead on blood lead wouldconsist of the contribution from the gasoline lead variable and the contribution from the variable repre-senting its interaction with smoking, which will change depending on the particular level of smoking.



against x1 in a graph like [Figure 3].The equation y � b0 � b1x1 would represent the line fit to our sampleof observations.The slope of the line is b1, themagnitude of the change in y that will result from a unitchange in x1, other things equal.The actual observationsdo not lie exactly on the line, however.Their verticaldistances from the line will be equal to the randomerror, represented in our model by e, incorporated ineach of the observations of y. If the values of e aresmall, our line will fit the data well in the sense that theactual observations will lie close to it.

How should we go about fitting the line? The mostcommonly used procedure is the method of ordinary least

squares (OLS).When we have only one independentvariable, so we can plot our data on a two-dimensionalgraph like [Figure 3], the OLS procedure picks the linefor which the sum of squared vertical deviations fromthe observed data is smallest.When we have morethan one independent variable, OLS determines thevalues of the coefficients (b0, b1, . . . , bk) that minimizethe sum of squared prediction errors.30 As long as thenumber of observations in our sample exceeds thenumber of coefficients we are trying to estimate, andnone of our independent variables can be expressed asa linear combination of the other independentvariables, the commonly available regression software

packages will enable us to use computers to find theOLS fitted coefficients.31

The estimates of coefficients that we obtain fromOLS will generally have a number of very desirableproperties. If the independent variables are alluncorrelated with the error term (e), then ourcoefficient estimators will be unbiased.32 (NOTE: anestimator is the formula we use to calculate aparticular estimate from our data.) To understandwhat it means for an estimator to be unbiased, wemust keep in mind that our particular estimatedepends upon the errors actually realized in oursample of data. If we were to select a new sample, wewould realize different errors and hence differentcoefficient estimates.When an estimator is unbiased,we expect that the average of our estimates acrossdifferent samples will be very close to the truecoefficient value. For example, if gasoline lead had notrue effect on blood lead, we would almost certainlyestimate its coefficient to be positive or negative,rather than exactly zero. Repeating OLS on a largenumber of samples and averaging our estimates ofthe coefficient of gasoline lead, however, wouldgenerally yield a result very close to zero.33 Indeed,by adding more and more samples, we could get theaverage as close to zero as we wanted.

30The prediction error is the observed value of the dependent variable minus the value we would predictfor the dependent variables based on our parameter estimates and the values of our independentvariables. For the ith observation the prediction error is given by:

The prediction error is also referred to as the residual of the observation. OLS selects the parameter estimates to minimize thesum of the squares of the residuals.31When one independent variable can be written as a linear combination of the others, we have a case ofperfect multicollinearity. A related and more common problem, which we can rarely do anything about,occurs when the independent variables in our sample are highly correlated. This condition, called multic-ollinearity, is not a problem with the specification of our model but with the data we have available toestimate it. If two variables are highly correlated, positively or negatively, OLS has difficulty identifyingtheir independent effects on the dependent variable. As a result, the estimates of the parameters associ-ated with these variables will not be very reliable. That is, they will have large variances, increasing thechances that we will fail to recognize, statistically speaking, their effects on the dependent variable. Oneway to deal with multicollinearity is to add new observations to our sample that lower the correlation.For example, if we had a high positive correlation between smoking and drinking in our sample, weshould try to add observations on individuals who smoke but do not drink and who drink but do notsmoke. Unfortunately, we often have no choice but to work with the data that are already available.32Strictly speaking, we must also assume that our independent variables are fixed in the sense that wecould construct a new sample with exactly the same observations on the independent variables. Ofcourse, even if the independent variables are fixed, we would observe different values of the dependentvariable because of the random error term. In addition, for complete generality, we must assume thatthe expected value of the error term is constant for all observations.

yi - (bN 0 + bN 1x1i + Á + bN kxki)



Unfortunately, we usually only have a singlesample for estimating coefficients. How do we decideif an estimate deviates enough from zero for us toconclude that the true value of the parameter is notzero? Making the fairly reasonable assumption thatthe error term for each observation can be treated asa draw from a Normal distribution with constantvariance, the OLS estimators will be distributedaccording to the Student-t distribution.34 That is, wecan interpret the particular numerical estimate of acoefficient as a draw from a random variabledistributed as a Student-t distribution centered [on]the true value of the coefficient. (The OLS estimatoris the random variable; the actual estimate based onour data is a realization of that random variable.)

Knowing the distribution of the OLS estimatorenables us to interpret the statistical significance of ourcoefficient estimate.We determine statisticalsignificance by asking the following question: Howlikely is it that we would observe a coefficientestimate as large as we did if the true value of thecoefficient were zero? We answer this question byfirst assuming that the true value of the coefficient iszero (the null hypothesis) so that the distribution of

our estimator is centered [on] zero.We thenstandardize our distribution to have a variance ofone by dividing our coefficient estimate by anestimate of its standard error (a by-product of theOLS procedure).The resulting number, called the t-ratio, can then be compared to critical values intabulations of the standardized Student-t distributionfound in the appendix of almost any statistics text.For example, we might decide that we will reject thenull hypothesis that the true value of the coefficientis zero if there is less than a 5 percent probability ofobserving a t-ratio (in absolute value sense) as largeas we did if the null hypothesis is true. (Theprobability we choose puts an upward bound on theprobability of falsely rejecting the null hypothesis.)35

To carry out the test, we look in the standardizedtabulations of the Student-t distribution forthe critical value corresponding to 5 percent.36

If the absolute value of our estimated t-ratio exceedsthe critical value, then we reject the null hypothesisand say that our estimated coefficient is statisticallysignificantly different from zero.

Fortunately, most regression software saves usthe trouble of looking up critical values in tables

33Our average will not be close to zero if gasoline lead is correlated with a variable excluded from ourmodel that does have an effect on blood lead. In this case, gasoline lead stands as a proxy for theexcluded variable. Other things equal, the stronger the true effect of the excluded variable on blood leadand the higher the absolute value of the correlation between gasoline lead and the excluded variable, thegreater will be the bias of the coefficient of gasoline lead. We might not worry that much about the biasif we knew that it would approach zero as we increase sample size. (If the variance of the estimator alsoapproached zero as we increased sample size, we would say that the estimator is consistent.) AlthoughOLS estimators are consistent for correctly specified models, correlation with an important excludedvariable makes an estimator inconsistent.34The Central Limit Theorem tells us that the distribution of the sum of independent random variablesapproaches the Normal distribution as the number in the sum becomes large. The theorem applies foralmost any starting distributions—the existence of a finite variance is sufficient. If we think of the errorterm as the sum of all the many factors excluded from our model, and further, we believe that they arenot systematically related to each other or the included variables, then the Central Limit Theoremsuggests that the distribution of the error terms will be at least approximately Normal.35Falsely rejecting the null hypothesis is referred to as Type I error. Failing to reject the nullhypothesis when in fact the alternative hypothesis is true is referred to as Type II error. We usually setthe probability of Type I error at some low level like 5 percent. Holding sample size constant, thelower we set the probability of Type I error, the greater the probability of Type II error.36The Student-t distribution is tabulated by degrees of freedom. In the basic OLS framework, the degrees offreedom is the total number of observations minus the number of coefficients being estimated. As the degreesof freedom becomes larger, the Student-t distribution looks more like a standardized Normal distribution.You should also note the difference between a one-tailed and two-tailed test. Because the standardizedStudent-t is a symmetric distribution centered on zero, a 5 percent test usually involves setting criticalvalues so that 2.5 percent of area lies under each of the tails (positive and negative). A one-tailed test,appropriate when the null hypothesis is that the true coefficient value is zero or less than zero, puts theentire 5 percent in the positive tail.



by directly calculating the probability under the nullhypothesis of observing a t-ratio as large as thatestimated.To do a classical test of hypothesis on thecoefficient, we simply see if the reported probabilityis less than the maximum probability of falselyrejecting the null hypothesis that we are willing toaccept. If it is smaller, then we reject the nullhypothesis.

Consider the regression results presented in[Table 3].They are based on data from 6,534 whitesin the NHANES II survey for whom blood leadmeasurements were made.37 The dependent variableis the individual’s blood lead level measured in µg/dl.The independent variables are listed under theheading “Effect.”The independent variable ofprimary interest is the national consumption ofgasoline lead (in hundreds of metric tons per day)in the month prior to the individual’s blood leadmeasurement.The other independent variables wereincluded in an effort to control statistically for otherfactors that might be expected to affect theindividual’s blood lead level.With the exception ofthe number of cigarettes smoked per day and thedietary factors (vitamin C, riboflavin, and so on),these other statistical controls are indicator, or“dummy,” variables that take on the value one ifsome condition is met and zero otherwise. So, forexample, if the individual is male, the variable“male” will equal one; if the individual is female,it will equal zero.The variables “vitamin C,”“phosphorus,”“riboflavin,” and “vitamin A,” whichare included as proxy measures for dietary intake oflead, each measure dietary intake in milligrams.

Otherwise, the other variables are intended to capturedemographic, income, occupational exposure,drinking habit, and locational effects.The reportedR2 indicates that, taken together, the independentvariables explain about 33 percent of the totalvariation in blood lead levels.38

The estimated coefficient for gasoline lead is2.14 µg/dl of blood per 100 metric tons per day ofnational gasoline lead consumption. Dividing thecoefficient estimate by its estimated standard errorof 0.192 yields a t-ratio of about 11.The probabilityof observing a t-ratio this large or larger if the truevalue of the coefficient were actually zero is less thanone chance in 10,000 (the 0.0000 entry in [Table 3]under “P-value”).We would thus reject the nullhypothesis in favor of the alternative hypothesis thatgasoline lead does contribute to blood lead. In otherwords, we would say that gasoline lead has astatistically significant effect on blood lead.

After finding a statistically significant effect, thenext question to ask is whether the size of thecoefficient is substantively significant. That is, does thevariable in question have an effect that is worthconsidering?39 One approach to answering this questionis to multiply the estimated coefficient by the plausiblechange in the independent variable that might occur. Forexample, by the end of the NHANES II survey, gasolinelead was being consumed at a rate of about 250 metrictons per day nationally. A strict policy might reduce thelevel to, say, 25 metric tons per day. Using the estimatedcoefficient of gasoline lead, we would expect a reductionof this magnitude to reduce blood lead levels on averageby about 4.8 µg/dl (the reduction of 225 metric tons

37The analysts estimated similar models for blacks and for blacks and whites together. Their estimatesof the coefficient for gasoline lead never deviated by more than 10 percent across the different samples.To conserve space, they reported in detail only their regression results for whites. They chose whitesbecause it was the largest subgroup and because it preempted the assertion that the relationship betweenblood lead and gasoline lead was due to changes in the racial composition of the sample over time.38R2 is a measure of the goodness of fit of the model to the particular sample of data. It is the squareof the correlation between the values of the dependent variable predicted by the model and the valuesactually observed. An R2 of one would mean that the model perfectly predicted the independentvariable for the sample; an R2 of zero would mean that the model made no contribution to prediction.39The standard errors of the coefficient estimates decrease as sample size increases. Thus, very largesamples may yield large t-ratios even when the estimated coefficient (and its true value) [is] small. Werefer to the power of a statistical test as 1 minus the probability of failing to reject the null hypothesis infavor of the alternative hypothesis. Other things equal, larger sample sizes have greater power, increasingthe chances that we will reject the null hypothesis in favor of alternatives very close to 0.



TABLE 3

Basic Regression Model for Estimating the Effects of Gasoline Lead on Blood Leada

Effect Coefficient Standard Error P-value

Intercept 6.15Gasoline 2.14 .142 0.0000Low income 0.79 .243 0.0025Moderate income 0.32 .184 0.0897Child (under 8) 3.47 .354 0.0000Number of cigarettes 0.08 .012 0.0000Occupationally exposed 1.74 .251 0.0000Vitamin C –0.04 .000 0.0010Teenager –0.30 .224 0.1841Male 0.50 .436 0.2538Male teenager 1.67 .510 0.0026Male adult 3.40 .510 0.0000Small city –0.91 .292 0.0039Rural –1.29 .316 0.0003Phosphorus –0.001 .000 0.0009Drinker 0.67 .173 0.0007Heavy drinker 1.53 .316 0.0000Northeast –1.09 .332 0.0028South –1.44 .374 0.0005Midwest –1.35 .500 0.0115Educational level –0.60 .140 0.0000Riboflavin 0.188 .071 0.0186Vitamin A 0.018 .008 0.0355aDependent variable: Blood lead (µg/dl) of whites in NHANES II Survey.

Source: Joel Schwartz et al., Costs and Benefits of Reducing Lead in Gasoline: Final Regulatory Impact Analysis (Washington, DC: EPA,1985), p. III-15.The original document reported incorrect standard errors.The standard errors reported here wereprovided by Joel Schwartz.

per day times the estimated coefficient of 2.14 µg/dlper 100 metric tons per day).

To get a better sense of whether a 4.8 µg/dlreduction is substantively important, we can look atthe blood lead levels for representative groups at the250 and 25 metric ton levels. For example, at the250 metric ton level a nonsmoking (number ofcigarettes equals zero), moderate drinking (drinkerequals one; heavy drinker equals zero),nonoccupationally exposed (occupationally exposedequals zero), adult female (child, teenager, male,male teenager, and male adult equal zero), living in alarge Northeastern city (Northeast equals one;small city, rural, South, and Midwest equal zero),

with moderate income, a college degree, and a high-nutrition diet (low income equals zero; moderateincome, educational level, vitamin C, phosphorus,riboflavin, and vitamin A equal one) would beexpected to have a blood lead level of 10.6 µg/dl.We would expect the same person to have a bloodlead level of only 5.8 µg/dl if the gasoline lead levelwere cut to 25 metric tons per day—a reduction ofabout 45 percent.Thus, the effect of gasoline lead onblood lead appears substantively as well asstatistically significant.

The study team was especially interested inestimating the contribution of gasoline lead to bloodlead in children. As a first cut, they developed a



logistic regression model40 for predicting theprobability that a child between the ages of sixmonths and eight years will have blood levels inexcess of 30 µg/dl, the definition of lead toxicity usedby the Centers for Disease Control at the time. Logistic

regression, which assumes a nonlinear relationshipbetween the dependent and the independentvariables,41 is usually more appropriate than linearregression when the dependent variable isdichotomous (Y equals one if the condition holds, Y

equals zero if it does not).42 The study team found astrong relationship in the NHANES II data betweengasoline lead and the probability that a child has atoxic level of blood lead. In fact, they estimated thatthe elimination of gasoline lead would have reducedthe number of cases of lead toxicity in the sampleby 80 percent for children under eight years of age.The study team used logistic regression and otherprobability models to estimate how reductions ingasoline lead would change the number of childrenhaving various blood lead levels.These estimateswere essential for their subsequent valuation of theeffects of gasoline lead reductions on the health ofchildren.

Reconsidering Causality

Finding that an independent variable in a regressionmodel has a statistically significant coefficient doesnot by itself establish a causal relationship.That is, itdoes not guarantee that changes in the independentvariable cause changes in the dependent variable.Even if the independent variable has no direct effecton the dependent variable, some other variable, notincluded in the model, may be correlated with both soas to produce an apparent relationship in the data

sample (remember the apparent relationship betweenbirth rates and the density of stork nests). Should thestrong relationship between gasoline lead and bloodlead be interpreted as causal?

The study team considered this question indetail. Although its demonstration was not legallynecessary, [team members] believed that adoptionof the proposed rule would be more likely if a strongcase for causality could be made.Their approachwas to apply the criteria commonly used byepidemiologists to determine the likelihood ofcausality. Not all of the criteria are directlyapplicable outside of the health area. Nonetheless,the way the study team applied the criteria illustratesthe sort of questioning that is valuable in empiricalresearch.Therefore, we briefly review the six criteriathat they considered.

Is the Model Biologically Plausible? The studyteam noted that lead can be absorbed through thelung and gut.They pointed out that gasoline lead, themajor source of environmental lead, is emittedpredominantly as respirable particulates inautomobile exhaust.These particulates can beabsorbed directly through the lungs.They alsocontaminate dust that can be inhaled through thelungs and absorbed through the gut.Therefore, theyargued, it is biologically plausible that gasoline leadcontributes to blood lead.

Biological plausibility is the epidemiologicalstatement of a more general criterion: Is the modeltheoretically plausible? Prior to looking throughdata for empirical relationships, you should specifya model (your beliefs about how variables arerelated). If you find that your data are consistent

40For an introduction to logistic regression, see Hanushek and Jackson, Statistical Methods, pp. 179–216.41The logistic regression model is written:

where P(Y) is the probability that condition Y holds, e is the natural base, and Z � b0 � b1x1 � . . . � bkxkis the weighted sum of the independent variables x1, x2, . . . , xk. The coefficients, b0, b1, . . . , bk, areselected to maximize the probability of observing the data in the sample. Note that the marginalcontribution of xi to the value of the dependent variable is not simply bi, as would be the case in a linearregression model. Rather, it is bi[1 � P(Y)]P(Y), which has its greatest absolute value when P(Y) � 0.5.42The logistic regression model, unlike the linear regression model, always predicts probabilities that liebetween zero and one (as they should).

P(Y) = ez/(1 = ez)



with your model, then you can be more confidentthat the relationships you estimate are not simplydue to chance.43

Is There Experimental Evidence to Support theFindings? The study team found reports of severalinvestigations specifically designed to measure thecontribution of gasoline lead to blood lead. One wasan experiment conducted in Turin, Italy, byresearchers who monitored changes in the isotopiccomposition of blood lead as the isotopiccomposition of gasoline lead varied.44 They foundthat at least 25 percent of the lead in the blood ofTurin residents originated in gasoline.Thus, theexperiment not only confirmed the biologicalplausibility of the contribution of gasoline lead toblood lead, but also suggested an effect on the orderof magnitude of that estimated by the study team.

Being able to find such strong and directlyrelevant experimental support is quite rare in policyresearch, much of which deals with the behavioralresponses of people. Controlled experiments in thesocial sciences are rare, not only because they arecostly and difficult to implement, but also becausethey often involve tricky ethical issues concerning theassignment of people to “treatment” and “control”groups. Nevertheless, there have been a number ofpolicy experiments in the United States over the lasttwenty years.45 It is unlikely, however, that any ofthese experiments will be directly applicable to yourpolicy problem.Therefore, you must typically broaden

your search for confirmation beyond experiments toother empirical research.

Do Other Studies Using Different Data Replicatethe Results? The study team reviewed severalstudies that also found relationships between gasolinelead and blood lead.These studies were based ondata collected in conjunction with community-widelead-screening programs funded by the Centers forDisease Control during the 1970s46 and on datacollected from the umbilical cord blood of over11,000 babies born in Boston between April 1979and April 1981.47 These studies reported statisticallysignificant relationships between gasoline lead andblood lead, and thus supported the study team’sanalysis based on the NHANES II data.

Does Cause Precede Effect? The study team usedinformation about the half-life of lead in blood to makepredictions about the strengths of relationshipsbetween lagged levels of gasoline lead and blood leadthat would be expected if gasoline lead contributes toblood lead. Lead has a half-life of about thirty days inblood. Noting that the average NHANES II blood testwas done at mid-month, they predicted that theprevious month’s gasoline lead (which on averagerepresents emissions occurring between fifteen andforty-five days before the test) should have a strongerimpact on blood lead than that of either the currentmonth (average exposure of zero to fifteen days) or themonth occurring two months prior (average exposureof forty-five to seventy-five days).They tested their

43Imagine that you regress a variable on twenty other variables. Assume that none of the twentyindependent variables has an effect on the dependent variable. (The coefficients in the true model are all zero.)Nevertheless, if you use a statistical test that limits the probability of falsely rejecting the null hypothesis to 5percent, then you would still have a 0.64 probability [1 – (.95)20] of rejecting at least one null hypothesis.In other words, if you look through enough data you are bound to find some statistically significant relation-ships, even when no true relationships exist. By forcing yourself to specify theoretical relationships before youlook at the data, you reduce the chances that you will be fooled by the idiosyncrasy of your particular datasample. For a brief review of these issues, see David L. Weimer, “Collective Delusion in the Social Sciences:Publishing Incentives for Empirical Abuse,” Policy Studies Review, Vol. 5, No. 4, May 1986, pp. 705–8.44S. Fachetti and F. Geiss, Isotopic Lead Experiment Status Report, Publication No. EUR8352ZEN(Luxembourg: Commission of the European Communities, 1982).45For a review of the major policy experiments conducted in the United States, see David H. Greenbergand Philip K. Robins, “The Changing Role of Social Experiments in Policy Analysis,” Journal of PolicyAnalysis and Management, Vol. 5, No. 2, Winter 1986, pp. 340–62.46Irwin H. Billick et al., Predictions of Pediatric Blood Lead Levels from Gasoline Consumption(Washington, D.C.: U.S. Department of Housing and Urban Development, 1982).47Michael Rabinowitz and Herbert L. Needleman, “Petrol Lead Sales and Umbilical Cord Blood LeadLevels in Boston, Massachusetts,” Lancet, January 1/8, 1983, p. 63.



predictions by regressing blood lead levels on current,one-month lagged, and two-month lagged gasoline leadlevels. As predicted, one-month lagged gasoline leadwas the most significant of the three. Also, consistentwith the thirty-day half-life, two-month lagged gasolinelead had a coefficient approximately one-half that ofone-month lagged gasoline lead.Thus, cause did appearto precede effect in the expected way.

Does a Stable Dose-Response RelationshipExist? The regression model used by the study teamassumed a linear relationship between gasoline lead andblood lead. Did this relationship remain stable as thelevel of gasoline lead changed?To answer this question,the study team took advantage of the fact that, onaverage, gasoline lead levels were about 50 percentlower in the second half of the NHANES II survey thanin the first half. If the relationship between gasoline leadand blood lead is stable and linear, then reestimating theregression model using only data from the second halfof the survey should yield a coefficient for gasoline leadcomparable to that for the entire sample. [The team]found that the coefficients were indeed essentially thesame. In addition, estimation of regression models thatdirectly allowed for the possibility of nonlinear effectssupported the initial findings of a linear relationshipbetween gasoline lead and blood lead.

Is It Likely That Factors Not Included in theAnalysis Could Account for the ObservedRelationship? The study team considered severalfactors that might confound the apparentrelationship between gasoline lead and blood lead:dietary lead intake, exposure to lead paint,seasonality, and sampling patterns.

The basic regression model included nutrient anddemographic variables as proxy measures for theintake of lead in the diet.Yet these variables may notadequately control for a possible downward trend indietary lead that could be causing the estimatedrelationship between gasoline lead and blood lead.Market basket studies conducted by the Food and DrugAdministration over the survey period, however, showedno downward trend in dietary lead intake. Also, lead

intake from drinking water is largely a function ofacidity, which did not change systematically over thesurvey period. Evidence did suggest that changes insolder reduced the content of lead in canned foods overthe period. But the study team was able to rule out thelead content in canned foods as a confounding factorwhen [it] added the lead content in solder as anindependent variable, reestimated the basic regressionmodel, and found that the coefficient of gasoline leadremained essentially unchanged.

The study team recognized changing exposure tolead paint as another potential confounding factor.[it] dismissed the possibility on three grounds.

First, paint lead is a major source in blood leadfor children (who eat paint chips) but not for adults.If declining exposure to lead paint were responsiblefor the estimated relationship between gasoline leadand blood lead, we would expect the reduction inblood lead to be much greater for children than foradults. In fact, the average reduction over the surveyperiod for adults was only slightly smaller than thatfor children (37 percent versus 42 percent).

Second, the ingestion of paint lead usually resultsin large increases in blood lead levels. If reducedexposure to paint lead were responsible for decliningblood lead levels, then we would expect to observe theimprovement primarily in terms of a reduction in thenumber of people with very high blood lead levels. Infact, blood lead levels declined over the survey periodeven for groups with low initial levels.48

Third, declining exposure to paint lead should be amore important factor in central cities than insuburbs because the latter tend to have newerhousing stocks with lower frequencies of peeling leadpaint.Yet the gasoline lead coefficient was essentiallythe same for separate estimations based on centralcity and suburban subsamples.

Blood lead levels in the United States are onaverage higher in the summer than in the winter.Torule out the possibility that seasonal variationconfounds the relationship between gasoline leadand blood lead, the study team reestimated the basicmodel with indicator variables included to allow for

48The study team also used data from the lead-screening program in Chicago to estimate theprobability of toxicity as a function of gasoline lead for children exposed and not exposed to leadpaint. They found that gasoline lead had statistically significant positive coefficients for both groups.



the possibility of independent seasonal effects.Thecoefficients of the seasonal variables were notstatistically significant when gasoline lead was keptin the model.Thus, it appeared that changes ingasoline lead could adequately explain seasonal aswell as long-term changes in blood lead levels.

As already mentioned, the study team estimatedthe basic model on a variety of demographicsubsamples and found no more than a 10 percentdifference across any two estimates of the gasolinelead coefficient. [Team members were] alsoconcerned, however, that changes in NHANES IIsampling locations over the survey period mighthave confounded the estimation of the gasoline leadcoefficient.Therefore, they reestimated the basicmodel with indicator variables for forty-ninelocations and found that the gasoline leadcoefficient changed by only about 5 percent.Further, they found that, even when includingvariables to allow for different gasoline leadcoefficients across locations, the coefficientrepresenting the nationwide effect of gasoline leadwas statistically and substantively significant.Together, these tests led the study team to dismissthe possibility of serious sampling bias.

The Weight of the Evidence

The study team [members] produced a very strongcase in support of an important causal relationshipbetween gasoline lead and blood lead. In many waystheir efforts were exemplary.They drew relevantevidence from a wide variety of sources to supplementtheir primary data analysis.They gave seriousattention to possible confounding factors, consideringboth internal tests (such as subsample analyses andmodel respecifications) and external evidence to see ifthey could be ruled out. As a consequence, opponentsof the proposed policy were left with few openings forattacking its empirical underpinnings.

Finalizing the Rule

The primary task facing the analytical team afterpublication of the proposed rule was to respond to

comments made by interested parties.Team membersparticipated in public hearings held in August andspent much of the fall of 1984 responding tocomments placed in the public docket, which closedon October 1. During this process they became moreconfident that the proposed rule would produce thelarge net benefits they predicated. At the same timethey discovered another benefit category—reductionsin adult blood pressure levels—that could potentiallyswamp their earlier estimates of benefits.

In 1983 Schwartz chanced upon a research articlereporting a correlation between blood lead andhypertension.49 He began work with researchers at theCenters for Disease Control and the University ofMichigan to determine if a relationship existed betweenblood lead and blood pressure levels. By the summer of1984 their analysis of the NHANES II data suggesteda strong link.50 Because high blood pressure contributesto hypertension, myocardial infarctions, and strokes, thepotential benefits from blood lead reductions wereenormous. Although the final rule was ultimately issuedwithout reference to quantitative estimates of thebenefits of lower adult blood lead levels, the teamprovided estimates in the supporting documents.

The one remaining issue was the complianceschedule.The costs of various lead standards wereestimated, using a model of the U.S. refining sectororiginally developed for the Department of Energy.Themodel represents the various types of refiningcapabilities that are available to convert crude oils tofinal petroleum products. It employs an optimizationprocedure for finding the allocations of crude oils andintermediate petroleum products among refining unitsthat maximizes social surplus, the sum of consumerand producer surpluses.This allocation corresponds tothat which would result from a perfectly competitivemarket operating without constraints on the utilizationof available units. Cost was estimated by looking atthe decline in social surplus resulting when the leadconstraint was tightened—for instance, from 1.1 gplgto 0.1 gplg.The manufacturers of lead additiveschallenged these results on the grounds that the model

49V. Batuman, E. Landy, J. K. Maesaka, and R. P. Wedeen, “Contribution of Lead to Hypertension withRenal Impairment,” New England Journal of Medicine, No. 309, 1983, pp. 17–21.50The results of their research were later published in J. L. Pirkle, J. Schwartz, J. R. Landes, andW. R. Harlan, “The Relationship between Blood Lead Levels and Blood Pressure and Its CardiovascularRisk Implications,” American Journal of Epidemiology, Vol. 121, No. 2, 1985, pp. 246–58.



assumed more flexibility in capacity utilization acrossdifferent refineries than was realistic.

The analytical team held meetings with staffersfrom other EPA offices to consider alternativecompliance schedules. Although a tentative decisionwas reached to set an interim standard of 0.5 gplg,to be effective July 1, 1985, and a final standard of0.1 gplg, to be effective January 1, 1986, severalstaffers feared that some refiners would be unable tocomply with their existing equipment. If these fearswere realized, the economic costs of the new rulewould be higher than estimated and perhaps raisepolitical problems.

A consultant to the project,William Johnson ofSobotka and Company, suggested a solution. If thephysical distribution of equipment among refineriesinterfered with the flexibility in petroleum transfersassumed in the model, he reasoned, perhaps asecondary market in lead rights could be created tofacilitate trading to get around specific bottlenecks.Taking the total permitted lead content from July 1,1985 to January 1, 1988 as a constraint, the keywas to create an incentive for refiners who couldmake the least costly reductions in lead additivesbelow the interim 0.5 gplg standard to do so.Theiradditional reductions could then be used to offsetexcess lead in gasoline produced by refiners whocould not easily meet the basic standards withthe equipment they had in place. Because currentreductions below the standard create a right toproduce above the standard some time in the future,the trading process was called “banking of leadrights.” Refiners would be free to buy and sell leadrights at prices that were mutually beneficial. Asa result, the aggregate cost of meeting the newstandard would be reduced.

Representatives from the various EPA officesinvolved with the lead rule agreed that bankingseemed to be a good way to deal with concerns aboutthe compliance schedule. Because it had not been

discussed in the proposed rule published in August,banking could not be part of the final rule.Nevertheless, by moving quickly to propose bankingin a supplemental notice, it would be availableshortly after the new standard became final.51

The remaining task for the team was to prepare theFinal Regulatory Impact Analysis, which would be publishedin support of the final rule.52 The resulting documentbegan by discussing the misfueling and healthproblems associated with lead additives along withalternatives (public education and stepped-up localenforcement to deal specifically with misfueling,pollution charges to deal generally with lead as anegative externality, and other regulatory standards)to the final rule. It then detailed the methods used toestimate the costs of tighter lead standards, the linkbetween gasoline lead and blood lead, the healthbenefits of reducing the exposure of children andadults to lead, the benefits of reducing pollutants otherthan lead, and the benefits from reduced vehiclemaintenance costs and increased fuel economy.

The present value of the net benefits of the finalrule was presented with various assumptions aboutmisfueling (the use of leaded gasoline in vehicles withcatalytic converters).The lower level of permittedlead would reduce the price differential betweenleaded and unleaded gasoline, thereby reducing theeconomic incentive for misfueling. It was not possibleto predict with confidence, however, how muchmisfueling would actually decline.Therefore, thereasonable approach was to consider net benefitsover the range of possibilities. [Table 4] presents theresults of this sensitivity analysis. Note that netbenefits were given both including and excluding theadult blood pressure benefits. Although the bloodpressure benefits appeared huge, they were the last ofthe benefit measures considered and hence had theleast-developed supporting evidence. Nevertheless,even assuming that the standard would produce noreduction in misfueling and no health benefits for

51Fed. Reg. 718 (January 4, 1985); and 50 Fed. Reg. 13116 (April 2, 1985).52Joel Schwartz, Hugh Pitcher, Ronnie Levin, Bart Ostro, and Albert L. Nichols, Costs and Benefitsof Reducing Lead in Gasoline: Final Regulatory Impact Analysis, Publication No. EPA-230-05-85-006(Washington, DC: Office of Policy Analysis, EPA, February, 1985). By the time the final report wasprepared, Jane Leggett left the project team. In the meantime, Albert Nichols, a Harvard Universityprofessor who was visiting the EPA as the acting director of the Economic Analysis Division, beganworking closely with the team to produce the final document.



adults, the present value of benefits appeared to bemore than double the present value of costs. Indeed,it appeared that maintenance benefits alone wouldmore than cover higher refining costs.

The Final Regulatory Impact Analysis was released inFebruary 1985. On March 7, 1985, the final rule waspublished in the Federal Register.53 The 0.1 gplg standardwould take effect on January 1, 1986, almost threeyears after work on the supporting analysis began.

Conclusion

We have described a case where statistical analysismade an important contribution to changing policy. Isthis case typical? Yes and No.You should not expectthat such a confluence of skill, time, data, resources,and interest will often arise to produce such definitiveempirical findings. At the same time, you should

expect to encounter empirical questions that at leastcan be approached, if not confidently answered, withthe sort of statistical methods used by the EPAanalysts. Consider a few prominent examples fromrecent years: Does the death penalty deterhomicide?54 Do higher minimum legal drinking agesand the 55-mile-per-hour speed limit reduce trafficfatalities?55 Do smaller class sizes improve studentperformance?56 Although widely accepted answers tothese empirical questions would not necessarily bedecisive in resolving policy debates, they would atleast move the debates beyond disputes overpredictions to explicit considerations of values. Suchhighly controversial issues aside, you are likely to findthat making empirical inferences and criticallyconsuming those of others often contribute inimportant ways to the quality of your policy analyses.

TABLE 4

Present Values of Costs and Benefits of Final Rule, 1985–1992 (Millions of 1983 Dollars)No Misfueling Full Misfueling Partial Misfueling

Monetized benefits

Children’s health effects 2,582 2,506 2,546Adult blood pressure 27,936 26,743 27,462Conventional pollutants 1,525 0 1,114Maintenance 4,331 3,634 4,077Fuel economy 856 643 788Total monetized benefits 37,231 33,526 35,987Total refining costs 2,637 2,678 2,619Net benefits 34,594 30,847 33,368Net benefits excluding blood pressure 6,658 4,105 5,906

Source: Joel Schwartz et al., Costs and Benefits of Reducing Lead in Gasoline (Washington, DC: EPA, 1985), table VIII-8, p.VIII-26.

5350 Fed. Reg. 9386 (March 7, 1985).54See Isaac Ehrlich, “The Deterrent Effect of Capital Punishment: A Question of Life and Death,”American Economic Review, Vol. 65, No. 3, June 1975, pp. 397–417; and Alfred Blumstein, JacquelineCohen, and Daniel Nagin, eds., Deterrence and Incapacitation: Estimating the Effects of CriminalSanctions on Crime Rates (Washington, DC: National Academy of Sciences, 1978).55See Charles A. Lave, “Speeding, Coordination, and the 55 MPH Limit,” American Economic Review,Vol. 75, No. 5, December 1985, pp. 1159–64; and Peter Asch and David T. Levy, “Does the MinimumDrinking Age Affect Traffic Fatalities?” Journal of Policy Analysis and Management, Vol. 6, No. 2,Winter 1987, pp. 180–92.56See Eric A. Hanushek, “Throwing Money at Schools,” Journal of Policy Analysis and Management,Vol. 1, No. 1, Fall 1981, pp. 19–41.



The Policy Issue Paper

A P P E N D I X

From Public Policy Analysis, Fifth Edition.William N. Dunn. Copyright © 2012 by PearsonEducation, Inc. All rights reserved.



The policy issue paper is a report that presents an analysis of a policy issue. Thekinds of questions addressed in policy issue papers are these: What actual orpotential courses of action are the objects of disagreement among policy

makers and other stakeholders? In what different ways may the problem be defined?What is the scope and severity of the problem? If nothing is done, is the problemlikely to get worse in future months or years? What goals and objectives should bepursued to solve the problem? How can the degree of success in achieving objectivesbe measured? What activities are now under way to resolve the problem? What newpolicy alternatives should be considered? Which are preferable and why?

In answering these questions, the analyst needs to acquire knowledge and skillsin developing structured written narratives and communicating them in policypresentations and other briefings. Only recently have such knowledge and skillsbeen included in instructional programs in public policy analysis. Policy analysismay be applied or basic.1 But the mission of policy analysis is mainly applied.

FOCUS AND FORMS OF THE POLICY ISSUE PAPERThe policy issue paper may focus on problems at one or more levels of government.Global warming and air pollution, for example, are international, national, and local inscope. The issue paper may take a number of specific forms, depending on the audienceand the particular issue at hand. Thus, issue papers may be presented in the form of

A P P E N D I X

1The terms basic and applied are used for convenience only. It is widely acknowledged that these twoorientations toward science overlap in practice. Indeed, there is much evidence to support the conclu-sion that many of the most important theoretical, methodological, and substantive advances in the basicsocial sciences originated in applied work on practical problems and conflicts. See, especially, Karl W.Deutsch, Andrea S. Markovits, and John Platt, Advances in the Social Sciences, 1900–1980: What,Who, Where, How? (Lanham, MD: University Press of America and Abt Books, 1986).



“staff reports,” “briefing papers,” “options papers,” or “white papers.” An illustrativelist of issues that may be addressed in a policy issue paper follows:

� Which of several alternative contracts should be accepted by a unionbargaining team?

� Should the mayor increase expenditures on road maintenance?� Should the city manager install a computerized management information

system?� Which public transportation plan should the mayor submit for federal funding?� Should a state agency establish a special office to recruit minorities and women

for civil service positions?� Should a citizens’ group support environmental protection legislation now

before Congress?� Should the governor veto a tax bill passed by the state legislature?� Should an agency director support a plan for flexible working hours (flextime)?� Should a legislator support a bill restricting the sale of hand guns?� Should the president withhold foreign aid from countries that violate human

rights?� Should the United Nations General Assembly condemn the violation of human

rights in a particular country?� Should the United States withdraw from Afghanistan?

ELEMENTS OF THE POLICY ISSUE PAPERA policy issue paper should be as complete as time and available informationpermit. An issue paper should

explore the problem at a depth sufficient to give the reader a good idea of itsdimensions and the possible scope of the solution, so that it might be possiblefor a decision maker to conclude either to do nothing further or to commissiona definitive study looking toward some action recommendation.2

In this author’s experience, most issue papers deal primarily with the formulationof a problem and possible solutions. Only rarely does the issue paper reachdefinitive conclusions or recommendations. Although an issue paper may containrecommendations and outline plans for monitoring and evaluating policy outcomes,it is essentially the first phase of an in-depth policy analysis that may be undertakenat a later time.

In preparing an issue paper, the analyst should be reasonably sure that all majorquestions have been addressed. Although issue papers will vary with the nature ofthe problem being investigated, most issue papers contain a number of standardelements.3 These elements have been organized around the framework for policyanalysis presented in the text.

2Edgar S. Quade, Analysis for Public Decisions (New York: American Elsevier Publishing, 1975), p. 69.3Compare Ibid., pp. 68–82; and Harry Hatry and others, Program Analysis for State and LocalGovernments (Washington, DC: The Urban Institute, 1976), Appendix B, pp. 139–43.



Observe that each element of the issue paper requires a different policy-analyticprocedure to produce and transform information. A policy issue paper, however, has onemajor characteristic not shared by integrated policy analysis. The issue paper is essentiallya prospective (ex ante) investigation that begins with limited information about pastpolicy outcomes and performance and ends with as much information as possible aboutthe nature of policy problems, expected policy outcomes, and preferred policies.

A CHECKLISTThis checklist is designed as a practical guide to the preparation and self-evaluation ofpolicy issue papers. Items in the checklist are based on the format for preparing policyissue papers presented earlier in this appendix. The checklist operationalizes this tem-plate, provides examples, and contains a rating scale for evaluating specific elements ofthe paper. Remember, however, that some elements in the checklist may not be appropri-ate for each and every paper. The checklist should therefore be used in a flexible manner.

Element Policy-Analytic

Letter of Transmittal

Executive Summary

I. Background of the problem MONITORINGA. Describe client’s inquiryB. Overview of problem situationC. Describe prior efforts to solve problem

II. Significance of the problem EVALUATIONA. Evaluate past policy performanceB. Assess the scope and severity of problemC. Determine the need for analysis

III. Problem statement PROBLEM STRUCTURINGA. Diagnose problemB. Describe major stakeholdersC. Define goals and objectives

IV. Analysis of alternatives FORECASTINGA. Describe alternativesB. Forecast consequences of alternativesC. Describe any spillovers and externalitiesD. Assess constraints and political feasibility

V. Conclusions and recommendations PRESCRIPTIONA. Select criteria or decision rulesB. State conclusions and recommendationsC. Describe preferred alternative(s)D. Outline implementation strategyE. Summarize plan for monitoring and evaluationF. List limitations and unanticipated consequences

ReferencesAppendices



Element of Policy Issue Paper Example/Note Rating

Letter of transmittal1. Are all relevant stakeholders

expected to act on the paperaddressed?

A paper on local law enforcement might beaddressed to the chief of police, the mayor, andthe director of public safety.

[ ]

2. Does the letter describe allattached materials?

“Enclosed please find a policy issue paper onoptions to increase the productivity of municipalemployees. An executive summary of this paperis also included, along with relevant statisticalmaterials appended to the text.”

[ ]

3. Does the letter specify who isexpected to take action, how,and when?

“We look forward to your reply well in advanceof the meeting scheduled for June 15.”

[ ]

Executive summary4. Are all elements of the issue

paper described in theexecutive summary?

Every element of the issue paper (Source andBackground of the Problem, the PolicyProblem, and so on) should be included in thesummary.

[ ]

5. Is the summary clear, concise,and specific?

“This paper reviews problems of air pollution overthe past ten years, showing that the level ofindustrial pollutants has increased by more than200 percent in the 2000–2009 period.”

[ ]

6. Is the summaryunderstandable to all who willread it?

The point here is to avoid unnecessary jargon,long sentences, and complex arguments asmuch as possible.

[ ]

7. Are recommendationsappropriately highlighted in thesummary?

“In conclusion, we recommend that funds in therange of $15,000–$20,000 be allocated toconduct an assessment of recreational needs inthe Oakland area.”

[ ]

I. Background of the problem

8. Are all dimensions of the problemsituation described?

Increasing crime rates may be analyzed inconjunction with data on unemployment, policerecruitment, citizen surveys, migration, and so on.

[ ]

(continued)

Box A1 Checklist for Evaluating a Policy Issue Paper

Rating Scale

1 � totally adequate2 � adequate3 � inadequate4 � totally inadequate0 � not applicable




9. Have outcomes of priorefforts to resolve problems inthe area been described?

“Similar training programs have been implementedin New York, Florida, and California, at a cost of$4,000–$5,000 per trainee.”

[ ]

10. Is there a clear assessment ofpast policy performance?

“Training programs in New York and Californiaresulted in. a 1 percent reduction in hard-coreunemployment and were judged to be highlysuccessful.”

[ ]

II. Scope and severity of problem

11. Is the scope and severity ofthe problem situation clearlydescribed?

“Problems of industrial pollution are moresevere today than at any time in our historyand affect the physical well-being of 60percent of the population. The resolution ofthese problems is an urgent matter that callsfor timely and decisive action.”

[ ]

III. Problem statement

12. Is the problem clearlystated?

“The problem addressed in this paper is how bestto satisfy the need for increased fiscal accoun-tability among local governments in the region.”

[ ]

13. Is the issue clearly stated? “The issue addressed in this paper is whether thestate’s Department of Community Affairs shouldincrease its technical assistance to localgovernments in the area of financial management.”

[ ]

14. Is the approach to analysisclearly specified?

“In addressing this issue we have employed cost-effectiveness analysis as a way to determinethe levels of investment necessary to train10,000 local officials in cash managementtechniques.”

[ ]

15. Are all major stakeholdersidentified and prioritized?

The task here is to identify all persons and groupswho significantly affect and are significantlyaffected by the formulation and implementationof policies. Stakeholders who are only marginallyinfluential (or affected) may be dropped from theanalysis, but this requires prioritization.

[ ]

16. Are goals and objectivesclearly specified?

“The goal of this policy is to improve thephysical security of the community.The objective is to reduce reported crimes by 10 percent or more in the 2005–10 period.”

[ ]

17. Are measures of effectivenessclearly specified?

Here there are many choices: benefit-cost ratios,net benefits, distributional benefits, and so on.

[ ]

18. Are all sets of potentialsolutions outlined?

“Any effort to reduce crime in the area shouldtake into account the effectiveness of lawenforcement programs, the unemploymentrate, population density, and urbanization.”

[ ]




IV. Policy alternatives

19. Are alternative solutionsspecified?

“Three policy alternatives are analyzed in thispaper: educational programs to alert citizensto the role they can play in crime control,policy training programs, and advanced crimecontrol technology.”

[ ]

20. Are alternatives systematicallycompared in terms of theirprobable costs and effectiveness?

“Program A has a benefit-cost ratio that istwice that of program B.”

[ ]

21. Are relevant spillovers andexternalities included in theanalysis of alternatives?

“The early childhood educational program is notonly cost-effective. It will also producespillovers in the form of increasedparticipation of family members in the school.At the same time there are importantexternalities in the form of increased costs fortransportation to and from school.”

[ ]

22. Are all relevant constraintstaken into account?

These may be financial (e.g., fixed costs), legal(e.g., laws proscribing certain actions), orpolitical (e.g., lack of political support).

[ ]

23. Have alternatives beensystematically compared interms of political feasibility?

The point here is to make informed judgments about the probable support, relevance, andinfluence of key stakeholders in gaining acceptancefor and implementing each alternative.

[ ]

V. Policy recommendations

24. Are all relevant criteria forprescribing alternativesclearly specified?

Criteria may include net welfare improvement,distributional improvement, cost effectiveness,and so on.

[ ]

25. Has the preferred alternativebeen clearly described?

“We therefore recommend that the secondalternative be adopted and implemented, sinceit is likely to be more cost-effective and ispolitically feasible under present conditions.”

[ ]

26. Is a strategy forimplementation outlined?

“The proposed policy should be implemented bythe Department of Labor and stateemployment offices. Implementation should bephased over a period of three years and focuson cities with more than 50,000 persons.”

[ ]

27. Are provisions made formonitoring and evaluatingpolicies?

“The state employment offices should establishspecial units to engage in process and outcomeevaluations, at intervals of 30 days. Results ofmonitoring and evaluation should be stored ina management information system, andreports should be filed quarterly.”

[ ]

(continued)




28. Are limitations and possibleunintended consequencestaken into account?

Here the point is to specify how limitations of theanalysis affect the confidence one has in theconclusions. Possible unintended consequences(e.g., abuses of social welfare programs, nuclearmeltdowns, and so on) should also be thoughtthrough.

[ ]

References

29. Are references cited in thetext included in theappropriate form?

It is preferable to use the “name–date”bibliographic form in a policy issue paper. Only thename of the author, the date of publication, andpage numbers will be included in the text, with nofootnotes at the bottom of the page. For example:(Jones, 2010: 20–23).The citation in thereferences will read: Jones, John (2010). A Report

on the Fiscal Crisis. Washington, DC: Fiscal Institute.

[ ]

Appendices

30. Is relevant supportinginformation (statistics,legislation, documents,correspondence) included?

All statistical tables should be properly numberedand headed and the source of data specified at thebottom of the table. Legislation, documents, andcorrespondence should be identified as to itssource, date, author(s), and so on.

[ ]


The PolicyMemorandum

A P P E N D I X

Policy issue papers require many months or years of research and analysis.Policy memoranda, by contrast, are prepared over a much shorter period oftime—often no more than one month, but sometimes a matter of weeks or

days. Policy memoranda are often based on syntheses of one or many issue papers,studies, or reports. The differences between the issue paper, on one hand, and thepolicy memorandum, on the other, are illustrated by contrasts between basic andapplied policy analysis.

Two Kinds of Policy AnalysisCharacteristic Basic Applied

Origin of problems • University community • Policy communityTypical methods • Quantitative modeling • Sound argumentationType of research • Original data collection • Synthesis of existing dataPrimary aim • Improve theory • Improve practiceCommunications media • Article or book • Policy memo or issue paperSource of incentives • University departments • Government departments

The policy memo (short for memorandum) is a structured written narrativeused in public, nonprofit, and business organizations. Policy memos are brief, wellorganized, and understandable to nonspecialists. In some cases, they includepolicy recommendations (but only if requested). The clients for policy memos maybe appointed or elected officials; citizens groups; or representatives of nonprofit,nongovernmental, and business organizations.



Policy memos take different forms, depending on the agency or organization. Forexample, some agencies, departments, or ministries have their own letterheadstationery that is preformatted, whereas others use word-processing templates.Despite these differences, there are several common elements. Most memos displaythe date, the name of the recipient, the name of the analyst, and the subject of thememo on separate lines at the top of the first page. Because memos are designed tocommunicate with busy policy makers, they should be brief (most are no more thanthree to five pages in length), clearly organized with appropriate headings, andunderstandably written with as little technical jargon as possible. The content of thememo should be concisely communicated on the subject line. Technical and analyticmaterial (tables, graphs, charts, legal documents, etc.) that supports the conclusionsand recommendations should be appended to the memo as an attachment or kept onhand to respond to questions that may be raised later.

Here are a few guidelines for writing an executive summary:

� The introductory paragraph should state the purpose of the memo and thequestion or issue to which the memo is a response.

� There may be a brief description of the methods used in doing the analysis.But remember that busy readers usually do not care very much about yourmethods (this does not mean that you will not have to justify your methodsto someone else later).

� As appropriate, longer memos with more than three pages should use headers,bullets, underlining, italics, and other stylistic devices to highlight material thatis especially important.

� Normally, the memo should end with information about where you can bereached for further questions (e.g., telephone, fax, e-mail).

Following are examples of a cross-section of policy memos.

POLICY MEMO

TO: The President-Elect’s National Security TransitionFROM: David Shorr, The Stanley Foundation

Vikram Singh, Center for a New American SecurityRE: A Unified International Affairs and National Security Budget

to Increase American Effectiveness Worldwide

The Center for a New American Security (CNAS) and Stanley Foundation recentlyconcluded an eight-month initiative focused on the anemic condition of theUnited States’ civilian international affairs agencies. There is nearly universalsupport among national security professionals for serious action to remedy thiscondition. Defense Secretary Gates continues to call for more resources for the civil-ian elements of national power and the Project on National Security Reform isexpected to recommend a doubling of State Department and United States Agencyfor International Development (USAID) funding and a minimum manpowerincrease of 50%. Yet this widely recognized need for stronger civilian agencies hasnot been paired with the political will to mandate and underwrite them. 94% of

The Policy Memorandum



supplemental funding for the wars in Iraq and Afghanistan has been for theDepartment of Defense.

The essence of the problem is inadequate financial and human resources tosustain US political influence in a fast-changing world. Just when US relations withthe rest of the world are in a deep slump, our capacity to turn things around remainsat historic lows. America needs foreign policy infrastructure investment, and notjust for special initiatives or boutique programs.

The complete findings of the CNAS � Stanley project are in four reportsavailable at http://www.stanleyfoundation.org/articles.cfm?ID�492. This memo,however, proposes a single major policy initiative to give immediate impetus and laythe foundation for the rebuilding of America’s international affairs capabilities. ThePresident-elect should require his new national security and budget teams to preparea joint FY2010 international affairs and national security budget.

The Obama Administration will, and should, consider specific measures toredress organizational weaknesses in the US government’s foreign policy apparatus,particularly post-conflict reconstruction, long-term economic development, andpublic diplomacy. It is critical to recognize, though, that these are all symptoms of asystemic civilian capacity deficit. Since this is a matter of the overall effectiveness of USforeign policy, the push for budgeting across interagency stovepipes must come fromthe top. In other words, whatever other reforms are undertaken, a presidentialmandate for a joint FY2010 budget is vitally important to initiate the process ofrebuilding organizational infrastructure.

The joint budget for international affairs and national security— starting with thecombining of the 050 and 150 budget functions for FY2010 and expanding in subse-quent years—would at first be a tool to force cross-department balancing. Withoutmandating changes in normal budget processes within agencies, an integrated WhiteHouse review will promote collaboration across overlapping budget areas. It will drive:

� Substantial investment in strengthening the State Department and USAID,to bolster their overstretched headquarters bureaus and overseas missions.

� Significant growth in their work forces.� Steady balancing of the relative increases in defense and international affairs

spending seen in recent years.

Detailed budget proposals and implementation plans are available from other initia-tives and organizations, such as the Foreign Affairs Budget for the Future reportfrom the American Academy of Diplomacy and the Stimson Center. We suggest thata combined international affairs and national security budget is the only lever thatcan spur more-than-marginal added investment.

This initiative should not be contingent upon, nor await, other reforms to theforeign policy apparatus. A number of related mechanisms will be needed toimplement this budgeting discipline, and we urge that they be incorporated into thestructures that the new administration is establishing for national security planningand decision making:

� As these joint budgets are prepared, presented, and enacted, they will needsubstantial White House staff support from the NSC and OMB. Implementing


such a joint budget will require a single dual-hatted NSC and OMB Directorfor international affairs and national security and integrated reviews of allagency budget submissions.

� Intensive cooperation will be needed among DoD, DoS, and USAID budgetoffices to help their principals fulfill the president’s mandate.

� The budget must be treated as a single package from the beginning to the endof the process. The principals on the president’s national security team musthew to shared positions all the way through to the final appropriations bills.

� While reforms of the congressional budget process may ultimately be needed,the administration should meanwhile consult as widely as possible with thebudget, authorizing, and appropriating committees. Congress will likely splitup such a combined budget upon receipt, but the cross-leveling achieved bythis new White House process will still be reflected in the budget bills forthe agencies.

Despite being a global power, the United States does not really seem to have itsfinger on the global pulse. In an era of stakeholder proliferation, every internationalproblem confronting the United States involves not just national governments but arange of actors from the private sector and powerful NGOs to criminal and terroristnetworks. To have any chance of shaping world events, Washington must be able toengage this panoply of actors around the globe.

Having drawn down US civilian capabilities faster than defense in the 1990sand increased them more slowly than defense since 2001, the American governmenthas crippled its ability to have insight into, and relationships with, the actors onwhom future peace and prosperity hinge. Just as the United States needs to invest ineducation and science to ensure that the American work force can compete andthrive in the globalizing world, it must likewise transform our government to becompetitive in the effort to sustain America’s global power in the 21st century.

Source: www.stanleyfoundation.org/articles.

August 28, 1986

The Honorable Butler DerrickChairman, Task Force on the Budget ProcessCommittee on the BudgetHouse of Representatives

Dear Mr. Chairman:

In response to your letter of May 8, 1985, we have examined the potential impactsof immigration on the federal budget over the next 10 years. This effort wasundertaken with dual objectives: obtaining the best possible estimate of budgetaryimpacts despite known data inadequacies and beginning the development of amethodology for dealing with long-term, crosscutting issues of budgetary signifi-cance, of which immigration is a prime example. On May 1, 1986, we presented abriefing to you and other task force members and staff on the results of our study.




This report is a written version of that briefing. We have also included additionaldetail on the budgetary impact of immigration on state and local governments.

Our study established fiscal year 1984 as a baseline. We identified, and wherenecessary estimated, major federal immigrant-related outlays and revenues for thatperiod. We then projected these data to yield 1990 and 1995 estimates under threedifferent sets of assumptions (scenarios) as to the future social, economic, andpolitical environment. These projections are not intended as forecasts but weredeveloped to provide illustrative ranges. We also sought similar data for selectedprograms and localities in the five most impacted states, for illustrative purposes, butwe made no projections at this level.

Although there are constraints and limitations to our analysis, it demonstrates thatsignificant budgetary impacts of immigration exist at the federal level, in terms ofboth outlays and revenues. Dollar amounts can be assigned to such impacts only ifmajor assumptions are made concerning immigrant participation in major socialand income security programs. This is due to the uncertainties of population sizeand characteristics in 1984, as well as to the lack of direct information regardingimmigrant-related outlays and revenues, data gaps which are even more apparent inthe out year projections. We assumed, therefore, that these participation rates wouldbe comparable, on the average, to those of U.S. citizens. We could not test thishypothesis, and expert opinion varies.

Using these assumptions, we found that for fiscal year 1984 in the programs weexamined

—per capita federal outlays for immigrants were roughly comparable to those foraverage U.S. residents;

—per capita, immigrants contributed fewer revenues to the federal government thanaverage U.S. residents; and

—total immigrant-related outlays differed only slightly from total immigrant-relatedrevenues.

Under all three scenarios examined, these observations remain valid for 1990 and1995, although uncertainty increases.

Regarding the broader methodological interests which it was designed to explore,this study of a crosscutting issue has both value and limitations. While specific totaldollar amounts cannot be assigned on any supportable basis, minimum outlays andrevenues can be identified, ranges can be estimated, and comparisons can be drawnbased on explicit assumptions. Because both federal outlays and revenues are signif-icantly affected by immigration and remain roughly in balance, the committee maywish to monitor the key conditions and trends to determine whether this balance isbeing maintained. Thus, we believe we have demonstrated the feasibility of thisapproach, but its usefulness will vary with the circumstances.

I would be pleased to discuss this information with you further at your convenience. Asagreed with your office, we have not obtained agency comments on this report. We


are sending copies of this report to interested parties, and copies will be made avail-able to others upon request.

If you or your staff have any questions on this report, please call me at 275-9455.

Sincerely yours,

Kenneth W. HunterAssistant to the Director

Source: Budget Issues: Immigration to the United States—Federal Budget Impacts,1984–1995 (Washington, DC: U.S. General Accounting Office, August 1986).

Another type of memo is the briefing memo that summarizes the results of anintelligence analysis. Following is the briefing memo on Osama Bin Laden’s plansfor the attack on the World Trade Center on September 11, 2001. This briefingmemo was presented to President George W. Bush on August 6, 2001.

DECLASSIFIED PRESIDENTIAL DAILY BRIEFING FOR 6 AUG 01For the President Only6 August 2001

Bin Laden Determined to Strike in US

Clandestine, foreign government, and media reports indicate Bin Laden since 1997has wanted to conduct terrorist attacks in the US. Bin Laden implied in US televisioninterviews in 1997 and 1998 that his followers would follow the example of WorldTrade Center bomber Ramzi Yousef and “bring the fighting to America.”

After US missile strikes on his base in Afghanistan in 1998, Bin Laden told followershe wanted to retaliate in Washington, according to a (redacted) service.

An Egyptian Islamic Jihad (EIJ) operative told a (redacted) service at the same timethat Bin Laden was planning to exploit the operative’s access to the US to mount aterrorist strike.

The millennium plotting in Canada in 1999 may have been part of Bin Laden’s firstserious attempt to implement a terrorist strike in the US. Convicted plotter AhmedRessam has told the FBI that he conceived the idea to attack Los AngelesInternational Airport himself, but that Bin Laden lieutenant Abu Zubaydah encour-aged him and helped facilitate the operation. Ressam also said that in 1998 AbuZubaydah was planning his own US attack.




Ressam said Bin Laden was aware of the Los Angeles operation.

Although Bin Laden has not succeeded, his attacks against the US Embassies inKenya and Tanzania in 1998 demonstrate that he prepares operations years inadvance and is not deterred by setbacks. Bin Laden associates surveilled ourEmbassies in Nairobi and Dar es Salaam as early as 1993, and some members of theNairobi cell planning the bombings were arrested and deported in 1997.

Al-Qa’ida members—including some who are US citizens—have resided in or trav-eled to the US for years, and the group apparently maintains a support structure thatcould aid attacks. Two al-Qa’ida members found guilty in the conspiracy to bombour Embassies in East Africa were US citizens, and a senior EIJ member lived inCalifornia in the mid-1990s.

A clandestine source said in 1998 that a Bin Laden cell in New York was recruitingMuslim-American youth for attacks.

We have not been able to corroborate some of the more sensational threat reporting,such as that from a (redacted) service in 1998 saying the Bin Laden wanted to hijacka US aircraft to gain the release of “Blind Shaykh” ’Umar ’Abd al-Rahman and otherUS-held extremists.

Nevertheless, FBI information since that time indicates patterns of suspiciousactivity in this country consistent with preparations for hijackings or other types ofattacks, including recent surveillance of federal buildings in New York.

The FBI is conducting approximately 70 full field investigations throughout the USthat it considers Bin Laden–related. CIA and the FBI are investigating a call to ourEmbassy in the UAE in May saying that a group of Bin Laden supporters was in theUS planning attacks with explosives.

Source: http://www.cnn.com/2004/images/04/10/whitehouse.pdf



Planning Oral Briefings

A P P E N D I X




A P P E N D I X

Written documents are only one means for communicating the results ofpolicy analysis. The other is the oral briefing. Although the oral briefingor presentation has most of the same elements as the issue paper, the

approach to planning and delivering an oral presentation is quite different.The elements of an oral briefing typically include the following:

� Opening and greeting to participants� Background of the briefing� Major findings of the analysis� Approach and methods� Data used as basis for the analysis� Recommendations� Questions from participants� Closing

An essential aspect of oral communications is knowing the audience. Here a se-ries of important questions should be asked prior to the briefing: How large is theaudience? How many members of the audience are experts on the problem you areaddressing in the briefing? What percent of the audience understands your researchmethods? Are you credible to members of the audience? Does the audience preferdetailed or general information? Where does the briefing fall in the policy-makingprocess? At the agenda setting or formulation and adoption phases? Or the imple-mentation and evaluation phases?

Answers to these questions govern the choice of a communications strategyappropriate for a particular audience. For example, if the audience is a medium-sized



group with twenty to twenty-five persons, the diversity of the audience calls for thesespecific communications strategies:1

� Circulate background materials before the meeting so that everyone willstart from a common base.

� Tell the group that the strategy selected is designed to meet the objectivesof the group as a whole but that it may not satisfy everyone’s needs.

� Focus on the agendas of key policy makers, even if you lose the rest ofthe audience.

Another common problem in giving oral briefings is that policy makers maybring their own staff experts to the briefing. In this case, a high percentage of theaudience has expertise in the area you are addressing. Here the following strategiesare appropriate:

� Avoid engaging in a “knowledge contest” with experts by focusing on the purpose of your own presentation.

� Capitalize on the group’s expertise by focusing on evidence bearing on findingsand recommendations.

� Avoid a long presentation of background material and lengthy descriptionof methods.

� Attempt to generate dialogue and a productive debate on potential solutionsfor the problem.

An equally important problem in oral briefings is understanding individualmembers of the group. For example, it is important to know if the immediate clientsfor policy analysis are

� Experts in the problem area� Familiar with your analytic methods� Influential participants in the policy-making process� Oriented toward detailed or broad information� Faced with high, medium, or low stakes in the problem� Politically important to the success of your recommendations

Finally, oral briefings employ various kinds of graphic displays. One of theseis the transparency sheet used with an overhead projector (also known as a“beamer”). In using overhead transparencies or computer graphics, it is importantto recognize that they are not appropriate if you want an informal style ofpresentation—they tend to make briefings formal, making open dialoguedifficult. Alternatively, you may want to limit discussion, and strict adherence totransparencies helps.

It is also important to observe the following guidelines in using transparenciesand slides:

� Keep transparencies and slides simple, with condensed and highlighted text.� Use clear, bold, uncluttered letters and lines of text.

1These guidelines are based on Presentation Planner (Pitsford, NY: Discus Electronic Training, EastmanTechnology, 1991).


� Highlight important points and use a few different colors (but no morethan three).

� Limit the text on any page to a maximum of ten lines.� Be sure the width of the screen is at least one-sixth of the distance between

the screen and the farthest viewer.� Make sure that the nearest viewer is a minimum of two screen widths from

the screen.� Remain on the same transparency or slide for a minimum of two minutes

to give viewers time to read plus an additional thirty seconds to study thecontent.

� Leave the room lights on—overheads are not films.� Turn off the overhead projector when you want to redirect attention back

to you and reestablish eye contact.

These are but a few of the strategies and tactics that are appropriate for differentaudiences and communications media. Given the goal of selecting communicationsmedia and products that match the characteristics of the audience, there is today arich body of theory and techniques to guide the visual display of information.2

Appropriate computer graphics, presentation packages, and statistical programs arenow widely available. These include the Statistical Package for the Social Sciences(SPSS), Microsoft Excel, Microsoft PowerPoint, Stata, and R.

Many types of visual displays are available for communicating complex ideas.Examples of these visual displays follow:

� Bar charts� Pie charts� Frequency histograms� Relative frequency histograms� Ogives� Scatter plots� Time-series graphs� Graphs with data tables� Decision trees and other diagrams

In most policy presentations, a key visual display is one that summarizesrelationships between two or more alternatives and their respective impacts. The options–impacts matrix (Goeller matrix), which is a rectangular array of policyalternatives and their anticipated impacts, is well suited for this purpose. The matrixprovides a concise visual map of the range of impacts associated with alternatives.The preferred impacts may be shaded, so that participants in a presentation caneasily examine the pattern of trade-offs between impacts before they have beenrepresented quantitatively in the form of coefficients, indexes, or ratios (e.g., abenefit-cost ratio), which can obscure issues and prevent informed discussion.


2Anyone who wishes to achieve competency in addressing these questions should read Edward R.Tufte’s award-winning trilogy: The Visual Display of Quantitative Information (Cheshire, CT: GraphicsPress, 1983); Envisioning Information (Cheshire, CT: Graphics Press, 1990); and Visual Explanations:Images and Quantities, Evidence and Narrative (Cheshire, CT: Graphics Press, 1997).


22.6Billion

20.2Billion

55 MPH 65 MPH

TRAVEL TIMEIN HOURS

81.6Billion

87.1Billion

GALLONS GASOLINECONSUMED

1 %Decrease

3 %Increase

CHANGE IN AIRPOLLUTION

45,200 54,100TRAFFICFATALITIES

1,800,000 2,000,000INJURIES

70 % 30 %PUBLIC SUPPORT

POLICY ALTERNATIVESPOLICY IMPACT

Options–Impacts Matrix

NOTE: Shaded boxes are preferred impacts.

0.43

Clinton

0.37

Bush

Bar Chart Display of Results of 1992 Presidential Election

0.20

Perot



43.00 %

Clinton

37.00 %

Bush

Pie Chart Display of Results of 1992 Presidential Election

20.00 %

Perot

120.00

17 22 27 32 37 42

Age

47 52 57 62 67

5.00

10.00

15.00

Freq

uenc

y

20.00

25.00

Frequency Histogram Displaying Ages of Women at Time of Divorce(N � 100)



120.00

17 22 27 32 37 42Age

47 52 57 62 67

0.05

0.10

0.15

Rel

ativ

e Fr

eque

ncy

0.20

0.25

Relative Frequency Histogram Displaying Ages of Women at Timeof Divorce (N = 100)

00.00

5 10 15 20 25 30 35Age

40 45 50 55 60 65 70

0.10

0.20

0.30

0.40

0.50

0.60

Cum

ulat

ive

Rel

ativ

e Fr

eque

ncy

0.70

0.80

0.90

1.00Ogive Displaying Age of Women at Time of Divorce (N � 100)

Note: Q and P are quartiles and percentiles.

Q1 (P25)

Q1 (P75)

Q2 (P60) � Median



00

2 4 6 8 10X AXIS

12 14 16 18 20

2

4

6

8

10

Y A

XIS

12

14

16

18

20

Scatterplot Displaying Pattern of Observations around RegressionLine

650

66 67 68 69 70 71 72 73 74 75 76 77

Years

Interrupted Time Series Graph Displaying Effects of the 55 mphSpeed Limit

78 79 80 81 82 83 84 85 86 87 88 89 90

1

2

3

Dea

ths

Per

100

Mill

ion

Mile

s

4

5

6

POLICYINTERVENTION



120.00

17 22 27 32 37 42

Age

Ages

47 52 57 62 67

2.00 3.00 6.00 10.00 16.00 21.00 18.00 11.00 9.00 4.00

5.00

10.00

15.00

Freq

uenc

y

20.00

25.00

Frequency Histogram Displaying Ages of Women at Time ofDivorce with Data Table Attached (N � 100)




Index

IndexPage references followed by "f" indicate illustratedfigures or photographs; followed by "t" indicates atable.

22008 election, 119

AAbortion, 43

right to, 43Access, 12, 31, 108, 178, 217, 438accessibility, 122Accountability, 33, 55, 148, 314, 386, 430Administration, 11, 14, 24, 32, 34, 37-39, 43, 50,

54-56, 60, 119, 148-149, 156, 184, 215-216,229-230, 244-245, 250-251, 259, 269,307-308, 315, 317, 348-349, 353, 374,378-379, 389, 400, 420, 435-436

defined, 14, 215advertising, 244

campaign, 244Advice and consent, 40, 59affirmative action, 43, 258Afghanistan, 73, 427, 435, 438age, 34-36, 55, 82, 92-93, 107-108, 218, 238, 255,

258, 265, 274-275, 278-279, 305, 308, 319,327, 408, 413, 418, 423, 446-447, 449

discrimination, 238drinking, 423unemployment and, 238

Agency for International Development, 434USAID, 434

Agenda setting, 42-44, 111, 385, 393, 442Agendas, 44, 48, 51-52, 54, 59, 386, 393, 443Aggregates, 94, 177Agriculture, 38, 108, 267aid, 43, 55, 80, 84, 94, 120, 191, 304, 389-390, 427,

439foreign, 191, 389-390, 427

Aid to Families with Dependent Children, 43, 304AFDC, 43, 304

Air pollution, 23, 70, 258-259, 280, 285, 296-297, 389,426, 429, 445

air quality, 71-72, 123, 248, 252, 282Alabama, 60, 148, 229Albania, 378Alienation, 149, 173, 205, 259, 303Alinsky, Saul, 318Allen, George, 8Amendments, 18, 407

process, 18, 407America, 14, 109, 177, 214, 257, 260, 389, 426,

435-436, 438American Society for Public Administration, 14, 315,

317Americans, 54, 67, 184Analogies, 89, 98-99, 118, 123, 270, 347, 366Appeal, 113, 123, 350, 375Appeals, 20, 35, 314Appointment, 130Appropriations, 436Appropriations bills, 436Aristotle, 33-34, 363arms control, 109arms race, 173Army, 38, 128, 371, 377, 380assembly, 37, 111, 390, 427Auditor, 334Austerity, 178Austin, 355Australia, 18Authoritarian, 354-355Authoritarian regime, 354Authority, 18-20, 35, 88, 118, 123, 166, 168, 181, 184,

187, 342, 346-350, 354, 395, 407

BBalkans, 374, 376, 378Bardach, Eugene, 390bargaining, 20, 73-74, 96-97, 206, 389, 427Beliefs, 52, 66, 78, 80, 112, 122-123, 147, 180, 223,

318, 344, 375, 418Bell, Daniel, 60, 257Berlin Wall, 119bias, 54, 213, 345, 415, 421Biden, Joseph, 381Bills, 436

appropriations, 436boards, 251, 317

municipal, 251Borders, 354, 378borrowing, 234-235Bosnia, 376, 378Bosnia-Herzegovina, 378Boston Globe, 349Boundaries, 32, 77, 79, 87-88, 90-91, 211, 214-215Britain, 405-406Brookings Institution, 40, 62, 139, 204, 209, 241,

248-249Budget, 9, 42, 81, 119, 178-179, 181-182, 184-185,

199, 230, 235, 244, 248, 251, 257, 259, 410,434-436, 438

defense, 9, 434-436preparation, 410

Budget deficit, 119budgets, 229, 435

balancing, 435deficit, 435

Bulgaria, 378Bureau of Indian Affairs, 38Bureau of Labor Statistics, 55, 124, 249, 281Bureau of the Census, 55, 93, 100, 124, 249, 275-276,

304bureaucracies, 52, 71, 149-152Bureaucracy, 15, 149-151, 355, 372

administration, 149American, 355and public policy, 355constraints, 15control, 149-151foreign policy, 355hierarchy, 149problems of, 355structure, 150-151structure of, 150-151understanding, 355, 372

Bureaucrats, 57Business, 28-29, 57, 60-61, 77, 80, 138, 238, 251,

260, 357, 364, 400, 433French, 138

Business cycles, 138busing, 313

CCalifornia, 15, 123, 312, 328, 390, 430, 439campaigns, 289Campbell, Angus, 257, 259Canada, 165, 258, 438Candidates, 336capital punishment, 423capitalism, 46, 71Causality, 13, 153, 306, 347, 360, 374, 418census, 38, 55, 93, 100-101, 103, 106, 110, 124, 192,

249, 253, 255, 275-276, 304, 377Census Bureau, 101Centers for Disease Control, 409-411, 418-419, 421Central cities, 69, 175, 420Central Intelligence Agency, 38centralization, 108, 149, 229, 254Chain of command, 108Chapter, 1-2, 8-9, 18, 21, 25, 31, 58, 61, 65-66, 90,

106, 108-109, 116, 117, 128, 153, 155, 167,

176-177, 179-181, 189-190, 216, 221, 223,240, 247, 257, 285, 291, 303, 308, 311, 324,328, 334-336, 339-340, 351, 373, 376, 383,398, 400, 402, 406

Charter, 403Children, 43, 92, 249-250, 252-253, 261, 263, 265,

279, 304, 327-328, 337-338, 400, 408,410-412, 417-418, 420, 422-423

education programs, 327-328China, 34CIA, 439circuits, 44Cities, 34, 41, 69, 82, 123, 153, 175, 264-265, 274,

276-278, 282, 296-297, 301-302, 413, 420,431

Citizens, 46, 48, 54, 62, 146, 156, 191, 200, 206,208-209, 211, 228, 231, 234, 240, 253,259-260, 313, 316-317, 347-348, 364, 366,389-390, 399, 427, 431, 433, 437, 439

City, 21, 32-33, 43, 63, 80, 109, 140, 160, 174, 181,184-185, 191, 213, 236, 262, 269, 274, 278,296, 298-299, 301, 328-329, 336-337, 367,390, 400, 417, 420, 427

commission, 390, 427manager, 140, 390, 427treasurer, 43

city manager, 390, 427City managers, 269city treasurer, 43Civil liberties, 61civil rights, 186, 337

civil rights movement, 337Civil rights movement, 337Civil service, 390, 427Civil society, 203, 316Civil War, 377-378Civilizations, 60, 138Clark, Wesley, 348Clean Air Act, 52, 407

1970, 407Cleavages, 62climate change, 112CNN, 439coal, 46-47, 130, 222, 230coalition, 9, 42, 52, 179, 379Coalitions, 177Cold War, 102, 378, 380Collective action, 46, 371collective goods, 206-209, 219, 227College students, 194Colorado, 268Commerce, 33, 61, 238, 275-276commissions, 206

regulatory, 206committees, 48, 100-101, 166, 436

joint, 436select, 166

Common Sense, 18-19, 86, 146, 342communications, 103, 113, 178, 386-389, 393, 395,

398, 400, 433, 442-444Communist parties, 119

Soviet, 119Community, 12, 15, 23, 40, 55, 63, 69, 156, 181, 186,

191, 199, 202, 210, 213, 219, 227, 231,234-235, 239, 255, 258, 313, 321, 337,352-353, 375, 399, 419, 430, 433

standards, 191, 313Comparable worth, 238comparative advantage, 120competition, 11, 20, 44, 154, 227Complexity, 4, 22, 33, 41, 44, 50-51, 58, 62, 73, 78,

90, 110, 112, 117, 120, 174, 179, 193, 200,386

computers, 414conditions, 5, 12, 19-20, 22, 36, 45, 47-48, 50, 52, 63,

66-67, 69-70, 72, 78, 112, 114, 119, 127,130, 135, 137, 148, 150, 156, 165-166, 172,


177-179, 193-194, 197, 202-203, 205, 223,239, 241, 252, 256-258, 260, 262-264, 267,270-271, 277, 291-292, 295, 302, 327, 330,342-343, 347, 359, 361-362, 368, 374, 388,406, 431, 437

social, 12, 19-20, 22, 36, 45, 63, 67, 69-70, 78,112, 119, 130, 165-166, 177, 202-203,205, 223, 239, 252, 256-258, 260, 262,264, 267, 270-271, 277, 292, 295, 302,343, 347, 359, 362, 368, 374, 406, 437

Confirmation, 86, 419Confrontation, 146Congress, 18, 24, 28, 30, 43, 102, 124, 191, 252, 308,

329, 334, 343, 364, 366, 386, 390, 427, 436competitive, 436First, 24, 390, 427in foreign policy, 191leadership, 102organization, 24, 390oversight, 43staff of, 386strategic, 102

Congress, U.S., 43, 252, 386staff and, 252

Congressional Record, 380Congressional Research Service, 124, 379Connecticut, 286, 288-291, 360Conservatism, 343, 398conservatives, 335Constable, 263Constituencies, 178constituents, 194, 358Constitution, 366Constructed, 70, 77-78, 82, 85, 112, 196, 203, 280Consumer goods, 284Consumer Price Index, 275, 280-281, 304

CPI, 280consumers, 62, 84, 206-207, 224, 228, 280, 410Consumption, 24, 26, 71, 83, 120, 127-128, 130,

132-135, 174, 206, 222-223, 230-232, 254,275, 303, 308, 413, 416, 419

controversy, 79, 196, 351Converse, Philip, 259Coordination, 91, 257, 423corporations, 15, 74, 267, 390Corrections, 15Corruption, 204cost of living, 253costs, 3, 5, 8, 13, 16-17, 21-22, 24-26, 42, 45-46, 49,

54, 74, 77, 82, 84, 106-107, 147, 149,154-158, 160-164, 173, 190-191, 195,197-203, 205-221, 224-236, 238-246, 266,274, 280, 284, 305, 313, 316, 324, 331, 347,368, 408-411, 417, 421-423, 431

of health care, 199Council for Excellence in Government, 42Council of Economic Advisers, 81Counties, 109, 184Courts, 43, 57Crime, 12, 21, 55, 62, 70, 74, 92, 109, 129-130, 135,

155-156, 175-176, 249, 253-254, 258,276-278, 280, 303, 376, 389, 423, 429-431

Criminal justice, 41, 218, 248, 252-253, 258, 262, 270,277-278

Criminal law, 249crisis, 11, 24, 26-29, 46, 62, 82, 103, 138, 148, 205,

229, 239, 274, 303, 355-357, 376, 379-380,387, 432

Crude oil, 24, 245Cuba, 27-29, 356-357Cuban Missile Crisis, 11, 27-29, 82, 103, 355-357culture, 86, 107, 257, 318, 367, 380Culture of poverty, 86Customs Service, 215

Ddeath penalty, 423Debates, 18, 24, 54, 106, 181, 198, 211, 223, 237,

368, 371, 380, 423Decentralization, 267Decision making, 4, 22, 27, 32, 40, 49-50, 56, 59, 64,

80, 101, 110, 191, 193, 205, 229, 314, 316,335-336, 378, 380, 401, 435

Decision-making, 49, 73-74, 80, 146, 252, 330Decisions, 2, 10, 17, 23, 34, 39, 43, 47-48, 50, 56-57,

62-63, 73-74, 80, 82, 100, 155, 172, 181,184, 197, 200, 238, 268, 351, 365, 379, 390,395-396, 402, 427

Decriminalization, 368

defendant, 96Defense, 8-9, 27, 33, 38-40, 62, 99-100, 180, 206,

218, 403-404, 407, 434-436Defense Department, 218Defense expenditures, 9Defense policy, 99Deficit, 119, 335, 435defined, 6, 14, 33, 69-72, 75-79, 82, 86, 88, 93-94,

112, 116, 122, 154, 171, 192, 196, 206,214-215, 253, 255, 271, 280, 295, 312, 328,343, 346, 354, 388, 396-398, 410, 426

Delaware, 380-381Democracies, 355, 367, 374democracy, 4, 7, 39, 42, 239, 263, 343, 374

assessment of, 7defined, 343

Democrats, 335, 381Demographic, 249, 416, 420-421

characteristics, 249Demography, 35Department of Agriculture, 38Department of Commerce, 275-276Department of Defense, 38, 40, 180, 218, 435

budget for, 435Department of Energy, 242, 421Department of Health and Human Services, 72Department of Housing and Urban Development, 4,

266, 419HUD, 266

Department of Justice, 303Department of Labor, 43, 257, 259, 281, 431Department of State, 386Department of Transportation, 86Departments, 4, 42, 206, 389, 433-434depression, 60, 257Detention, 97Deterrence, 99, 423Detroit, 337Devaluation, 322Dictators, 380diplomacy, 348-349, 381, 402, 404-406, 435disability, 245Discontinuities, 135, 138Discrimination, 43, 198, 204, 238, 303, 335, 364,

366-367, 369-370against women, 43, 364, 366-367age, 238employment, 367racial, 198, 303, 366wage, 238, 335

Distribution, 32-33, 59-61, 77, 90, 96, 103, 107, 109,161, 177, 202-204, 251, 266, 274-277,296-297, 302-304, 316, 335, 353, 368-369,401, 415, 422

Distribution of income, 276, 304, 316, 335, 368-369Diversity, 267, 388, 393, 443Division of labor, 34Docket, 411, 421Dole, Bob, 381dollar, 23, 26, 201-202, 213, 217, 219, 226, 228,

232-233, 235, 252, 266, 280, 284, 291,349-350, 437

domestic policy, 250, 382drinking age, 423Drinking water, 420drug abuse, 167Drug Enforcement Administration, 215Drug policy, 34, 368

EEaston, David, 99, 312Economic development, 109, 269, 375, 435Economic growth, 83, 128, 257economic issues, 311economic policy, 217, 240Economic recession, 194Economic sanctions, 377, 379, 381-382Economy, 37, 48, 71-72, 75, 113, 120, 218, 241, 244,

257, 308, 343, 346, 349, 374, 422-423global, 218groups in, 48issues, 48, 71-72, 423management, 257, 423stability in, 37systems in, 71

education, 1, 31, 37, 41, 55, 61-62, 65, 84, 93, 101,109, 111, 117, 123, 153-154, 165, 189, 206,209, 247-249, 257-259, 261-263, 265, 267,289, 292, 311, 316-317, 327-329, 335, 339,

341, 383, 389, 410, 422, 425, 433, 436, 441Education Act, 327Educational level, 417Efficacy, 46, 237, 372, 387efficiency, 4-5, 7, 13, 15, 24-25, 37, 45, 56, 148, 163,

192, 197-199, 203-206, 210-220, 222,224-225, 229, 238-242, 245, 312-314, 316,324, 326, 329, 335, 344-345

Egypt, 108elderly, 92, 204, 224, 250, 284election, 119, 174, 230, 374, 399, 445-446

of 1992, 445-446presidential, 445-446

Elections, 48, 154, 196Elementary and Secondary Education Act, 327

1965, 327Elementary education, 206Elite, 104Elites, 48, 77, 96, 177, 397e-mail, 111, 186, 434Emancipation, 205Embassies, 439Employment, 18, 26, 36, 92-93, 123, 249, 258-259,

262, 279, 291, 307, 309, 337, 367-368, 431Employment discrimination, 367Employment opportunities, 123Encyclopedia of Associations, 41, 111energy, 8, 12, 22, 24-25, 46-47, 52, 62, 69, 71, 83,

120, 128, 132-135, 146, 174, 197-198, 206,220, 222-223, 225, 229-230, 241-242,244-245, 249, 254, 278, 303, 329, 349-350,389, 421

sources of, 249, 329, 350energy crisis, 24, 46, 303Energy Department, 244energy policy, 8, 25, 71, 146, 174, 220, 222, 229-230,

245Engagement, 156England, 35-37, 421Enlightenment, 35, 57, 395Entrepreneurs, 15

policy, 15Environment, 41, 47, 56, 61-62, 71, 78, 99, 117-118,

209, 222, 259, 284-285, 377, 410, 437nuclear power, 71, 222

environmental policy, 52, 55, 58, 174Environmental Protection Agency, 80, 153, 182, 242,

244, 248, 251, 258, 282, 284-285, 296, 407EPA, 282, 407

environmentalists, 222-223, 375Equal opportunity, 335Equal rights, 343, 364, 366Equal Rights Amendment, 343, 364, 366

ERA, 364, 366Equality, 7, 204, 239, 241, 265, 276, 303, 313, 319,

364, 368political, 204, 239, 241

Establishment, 100, 102, 215, 314Ethnic cleansing, 19-20, 374, 376, 378Ethnicity, 11, 329Europe, 34-35, 41, 62, 71, 119, 123, 308, 312, 378European Union, 41-42, 248-249Ex post facto, 327Executive, 38, 43, 57, 62, 101, 103, 111, 206, 257,

366, 384-387, 391-392, 410, 428-429, 434Executive agencies, 43, 62Executive branch, 38Executive departments, 206Executive Office, 257Executive Office of the President, 257Executive order, 366, 410executive orders, 43Expansion, 34, 62, 184expenditures, 9, 11, 36, 54, 135, 144, 151, 153-154,

156, 178, 199, 213, 240, 250, 252, 255, 258,260, 266, 269, 327, 389, 427

health, 54, 153, 199, 250, 255, 258, 389other, 9, 36, 135, 144, 153-154, 178, 213, 250, 252,

255, 258, 266tax, 154, 178, 427transportation, 153, 199, 427

expertise, 58-59, 61, 126, 166, 375, 394, 401, 443technical, 59, 61, 166

Exponential growth, 61, 138exports, 381, 402Externalities, 54, 211, 218-219, 228-230, 392, 407,

428, 431

F


Fairness, 4, 202, 204, 239, 316-317, 320-321, 335,369

farmers, 208, 412federal agencies, 40, 125, 249, 257Federal budget, 436, 438Federal Bureau of Investigation, 258, 278, 303Federal bureaucrats, 57Federal funding, 390, 427federal government, 38, 40, 62, 138-139, 191, 231,

244, 257, 407, 437power of, 62understanding, 407

Federal Highway Administration, 244Federal Register, 423federal revenue, 248, 251Feedback loop, 44, 152finances, 34Firefighters, 63Fiscal accountability, 430fiscal policy, 179Food and Drug Administration, 353, 420food stamps, 93Ford Foundation, 41, 186Foreign Affairs, 249, 377, 435foreign aid, 191, 389-390, 427Foreign policy, 27, 103, 110, 191, 193, 250, 340,

355-356, 380, 435bureaucracy, 355State department, 435State Department and, 435understanding, 340, 355-356

Formal agenda, 229Founding fathers, 375France, 36, 71, 258, 380fraud, 43, 204, 238, 369-370free press, 4, 46, 48-49, 61, 74, 98, 147, 177, 193,

202, 254, 267, 314, 318-319, 351, 356, 363,371, 375

freedom, 192, 216, 253, 319, 321, 371-372, 380, 415Free-rider problem, 209, 228Friedman, Milton, 4, 240, 312

GGarbage can model, 44Gas prices, 175Gasoline prices, 30, 156, 173, 236, 238GDP, 374Gender, 11, 252, 364Gender equality, 364General Accounting Office, 43, 182, 267, 307, 438General law, 62Generalizations, 264, 268-271, 346Genocide, 374, 378Georgia, 182, 381, 411Germany, 36, 71, 258, 382global warming, 5, 66, 112, 225, 376, 389, 426Goods and services, 77, 206, 211, 213, 231, 280, 284Government, 5, 11, 15, 27, 32-35, 38, 40-43, 56,

60-62, 64, 66-67, 71-72, 74, 78, 80, 92-93,95, 100, 108, 118, 120, 124-126, 131,138-139, 144, 147-148, 152, 165, 182, 191,198, 206, 208, 218, 224-225, 227-229, 231,234-235, 241, 244-245, 247-248, 257,259-260, 265, 269, 303, 317, 343, 347-350,354-355, 364, 366-367, 374-375, 379, 389,399-400, 407-408, 426, 433, 435-438

coalition, 42, 379corporations, 15, 74democratic, 61, 148, 198, 241divided, 92, 95, 206, 317nature of, 66-67, 71, 74, 78, 92, 108, 148, 206, 231objects of, 124, 426territorial, 33tribal, 33waste, 78

Government employees, 260Government regulation, 5, 72, 400Government spending, 374governor, 41, 286-289, 291, 360, 390, 427grants, 37, 186, 248

federal, 248project, 186

Great Depression, 257Greece, 378Gross domestic product, 9growth, 9, 33-35, 37-38, 40, 58, 61-62, 83, 91, 118,

120, 125-126, 128-130, 137-142, 144, 152,241, 243, 257, 274, 322, 374, 435

small, 33, 139, 144

Gulf War, 402Gun control, 12guns, 312, 375, 427

HHarvard University, 22, 36, 39-40, 46, 57, 59, 64, 124,

197, 203, 267, 307, 316, 336, 359, 401, 422Head Start, 123, 265, 328, 361Head Start program, 265Health, 37, 41, 43, 54-55, 61-62, 69-70, 72-73, 90, 93,

114-115, 117, 122-123, 130, 153, 192,197-199, 209-211, 217-218, 229, 231-232,235, 241, 249-251, 254-255, 257-259, 262,270, 280, 282, 284-285, 303, 313, 323, 329,331, 335, 389, 399-400, 407-409, 411, 418,422-423

Health care, 54, 69, 73, 90, 122, 197-199, 217, 229,254-255, 258, 280, 331, 335

health care policy, 54national, 54

Health care reform, 90health insurance, 54, 122health maintenance organizations, 217Hearing, 244Heclo, Hugh, 90higher education, 123highways, 24, 108, 126-127, 209, 237, 243-245, 308Hitler, Adolf, 380Hofstadter, Richard, 110Hold, 103, 108, 119, 185, 320, 413Holland, 2, 101, 110Hoover, Herbert, 37House of Commons, 41, 59House of Representatives, 54, 436Households, 23, 55, 92-93, 185, 259-260, 369Housing, 4, 37-38, 55, 62, 93, 111, 165, 184, 206, 213,

231, 248, 259, 262, 266, 338, 368, 419-420Housing and Urban Development, 4, 266, 419Hull House, 36human nature, 66, 78, 150, 205, 316human rights, 111, 376-378, 381, 389-390, 402, 427

groups, 111, 376-377, 389Human Rights Watch, 111Hussein, Saddam, 375, 379, 406

IIdaho, 113Identity, 372Ideology, 63, 77-79, 119, 358Illinois, 108, 113, 197, 263Immigrants, 437Immigration, 436-438imports, 381, 402incentives, 31, 52, 177, 262, 303, 389, 419, 433Inclusion, 81, 248income, 23, 36, 55, 81, 92-93, 95, 106-107, 144,

153-154, 184-186, 191, 194-196, 201-202,204, 209-213, 216, 218-219, 224, 228, 238,240, 253, 255, 258-259, 262-263, 266,273-274, 276-278, 280, 295, 303-304, 316,318, 320-321, 325, 328, 331, 335, 337-338,368-369, 375, 416-417, 437

distribution, 107, 202, 204, 266, 274, 276-277,303-304, 316, 335, 368-369

minimum wage, 194-195, 337Income distribution, 303, 335, 369Income inequality, 55, 369Income security programs, 437Income tax, 262Incorporation, 46Incremental policy, 146-147, 368incrementalism, 48-50, 58, 147, 190Independence, 48Independent, 14, 33, 48, 69-71, 83, 135, 154-157,

163, 174, 179, 181, 226, 262, 271, 289-290,306, 313, 318-319, 413-416, 418-421

India, 34Individual, 34, 45-49, 57, 70, 74, 81, 88, 93-94, 96-97,

99, 101-102, 106, 108, 114, 146, 154, 160,166-167, 196, 203-204, 210-211, 223,234-235, 262, 308, 313, 317, 319, 322, 347,371, 375, 394-396, 416, 443

Industrial Revolution, 35, 37Industrialization, 154inflation, 70, 140, 195, 216, 303influence, 2, 8, 15, 21-22, 25-26, 34-36, 40, 53, 58,

61-62, 72, 77, 96, 102, 104, 127, 151, 168,175, 178, 195, 257, 267-268, 328, 335, 384,

395, 397-398, 402, 407, 431, 435Information, 2, 5-8, 10-15, 17-21, 24, 27, 31-33, 38,

42, 46, 48-49, 53-55, 57-58, 60-64, 66-67,75, 86, 91, 105-108, 113, 118, 125-127, 146,149-152, 160, 163-165, 171-172, 176, 178,182, 186, 191-194, 198, 214-215, 221,224-226, 228, 231, 237, 239-241, 244,248-250, 253-256, 259-260, 263, 266-272,274, 277-279, 291-292, 304-305, 311, 318,322-323, 325-327, 329-334, 340-357,360-361, 363-369, 373, 375, 381, 383-388,390-391, 393-399, 401-402, 407-408, 411,419, 427-428, 431-432, 434, 437, 439,442-444

Information costs, 17Inglehart, Ronald, 62Initiatives, 54, 257, 317, 435Institute for Research on Poverty, 93Institutionalization, 37, 59, 198, 401insurance, 54, 122Integration, 63, 103, 313, 355Intelligence, 28-29, 38, 69, 77, 357, 438intensity, 76, 398interdependency, 69, 71interest groups, 42, 62, 77

activities of, 62economic, 42, 77role of, 62

interest rates, 81, 216, 235International Labor Organization, 390international organizations, 41, 257Internet, 18, 61, 64interstate highways, 24, 108, 243-245, 308Invasion, 27, 102, 356, 403-405

Britain, 405Japan, 102

involvement, 39, 56, 60, 62, 268, 381Iran, 354, 379Iraq, 33, 73, 354, 375, 379-382, 402-406, 435Israel, 379, 381, 402-404issues, 3, 13, 16-17, 22, 34, 40-41, 44, 48, 55, 61,

63-64, 71-73, 79, 105, 117, 124, 146,167-168, 170-172, 178, 184, 191, 194,197-198, 242, 247, 252, 260, 279, 311, 313,316, 331, 334-335, 337, 359, 371-372, 382,389, 397, 403, 407, 419, 423, 427, 436, 438,444

advocacy, 41, 191Italy, 419

JJails, 96Japan, 34, 102, 165, 258Japanese, 41Jencks, Christopher, 260Jihad, 438Jobs, 3, 23, 195, 238Johnson, William, 422Journalists, 37judges, 97judicial system, 96

structure of, 96Jurisdiction, 211-213, 230-231, 286Jury trial, 103Justice, 3, 5, 41, 117, 124, 181, 184, 186, 202-204,

218, 224, 238-239, 248, 252-253, 258, 262,270, 277-278, 303, 311, 313, 315-316, 320,335-336, 369

justices, 43

KK Street, 42, 59Kautilya, 33-34Kenya, 439Keohane, Robert, 371Kingdon, John, 59Kurds, 379Kuwait, 379-381, 402-406Kyoto treaty, 343

LLabor, 34, 43, 55, 60, 124, 186, 194-195, 208, 243,

248-249, 257-259, 270, 281, 369, 389-390,431

labor unions, 124Language, 16, 18-20, 25, 81, 84, 86, 94-95, 113, 193,

341-342, 345, 370, 372-373, 381, 386, 396Lasswell, Harold, 3


Law enforcement, 107, 156, 231, 244, 258, 429-430Laws, 19-20, 33, 36, 118, 126, 147-148, 153, 198,

216, 238-239, 286, 346, 356, 358-359, 371,431

civil, 118implementation of, 216, 238

Le Monde, 109leadership, 102, 355

congressional, 102Left, 11, 13, 79, 140, 154, 161, 201, 234, 274, 327,

362, 364, 387, 421-422Legislation, 123, 191, 194, 201, 209, 326, 366, 385,

390, 399, 427, 432veto, 390, 427

Legislators, 12, 194-195Legislature, 195, 375, 390, 427Legislatures, 43, 57, 248, 395Legitimacy, 102, 178Liberalism, 343liberals, 335liberty, 4, 13, 239Library of Congress, 124Libya, 377life expectancy, 216local government, 182, 225, 347, 367

scope of, 182Local governments, 35, 182, 235, 248-249, 390, 427,

430, 437Los Angeles Times, 376, 378Lowi, Theodore, 251Luxembourg, 55, 419

MMajority, 38, 43, 46-48, 152, 185, 240, 313, 336-337,

364, 375opinion, 364rule, 47-48, 240, 313status, 43, 48

majority rule, 47-48, 240Mandate, 434-436Marijuana, 167Market economy, 374Market failures, 407-408Maryland, 17, 42Mass media, 165, 179, 387Massachusetts, 337, 419media, 165, 179, 387, 389, 395, 433, 438, 444Median age, 238Medicaid, 93Medical costs, 228Medicare, 54, 93Medicare Trust Fund, 54Michigan, 3, 74, 259-260, 387, 421Microsoft, 376, 394-395, 444Middle class, 37, 216, 355Middle East, 119, 381Migration, 108, 429military, 38, 40, 45, 60, 100, 165, 218, 357, 364, 371,

373, 379-382, 402-406draft, 403

Military force, 379Military forces, 379Military intervention, 381, 402-406minimum wage, 194-195, 337mining, 109, 112minorities, 43, 123, 184, 238, 367, 378, 390, 427

discrimination against, 367Missouri, 24mobilization, 62, 337Mobilize, 43money, 19, 81, 118, 216, 228, 232, 234-235, 251, 346,

381, 423Moynihan, Patrick, 381multinational corporations, 390Myths, 12, 77, 108

NNation, 70, 257, 378, 405national, 8-9, 11-12, 14-15, 22, 24-27, 38-41, 43,

54-55, 57, 62-63, 70, 72, 106, 109, 111, 113,144, 167, 220, 222, 229-230, 237-240,242-243, 245-246, 248, 257, 259, 263, 265,272, 284, 303, 305, 307-308, 317, 319, 321,328, 348-349, 352, 378, 386-387, 389, 396,398, 409, 416, 423, 426, 434-436

national defense, 8, 27, 39, 62National League of Cities, 41national power, 434

National Recovery Administration, 38National Review, 109National Rifle Association, 111National Science Foundation, 15, 40, 257, 259National security, 9, 12, 22, 38, 40-41, 70, 109, 240,

303, 319, 321, 348-349, 389, 434-436National Security Advisor, 348-349National Security Council, 348Nationalism, 52, 380

German, 380NATO, 19-20, 348natural disasters, 131, 255Natural resources, 235Nebraska, 380Netherlands, 36, 75, 367Netherlands, The, 36, 75, 367New Deal, 38, 262New institutionalism, 75New Jersey, 113, 262-264, 328, 381New Republic, 109New York, 2-4, 6-7, 10, 17-18, 22-24, 31-32, 34-35,

38, 40, 42, 44, 46, 48-50, 55-56, 59-63, 67,70, 74, 76-78, 80, 88, 90, 92, 94, 97-101,103, 107-110, 116, 119, 123-124, 128, 131,135, 138, 145, 148, 153, 166, 177, 181-182,191, 193, 200, 202, 210, 216, 218, 220, 242,248-250, 252, 254, 256-260, 262-264, 267,269, 274, 279, 307, 314-315, 318-320, 328,335-337, 341, 351-352, 355-357, 363, 365,370-371, 377-378, 381, 387, 390, 396-397,401, 413, 427, 430, 439

New York City, 63, 274, 337New York Times, 24, 108-109News, 109, 111, 347, 361, 384-385, 387newspapers, 109, 376, 386Newsweek, 109No Child Left Behind, 327nomination, 174

candidate, 174Nominee, 55Nongovernmental organization, 265nongovernmental organizations, 250North Carolina, 55, 148, 248, 307North Dakota, 264, 397Norway, 258NSC, 435-436nuclear power, 71, 146, 222, 230, 349-350nuclear power plants, 71, 349

OOccupational health, 251Occupational safety, 250, 400Occupational Safety and Health Administration, 250,

400OSHA, 250

Occupations, 60, 337Office of Economic Opportunity, 262Office of Management and Budget, 42, 235, 257, 259,

410Office of Strategic Services, 38Ohio, 99Oil, 22, 24, 26, 98, 113, 130, 174-175, 223, 239, 245,

308, 381, 400oil companies, 175Olson, Mancur, 46OPEC, 22, 24, 26, 308opinions, 36, 118, 146, 364Opposition, 5, 12, 24, 124-125, 178, 181, 195,

229-230, 308, 365, 410order, 12, 22, 33, 37, 55-56, 68-69, 71, 73, 75, 77, 94,

96, 106-107, 112, 114, 119, 129-130, 139,145-146, 170, 172, 178, 198, 203-204,238-240, 245, 256-257, 261, 290, 302, 313,319, 331, 351, 366, 410, 419

Organization for Economic Cooperation andDevelopment, 176

Organization of Petroleum Exporting Countries, 24OPEC, 24

Outlays, 437Oversight, 43

PPalestinian Authority, 354Parliaments, 57Parties, 46, 119, 226, 329, 331, 345, 370, 421, 438

political, 46, 345Partisanship, 58Party, 11, 154, 174, 231, 344-345, 372, 378, 381

identification, 11party competition, 11, 154Patriot missiles, 379Patriotism, 373Patrons, 81peace dividend, 378Pennsylvania, 54, 113, 262-264, 328, 338Pentagon, 52People, 36, 76, 87, 98, 109, 116, 120, 185-186,

196-198, 228, 232, 241, 243, 313-314, 335,360, 364, 378, 380, 388, 412, 419-420

Perestroika, 79permits, 113, 119, 142, 154-155, 160, 163, 176, 201,

209-210, 217, 226, 248, 261, 282, 289,327-328, 358

Persian Gulf, 379-380, 402Persian Gulf War, 402Plato, 34plea bargaining, 96-97Plea bargains, 97police, 12, 32, 131, 137, 155-156, 160, 213, 219, 249,

253, 272, 278, 303, 379, 429department, 303

policies, 4-14, 16-18, 20-24, 31-33, 36, 38, 41-44,50-59, 63-64, 71-74, 84, 86, 90, 97, 102,104, 111-112, 118-119, 121-125, 146, 153,155, 177, 180, 189-246, 247-251, 253,258-262, 264-270, 276, 286, 296, 311,315-316, 322-324, 326-327, 329-330,336-337, 340, 343, 347, 355, 367-368, 372,383-384, 388, 406, 428, 430-431

continuing, 4offset, 97social welfare, 9, 38, 43, 55, 202-203, 206,

210-211, 218-219, 262, 267Policy, 1-30, 31-64, 65-88, 90-116, 117-142, 144-158,

160-168, 170-182, 184-187, 189-194,197-206, 208-211, 214-218, 220-224, 226,228-232, 234-242, 245-246, 247-282,284-292, 294-297, 300-310, 311-336, 338,339-382, 383-423, 425-432, 433-439,441-445, 448

categories, 42, 64, 92-93, 228, 251, 267-268, 274,282, 286, 303, 408, 413

entrepreneurs, 15process model, 27, 355-356

Policy adoption, 42-43, 54, 111, 177, 385policy agenda, 349Policy evaluation, 11, 111, 250, 253, 311-312, 316,

323-324Policy formulation, 42-43, 54, 110-111, 267, 385policy impacts, 11, 24, 250, 256policy implementation, 42-43, 55, 111-112, 177, 221,

255, 269, 385Policy makers, 11-12, 19-20, 22, 31-32, 35, 38, 41,

48-50, 54, 57-58, 61-64, 80-82, 96, 104, 108,112-113, 119, 121, 126, 146, 155-156, 164,172, 181-182, 184, 197, 222, 234, 249, 255,260, 290, 297, 315, 325-326, 329, 334,340-341, 357, 359, 365, 379, 383, 386,393-399, 426, 434, 443

Policy making, 4, 6, 15, 20, 22, 31, 34-35, 41-42, 45,50, 52-53, 55-64, 74, 102, 110, 146, 148,205-206, 208, 238, 242, 253, 329, 340, 347,355, 358, 363, 365, 367-368, 377, 383, 387,395-396, 398, 401

Policy outcomes, 5-9, 11-12, 24, 26, 32, 51, 54-55, 64,108, 112, 117-142, 144-158, 160-168,170-182, 185-187, 190-191, 201, 221,247-282, 284-292, 294-297, 300-310,311-312, 322-326, 328-332, 346, 384-385,390, 427-428

Policy outputs, 153, 250-251, 255-256, 260, 262, 264,326

Policymaking, 5, 45, 52, 56, 64, 103, 110, 182, 192,315, 398

Policy-making process, 31-59, 61-63, 77, 88, 177-178,364, 376, 384, 393, 395, 399, 401, 442-443

political conflict, 177-178Political culture, 367political participation, 154, 267Political science, 3-4, 9, 13-14, 16, 27, 31-32, 39-40,

59-60, 62, 67, 75, 77, 79, 82, 90, 100, 112,341, 355, 362, 377, 401

Political values, 239, 260politicians, 15, 34, 37, 62Politico, 60Politics, 15, 27, 32-35, 52, 59, 61-63, 119, 129-130,

149, 154, 196, 229, 238-239, 241, 251, 259,


321, 327, 351, 355, 358, 368budgetary, 229, 238, 368information about, 239, 259machine, 32policy and, 32, 368

polls, 366pollution, 23, 67, 70-72, 84, 92, 112, 135, 184, 191,

229-230, 234, 248-249, 253-254, 258-259,280, 282, 285, 296-297, 301-303, 347, 367,376, 389, 422, 426, 429-430, 445

pollution control, 254poor, 21, 36, 77-78, 184, 203-204, 213, 216, 222-224,

229-230, 260-261, 263, 327, 400Population, 35, 52, 94, 97, 111, 124, 128-130, 138,

144, 147, 161, 238, 248-249, 258-259,262-263, 273-274, 276-278, 280, 290, 292,294-295, 303-304, 309, 346, 351-353, 374,409-410, 430, 437

density, 138, 238, 430Positivism, 3, 16, 312, 322, 372Postindustrial society, 60-61Poverty, 16, 36, 45, 55, 70, 72, 77-78, 86, 92-96, 104,

209, 211, 240, 248, 259-260, 274-276,278-279, 286, 303-304, 331, 335, 337, 376,386

levels, 55, 70, 72, 77, 93, 95, 248, 279war on, 16, 45, 92, 240

Poverty line, 92-93, 95, 248, 259-260, 274Poverty rates, 278-279power, 3, 35, 48, 56, 58, 61-63, 71, 77, 96, 99, 109,

116, 118, 126, 141-142, 144, 146, 177-178,204, 208, 210, 222, 230, 280-281, 304, 314,316, 335, 346-347, 349-350, 363, 365, 380,398, 416, 434, 436

commerce, 61Doctrine, 35expressed, 118, 126, 178, 210nuclear, 71, 99, 146, 222, 230, 349-350, 380war, 99, 380

powers, 59, 82, 149, 153, 165, 346, 355, 380, 401Pragmatism, 3, 351, 370Precedent, 146presidency, 102, 174

election, 174President, 4, 16, 54-55, 81-82, 92, 102, 111, 174, 257,

308, 327, 375, 377, 379, 381, 390, 402,404-405, 427, 434-436, 438

press, 2-5, 9, 15, 17-18, 22, 32-34, 36, 39-40, 42, 46,48-49, 51-52, 55-57, 59-63, 67-68, 74-77,79, 88, 90-91, 98-100, 110, 119, 124-125,138, 147-148, 155, 177-178, 181-182, 191,193, 196-198, 202-204, 229, 231, 238-241,248-251, 254, 257, 259, 262, 265, 267, 269,272, 279, 302, 307-308, 312, 314-316,318-321, 336, 340-341, 349-351, 355-356,358-359, 363, 371, 375, 377-378, 380,386-387, 389-390, 401-402, 413, 426, 444

prices, 30, 77, 156, 173-175, 206-207, 209, 211, 213,218-219, 222, 224, 227-228, 230, 235-236,238-239, 280, 410, 422

Prisons, 367Privacy, 61, 198private property, 223Privatization, 344-345, 355Probation, 74problems, 2-15, 21, 25, 27, 31-32, 35-45, 48, 50-51,

53, 55-62, 65-88, 90-116, 118, 121, 123,144, 148, 152, 155, 165-167, 176-177, 184,197, 199-200, 202-203, 206, 208-211, 218,221, 229, 244, 247, 251, 253, 255, 259-261,284, 290-291, 303-305, 314-315, 323, 327,334, 355, 363, 365, 367, 379, 381, 384-385,389, 395-398, 402, 407, 422, 426, 428-430,433

procurement, 72Program budgeting, 218Prohibition, 54, 130Proletariat, 37property, 25, 33, 43, 178, 203, 219, 223, 229, 246,

258, 303, 369, 408taxes on, 43

property rights, 33, 229proposals, 182, 435Propositions, 13, 62-63, 118, 126, 147, 153, 346, 356,

358-359Prosecutor, 74, 96Prosecutors, 97Public, 1-5, 11, 14-18, 23, 31-33, 37-46, 49-50, 52,

55-57, 59-61, 63-64, 65, 68, 71, 73-75, 77,

79, 81-82, 85, 90, 93, 97, 100, 102-103, 107,109-112, 117, 124, 129-130, 135, 144, 146,148-154, 165, 170-171, 177-178, 182, 185,189, 193, 197-203, 205-210, 216-217, 225,227, 229, 231, 234, 238-239, 242, 244,247-253, 258-260, 262, 267-269, 280, 289,307, 311, 313-318, 321, 323-324, 326-327,329, 335-336, 339, 344, 347, 349, 351, 355,359, 362-363, 365-368, 370, 372, 377, 383,387, 389-390, 395, 398-399, 407, 410,421-422, 425-427, 429, 433, 435, 441, 445

going, 42, 97, 407goods, 46, 60, 77, 206-209, 217, 227, 231, 250,

280, 321, 335, 407Public agenda, 43-44, 399Public choice, 61, 148-149, 152-154, 208Public diplomacy, 435Public employees, 149-150, 178public goods, 46, 206, 208-209, 407Public health, 37, 249, 262, 407Public hearings, 146, 185, 421Public Interest, 56, 206public opinion, 52, 146, 262, 366, 370Public opinion polls, 366public order, 33, 129-130public policies, 11, 18, 44, 52, 59, 74, 97, 112, 124,

146, 153, 229, 248, 251, 253, 258, 267-268,315, 323-324, 326, 329

agricultural, 146economic, 18, 59, 74, 153, 248, 253, 258, 323energy, 52, 146, 229, 329environmental, 52, 153, 229, 248, 251, 258health care, 229, 258implementation of, 248, 267, 329social welfare, 267

Public policy, 1, 3-5, 11, 14, 17-18, 31, 37, 39-42, 49,57, 59-60, 63-64, 65, 68, 71, 75, 85, 90,102-103, 107, 109-110, 117, 124, 144, 146,148, 152-154, 177, 182, 189, 193, 197,201-203, 206, 208, 216, 238-239, 247,250-253, 262, 269, 280, 311, 315-317, 321,335, 339, 347, 351, 355, 359, 362-363,367-368, 370, 377, 383, 387, 389, 398,425-426, 433, 441

domestic, 250effective, 17, 37, 40, 59, 63, 90, 109, 193, 202,

251, 347, 387, 398formulation of, 71, 85, 239policy evaluation, 11, 250, 253, 311, 316policy implementation, 42, 177, 269values in, 317, 377

public services, 149, 202, 344public support, 146, 367, 445punishment, 74, 319, 367, 423purchasing, 74, 280-281, 304

QQuarrels, 113

RRace, 54, 173, 258, 278-279Racial discrimination, 198, 303, 366Racial segregation, 276Rainfall, 85-86, 155Ranney, Austin, 355ratings, 106Realignment, 62Recall, 139, 156, 161, 163, 327Recession, 26, 30, 127, 156, 194, 235, 238Recessions, 52Recidivism, 328reconciliation, 264, 266Reconstruction, 15, 36, 435Recovery, 38, 316Redacted, 438-439Redistribution, 211, 213-214, 240Referendum, 184reform, 35, 54, 90, 262, 269, 278, 375, 434Reforms, 18, 92, 123, 131, 239, 261, 285-286,

288-290, 294, 296, 359, 377, 435-436Registration, 375Regulation, 5, 72, 84, 215, 239, 251, 400, 410

of business, 251Regulations, 43, 216, 229, 251, 408, 410Regulatory agencies, 248, 410Regulatory commissions, 206representation, 26, 78-79, 85, 91, 145, 155, 274, 277,

351, 380

congressional, 380Representative, 16, 36, 154, 166, 168, 178, 258-259,

267, 294, 346, 351, 364, 409, 417Representative government, 364Representatives, 54, 101, 364, 411, 422, 433, 436Republic, 109, 364, 378Research and development, 38, 80, 165, 173, 241,

384residential segregation, 276revenue, 54, 182, 184, 206, 208, 248, 251

sales tax, 184sources, 184

Revenue sharing, 248Revenues, 23, 156, 184, 206-207, 240, 251, 437Revolutions, 77, 350, 356Rider, 185, 209, 228Riders, 185-186Right, 4, 10, 13-14, 24, 26, 43, 67-71, 75, 78-79, 85,

87, 92, 96, 106, 108, 161, 180, 191, 193,206, 224-225, 239-240, 274, 315, 320-321,343, 363, 369, 371, 378, 422

rights, 1, 31, 33, 46, 65, 108, 111, 117, 123, 186, 189,195, 198, 203, 223, 229, 247, 311, 337, 339,343, 364, 366, 376-378, 381, 383, 389-390,402, 422, 425, 427, 433, 441

civil, 186, 203, 337, 377-378, 390, 427individual, 46, 108, 203, 223natural, 123privacy, 198property, 33, 203, 223, 229to privacy, 198

Roe v. Wade, 43rogue states, 349Rose, Richard, 90, 355Rule, 36, 47-48, 75, 77, 90, 92-94, 115, 146, 217, 240,

272, 289, 291, 313, 316, 335, 349-352, 354,368-370, 373, 407, 410-411, 418, 420-423

Rules, 19, 36, 49, 72, 92-93, 95, 97, 102, 113, 146,198, 202, 204, 253, 304, 312-314, 318,320-321, 335-336, 345, 349, 351, 370, 387,392, 428

formal, 36, 204, 320-321, 349, 370, 392Russia, 165

SSafety, 43, 55, 109, 114, 201, 206, 237-238, 245,

250-253, 259, 262, 272, 289, 291, 307, 360,374, 400, 408, 429

Sales tax, 184, 186Sample, 2, 38, 51, 90, 111, 115, 135, 161, 168, 255,

262, 268-270, 284, 294, 296, 302, 304, 333,346, 351-353, 374, 409, 411-416, 418-420

sampling bias, 421sanctions, 377, 379, 381-382, 402, 405-406, 423Saudi Arabia, 379School busing, 313school districts, 327schools, 5, 14, 269, 313, 316-317, 351, 389, 423Scriven, Michael, 341searches, 99Seattle, 264Secondary education, 327Secretaries, 101Secular, 129-131, 135, 137-138Securities and Exchange Commission, 38security, 4, 7-9, 12-13, 22, 25, 33, 38, 40-41, 45, 70,

81, 88, 93, 109, 111, 192, 201, 219, 235,239-240, 251, 303, 313, 319, 321, 348-349,357, 379, 381, 389, 402-406, 430, 434-437

military, 38, 40, 45, 357, 379, 381, 402-406Segregation, 276

residential, 276Selective perception, 77self-interest, 102, 150-151, 378Sen, Amartya, 336Senate, 377, 379-382, 402-406Senate, U.S., 379-380, 405Senator, 24, 111, 308, 344-345, 375, 380-381Senators, 381Settlements, 33Sierra Club, 111Sit-ins, 146Social change, 69, 78, 101, 128, 166, 171, 256-258,

267-270, 397Social contract, 320Social costs, 209, 234-235Social groups, 37, 222, 234Social inequality, 11, 260Social issues, 40


Social mobility, 259-260Social policy, 10, 12-13, 16, 39-40, 57, 249, 261-262,

328Social Security, 93, 251Social Security Administration, 251Social services, 9, 93, 229, 248social welfare policies, 55Socialism, 119

Soviet, 119Socialists, 375Socialization, 15-16solar energy, 47, 229sources of information, 17, 249-250South, 108, 214, 341, 349, 378, 413, 417South Africa, 341South America, 214South, The, 413Sovereignty, 48, 378Soviet Union, 27-29, 102, 119, 356-357Specialization, 18, 37, 108speech, 346spending, 312, 374, 435

choices, 312stability, 37, 52, 381staff, 38, 43, 56, 60, 101, 116, 154, 178, 252, 255,

264, 266, 268, 327, 329, 333, 352, 386, 389,394, 400, 427, 435-436, 438, 443

agencies, 38, 43, 56, 60, 327, 329, 386, 435-436committee, 43, 436congressional, 252, 400, 436personal, 56, 101, 386White House, 435-436

standard operating procedures, 72, 90, 102, 357, 387bureaucratic, 102

Standardization, 265, 331Standards, 3, 33, 37, 42, 67, 71-72, 191, 198, 230,

248, 251, 313-315, 317, 347, 367, 370, 388,396, 407-408, 411, 421-422

Standing, 56, 239, 345, 371Stanford University, 39START, 69, 123, 125, 220, 265, 328, 345, 361, 393,

443state, 3, 20, 22, 32-33, 35, 37, 41, 43, 59-62, 72, 86,

99-100, 102, 108-109, 111, 113, 115, 119,121, 146, 165, 171, 191, 203, 216, 223, 225,228, 231, 235, 248-249, 251, 257-259, 263,269, 286, 313, 330, 334-336, 338, 346,348-349, 354, 363, 368-369, 377-378, 380,382, 386, 390-392, 400-401, 403, 427-428,430-431, 434-435, 437

creation of, 41, 60, 377origins of, 32understanding, 35, 37, 86, 119, 363

State Department, 43, 86, 348-349, 378, 434-435state governments, 191state legislature, 390, 427statehood, 378states, 22, 24, 27-29, 34-36, 38, 40-42, 46, 54-56, 60,

70, 73, 77, 81-82, 92-94, 113, 118, 121,123-124, 128-129, 131, 139, 147, 153, 165,174, 191, 193, 203, 216-217, 224, 238, 244,248-249, 251, 255, 258, 274, 276, 278, 281,284, 290-291, 303-304, 307-308, 310, 318,335, 337, 343, 345, 347-350, 354, 356-357,361, 364-365, 367-369, 374, 376, 378-379,381, 390, 402-404, 411, 419-420, 427, 434,436-438

Stock exchanges, 138Stone, Deborah, 4, 22, 191, 314, 335-336strikes, 19-20, 27, 131, 149, 255, 262, 438Suburbs, 420Supreme Court, 43Supreme Court justices, 43Surplus, 37, 421surpluses, 421Surveys, 16, 37, 209, 228, 255, 267, 324, 333, 370,

374, 429Sweden, 258Switzerland, 265

TTax, 23, 43, 154, 178-179, 184, 186, 194-195, 216,

234, 262, 338, 390, 427income, 23, 154, 184, 186, 194-195, 216, 262, 338

Tax burden, 194, 216Taxation, 368taxes, 23, 43, 120, 178, 184, 191, 239, 338, 344-345,

390sales, 23, 184

Technocrats, 58, 60, 257technology, 8, 32, 35, 37, 41, 43, 55, 59-61, 100-101,

103, 108, 126, 138, 165, 173-174, 176, 208,228-230, 257, 259, 267, 337, 381, 388, 401,431, 443

telecommunications, 55, 61Television, 438Terms, 3, 5, 8, 10, 12, 20, 23, 27, 46, 51, 61-63, 69-70,

77-79, 86, 94, 101-102, 105, 122, 131, 141,147, 152, 154, 162, 168, 177, 192, 197-198,200-201, 210-211, 213-214, 216, 218-220,238, 254-255, 258-259, 262, 265, 278, 280,295, 312, 314, 318, 322, 325, 328-329, 334,343, 349, 355-356, 363, 372-373, 376, 380,389, 391, 396-397, 415, 420, 426, 431, 437

Territory, 88terrorism, 354, 376-377, 389The Internet, 18, 61, 64The Netherlands, 36, 75, 367The View, 72, 315, 378, 405Think tank, 64Think tanks, 15, 40-41, 100, 127, 250Third parties, 226Third World, 380Time, 9, 15-17, 22-24, 26-27, 29, 31, 34-35, 38, 40,

42, 45, 48-49, 51-52, 60, 63-64, 67, 74,76-78, 82, 84, 86, 90, 97, 104, 106-107, 109,116, 118, 120, 122, 125, 128-135, 137-147,152, 157-158, 161-162, 171, 173-174, 176,178-179, 181, 191, 194-197, 202, 205, 214,216, 224-226, 228, 232-234, 238, 242-244,246, 247, 249-251, 255, 257-258, 262-264,271-274, 276, 278, 280, 282-283, 285-292,294, 296, 301, 305-307, 312, 323, 326-328,333-334, 337, 343, 345, 356-357, 359-361,368, 381, 386, 388, 390-391, 394-395,406-408, 410, 412, 416, 418, 421-423, 427,430-431, 433, 438-439, 444-449

Torture, 371Towns, 108, 368toxic wastes, 321trade, 4, 17, 21, 33, 52, 214, 241, 438, 444

international, 438training, 15, 39, 43, 60, 195, 210, 218, 226-227, 236,

239, 266, 291, 294, 430-431, 443transfer payments, 95Transportation, 23-24, 27, 62, 86-87, 109, 112, 153,

165, 174, 184-186, 199-200, 206, 210, 218,228, 236, 243, 259, 308, 338, 344, 390, 410,427, 431

Treaty, 343Trial, 96-97, 103, 343trials, 97Tribe, Laurence, 79Turkey, 378Turnover, 255, 303

UUnderclass, 36Unemployment, 5, 21, 26, 30, 35, 69-70, 86, 96, 104,

113, 137, 194, 238, 240, 309, 374, 376,429-430

Unions, 124United Kingdom, 41-42, 59, 165, 258United Nations, 41, 201, 248-249, 257, 379, 390, 427United States, 22, 24, 27-29, 35-36, 38, 40-42, 54-55,

60, 73, 81-82, 92-94, 123-124, 139, 165,191, 217, 248-249, 276, 278, 281, 284,303-304, 307-308, 335, 337, 343, 348,356-357, 367-369, 374, 376, 378-379, 390,411, 419-420, 427, 434, 436, 438

Universities, 41, 108, 249-250, 253, 317British, 41

Urban areas, 21, 36, 138, 240, 274, 284Urban Institute, 225, 274, 390, 427Urbanization, 35, 135, 154, 430U.S. Bureau of the Census, 93, 100, 124, 249, 304User fees, 184, 186Utah, 381Utilitarianism, 320

Vvalues, 4-7, 13, 15-17, 32, 46, 51-52, 55-56, 59-60,

62-64, 67, 73-74, 79-81, 83, 85-86, 102, 104,112, 118-119, 121-123, 128, 130-135, 138,140-142, 144, 152-153, 155-156, 158,160-162, 165, 181, 183, 191-194, 196,202-206, 211, 216, 219, 223, 228, 233-236,

239-240, 242, 248, 252, 255-256, 259-260,263-264, 271-273, 280, 282, 287-288,305-306, 311-314, 316-319, 321-324,329-331, 335, 343, 347, 352, 357, 361,363-364, 368, 370, 377, 396, 401, 413-416,423

Verba, Sidney, 371Veto, 308, 390, 427Victim, 78Vietnam War, 52Violence, 380Virginia, 153, 381Vote, 308, 336, 379, 381, 405

choice, 336voting, 46, 146

Wwages, 194-195, 226, 280-281, 335, 337

minimum wage, 194-195, 337Wall Street Journal, 109War, 16, 33-34, 37-39, 45, 52, 60, 82, 92, 99, 102,

127, 138, 240, 262, 343, 371-374, 377-382,402

declaration of, 377War on terror, 240Warner, John, 381Warrant, 7, 18-20, 27, 106-107, 125-127, 139, 148,

165, 224-225, 237, 291-292, 315, 342-357,359-361, 363-369, 381, 402-406

Warrants, 18-20, 22, 106, 342-343, 346, 349, 376Washington, D.C., 409, 419Washington, George, 38, 51Washington Post, 24, 109, 308Washington University, 38wealth, 40, 61, 77, 96, 276-277, 314, 316, 335weapons, 218, 354

nuclear, 218Weapons systems, 218Weber, Max, 34Welfare, 9, 38, 41, 43, 55, 61-62, 92-93, 111, 117, 123,

130, 153-154, 195, 202-206, 210-211,218-219, 235, 249-250, 252, 257, 259, 262,267, 278, 316, 320-322, 351, 368-370, 375,389, 407, 431-432

welfare policies, 55Welfare reform, 262, 278, 375Western Electric, 263Westinghouse, 265White House, 108, 380, 435-436White House staff, 435Whites, 185, 416-417Wildavsky, Aaron, 3, 57, 116, 368Wilson, Woodrow, 37Wisconsin, 3, 93, 377WISH, 81-82, 135, 155, 259, 272, 277-278, 280, 288,

331, 354, 381, 437women, 43, 111, 123, 216, 238, 252, 258-259, 364,

366-367, 390, 427, 446-447, 449Workers, 35-37, 62, 123-124, 194-195, 250, 252, 263,

280, 284Workforce, 37, 93World Bank, 41, 248, 336, 343World Health Organization, 123

WHO, 123World Trade Center, 52, 438World War I, 37World War II, 38-39, 262, 374, 382

YYale University, 2, 4, 18, 22, 59, 63, 79, 90, 124, 196,

204, 314-315, 336, 340, 351, 377, 387, 401Yugoslavia, 348, 378


Documents

1VCMJD1PMJDZOBMZTJT %VOO 'JGUI&EJUJPOirpublicpolicy.ir/wp-content/uploads/2017/09/Public-Policy-Analysis... · normative, a term that refers to value judgments about what ought to