The Standard Error of Regressionsfaculty.smu.edu/millimet/classes/eco7377/papers/mccloskey ziliak.pdf · Volume I, Zvi Griliches and Michael D. Intriligator, ed. 1983). In the 762

Journal of Economic LiteratureVol. XXXIV (March 1996), pp. 97-114

The Standard Error of Regressions

By D E I R D R E N . M C C L O S K E Y

andSTEPHEN T. ZILIAK

University of Iowa

Suggestions by two anonymous and patient referees greatly improved the paper. Our thanksalso to seminars at Clark, Iowa State, Harvard, Houston, Indiana, and Kansas State universi-ties, at Williatns College, and at the universities of Virginia and Iowa. A colleague at Iowa,Calvin Siehert, was materially helpful.

THE IDEA OF Statistical significance isold, as old as Cicero writing on fore-

casts (Cicero, De Divinatione, 1. xiii. 23).In 1773 Laplace used it to test whethercomets came from outside the solar sys-tem (Elizabeth Scott 1953, p. 20). Thefirst use of the very word "significance"in a statistical context seems to be JohnVenn's, in 1888, speaking of differencesexpressed in units of probable error;

They inform us which of the differences inthe above tables are permanent and signifi-cant, in the sense that we may be tolerablyconfident that if we took another similarbatch we should find a similar difference; andwhich are merely transient and insignificant,in the sense that another similar batch isabout as likely as not to reverse the conclu-sion we have obtained. (Venn, quoted in Lan-celot Hogben 1968, p. 325).

Statistical significance has been muchused since Venn, and especially sinceRonald Fisher.

The problem, and our main point, isthat a difference can be permanent (asVenn put it) without being "significant"in other senses, such as for science orpolicy. And a difference can be signifi-

cant for science or policy and yet be in-significant statistically, ignored by theless thoughtful researchers.

In the 1930s Jerzy Neyman and EgonS. Pearson, and then more explicitlyAbraham Wald, argued that actual inves-tigations should depend on substantivenot merely statistical significance. In1933 Neyman and Pearson wrote of typeI and type II errors:

Is it more serious to convict an innocent manor to acquit a guilty? That will depend on theconsequences of the error; is the punishmentdeath or fine; what is the danger to the com-munity of released criminals; what are thecurrent ethical views on punishment? Fromthe point of view of mathematical theory allthat we can do is to show how the risk oferrors may be controlled and minimised. Theuse of these statistical tools in any given case,in determining just how the balance shouldbe struck, must be left to the investigator.(Neyman and Pearson 1933, p. 296; italicssupplied)

Wald went further:

The question as to how the form of theweight [that is, loss] function . . . should bedetermined, is not a mathematical or statisti-cal one. The statistician who wants to test

97

98 Journal of Economic Literature, Vol. XXXIV {March 1996)

certain hypotheses must first determine therelative importance of all possible errors,which will depend on the special purposes ofhis investigation. (1939, p, 302, itahcs sup-plied)

To date no empirical studies havebeen undertaken measuring the use ofstatistical significance in economics. Wehere examine the alarming hypothesisthat ordinary usage in economics takesstatistical significance to be the same aseconomic significance. We compare sta-tistical best practice against leading text-books of recent decades and against thepapers using regression analysis in the1980s in the American Economic Review.

I, An Example

The usual test of purchasing powerparity regresses prices at home (P) onprices abroad {P*), allowing for the ex-change rate {e). Thus: P = a + ^ {eP*) +error term (cf,, McCloskey and J, Rich-ard Zecher 1984), The equation can bein levels or rates of change or in somemore complex functional form. An esti-mated coefficient P is of course a ran-dom variate, and the accuracy of its esti-mated mean depends on the propertiesof the error term, the specification of themodel, and so forth. But to fix ideas sup-pose that all the usual econometric prob-lems have been solved. In tests of pur-chasing power parity the null hypothesisis usually thought of as "P equal to 1,0,"Suppose an unbiased estimator of Pyields not exactly 1,000 but very close,say 0,999, That is, prices at home rise byvery nearly the same rate as pricesabroad for most purposes of science orpolicy, (Not for all purposes: the point ofthinking as Wald did in terms of lossfunctions is that for some purposes a dif-ference that in some metric looks "small"might in another metric be important,)If the sample size were large enough,however, even a coefficient of 0,999

might prove to be a statistically signifi-cant divergence from exactly 1,000, Un-der purely statistical procedures the in-vestigator would conclude, as many have,that the hypothesis of purchasing powerparity had failed.

The hypothesis does not in truth pre-dict that the coefficient will be 1,000 tomany decimal places. It predicts that Pwill be "about 1," The econoniically rele-vant null hypothesis is a range around1,000, not the point itself in isolationfrom its neighborhood. The investigatorwould not want to assert that if p = 0,999with a standard error of 0,00000001 weshould abandon purchasing power parity,or run our models of the American econ-omy without the world price level. Yetthe literaiture on purchasing power parityhas ordinarily used the null of 1,000 ex-actly. The procedure is not defensible instatistical theory. The table of t will nottell what is "close," Closeness depends,in Wald's words, on the special purposesof the investigation—good enough for in-flation control, say, if P = 0,85, thoughnot good enough to make money on theforeign exchange market unless P =0,99998,

Just how the balance should be struck,as Neyman and Pearson put it, must beleft to the investigator, A coefficient of0,15, say, would for most purposes reject"P = about 1," Accepting the null hy-pothesis may be reasonable or unreason-able, but it depends on economic con-text. The point is that it does not mainlydepend on the value of the test statistic.The uncertainty of the estimate thatarises from sampling error—the onlykind of uncertainty the test of signifi-cance deals with—is still of scientific in-terest. But low or high uncertainty (moreor less "permanence" in Venn's terms)does not by itself answer the questionhow important the variable is, how largeis large. In tests of purchasing powerparity, for example, one should ask if P =

McCloskey and Ziliak: The Standard Error of Regressions 99

0.999 is close enough for scientific pur-poses to the null. How should the answerbe adjusted if there are 20,000 observa-tions (cf. Richard Rudner 1953, p. 3;Scott Gordon 1991, pp. 664-65)? If theestimate is not taken to be close to thenull, what makes it "interestingly differ-ent" or what is the "scientific intuition"of one's "public" (Edwin Boring 1919, p.337)? These are not easy questions. Butthey are the questions relevant to scien-tific discovery.

II. The Evidence: Textbooks

The late Morris DeCroot ([1975]1989), a statistician with sophistication ineconomics, was emphatic on the point:

It is extremely important . . . to distinguishbetween an observed value of U that is statis-tically significant and an actual value of theparameter . . . In a given problem, the tailarea corresponding to the observed value ofU might be very small; and yet the actualvalue . . . might be so close to [the null] that,for practical purposes, the experimenterwould not regard [it] as being [substantively]different from [the null], (p. 496)[I]t is very likely that the (-test based on thesample of 20,000 will lead to a statisticallysignificant value of (7 . . . [The experimenter]knows in advance that there is a high prob-ability of rejecting [the null] even when thetrue value . . . differs [arithmetically] onlyslightly from [the null], (p. 497)

But few other econometrics textbooksdistinguish economic significance fromstatistical significance. And fewer em-phasize economic significance. In theeconometrics texts widely used in the1970s and 1980s, when the practice wasbecoming standard, such as Jan Kmenta'sElements of Econometrics (1971) andJohn Johnston's Econometric Methods([1963] 1972, 1984), there is no mentionof economic as against statistical signifi-cance. Peter Kennedy, in his A Guide toEconometrics (1985), briefly mentionsthat a large enough sample always givesstatistically significant differences. This

is part of the argument but not all of it,and Kennedy in any case relegates thepartial argument to an endnote (p. 62).

Among recent econometrics books Ar-thur Goldberger's is the most explicit.His A Course in Econometrics (1991)gives the topic "Statistical versus Eco-nomic Significance" a page of text (pp.240-41), quoting McCloskey's little arti-cle of 1985. Coldberger's page has beennoticed as unusual. Clive Granger, re-viewing in the March 1994 issue of thisJournal four leading books (Coldberger;Russell Davidson and James C. MacKin-non 1993; William H. Creene 1993; Wil-liam E. Griffiths, R. Carter Hill, andGeorge G. Judge 1993), notes that

when the link is made [in Goldberger be-tween the economics and the technical statis-tics] some important insights arise, as for ex-ample the section discussing "statistical andeconomic significance," a topic not men-tioned in the other books. (1994, p. 118)

That is, most beginning econometricsbooks even now, unlike DeGroot andGoldberger and before them the modernmasters of statistics, do not contrast eco-nomic and statistical significance.

Nor do the present-day advancedhandbooks and textbooks. The three vol-umes of the Handbook of Econometricscontain one mention of the point, unsur-prisingly by Edward Leamer (p. 325 ofVolume I, Zvi Griliches and Michael D.Intriligator, ed. 1983). In the 762 pagesof the recent companion work. Volume11 of the Handbook of Statistics (1993),there is one sentence about the levelof the test in its relation to samplesize (Jean-Pierre Florens and MichelMouchart 1993, p. 321).

One might defend contemporary usageby arguing that the advanced texts as-sume their readers already grasp the dif-ference between economic and statisticalsignificance. Economy of style woulddictate the unqualified word "signifi-cance," its exact meaning, economic or


statistical, to be supplied by the sophisti-cated reader. Under such a hypothesisthe contemporary usage would be nomore than a shorthand way to refer to anestimated coefficient. The impliedreader would be educated enough tosupply the appropriate caveats about eco-nomic significance.

The hypothesis is not borne out by theevidence. To take one example amongmany, Takeshi Amemiya's advanced text-book in econometrics does not itselfdraw a distinction between economicand statistical significance (Amemiya1985), The book makes little claim toteaching empirical methods, but presum-ably the theory of econometrics is sup-posed to connect to empirical work,Amemiya recommends that the studentprepare "at the level of Johnston, 1972"(preface). Does the recommendationcover the matter of statistical versus sub-stantive significance?

No, Johnston as we have seen makesno mention of the point. He uses theterm "economic significance" once only,without contrasting it to the statisticalsignificance on which he lavishes atten-tion: "It is even more difficult to attacheconomic significance to the linear com-binations arising in canonical correlationanalysis than it is to principal compo-nents" (p, 333), In an extended exampleof hypothesis testing, spanning pages 17to 43, Johnston tests in the conventionalway the hypothesis that "sterner penal-ties" for dangerous driving caused fewerroad deaths, concluding "[t]he computedvalue [of the f-statistic] is suggestive of areduction, being significant at the 5 percent, but not at the one per cent, level"(p, 43, italics supplied). He is saying thatat a high level of rigor the policy ofsterner penalties might be doubted tohave desirable effects. Statistically theusage is unobjectionable (except that heuses the universe of road casualtiesin the United Kingdom 1947-1957 as

though it were a sample of size 11 fromsome universe). But the 100,000 livesthat were saved in the reduction asmeasured are not acknowledged as "sig-nificant," Johnston has merged statisticaland policy significance. At what level thesignificance level should be set, consid-ering the human cost of ignoring the ef-fect of sterner penalties, is none ofJohnston's concern. He leaves the ques-tion of how large is large to statistics. AsWald said in 1939, however, the question"is not a mathematical or statistical one,"

Johnston does recommend "TheCaimcross Test" (1984, pp, 509-10).That is, after computing assorted teststatistics the researcher should ask if themodel would satisfy the discerning judg-ment of Sir Alec Caimcross, "Would SirAlec be willing to take this model to theRiyadh?" But that is our point. If judg-ments about economic significance arenot made at the keyboard they need tobe brought into the open, before reach-ing Sir Alec, The researcher wastes thetime of Caimcross if the statistically sig-nificant does not correspond to what SirAlec, and the Riyadh, want: economicsignificance,

A tenacious defender of contemporaryusage might argue further that Johnston,in turn, presumes the reader already un-derstands the difference between eco-nomic and statistical significance, havingacquired it in elementary courses on sta-tistics. The argument is testable. In hispreface Johnston directs the reader whohas difficulty with his first chapter toexamine a "good introductory" book onstatistics, mentioning Paul G, Hoel's In-troduction to Mathematical Statistics(1954), Alexander M, Mood's Introduc-tion to the Theory of Statistics (1950),and Donald A, S, Fraser's Statistics: AnIntroduction (1958) (p, ix). These arefine books: Mood, for example, givesa good treatment of power functions,pointing to their relevance in applied


work. But none of them make a distinc-tion between substantive and statisticalsignificance. Hoel writes that

[t]here are several words and phrases used inconnection with testing hypotheses thatshould be brought to the attention of stu-dents. When a test of a hypothesis produces asample value falling in the critical region ofthe test, the result is said to be significant;otherwise one says that the result is not sig-nificant, (p. 176, his italics)

The student from the outset of her statis-tical education, therefore, is led to be-lieve that economic (or substantive) sig-nificance and statistical significance arethe same thing. Hoel explains: "Thisword ['not significant'] arises from thefact that such a sample value is not com-patible with the hypothesis and thereforesignifies that some other hypothesis isnecessary" (p. 176). The elementarypoint that "[tjhere is no sharp border be-tween 'significant' and 'insignificant,'only increasingly strong evidence as theP-value decreases" (David S. Moore andGeorge P. McCabe 1993, p. 473) is notfound in most of the earlier books fromwhich most economists learned statisticsand econometrics. The old classic by W.Allen Wallis and Harry V. Roberts, Sta-tistics: A New Approach, first publishedin 1956, is an exception:

It is essential not to confuse the statistical us-age of "significant" with the everyday usage.In everyday usage, "significant" means "ofpractical importance," or simply "important."In statistical usage, "significant" means "sig-nifying a characteristic of the populationfrom which the sample is drawn," regardlessof whether the characteristic is important.(Wallis and Roberts [1956] 1965, p. 385)

The point has been revived in elemen-tary statistics books, though most stilldo not emphasize it. In their leadingelementary book the statisticians DavidFreedman, Robert Pisani, and RogerPurves (1978) could not be plainer. In

one of numerous places where theymake the point they write:

This chapter . . . explains the limitations ofsignificance tests. The first one is that "sig-nificance" is a technical word. A test can onlydeal with the question of whether a differ-ence is real [permanent in Venn's sense], orjust a chance variation. It is not designed tosee whether the difference is important, (p.487, italics supplied)

The distinction is also emphatic inRonald J. Wonnacott and Thomas H.Wonnacott (1982, p. 160) and in Mooreand McCabe (1993, p. 474).

III. The Instrument: A Survey ofPractice in Significance

The evidence, then, is that econo-metricians are not in their textbooksemphasizing the difference betweeneconomic significance and statistical sig-nificance. What is practice?

We take the full-length papers pub-lished in the American Economic Reviewas an unbiased selection of best practice(we will not say "sample" and will nottherefore use tests of statistical signifi-cance). We read all the 182 papers in the1980s that used regression analysis (andrecord our impression that in most mat-ters these are superb examples of eco-nomic science). Each paper was asked 19questions about its use of statistical sig-nificance, to be answered "yes" (soundstatistical practice) or "no" (unsoundpractice) or "not applicable."

The survey questions are:1. Does the paper use a small number

of observations, such that statisticallysignificant differences are not found atthe conventional levels merely by choos-ing a large number of observations? Thepower of a test is high if the significancelevel at N = 30,000 is carried over fromsituations in which the sample is 30 or300. For example, in Glen C. Blomquist,Mark C. Berger, and John P. Hoehn, N =


34,414 housing units and 46,004 indi-viduals (Mar, 1988, p, 93), At such largesample sizes the authors need to pay at-tention to the tradeoff between powerand the size of the test, and to the eco-nomic significance of the power againstalternatives,

2, Are the units and descriptive statis-tics for all regression variables included?Empirical work in economics is measure-ment. It is elementary to include units ofthe variables, and then also to givemeans,

3, Are coefficients reported in elastic-ity form, or in some interpretable formrelevant for the problem at hand andconsistent with economic theory, so thatreaders can discern the economic impactof regressors? Wallis and Roberts longago complained that "sometimes authorsare so intrigued by tests of significancethat they fail even to state the actualamount of the effect, much less to ap-praise its practical importance" (1956, p,409), In some fields (not much in eco-nomics, though we did find one example)the investigator will publish tables thatconsist only of asterisks indicating levelsof significance,

4, Are the proper null hypothesesspecified? The commonest problemwould be to test against a null of zerowhen some other null is to the point.Such an error would be the result of al-lowing a canned program to make scien-tific decisions. If a null hypothesis is Pi +p2 = 1, there is not much to be gainedfrom testing the hypothesis that each co-efficient is statistically significantly dif-ferent from zero. The most fruitful appli-cation of the Neyman-Pearson testspecifies the null hypothesis as some-thing the researcher believes to be true.The only result that leads to a definitiveconclusion is a rejection of the null hy-pothesis. Failing to reject does not ofcourse imply that the null is thereforetrue. And rejecting the null does not im-

ply that the alternative hypothesis istrue: there may be other alternatives (arange that investigators agree is relevant,for example) which would cause rejec-tion of the null. The current rhetoric ofrejection promotes a lexicographic pro-cedure of "regress height income coun-try age"; inspect f-values; discard as un-important if f < 2; circulate as importanti f t > 2 ,

5, Are coefficients carefully inter-preted? Goldberger has an illustrationsimilar to many issues in economic policy(Goldberger 1991, p, 241), Suppose thedependent variable is "weight inpounds," the large coefficient is on"height," the smaller coefficient is on"exercise," and the estimated coefficientshave the same standard errors. Neitherthe physician nor the patient wouldprofit from an analysis that says height is"more important" (its coefficient beingmore standard errors away from zero inthis sample), offering the overweight pa-tient in effect the advice that he's nottoo fat, merely too short for his weight,"The moral of this example is that statis-tical measures of 'importance' are a di-version from the proper target of re-search—estimation of relevantparameters—to the task of 'explainingvariation' in the dependent variable"(Goldberger, p, 241),

6, Does the paper eschew reporting allt- or F-statistics or standard errors, re-gardless of whether a significance test isappropriate? Statistical computing soft-ware routinely provide f-statistics forevery estimated coefficient. But that pro-grams provide it does not mean that theinformation is relevant for science. Wesuspect that referees enforce the prolif-eration of meaningless t- and F-statistics,out of the belief that statistical and sub-stantive significance are the same,

7, Is statistical significance at the firstuse, commonly the scientific crescendo ofthe paper, the only criterion of "impor-


tance"? By "crescendo" we mean thatplace in the paper where the authorcomes to what she evidently considersthe crucial test.

8. Does the paper mention the powerof the tests? For example, Frederic S.Mishkin does, unusually, in two foot-notes (June 1981, pp. 298 nil , 305 n27;lack of power is a persistent difficulty incapital-market studies, but is seldomfaced). As DeGroot pointed out, thepower of a test may be low against anearby and substantively significant al-ternative. On the other hand, power maybe high against a nearby and trivial alter-native.

9. If the paper mentions power, doesit do anything about it? It is true thatpower can only be discussed relative toan explicit alternative hypothesis, makingpower analysis difficult for some of thealternatives. An example is the Durbin-Wu-Hausman test for whether two esti-mators are consistent. (The survey ac-counts for the difficulty by coding therelevant papers "not applicable.")

10. Does the paper eschew "asteriskeconometrics," that is, ranking the coeffi-cients according to the absolute size of^-statistics?

11. Does the paper eschew "signeconometrics," that is, remarking on thesign but not the size of the coefficients?There is a little statistical theory in theeconometrics books lying behind thiscustomary practice (Goldberger, ch. 22;Greene, ch. 8), though for the most partthe custom outstrips the theory. But signis not economically significant unless themagnitude is large enough to matter.Statistical significance does not tellwhether the size is large enough to mat-ter. It is not true, as custom seems to bearguing, that sign is a statistic indepen-dent of magnitude.

12. Does the paper discuss the size ofthe coefficients? That is, once regressionresults are presented, does the paper

make the point that some of the coeffi-cients and their variables are economi-cally influential, while others are not?Blomquist, Berger, and Hoehn do inpart, by giving their coefficients on hous-ing and neighborhood amenities in dollarform. But they do not discuss whetherthe magnitudes are scientifically reason-able, or in some other way important.Contrast Christina Romer, in a 19-page,exclusively empirical paper: "Indeed,correcting for inventory movements re-duces the discrepancy . . . by approxi-mately half. This suggests that inventorymovements are [economically] impor-tant" (June 1986, p. 327). M. Boissiere,J. B. Knight, and R. H. Sabot reflect themore typical practice: "In both countries,cognitive achievement bears a highly sig-nificant relationship to educational level. . . In Kenya, secondary education raisesH by 11.75 points, or by 35 percent ofthe mean" (Dec. 1985, p. 1026). Theymake ambiguous use of the word "sig-nificance," then draw back to the rele-vant question of economic significance.Later in the paragraph they recur to de-pending on statistical significance alone:"significantly positive" and "almost sig-nificantly positive" become again theironly criteria of itnportance.

Daniel Hamermesh, by contrast, esti-mates his crucial parameter K, and at thefirst mention says, "The estimates of Kare quite large, implying that the firmvaries employment only in response tovery large shocks. . . . Consider what anestimate this large means" (Sept. 1989,p. 683). The form is here close to ideal:it gets to the scientific question of whatthe size of a magnitude means. Twoparagraphs down he speaks of "fairlylarge," "very important," "small," and"important" without merging these withstatistical significance. In Goldberger'sterms, he focuses on "the proper targetof research—estimation of relevant pa-rameters." (Later, though, Hamermesh


falls back to average practice: "The K forthe aggregated data in Table 2 are insig-nificant," though he adds wisely, "andvery small; and the average values of thep are much higher than in the pooleddata"; p. 685.)

13, Does the paper discuss the scien-tific conversation within which a coeffi-cient would be judged "large" or "small"?Romer, for example, remarks that "Theexistence of the stylized fact [that is, thescientific consensus] that the economyhas stabilized implies a general consen-sus" (p, 322),

14, Does the paper avoid choosingvariables for inclusion solely on the basisof statistical significance? The standardargument is that if certain variables en-ter the model significantly, the informa-tion should not be spumed. But such anargument merges statistical and substan-tive significance,

15, After the crescendo, does the pa-per avoid using statistical significance asthe criterion of importance? The refe-rees will have insisted unthinkingly on asignificance test, the prudent author willhave acceded to their insistence, but willafter reporting them turn to other andscientifically relevant criteria of impor-tance,

16, 7s statistical significance decisive,the conversation stopper, conveying thesense of an ending? Romer and JeffreySachs (Mar, 1980) both use statisticalsignificance, and misuse it—in bothcases looking to statistical significance asa criterion for how large is large. But inneither paper does statistical significancerun the empirical work. The misuse inMichael Darby (June 1984) is balder: hisonly argument for a coefficient when heruns a regression is its statistical signifi-cance (pp, 311, 315), but on the otherhand his findings do not turn on the re-gression results,

17, Does the paper ever use a simula-tion (as against a use of the regression as

an input into further argument) to deter-mine whether the coefficients are reason-able? To some degree Blomquist, Ber-ger, and Hoehn do. They simulate therankings of cities by amenity, and if thecoefficients were quite wrong the rank-ings would be themselves unreasonable,Santa Barbara does rank high, thoughthe differential value of amenities worstto best, at $5,146, seems low (Mar, 1988,p, 96), Simulations using regression coef-ficients can be informative, but of courseshould not use statistical significance as ascreening device for input,

18, In the "conclusions" and "implica-tions" sections, is statistical significancekept separate from economic, policy, andscientific significance? In Boissiere,Knight, and Sabot (Dec 1985) the effectof ability is isolated well, but the eco-nomic significance is not argued,

19, Does the paper avoid using theword "significance" in ambiguous ways,meaning "statistically significant" in onesentence and "large enough to matter forpolicy or science" in another? ThusDarby (June 1984): "First we wish to testwhether oil prices, price controls, orboth has a significant influence on pro-ductivity growth" (p, 310), The meaningsare merged.

IV, Results of the Survey of theAmerican Economic Review

Some of the AER authors, such asRomer and Hamermesh, show that theyare aware of the substantive importanceof the questions they ask, and of the fu-tihty of relying on a test of statistical sig-nificance for getting answers. Thus KimB, Clark: "While the union coefficient inthe sales specification is twice the size ofits standard error, it is substantivelysmall; moreover, with over 4,600 obser-vations, the power of the evidence thatthe effect is different from zero is not


TABLE 1THE AMERICAN ECONOMIC REVIEW IN THE 1980S HAB NUMEROUS ERRORS IN THE USE OF STATISTICAL

SIGNIFICANCE

Survey QuestionDoes the Paper , , ,8. Gonsider the power of the test?

6. Eschew reporting all standard errors, t-, and F-statistics, whensuch infonnation is irrelevant?

17. Do a simulation to detennine whether the coefficients arereasonable?

9. Examine the power function?13. Discuss the scientific conversation within which a coefficient

would be judged large or small?16. Gonsider more than statistical significance decisive in an empirical

argument?18. In the conclusions, distinguish between statistical and substantive

significance?2. Report descriptive statistics for regression variables?

15. Use other criteria of importance besides statistical significanceafter the crescendo?

19. Avoid using tlie word "significance" in ambiguous ways?

5. Carefully interpret coefficients? For example, does it pay attentionto the details of the units of measurement, and to the limitationsof the data?

11. Eschew "sign econometrics," remarking on the sign but notthe size of the coefficients?

7. At its first use, consider statistical significance to be one amongother criteria of importance?

3. Report coefficients in elasticities, or in some other useful formthat addresses the question of "how large is large"?

14. Avoid choosing variables for inclusion solely on the basis ofstatistical significance?

10. Eschew "asterisk econometrics," the ranking of coefficientsaccording to the absolute size of the test statistic?

12. Discuss the size of the coefficients?1. Use a small number of observations, such that statistically

significant differences are not found merely by choosing a verylarge sample?

4. Test the null hypotheses that the authors said were the ones of 180 97.3interest?

Source for Tables 1-5: All full-length papers using regression analysis in the American Economic Review, 1980-1989, excluding the Proceedings.Notes: "Percent Yes" is the total number of Yes responses divided by the relevant number of papers (neverexceeding 182). Some questions are not generally applicable to particular papers and some questions are notapplicable because they are conditional on the paper having a particular characteristic. Question 3, for example, wascoded "not applicable" for papers which exclusively use nonparametric statistics. Question 19 was coded "notapplicable" for papers that do not use the word "significance."

Total for which thequestion applies

182

181

179

12181

182

181

178182

180

181

181

182

173

180

182

182182

PercentYes

4.4

8.3

13.2

16.728.0

29.7

30.1

32.440.7

41.2

44.5

46.7

47.3

66.5

68.1

74.7

80.285.7


overwhelming" (Dec 1984, p, 912), AndCriliches:

Here and subsequently, all statements aboutstatistical "significance" should not be takenliterally. Besides the usual issue of data min-ing clouding their interpretation, the "sam-ple" analyzed comes close to covering com-pletely the relevant population. Tests ofsignificance are used here as a metric for dis-cussing the relative fit of different versions ofthe model. In each case, the actual magni-tude of the estimated coefficients is of moreinterest than their precise "statistical signifi-cance," (Dec 1986, p, 146)

Griliches understands that populationsshould not be treated as samples, andthat statistical significance is not a sub-stitute for economic significance, (Hedoes not say why statistical significanceis a scientifically relevant "metric for dis-cussing the relative fit of the differentversions of the model,")

But most authors in the AER do notunderstand these points. The results ofapplying the survey to the papers of the1980s are displayed in Table 1,

The principal findings of the surveyare:

• 70 percent of the empirical papersin the American Economic Reviewpapers did not distinguish statisticalsignificance from economic, policy,or scientific significance,

• At the first use of statistical signifi-cance, typically in the "Estimation"or "Results" section, 53 percent didnot consider anything but the size oft- and F-statistics, About one thirdused only the size of t- and F-teststatistics as a criterion for the inclu-sion of variables in future work,

• 72 percent did not ask "How large islarge?" That is, after settling on anestimate of a coefficient, 72 percentdid not consider what other authorshad found; they did not ask whatstandards other authors have usedto determine "importance"; they did

not provide an argument one way oranother whether the estimate p =0,999 is economically close to 1,0and economically important eventhough "statistically different fromone," Awareness that scientific in-quiry takes place in a conversationabout how large is large seemed toimprove the econometric practice.Of 131 papers that did not mentionthe work of other authors as a quan-titative context for their own, 78percent let statistical significancedecide questions of substantive sig-nificance. Of 50 papers that didmention the work of other authorsas a context, only 20 percent let sta-tistical significance decide,

• 59 percent used the word "signifi-cance" in ambiguous ways, at onepoint meaning "statistically signifi-cantly different from the null," atanother "practically important" or"greatly changing our scientificopinions," with no distinction,

• Despite the advice proffered intheoretical statistics, only 4-percentconsidered the power of their tests.One percent examined the powerfunction,

• 69 percent did not report descriptivestatistics—the means of the regres-sion variables, for example—thatwould allow the reader to make ajudgment about the economic sig-nificance of the results,

• 32 percent admitted openly to usingstatistical significance to drop vari-ables (question 14), One would haveto have more evidence than explicitadmissions to know how prevalentthe practice is in fact. One-third is alower bound,

• Multiple-author papers, as onemight expect from the theory ofcommon property resources, more


TABLE 2MULTIPLE AUTHORS APPEAR TO HAVE COORDINATION PROBLEMS, MAKINC THE ABUSES WORSE

MEASURED BY PERCENT YES

Survey QuestionMultiple

Author PapersSingle

Author Papers

Does the paper . . .7. At its first use, consider statistical significance to be one among

other criteria of importance?10. Eschew "asterisk econometrics," the ranking of coefficients

according to the absolute size of the test statistic?12. Discuss the size of the coefficients?1. Use a small number of observations, such that statistically

significant differences are not found merely by choosing a verylarge sample?

42.2

68.8

76.777.8

53.4

79.2

84.184.8

Notes: "Percent Yes" is the total number of Yes responses divided by the relevant number of papers.

often spoke of "significance" in am-biguous ways, used sign econo-metrics, did not discuss the size ofestimated coefficients, and foundnothing more than the size of teststatistics to be of importance at thefirst use of statistical significance(Table 2).

• Authors from "Tier 1" schools did insome respects a little better, butwhether the difference justifies theinvidious terminology of "tiers" is ascientific, not a statistical, questionand must be left to the investigator(Table 3; the terminology is that ofthe most recent National ResearchCouncil assessment and includesChicago, Harvard, MIT, Princeton,Stanford, and Yale.)

Though we do not here report the re-sults, we found on the other hand thatpapers written by faculty at Tier 1schools were proportionally more likelyto use sampling theory on entire popula-tions, and to treat as probability sampleswhat are in fact samples of convenience.

The substantive significance of suchpractices can be made more vivid by ex-

amining a few of the papers in somedepth.

The first is a case of not thinkingabout the economic meaning of a coeffi-cient. The authors estimate benefit-costratios for the state of Illinois followingthe implementation of an unemploymentinsurance experiment. In one experi-ment a control group was given a cashbonus for getting a job quickly and keep-ing it for several months. In another ex-periment, the "Employer Experiment,"employers were given a cash-bonus ifclaimants found a job quickly and re-tained it for some specified amount oftime (Sept. 1987, p. 517). The intent ofthe "Employer Experiment" was to "pro-vide a marginal wage-bill subsidy, ortraining subsidy, that might reduce theduration of insured unemployment" (p.'517). Here is how the conclusion is pre-sented:

The fifth panel also shows that the overallbenefit-cost ratio for the Employer Experi-ment is 4.29, but it is not statistically differ-ent from zero. The benefit-cost ratio forwhite women in the Employer Experiment,however, is 7.07, and is statistically differentfrom zero. Hence, a program modeled on theEmployer Experiment also might be attrac-


TABLE 3AUTHORS AT TIER 1 DEPARTMENTS DO BETTER THAN OTHERS IN MANY CATEGORIES


Survey QuestionTierl

DepartmentsOther

Departments

Does the paper . . .

1, Use a small number of observations, such that statisticallysignificant differences are not found rherely by choosing a verylarge sample?

12, Discuss the size of the coefficients?10, Eschew "asterisk econometrics," the ranking of coefficients

according to the absolute size of the test statistic?

7, At its first use, consider statistical significance to be one amongother criteria of importance?

5, Carefully interpret coefficients? For example, does it pay attentionto the details of the units of measurement, and to the limitationsof the data?

19, Avoid using the word "significance" in ambiguous ways?18 In the conclusions, distinguish between statistical and substantitive

significance?

91,3

87,0

84,8

65,5

60,0

52,450,0

83,9

78,971,4

41,2

37,5

37,523,1

Notes: According to the most recent National Research Council assessment, the tier 1 departments are Chicago,Harvard, MIT, Princeton, Stanford, and Yale,"Percent Yes" is the total number of Yes responses divided by the relevant number of papers.

tive from the state's point of view if theprogram did not increase unemploymentamong nonparticipants. Since, however, theEmployer Experiment affected only whitewomen, it would be essential to understandthe reasons for the uneven effects of thetreatment on different groups of workers be-fore drawing conclusions about the efficacy ofsuch a program, (p, 527)

Here "affected" means that the esti-mated coefficient is statistically signifi-cantly different from a value the authorsbelieve to be the relevant one. The 4,29benefit-cost ratio for the whole Em-ployer Experiment is, according to theauthors, not useful or important for pub-lic policy. The 7,07 ratio for whitewomen is said to "affect"—to be impor-tant—because it passed an arbitrary sig-nificance test. That is, 7,07 affects, 4,29does not. It is true that 4,29 is a realiza-tion from a noisy random variable,whereas 7,07 is from a more quiet one.

Though the authors do not say so, the4,29 benefit-cost ratio is marginally dis-cernible from zero at about the 12 per-cent level (p, 527), Yet for policy pur-poses even a noisy benefit-cost ratio isworth talking about. The argument thatthe 4,29 figure does not "affect" is un-sound, and could be costly in employ-ment foregone.

Another paper offers "an alternativetest of the CAPM and report[s] , , , testresults that are free from the ambiguityimbedded in the past tests" (Jan, 1980, p,660), The authors are taking exception,they say, to Richard Roll's comment that"there is practically no possibility thatsuch a test can be accomplished in thefuture" (p, 660), So they test five hy-potheses: the intercept equals zero; theslope coefficients differ from zero; theadjusted coefficient of determinationshould be near one; there is no trend in


the intercept; and there is no trend inthe adjusted coefficient of determination(pp. 664-65). On several time-series theyrun least squares regressions to estimatecoefficients. Nowhere in the text is thesize of the estimated coefficients dis-cussed (a common mistake in the capital-market literature). Instead, the authorsrank their results according to the num-ber of times the absolute value of the t-statistic is greater than two (p. 667).Three out of four of their tables of esti-mation results have a column called "No.of Times t > 2," another coltimn with"Average i-statistics," and one with "Ad-justed R2" They do not report coeffi-cient estimates in the three tables,merely the t-statistics (Tables 1, 2, and 3,pp. 667-68). The only "Yes" that the pa-per earned in our survey was for specify-ing the null according to what their the-ory suggests.

Using ambiguously the very word "sig-nificance" implies there is no differencebetween economic significance and sta-tistical significance, that nothing or littleelse matters. Of the 96 papers that useonly the test of statistical significance asa criterion of importance at its first use,90 percent imply—or state—that it is de-cisive in an empirical argument, and 70percent use the word "significance" am-biguously. Of the other 86 papers in thesurvey less than half use the word am-biguously. The 96 unsound papers con-tinue making inappropriate decisions at ahigher rate than the 86 papers thatacknowledge some criterion other thanstatistical significance. Only seven of the96 distinguish statistical significancefrom economic or policy or scientific sig-nificance in the concltisions and implica-tions sections, while 47 of the 86 makethe distinction (Table 4).Here is an extreme case of ambiguity:

The statistically significant [read: (1) sam-pling theory] inequality aversion is in addi-tion to any unequal distribution of inputs re-

sulting from different social welfare weightsfor different neighborhoods. The KP resultsallowing for unequal concern yield an esti-mate of q of -3.4. This estimate is signifi-cantly [read: (2) some numbers are smallerthan others] less than zero, indicating aggre-gate outcome is not maximized. At the sametime, however, there is also significant [read:(3) a moral or scientific or policy matter] con-cern about productivity, as the inequality pa-rameter is significantly [read: (4) a joint ob-servation about morality and numbers]greater than the extreme of concern solelywith equity. (AER Mar. 1987, p. 46)

In a piece on Ricardian Equivalence, sta-tistical significance decides nearly every-thing:

Notice the least significant of the variables inthe constrained estimation is the secondlagged value of the deficit in the governmentpurchases equation. A natural course wouldbe to reestimate the modal for the case oftwo lagged values of government spendingand one lagged value of the government defi-cit. . . . Although the elimination of [the vari-able] raises the confidence level at which thenull hypothesis can be rejected, it remainsimpossible to argue that the data providesevidence against the joint proposition of Ri-cardian equivalence and rational expectationsat conventional levels of significance. (AERMar. 1985, p. 125)

Another paper reports "significant" re-sults on the relation between unemploy-ment and money:

The coefficient is significant at the 99 per-cent confidence level. Neither the currentmoney shock nor all 12 coefficients as agroup are significantly different from zero.The coefficient on c is negative and signifi-cant and the distributed lag on c is significantas well. In column (2) we report a regressionwhich omits the insignificant lags on moneyshocks. The c distributed lag is now signifi-cant at the 1 percent confidence level. . . .We interpret these results as indicating that

the primary factor determining cyclical vari-ations in the probability of leaving unemploy-ment is probably heterogeneity. Inventory in-novations appear to play some role andsurprisingly, money shocks have no signifi-cant impact. (AER Sept. 1985, p. 630)


TABLE 4IF ONLY STATISTICAL SIGNIFICANCE IS SAID TO BE OF IMPORTANCE AT ITS FIRST USE (QUESTION 7),

THEN MANY OTHER INAPPROPRIATE DECISIONS ARE MADE


Survey Question

Does the paper . . .12, Examine the power function?

6, Eschew reporting all standard errors, t-, and F-statistics, when suchinformation is irrelevant

8, Consider the power of the test?

17, Do a simulation to determine whether the coefficients arereasonable

18, In the conclusions, distinguish between statistical and substantivesignificance

16, Consider more than statistical significance decisive in anempirical argument?

5, Carefully interpret coefficients? For example, does it pay attentionto the units of measurement, and to the limitations of the data?

13, Discuss the scientific conversation within which a coefficientwould be judged large or small?

11, Eschew "sign econometrics," remarking on the sign but not the sizeof the coefficients?

2, Report descriptive statistics for regression variables?15, Use other criteria of importance besides statistical significance

after the crescendo?

19, Avoid using the word "significance" in ambiguous ways?3, Report coefficients in elasticities, or in some other useful form that

addresses the question "how large is large?"

14, Avoid choosing variables for inclusion solely on the basis ofstatistical significance?

10, Eschew "asterisk econometrics," the ranking of coefficientsaccording to the size of the test statistic?

12, Discuss the size of the coefficients, making points of substantivesignificance?

1, Use a small number of observations, such that statisticallysignificant differences are not found merely by choosing a verylarge sample?

4, Test the null hypotheses that the authors say are the ones of 94,7 100interest?

Notes: "Percent Yes" is the total number of Yes responses divided by the relevant number of papers. Some questionsare not generally applicable because they are conditional on a paper having a particular characteristic. Question 3,for example, was coded "not applicable" for papers which exclusively use nonparametric statistics. Question 19 wascoded "not applicable" for papers that do not use the word "significance,"

If only statisticalsignificance is

important

03,2

426,3

7,3

10,4

13,7

17,7

21,9

26,330,2

29,551,6

59,0

66,7

66,7

86,5

If more thanstatistical significance

is important

28,614,0

4,717,9

55,3

51,2

77,9

38,8

74,1

36,152,3

52,980,0

77,7

83,7

96,5

84,8


TABLE 5THE EASE OF COMPUTING STATISTICAL SIGNIFICANCE IN THE LATE 1970S MAY HAVE HAD I I I EFFECTS ON THE

USE OF REGRESSION ANALYSIS


Date of Ph.D.Conferral

1940-19691970-19741975-19791980-1984

Does the Paper , , ,

Distinguish Among Kinds ofSignificance in the

Conclusions (Question 18)

29331733

Eschew Ambiguous Usage ofthe Very Word (Question 19)

61372945

Notes: The number of papers published by each cohort is 31, 48, 24, and 24. Multiplethe first name listed on the published article.

Consider More ThanStatistical SignificanceDecisive in Empirical

Argument (Question 16)

26311333

author papers were dated by

Such misuses of statistical significanceappear to depend in part on a vintageeffect, measured by date of Ph.D. con-ferral. The papers authored by Ph.D.'sconferred between 1975 and 1979, wheninexpensively generated f-tests firstreached the masses, were considerablyworse than the papers of others at mak-ing a distinction between economic andstatistical significance. They used theword "significance" in ambiguous waysmore often than did early or laterPh.D.'s and they were less likely to sepa-rate statistical significance from otherkinds of significance in the sections onscientific and policy implications (Table5).

V. Taking the Con Out of ConfidenceIntervals

In a squib pubHshed in the AmericanEconomic Review in 1985 one of usclaimed that "[r]oughly three-quarters ofthe contributors to the American Eco-nomic Review misuse the test of statisti-cal significance" (McCloskey 1985, p.201). The full survey confirms the claim,and in some matters strengthens it.

We would not assert that every econo-

mist misunderstands statistical signifi-cance, only that most do, and these someof the best economic scientists. By wayof contrast to what most understand sta-tistical significance to be capable of say-ing, Edward Lazear and Robert Michaelwrote 17 pages of empirical economics inthe AER, using ordinary least squares ontwo occasions, without a single mentionof statistical significance (AER Mar.1980, pp. 96-97, pp. 105-06). This is no-table considering they had a legitimatesample, justifying a discussion of statisti-cal significance were it relevant to thescientific questions they were asking. Es-timated coefficients in the paper are in-terpreted carefully, and within a conver-sation in which they ask how large islarge (pp. 97, 101, and throughout).

The low and falling cost of calculation,together with a widespread though unar-ticulated realization that after all the sig-nificance test is not crucial to scientificquestions, has meant that statistical sig-nificance has been valued at its cost. Es-sentially no one believes a finding of sta-tistical significance or insignificance.

This is bad for the temper of the field.My statistical significance is a "finding";yours is an ornamented prejudice. Con-


trary to the decisive rhetoric of rejectionin the mechanical test, statistical signifi-cance has not in fact changed the mindsof economic scientists. In a way the in-significance of significance tests in scien-tific debate is comforting. Economistshave not been fooled, even by their ownmistaken beliefs about statistical signifi-cance. To put it another way, no econo-mist has achieved scientific success as aresult of a statistically significant coeffi-cient. Massed observations, clever com-mon sense, elegant theorems, new poli-cies, sagacious economic reasoning,historical perspective, relevant account-ing: these all have led to scientific suc-cess. Statistical significance has not.

What should replace a lessened atten-tion to statistical significance is seriousattention to the scientific question. Thescientific question is ordinarily "Howlarge is large in the present case?" Thisis the question that geologists thinkingabout continental drift and astrophysi-cists thinking about stellar evolutionspend their days answering.

The question "How large is large?" re-quires thinking about what coefficientswould be judged large or small in termsof the present conversation of the sci-ence. It requires thinking more rigor-ously about data—for example, askingwhat universe they are a "sample" from,(Carelessness in such matters is morecommon than one might have expected.Of the 107 papers using cross-sectionaldata, for example, 20 percent used testsof statistical significance on the entirepopulation or on a sample of conve-nience. Only two of these offered somejustification for the usage,)

Most scientists (and historians) usesimulation, which in explicit, quantitativeform is becoming cheaper in economics,too. It will probably become the mainempirical technique, following other ob-servational sciences. Econometrics willsurvive, but it will come at last to empha-

size economic rather than statistical sig-nificance. We should of course worrysome about the precision of the esti-mates, but as Leamer has pointed outthe imprecision usually comes fromsources other than too small a sample.

Simulation, new data sets, and quanti-tative thinking about the conversation ofthe science offer a way forward. The firststep anyway is plain: stop searching foreconomic findings under the lamppost ofstatistical significance.

REFERENCES

AMEMIYA, TAKESHI, Advanced econometrics.Cambridge: Harvard U, Press, 1985,

AMES, EDWARD AND REITER, STANLEY, "Distri-butions of Correlation Coefficients in EconomicTime Series," /. Amer. Statist. Assoc, Sept,1961, 56(295), pp, 637-56,

BAKAN, DAVID, "The Test of Significance in Psy-chological Research," Psychological Bulletin,Dec, 1966, 66(6), pp, 423-37,

BARRETT, WILLIAM, The illusion of technique.Carden City, NY: Anchor Press, 1978,

BEHRMAN, JERE R, AND CRAIG, STEVEN G, "TheDistribution of Public Services: An Explorationof Local Covemment Preferences," Amer.Econ. Rev., Mar, 1987, 77(1), pp, 37-49,

BLOMQUIST, CLENN C ; BERGER, MARK C, ANDHOEHN, JOHN P, "New Estimates of Quality ofLife in Urban Areas," Amer. Econ. Rev. Mar1988, 7S(1), pp, 89-107,

BOISSIERE, M,; KNIGHT, J ,B, AND SABOT, R ,H ,"Earnings, Schooling, Abihty, and CognitiveSkills," Amer, Econ. Rev., Dec, 1985, 75(5) pp,1016-30,

BORING, EDWIN C , "Mathematical versus Scien-tific Significance," Psychological Bulletin, Oct,1919, i6(10), pp, 335-38,

CICERO, MARCUS TULLIUS, De divinatione [45BC]; in De senectute; De amicitia; De divina-tione. Ed, and trans, WILLIAM A, FALCONER,Cambridge: Harvard U, Press, 1938,

CLARK, KIM B, "Unionization and Firm Perfor-mance: The Impact on Profits, Crowth, andProductivity," Amer. Econ. Rev., Dec, 198474(5), pp, 893-919,

COHEN, JACOB, "The Statistical Power of Abnor-mal-Social Psychological Research: A Review,"/, Abnormal and Social Psychology, Sept, 196265(3), pp, 145-53,

CooLEY, THOMAS F , AND LEROY, STEPHEN F ,"Identification and Estimation of Money De-mand," Amer, Econ. Rev., Dec, 1981, 7i(5), pp,825-44,

DARBY, MICHAEL R, "The U,S, Productivity Slow-down: A Case of Statistical Myopia," Amer.Econ. Rev., June 1984, 74{3), pp, 301-22,


DAVIDSON, RUSSELL AND MACKINNON, JAMESG. Estimation and inference in econometrics.Oxford: O.\ford U. Press, 1993.

DAVIS, PHILIP J. AND HERSH, REUBEN. "Rhetoricand Mathematics," in The rhetoric of the humansciences. Eds.: JOHN S. NELSON, ALLANMEGILL, AND DONALD N. MCGLOSKEY. Madi-son: U. of Wisconsin Press, 1987, pp. 53-68.

DEGROOT, MORRIS H . Probability and statistics.Reading, MA: Addison-Wesley, [1975] 1989.

DENTON, FHANK T. "Data Mining as an Industry,"Rev. Econ. Statist., Feb. 1985, 67(1), pp. 124-27.

"The Significance of Significance: Rhetori-cal Aspects of Statistical Hypothesis Testing inEconomics," in The consequences of economicrhetoric. Eds.: AKJO KLAMER, DONALD N.MCCLOSKEY, AND ROBERT SOLOW. New York:

Cambridge U. Press, 1988, pp. 163-83.FEIGE, EDGAR. "The Consequences of Journal

Editorial Policies and a Suggestion for Revi-sion," /. Polit. Econ., Dec. 1975, 83(6), pp.1291-95.

FISHER, RONALD A. Statistical methods for re-search workers. Edinburgh: Oliver and Boyd,1925.

FLORENS, JEAN-PIERRE AND MOUCHART,MICHEL. "Bayesian Testing and TestingBayesians," in Handbook of statistics. Vol. 11.Econometrics, of Eds.: G. S. MADDALA, C. R.RAO, AND H . D . VINOD. Amsterdam: North-Holland, 1993, pp. 303-91.

FRASER, DONALD A. S. Statistics: An introduc-tion. New York: Wiley, 1958.

FREEDMAN, DAVID; PiSANI, ROBERT AND PUR-VES, ROGER. Statistics. New York: Norton,1978.

GiGERENZER, GERD. "Probabilistic Thinking andthe Fight Against Subjectivity." Unpublishedpaper. Department of Psychology, UniversititatKonstanz, no date.

GOLDBERGER, ARTHUR S. A course in econo-metrics. Gambridge: Harvard U. Press, 1991.

GORDON, SGOTT. The history and philosophy ofsocial science. London: Rontledge, 1991.

GOULD, STEPHEN JAY. The mismeasure of man.New York: Norton, 1981.

GRANGER, GLIVE W. J. "A Review of Some Re-cent Textbooks of Econometrics," /. Econ. Lit.,Mar. 1994, 32(1), pp. 115-22.

GREENE, WILLIAM H . Econometric analysis. NewYork: Macmillan, [1990] 1993.

GRIFFITHS, WILLIAM E.; HILL, R. GARTER ANDJUDGE, GEORGE G. Learning and practicingeconometrics. New York: Wiley, 1993.

GRILICHES, ZVI. "Productivity, R&D, and BasicResearch at the Firm Level in the 197O's,"Amer. Econ. Rev., Mar. 1986, 76(1), pp. 141-54.

GRILIGHES, ZVI AND INTRILIGATOR, MICHAELD. Handbook of econometrics. Vols. I, II, andHI. Amsterdam: North-Holland, 1983, 1984,1986.

GUTTMAN, LOUIS. "What Is Not What in Statis-

tics?" in Multidimensional data representations:When and why. Ed.: INGWER BORG. Ann Ar-bor: Methesis Press, 1981, pp. 20-46.

"The iUogic of Statistical Inference forGumulative Science," Applied Stochastic Mod-els and Data Analysis, July 1985, 1(1), pp. 3-10.

HAMERMESH, DANIEL S. "Labor Demand and theStructure of Adjustment Costs," Amer. Econ.Rev., Sept. 1989, 79(4), pp. 674-89.

HOEL, PAUL C. Elementary statistics. New York:Wiley, 1966.

HOGBEN, LANCELOT T. Statistical theory: The re-lationship of probability, credibility, and error.New York: Norton, 1968.

HOGG, ROBERT V. AND CRAIG, ALLEN T. Intro-duction to mathematical statistics. 4th ed. NewYork: Macmillan, 1978.

JOHNSTON, JOHN. Econometric methods. 2nd ed..New York: McGraw-Hill, [1963] 1972.

. Econometric methods. 3rd ed. New York:McGraw-Hill, 1984.

KENDALL, MAURICE G. AND STUART, ALAN.T/ieadvanced theory of statistics. Vol. 2., 3rd ed.London: Griffin, 1951.

KENDALL, MAURIGE G.; STUART, ALAN ANDORD, J. KEITH. The advanced theory of statis-tics. Vol. 3, 4th ed. Design and analysis, andtime-series. New York: Macmillan, 1983.

KENNEDY, PETER. A guide to econometrics. Gam-bridge: MIT Press, [1979] 1985.

KMENTA, JAN. Elements of econometrics. NewYork: Macmillan, 1971.

KRUSKAL, WILLIAM. "Significance, Tests of," In-ternational encyclopedia of statistics. Eds.:WILLIAM H. KRUSKAL AND JUDITH M. TANUR.New York: Macmillan, [1968] 1978a, pp. 944-58.

. "Formulas, Numbers, Words: Statistics inProse," The American Scholar, Spring 1978b,47(2), pp. 223-29.

KURTZ, ALBERT K. AND EDGERTON, HAROLD A.,eds. Statistical dictionary of terms and symbols.New York: Wiley, 1939.

LAZEAR, EDWARD P. AND MIGHAEL, ROBERT T."Family Size and the Distribution of Real PerCapita Income," Amer. Econ. Rev., Mar. 1980,70(1), pp. 91-107.

LEAMER, EDWARD E . Specification searches: Adhoc inferences with nonexperimental data. NewYork: Wiley, 1978.

. "Let's Take the Con Out of Econo-metrics," Amer. Econ. Rev., Mar. 1983, 73(1),pp. 31-43.

LOVELL, MIGHAEL G. "Data Mining," Rev. Econ.Statist., Feb. 1983, 65(1), pp. 1-12.

MADDALA, G. S. Introduction to econometrics.New York: Macmillan, [1988] 1992.

MAYER, THOMAS. "Selecting Economic Hypothe-ses by Goodness of Fit," Econ. J., Dec. 1975,S5(340), pp. 877-83.

MCGLOSKEY, DONALD N . "The Loss FunctionHas Been Mislaid: The Rhetoric of SignificanceTests," Amer. Econ. Rev., May 1985, 75 (2), pp.201-05.


MCCLOSKEY, DONALD N , AND ZECHER, J, RICH-ARD, "The Success of Purchasing Power Parity,"in A retrospective on the classical gold stan-dard, 1821-1931. Eds,: MICHAEL D , BORDOAND ANNA J, SCHWARZ, Chicago and London:U, of Chicago Press, 1984, pp, 121-50,

MEEHL, PAUL E , "Theory Testing in Psychologyand Physics: A Methodological Paradox," Phi-losophy of Science, June 1967, 34(2), pp, 103-15.

MiSHKiN, FREDERIC S, "Are Market ForecastsRational?" Amer. Econ. Rev., June 1971, 7i(3),pp, 295-305,

MOOD, ALEXANDER M, Introduction to the theoryof statistics. 1st ed. New York: McCraw-Hilf,1950,

MOOD, ALEXANDER M, AND GRAYBILL, FRANK-LIN A, Introduction to the theory of statistics.2nd ed. New York: McGraw-Hill, 1963,

MOORE, DAVID S, AND MGCABE, CEORGE P, In-troduction to the practice of statistics. 2nd ed.New York: Freeman, 1993,

MORRISON, DENTON E, AND HENKEL, RAMONE, "Significance Tests Reconsidered," Amer.Sociologist, May 1969, 4(2), pp, 131-39,

, The significance test controversy: Areader. Chicago: Aldine, 1970,

MOSTELLER, FREDERICK AND TUKEY, JOHN W,Data analysis and regression. Reading, MA: Ad-dison-Wesley, 1977,

NEYMAN, JERZY AND PEARSON, EGON S, "On theProblem of the Most Efficient Tests of Statisti-cal Hypotheses," Philosophical Transactions ofthe Royal Society A, 1933, 231, pp, 289-337,

OHTA, MAKOTA AND CRILICHES, ZVI, "Automo-bile Prices Revisited: Extensions of the He-donic Hypothesis," in Household production

and consumption. Studies in Income andWealth, vol, 40, Ed,: NESTOR E, TERLECKYJ,New York: National Bureau of Economics Re-search, 1976, pp, 325-90,

ROMER, CHRISTINA D , "IS the Stabilization of thePostwar Economy a Figment of the Data?"Amer. Econ. Rev., June 1986, 76(3), pp, 314-34,

RUDNER, RICHARD, "The Scientist Qua ScientistMakes Value Judgments," Phil. Science, Jan,1953, 20(1), pp 1-6,

SACHS, JEFFREY D , "The Changing Cyclical Be-havior of Wages and Prices: 1880-1976," Amer.Econ. Rev., Mar, 1980, 70 (1), pp, 78-90,

SCOTT, ELIZABETH, "Testing Hypotheses," in Sta-tistical astronomy. Ed,: ROBERT J, TRUMPLERAND HAROLD F , WEAVER, New York: Dover1953, pp, 220-30,

TUKEY, JOHN W, "Sunset Salvo," The AmericanStatistician, Feb, 1986, 40(1), pp, 72-76,

TULLOGK, CORDON, "Pubhcation Decisions andTests of Significance—A Comment," /, Amer.Statist. Assoc, Sept, 1959, 54(287), p, 593,

, The organization of inquiry. Durham,NC: Duke U, Press, 1966,

WALD, ABRAHAM, "Contributions to the Theory ofStatistical Estimation and Testing Hypotheses,"Annals of Mathematical Statistics, Dec, 1939,10(4), pp, 299-326,

WALLIS, W, ALLEN AND ROBERTS, HARRY V, Sta-tistics: A new approach. New York: Macmillan1956,

WONNACOTT, RONALD J, AND WONNACOTT,THOMAS H , Statistics: Discovering its power.New York: Wiley, 1982,

WOOFTER, T, J,, jR, "Common Errors in Sam-pling," Social Forces, May 1933, 11{4), pp, 521-25,

Documents

The Standard Error of Regressionsfaculty.smu.edu/millimet/classes/eco7377/papers/mccloskey ziliak.pdf · Volume I, Zvi Griliches and Michael D. Intriligator, ed. 1983). In the 762