On the Design of Sampling Investigations

Professor Snedecor's discussion of the design of sam-pling investigations was prepared early last fall as a speech which was delivered at meetings of the Chicago and Central Indiana Chapters of the American Statistical Association. No basic revisions have been made since the 1948 presidential elections.

Sampling is one of those things we do every day, almost every hour. We sample our food to learn if it is too hot or if it needs more salt. The doctor tests a single drop of our blood to diagnose our ills. At the blast furnace, the run is sampled for its content of carbon and other chemi- cals. We read in our papers the reports of samplings of public opinion and of the cost of living. In fact: we get far snore information from sampling than from making complete enumerations.

This wide-spread use of sampling means that people have built up a body of common sense about the way to do it. As examples, there are very few who would limit their sampling to a single individual; usually the sample is widely spread and as large as convenient: proportional sampling seems obvious to those who have had their atten- tion called to it. It is clear that since sampling is so common. one must conclude that it is generally successful.

Why, then, should I take up your time for a discussion of the design of sampling investigations? If sampling is merely a matter of common sense, there is no object in further talk. One answer is that common sense isn't very common. Coinmon sense is wisdom about ordinary affairs. and wisdom is always a scarce commodity. RiIore than that, coininon sense is based on experience of the past, whereas uTe need to anticipate the future. Let me illustrate. You will doubtless remember the famous Lit- erary Digest fiasco, in which a sample of millions of voters led to a disastrously erroneous conclusion. Here was a bias caused by failure to include in the sample a large and, as it turned out, a determining seginent of the population. A far smaller sample properly designed would have produced more reliable results. This is in- corporated into our body of common sense since 1936, but then it was not. The theory, however, was available then.

Another illustration: mailed questionnaires are known to be almost always biased, yet they are in general use

THE AI\IIERICAN STATISTICIAN. DECEMBER

by GEORGE W. SNEDECOR

Professor of Statistics, Iowa State College

because of their cheapness. Those agencies which have used this form of sampling for years have learned how to correct for the bias i r i ordinary times and for the usual run of questions. But when the population is changing rapidly, or when new types of questions are asked, the bias is likely to be notably misleading. During the last war, I was told of one questionnaire mailed to a selected list of farmers, the purpose being to evaluate the need for farm laborers. All the big operators returned the questionnaire listing their shortages. The vast ma-jority of little fellows were too busy doing their own work to bother about replying. The result was a fantastic estimate of 3 or 4 laborers needed on the average farm. Now, the investigating agency wasn't blessed with enough common sense to foresee the disability of such a question- naire, but fortunately it had enough experience to recog- nize the returns as absurd, so suppressed the report. I suspect it will not be far i n the future that so-caller1 common sense will lead us to distrust the results of most mailed questionnaires. Meanwhile, both knowledge and wisdom are handy.

But scarcity of w i s d ~ m is not the only reason for look- ing carefully at sampling designs. Another reason is the increasing magnitude of the consequences. If you burn your tongue because of inadequate sampling of your cocoa, the suffering is only temporary. So far as I can judge, it is of no consequence at all if you err in your sampling of brands of cigarettes. Small or even large errors in every day samplings may not be very costly. But if you plan to spend $100,000 on a national sample to guide J ~ O Uin a manufacturing and marketing program, anything less than optimum design means hundreds of thousands of dollars wasted on the survey. Even so, that loss may be small compared to the loss due to unsound policy decisions based on the results of the inadequate or inefficient sampling. Another example is the strategic position of the consumers' price index in labor-manage- ment relations. Poor sampling design for this index would mean millions to the negotiators. I t is this increas- ing iinportance of the consequences that raises questions of design above the reach of the common run of experi- ence.

No worthwhile discussiorl of sampling design is pos- sible without an agreement on the fundamental nature of -

the process. Sampling consists in examining a portion,

1948 6

usually a small portion, of some aggregate of individuals, then making inferences about the entire body of indi-viduals. We learn the facts about the sample but we can only infer the desired facts in the population which is sampled. This indicates the importance of designing the sample so that it shall be representative of the population. Otherwise, the sample facts may lead us to incorrect in- ferences, as was the case in the farm-labor investigation quoted before.

The crucial question, then, is this: "How can a sample be designed so as to be representative of the population?" The answer is in two parts. First, every individual in the population must have a chance of being drawn in the sample; and second, the choice of the individuals in the sample must be random. Unless these requirements are met, there is no way to know whether the sample is repre- sentative; that is, there is no way to judge the conclusions about the population. Let us examine these two require- ments.

First, every individual in the population must have a chance of appearing in the sample. If the design of the investigation is such that some individuals cannot be drawn, then unknown biases may affect the sample. This was one of the causes of the fa,ilure of the Literary Digest prediction of the 1936 election. The mailing lists were taken from telephone directories, lists of registered voters, etc. This was all very well so long as the excluded portion of the population voted like those who were in- cluded; but failure resulted when there came a segrega- tion according to level of income. The sample which ex- cluded a substantial segment of the voting population was insensitive to changes affecting differentially the excluded and included portions.

Another way of saying this is that the population repre- sented is lim,ited to those individuals who have a chance of being drawn in the sample. This is frankly admitted in the Hooper ratings. The sampling is altogether by tele- phone so that the population sampled is confined to those people who happen to be at home and who have both telephones and radios. If there are programs especially popular with non-telephone subscribers, these programs will be under-rated. Another illustration is the ordinary public opinion poll in which all sampling is confined to certain cities and counties selected on a judgment basis. . -

According to my definition, the sample is not represen- tative of any larger population because people in other places have no chance of being included. The population sampled is not the state or the nation at all, but only those regions chosen by the samplers.

The second requirement for representativeness is that the sampling must be random. If the sampler exercises any selection among places or individuals he opens the door to biases due to his selections. This is a problem faced by the samplers of opinion and marketing research

organizations. The respondents are selected by the in- terviewers according to certain criteria which do not include random choice.

If you do not consider this important, let me tell you of a recent incident that may serve as food for thought. It is the failure of the Iowa poll of 1948 to predict the Republican nominee for governor. The poll indicated 53-58% of the electorate as favorable to Governor Blue whereas the nomination went to Beardsley by 60'$. In this instance it is not known how much of the bias can be attributed to interviewer selection and how much was due to changes in intention after the poll was taken. The important lesson to be learned is this: properly designed samples are not vulnerable to this type of interview bias, so that any deviation from prediction, that is, any devia- tion which turns out to be greater than expected sam-pling variation, can be used as a measure of change in intention on the part of the voters.

I suspect that every interviewer has his peculiar biases in choosing respondents. Ordinarily there may be com- pensation among the members of an organization. It is the unpredictable circunlstance of accumulation of biases that may cause trouble.

The advocates of non-random sampling often admit the possibility of bias but plead the high cost of random sampling. This may turn out to be the vital question- which method gives the greatest amount of accurate in- formation per dollar spent? It may be that a smaller sample randomly drawn would give the required infor- mation at the same or even less expense than the larger sample necessitated by interviewer biases; avoiding, at the same time, the possibility of non-cancelling biases. One advantage of the random sample is that i t is essen- tially unbiased, whereas biases may suddenly and un-accountably appear in any selected sample. This means that randomness in sampling reduces the element of risk. The sampler's reputation is always mortgaged to inevit- able sampling variation; why should he stick his neck in the noose of avoidable bias?

So far the discussion has been mainly about sampling from a single population. Many experiments in biology fulfill this condition, at least approximately. Animal and plant populations are fairly stable in many of their char- acteristics. If one assays the concentration of a vitamin by feeding it to a sample of rats, he has reasonable as-surance that the experiment is repeatable next year. Such populations are often roughly normal and do not change notably in time. On the contrary, sampling in economics, sociology and engineering is usually from mixed popula- tions or from those which vary more or less continuously and erratically with time.

If the population is heterogeneous but effectively in- variable, it is advisable to design the sampling so as to insure a wide scattering of its elements throughout the

aggregate. This is the opposite of the device that has been widely used in the past where the sampling has been con- fined to selected segments which are considered "repre- sentative". The most extreme case I ever encountered was the complete enumeration of the farms in a single township under the impression that the findings in this 36-square-mile area would apply to the entire state of Iowa. This kind of sainple produces all the information about this particular township but no information at all about the rest of the state. Most people, after having their attention called to it, agree that common sense lines up with authentic theory to dictate as wide a scattering of the sample as feasible.

One device for insuring scatter is called stratification. If nothing is known about the composition of the popula- tion, mere geographic stratification is advisable: for ex- ample, it may be required that some sampling shall be done in every county of a state. But most investigators know a good deal about their populations and use this knowledge to subdivide them into more homogeneous strata. The public opinion polls allocate the sample to strata such as economic level, sex, etc. Rural dwellers are usually put into strata different from urban, while farming areas are separated from industrial. These are all schemes for subdividing a heterogeneous population into more homogeneous sub-populations. After the strata are selected, each should be sampled according to the prin- ciples stated before-random drawing with the oppor- tunity of being included guaranteed to every member of the stratum.

The designs which I have been describing insure cor- rect sample-to-population inferences but need some further specification for maximum efficiency. Maximum efficiency can be defined in two ways; the most information for the money spent, or the least cost of a required amount of information. Some organizations, operating under a fixed budget, may be limited to the amount of information they can get for, say, $10,000. They may have to restrict themselves to chosen segments of the population in order to get reliable inforiliation about the parts sampled. Other organizations may be more flexible. They may specify a national sample with a certain stated reliability, then enter into a contract to pay the necessary cost. Either way, the efficiency of the sampling is governed by the design.

Neyinan, in the Journal of the Royal Statistical Society in 1934, derived two rather simple rules for the efficient sampling of stratified populations. The first is in accord with ordinary good sense, that the sample be allocated . -

in proportion to the size of the stratum. I t is assumed that both the variation and the cost per interview in the several strata are the same. The means of the strata may be all different, but there must be a constant standard deviation. If these conditions are met, then proportional sampling is the most efficient.

THE AMERICAN STATISTICIAN, IIECEMBER

The strata seldoin have identical standard deviations any inore than they have the same means. Neyman's sec-ond rule is that the intensity of sampling should be pro- portional, also, to the standard deviation. Suppose, as a fanciful exainple, one is sampling the electorate in Iowa for the corning election. He has stratified according to econonlic level and occupation. Consider what might be the outcome in the high economic level among farmers as contrasted with common laborers in industry. Some 90% of the well-to-do farmers might be expected to vote in the traditional manner of our rock-ribbed Republican state, whereas only about 50% of the laborers may follow suit.

Then, according to the well-known formula, a =vlS$ the standard deviations are respectively 0.3 and 0.5; that is, the second is almost double the size of the first. This means that, for maximum efficiency, the sampling among laborers should be almost twice as intensive as among wealthy farmers. A moments thought will con-vince you that this also is just good sense. If rich farm- ers tend to vote alike, then it will take little sampling to evaluate their preference. Among industrial laborers, however, who may differ markedly in their opinions, almost double the sampling rate is required to reach an equally good evaluation. A fantastic extreme is the case in which the investigator might know that all the mein- bers of a stratus will vote alike. Clearly, it would be necessary to poll only a single voter to learn how the entire stratum will vote.

This sounds simple but in practice it is not easy. Esti-mates of population standard deviations cannot be exact. Furthermore. not one but several questions often are asked and no corninon allocation of the sampling yields maximum efficiency for them all. Here, again, the ex-perimenter in biology usually has the advantage. The standard deviations of many of his sampled populations are known with some fair degree of approximation, and often only one question is asked of an experiment. The reliability of the eventual mean can be predicted with considerable certainty. The experimenter, then, can be quite sure of the money it will take to get an answer of specified reliability; or of the reliability of the answer he can get from expenditure of a specified sum of money. While maximum efficiency can seldom be attained in the sampling of human populations, yet the principles are the same and can be used to regulate the design of such samplings.

A necessary part of any sampling desig~l is the statis- tical method for summarizing the results. There would be no object in getting hformation into data unless there were efficient methods of extracting the information for eventual use. For this reason it is worthwhile to consider the formulas applying to a stratified random sample such ds we have been discussing. For illustration, the table

1948 8

-- - - - - - -- - - - - - - -- -- --

- - -- -- - - -

--

--

- - -- - --

Data on Iowa Survey of Food Production in the Home (1943) -

Number of Opt~mum Schedules Qts. per Estimated Stratum Famllies Weight Allocation Obtained Famlly Std. Dev.

Urban 312,000 0.445 295 300 165.1 153

Rur a1 161,000 0.230 159 155 201.4 160

Farm . 228,000 0.325 247 237 297.8 175

Total 701,000 1.000 701 692 -- -- - -- .--- - - --

Total quarts in Iowa (thousands) : (312) (165.1) + (161) (201.4) + (228) (297.8) =151,835 thousands of quarts.

151,835 Mean quarts per Iowa family: 701 = 217 quarts

(153)\ (160) (175) Variance of mean: ( 0.445) * joO- + (0.230) 155 + ( 0 . 3 2 5 ) 2 2 3 7 = 37.84.

Standard deviation of mean = v37.84 = 6.15 quarts. - - - --

contains the data from a wartime survey of Iowa to learn sarily expensive. Experience has shown that it often pays the extent of the production and preservation of food to form clusters of families, then chose a random sample products by private individuals. I shall use only the re- of clusters. There may be a notable saving in traveling sults of the question about the number of quarts of food expense and little loss due to correlation among indi-canned, frozen or stored for home consumption. vidual families of a cluster. Samples of this type are

The rate of sampling was designed as one per thousand available from a project known as the Master Sample.

families. From the slight amount of information avail- This was constructed by a cooperation among the Bureaus

able, we guessed that the variation would be greatest of the Census and Agricultural Economics together with

among farm families, so we allocated somewhat more the Statistical Laboratory of Iowa State College. The

than a proportional number of families to the farm materials of the master sample are available to all.

stratum. Results indicated that the guess was good; but It is of particular interest to observe that a sample de-

the variation proved to be so nearly uniform that pro- signed like that of the food survey, using the materials

portional sampling would have been satisfactory. of the master sample, allows of population estimates

Notice particularly the calculation of the total produc- without direct use of stratum size. The sample can be

tion, 151,835,000 quarts. The stratum means are used expanded from a knowledge of the sampling rates.

but not the stratum sample sizes. Instead, the stratum So much for fixed populations, whether single or mul- tiple. I turn now for a look at populations in which the sizes or population figures are used. This indicates that

moderate deviations from optimum allocations are not variable is changing in time. Public opinion is such a

important, but that errors in stratum sizes, that is, errors variable, and consumer acceptance is another. The cost-

in weights, can be disastrous. This is a point which the of-living is a third. In fact, most of our economic, in-

poll takers and the marketing research people seem to dustrial and political variables are of this kind. In some,

have overlooked. They insist on adherence to quotas, the changes are fairly regular so that trends can be pre-

which is relatively unimportant; but they have no way to dicted. In many, the changes appear to be erratic.

make reliable determination of the sizes of strata such as For these changing populations, efficient sampling de-

economic levels. Unless the weights are accurately known, signs and statistical methods have not in general been

the use of strata may be less efficient than completely ran- developed. Even the ordinary methods of curve fitting

dom sampling. Similarly, it is clear that biases in the f8il because of correlated variances. On this subject, I am

sample means in the strata contribute directly to bias in not competent to say much. Some progress has been made,

the total, emphasizing the necessity of sound sampling but the difficulties are great. The most successful effort in this field has been in the methods.

control of the quality of manufactured products. TheI have been talking as though randomness of sampling

reason for the success is that the measurement of quality were easy to accomplish. This is not the case in much of

can usually be stabilized by discovery and adjustment of our sampling. In a survey of Iowa families, for instance,

the operation responsible for poor quality. Here it is not even if one had all the names and addresses and could

desired to evaluate and predict trends but to eliminate select a random sample from it, searching them out in

them. Lack of stability is discovered by sampling. The every nook and cranny of the state would be unneces-

CONTINUED ON PAGE 13

show that he understands the conditions in the region and partly that he is interested in acquiring new knowl- edge. It mill not do for him to make it plain that his interest is to obtain statistical information. When the conversations run smoothly, the respondent will often volunteer information which is unexpected, but which gives important background data. Usually the interviewer has to be very tactful in leading the conversation back to the topics about which he is gathering statistical informa- tion, when the respondent talks too much at random.

It will not do for the interviewer to ask one question after another even when the respondent has shown a will- ingness to talk. It is usually a good practice to sandwich his question among a general discussion of other topics. This requires a good deal of tact and patience on the part of the interviewer, but it usually produces the best results. Sometimes several questions worded differently have to be asked in order to obtain one answer, if the first or first few answers are not satisfactory. In such cases these questions, as has been pointed out above, must not follow one after another, but other questions or general discussion should intervene in order to take the respondent off guard, or to make him understand exactly what infor- mation is wanted. As a rule, the respondents will give truthful replies, but if they are not approached properly they may either refuse to answer or give anything for a reply, just to get rid of a bore.

The interviewer should, whenever possible, be prepared for the interview by obtaining information from indirect sources beforehand. For instance, if he wants to interview the manager of a factory, he should first find out from several other factories, usually of the same industry, the general conditions in the industry and the factory con-cerned. When he has such general knowledge, especially of that particular factory, it is easier for him both to carry on a general conversation with the manager and to check upon the accuracy of his answer. This does not mean that inEorination obtained by indirect methods is always reliable, but it gives him some sort of background

Sampling Investigations engineer then seeks the cause or causes of excessive varia- tion and takes steps to remedy the defects in the produc- tion line. Once the quality variable is under control, the characteristics of the stable output population are deter- mined. These can be made the basis for contracts safe- guarding the interests of both producer and purchaser. The sampling designs used in quality control are rather simple and have become astonishingly popular. Their widespread use is almost certain to be followed by the rapid introduction of other sampling and experimental designs into industry.

about the factory concerned. The same holds true with obtaining information from farmers. This kind of pre- liminary information may sometimes be obtained from research reports, statistics of previous periods, etc.

One single respondent may have to be approached more than once. I t may be because the person is too busy to spend enough time with the interviewer or unwilling to give any information during the first interview. If the interviewer is tactful he will soon establish a sort of repu- tation as a good fellow in whom one may confide. When that is the case, any particular respondent who has been unwilling to talk may find out from other respondents that talking with this pardcular interviewer will do no harm, and he will usually be more frank during the second or third interview.

If written schedules are sent to the respondent, which is only done in the case of large companies, it usually requires follow-up work by personal calls on the part of the interviewer. Follow-up letters sometimes help, but are not of much use in China.

The usual way of preserving secrecy in the West by leaving out the name of the respondent or that of his firm can seldom be adopted in China. The Chinese respondent is not in the habit of mailing in questionnaires, and even if he cares to do so, he will thank that the number or some identifying mark on the questionnaire will "betray" him. Either he will talk quite freely, if the interviewer approaches him in the proper way, or he will not supply any information, no tnatter how much secrecy is promised.

In some cases some sort of pressure has to be exercised on the respondent. The pressure must not be so great as to make the respondent feel he is under compulsion to supply information, nor should it be so slight that he may disregard it entirely. It need not be from official sources. A field worker of a private research institute may also exercise such pressure if he knows how to handle his respondents. It consists in the tone of the conversation, the way he behaves and other indefinable mannerisms. - -.

Reprinted by permission of t h e STATISTICAL REPORTER, October, 1948, No. 130, Ui\'ision of St:itistical S tandards , Bureau of t h e Budget.

The sampler of public opinion or consumer acceptance may take successive samples in order to learn of changes that may be going on in the population. He should always be careful to designate the date of a sample and avoid any implication that the facts apply to any other date.

S'ince sampling has transcended the routine of every- day living and has become economically, socially and politically momentous, it behooves us to examine and re- examine our sampling procedures to make sure that they are'not only competently executed but also theoretically sound.