Area Sampling or Quota

Con trol?-Three Sampling

hxperiments BY JOSEPH R. HOCHSTIM AND DILMAN M. K. SMITH

Since area sampling is considerably more expensive than quota sampling, it is pertinent to inquire whether increased costs are compensated by suficiently increased accuracy. Three experiments, conducted by the Opinion Research Corporation in 1945 and 1946, shed considerable light on this question by com- paring the results of quota control sampling, area sampling, and "domal sam-pling" with known data. Area sampling is found to yield the most accurate results, although the less expensive "domal method" is adequate for most cases. In the last analysis, however, the most efficient sampling design (from the point of view of expense) must be determined according to the nature of the problem under investigation.

Dr. Hochstim, Chief Statistician for Opinion Research Corporation, holds a Ph.D. in political science from the University of Vienna, and was formerly on the stag of the Psychological Corporation. MY. Smith, Vice-President of Opinion Research Corporation since I9j9, entered the market research field in 1931. Cur-rently, he serves as research consultant to several large industrial firms and trade associations.

Recent years have seen heated discus- merits and that each has a sphere in sion about the efficiency and accuracy which it may be properly used.2 of different methods of survey sam-

The authors are indebted to Mr. Kendrick pling. Area sampling is one of the S. Few for his many helpful suggestions. methods proposed as the answer to a In all discussions of sampling techniques, great many problems. Its proponents one general caution should be kept in mind. have been on the offensive. Quota sam- It would be unreasonable to aim for perfec-

pling, on the other hand, even though tion in sampling when errors due to wording

put on the defensive, still is very widely and sequence of questions and to reliability

used in market and public opinion and validity of the answers might affect the results of a survey to a much greater extent

research? than errors from sampling fluctuations. We Although discussion has been ex- should always strive For the most efficient

tensive, no definitive data on the relative sample, but it would not be wise to double

efficiency of the two sampling methods costs in order to reduce a sampling error by I per cent or 2 per cent, especially if little is

have been published. Actually, there is done about question wording and other Factors evidence that each method has certain which might possibly affect the results.

74 PUBLIC OPINION QUA LRTERLY, SPRING 1948

As for the relative merits of the two sampling techniques, the defenders of each method have powerful arguments. Quota samplers rightfully cite:

a. Considerably lower sampling and interviewing costs.

b. Greater speed in execution of sample design and of field work.

On the other hand, area samplers justly mention:

a. Freedom from the necessity of estimating the composition of the universe and of establishing quotas. (The longer the time since the last census, the greater is the possibility that the composi- tion of the universe may have changed. Thus, the sampler is frequently at a loss to make an intelligent estimate of how large these changes are and even in what direction they go.)

b. Drastic reduction in bias due to interviewer selection of respond-ents. (When final selection of re- spondents is left to the inter-viewer, bias may be introduced because of his personal prefer-ences, convenience, or special ef- forts to locate respondents likely to give "informed" comments.)

Criteria for Selection of Sampling Technique

The principal object of each survey sample is to provide the user of the data with the most information at the lowest cost. Consequently, the choice between area sampling and quota con- trol will ultimately be determined by the dollar efficiency of each technique in each individual case.3 Efficiency is defined as the quotient of information

(described by R. A. Fisher as the re-ciprocal of the average of [error]') over cost.

The nature of the survey, in turn, usually determines efficiency from the point of view of the consumer. In sur- veys dealing with issues and reasons for people's opinions (qualitative surveys), where the condusions of the user of the data will be the same whether 70 per cent, 75 per cent, or 80 per cent of the respondents hold a certain opinion, quota samples will usually be better- that is, more efficient. If, however, the purpose of the survey is to measure exact quantities, as in a sample census or in a survey measuring sales poten- tials (quantitative surveys), the area method probably yields more accurate results. In other words, where we paint with a broad brush quota sampling is more efficient, but where we etch with a fine needle area sampling is refera able.^

Ballot length is another factor in-fluencing the choice of the sampling technique. Area sampling will probably be more efficient with long ballots, quota sampling with short ballots. Bal- lot costs are determined by interview- ing time, which consists of time spent in (a) locating the respondent, and (b) actual interviewing. Actual inter- viewing time is the same regardless of the type of sampling used, but time

a J. Stevens Stock will show in a forth-coming article how the dollar efficiency of a sample can be computed algebraically.

4There are times, however, when a quali-tative survey might become a quantitative one, as is the case when an opinion survey is re-peated for the purpose of measuring trends. Here the sample must be kept strictly com-parable from one survey to the next in the same way that question wording and se-quence must remain unchanged for accurate time-series measurement.

AREA SAMPLING OR QUOTA CONTROL?

spent in locating respondents is con-siderably less when interviews are as- signed by quota. Since the relative importance of this time increases in- versely to the length of the ballot, the ratio of the cost of locating respondents to total ballot expense is more favorable in the case of a long ballot.

In practice, a compromise solution such as the dwelling unit or doma15 sample described below may often be the most eficient one in market and public opinion surveys. Its advantages are:

a. Elimination of the worst biases of quota sampling.

b. Relatively low cost.

Area sampling is considerably more expensive than quota sampling. Since area sampling is being used increas-ingly, it is important to determine if the additional expense is justified. Ac- cordingly, Opinion Research Corpora- tion has conducted a number of ex-periments, three of which are reported here. The problems investigated in these experiments were:

I. What are the differences in com- position between a quota-control sample and a stratified block sample? (Experiment I )

2. What happens if the freedom of the interviewers in selecting households within the block is restricted? (Experiment 2)

3. T o go one step further, what is the result when we control the selection of respondents within households and require callbacks if respondents are not at home? (Experiment 3)

Experiment I: Block Sample versus Quota Sample

Eleven cities, each with a population of more than 50,000, were selected. Two interviewers in each city were assigned the same number of ballots. While one interviewer was given a simple quota assignment by sex, age, and socio-economic status, the other in- terviewer received a block assignment with quotas for sex and age superim- posed. The blocks were stratified by census tracts and by average monthly rent. Blocks within each stratum were selected systematically and interviews were assigned in proportion to block size. Instructions to the interviewers called for respondents in poor, average, and prosperous homes in the same pro- portions as they occur in the blocks.

In order to keep the variable of in- dividual interviewer ability constant, the experiment was repeated in a sec- ond survey. The same eleven cities and the same twenty-two interviewers were used, but the quota and block assign- ments were reversed. A number of questions designed to measure certain known population characteristics were included, and the samples obtained were compared with Census data.

Results: In both surveys, the block sample showed less bias on education than did the quota sample. (See Table I ) Even in the block sample, however,

"Domal" from the Latin domus is defined by Webster as pertaining to the house. The term "domal sampling" as introduced here refers to a sample in which not only blocks but houses and dwelling units within the blocks are predetermined in a systematic way with quotas for one or more characteristics such as sex, age, etc. being assigned.

PUBLIC OPINION QUARTERLY, SPRING 1948

TABLE I

Actual Figures Survey Figures 1945 and 1946

( I 940 Census) Block Sample Quota Sample Respondents 293 298 Monthly rent-fourth decile $26.15 $33.24 $34.51

median 29.69 36.69 39-71 sixth decile 34-04 40.34 47.69

Median number of school years completed 9

Per cent having grammar-school education or less 45

Per cent having high-school in- complete I8

Per cent having high-school complete 23

Per cent having college incom- plete 8

Per cent having college com-plete 6

Per cent owning home 44" Per cent having telephone 63t Per cent having mechanical re-

frigeration 69' Per cent having automobile -$

*Opinion Research Corporation estimate based upon U.S. Census releases, 1945 and 1946. t Opinion Research Corporation estimate based upon AT&Treleases, 1945. S No data available.

there were still too many respondents thoritative figures on automobile own- with college and high-school education ership in the eleven cities were available and too few with grammar-school edu- for comparison. The results from the cation or less. two surveys, however, are identical.

The block sample also showed some One more point remains for con-improvement over the quota sample sideration. Even though discrepancies with respect to average monthly rent. between the two samples might notCensus data on rent shown in Table I

refer to 1940. Since the surveys were appear unreasonable, it is still possible

made in 1945 and 1946, the monthly that bias might be present in either or

rent reported by respondents was ex- both of them. In fact, when respondents

pected to be higher than that shown were asked if they had ever been inter-

by the 1940 Census. viewed before, 13 per cent in the block

On three other variables-home own- sample and 19 per cent in the quota

ership, telephone ownership, and me- sample said yes-a possible indication chanical refrigeration-the two samples that quota interviewers are inclined to compare reasonably well with each other approach the same persons over and and with the estimated actual. No au- over again.

77 AREA SAMPLING OR QUOTA CONTROL?

Experiment a: Block Sample versus Domal Sample

The purpose of this experiment was to compare a block sample of the type described in Experiment I with a sam- ple wherein both the blocks and dwell- ing units within the blocks were pre-determined systematically so that bias due to interviewer selection would be minimized.

The blocks of a Midwestern city of about ~oo,ooo population were stratified as described in Experiment I and the sample blocks were numbered consecu- tively. Assignments in each odd-num- bered block called for selection of poor, average, and well-to-do homes propor- tionate to their occurrence in the block. In even-numbered blocks, however, in- terviewers were instructed to interview in every nth dwelling unit, proceeding clockwise from a predetermined starting point. A different starting point was selected for each block (for example, house at the NW corner, second house from the NE corner). Corner houses and different sides of the block were given proper representation.

When interviewers found nobody at home, or were refused, the interview was taken in the next dwelling unit. The count from one sample dwelling unit to the next was carried on as if the interview were taken in the home orig- inally designated. Because of the na-ture of the survey, interviewing was restricted to employed persons. Conse- quently, most of the interviewing was done during late afternoons, evenings, and weekends. Therefore, the number of substitutions was relatively small.

In both samples sex and age quotas were superimposed, and to eliminate variations due to interviewer ability

both types of assignments were given to all interviewers.

Results: The domal sample came closer than the block sample to Census data in the variables tested: rent, edu- cation, home ownership, refrigeration. (See Table 2)

Thus, the less freedom the inter-viewer has in selecting dwelling units, the more representative is the sample.

Experiment 3: Domal Sample without callbacks versus Area Sample with callbacks6

The area to be sampled in this ex-periment covered two adjacent coun-ties in Ohio comprising:

a. A city of over 200,000 population. b. A number of smaller towns be-

tween 2,500 and 25,000 popula- tion.

c. Villages, open country, and farms in rural areas.

Interviews to be taken in each of these areas were assigned in proportion to their populstion.

The blocks in the central city were stratified by Census tracts, rent, and number of dwelling units per block, in accordance with 1940 b'lock statistics. Since 12 per cent of all dwelling units in this city were constructed since the 1940 Census, data on their number and location were obtained from the city planner. A sample of blocks was taken in such a way that old and new dwell- ing units were represented in their proper proportions.

Block statistics were not available for the other urban places in the sam-

e The authors are indebted to Mr. Frank C. Weber, Vice-President, D'Arcy Advertising Agency, for his suggestions in this sampling experiment and for his permission to publish the results.

PUBLIC OPINION QUARTERLY, SPRING 1948

TABLE 2

Survey Figures 1946

Domal Sample Block Sample (Automatic (Interviewer

Actual Figures selection of selection of (1940 Census) households) households)

Respondents 452 495 Monthly rent-fourth decile $24.62 $35.49 $37.83

median 27.55 39.15 41.94 sixth decile 30.71 44.35 47.23

Median number of school years completed 8 10 I I

Per cent having grammar-school education or less 5O 33 33

Per cent having high-school in- complete

Per cent having high-school complete

Per cent having college incom- plete

Per cent having college com-plete

Per cent owning home Per cent having mechanical re-

frigeration Per cent union members Per cent Negroes

* Opinion Research Corporation estimate based upon Census releases, 1945, 1-No data available.

ple. Blocks in these towns were num- Within the minor civil divisions, bered consecutively on maps in ser- separate samples were taken in vil-pentine fashion, and every nth block lages, in unincorporated rural nonfarm was selected for the sample. territory, and on farms. In villages, in-

For rural areas, minor civil divisions terviewers were assigned every nth were used as primary sampling units dwelling unit on one side of the street. and were stratified into groups by size, For the open country and farm assign- local option for alcoholic beverage^,^ ment, square-mile sections were num-and distance from the nearest town. bered on state highway maps, and every Within each group the minor civil di- nth section was selected. Interviews visions were selected at random for the were taken at every nth dwelling unit sample. Altogether, seventeen minor within each section in the sample. civil divisions were chosen, and inter- Interviewing Procedure. In the city, views were assigned in proportion 7 This stratification was necessary because the size of each group. of the nature of the survey.

79 AREA SAMPLING OR QUOTA CONTROL?

towns, and rural areas, the sample blocks and square-mile sections were numbered consecutively. Split-sample technique was used. All odd-numbered blocks and square-mile sections were put in Group A (the domal sample) and all even-numbered in Group B (the area sample). Interviewers were given starting points and instructions on how to proceed within each block.

In the domal sample, interviewers were told to start at the nth house from the N W (or N E or SW or SE) corner, to proceed clockwise, and to interview in every nth dwelling unit. Only one interview was to be taken in each dwelling unit. In cases of noninterview or refusal, interviewers were instructed to substitute the next dwelling unit, but the count was carried on from the dwelling unit originally designated. Sex and age quotas were assigned.

In the area sample, interviewers were given starting points and directions on how to proceed within the block, as in the domal sample. There were, how-ever, no sex and age quotas. Instruc- tions were to interview all persons 18 years old and over in each dwelling unit assigned. Two callbacks were made wherever necessary. In no instance was it left to the interviewer to decide where and whom to interview. Only 2 per cent of the persons in this sample were not reached after two callbacks.

All interviewers participated in both parts of the survey. Interviewing was equally divided throughout the week.

Results. A comparison of the two ob- tained samples, as set forth in Table 3, shows very close agreement. This is the more remarkable since the ~opu la - tion had increased sharply in the survey area. Consequently, changes in the char-

acteristics of the population were di&- cu~lt to estimate.

An experiment on callbacks reported by Ernest R. Hilgard and Stanley L. Paynes has yielded results which showed differences between first call and later calls. It should be kept in mind, however, that even if there are differences between respondents reached at the first call and those reached at later calls, these differences may not affect materia~lly the total re-sults of a survey unless a relatively large proportion of respondents is not reached at first.

Conclusions

In summary, the conclusions from our three experiments are:

I. Block samples tend to result in a more representative cross section than quota samples.

2. Mechanical and automatic selec- tion of dwelling units makes the cross section more representative and less subject to bias.

3. The extra expense of callbacks is not always necessary. Experiment 3 indicates that under certain conditions a carefully selected domal sample will produce about as good a cross section as will an area sample.

The results of our three experiments are reported here for the benefit of those who are interested in this prob- lem. Each technique reported has its merits. But there will be cases in which one technique has definite advantages over the others; and there also will be cases where a compromise solution,

8 "Those Not at Home: Riddle for Poll-sters," Public Opinion Quarterly, 8, No. 2

(1944), 254-261.

80 PUBLIC OPINION QUARTERLY, SPRING 1948

such as domal sampling, is most ef- determine which technique is to be ficient. In any instance the problems used-domal sampling, area sampling, presented by the individual survey will or quota control.

TABLE 3 Survev Fi~ures 1946

Area Sample Domal Sample (Automatic (Automatic selection of zelection of

households and households; respondents; no callbacks) callbacks)

Respondents Monthly rent-fourth decile

median sixth decile

Median number of persons per household

Median number of children under 18 years

Median number of school years completed

Per cent with grammar-school education or bless

Per cent with high-school in-complete

Per cent with high-school com-plete

Per cent with college incomplete Per cent with college complete Per cent manual workers Per cent employed in manufac-

turing industries Per cent foreign born Per cent with foreign-born father Per cent with foreign-born mother Per cent Negroes Per cent married Per cent owning home Per cent having telephone Per cent having mechanical re-

frigeration Per cent having automobile

Median monthly grocery bill