COMMUNICATIONS The purpose of this new section is to introduce the lighter side of

marketing, where readers can discuss the less technical aspects of mar- keting research, and comment on material appearing in our journal or elsewhere relating to marketing research. Readers are invited to submit material for this section.

Are Open-ended Questions Worth the Effort? STANLEY L. PAYNE*

>Free-answer questions are so troublesome that banning them from full-scale surveys has been suggested.

A recent experiment finds them inefficient and no more productive of depth or of valid answers than check-

box questions ore. Their indicated use may b e confined to the development and pretesting phases of surveys.

IN private discussion, Ben Gaffin sometimes con-tended-that open questions had no place in large-scale surveys.l He argued that the open, or free-answer, ques- tion was inordinately inefficient for whatever contribu- tion it might make to understanding consumer views and attitudes. Problems of interviewer variability, coding inconsistency, lack of comparability in trend surveys, and tabulating complexity combined to make the open question troublesome, time consuming, and expensive beyond its value.

Even if the transfer from open questions in the devel- opmental phase to closed questions in the final form meant increasing the total number of questions, he still championed the closed questions. For example, from answers to the single, exploratory open question, "What do you like about it?" a number of categorical questions might evolve for the final interviews: "Do you like the taste or not?", "Do you like the color or not?", "Do you like the thickness or not?", "Which is the most important to you-its taste, its color, or its thickness?", etc.

Having written a chapter on the free-answer question and its demerits [2], I had some sympathy for the point of view he expressed. At the same time, it seemed almost heresy to think of eliminating that basic type of question from final questionnaires. In writing of its merits, I said that "the free-answer (question) is uninfluenced, it elicits a wide variety of responses, it makes a good introduction to a subject, it provides background for interpreting answers to other questions. It can be used to solicit sug- gestions, to obtain elaborations, to elicit reasons, to evaluate arguments, to explore knowledge and memory, and to classify respondents." The discussions with Ben did little more than increase my tendency to look at open questions critically; I continued to use many of them in full-scale surveys.

Now, from an unexpected source comes an incisive indictment of open questions based upon a well-con-

* Stanley Payne is a survey director and consultant, Payne Survey Research, Chicago.

IBenjamin H. Gaffin, who died in 1959, was the founder of Ben Gaffin & Associates, and United States Interviewing Corpo- ration.

ceived, carefully executed, evidently unbiased, con-trolled experiment. Because Human Organization, offi-cial journal of the Society for Applied Anthropology, is not in the normal purview of most market researchers, that report by Barbara Dohrenwend merits special men- tion here [I]. Among the conclusions from that experi- ment are the following:

The experimental interviews lend support to the criticism that open questions are less efficient than closed questions. At the sime time, there is no evidence that open questions possess the advantage of being more productive of depth. . . . Neither is there any direct evidence that open questions produce more valid answers. . . . The general conclusion to which we are led by the evidence in the experimental inter- views is that closed questions offer more definite advantages than open questions in research interviews.

In' comparison with most marketing studies, this ex- perimental study was small and limited in coverage- only 32 respondents chosen at random from women undergraduates at Cornell University. Its lack of size and scope was balanced by careful design and rigorous pro- cedure. The design employed a 4 X 2 x 2 x 2 pattern. Each of four interviewers, all having master's degrees and an average of two years of full-time work as professional interviewers, made eight two-part interviews; one part was factual and one part attitudinal, one part by open and one by closed questions, with the order of presentation of the parts and question types rotated over all possible combinations. Each respondent had earlier been exposed to a contrived standard situation which was the subject for both fact and attitude questions.

Nine "response properties" were measured for the two parts of each interview. The findings on six of those properties may be briefly summarized as follows:

( 1 ) Length of response averaged about nine lines for open questions and three lines for closed questions.

(2) Usability of response-meaningful, relevant, unam-biguous, nonrepetitive proportions of the answers-did not differ significantly for the two question types.

(3) Pertinence of answers was at about the same level for both types of questions, somewhat relieving open ques-

417

Journal of Marketing Research, Vol. I1 (November 1965), 417-19

418 JOURNAL OF MARKETING RESEARCH, NOVEMBER 1965

tions from the criticism that they allow respondents to wander from the topic.

(4) Self-revelation in answers to open questions was higher than in answers to closed questions on facts (Who was there? Who said what?, etc.), but the reverse was found in answers to questions on attitudes (Feelings about participating? Feelings of others?, etc.). The latter re- sult goes directly counter to most expectations.

(5) Correctness of factual reporting, or known validity, was higher with the closed questions.

(6) Reasonableness of subjective responses, a judgmental measure, appears to be more affected by subject matter and position in the interview than by question type. Dr. Dohrenwend reaches the conclusion here that "in the face of resistance" closed questions yield more valid responses on subjective topics than do direct open ques- tions.

The full report on the experiment analyzes interac- tions of form, subject matter, part of interview, and interviewer in considerable detail. It warns of limitations and is cautious in generalizing to normal survey opera- tions. The above summary of six properties contains the most pertinent findings, however, and is presented as fairly as I find possible in a condensed version.

One response property not referred to in that report is the relative number of different ideas mentioned by respondents when answering open and closed questions. My experience is that the longer responses for open questions tend to consist largely of elaboration and ex- planation of the few ideas originally expressed rather

than a proliferation of additional ideas. In an unpub- lished experiment where I was able to compare depth probing with lone reason-why questions, much more verbiage but an increase of only about a sixth in reasons per respondent was elicited through probing. The addi- tion of unusual ideas was negligible for the overall sur- vey.

As a consequence of studying Dr. Dohrenwend's re-port, I am much more inclined toward the view that open questions should be eliminated from full-scale surveys wherever possible. To quote from her last sen- tence "our results suggest that it is generally to the in- vestigator's advantage to use well-tested closed questions rather than open questions."

The open question does have important application in the preliminary phases of interview researches. I t should be employed there for the development of cate- gorical, checkbox questions which can anticipate the range of relevant views or attitudes likely to be ex-pressed, or to eliminate a need for asking reason-why questions of every respondent, or to provide quotes which may add interest to the report.

REFERENCES

1. Barbara Snell Dohrenwend, "Some Effects of Open and Closed Questions on Respondents' Answers, Human Organi- zation, 24 (Summer 1965), 175-84.

2. Stanley L. Payne, The Art of Asking Questions, Chapter 3, Princeton, New Jersey: Princeton University Press, 1951.

Should the AMA Encourage Arbitration

of Trade Disputes ? SIDNEY HOLLANDER, Jr .*

>The advantages of AMA arbitration of trade disputes are brought out by means of an actual case history.

M EMBERS of the American Marketing Associ- ation may be interested in how the ~ssociat iok was in- strumental in effecting the friendly settlement of a trade dispute. Arbitration under the auspices of a business or professional association is fairly common in certain fields (i.e., American Institute of Architects, the Air Transport Association, and some textile and insurance groups). But as far as I, and some knowledgeable elder statesmen whom I asked, know, this is the first instance

* Sidney Hollander, Jr., is a senior partner of Sidney Hol- lander Associates, Baltimore.

where this has happened within our group. I t appears such a logical extension of the Association function, however, that it is presented here with the idea that it may serve as a useful precedent.

The content of the dispute is unimportant for the present report; briefly, the issue was whether a research agency can refuse to pay for admittedly faulty inter- viewing purchased in another city without giving the field interviewing service a chance to make good. After heated correspondence had deteriorated to mere reitera- tion of each party's position, it was agreed to submit the dispute to arbitration. Since both parties were AMA