251

Quality and Quantity, 10 (1976) 25 l-260 @ Elsevier Scientific Publishing Company, Amsterdam - Printed in The Netherlands

A CRITIQUE OF GUTTMAN SCALING: WITH SPECIAL ATTENTION TO ITS APPLICATION TO THE STUDY OF

COLLEGIAL BODIES

JOHN A. STOOKEY and MICHAEL A. BAER

University of Kentucky

I. Introduction

Guttman scaling, also called cumulative scaling or scalogram analysis, was first developed by Louis Guttman during the early part of World War II to determine which of a series of attitude questions on interview schedules tapped the same underlying attitude. Guttman (1944) hypo- thesized that if the questions did tap the same attitudinal dimension the questions and responses could be arranged so that there would be an attitudinal continuum that indicated varying degrees of the under- lying dimension. (For a discussion of Guttman scaling, see Goldman and Jahnige, 197 1.) The two primary characteristics of Guttman scales can be drawn from this primary hypothesis (Nie et al., 1970):

(I) Guttman scales are unidimensional; “that is, the component items (questions) must all measure movement toward or away from the same single underlying object.” (2) Guttman scales are cumulative. “Operationally, a cumulative scale implies that the component items can be ordered by degree of difficulty and that the respondents who reply positively to a difficult item (question) will always respond positively to less difficult items and vice versa.

The conventional example of a Guttman scale is one in which the questions deal with the person’s height, and the underlying object or dimension is therefore height. The following set of questions will pro- duce a perfect scale: (1) Are you five feet tall or more? (2) Are you five feet, eight inches tall or more? (3) Are you six feet tall or more? The scale which would result from giving this set of questions to a group of respondents appears as in Table I.

This scale meets both of the previously delineated characteristics of a Guttman scale. (1) The questions are unidimensionally ranked on a single underlying dimension, height. (2) The scale is cumulative in that

2.52

TABLE I Hypothetical Guttman Scale Response Pattern.

Respondent Questions

1 (> 5’) 2 (> 5’8”) 3 (2 6’)

A B C D

Yes Yes Yes Yes Yes No No No

Yes No No No

none of the respondents had a “No” answer before a “Yes” response, or a “Yes” response after a “No” answer: Thus, the scale is perfectly cumulative and knowledge of the position of a respondent’s last posi- tive response will allow the prediction of all of his other answers. For example, knowing that respondent B has a scale position (last consis- tent positive answer) of two (2) permits one to correctly predict that he also answered “Yes” to question one (1) but answered “No” to ques- tion three (3).

In actual research situations it is highly unlikely that a perfect scale will be found. Rather in most situations inconsistent responses will be encountered, i.e., a negative response before a positive one. The possi- bilities of inconsistent responses can be examplified by the simplistic example in Table II.

If the researcher is attempting to scale three questions, there are eight (8) possible response patterns. Four of those patterns contain in- consistencies. (The inconsistencies are circled.)

In actual research it is likely that some of these inconsistent patterns

TABLE II

Possible Scale Response Patterns.

Response patterns Questions

1 2 3

A + + + B + 8 + C + + - D 8 + + E + - - F - - tB G - 63 - H - - -

253

will be found. Thus the question which must be faced is what degree of inconsistency can be accepted without violating the unidimensional and cumulative assumptions of Guttman scale theory. Two mathematical coefficients have been formulated to answer this question.

(2) Coefficient of Scalability Inconsistencies (CS) = 1 - -- ___-- (Menzel, 1953) Maximum Inconsistencies Possible

(The maximal inconsistencies possible is equal to the sum of the non- modal responses of each of the respondents.) A CR of .90 and a CS of .60 have been established as the minimum standards for accepting a scale as unidimensional.

Given this explanation of Guttman scaling, it would appear that the technique claims two abilities: 1. Guttman scaling can differentiate be- tween items which are characterized by a common underlying unidi- mensionality and those which are nondimensional. 2. It can differen- tiate between items which are characterized by a common underlying unidimensionality and those that are characterized by an underlying multidimensionality.

Each of these claims will be evaluated in relation to both hypothe- tical data and judicial behavior data. Both judicial and legislative behav- ior have been selected for special attention because of the extensive use of Guttman scaling in these subdisciplines. When judicial behavioralists use Guttman scaling, judges are the respondents; cases pose the ques- tions; and votes cast by judges provide the answers.

After completing the consideration of the mathematical viability of Guttman scaling, this paper will conclude with an evaluation of the con- ceptual utility of Guttman scaling for explaining the complex voting be- havior of Supreme Court Justices.

II. Unidimensionality V. No Dimension

Several authors have suggested that under certain circumstances CR's and CS’s above the minimum required can be achieved by chance (Schuessler, 1961; Sagi, 1959). If these charges are in fact true, the abi- lity of Guttman scales to differentiate between the existence of a uni- dimensional pattern and no pattern at all must be called into question. This charge is based upon the fact that the probability of a particular

254

response pattern is based upon the marginal frequencies of the re- sponses to each question. This point can be exemplified by the exami- nation of some simplistic hypothetical data. If, as discussed earlier, the researcher is attempting to determine if three questions constitute a scale and are therefore tapping a single underlying dimension, there are eight possible response patterns. If we assume that the marginals are as those of Table III the probability of the chance occurrence of each response pattern can be determined. For example, the probability of response pattern A (Table II) is equal to (.75 X .50 X .25) = 9.4%. Using the same procedure for each of the patterns, the probabilities in Table IV were determined for each response pattern. It will be noted that re- sponse patterns A, C, E and H contain no inconsistencies. Thus by sum- ming their probabilities, we can determine the percentage of respon- dents who should be perfect scale*types due to chance for this particu- lar case:

A(.094) + C(.281) +E(.281) + H(.094) = 75%.

Similarly\, we should expect that approximately 25% of the respondents will fall into response patterns which contain inconsistencies (B, D, F, and G). However, only one out of the three responses in each of these patterns is inconsistent. Therefore, 66% of all the responses in all of these remaining patterns will also be consistent. This 16.7% (2/3 X 25 = 16.7) constitutes additional consistent response behavior that would be expected by chance. Thus, given the number of questions and marginal frequencies specified, we should expect a CR of .92 by mere chance alone. Similarly, a CS of greater than .60 should also be expected by chance [ 1 I.

Schooler (1968) has found that when the number of questions is low (ten or less) the probability of achieving a CR and CS above .90 and .60 respectively is quite high. Additionally, even when the number of

TABLE III

Hypothetical Response Frequencies.

Question Response Frequency (%)

Yes No

1 15 25 2 50 50 3 25 15

25.5

TABLE IV

Probability of All Possible Response Patterns.

Response patterns Questions Probability

1 2 3

+ + + 0.094 + - + 0.094 + + - 0.281 - + + 0.031 + - 0.281 - + 0.031 - + 0.094 - - - 0.094

questions is above 10, if the marginal frequencies are extreme, the pro- portion of the CR and CS due to chance increases.

Two alternative solutions to this problem present themselves. First, the difference between the expected and the observed CR could be tested for statistical significance. Second, it can be established that no scale of less than ten dichotomous items can be accepted and that items with extreme marginals should be eliminated. The latter alterna- tive has been recognized by scholars of judicial behavior. Schubert (1959) has indicated that no scale should be formed with less than ten cases, and that unanimous and single dissent cases should be excluded from the calculation of the CR. The exclusion of unanimous and single dissent cases puts a ceiling of 78% (7/9) on the marginal frequencies of cases included in judicial Guttman scaling. Thus, it appears that the judicial behavioralist has dealt with this particular problematic aspect of Guttman Scaling.

III. Unidimensionality V. Multidimensionality

We have seen that it is possible to differentiate between a unidimen- sional set of responses from a single underlying object and a set of chance response patterns by care of utilization of the method. It still remains to be seen if Guttman scaling can distinguish between a unidi- mensional set of response patterns and a multidimensional set. We have seen that increasing the number of items (questions) in a scale increases the reliability of the CR and C’S, with respect to differentiating between unidimensionality and nondimensionality. However, such an increase

256

may have negative ramifications for differentiating between a unidi- mensional array and a multidimensional array. For example, as the number of items increases, it becomes increasingly possible for extrane- ous items to be included, but to go undetected because of their relative- ly small frequency in comparison to the dominate set of items. Thus, it appears that a CR and C’S significantly better than might be expected by chance does not ensure that the items are totally unidimensional.

This problem is particularly acute in judicial behavior, where cases constitute the questions (or items) and may easily number up to one hundred or more. Such masking of a multidimensional array by a Gutt- man scale will be exemplified with a set of data dealing with the Nixon appointtees to the Supreme Court. The scale is designed to measure support for governmental authority. As can be seen the scale has a CR

TABLE V

Guttman Scale a: Governmental Authority and the Nixon Court

Cases Justices b

RE BU BL PO Total

1 2 3 4 5 6 I 8 9

10 11 12 13 14 15 16 17 18 19 20

X X X X X X X X X

7 X X X X - -

X

X X X X X X X X X X X X

I X X X X X X - X

l-3 l-3 2-2 1-3 l-3 l-3 l-3 l-3 2-2 2-2 2-2 2-2 l-l 3-l 3-l 3-l 3-l 2-2 2-2 3-l

a CR = 1 - 6178 = 0.92; CS = 1 - 6127 = 0.71. b RE = REHNIQUIST; BU = BURGER; BL = BLACKMUN; PO = POWELL; / = Justice not a member of the Court at time case was decided; X = Consistent vote in favor of governmental authority;@= Inconsistent vote in favor of govern- mental authority; Blank represents consistent negative vote (against governmental authority); - = Inconsistent negative vote (against governmental authority).

257

and CS above the required level and appears to be a good scale. IIew- ever, examination of the cases themselves reveals an interesting pattern. While all of the cases deal with governmental authority, they can be subdivided on another dimension: those cases dealing with economic issues and those dealing with civil libertarian issues. This analysis re- veals that cases 1, 2, 12, 15, 17, 18, 19 and 20 deal with economic is- sues, while the other twelve deal with civil libertarian issues. By com- paring the two subsets of cases it becomes manifest that the economic cases account for 5 out of the 6 errors in the scale. If the eight econo- mic cases are removed, an almost perfect twelve items scale results (one error). The economic cases, on the other hand, do not meet the CR criteria of scalebility.

Thus, although the original scale indicated that the Nixon Court Jus- tices reacted to a unidimensional concept of governmental authority, closer examination reveals a two-dimensional array: Governmental authority in economic issues and governmental authority in civil liber- ties issues. Here again, this type of error can be prevented, but only by careful examination of the inconsistencies and thorough knowledge of the subject area. The investigator must subjectively decide if the incon- sistencies have any common dimensionality other than that being tap- ped by the scale. This procedure appears to be adhered to by all judicial behavioralists whose articles were examined by the authors (Ford, 1954).

IV. What Dimension is being Tapped?

We have seen that Guttman scaling appears to be mathematically sound. However, the application of Guttman scaling to the study of legislative and judicial behavior presents serious conceptual problems which do not confront the user who is merely attempting to tap an un- derlying value dimension of a subject of the general population. For example, in the latter case attitudes, such as alienation, are being de- termined by means of the scale and then related to other behavior, as either an independent variable (alienation + voting) or as a dependent variable (social background + alienation). Conversely, the legislative and judicial behavioralists use Guttman scaling to derive the indepen- dent variable (attitudes of justices) from the dependent (decisions). Such a procedure leads to conceptual difficulties.

It is clear that in Schubert’s work (1965) he has tapped some under- lying dimension in the voting behavior of the justice by means of Gutt- man scaling. However, the question remains “What dimension is it?“.

258

Schubert contends that the dimension is attitude toward civil liberties claims. However, Grossman (1962) makes the argument that much of what is being measured and called attitudes is really an outward mani- festation of role behavior. He gives as an example Justice Felix Frank- furter, who in all of the Guttman scale analyses is described as very conservative. Grossman questions this finding. Rather he contends that Frankfurter’s voting behavior is a function of his judicial role percep- tion. It is well documented that Frankfurter felt that the judicial should defer to the elected branches of government.

Combining this fact with an examination of Frankfurter’s non-deci- sional attitudes (pre-court speeches, off-the-bench speeches and arti- cles), which reveals strong liberal feelings, leads Grossman to conclude that Schubert’s Guttman scales have been tapping the judicial restraint of Frankfurter, rather than his conservative attitudes. A close examina- tion of the scales reveals that there is nothing in them which prevents this interpretation, but rather the conclusions of the scales are based upon Schubert’s assumption that “attitudes” was the appropriate apellation for the discovered underlying dimensionality. Instead of measuring the variance from liberal to conservative, the variance may be from judicial activism to judicial restraint. Thus, in contrast to scaling responses to questions given to members of the public, the legislative and judicial behavioralists are dealing with a complex set of output data which could be the result of many different independent variables, i.e., role perception, organizational constraints, or group decision- making. As Anderson et al. (1966) note in relation to legislative analy- sis, “the conceptual meaning of. . . scalar patterns in votes . . . is by no means clear.” They go on to observe that legislative voting behavior is the result of a complex set of factors in addition to the attitudinal vari- able.

The primary problem appears to be that legislative and judicial votes are not a direct analog to responses to explicit questions. There are a multiplicity of organizational and role factors which may distort the re- lationship between a judge’s actual attitudes and his voting behavior. This conclusion does not mean that scaling is useless for the study of legislative and judicial behavior. However, it does point to the fact that inferences drawn from Guttman scales in these two subdisciplines must be tempered with the realization that the behavior being scaled is a result of a complex set of variables and cannot be as easily conceptua- lized as answers to a questionaire. Thus, both legislative and judicial behavioralists must be particularly alert to the possibility of spurious results being derived from Guttman scales, due to the effect of inter- vening variables which are inherent in the organizational setting in

259

which the decisions are being made, i.e., Frankfurter’s perception of the judicials proper role in American society.

In this article we have examined the mathematical problems inherent in forwarding claims of unidimensionality of a set of decisions based upon legislative and/or judicial “votes”. In addition we have indicated how one can easily be mislead in the interpretation of results obtained by Guttman scale and analysis due to the inability of scaling techniques to offer more than an objective mathematical pattern, rather than enlightening subjective knowledge. While these problems may be gener- ally familiar to scholars utilizing scaling techniques, the authors felt it appropriate to offer specific evidence of the weakness of this method, especially with reference to behavioral studies of decision-making in a collegial body.

Notes

1 Assuming CR of .92 (that expected by chance), we would obtain a CS of .78 for this data. Similarly the minimal marginal reproducibility would be .667; the percent improvement .26, see Nie et al. (1970, p. 201) for description of mea- surements.

References

Anderson, L., Watts, M. and Wilcox, A. (1966). Legislative Roll-Call Analysis, p. 119. Evanston, Illinois: Northwestern University Press.

Ford, R.N. (1954). “A Rapid Scoring Procedure for Scaling Attitude Questions,” pp. 59-67 in M.W. Riley et al., Sociological Studies in Scale Analysis, New Brunswick, NJ: Rutgers University Press.

Goldman, S and Jahnige, T. (1971). The Federal Courts as a Political System, pp. 159-167. New York: Harper & Row.

Grossman, J.B. (1962). “Role Playing and the Analysis of Judicial Behavior: The Case of Mr. Justice Frankfurter,” Journal ofPolitics 11: 285-309.

Guttman, L. (1944). “A Basis for Scaling Qualitative Data,” American Sociological Review 9: 139-150.

Menzel, H. (1953). “A New Coefficient for Scalogram Analysis,” Public Opinion Quarterly 17: 268-280.

Nie, N., Bent, D.H. and Hull, G.H. (1970). Statistical Package For the Social Sciences, pp. 196-199. New York: McGraw-Hill.

Sagi, P.O. (1959) “A Statistical Test for a Coefficient of Reproducibility,” Psycho- metrica 24: 29-42.

Schooler, C. (1968). “A Note of Extreme Caution on the Use of Guttman Scales,” The American Journal of Sociology 74: 296-301.

Schubert, G. (1959) Quantitative Analysis of Judicial Behavior, chpt. 5. New York: Free Press.

260

Schubert, G. (1965). The Judicial Mind: Attitudes and Ideologies of Supreme Court Justicies, 1946-1963, pp. 99-127. Evanston, Illinois: Northwestern University Press.

Schuessler, K.F. (1961). “A Note on Statistical Significance of Scalogram,” Socio- metry 24: 312-318.