Acta Psychologica 56 (1984) 49-57 North-Holland

49

THE EVALUATION OF MANAGERIAL DECISIONS’ QUALITY BY MANAGERS

Dan ZAKAY * Tel - Auio University, Israel

The purpose of the study was to explore the criteria used by mid-level managers for the evaluation of managerial decisions’ goodness. A questionnaire composing of 25 items was administered to 145 managers who were asked to assign each item a score indicating its suitability to serve as a

criterion for decisions’s goodness. A factor analysis performed on these scores yielded eight

meaningful factors. The factors were interpreted as representing the following criteria, which are ordered according to the values of the factors’ scores: (1) goodness of outcomes, (2) correctness of

the decision process, (3) information utilization, (4) realism and resources, (5) ethics, (6) subjective

rationality, (7) acceptance, and (8) feelings and social compromise.

Pfeifer’s Principle: “No one keeps a record of decisions you could have made but didn’t. Everyone keeps a record of your bad ones” (Arthur Block 1980).

Managerial decision making is done today in an environment more complex than ever before (Keeney 1982). This situation is posing a severe threat to decisions’ quality as systematic divergences from opti- mal decision procedures by intuitive decision makers have often been demonstrated (e.g. Nisbett and Ross 1980; Hogarth 1981; Kahneman et al. 1982). In order to cope with this threat, attention is focused today on the training of decision makers in the use of procedures which aim to improve their decision making performance (e.g. Nickerson and Feerer 1975; Fischhoff 1982). A clear definition of a good decision is essential for the success of this new trend of research and application. Developers of such procedures (or “decision aids” as they are often called, e.g. Howard 1968; Kepner and Tregoe 1965; Edwards 1977; Zunker 1981), must define what a good decision is in order to test the

* Author’s address: D. Zakay, Dept. of Psychology, Tel-Aviv University, Ramat Aviv 69978, Israel.

OOOl-6918/84/$3.00 0 1984, Elsevier Science Publishers B.V. (North-Holland)

50 D. Zakay / Managerial decisions’ quality

efficiency of their recommended methods, relative to the use of intuitive models. Successful implementation of these normative models depends on the acceptance of them by its consumers. However, the chances of such an acceptance depend to a large extent on the decision makers’ perceived benefit of using such models relative to that of using intuitive ones. The decision makers’ way of evaluation that relative benefit is, by definition, related to the way they evaluate what a good decision is. The fate of decision analysis (e.g. Brown et al. 1974) can serve as an example. Brown (1970) found that the utilization of decision analysis by managers was problematic, mainly because they were not sure about the practicality and benefit of its usage. As a result “most analyses of important decision problems have left the incorporation of judgments and values to informal procedures with unidentified assumptions and to the intuition of the decision makers” (Keeney 1982: 830).

As with regard to the whole decision process, so with regard to defining what a good decision is, two approaches are evident: What is the normative definition of a good decision vs. what is the intuitive definition of a good decision. Usually, normative definitions are focus- ing on some relationship between decisions and some criteria. However, a definition of an acceptable set of criteria for decisions’ goodness is problematic, and many approaches can be found in the literature. One such approach is that of axiomatic rationality which is based on the concepts of “economic man” and of utility maximization. Accordingly, a good decision is one in which the requirements of rationality are met and positive utility is maximized (e.g. Lee 1971). However, today, the suitability of axiomatic rationality to serve as a criterion for decisions’ goodness is questionable in light of recent empirical findings about decision making behaviour. The concept of “bounded rationality” which was introduced by Simon (1957, 1978) might serve better the purpose of defining decisions’ goodness. According to this line of reasoning Lee defined a good decision as “the best decision for a certain individual in a specific decision situation” (1971: 7) and Einhorn and Hogarth defined decision optimality as “decisions or judgments that maximize or minimize some explicit and measurable criterion (e.g. profits, errors, time) conditional on certain environmental assumptions and a specified time horizon” (1981: 57). Those definitions emphasize the dependency of decision goodness or optimality on environmental and situational conditions. A different approach is the one which suggests the amount of deviation from some normative model as a

D. Zakay / Managerial decisions’ quality 51

criterion for decisions’ goodness. This criterion was criticized because deviation from a normative model might not be a reflection of any real “error” et all. Edwards (1962) as well as Winkler (1973) claim that unlike experimental situations, in realistic ones, no single normative model can serve as a basis for the evaluation of decisions’ goodness. Empirical findings regarding the dependency of conservatism behaviour on the exact nature of the situation (e.g. Slavic and Lichtenstein 1971) are in support of this criticism.

Another different approach is the one which sees the value of decisions’ outcomes as a criterion for its goodness. This criterion, despite its being appealing intuitively, might be misleading since un- satisfactory outcomes cannot necessarily be traced to bad decision making. “Since good decisions can lead to bad outcomes (and vice versa) decisions makers cannot infallibly be graded by their results” (Brown et al. 1974: 4).

Less effort has been devoted to explore the intuitive definition of good decisions. However, this is of the utmost importance since it is crucial for any evaluation model to be accepted by relevant decision makers in order to be effectively used. This was found to be true in the area of clinical decision making (Dawes 1976). Hence, revealing the intuitive criteria might help in building acceptable evaluation models.

The purpose of the present study was the exploration of the criteria used intuitively by managers for the evaluation of managerial decisions’ goodness.

Method

Sample

One hundred eigthy five mid-level Israeli male managers with ages ranging between 30 to 45 years, from three high technology electronic industries, volunteered to participate in the study.

Procedure

Decisions’ Goodness Questionnaire (DGQ)

The DGQ was built by the following method. Forty managers were selected so that all three industries were properly represented. Each one of these managers was asked to answer in writing the following questions: (1) what is the definition of a good decision as he sees it, and (2) how and in what ways was he actually evaluating the goodness of

52 D. Zakay / Managerial decisions’ quality

decisions reached by himself or by his subordinates. The descriptions obtained were subjected to content analysis, done independently by three qualified organizational psychologists. Twenty-five criteria were defined and agreed by all three judges. The criteria are presented in table 1.

In the second phase of the study the DGQ was administered to the remaining 145 managers, who were asked to rate each criterion on a 10 point scale with 0 denoting that the criterion was not a good one and 10 denoting that it was a very good one for evaluating decisions’ goodness. It was emphasized that there were no correct or wrong answers. All managers completed the DGQ in their own offices and secrecy was ensured.

Table 1

The Decision’s Goodness Questionnaire.

Item

no.

Mean score

Content

I-1 I-2

1-3

1-4

I-5

9.29 The outcomes of the decision are good

6.67 The decision is accepted by those who need to implement it

7.82 All of the existing information is used

8.47 Attempts are made to obtain missing information

6.52 The decision maker is aware of the possibility of personal biases

I-6 7.52 I-7 8.68

I-8 7.26 I-9 8.08

I-10 8.82 I-11 7.21 I-12 6.39 I-13 4.17

I-14 4.82 I-15 7.56 I-16 5.54 I-17 7.69 I-18 6.95

I-19 6.00

I-20 8.73

I-21 8.68

I-22 6.91

I-23 4.49 I-24 7.93 I-25 4.21

and tries to avoid them Decision aids are used when necessary The decision maker possesses a picture of the situation, the

contingencies and the possible results of the decision New alternatives were actively searched for The results of all the potential outcomes of all the alternatives

are weighted and considered

Relevant professional knowledge is used The decision is made at the right time The decision process is clear and easy to explain

The decision maker has a good feeling about the decision The decision is satisfactory to superiors The decision could be carried out

The cost of the decision is low The decision is logical in the light of the available data

Internal consistency among all the values and probabilities used is kept

The majority of the decision maker’s peers agree upon the ethical

social, and cultural values underlying the decision Before reaching the decision there is an attempt

to understand the problem and its causes At the time of the decision all of the potential risk factors

are weighted The decision maker does not make the decision based on personal

utility Lesson learned in previous similar decisions was applied All relevant alternatives have been compared Political pressures were considered and handled

D. Zakay / Managerial decisions’ quality 53

Table 2

Eigenvalues, Pet. of explained variance and communality Pet. of the factors.

Factor

A B

C

D

E

F

G

H

I

J

1.72

3.18

2.28

2.04

1.75

1.66

1.36

1.09 0.91

0.83

Pet. of explained

variance

Communality

Pet.

33.75 33.75

13.90 41.65

9.96 57.61

8.94 66.55

7.11 74.32

7.26 81.58

5.89 87.49

4.85 92.32

4.10 96.42

3.58 100.00

Results

The three industries were compared in regard to each criterion and since no significant differences were found the different industries were combined in further analyses. The scores obtained for the criteria were subjected to factor analysis with an orthogonal rotation (the mean scores of the 25 criteria are presented in table 1). Ten factors were obtained after a varimax rotation with Kaiser normalization. The eigenvalues are presented in table 2.

Since an eigenvalue of 1.0 was selected as a criterion only the first 8 factors were defined as meaningful. The criteria utilized for the inclusion of an item in a factor were: (1) that its loading on the factor is higher than 0.30, and (2) that its loading on any other factor is lower than 0.30. Only items 10 and 21 have loadings higher than 0.30 on two factors. Factor scores were calculated for each S as a weighted average of the scores given to items composing each factor, with the loadings as weights. The structure of the 8 factors is presented in table 3.

Discussion

The eight meaningful factors which were obtained were interpreted on the basis of the contents of the items composing each one. Factor A was interpreted as representing realism of the decision and amount of resources invested in the decision process as criteria for decisions’ goodness. This factor emphasizes as well the correctness of the decision making process itself. Factor B was interpreted as representing mainly the utilization of information as a criteria for decisions’ goodness. It is of interest to note that utilization of decision aids is related to this

Tab

le 3

T

he

stru

ctu

re o

f th

e 8

mea

nin

gfu

l fa

ctor

s.

B

Ia

Lb

10

0.34

11

0.

70

12

0.49

15

0.

70

16

0.46

21

0.

60

24

0.34

Mea

n f

acto

r sc

ore

7.59

8.

11

6.89

4.

33

S.D

. 1.

66

1.28

1.

68

1.89

I L

3 0.

67

4 0.

84

5 0.

51

6 0.

38

20

0.40

21

0.

46

C

D

I L

I

L

17

0.66

13

0.

72

18

0.80

14

0.

80

19

0.59

25

0.

35

E

F

G

H

I L

I

L

1 L

I

L

P

7 0.

75

2 0.

60

1 0.

69

22

0.47

N

9

0.66

%

9

10

0.38

\

$ : 0 9.

%

9 8.

50

6.67

9.

29

6.91

a.

‘“

. 1.

29

1.61

2.

11

2.45

0 %

Nam

e gi

ven

’ R

eali

sm

and

reso

urc

es

Info

rmat

ion

S

ubj

ecti

ve

Fee

lin

gs

Cor

rect

nes

s A

ccep

tan

ce

Ou

tcom

es’

Eth

ics

“:

uti

liza

tion

ra

tion

ali-

an

d so

cial

%

of

th

e go

odn

ess

Q

tY

com

prom

ise

deci

sion

pr

oces

s

a I

-Ite

ms

incl

ude

d in

th

e fa

ctor

b

L -

L

oadi

ng

C

- S

ee t

he

disc

uss

ion

sec

tion

D. Zakay / Managerial derisions’ quality 55

factor. In other words, managers see the main advantage of decision aids in relation to information utilization. This might indicate that the most acceptable use of computers in decision making is as data bases. Factor C was interpreted as representing subjective rationality since the emphasis is on the logic and internal consistency of the values used and on those values being generally acceptable. All these criteria represent subjective values in terms of specific group, culture or a society. Factor D was interpreted as representing emotional criteria, i.e. having good feelings about the decision. Another emphasis here is on acceptance of the decisions by superiors and by not causing unnecessary turbulence. Factor E is similar to factor A, and indeed the criterion of using relevant professional knowledge is common to both. However, factor E is dealing mainly with the correctness of the decision process. Ir~~ctor F includes only one criterion, that of the decision being accepted by those who should implement it. This criterion is of course necessary in an industrial environment. Fuctor G includes only one criterion, that of goodness of outcomes. Factor H includes only one criterion, that of organizational utility being utilized in the decision process rather than personal utility. This might be considered as representing an ethical criterion. The overall picture, based on the ranks of the mean factor scores is that managers use outcomes as the most important criterion for decisions’ goodness. This criterion (factor G) obtained the highest score. A second group of criteria (factors E, B, A,) which were rated as “second best” includes correctness of the decision process, utilization of all relevant information while investing minimal resources in the decision process and being aware of environmental constrains like time. A third group of criteria which were rated third in importance (factor C, H) deals with the values used in the decision process. The emphasis here is on using values which represent organizational utility rather than personal utility, and which are generally accepted by the social environment. Another demand is that internal consistency will be kept among the values in the decision process. The least important criterion (factor D) was that of having good feelings about the decision. How- ever, this might reflect the climate in sophisticated industries and might be more important in public services, etc.

Some interesting conclusions might be drawn from the findings. It is clear that an evaluation model should include social, economical, environmental, and ethical criteria in addition to those which are directly related to decision theory. Nevertheless, it is encouraging that

56 D. Zakay / Managerial decisions’ quality

managers understand the importance of correct decision process, which is the main criterion for decisions’ goodness according to the methodol- ogy of decision analysis (e.g. Brown et al. 1974). On the other hand, the emphasis of outcomes as a criterion is worrying because it indicates that managers do not understand the inherent character of decision making under uncertainty, where a good decision process may result in an undesired outcome and vice versa. In addition, a naive reliance on outcomes can seriously prejudice the evaluation of decisions made in the past and limit what is learned from experience (Fischhoff 1975). This finding calls for the development of specific training methods aimed at teaching the problematic nature of the outcomes’ criterion.

A general conclusion, which is evident from the results is that the intuitive perception of good decisions is much more similar to the definitions which emerge from the concept of bounded rationality than to those which are anchored on the concept of axiomatic or normative rationality.

The findings obtained in this study should be validated in future research which should incorporate various types of decision makers in order to find out if intuitive models of decisions’ evaluation are dependent on organizational and technological factors and on the nature of the decisions which should be taken.

Block, A., 1980. Murphy’s law. Book two. Los Angeles, CA: Rice, Stem, Slow. Brown, R.V., 1970. Do managers find decision theory useful? Harvard Business Review 48, 78-89. Brown, R.V., AS. Kahr and C. Peterson, 1974. Decision analysis for the manager. New York:

Holt, Rinehart & Wiston. Dawes, R.M., 1976. ‘Shallow psychology’. In: J.S. Carroll and J.W. Payne (eds.), Cognition and

social behavior. Hillsdale, NJ: Lawrence Erlbaum. Edwards. W., 1962. Dynamic decision theory and probabilistic information processing. Human

Factors 4, 59-73.

Edwards, W., 1977. ‘Use of multiattribute utility measurement for social decision making’. In: D.E. Bell, R.L. Keeney and H. Raiffa (eds.), Conflicting objectives in decisions. New York: Wiley.

Einhom, H.J. and R.M. Hogarth, 1981. Behavioral decision theory: processes of judgment and choice. Annual Review of Psychology 32, 53-88.

Fischhoff, B., 1975. Hindsight # foresight: the effect of outcome knowledge on judgment under uncertainty. Journal of Experimental Psychology: Human Perception and Performance 1,

288-299. Fischhoff, B., 1982. ‘Debiasing’. In: D. Kahneman, P. Slavic and A. Tversky (eds.), Judgment

under uncertainty: heuristics and biases. New York: Cambridge University Press.

D. Zakay / Managerial decisrons’ quality 57

Hogarth, R., 1981. Judgment and choice. New York: Wiley. Howard, R.A., 1968, The foundations of decision analysis. IEEE Transactions in Systems Science

and Cybernetics SSC-4, 211-219. Kahneman, D., P. Slavic and A. Tversky, 1982. Judgment under uncertainty: heuristics and biases.

New York: Cambridge University Press. Keeney, R.L., 1982. Decision analysis: an overview. Operational Research 30, 803-838. Kepner, C.H. and B.B. Tregoe, 1965. The rational manager. New York: McGraw-Hill. Lee, W., 1971. Decision theory and human behavior. New York: Wiley. Nickerson, R.S. and C.E. Feerer, 1975. Decision making and training. Cambridge, MA: Bait

Beranek and Newman, Inc. Nisbett, R. and L. Ross, 1980. Human inference: strategies and shortcomings of social judgment.

Englewood Cliffs, NJ: Prentice-Hall. Simon, H.A., 1957. Models of man. New York: Wiley. Simon, H.A., 1978. Rationality as process and as product of thought. American Economic Review

68, 1-16. Slavic, P. and S. Lichtenstein, 1971. Comparison of Bayesian and regression approaches to the

study of information processing in judgment. Organizational Behavior and Human Perfor- mance 6, 649-744.

Winkler, R.L. and A.H. Murphy, 1973. Experiments in the laboratory and the real world. Organizational Behavior and Human Performance 10, 252-270.

Zunker, V.G., 1981. Career counseling: applied concepts of life planning. Monterey: Brooks, Cole.