CONFIDE: A Collective Decision-Making Procedure Using Confidence Estimates of Individual Judgements

179CONFIDE: COLLECTIVE DECISION-MAKING PROCEDUREGroup Decision and Negotiation 7: 179–194, 1998

© 1998 Kluwer Academic Publishers. Printed in the Netherlands

CONFIDE: A Collective Decision-Making ProcedureUsing Confidence Estimates of Individual Judgements

DENNIS P. SLEVINKatz Graduate School of Business, University of Pittsburgh

LARRY W. BOONEDepartment of Management, St. John’s University

EILEEN M. RUSSOHRDQ, King of Prussia, PA

RICHARD S. ALLENSchool of Business Administration, University of Tennessee at Chattanooga

Abstract

An algorithm (termed CONFIDE) is proposed that may capture many of the benefits of group decision makingwithout the necessity of face-to-face interaction. The algorithm allows individual decision makers to differentiallyweight the contributions from members according to the confidence with which each member holds to theiropinions. The CONFIDE algorithm is compared to both face-to-face group decisions and simple averaging ofgroup members opinions on the Lost-at-Sea ranking task. Results indicate that, in terms of decision quality, theCONFIDE algorithm produces solutions equal to that of the face-to-face group decision method and significantlybetter than the solution achieved by simple averaging of group members’ responses.

Key words: group decision making, individual decision making, collective decision making, confidence estimates,self assessments, lost at sea, ranking exercises

Managers are increasingly faced with the task of making complex decisions in turbulentorganizational environments (Huber 1984; Kilmann 1989). Two key forces are currentlyhaving a profound impact on managerial decision making: (1) a move towards more team-based organizations which necessitates an increased use of group decision-making methods(Lawler 1986, 1992; Stewart 1989; Hackman 1990) and (2) the growth of telecommutingwhich increases the geographical and temporal dispersion of organizational members (Olson1985; Phelps 1985; Ramsower 1985; Silver 1989). A net result of these two key forces isan increased reliance on technology-assisted methods to facilitate group decision makingsuch as e-mail, fax, and Group Decision Support Systems (GDSS) which organizationsimplement to facilitate group interaction in settings where the work force is geographicallyor temporally dispersed (Kiesler and Sproull 1992; Er and Ng 1995).

Group decision making using electronic media is substantially different from face-to-face decision making. Electronic media alter the informational, temporal and interactionalprocesses by which groups work (Kiesler and Sproull 1992; Hollingshead and McGrath

180 SLEVIN, BOONE, RUSSO AND ALLEN

1995). Certain advantages and disadvantages are a result of these changes. Advantagesinclude the potential for improving access to information and information-processingcapabilities. Also there is the potential for increasing the participation of group membersby reducing temporal and geographic impediments. Some disadvantages of using electronicmedia in group decision making include a drastic reduction in the number and variety ofmodalities of communication such as the loss of nonverbal, paraverbal, and social-statuscues. There can be a resultant reduction in the “richness” of information (Daft and Lengel1986).

Recent reviews (Fulk and Boyd 1991; Hollingshead and McGrath 1995) comparingtechnology-assisted decision making versus traditional face-to-face decision makingconclude that, in general, technology-assisted groups take longer to make decisions anddo not consistently produce decisions of higher quality than groups which meet face-to-face. Obviously these findings do not bode well for practitioners who must increasinglyrely on the use of technology to facilitate group decision making in today’s geographicallyand temporally dispersed organizations.

This paper proposes and tests a group decision-making technique, termed CONFIDE,which is appropriate for technology-assisted decision-making environments. This techniquehelps to overcome the speed and quality shortcomings of technology-assisted group decisionmaking. The CONFIDE technique allows the collection of opinions and estimates ofconfidence from individuals on a decision problem. The opinions and confidences arecombined using an algorithm and are tested in comparison to face-to-face consensus andsimple averaging of individual opinions in order to assess the quality of these three differentdecision-making methods. Results indicate that the CONFIDE approach provides decisionsequal in quality as those arrived at via face-to-face decision making without the necessityof time consuming, face-to-face, group meetings.

1. Group decision making

Humans are imperfect decision makers. Our spans of absolute judgement and of immediatememory impose severe limitations on the amount of information that we are capable ofreceiving, processing, and remembering (Miller 1956). As a consequence, decision makersselect alternatives based upon a simplified model of a situation, and this model is affectedby the unique frame of reference or psychological set of the decision maker (Simon 1979).The task then becomes searching for ways to compensate for the limitations of the individualdecision maker.

Managers may find it useful when facing a complex decision to gather informationfrom several individuals who have various perceptions of a problem and different areas ofexpertise. Because individual managers lack requisite knowledge to make complex decisionsalone, they may be led to convene a small group of individuals, each of whom is believedto hold important information or opinions relevant to the decision problem. While thismethod of decision making may provide information, it may also add complexities stemmingfrom interactions within small groups.

The manager in this case has an extremely intricate job, managing numerous small

181CONFIDE: COLLECTIVE DECISION-MAKING PROCEDURE

group variables such as power relationships, differential ability of group members, andgroup norms, and information flow while simultaneously attempting to maximize the qualityof the group decision (Vroom and Yetton 1973; Dyer 1977; Yukl 1981; Mannix 1993).The leader of a small group must make sub-decisions as to what information should begathered from whom and how this information should be weighted in the final decision.Thus, while use of a group decision-making process typically results in higher confidencein the decision than the use of individual decision making, the quality of the group decisionis not always necessarily superior to that of decisions made by individuals (Lorge andSolomon 1955; Steiner 1972; Davis 1992; Sniezek 1992) and it is not certain that theincreased investment associated with group decision making has a payback in increaseddecision quality.

Groups sometimes may not produce decisions equal to their potential as suggested bystatistical pooling methods (Lorge and Solomon, 1955). This is due in part to processlosses stemming from poor methods of information integration and social process (Steiner1966, 1972). Hidden profile research conducted by Stasser and Titus (1985) and Stasser(1992) support the notion that decision-making groups often fail to use key informationthat is held by specific group member experts. The information sampling dynamics inunstructured, face-to-face discussions tend to prevent the effective pooling of unsharedinformation.

An alternative method of decision making that may avoid the expenditure of time andenergy in the management of a small group process, but still benefit from the increasedknowledge base of several individuals is the aggregation of individual opinions withoutface-to-face interaction. This option may provide some of the benefits of group interactionwhile defraying many of the costs and communication barriers inherent in group decisionmaking. Several studies have demonstrated the improvement realized by the aggregationof subjective individual forecasts (Goldberg 1970; Ashton and Ashton 1985; Libby andBlashfield 1986). Additionally, a significant amount of research suggests that much of thegain in accuracy that can be achieved by aggregation is attained by combining just a smallnumber of forecasts or opinions (Einhorn et al. 1977; Hogarth 1977, 1978; Ashton andAshton 1985).

Computer-based information systems that provide the capability for processingindividual inputs to produce new information and group decisions have been developedand are currently being implemented and refined in organizational environments. Thesegroup decision support systems (GDSS) offer the potential for increasing the efficiencyand effectiveness of group decision making (Vroom and Yetton 1973; Delbecq et al. 1975;Mockler and Dologite 1991; Eckerson 1992; Jacob and Pirkul 1992; Reisman et al. 1992;Migliarese and Paolucci 1993; Gessner et al. 1994; Walker 1994).

Even if one has the technical capability of combining information from several sources,however, one must have a basis for weighting the gathered information. Many of the studiesaddressing this issue employ an equal weighting aggregation method. That is, aggregateopinions are formed by taking simple averages of individual opinions. However, Yettonand Bottger (1982) have demonstrated the improved performance attainable by weightingmore heavily those individual opinions expressed by more expert or skilled group members.Ashton and Ashton (1985) examined various differential weighting methods derived from


ex post and ex ante accuracy assessments, and in general found these aggregation methodsto produce more accurate forecasts than equal weighting methods. However, the weaknessof this procedure, as the authors discuss, involves the general lack of availability of accuracyinformation upon which to base differential weighting factors in realistic decision-makingscenarios, where correct answers are not known.

2. Confidence estimates

An alternative method is to utilize a group member’s own assessment of confidence as aweighting criterion in the combination of opinions. Empirical work in forecasting suggeststhat a confidence based consensus measure may be more accurate than a traditionalconsensus measure. Dalkey (1969) and Dalkey et al. (1979) found that using confidenceestimates of individuals increased the accuracy of decision results. Inclusion of only theresponses of highly confident individuals produced superior decisions in Delphi exercises.

Research concerning individual decision-maker confidence has demonstrated a lessthan perfect correlation between decision accuracy and confidence in decisions (Einhornand Hogarth 1978, 1981). Specifically, individuals tend to express unwarranted confidencein their fallible judgements. For example, decision makers should ideally be accurate on80% of the judgments to which they assign confidence estimates of 80%. Laboratory studiesindicate, however, that overconfidence is common. For a large number of two-choicequestions for which subjects were 75% confident that their judgement was correct, only60% of the questions had been answered correctly. When subjects expressed 100%confidence, typically they were correct only 85% of the time (Fischhoff 1982). Moreover,the more difficult the task, the more likely a subject is to be overconfident (Keasey andWatson 1989).

While studies of individual decision makers have shown that personal confidenceestimates are not perfectly correlated with accuracy (since there is a general tendency tobe overconfident in judging one’s own chance for success), these investigations havedemonstrated that a positive correlation exists between confidence and accuracy. Shraugerand Osberg (1982) have found that self-confident individuals are better able to predicttheir behavior than are individuals low in self-confidence. Moreover, if overconfidence isa constant and does not interact with other variables, the correlation between accuracy andconfidence remains useful.

3. Self-assessments

The use of individual self-assessments of ability in lieu of external indices of ability has beengenerally overlooked (Mabe and West 1982). The belief has been that individuals are eitherunable or unwilling to provide objective assessments of their ability. In addition to thepreviously discussed tendency to overrate confidence, there are a host of other cognitive andjudgemental biases which can come into play when individuals are asked to make self-assessments (e.g., Tversky and Kahneman, 1973; Fischhoff 1975; Agnew and Brown, 1986).


Recently, however, the assumption that individuals are unable or unwilling to provideobjective self-assessments has been called into question (Mabe and West 1982; Shraugerand Osberg 1982; Osberg and Shrauger 1986; Keasey and Watson 1989). Self-assessmentscan prove useful because individuals possess unique self-knowledge. As Shrauger andOsberg (1982) point out, there is empirical and conceptual support for the notion that self-assessors frequently have the appropriate information and motivation to make as effectivejudgments about their own behavior as can be made by any other means. Keasey andWatson (1989), in a study which incorporates experts’ confidence in the correctness oftheir input into group consensus decisions, conclude that inclusion of self-estimates ofconfidence provide important additional information with respect to explaining subjectsaccuracy. Mabe and West (1982) in a meta-analysis found a modest positive correlationbetween self-assessments and objective indices of ability (r = 0.29). This would suggestthat relying on subjects to rate themselves on ability dimensions may be a cost-effectivemanner in which to gather data.

This study seeks to examine the potential for extracting meaningful contributions ofindividuals on a decision problem through their related confidence estimates withoutresorting to the complex small group process.

4. Theoretical framework

Hollingshead and McGrath (1995) propose a conceptual model of the flow of decision-making effects for technology aided work groups. This model includes many variablesthat affect group decision making including input variables, operating conditions, processvariables and outcome variables. The flow of the model is as follows:

Input → Operating → Process → OutcomeVariables Conditions Variables Variables

The first group, input variables, consists of three major classes of variables: group andmember characteristics; tasks, projects and purposes; and communication systems. Thesecond group, operating conditions, consider the prevailing operating conditions underwhich the group is working such as the degree of anonymity, time pressure, changes intask, etc. The third group, process variables, reflect the ongoing group activity while thegroup is doing its work. These include such aspects as the distribution and amount ofparticipation by group members. The final group, outcome variables, are the final outputsof the group decision-making process such as time to reach decision, number of solutionsgenerated, quality of solutions, user reactions and member relations.

Our study utilizes Hollingshead and McGrath’s (1995) theoretical framework bymanipulating input variables in a group decision-making task (specifically through thetype of decision-making technique employed) and determining the impact on outcomevariables (namely decision quality and speed). This study limits itself to the testing of asubset of the variables proposed by Hollingshead and McGrath (1995). Our selection ofvariables was driven by criteria that are most pertinent to the previously mentioned current


developments in organizational group decision making including increased use oftechnology and telecommuting as well as changes in organization structure being broughton by the move towards more team-based organizations. The variables in our subset includethe following:

(a) Input Variables – This study attempts to hold the task and group member attributesconstant while varying the type of decision-making technique being utilized. Thedecision-making technique in this case is not determined by differences in computerhardware or software, but by the differences in structuring of the form and sequenceof member inputs to the group decision imposed by the decision-making techniqueemployed.

Three types of decision-making options were tested: (1) a simple average ofindividual confidence estimates, (2) group consensus arrived at via a face-to-facemeeting and, (3) a weighted average of individual confidence estimates based onindividual levels of confidence with no face-to-face meeting of the group.

(b) Process Variables – The distribution of participation among group members is variedin the three different conditions. In the individual situation all members participateequally. In the face-to-face situation members with more status, expertise or negotiationskills may have a greater influence on the group decision. In the no face-to-faceinteraction with confidence estimates situation the members with relatively higherconfidence in their position have a greater influence over the group decision. Otherprocess variables such as participation over time or among functions, or politicalimplications are not explored in this study.

(c) Outcome Variables – This study examines the impact of the three input conditions ontime to reach solution and quality of decisions. It does not address other potentialoutcome variables such as user reactions (e.g., satisfaction or commitment to the processor outcomes, ease or effectiveness of implementation), creativity levels, or memberrelations (e.g., attraction or feelings of impersonality or alienation).

5. CONFIDE model

An algorithm for combining individual confidence estimates was needed for this study.Agnew and Brown (1986) have proposed a lag regression model to suggest how individualsuse confidence to arrive at decisions. They suggest:

Ef = f(C, Ei + (1 – C ) g (Es))

or, Final Estimate (Ef) is a function of Confidence (C) in Initial Estimate (Ei), adjusted bythe distribution of remaining confidence (1 – C) over Subsequent Estimates (Es) througha lag regression function (g), or recency weightings (Agnew and Brown 1986, p. 150).

This model identifies three main sources of variance for any estimate or decision: (1)the initial estimate (Ei); (2) the decision maker’s confidence (C) in the initial estimate; and(3) a weighting and compilation (g) of other estimates (Es). We have adapted this model of


individual decision making to groups, by defining other estimates (Es) to be the informationand judgements made known to individuals by other members of a decision-making group.

6. A test of three methods of decision making

A wide range of studies (e.g. Miner 1984) have indicated that group error score in typicalranking tasks (e.g., Lost at Sea) is usually lower (better) than the error score obtained byusing a nominal group and simply averaging the individual estimates. In addition, in anumber of cases the group error score is lower than the best individual score. It appearsthen that a well-managed group process which includes exchange of information amongindividuals, exploration of assumptions, and testing of member confidences improvesdecision quality. A portion of this improvement could be gained by the explication and useof confidences of individuals in their various estimates. A comparison of group consensusversus averaging versus the CONFIDE model of decision making was performed usingthe following equations. We begin with a definition of terms.

Eij = Estimate of individual i concerning issue j.

Cij = Confidence in Eij.

Dj =∑

iCij.

n = Number of individuals in the group.

Rg = (Rgj) = Vector of final estimates obtained by group process.

Ra = (Raj) = Vector of final estimates obtained by simple averaging of estimates.

1Raj = — ∑

iEij.

n

Rc = (Rcj) = Vector of final estimates obtained by a normalized confidence weightingof individual estimates. Cij

Rcj =∑

iEij ——

Dj

Thus, Rg represents group consensus, Ra represents a simple averaging nominal groupapproach, and Rc represents the use of a normalized confidence weighting algorithm(CONFIDE) for producing enhanced nominal group results.

It is hypothesized that CONFIDE will capture some of the benefits of the small groupprocess, even though a nominal group approach is used with no face-to-face group meeting.Utilizing the Hollingshead and McGrath (1995) theoretical framework, this effect ishypothesized to be due to the manipulation of the operating conditions (i.e. degree ofanonymity) and process variables (i.e. amount and distribution of participation amonggroup members) by the differences in structuring of the form and sequence of memberinputs to the group decision imposed by the three methods of decision making.


More specifically, three hypotheses will be tested:

Hypothesis 1. Individual error scores will be inversely related to expressed confidence(cf. Shrauger and Osberg 1981). That is, we expect that those with high confidence willhave lower error scores than will those with medium or low confidence.

Hypothesis 2. The error scores obtained from face-to-face group process technique willbe lower than the error scores obtained from the CONFIDE technique.

Hypothesis 3. The error scores obtained from the CONFIDE technique will be lowerthan the error scores obtained from aggregation of opinions by simple averagingtechnique.

7. Methods

7.1. Subjects

A total of 177 students in the University of Pittsburgh Masters of Business Administrationexecutive and evening programs participated in this study as part of a class exercise in agroup decision-making class. All subjects were employed full time in the business sectoror had previously been employed therein. Ninety subjects participated in the first part ofthe study and 87 in the second part of the study.

7.2. Materials

Subjects were administered the Lost-at-Sea simulation (Jones and Pfeffer 1975). In thisexercise, subjects are told to imagine that they are adrift on a sinking yacht in the SouthPacific. Respondents are then asked to rank order 15 available items (a sextant, shavingmirror, fishing kit, etc.) in terms of their importance for survival of the crew. The Lost-at-Sea simulation was chosen because of its extensive use in group decision-makingstudies.

7.3. Procedure

Part I. Ninety subjects were presented with the Lost-at-Sea simulation and asked to rankorder the 15 items according to their importance. In addition, subjects were asked to providean estimate of the confidence with which they held their ranks for each item.

An initial pilot test asked subjects to rate their confidence on a continuous scale (i.e.0–100%) for each item. Subjects found it very difficult to rate their confidence usingthis scale. The initial data was difficult to interpret and deemed unusable due to theseinterpretation problems. Based on this pilot test the rating scheme was changed to a


categorical scale for the actual study. Subjects circled high, medium, or low confidencefor their rank of each item. High confidence was defined as “You are very sure yourknowledge and opinions concerning this item are correct and valuable.” Mediumconfidence was defined as “You are somewhat sure of your knowledge and the value ofyour opinions concerning this item.” Low confidence was defined as “You are not at allsure of your knowledge and the value of your opinions concerning this item.” Subjectswere better able to intuitively grasp this categorical scale format and did not have aproblem estimating their confidence levels.

After completing the Lost-at-Sea exercise independently, subjects were randomlyassigned to 18 five-person groups. They were then instructed to discuss the Lost-at-Seasituation in the group and to arrive at a consensual ranking of the items. Care was takento instruct the subjects to reach consensus regarding the items and to avoid merely votingon the ranking.

Following the group exercise, each subject computed their own individual error scoreand the group’s error score. Error scores were calculated by comparing rankings withthe ranking produced by officers of the United States Merchant Marines (Jones andPfeffer 1975).

Part II. A second independent sample of 87 Masters of Business Administration studentsat the University of Pittsburgh completed the same Lost-at-Sea exercise as those in Part I,including the categorical confidence ratings for each item. However, prior to discussingthe Lost-at-Sea situation in groups or seeing the correct rankings, these subjects indicatedthe weight each confidence category (high, medium, low) should be given in the aggregationof scores of group members. This was accomplished by instructing subjects to distribute100 points among the three confidence categories. These weights were then averagedacross subjects to arrive at independent weights for use in the aggregation of Part I subjectrankings in the CONFIDE algorithm to produce a nominal group decision.

This was done to insure that the algorithm gives more weight to the individuals in thegroup who express a higher confidence relative to other group members. The point ofCONFIDE is to capture the relative differences in confidence between group members.The net result of using this algorithm is that if all group members express correspondinglylow or high confidence then no individual’s selection will be given extra weight in thecomposite group selection.

8. Results

8.1. Confidence weightings

Table 1 contains the averages of the confidence weightings provided by Part II subjects.By standardizing the means of each category, the following weights were developed: HighConfidence = 3.1, Medium Confidence = 1.6, Low Confidence = 1.0. These relative weightswere used in the CONFIDE algorithm to aggregate individual rankings from the first setof data to produce nominal group decisions.


8.2. Hypothesis one

The 90 participants in Part I of the experiment produced 89 usable responses, and theresults of their individual rankings for each of the 15 items generated a pool of 1335opinions. Four hundred ninety-eight (37.3%) of the opinions were expressed with highconfidence, 565 (42.3%) with medium confidence, and 272 (20.4%) with low confidence.

Hypothesis one proposed that individual error scores would be inversely related toexpressed confidence. Partial support for this hypothesis was obtained. A Scheffe analysisindicated that the mean error of low confidence opinions (X

– = 4.47) was significantly

greater than that of high confidence (X– = 3.57) and medium confidence (X

– = 3.70) opinions,

F (2, 1332) = 7.16, p < 0.001. No significant difference in error was found, however,between high confidence and medium confidence opinions (see Table 2).

8.3. Hypotheses two and three

Table 3 represents a compilation of results for the 18 five-person groups participating inPart I of the experiment.

Error scores for actual interacting groups were obtained by comparing the consensualranking of the group with the correct ranking.

The simple average error was calculated by taking an unweighted simple average ofmembers’ individual rankings for each item. These averages were then used to rank the 15items, producing a nominal group score. In the case of ties, ranks were shared. For instance,if two items were tied for 4th and 5th place, each was assigned the rank of 4 and 1/2.Average ranking, not average individual error scores, were used to eliminate the bias inaverage individual error scores as analyzed earlier by Slevin (1978).

The error calculated with the CONFIDE algorithm employed the confidence assignmentsprovided by each individual for the item rankings. Weights of 3.1, 1.6, and 1.0 were usedfor high, medium, and low confidence designations, respectively. A score was assigned to

Table 2. Individual rankings by confidence categories

Confidence category N X–

error Error

High 498 3.57 3.22Medium 565 3.70 3.11Low 272 4.47* 3.75

*Number is statistically significantly different from the other two conditions atthe 0.001 level.

Table 1. Weighting of confidence categories

Confidence category X–

Standardized weight

High 54.4 12.6 3.1Medium 28.3 9.4 1.6Low 17.3 10.9 1.0


each of the 15 items by multiplying the rank assigned to the item by the normalizedconfidence factor of the individual subject. The normalized confidence factor was obtainedby dividing the individual’s confidence factor by the total of all confidence factors of theindividual’s group for that particular item. Thus, individuals expressing higher confidencecontributed more heavily to the score of an item than did individuals expressing lowerconfidence. The item score was then calculated by combining all ranks multiplied bynormalized confidence weighting. (See Table 4 for an example of the weighting procedure).The fifteen item scores were then ranked to produce a nominal group decision based onnormalized confidence factors.

Table 3. Error scores for group decision methods

Error scores

Group Actual group Simple average CONFIDE

1 48 65 622 42 34 423 52 43 404 56 67 665 44 54 506 34 52 407 54 48 468 62 57 569 20 36 32

10 46 41 3811 46 42 3812 60 48 5213 56 51 4814 52 60 6215 42 37 3416 48 70 6617 60 54 5418 54 62 56

Table 4. Example of weighting procedure using the CONFIDE algorithm

Group 1Item 14 – Two boxes of chocolate bars

Respondent Rank Confidence Weight Normalized rank

1 10 Medium 1.6 (1.6/10.4)*10 = 1.542 12 Low 1.0 (1.0/10.4)*12 = 1.153 1 High 3.1 (3.1/10.4)* 1 = 0.304 7 Medium 1.6 (1.6/10.4)* 7 = 1.085 3 High 3.1 (3.1/10.4)* 3 = 0.89

Total confidence = 10.4

Item Score = 1.54 + 1.15 + 0.30 + 1.08 + 0.89 = 4.96


A Wilcoxon Matched-Pairs Signed-Ranks test was performed in order to determine ifsignificant differences existed among the two nominal group methods (simple averageand CONFIDE algorithm) and the group consensus method. Hypothesis two proposedthat error scores obtained through group consensus would be lower (better) than thoseobtained through the CONFIDE method. This hypothesis was not supported. In 9 of our18 cases, the CONFIDE method produced lower error scores than did the group consensusmethod. There was one tie. Consequently, no significant differences in error scores betweenCONFIDE and group consensus were obtained (Z = –0.189, p < 0.850) and hypothesistwo was rejected.

Hypothesis three proposed that error scores obtained through CONFIDE would be lowerthan those obtained by simple averaging. Hypothesis three was confirmed. In 14 out of 18cases, the CONFIDE method produced lower error scores than did simple averaging. Therewas one tie. Therefore, the CONFIDE method produced significantly lower error scoresthan did the simple averaging method (Z = –2.130, p < 0.05). Thus it appears that while theCONFIDE results were essentially indistinguishable from the small group interaction, theywere significantly better than the nominal group simple average technique.

9. Discussion and limitations

The results of this study indicate that use of confidence as a weighting criterion may bebeneficial in providing some of the benefits of small group information integration withoutthe necessity of face-to-face discussion. The CONFIDE method produced error scoressignificantly better than those produced using a simple average of individuals’ opinions,and virtually indistinguishable from scores produced in face-to-face group interaction.The implication of this result is that with regards to decision quality, confidence estimates,which took approximately 2–5 minutes to complete, may adequately replace a 45-minutegroup interaction. This is a tenfold increase in efficiency that is gained by using theCONFIDE algorithm.

These findings are subject to important limitations. We do not propose that CONFIDEis an appropriate technique to be used for all types of group tasks. Hollingshead and McGrath(1995) categorize group tasks as being of four general types:

(1) Generate – including creativity and planning tasks which typically involve a divergentprocess of generating ideas, plans or solutions.

(2) Negotiate – including cognitive conflict and mixed motive tasks which typically involveresolving conflicts of viewpoint or interest.

(3) Choose – including intellective and decision-making tasks which typically involve thegroup integrating information and converging on a decision from a list of potential solutions.

(4) Execute – including contests/competitive and performance/psychomotor tasks whichtypically involve resolving conflicts of power or executing performance tasks.

The use of CONFIDE seems most appropriate for the “choose” type of task. The choosetype of task is analogous to situations in which a judgemental strategy must be utilized


(Thompson 1967). This is a situation in which outcome preferences are clear, but there isuncertainty with regards to cause and effect relationships. The use of CONFIDE in choosingtasks may help to avoid problems often found in group decision making such as groupthink(Janis 1973) or minority influence issues like the inhibition of lower status members toexpress their opinions (Huber 1982) or be unduly influenced by higher status members(Applegate et al. 1986) by keeping group members separated and their confidence estimatesanonymous.

CONFIDE by its very nature seems ill-suited for group tasks such as generation ornegotiation. These forms of group tasks typically require the face-to-face interaction ofgroup members to allow the emergence of creativity through member interplay and thepolitical dynamics of negotiation. Furthermore, this study did not examine the impact ofCONFIDE on the emergence of member commitment for execution of group decisions. Itcould be that face-to-face interaction is necessary in order for feelings of ownership andbuy-in to develop. These remain interesting areas for future research.

Additionally this study did not look at the impact of CONFIDE on other outcomevariables such as user satisfaction with the process and effects on member attitudes suchas attraction to the group or feelings of impersonality or alienation. These are also interestingareas for future research.

Despite the usefulness of confidence estimates in this study, a number of other limitationsshould also be noted. The task was performed in a classroom setting in which subjects hadno personal involvement with the decision and did not have to suffer the consequences ofthe decision. Groups are more likely to reach unanimous agreement on less importantissues than on more important issues (Kerr 1992). In other words, the stakes were not ashigh as in a real-life situation. It is not certain that the results obtained here would generalizeto an organizational setting. The findings to date, therefore, should not yet be extrapolatedto an organizational setting where the participants have a stake in the decision. Likewisethe sample groups were selected from typical MBA classes and thus were not representativeof the population at large in terms of gender, age, income level, intelligence, etc. Thereforethe results from this study cannot necessarily be extrapolated to the general population.These also remain as areas for future research.

Finally, although the differences between the methods of decision making werestatistically significant, they were small. However, the use of confidence estimates may befurther enhanced by training individuals in confidence estimation. In addition, rewardingindividuals for accurate assessments of confidence might improve assessments. In fact,Einhorn and Hogarth (1978) have developed a model for learning confidence in judgmentsthat accounts for experience and feedback. Training individuals to assess the quality oftheir opinions could only enhance the use of confidence estimates as a means of weightinggroup information.

Another related and potentially fruitful avenue for future research is to examine theefficacy of combining the use of confidence estimates with the Delphi technique (Dalkey1969) which currently includes only point estimates from individuals.

The comparability of CONFIDE and group discussion may be due to (a) the gainsachieved by eliciting confidence estimates and/or (b) avoidance of the process losses ofgroup interaction. As pointed out earlier, self-assessments of confidence may be a useful


device for rating the quality of an individual’s contribution to the group. Our subjects hadlittle difficulty rating the value of their opinions in this task. On the other hand, groupproductivity may have suffered in the discussion method of decision making. As Steiner(1972) stated, actual productivity is equal to maximum productivity minus losses due tofaulty process. The groups in the present study may not have been operating at theirmaximum potential.

The potential of the CONFIDE method should not be overlooked. As collective decisionmaking in large, geographically dispersed, modern organizations becomes more a matterof computer networking and less a matter of face-to-face interaction, finding efficient andeffective ways of combining multiple information inputs to reach group decisions becomesincreasingly relevant. The introduction of technology into group decision making can helporganizations cross physical, social and psychological boundaries. The use of CONFIDEmay provide a useful means of combining group information efficiently and effectively.

References

Agnew, N.M., and J.L. Brown. (1986). “Bounded Rationality: Fallible Decisions in Unbounded Decision Space,”Behavioral Science 31, 148–161.

Applegate, L.M., B.R. Konsynski, and J.F. Nunamaker. (1986). “A Group Decision Support System for IdeaGeneration and Issue Analysis in Organization Planning,” Proceedings of the Nineteenth Annual HawaiiInternational Conference of Systems Sciences.

Ashton, A.H., and R.H. Ashton. (1985). “Aggregating Subjective Forecasts: Some Empirical Results,”Management Science 31, 1499–1508.

Daft, R.L., and RH. Lengel. (1986). “Organizational Information Requirements, Media Richness and StructuralDesign,” Management Science 32, 554–571.

Dalkey, N.C. (1969). “The Delphi Method: An Experimental Study of Group Opinion,” working paper 5888-PR, Rand Corporation.

Dalkey, N.C., B. Brown, and S. Cochran. (1979). “The Delphi Method IV: Effect of Percentile Feedback andFeed-in of Relevant Facts,” working paper 6118-PR, Rand Corporation.

Davis, J.H. (1992). “Some Compelling Intuitions about Group Consensus Decisions, Theoretical and EmpiricalResearch, and Interpersonal Aggregation Phenomena: Selected Examples, 1950–1990,” OrganizationalBehavior and Human Decision Processes 52(1), 3–38.

Delbecq, A.L., A.H. Van de Ven, and D.H. Gustafson. (1975). Group Techniques for Program Planning: AGuide to Nominal Group and Delphi Processes. Glenview, IL: Scott, Foresman & Company.

Dyer, W.G. (1977). Team Building: Issues and Alternatives. Reading, MA: Addison-Wesley.Eckerson, W. (1992). “Users Enthused About Electronic Meetings,” Network World 9(24), 43.Einhorn, H.J., and R.M. Hogarth. (1978). “Confidence in Judgement: Persistence of the Illusion of Validity,”

Psychological Review 85, 395–416.Einhorn, H.J., and R.M. Hogarth. (1981). “Behavioral Decision Theory: Process of Judgments and Choice,”

Annual Review of Psychology 32, 53–88.Einhorn, H.J., R.M. Hogarth, and E. Klempner. (1977). “Quality of Group Judgement,” Psychological Bulletin

84, 158–172.Er, M.C., and A.C. Ng. (1995). “The Anonymity and Proximity Factors in Group Decision Support Systems,”

Decision Support Systems 14(1), 75–83.Fischhoff, B. (1975). “Hindsight and Foresight: the Effect of Outcome Knowledge on Judgement Under

Uncertainty,” Journal of Experimental Psychology: Human Perception and Performance 1, 288–299.Fischhoff, B. (1982). “Debiasing,” in D. Kahnenman, P. Slovic, and A. Tversky (eds.), Judgement Under

Uncertainty: Heuristics and Biases. New York, NY: Cambridge University Press, 422–444.


Fulk, J., and B. Boyd. (1991). “Emerging Theories of Communication in Organizations,” Journal of Management17(2), 407–446.

Gessner, S., M. McNeilly, and W. Leskee. (1994). “Using Electronic Meeting Systems for Collaborative Planningat IBM Rochester,” Planning Review 22(1), 34–39.

Goldberg, L.R. (1970). “Man Versus Model of Man: A Rationale, Plus Some Evidence for a Method of Improvingon Clinical Inferences,” Psychological Bulletin 73, 422–432.

Hackman, J.R. (ed.). (1990). Groups That Work (And Those That Don’t). San Francisco, CA: Josse-Bass.Hogarth, R.M. (1977). “Methods for Aggregating Opinions,” in H. Jungerman, and G. de Veouw (eds.), Decision

Making and Change in Human Affairs. Dordrecht, Holland: D. Reidel, 231–255.Hogarth, R.M. (1978). “A Note on Aggregating Opinions,” Organizational Behavior and Human Performance

36, 209–228.Hollingshead, A.B., and J.E. McGrath. (1995). “Computer-Assisted Groups: A Critical Review of the Empirical

Research,” in Richard A. Guzzo, Eduardo Salas, and Associates (eds.), Team Effectiveness and DecisionMaking in Organizations. San Francisco, CA: Josse-Bass, 46–78.

Huber, G.P. (1982). “Group Decision Support Systems as Aids in the Use of Structured Group ManagementTechniques,” Management Science 28, 96–108.

Huber, G.P. (1984). “The Nature and Design of Post-Industrial Organizations,” Management Science 30, 928–951.

Jacob, V.S., and H. Pirkul. (1992). “A Framework for Supporting Distributed Group Decision-Making,” DecisionSupport Systems 8(1), 17–28.

Janis, I. (1973). Victims of Groupthink: A Psychological Study of Foreign Decisions and Fiascos. Boston, MA:Houghton Mifflin.

Jones, J.E., and J.W. Pfeffer (eds.). (1975). The 1975 Annual Handbook for Group Facilitators. La Jolla, CA:University Associates.

Keasey, K., and R. Watson. (1989). “Consensus and Accuracy in Accounting Studies of Decision-Making: ANote on a New Measure of Success,” Accounting, Organizations and Society 14(4), 337–345.

Kerr, N.L. (1992). “Group Decision Making at a Multialternative Task,” Organizational Behavior and HumanDecision Processes 52(1), 64–95.

Kiesler, S., and L. Sproull. (1992). “Group Decision Making and Communication Technology,” OrganizationalBehavior and Human Decision Processes 52(1), 96–123.

Kilmann, R.H. (1989). Managing Beyond the Quick Fix. San Francisco, CA: Josse-Bass.Lawler, E.L. III. (1986). High Involvement Management. San Francisco, CA: Josse-Bass.Lawler, E.L. III. (1992). The Ultimate Advantage: Creating the High Involvement Organization. San Francisco,

CA: Josse-Bass.Libby, R., and R.K. Blashfield. (1986). “Performance of a Composite as a Function of the Number of Judges,”

Organizational Behavior and Human Performance 21, 121–129.Lorge, I., and H. Solomon. (1955). “Two Models of Group Behavior in the Solution of Eureka-type Problems,”

Psychometrika 20, 139–148.Mabe, P.A. and S.G. West. (1982). “Validity of Self-Evaluation of Ability: A Review and Meta-Analysis,”

Journal of Applied Psychology 67, 280–296.Mannix, E.A. (1993). “Organizations as Resource Dilemmas,” Organizational Behavior and Human Decision

Processes 55(1), 1–22.Migliarese, P., and E. Paolucci. (1993). “A System for Group Production Planning in Manufacturing,” Interfaces

23(3), 29–40.Miller, G.A. (1956). “Magical Number Seven Plus of Minus Two: Some Limits on Our Capacity for Processing

Information,” Psychological Review 63, 81–97.Miner, F.C. Jr. (1984). “Group Versus Individual Decision Making: An Investigation of Performance Measures,

Decision Strategies, and Process Losses/Gains,” Organizational Behavior and Human Performance 33, 112–124.

Mockler, R.J., and Dologite, D.G. (1991). “Using Computer Software to Improve Group Decision-Making,”Long Range Planning 24(4), 44–57.

Olson, M.H. (1985). “The Potential of Remote Work for Professionals,” in Office Workstations in the Home.


Washington, DC: National Academy Press, 125–132.Osberg, T.M., and J.S. Shrauger. (1986). “Self-prediction: Exploring the Parameters of Accuracy,” Journal of

Personality and Psychology 51, 1044–1057.Phelps, N. (1985). “Mountain Bell: Program for Managers,” in Office Workstations in the Home. Washington,

DC: National Academy Press, 33–37.Ramsower, R.M. (1985). Telecommutinq: the Organizational and Behavioral Effects of Working at Home. Ann

Arbor, MI: University of Michigan Research Press.Reisman, S., T.W. Johnson, and B.T. Mayes. (1992). “Group Decision Program,” Decision Support Systems

8(2), 169–180.Shrauger, J.S., and T.M. Osberg. (1981). “The Relative Accuracy of Self-predictions and Judgements by Others

in Psychological Assessment,” Psychological Bulletin 90, 322–351.Shrauger, J.S., and T.M. Osberg. (1982). “Self-Awareness: The Ability to Predict One’s Future Behavior,” in G.

Underwood (ed.), Aspects of Consciousness, Vol. 3. London: Academic Press, 267–313.Silver, H. (1989). “The Demand for Homework: Evidence from the U.S. Census,” in Eileen Boris, and Cynthia

R. Daniels (Eds.). Homework: Historical and Contemporary Perspectives on Paid Labor at Home. Urbana,IL: University of Illinois Press, 103–129.

Simon, H.A. (1979). “Rational Decision Making in Business Organizations,” American Economics Review 69,493–513.

Slevin, D.P. (1978). “Observations on the Invalid Scoring Algorithm of ‘NASA’ and Similar Consensus Tasks,”Group and Organizational Studies 3, 497–507.

Sniezek, J.A. (1992). “Groups Under Uncertainty: An Examination of Confidence in Group Decision Making,”Organizational Behavior and Human Decision Processes 52(1), 124–155.

Stasser, G. (1992). “Information Salience and the Discovery of Hidden Profiles by Decision-Making Groups: AThought Experiment ,” Organizational Behavior and Human Decision Processes 52(1), 156–181.

Stasser, G., and W. Titus. (1985). “Pooling of Unshared Information in Group Decision Making: BiasedInformation Sampling During Group Discussion,” Journal of Personality and Social Psychology 48, 67–78.

Steiner, I.D. (1966). “Models of Inferring Relationships Between Group Size and Potential Group Productivity,”Behavioral Science 11, 273–283.

Steiner, I.D. (1972). Group Process and Productivity. New York, NY: Academic Press.Stewart, A. (1989). Team Entrepreneurship. Newbury Park, CA: Sage Publications.Tversky, A., and D. Kahneman. (1973). “Availability: A Heuristic for Judging Frequency and Probability,”

Cognitive Psychology 4, 207–232.Thompson, J.D. (1967). Organizations in Action. New York, NY: McGraw-Hill.Vroom, V.H., and P. Yetton. (1973). Leadership and Decision Making. Pittsburgh, PA: University of Pittsburgh

Press.Walker, R.G. (1994). “Virtually Interactive Brainstorming,” Industrial Engineering 26(9), 20–21.Yetton, P., and P. Bottger. (1982). “Individual Versus Group Problem Solving: An Empirical Test of Best-member

Strategy,” Organizational Behavior and Human Performance 29, 307–321.Yukl, G. (1981). “Leadership in Organizations,” Englewood Cliffs, NJ: Prentice-Hall.

Documents

CONFIDE: A Collective Decision-Making Procedure Using Confidence Estimates of Individual Judgements