III.C. 3. A Delphi on the Future of the Steel and ...turoff/pubs/delphibook/ch3c3.pdfanalyzing the Delphi, (2) the contributions by the respondents, and (3) the flow of information

204

III.C. 3. A Delphi on the Future of the Steel and Ferroalloy Industries*

NANCY H. GOLDSTEIN Introduction In the spring and fall of 1970 a Delphi on the U. S. ferroalloy industry was conducted by the National Materials Advisory Board (NMAB) of the National Academies of Science and Engineering. The Board, concerned about a possible shortage of certain critical and strategic materials within the next decade or two, turned to the Delphi as a means of assessing the implications of technological change on usage trends of ferroalloys. The trends brought out by the Delphi could serve as a long-range planning guide for policy issues affecting the use of ferroalloys in steel making and certain other alloy production. This article will discuss the format of the Delphi, the selection of respondents, the manpower required to carry out the exercise, and the round-by-round method of conduct ing the Delphi. The article will then present a comparison of the Delphi exercise with a conventional panel study 1 which was conducted simultaneously with the Delphi exercise and conclude with some advice to prospective Delphi designers.

Form of the Delphi. The Steel and Ferroalloy Delphi included three rounds. The questions and exercises presented in each round were divided into three sections: Section I, Steel; Section II, Alloys; and Section III, Key Developments. Sections I and II generally presented trend lines for extension by the respondents and the assumptions underlying these extensions. Section III indicated future developments thought by the respondents to have a potential role in the steel and/or ferroalloy industry in the next two decades. More detailed descriptions of these three sections will be given in the round-by-round discussions which follow.

Selection of Respondents. The original Delphi respondents were not chosen randomly but were carefully selected from all sectors of the industry, government, the universities, institutes, and trade publications. Members of the NMAB Panel on ferroalloys submitted suggestions for respondents and the panel as a whole discussed each suggestion. One hundred names were chosen for the initial Delphi round. These one hundred potential respondents received a letter inviting them to participate, a card to return to the panel indicating their preference for participating or not, and a copy of the first-round questionnaire of the Delphi. Of the one hundred potential respondents,

* The full report on this exercise is available from the National Technical Information Service, Springfield, Va., as "A Delphi Exploration of the U. S. Ferroalloy and Steel Industries," by Nancy H. Goldstein and Murray Turoff, NMAB-277, July 1971. 1Available as "Trends in the Use of Ferroalloys by the Steel Industry of the United States," NMAB-276, July 1971, by the Panel on Ferroalloys of the NMAB.

General Applications: Steel and Ferroalloy Industries 205

fortytwo returned the card stating that they wished to participate and thirty-three actually responded to the first round. Response to the exercise was voluntary and no compensation was provided. This resulted in a much higher percentage response from industry-associated respondents who, as members of planning staffs, could consider the effort part of their job function. A much lower percentage response occurred from university people who probably considered this request as an uncompensated consulting effort.

The summary below shows the makeup of the final respondent group and the number of respondents replying to rounds one and two, one and three, and one, two, and three:2

OCCUPATIONS OF DELPHI RESPONDENTS

Occupation Number of Respondents Ferroalloy Producer Nonferrous Alloy Producer Specialty Metals Powder Metals Specialty Steel Producer Steel Producer Polymers Institutes University Government Technical Journal Consultant

6 1 2 2 4 4 3 4 4 1 2 1

—— 34

PATTERN OF RESPONSES TO THE ROUNDS

Rounds

Number of Respondents

1 and 2 1 and 3 2 and 3

1, 2, and 3

3 2 3 28

Manpower. The manpower required for this De lphi included two full-time professionals -a senior professional and his assistant-and intermittent temporary clerical and secretarial help. The exercise was conducted in cycles: one to two months waiting for the results of the previous round and making preparations for the handling of these

2 No respondent replied to fewer than two rounds.

206 Nancy H. Goldstein

results, and one to two months actually handling the results and preparing them in a form suitable for the next round. The requirements for secretarial help were also cyclical: little or no help was required during the waiting period but two full-time secretaries were required during the week-long rush period when the results had been tabulated and were being typed up for inclusion in the next round.

The Delphi ran for three rounds. Each round will be discussed below in terms of the design of that round and the handling of the results. It is, however, somewhat artificial to separate the design of a new round from the handling of the results of the previous round, since the form of the new round determines the method of handling and presenting the old round.

Tables 1 and 2 summarize (1) the effort involved in designing, monitoring, and analyzing the Delphi, (2) the contributions by the respondents, and (3) the flow of information in the Delphi rounds. While the clerical effort is broken out separately, a significant portion of this was actually done by the professionals involved. The availability of clerical-type support was a random process that did not always conform to requirements. One key element of both clerical and secretarial support is the benefit of having the same individuals to aid on every round, since there is a learning curve on the explicit procedures to be followed.

Round One Design. Round one was divided into three sections. The first section entitled "Steel," presented graphs covering various aspects of the steel industry-total steel shipments, ratio of shipments to production, etc. A trend line, usually running from 1960-69, was shown on the -graph and the respondents were asked to extend the line through to 1985. Three questions were associated with each graph:

(1) How reliable did the respondent consider his graph extension to be? (2) What key developments (i.e., his assumptions) did the respondent assume in

making his extension? (3) What other developments (i.e., his uncertainties) might result in major

revisions in the extension? A flow chart of the steelmaking process was also presented at the end of Section

1. Figures were given for 1969, and the respondent was asked to supply the corresponding figures for 1980.

Section 11, entitled "Alloys," presented a number of graphs in the same manner as Section 1. The graphs of Section 11 were concerned with aspects of the ferroalloy industry - U. S. consumption of chromium, tungsten, etc. and exports and imports of these materials. The exercise was separated into a Steel Section and a Ferroalloy Section because as majority of the expertise in the respondent group broke down into specialists in these two areas. While most respondents had something to contribute to both sect ions, it was clear, when the



results came in, that a given individual usually focused most of his effort on one of the two sections.

Section III, entitled "Suggested Additional Variables and Key Developments," offered blank graph sheets and blank key development tables for respondents wishing to add to the items presented in Sections I and 11. Figure 1 indicates the format.

The selection of categories to be presented in Sections I and II was made by the questionnaire designers, with suggestions and assistance from the Panel on Ferroalloys.

Handling the Results. Round one was mailed to forty-two respondents on June 16, 1970; thirty-three responded to this round.

There were two principal elements to handling the results of round one. First, a determination was made, for each graph, as to the location of upper and lower limits of the extensions which would include 50 percent of the responses to that graph.



The second element involved the gathering and synthesizing of comments presented under (Questions 2 and 3 of the graph sheets. The comments of all the respondents were collected, and each comment was then studied and determined to be either a forecasting assumption, an economic and international consideration, a key development, or a comment to be associated with that particular graph page. The comments were grouped accordingly, and the final product was retyped for inclusion on the second round. It was quite apparent that in many instances one respondent's assumption was another's uncertainty. The frequency with which a topic was brought up influenced the judgment on its choice as a key development for round two.

The process of collecting and editing the large number of comments obtained on the first round represented the largest single task in the exercise, in terms of both clerical time and professional judgment. Assumptions from all respondents were initially xeroxed, cut out, and taped on large sheets. Each sheet repre sented different topics or curves. On these large sheets duplications were crossed out and editing of assumptions to produce shorter wordings took place. This conglomeration was then retyped once and put through a final polishing, editing, and reordering before the final typing for the second-round questionnaire. The process of putting each set of assumptions through a two -stage editing process allowed each professional to check the other's work. It is noticeable to a certain extent that the availability of the xerox machine is a key feature in making large-scale Delphis possible via paper-and-pencil approaches. This is particularly true where one is handling a large volume of textual comments on the part of the respondents.

Section III in round one, "Suggested Additional Variables and Key Devel-opments," produced few responses from the participants. Some of the responsesin this section became key developments for Section III of round two, some became assumptions for round two, and the remainder were dropped from the exercise. Three additional curves suggested by the respondents were prepared for inclusion in round two.

The flow chart appended to Section I in the first round also received scant response (i.e., about ten respondents). The responses that were supplied were averaged for each entry in the flow chart and standard deviations were provided through the use of a simple computer program.

Round Two Design. The questionnaire sent to the respondents in round two was patterned after the results described in the handling of round one.

The new Section I, "Steel," contained thirty -six "Fo recasting Assumptions," thirty-five "Economic and International Considerations," and all the graphs contained in round one (Section 1) with their associated reasons. The respondent was asked to associate a validity score with each forecasting assumption and


economic and international consideration presented. The scores were based on the validity codes which are shown in Fig. 2.

For all graphs the original trend line was presented and the 50 percent confidence limits were indicated. The respondent was asked to reestimate his previous extension, after viewing the 50 pet-cent limits, and to identify his estimate as reliable, as good as anyone's, or risky. Each graph also contained associated reasons given by the respondents for increasing or decreasing the graph extension. In Section I, a total of 116 reasons associated with the graphs were presented for evaluation. The respondent was asked to assign a rank of I to 6 to each reason given according to the validity scale. He was also invited to show additional reasons if he wished. A sample of a typical round-two question is presented in Fig. 3. This is exactly the same basic form as used in round one.

At the end of Section I, the flow charts were again presented. Two charts. for 1969 and 1980, showed the means and standard deviations for cacti chart entry and also showed some additional boxes and paths not included in round one. "These modifications were suggested by the respondents and incorporated by the senior professional. The respondent was asked to circle the estimate presented if he agreed with it, or to cross it out and provide a new figure on the blank charts provided if he disagreed. The absence of either action was considered to be a No judgment vote. It was a surprise to the designers that almost all the respondents to the flow chart chose to modify it, since this was not an action suggested in the instructions.

Section II, "Alloys," was similar to Section I in design. The category of forecasting assumptions included sixty-nine assumptions about individual alloys studied, as well as seventeen general assumptions. The respondents were again asked to assign each statement a validity score of from 1 to 6. There was no category of Economic and International Considerations, but all the graphs from Section II of the first round were included with the 50 percent confidence limits and 128 associated forecasting reasons. The respondents were asked to provide the same information requested in Section I.

Section II was entitled "Key Developments and Added Curves." Under Key Developments, thirty -six items were presented for scoring by the respondents. For each item, the respondent was asked to evaluate the likelihood of occurrence by 1975 on a scale of 1 to 6 and to indicate, on a scale of 1 to 4, the impact on the steel industry if the development were to occur. Figure 1 provides a sample of the form utilized for key developments.

In addition to scoring the developments, the respondents were asked to describe the nature of impacts they had characterized as strong or moderate. Three new curves, supplied by respondents to round one, were also included in this section. The respondents were asked to handle these new curves in the same way they treated the original curves in round one (Sections I and II).


Numeric

Scale

1

CERTAIN (Average of 1 to 1.5) • Low risk of being wrong. • Decision based upon this will not be wrong because of this 'fact. ' • Most inferences drawn from this will be true.

2

RELIABLE (Average of 1.6 to 2.5) • Some risk of being wrong. • Willingness to make a decision based upon this. • Assuming this to be true but recognizing sonic chance of error • Some incorrect inferences can be drawn.

3 NOT DETERMINABLE: (at this time) (Average of 2.6 to 3.5) • The information or knowledge to evaluate the, validity of this assertion

is not available to anyone --expert or decisionmaker.

4

RISKY (Average of 3.6 to 4.5) • Substantial risk of being wrong. • Not willing to Make a decision based upon this alone. • Many incorrect inferences can be drawn. • The converse, if it exists, is possibly RELIABLE.

5

UNRELIABLE (Average of 4.6 to 5) • Great risk of being wrong. • Worthless as a decision basis. • The converse, if it exists, is possibly CERTAIN.

6

NOT PERTINENT (Used to eliminate some assumptions from exercise) • Even if the assertion is CERTAIN or UNRELIABLE it has no

significance for the basic- issue. • It cannot affect the variable under question an observable amount.

blank

NO JUDGMENT • No knowledge to judge this item, but the appropriate in dividual (expert,

decisionmaker) should he able to provide an evaluation I would respect.

Fig. 2. Validity or confidence scale



Handling the Results. Round two was sent to fifty-two potential respondents; thirty-four replies were received. Several respondents, representing the polymer industry, were added during this round. They had not been represented in round one and were introduced for the purpose of addressing specific issues on the substitution of plastics which had been generated in round one.

The results of round two required three separate types of handling: (1) new 50 percent confidence limits were supplied for the graphs; (2) verbal comments associated with the assumptions, the graphs, and the Key Development section were collected and considered; and (3) the numerical results, i.e., validity choices, Key Development scores, and flow chart inputs, were collected and tabulated.

There were several steps necessary in handling the large amount of data that was generated by the results of round two. It was first determined that several statistical calculations on the data would be desirable -specifically, the mean and standard deviation of the validity choices for each statement, a distribution showing the percentage of responses falling under each of the scores front 1 to 6 for each of the statements, and a matrix comparing the distribution, by numbers, of the different occupation categories represented by the respondents with the range of scores from 1 to 6. These occupation categories included: primary steel producers, research institutions, steel producers, ferroalloy pro ducers, research institutions, government, and the universities. A computer program was written to carry out these computations. T his breakdown allowed us to observe if there were any differences in judgment which may have reflected differences in affiliation of the respondents.

Examples of the statistical presentations are shown in Fig. 4. The flow chart included in Section I received very little additional informa-

tion in round two. The scant information received was averaged in to previous information on the flow chart and presented in a summary for round three. Due to the differences of opinion among the respondents on how actually to model the flow of steel-making materials, it was felt by the designers that , this one question could have constituted the total Delphi exercise among a select smaller group of respondents.

Round Three Design. Round three again consisted of three major sections: Section I: A Summary of Round Two, Section I, Steel; Section If: A Summary of Round Two, Section II, Alloys; and Section III: Key Developments and Added Curves.

Sections I and II were provided for summary purposes and required no further input by the respondents. For each statement, the mean and standard deviation as calculated from round two results were shown. Several new statements were added in Sections I and II; a few new developments were added to Section III to be assigned a validity score by the respondents. The new 50 percent confidence limits, taken from the results of round two, were also presented in a final form for each graph.


Section III, the only section to be returned by the respondents, contained all the new

assumptions and all previously evaluated assumptions which exhibited a large standard deviation (i.e., disagreement). It also called for a reevaluation of all key developments.

The first portion of Section III indicated the percentage distribution of the scores from round two on the likelihood and impact of potential key developments. The estimated average score for each development was indicated and a summary of all verbal comments associated with each development was given. The respondent was asked again to give his preference on the likelihood and impact of each potential development.

The second portion of Section III presented the three curves shown in this section of round two and included a number of reasons given by the respondents of round two for their curve extensions. The respondent was asked to reestimate the curves after reading the associated past reasons and to rate the reliability of his estimation. He was also asked to vote on the reasons given for each curve using the validity scale from 1 to 6 described earlier.


The third portion of Section III contained all assumptions from Sections I and II which exhibited a considerable degree of disagreement. This category generally, although not exclusively, included assumptions with a standard deviation of 13 or greater. The respondent was asked to reevaluate his previous validity choice and submit a new score. Several new assumptions were also added and the respondent was requested to provide a validity choice for these new assumptions.

In the final portion of Section III a new chart was introduced showing percentage breakdowns of inputs and outputs for three major steel processes. The figures were supplied for 1969 and a blank sheet was provided for 1980. The respondent was asked to fill in the sheet for 1980 and to change any 1969 figures with which he disagreed. A space was provided for an explanation of any disagreements with 1969 figures. The results of this chart were to be considered a summary response, as the monitors did not plan to feed back the responses for changes.

The monitor also surveyed briefly the attitudes of the respondents toward the Delphi approach by asking the respondents a number of questions, e.g., was the time spent in participating in Delphi well used; what organizations should sponsor an exercise of this type on a regular basis, etc.

Handling the Results. Round three was sent to thirty-eight respondents on December 10, 1970. Thirty -three respondents actually replied to Round 3.

A computer program provided the means, standard deviations, percentage distributions, and industry category matrices for all key developments, assump -tions to be reevaluated, and new assumptions. The percentage distributions were then examined by the senior professional. If 20 percent or more of the vote fell into the "not pertinent" category (a validity score of 6), the items were dropped from the exercise. Eight items were dropped for this reason. The remaining items were regrouped so that every assumption was associated with a curve. The assumptions and reasons for each curve were then reordered according to their mean validity scores.

The final report was then prepared for the National Materials Advisory Board of the National Academies of Science and Engineering.

Comparison of Delphi and Panel Studies A separate panel appointed by the National Materials Advisory Board approached the same problem considered by the Delphi exercise in a more "conventional" vein. The conventional study was carried out simultaneously with the Delphi, but the results were not compared until both exercises had been completed.

In the panel approach, individual members of the Panel on Ferroalloys reviewed portions of the problem with which they were most familiar. The Panel Report, NMAB-276, "Trends in the Use of Ferroalloys by the Steel Industry of the United States" consists of chapters, each of which is logical, comprehensive, and


definitive with respect to its topic. While appropriate caveats exist, its forecasts are precise. The recommendations and conclusions therein represent the unanimous agreement of the panel and no areas of disagreement are spelled out. The result is typical of a competent panel (or committee) activity. Based upon the expertise of the carefully selected participants, the report is a reliable and comprehensive account of known information and of projections based on this information and on current research and development. In contrast, this Delphi was designed to complement the panel report. The planned approach was to provide an opportunity to indicate uncertainties or disagreements about the subject and to evaluate quantitatively the degree of uncertainty which exists within a large group of experts. The Delphi product attempts to present an awareness of the areas which are subject to differences of view and to highlight the topics which appear to concern the respondent group. The Delphi provides a group evaluation of every statement advanced by the respondents who, presumably, express their beliefs. Although the results of this exercise include a number of statements which were rated uncertain, risky, or unreliable by the whole group, this variation does not imply that one dissenter from the group will be incorrect in retrospect. The group view has a higher probability of being correct than the view of any one individual. However, in the past, developments that significantly affected industries were often unforeseen by most of the involved experts. Therefore, the reader is cautioned not to extrapolate blindly from the group judgments exhibited in a Delphi to assumed facts.

In this case, the Delphi exercise is a literal exploration of the minds of experts in the steel and ferroalloy industries regarding their views on individual items. This exploration allowed a broader coverage of the subject area than was possible in the panel report. The presentation of the Delphi results allows the reader to compare easily his judgments with those of the group. No attempt is made to arrive at conclusions or recommendations, or to present a definitive view as was done in the panel activity.

Where the panel and Delphi activities overlap, there is considerable agree-ment in their forecasts. Figure 5 compares the consumption in steel of a number of alloys, as predicted by the panel and the Delphi, and some of the qualitative features of the two methodologies.

Other comparisons could be made between information presented in the panel report and the Delphi predictions. For example, NMAB-276 projected that carbon steel shipments would increase by 22 percent in the next ten years; the increase projected on the Delphi graph for the next decade was 22 to 26 percent over the current figure. Also, the panel report stated that High Strength Low Alloy Steel (HSLA) is -the fastest growing segment of the steel industry; the Delphi results were that HSLA is one of the two fastest growing segments of the industry.


As was mentioned earlier, the panel report did not indicate the areas of disagreement. In the Delphi category, "not determinable" with respect to validity reflected either the inability of the entire group to determine the validity of an assumption or the averaging of opposing judgments on the validity of a given assumption. Of the 135 statements that fell into this classification, seventy-three reflected an actual disagreement among the re-spondents by exhibiting a high standard deviation.


The following assumptions exemplify those that fell into the "not determinable" category of the Delphi as a result of disagreement:

• Cobalt-iron base tool steels will be marketed. • Continuing nickel shortages will establish permanent substitution. • New techniques will allow significantly greater flexibility of substitution among

alloying elements based upon price changes. • Critical shortages of nickel will reoccur. • No important new use for cobalt. • Present and projected investment in ocean studies is too large to exclude

development of economical offshore mining except in short term (i.e., next five years).

• Alloy steels will increase in nickel content. • Tungsten content of carbides will decrease because of cost. • Full alloy shipments will parallel automotive production. • Low-cost method of preparing high-purity iron powder will be developed by

1975. • More and cheaper scrap will result from urban waste recycling. • Shortages of natural gas will be a primary limiting factor in the expansion or

modernization of the steel industry. • Electron-beam refining will grow significantly. An additional fifty-seven assumptions were rejected by the Delphi respondents as

either risky or unreliable. One must reflect that while each of the 667 assumptions were suggested by at least one expert in the Delphi respondent group, approximately two hundred of these, or 30 percent, were considered less than reliable by the group as a whole. Furthermore, there is often considerable value to decisionmakers in observing the nature of rejected assumptions: For this reason the final Delphi report listed all the evaluated assumptions pertaining to any curve in the order of decreasing validity so one may observe the complete span of the Topics covered. Probably the most significant difference between the Delphi and committee approaches is the itemization of what the Delphi group could not agree on or what they rejected. Usually the psychological process in a committee of experts tends to eliminate these categories of information from the final report.

In summary, the two reports did not always cover the same subject matter. However, when they did touch upon the same subject matter, the results were generally compatible.


Advice to Future Delphi Monitors and Designers The monitor's experience with the Steel and Ferroalloy Delphi gives rise to a number of observations and advice to those planning to monitor Delphis in the future.

(1) When presenting statements for a vote, or synthesizing the respondents'

suggestions, be alert for ambivalent wording. Two separate statements may appear as one, leading to confusion as to what should be voted upon. Vague wording or easily misinterpreted wording may also lead to confusion.

(2) When editing respondents' comments for clarity, try to preserve the intent of the originator. When editing from round to round, avoid changing a statement so that it has one meaning in round one and another in round two.

(3) Lay out the expected processing of the data throughout all the rounds of the Delphi before you finalize the design. You may, by circumstance, be forced later to modify the procedure, but the process of planning ahead will usually turn up any gross problems in your initial questionnaire design and its impact on following rounds.

(4) Design the handling of your data so that each response can be processed (or punched for processing) as it comes in. Thus you will not have a frantic rush to analyze all the responses at once when the last tardy return comes in.

(5) Keep track of how different subgroups in your respondent group vote on specific items. This can be very useful in analyzing the results and will occasionally produce situations where you wish to let the respondent group know that polarizations or differences based upon background exist.

(6) If you are covering a number of fields of expertise, make sure that each field is adequately represented in your group.

(7) It should be mandatory that at least two professionals work on monitoring any one Delphi exercise, particularly when the abstracting of comments is a notable portion of the exercise. With two individuals one can always review what the other has done.

(8) Pretest your questionnaire on any willing guinea pigs you can find outside your respondent or monitor group. If you have a sponsor, it is useful to go over the design of each round with some of his people before finalizing it.

Documents

III.C. 3. A Delphi on the Future of the Steel and ...turoff/pubs/delphibook/ch3c3.pdfanalyzing the Delphi, (2) the contributions by the respondents, and (3) the flow of information