positive contribution to social progress, provided it is applied with the necessary methodological rigour and with a

More than 50 years have passed since the first Delphi experiment took place and more than 40 since

the first article came to light describing its use and procedure. Since then, the Delphi Method has become

Technological Forecasting & Social Change 73 (2006) 467482good knowledge of the social medium in which it is being applied. Finally, there is an explanation of a number of

lessons learned from the theory and aforementioned experiences, which may contribute to the successful outcome

of a Delphi exercise.

D 2005 Elsevier Inc. All rights reserved.

Keywords: Delphi method; Consensus; Policy Delphi; Expert judgement elicitation; Delphi validity

1. IntroductionCurrent validity of the Delphi method in social sciences

Jon Landeta T

Institute of Applied Business Economics of the University of the Basque Country (UPV/EHU), C/ Lehendakari Aguirre,

83 48015 Bilbao, Spain

Received 7 July 2005; received in revised form 7 September 2005; accepted 8 September 2005

Abstract

The Delphi method is a popular technique for forecasting and an aid in decision-making based on the opinions

of experts, which has been in existence for over half a century. This work evaluates its methodology and reviews

its validity in the present day, especially in the area of Social Sciences. Three recent applications in this field are

also explained, professional in nature, which have some characteristics that are not frequent with respect to other

Delphi studies published. The main aim of two of these studies was to provide input for economic or statistical

quantitative models, using the judgement of expert groups, while the third study aimed to analyse a complex social

reality by means of a Policy Delphi in order to obtain reliable information before taking a policy decision. These

applications highlight how this technique may be adapted to different social realities and requirements, making a0040-1625/$ -

doi:10.1016/j.t

T Tel.: +34 6E-mail add013705; fax: +34 6013710.see front matter D 2005 Elsevier Inc. All rights reserved.

echfore.2005.09.002

ress: jon.landeta@ehu.es.

technique for non-military purposes [5], where a proposal was made for it to be used in planning

developing economies.In the sixties, and a result of the declassification of the Delphi method by the American Armed Forces

from its previous category as reserved for military use, its use spread rapidly, both geographically and

thematically, particularly towards technological forecasting and towards the evaluation of complex social

problems.

The Delphi method was conceived as a group technique whose aim was to obtain the most reliable

consensus of opinion of a group of experts by means of a series of intensive questionnaires with

controlled opinion feedback [4].

Later applications of the technique have eliminated the restriction of the obligatory search for

consensus, so that today it might be defined as a social research technique whose aim is to obtain a

reliable group opinion using a group of experts. It is a method of structuring communication between a

group of people who can provide valuable contributions in order to resolve a complex problem [6].

Its main characteristics are as follows:

It is a repetitive process. The experts must be consulted at least twice on the same question, so that

they can reconsider their answer, aided by the information they receive from the rest of the experts.a widely used and recognised instrument to make predictions and help decision-making. In this process

of diffusion it has gone through several different stages since its birth. From the first stage of secrecy that

accompanied its genesis with military aims, it has advanced through successive phases of novelty,

popularity, criticism and re-examination, until reaching the current phase of continual but relatively

infrequent use, in which it has remained since the 80s [1].

Throughout these years, the Delphi Method has broadened its fields of application and has been the

object of fierce criticism and numerous evaluations, in which its weaknesses and strengths have been

highlighted, as well as its potential. It is possibly the best known forecasting technique with its own

name, based on the opinion of experts, and about which most has been written, but is it still valid? Is it

still being used for academic and professional purposes or does it now form part of history? Has it

overcome its weaknesses? What has been learned throughout these years?

In this article we will note some answers to these questions, paying particular attention to the use of

this technique in social sciences and also explaining some recent applications run by the author, which

will serve to compare the contributions of other authors and to present possible fields of development

and application for the technique.

2. The Delphi method

At the end of the 40s, researchers at the RAND Corporation (Santa Monica, California) started to

investigate the scientific use of expert opinion. Studies were published on the superiority of group

opinion over individual [2] and on the justification of expert opinion in inexact sciences and its scientific

use [3]. The Delphi method, with all its defining characteristics, was first used at the headquarters of this

corporation as part of a military defence project, which for security reasons was published twelve years

later [4]. From the RAND Corporation also came the first documented proposal to use the Delphi

J. Landeta / Technological Forecasting & Social Change 73 (2006) 467482468 It maintains the anonymity of the participants or at least of their answers, as these go directly to the

group coordinator. This means a group working process can be developed with experts who do not

coincide in time or space and also aims to avoid the negative influence that could be exercised by

factors in the individual answers in terms of the personality and status of the participating experts.

Controlled feedback. The exchange of information between the experts is not free but is carried out

by means of a study group coordinator, so that all irrelevant information is eliminated.

Group statistical response. All the opinions form part of the final answer. The questions are

formulated so that the answers can be processed quantitatively and statistically.

3. Evaluation of the Delphi method

The Delphi method was quickly accepted and spread rapidly because it provided valuable solutions to

problems inherent in the traditional group opinion based on direct interaction: a reduction in the

influence of some undesirable psychological effects among the participants (inhibition, dominant

personalities, etc.), selective feedback of the relevant information, more extensive consideration thanks

to the repetition, statistical results, flexible methodology and simple execution.

However, the method is not exempt from significant methodological weaknesses, highlighted in

numerous works, such as its basic source of information (who is expert, what biases each expert has,

etc.) [710], the use of consensus as a way to approach the truth [8,9], the limitation of the interaction

involved in written and controlled feedback [10,11], the restriction to the possibility of social

compensation for individual contribution to the group (the reinforcement and motivation normally

provided by the support and social approval of the other expert group members are removed) [12], the

impunity conferred by the anonymity with respect to irresponsible actions on the part of the experts [13],

the ease inherent in the methodology of interested manipulation by the person running the study [10,14],

the difficulty of checking the methods accuracy [4,15] and reliability [9,16], the time required to carry it

out [7,17], the effort required on the part of the participants and the non-consideration of possible inter-

relations between the forecast incidents [18].

In addition to these methodological weaknesses, the Delphi has also received criticism that is not due

to the technique itself but to its deficient application [19,20], such as the not very rigorous selection of

experts, the lack of explanation concerning its evolution and dropout, questions and problems that are

badly formulated, insufficiently analysed results, etc.

In despite of this weaknesses, the comparisons made of this technique with statistical groups and classic

direct interaction groups show mostly positive results for the Delphi method, although the comparisons

made with comparable techniques, such as the Nominal Group, based on different criteria (accuracy,

confidence, creativity, etc.) do not show conclusive results, neither in favour nor against [12,2127].

On the other hand, we have observed that, when this method is used without the required knowledge,

the characteristics inherent in the methodology usually result, on occasion, in a certain bdisappointmentQwith the technique, both among researchers running a Delphi exercise for the first time as well as in

those who take part as experts in this research. Consequently, although it is a relatively well-known and

widely used technique, it is not very attractive for use on more than one occasion and, among those who

know it superficially, it often has an image of a btroublesomeQ technique and relatively bworn-outQ orbout of fashionQ.

In our opinion, this sensation is due principally to the contrast existing between the apparent simplicity

J. Landeta / Technological Forecasting & Social Change 73 (2006) 467482 469of the technique, which motivates neophytes to use it the first time, compared with the work and difficulty

involved in its execution (ensuring the initial collaboration of the experts and particularly their responsible

answers in different rounds). With respect to the participating experts, the negative sensation is usually

caused because, on the one hand, they are asked to do a lot, since they must collaborate in a technique that

they often dont know much about, having to answer practically the same question more than once, with a

relatively long period of time in between and without really seeing how this can be totally justified, being

asked again about something they have already answered. While, on the other hand, the interaction they

have with the rest of the experts is very limited and mediated, so that the compensation for their

participation is often reduced to a series of statistics they are sent (or that they have to look for on the

internet) after a long time, which practically no longer interests them. They are often left with the feeling

that they have been used and that they have received practically nothing in return.

articles published between 1975 and 1994, distinguishing methodological articles from those of

application, and those in which the Delphi method appeared as the main subject of the article and where

J. Landeta / Technological Forecasting & Social Change 73 (2006) 467482470it was a secondary element. The result of their search was 463 articles, 254 of which dealt with Delphi as

a main theme and the remaining 209 as a secondary element. Of these main 254, 75 were methodological

and 179 applied. Analysing in numbers and by periods of five years, with the exception of the first five-

year period (19711975), where the number of articles with the Delphi as the main theme only numbered

21, in the remaining four five-year periods the number of articles was relatively constant (56, 56, 57 and

53 articles, respectively), indicating that, during this period, interest in the technique as a theme or

research instrument did not fall.

Table 1

Frequency of Delphi articles published over the study period

Database Period Delphi articles

Science Direct 199599 367

Science Direct 200004 571

ABI Inform 199599 47

ABI Inform 200004 106

Psycho 199599 86

Psycho 200004 162

Medline 199599 361These problems increase in the area of Social Sciences, particularly in the professional applications on

experts and directors of companies and institutions, always with problems of time and with whom there

is no bemotionalQ or professional link to connect their participation in this kind of study.With this background, and taking into account the time that has passed since the technique was

conceived, it needs to be asked whether this method is still being investigated, if it is assiduously

employed in academic and professional spheres, if it has overcome its weaknesses or if it has definitively

been overtaken by other more developed techniques.

4. Validity of the Delphi method

To evaluate the historical and current validity of the Delphi method we can use the publications

carried out on this area over time as a reference.

There have been various studies reviewing the literature on this area, but of particular note is that

carried out by Gupta and Clarke [28], who carried out an intensive search and classification of theMedline 200004 547

Table 2

Frequency of Delphi dissertations and theses

Years Delphi dissertations and theses Delphi d&t / total d&t

J. Landeta / Technological Forecasting & Social Change 73 (2006) 467482 471We have continued this search for articles dealing with the Delphi method as a main or secondary

theme from 1995 to 2004. We have reviewed the Science Direct database which, according to its

presentation, contains more than 25% of the bibliographical information and scientific, technological and

medical texts in the world, and also three significant databases specialised in the areas with greatest use

of this technique: ABI inform (Economy, Finance and Business), Medline (Medicine and Nursing) and

Psycho (Psychology). The result reflects a growing degree of use of the technique, in the light of the

number of articles published (Table 1) including the terms Delphi Method, Delphi Technique or Delphi

Study in their titles or abstracts.

In the area of technological forecasting, significant and massive applications of Delphi at a national

level have been noted, such as the studies carried out in Japan [29] (Cuhls and Kuwahara, 1994),

Germany [30] (Blind, Cuhls and Grupp, 2001), Austria [31] (Tichy, 2004), India [32] (Chakarvartyy et

al., 1998) and Korea [33] (Shin, 1998) and the studies promoted by the Institute for Prospective

Forecasting Studies (IPFS)1 in most of the countries of the European Union [34], reflecting a strong

institutional commitment to this method in its field of use.

Although the large majority of the publications concern applications of the technique, in the

methodological area interesting contributions are still being published [20,27,30,31,3541].

Finally, additional examples of the validity of the Delphi method as a research instrument can be

found based on the number of Dissertations and Theses that have employed this technique. We have

extracted this data from the ProQuest Dissertations and Theses database. We have also calculated the

total number of research studies contained in the aforementioned database in each five-year period

196569 0 0.000000

197074 44 0.000152

197579 99 0.000487

198084 298 0.001320

198589 335 0.001398

199094 335 0.001157

199599 281 0.000924

200004 276 0.000936analysed, in order to establish the relative importance of doctoral research employing the Delphi

technique compared to the total number of studies. The results of this search are shown in Table 2.

The data show that, in the area of doctoral research, the technique reached maximum interest during

the decade of the 80s, but currently maintains a notable level of employment, relatively stable, seeming

to indicate that, once the effect of fashion or novelty had been overcome, the scientific community has

accepted this technique as another research technique, with present-day validity and use.

Analysing the data as a whole, articles and doctoral theses, a growing use of the technique is observed

in research published in articles, while in theses its use is decreasing slightly.

1 The Institute for Prospective Technological Studies (IPTS) is one of the seven scientific institutes of the European

Commissions Joint Research Centre (JRC). The mission of the IPTS is to provide customer-driven support to the EU policy-

making process by researching science-based responses to policy challenges that may have both a socio-economic as well as a

scientific/technological dimension.

5. Some recent applications in social science

J. Landeta / Technological Forecasting & Social Change 73 (2006) 467482472In this section we are going to present a summary of some of our recent experiences within the area of

social sciences, carried out by external and professional request, which contain some relatively rare

characteristics compared with other Delphi studies published (due to their scope of application,

methodology or size). These studies show how this technique can be adapted to different social realities and

requirements, making a positive contribution to social progress, provided it is applied with the necessary

methodological rigour and with a good knowledge of the social medium in which it is being applied.

5.1. Delphi application 1: model for imputing individual tourist expenditure in Catalonia2

5.1.1. Objective

This work [42] analysed the minimum tourist expenditure of visitors to Catalonia coming from the

other autonomous communities in the state of Spain. The tourist expenditure per person was analysed

based on its breakdown into four sources of expenditure: accommodation, board, transport and other

expenditure (sundry expenses).

The aim of the study was to obtain reliable estimates of minimum expenditure by tourists visiting

Catalonia by means of the interaction of a group of experts in the tourist area employing a Delphi

methodology. The aim was not to obtain all the estimates that would lead to a quantification of this

expenditure but only those that could not be achieved based on the available data ormore objective sources.

The ultimate aim of the estimates provided by this work was to complete the input for a model of

imputing tourist expenditure that could be used by the Statistics Institute of Catalonia (IDESCAT) in the

case of not obtaining a reliable answer to this question in the periodic surveys carried out.

5.1.2. Participants

The coordinating team was made up of four researchers from the Institute of Applied Business

Economics (IABE) of the University of Basque Country (UPV/EHU) and of four specialists from

IDESCAT and from the Department of Industry, Trade and Tourism of the government of Catalonia (two

per body). The panel of experts was made up of 14 agents with in-depth knowledge of the Catalan tourist

sector: tour operators, travel agencies, tourism offices, representatives from the hotel, restaurant,

apartment and campsite sectors, people who travel intensively in Catalonia, university specialists and

tourism consultants.

5.1.3. Process and dates

The questionnaire had 90 quantitative items and, in each of the questionnaires sections, the experts

were required to provide any information or comment they felt was relevant to the questions asked and

the answers given (qualitative information).

The experts had to assess their own degree of knowledge or mastery (high, medium or superficial) for

each question, and this self-assessment as used to weight their answers.

The questionnaire pre-test was carried out by Catalan members of the coordinating team at their

respective organisations.

2 Catalonia is an important Autonomous Community in Spain. It has over six million inhabitants and highly significant in

both economic and cultural terms. Its GDP accounts for 20% of Spains GDP. Its capital is Barcelona.

Three Delphi rounds were carried out. The first started on the 16th of November 2000 in Barcelona,

with a meeting to present, explain and involve the participants, almost all of the panel attending, as well

as the Catalan members of the coordinating group, the author representing the IABE team and authorities

from IDESCAT and from the Department of Industry, Trade and Tourism of the government of

Catalonia.

J. Landeta / Technological Forecasting & Social Change 73 (2006) 467482 473The feedback was structured by the coordinating group, which the experts received, and, together

with the questionnaires of the next two rounds, were statistical data by item obtained from the

distribution of the answers (individual estimate in the previous roundspecific data from each expert

median, lower quartile, upper quartile, minimum and maximum) and additional information, qualitative

or quantitative, coming originally from the experts themselves.

The process ended on receipt of the third questionnaire from the last of the 14 experts on the 25th of

December 2000.

5.1.4. Results

The main trend and spread were obtained of the 90 items requested, together with valuable qualitative

comments.

With respect to the intrinsic quality of the exercise, various indicators allow us to deduce that this was

good:

The quality and stability of the panel of experts: all those selected took part until the end.

The time between rounds: less than two months.

Comments gathered from the experts: abundant and high quality qualitative information.

Stability of the results between rounds: the degree of group [43] and individual stability [44] in

the answers between rounds 2 and 3 was much higher than that between rounds 1 and 2. In

70% of the answers in the last round, group stability was higher than that in the previous round

(i.e., less variation in the relative interquartile range3) and individual stability was also higher in

80% of these answers (a smaller proportion of experts had modified their answer). The professional

nature of the study ruled out any further rounds since, during the preparatory phase of the study, an

agreement had been reached with the experts that their collaboration would last a maximum of three

rounds.

Consensus4 and convergence of opinion: Consensus was achieved in over 50% of the items, with a

spread rate below 20%, and for 89 of the 90 questions the degree of consensus was higher than that

shown in the first round.

With respect to the social validity of the study, this was clearly positive as the results have been used

successfully by IDESCAT for the purpose for which they were obtained, and they were so satisfied that

they promoted new uses of Delphi oriented at inputting quantitative economic and statistical models.

3 Variation of the relative interquartile range: measures the stability of the group answer and is obtained by calculating the

difference between the relative interquartile range of the group answer of two successive rounds.

RIR variation k=k1 RIRk RIRk1 k round ordinal number:4 Consensus was measured by means of the relative interquartile rangeRIRRIR Q3 Q1 =Mean 100:

The success of the project culminated with a public presentation of the study and a lunch with the

participation of all the experts and the study coordinators.

5.2. Delphi application 2: drawing up the inputoutput tables for Catalonia

5.2.1. Objective

In this study [45], the Delphi technique was also used as input for a quantitative economic model,

specifically in drawing up the InputOutput Tables for Catalonia (TIOC 2001). The InputOutput Tables

[46] are a statistical-accounting instrument that represent the whole of the economic relations between

the different agents in a zone over a specific period of time. They are widely used, both for statistical and

analytical purposes.

In order to prepare these tables, information is required on the functions of production (inputs with

their corresponding origins and proportions) of the homogenous products of the economy under study, in

this case that of Catalonia.

Traditionally, this information had been obtained by postal surveys to random samples of firms. In this

project it was decided to use the Delphi method as an alternative to the classic procedure of survey and

statistical inference for three main reasons:

To improve the quality of the information provided by companies. The traditional system guarantees

the number of answers but not their quality. By means of the Delphi, direct contact was set up with

business experts who had a better knowledge of such specific information, such as the functions of

production of their products, as well as the origin of their inputs and of the services used. These were

normally the purchasing directors, supported by the finance directors.

To eliminate rejection due to the communication, on the part of companies, of information of a

strategic and reserved nature, such as the cost structure of their products. This was achieved by asking

about the function of the production characteristic of the product in general, not of the companys

product in particular. This also helped to obtain more reliable information.

To reduce costs. A traditional survey has very high costs. It involves getting information from a

minimum of 12,000 firms (to be broken down into 120 homogenous products), while the Delphi

method means that this information could be obtained with a much lower number of companies

(approximately four per product).

This way of obtaining information is supported by the methodology of the SEC-95 [47], when it is

proposed to draw up symmetrical inputoutput tables by using the option to resolve the problem of

obtaining internal information from companies bexploiting, as far as possible, qualitative informationfrom expertsQ. However, this is the first real experiment we have heard of using the group opinion ofexperts instead of a traditional survey.

5.2.2. Participants

The coordinating team was made up of three researchers from the Institute of Applied Business

Economics (IABE) and of a team of specialists from the Statistics Institute of Catalonia (IDESCAT). The

panel of experts, divided into 121 sub-panels (involving de facto 121 Delphis), one for each homogenous

J. Landeta / Technological Forecasting & Social Change 73 (2006) 467482474product analysed, was made up of 465 directors, normally purchasing directors, representing as many

Catalan firms.

5.2.3. Process and dates

J. Landeta / Technological Forecasting & Social Change 73 (2006) 467482 475A different questionnaire was designed for each of the 121 industrial products it was decided to use

the Delphi technique for, asking about the consumption of the main raw materials normally used in

making the product (20 estimates or boxes, maximum), giving them a guiding reference taken from

some American IO tables, and about the geographic origin of the purchases of raw materials

(percentages from Catalonia, Rest of Spain, Rest of European Union and Rest of World). Another

additional questionnaire was also prepared per product regarding the consumption of external services

(42 estimates) and their geographic origin.

All the people providing information or experts were invited to attend a session to present the study

and explain to them the purpose of the information they were going to be requested and the repetitive

mechanism to be followed. To facilitate the presentations and organisation, the 121 products from the

manufacturing sector were distributed into 11 groups and 23 presentations were made in total.

The study was designed with two rounds, as the number and characteristics of the experts made a

greater number of rounds unfeasible.

The process started with a pilot Delphi in June 2002. The experts were asked on two occasions (two

Delphi rounds), receiving in the second round the group results for their product, as well as the specific

contributions made by the rest of the experts.

In October 2002 the general industrial Delphi was progressively launched. The second round of the

study was carried out approximately two months later, depending on the group, and the data from this

second round was complete by June 2003.

5.2.4. Results

Measurements of the main trend were obtained (those required as input to the InputOutput tables) for

the raw materials, components and auxiliary services of the 121 products analysed, together with

valuable qualitative comments.

Different indicators allowed us to evaluate positively the quality of the execution of this Delphi

application:

The quality and stability of the panel of experts: 465 experts took part from the most relevant Catalan

firms, of which 85% answered both rounds.

The time between rounds: less than three months, approximately 5 months from the launch of the first

round to receiving all the answers from the second round.

Comments gathered from the experts: copious and high quality qualitative information. Numerous

raw materials provided by the experts that had not been previously suggested, suggestions for

reclassification, positive ratings of the Delphi method for obtaining information.

Modification of the initial opinions as a consequence of repetition and feedback: 65% of the experts

modified their opinion, on average modifying 25% of their individual estimates.

Consensus and convergence of opinion. The average consensus5 in the second round was greater than

in the first. The average typical deviation r of the different distributions from the first round was3.35, while it was 2.43 in the second round.

5 On this occasion, consensus was measured using the typical deviation of different resulting distributions. The small number ofopinions per item and the differences in size between the average values for each item led us to choose this statistical method instead of

other, more traditional ones, such as the relative interquartile range or the coefficient of variation (mean/ typical deviation).

With respect to the social validity of the study, the results obtained (means of the contributions of the

experts per item and product) have been effectively employed in drawing up the aforementioned tables,

J. Landeta / Technological Forecasting & Social Change 73 (2006) 467482476after processing the data with weightings that adjust the relative value of the experts answers according

to their importance in terms of the turnover of the company they represent.

We may assume that the data obtained are of equal or greater quality than those obtained by means of

traditional surveys and it is certain that these have been achieved at a lower financial cost.

The technique has also revealed itself as valid for giving IDESCAT and the Departments of the

Economy and of Industry and Trade of the government of Catalonia a better image and bringing them

closer to the Catalan industrial firms.

5.3. Delphi application 3: Delphi consultation on the Ley de Ordenacion Universitaria (University

Organisation Act) for the Autonomous Community of the Basque Country6

5.3.1. Objective

In 2002 the Basque government needed to promote its own University Organisation Act (LOUV

in Spanish) to replace that which, since 1998, had governed the functioning of the Basque university

system, adapting this to the new context arising as a result of the commencement of the Ley

Organica Universitaria or Public General University Act (LOU in Spanish), of higher rank. This

Basque act aimed to contain, as far as possible, the feelings, opinions and interests of those social

agents most involved or affected by it, so it was necessary to begin a consultative process with these

agents.

The characteristics of the group involved and of the consultation context led us to select a technique

with indirect interaction, specifically the Delphi method, in order to achieve the objectives of obtaining

quality information and opening up channels of social involvement, as desired by the Basque

government, and which also could be favourably accepted by the people involved in the consultation,

also overcoming the limitations that would arise in this social consultation process.

The aim of this technique was to avoid the problems resulting from the multi-campus location of

experts, their frequent travelling and stays abroad, their different timetables, the differences between

status and personality, the influence of their group or class interests and possible prejudices or

concerns of a political nature (it is a highly political university, a reflection of Basque society). It was

also necessary to complete consultation within a limited period of time and of practical duration (the

new act had to be brought out within one year), at the end of the process obtaining results with a high

degree of specificity and clarity that could be communicated to the rest of society, with which the

legal and political levels of the government could work, and which were a faithful reflection of the

feelings and knowledge of all the university community. It was a clear example of the Delphi Policy

[48,49].

5.3.2. Participants

The coordinating team was made up of the Vice-Minister of Universities of the Basque government

and by this researcher of the Institute of Applied Business Economics, supported by an external team to

6 Together with Catalonia, the Basque Country is the oldest Autonomous Community in Spain. It has over two millioninhabitants and is of great importance both industrially and economically. Its cultural history and own language (Euskera or

Basque) has led to the development of a significant nationalist movement. Its capital is Vitoria and its main city is Bilbao.

the government and University with experience in university consultation, which has basically been

J. Landeta / Technological Forecasting & Social Change 73 (2006) 467482 477responsible for the technical and operational process of the Delphi study.

The experts were selected from the group of lecturers of the University of the Basque Country7

(directly and through their union representatives), paying attention to their experience, prestige and

academic and geographic assignment, students (through their student representatives), administration

and service staff (through their union representatives), relevant persons from other Basque universities

(private) and other experts related to Basque universities. In total 115 experts took part in the first round.

In the second round their number fell to 50.

5.3.3. Process and dates

The process started with a visit by the Vice-Minister of Universities to the main centres and groups in

order to present the consultation project and to ascertain university feeling, at the same time as making it

clear to the universities the importance placed on their opinion and the consultation process about to be

started by the government.

The initial questionnaire, which contained 22 questions, was drawn up by the coordinating team and

contrasted in restricted university and government circles. The questionnaire asked about different

aspects related to the organisation of the Basque university system, selected due to their particular

relevance with respect to the areas to be governed by the new Basque law.

Instead of the classic postal or electronic questionnaire, we employed qualified interviewers, aware of

the idiosyncrasies of the university community and trained for this specific study. The interview process

took place during the month of July 2002. Each interview lasted 60 minutes so that quantitative opinions

could be gathered (generally on a Likert scale), as well as valuable and numerous rationales and

comments.

The second questionnaire was sent and collected by email during the months of September and

October. It is important to point out that, with each question, a large amount of qualitative information

was fed back on the contributions of the group, but no quantitative information. In general, they were

provided with the reasons given by the different experts in favour of and against the proposal for each

question. In other words, unlike the vast majority of Delphi studies published, in this research the experts

were not provided with any measurement of the main trend nor of the spread of the distributions of the

answers in the previous round. Not even their own answer in the previous round. The objective of this

work was not to obtain the greatest degree of consensus of the group of panelists but to encourage the

different opinions and feelings latent in Basque university, making them exchange their rationales and

helping everyone learn from each other so that, ultimately, an effective and sufficiently accepted act

could be achieved.

5.3.4. Results

The study results have satisfactorily met the objectives and initial expectations, as the opinions of the

university community have been gathered, both group as well as by professional group and class, on the

aspects consulted with respect to the drawing up of the act on the organisation of Basque university, with

a detailed and ordered list of the reasons in favour and against the positions of the participants on each

question and, by extension, of the professional classes to which they belong.

7 The University of the Basque Country is the only public university in the Basque Country. It has approximately 50,000

students and 4000 lecturers.

As in previous cases, we used different indicators in order to evaluate the quality of the Delphi

exercise:

Modification of the initial opinions as a consequence of repetition and feedback: On average and per

question, 25.39% of the experts who answered in the first round had modified their question to some

J. Landeta / Technological Forecasting & Social Change 73 (2006) 467482478extent with respect to the first answer. If we only consider the experts who answered in the second

round, 57.33% of these modified their previous estimate.

With respect to the social validity of the study, this has been a very positive experience in

AdministrationSocietyUniversity communication, starting with the visits by the Vice-Minister. The

final report on the study was circulated to all the community on the university website, with express

announcements of its union representatives to see it. The report circulated throughout the unions and

political groups, which had the chance to get to know the opinions and feelings of the university at first

hand and without bias.

The Basque University Organisation Act was finally drawn up and passed. This fact, which

apparently has no direct relationship with the success of the study since acts are approved by

parliamentary majorities irrespective of their social referendum, does have a direct relationship in the

particular case of the Basque political reality. The autonomous government had an insufficient

parliamentary majority to pass laws if the rest of the political forces were against them. Due to the

situation of political tension experienced over the last few years, in this last legislature very few acts

have been passed, this being one of them (on the 25th of February 2003).

6. Conclusions and lessons learned

By searching through and reviewing the literature, we have been able to confirm that the Delphi

method continues to be used and is a valid instrument for forecasting and supporting decision-making.

The comparisons made to date with other techniques that share its scope and conditions of action do

not provide decisive findings either in favour of or against the Delphi method, although comparisons

with statistical groups and with classic groups using direct interaction do show results that are, in their

majority, positive for this technique.

8 By group, the highest response rate in the second roundwas that of the lecturers (47%), followed by the union representatives (41%)

and the student representatives (40%). The response rate among external experts to the University of the Basque Country (rectors of The quality and stability of the panel of experts: The vast majority of the experts who were invited to

participate did so. The number of contributions in the second round fell by half, but the experts were

warned that, if they did not reply, their answers from the first round would remain.8

Time between rounds: The answers to the two questionnaires were collected in less than three

months.

The quality and intensity of the participation: Each interview lasted an average of one hour. Apart

from the answers to the 22 Likert-type questions, each expert also made an average of 1.26

qualitative contributions per question.private Basque universities and other relevant people from Basque society) was the lowest (27%). It can be observed that those groups

that were going to be most directly affected by the university law provided a higher degree of response in the second round.

It extent of use in research has not fallen over the last 30 years, and over the last 5 years there has

even been a greater proliferation of articles using this technique as an instrument, particularly in the

fields of Social Sciences (Business, Economics, Psychology) and Health Sciences (Medicine and

Nursing). It is a usual instrument in the areas of technological and social forecasting, social diagnosis,

consensus interpretations of social or health realities, communication and participation. We should also

note that, in spite of the techniques maturity, articles are still being published about the Delphi

methodology.

However, there are characteristics or weaknesses inherent in the methodology or its application that

have not been completely corrected. These problems are greater in the area of Social Sciences, especially

in professional applications carried out on experts who do not have an emotional or professional link or

commitment to those running the study.

The experiences we have presented here have provided us with evidence of the present and potential

of the Delphi method in the areas of input for quantitative models by means of expert opinion, diagnosis

of complex social situations, social forecasting, obtaining a consensus with respect to social needs and

improvement in institutional participation and communication.

On the other hand, the aforementioned experiences, and other earlier ones, have allowed us to learn

certain lessons about effectively running professional Delphi exercises in Social Sciences, some of them

already discussed in classic publications [50]. In summary, the lessons or aspects we feel particular

attention should be paid to are as follows:

1. Having institutional support to support and guarantee the beneficial purpose for society (and not an

exclusively profit-seeking aim for the team running the study) facilitates expert collaboration. This

support must be visible in the acts and instruments of communication with the experts. In most cases,

experts take part due to their pride and their desire to collaborate in research that benefits society.

2. Setting up a plural team to run the study with a perfect understanding of the technique but also with a

very good knowledge of the area being studied, the experts in the area and their motivations and

psychology.

3. Selecting a panel of experts who can make an effective contribution to the theme being studied, with

their degree of motivation to take part also being a key criterion.

4. Putting oneself in the place of the expert. Thinking about what could motivate him or her to take part

and to collaborate actively until the end of the process, what we could offer him or her and the effort

involved for the expert. Generally it is necessary to sacrifice questions and rounds in order to

guarantee panel participation and continuity.

5. When possible, it is advisable to carry out a presentation of the study and emphasise its importance

with the attendance of the experts. The expert becomes aware of the seriousness and relevance of the

study, sees the people who are behind it, understands its methodology, any doubts can be clarified and

fears removed, the experts can meet each other and commit themselves to taking part. Participants

appreciate personal contact [51]. This meeting also serves to differentiate, in principle, those who

have a real interest in taking part (they come) from those who dont.

6. Carrying out a pilot application to improve the precision and comprehension of the questionnaire and

to calibrate any difficulties in managing and motivating the panel of experts and in administrating the

study [52].

J. Landeta / Technological Forecasting & Social Change 73 (2006) 467482 4797. Encouraging the contribution and sending of qualitative feedback: comments, explanations and

suggestions from the experts. Feeding this back so that the change in the estimate, if it takes place, is

the expert must be aware when the study has finally finished and that his or her contribution has been

of some use. It is vital to send the experts the results of the study as soon as possible, with a

also achieving relatively high levels of reliability and validity for a technique of these characteristics.

As a final conclusion we may state that the opinion of experts and the techniques that help to improvethe selection of experts, the externalising of their know-how and the integration and improvement

thereof, as is the case of the Delphi method, are still fully valid in a context where the speed of change

means that the future depends increasingly less on the past and more on the will of the agents of the

present. This is particularly true in the field of Social Sciences, where the intervention of human beings,

with all their complexity and variability, means that, on many occasions, objective data and relations and

models based on these are insufficient to explain and forecast social actions.

Acknowledgments

The author would like to acknowledge and thank the work of the IABE team that carried out the

studies performed for IDESCAT; Jesus Matey, Vicente Ruz and Oskar Villarreal, as well as the work by

those responsible for these studies on the part of IDESCAT itself, namely Alex Costa and Josep Mara

Bas, and also the Vice-Minister of the Universities of the Basque Country, Jon Barrutia, who took part in

designing and implementing the third study on the University Organization Act.

References

[1] W.R. Rieger, Directions in Delphi developments: dissertations and their quality, Technol. Forecast. Soc. Change 29 (1986)

195204.

[2] A. Kaplan, A.L. Skogstad, M. Girshick, The prediction of social technological events, Rand Corp. P-93 (1949).

[3] O. Helmer, N. Rescher, On epistemology of inexact sciences, Manage. Sci. 6 (1) (1959) 2552.

[4] N. Dalkey, O. Helmer, An experimental application of the Delphi method to the use of experts, Manage. Sci. 9 (3) (1963)

458467.personalised letter of thanks and, if possible, holding an event to present the results by way of

gratitude and as a farewell.

Our experiences have shown us that, together with the methodological guidelines usually contained in

the literature on the selection of experts, on the formulation of questions and processing of data,

following the logical guidelines of conduct indicated in this section means that both the researchers and

the experts feel satisfied with having taken part in a Delphi study that has been effectively carried out,due to a real change or modification in opinion, caused by the new information. In other words, the

interaction of qualitative information. This must help, on the one hand, in improving the quality of the

group opinion [40] and, on the other hand, in compensating emotionally those who contribute their

knowledge, as in the following round they see their suggestions included in the questionnaire sent to

all the experts. The expert must have a real sensation of taking part, both in providing information and

constructing the study in collaboration with the rest of the panel and the people running the research.

8. The study must not finish for the expert when the last completed questionnaire has been sent in but

J. Landeta / Technological Forecasting & Social Change 73 (2006) 467482480[5] O. Helmer, E.S. Quade, An approach to the study of a developing economy by operational gaming, Rand Corp. P-2718

(1963).

[6] H. Linstone, M. Turoff, The Delphi Method. Techniques and Applications, Addison-Wesley, Reading, MA, 1975.

[7] T.J. Gordon, O. Helmer, Report on a long-range forecasting study, Rand Corp. P-2982 (1964).

[8] T.W. Weaver, Delphi, a Critical Review, Educational Policy Research Center, Report, vol. RR-7, Syracuse University

Research Corporation, Syracuse NY, 1971.

[9] H. Sackman, Delphi assessment: expert opinion, forecasting and group process, Rand Corp. R-1283-PR (1974).

[10] H. Linstone, Eight basic pitfalls: a checklist, in: H.A. Linstone, M. Turoff (Eds.), The Delphi Method. Techniques and

Applications, Addison-Wesley, Reading, MA, 1975.

[11] G.T. Milkovich, A.J. Annoni, T.A. Mahoney, The use of Delphi procedures in manpower forecasting, Manage. Sci. 19 (4)

(1972).

[12] A.H. Van de Ven, A.L. Delbecq, The effectiveness of nominal, Delphi, and interacting group decision making processes,

Acad. Manage. J. 17 (4) (1974) 605621.

[13] G. Becker, D.A. Bakal, Subject anonymity and motivational distortion in self report data, J. Clin. Psychol. 26 (2) (1970)

207209.

[14] G. Welty, A critique of Delphi technique, The Joint Statistical Meeting of American Statistical Association, Fort Collins,

Colorado, August, 1971, pp. 377382.

[15] J.P. Martino, The consistency of Delphi forecast, Futurist 4 (1970) 63.

[16] J.P. Martino, Technological Forecasting for Decisionmaking, American Elsevier, NY, 1972, third edition in 1993, by

McGraw-Hill, NY.

[17] V.E. Huckfeldt, R.C. Judd, Issues in large scale Delphi studies, Technol. Forecast. Soc. Change 7 (1974) 175184.

[18] T. Gordon, H. Hayward, Initial experiments with the cross impact matrix method of forecasting, Futures, dec., 1968,

pp. 100116.

[19] W.G. Rieger, Directions in Delphi developments: dissertations and their quality, Technol. Forecast. Soc. Change 29 (1986)

195204.

[20] J. Landeta, El Metodo Delphi. Una Tecnica de Prevision Para la Incertidumbre, Ariel, Barcelona, 1999.

[21] D.H. Gustafson, R.K. Shukla, D. Delbecq, G.W. Walster, A comparative study of differences in subjective likelihood

estimates made by individuals, interacting groups, Delphi groups, and nominal groups, Organ. Behav. Hum. Perform. 9

(1973) 280291.

[22] W.E. Rigss, The Delphi technique: an experimental evaluation, Technol. Forecast. Soc. Change 23 (1) (1983) 8994.

[23] R.J. Parente, J.K. Anderson, P. Myers, T. OBrien, An examination of the factors contributing to Delphi accuracy, J.

Forecast. 3 (1984) 173182.

[24] J.M.G.M. Van Dijk, Delphi questionnaires versus individual and group interviews, Technol. Forecast. Soc. Change 37

(1990) 293304.

[25] R.C. Erffmeyer, I.M. Lane, Quality and acceptance of an evaluative task: the effects of four group decision-making

formats, Group Organ. Stud. 9 (4) (1984) 509529.

[26] J. Sniezek, A comparison of techniques for judgmental forecasting by groups with common information, Group Organ.

Stud. 15 (1) (1990) 519.

[27] G. Rowe, G. Wright, The Delphi technique as a forecasting tool: issues and analysis, Int. J. Forecast. 15 (1999) 353375.

[28] U.G. Gupta, R.E. Clarke, Theory and applications Delphi technique: a bibliography (19751994), Technol. Forecast. Soc.

Change 53 (1996) 185211.

[29] K. Cuhls, T. Kuwahara, Outlook of Japanese and German Future Technology. Comparing Technology Forecast Surveys,

Phisica-Verlag, Heidelberg, 1994.

[30] K. Blind, K. Cuhls, H. Grupp, Personal attitudes in the assessment of the future of science and technology: a factor

analysis, Technol. Forecast. Soc. Change 68 (2) (2001) 131149.

[31] G. Tichy, The over-optimism among experts in assessment and foresight, Technol. Forecast. Soc. Change 71 (4) (2004)

341363.

[32] A.K. Chakravarti, B. Vasanta, A.S.A. Krishnan, R.K. Dubash, Modified Delphi methodology for technology forecasting:

case study of electronics and information technology in India, Technol. Forecast. Soc. Change 58 (12) (1998) 155165.

[33] T. Shin, Using Delphi for a long-range technology forecasting, and assessing directions of future R and D activities: the

Korean exercise, Technol. Forecast. Soc. Change 58 (12) (1998) 125154.

[34] Institute for Prospective Technological Studies (IPTS) www.jrc.es/home/pages/list.cfm.

J. Landeta / Technological Forecasting & Social Change 73 (2006) 467482 481[35] M.K. Rayens, E.J. Hahn, Building consensus using the Policy Delphi method, Policy Polit. Nurs. Pract. 1 (4) (2000)

308315.

[36] H. Dransfeld, J. Pemberton, G. Jacobs, Quantifying weighted expert opinion: the future of interactive television and

retailing, Technol. Forecast. Soc. Change 63 (1) (2000) 8190.

[37] S. Keeney, F. Hasson, H.P.W.R. McKenna, A critical review of the Delphi technique as a research methodology for

nursing, Int. J. Nurs. Stud. 38 (2) (2001) 195200.

[38] J.P. Martino, A review of selected advances in technological forecasting, Technol. Forecast. Soc. Change 70 (8) (2003)

719733.

J. Landeta / Technological Forecasting & Social Change 73 (2006) 467482482[39] C. Okoli, S.D. Pawlowski, The Delphi method as a research tool: an example, design considerations and applications, Inf.

Manage. 42 (1) (2004) 1529.

[40] G. Rowe, G. Wright, A. McColl, Judgment change during Delphi-like procedures: the role of majority influence, expertise,

and confidence, Technol. Forecast. Soc. Change 72 (4) (2005) 377399.

[41] R.J. Parente, T.N. Hiob, R.A. Silver, C. Jenkins, M.P. Poe, R.J. Mullins, The Delphi method, impeachment and terrorism:

accuracies of a short-range forecast for volatile world events, Technol. Forecast. Soc. Change 72 (2) (2005) 401411.

[42] J. Landeta, J. Matey, V. Ruiz, O. Villarreal, Alimentacion de modelos cuantitativos con informacion subjetiva: aplicacion

Delphi en la elaboracion de un modelo de imputacion del gasto turstico individual en Catalunya, Questio 26 (12) (2002)

175196.

[43] J.S. Dajani, M.Z. Sincoff, W.K. Talley, The Delphi stability and agreement criteria, Technol. Forecast. Soc. Change 13

(1979) 8190.

[44] M. Scheibe, M. Skutsch, J. Schofer, Experiments in Delphi methodology, in: H.A. Linstone, M. Turoff (Eds.), The Delphi

Method. Techniques and Applications, Addison-Wesley, Reading, MA, 1975.

[45] J. Landeta, J. Matey, V. Ruiz, A. Costa, Resultados de una aplicacion Delphi en la elaboracion de las tablas InputOutput

de Catalunya, XIV Jornadas Luso Espanholas de Gestao Cientfica. CD of Papers, Universidade dos Acores, Ponta

Delgada (Azores-Portugal), 2004.

[46] W. Leontief, InputOutput Economics, Oxford University Press, New York, 1966.

[47] SEC-95. Regulation (EC) 2223/96, pg. 285.

[48] M. Turoff, The design of a Policy Delphi, Technol. Forecast. Soc. Change 2 (1970) 149171.

[49] P. Tapio, Disaggregative Policy Delphi. Using cluster analysis as a tool for systematic scenario formation, Technol.

Forecast. Soc. Change 70 (1) (2002) 83101.

[50] J. Ludlow, Delphi inquiries and knowledge utilization, in: H.A. Linstone, M. Turoff (Eds.), The Delphi Method.

Techniques and Applications, Addison-Wesley, Reading, MA, 1975.

[51] H.P. McKenna, The Delphi technique: a worthwhile approach for nursing? J. Adv. Nurs. 19 (1994) 12211225.

[52] N. Jairath, J. Weinstein, The Delphi methodology (part one): a useful administrative approach? Can. J. Nurs. Adm. 7 (3)

(1994) 2940.

Jon Landeta is a graduate in Economic and Business Sciences, Doctor of Economic and Business Sciences (University of

the Basque Country, 1992), with the thesis bSubjective information for decision-making: the Delphi methodQ. Full professorof the University of the Basque Country since 1987 and Director of the Institute of Applied Business Economics of this

university (www.ehu.es/ieae) since 1998. He also runs the management magazine "Cuadernos de Gestion" (www.ehu.es/

cuadernosdegestion) and the Master programme in Business Administration for Executives of the University of the Basque

Country (www.ehu.es/gestion.html). He has carried out several Delphi studies, both academic and professional, and has

published two books: El Metodo Delphi. Una tecnica de prevision para la incertidumbre (The Delphi Method. A forecasting

technique for uncertainty), published by Ariel, Barcelona, 1999 and Direccion Estrategica de los Recursos Humanos

(Strategic Human Resource Management), published by Piramide, Madrid, 2001, in addition to numerous articles and

conferences on forecasting, strategy and human resources.

Current validity of the Delphi method in social sciencesIntroductionThe Delphi methodEvaluation of the Delphi methodValidity of the Delphi methodSome recent applications in social scienceDelphi application 1: model for imputing individual tourist expenditure in Catalonia2ObjectiveParticipantsProcess and datesResults

Delphi application 2: drawing up the input-output tables for CataloniaObjectiveParticipantsProcess and datesResults

Delphi application 3: Delphi consultation on the Ley de Ordenacin Universitaria (University Organisation Act) for the Autonomous Community of the Basque Country6ObjectiveParticipantsProcess and datesResults

Conclusions and lessons learnedAcknowledgmentsReferences