1

Delphi for Foresightand

Large Group Communications

Murray Turoff turoff@njit.eduhttp://is.njit.edu/turoff

© 2009 Turoff 2

Definition from 1975

Design of a group communication process structured/tailored around the nature of the application and the nature of the group Original paper and pencil rounds Anyone can change their view Anonymity or pennames Scaling theory to promote understanding Voting to focus discussion Select “knowledgeable” people A round took a month – three to five rounds Respondents 15 to 500

Prediction, policy analysis, conditional forecasts, planning, significance of contributions, new product characteristics, etc, etc. (book has many examples)

© 2009 Turoff 3

Added definitions and analogs today A rose by any other name!

Recommender systems Marketplace Systems Collaborative Systems Knowledge structures and systems Collaborative tagging and Folksonomies

ESPgame.org A beautifully simple Delphi process

Structural Modeling (user created models)

Computer Mediated Communications The design of inquiry systems

Scientific: Leibnitz, Locke, Kant, Hegel, Singer

Non Scientific! Heidigger (Negotiated reality and marketing)

Virtual Organizations, teams, groups

© 2009 Turoff 4

Delphi does not make decisions

Delphi should never make a decision, it should be used to analyze a potential decision or action and provide that to a single individual or role that is accountable for the decision to be made. The decision maker for real time decision

should always be one of the participants In Emergency management many different

roles make decisions that can interfere with one another and an Emergency System must track the decision process to insure everyone has access to the most current relevant information This type of system alerts people to conflicts and

those responsible for oversight. The problem solving Delphi (later) is useful in that type of situation.

It is unpredictable what Roles (active 24 hours 7 days a week) are needed at a given time for a given problem but it is always a group that is needed.

© 2009 Turoff 5

Characteristics of Delphi I

A content structure appropriate to the problem being addressed Automatic organization of content Dynamic knowledge base creation

Anonymity of voting and degrees of anonymity for contributions Forced, voluntary, and pen names Some groups are told who other members are to

convince them of quality of group (peers) Anyone can change their original vote Primary goal of votes to expose different views and

generate discussion of causes Ambiguities, uncertainties, misunderstandings, and

disagreements

© 2009 Turoff 6

Characteristics of Delphi II Classifying participants so that votes can be seen

viewed by backgrounds of voters very important for large studies with very heterogeneous groups

Use of voting and comments is to encourage participants to change their views based upon relevant contributions of others. This is a different objective than surveys and

associated assumption about only measuring a fixed state of mind (psychological characteristics) that do not change for an adult

Not all survey guidelines and associated statistics apply to a process intended to deal with complex problems and causing people to rethink their viewpoint

Pennames extremely useful for allowing continuity of thought across a set of comments and for execution of role playing learning games or story telling For example: comment on the following policy

resolutions as the company auditor would

© 2009 Turoff 7

Ultimate Delphi Goal: Collective Intelligence

Group problem solving result is better than any single member could have done before the discussion process Possible with even paper and

pencil Group can solve problems as fast as

any single member considering the same relevant materials (asynchronous operation) Possible with computers and

Structured Computer Mediated Communications

© 2009 Turoff 8

Creativity ExperimentOpen ended idea generation

Distribution of total Raw, Unique, and Rare Ideas Online Discussion System alone and with Delphi

Voting imposed x group size. Problem: product ideas for a pill sized data

responder, ranked list report. Notes: Unique ideas (duplications removed), rare

ideas (occurred in 3 or less groups); Statistically significant on per group and per person that Delphi leads to more improvisation or creativity.

Cho, H. (2004) The effect of Delphi structure on small and medium-sized asynchronous groups, Ph. D. Dissertation, January 2004, New Jersey Institute of Technology, Information Systems Department. http://www.library.njit.edu/etd/index.cfm

© 2009 Turoff 9

Number of Idea Types

Structure x Group Size44 groups (11 groups in each condition)

Small5-7persons

Medium11-14persons

Delphi: Raw ideas (duplicates)

157 247

Unique ideas 67 111

Rare ideas (g<=3)) 40 57

Unstructured: Raw ideas (duplicates)

108 192

Unique ideas 52 86

Rare ideas (g<=3) 19 48Results for total ideas and ideas per personare both statistically significant

© 2009 Turoff 10

Easy Online Delphi

Bulletin Board System imposed structure by monitor Good discussion thread structure

Root items for main ideas Replies for discussions of each idea

Allowed anonymous or real signatures Web survey system where monitor put

the ideas for a vote and provided summaries online Once a week for three weeks

Non structured condition Same system but no imposed structure

by monitor and no surveys done

© 2009 Turoff 11

Why Delphi Too many participants needed for face-to-face

Three to five in any given type of expertise and/or experiences to ensure all rationales are exposed

Early experiments by Dalkey on rationales for specific subjective

With N types needed for a complex problem this is 3xN to 5xN participants required

Different backgrounds that require elimination of ambiguity and misinterpretations as well as translations of concepts

Severe disagreements to be mediated Freedom of expression and improvisation of ideas

without loss of face One out of ten ideas valuable High status participants most concerned about this

Equal Participation allowed Minimize group process losses An idea can be brought up at anytime

© 2009 Turoff 12

Critical Success Factors Composition and quality of the participants

Results only as good as the people involved Compensation for Effort

Communication with a peer group Letting participants nominate others helps

(snowball) Will learn from others in effort Results will serve useful and important purpose Consulting pay if above not strong enough

What is obvious already included, participants used for what is not obvious (no blank sheets of paper)

A morphological structure to automatically organize input material

Important functions and objectives: exploration, understanding one another, exposure of issues and uncertainties, examination of disagreements, and generation of agreements when possible

© 2009 Turoff 13

Why Delphi: operational challenges Subjective judgments required

Models to support consistency of judgments by individuals and group

Building collective models (e.g. cross impact modeling, structural modeling)

Producing collective Group views (e.g. Arrow’s paradox and scaling methods)

Individual human biases (~10) Group process losses and gains (~20) Avoiding consensus (Asch & Hawthorne effects)

pressure Participant effort and shadow time

Asynchronous flexibility to use any time Language or cultural difficulties Virtual teams

Detecting differences by backgrounds for feedback Multitasking, Cognitive Limitations, Information

Overload

© 2009 Turoff 14

Delphi technology and methods

Exploring Capturing individual knowledge

Process of design, problem solving, derivation, knowledge structures

Animation methods Promoting Understanding

Forming a group synthesis Scaling methods

Feedback to group Social judgment & voting theory

Finalizing Viewpoints (no more changes) Evaluation by the group

Collaborative processes

© 2009 Turoff 15

Rank Scale Disaster Damage Dimension 20 20.00 Causalities and fatalities 19 18 18.00 Utilities Impact 17 16 16.00

15.90 Potential to spread Ability of local response adequacy

15 15.43 15.40 15.40 15.38

Loss of command and control Infrastructure damage Resources for aid or containment Time needed to respond

14 14.82 Duration of disaster 13 13.09 Public reaction 12 12.95 Geographic impact 11 10 10.07 Time to return to normal

9 8 8.61 Chance of imminent reoccurrence 7 6 5 4 4.70 Financial Loss 3 2 1 0 0.01 Financial recovery costs

Feedback: a Thurstone Scale for RelativeImportance of Measures ofDisaster Impact

Graduate Student class Delphi in an Emergency Management class with about 30% with real experience in the area.

© 2009 Turoff 16

Paired Comparison for Online Version

© 2009 Turoff 17

Thurstone Scale for Practice Problem

System being builtin Sahana open sourceeffort for real time Decision analysis

© 2009 Turoff 18

Online Delphi’s Today

Dynamic: entering main voting items, voting, making vote changes, and pro, con, neutral discussion comments, can all go on at once.

Each individual can focus on what they want to System notifies user of changes Termination of vote changes is a sign of finalization of the

results People not confident about a given item can always not

vote or defer their vote to later People can propose rewording (Roberts Rules of Order)

What took three months can happen in three weeks for complex problems and large groups

Straight forward problems (e.g. emergency allocation priorities) can be done in real time by dispersed virtual groups of 5-20 professionals.

Typically voting anonymous, comments signed Voting really serves to focus discussion on differences of view Used as a collaborative learning tool in online class discussions System designed as a Social Decision Support System or

Dynamic Delphi

© 2009 Turoff 19

Approaches to the FutureGoals affect what questions to ask

Automation Reduce people time and effort Eliminate jobs

Productivity Increase quantity Increase quality

Opportunity Do new things Do things differently & Better

© 2009 Turoff 20

Approaches to the FutureDimensions of Group Communications

Information Exchange Pooling of current data and information

Cooperation Informing of plans and actions/efforts

Coordination Joint planning of actions/efforts

Collaboration Working together on the same

actions/effort Too often above words are used as if they

were all the same

© 2009 Turoff 21

Examples of general types of Delphi and

Dynamic Delphis

© 2009 Turoff 22

Examples of General Delphi Designsand

Dynamic Delphis These have all been done with paper a number of

times Some have been done partially or totally online or

in combination They could all be done online in a mode where

anyone can work on any part at anytime Some of these still require a human monitor to

perform certain functions and some can be fully automatic

Users can: 1. Add new items or content 2. vote on any item, change their vote, and/or

choose not to vote, and/or vote later after discussion

3. Suggest alternative wordings to existing item or add a new version of an existing item.

© 2009 Turoff 23

System operations

Allow lists to which options, solutions, goals, etc can be added

Allow voting scales for such lists Allow comments on list items Provide each user with new items of any type and

also any whole discussion they want to see Provide vote summaries to indicated the relative

status of voted items and highlight differences Determine any different vote patterns by

participants with different expertise and/or experience, and/or backgrounds

Show for any item Number voting Number of vote changes Number who might yet vote Number of discussion entries about an item

© 2009 Turoff 24

Delphi Examples

Trend Delphi: produces a forecast of a trend along with the mental model of the group making the extrapolation of the trend curve into the future

Problem Solving Delphi: Collects solutions to the problem which are rescaled to a group interval scale based upon individuals ranking or paired comparisons. Use voting to focus discussion on items that need it.

Policy Delphi: seeks policy resolutions and the strongest pro and con evidence or arguments to support each policy resolution

Cross Impact Modeling: Collaborative building of a model of the future possible outcomes of a set of unique events.

© 2009 Turoff 25

Trend Delphi

System Functions Participants Responses System Actions

Present a historical trend to be extrapolated by the participants

Draw a future curve or redraw a new one when a change has occurred in viewpoint.

Present summary of 50% median and 0%, 25%, 75% and 100% boundaries

Request assumptions and uncertainties used to make above estimate

Turn all these into potential assumptions

Vote on validity scale for each potential assumption.

Scale is from completely true to completely false.

Reorder assumptions from true to false.

Focus on middle range (maybe) and ask which can be influenced or measured for occurrence

Assume these can reduce the future uncertainty in the curve

Ask for a redrawing of curve extrapolation based upon assumption list for each trend curve in the study

Supply suggestions on how to influence or measure the maybe assumptions causing significant uncertainty in the projected curve.

Summarize important findings at any time:

Trend, true and false assumptions, assumptions that cause uncertainty, and their potential actions, and measurements

© 2009 Turoff 26

Problem Solving Delphi

System Functions Participants Responses System Actions

State the problem and request solution options

Provide options to solve the problem

Present options in order of occurrence

Request paired comparisons to measure individual preferences for options

Make comparisons for option pairs that a participant feels confident about judging at any time.

Use Thurstone's law of comparative judgment (using incomplete information) to derive a single group interval scale.

Calculate uncertainty due to those who have not yet voted with same type of scale.

Show interval scale.This indicates disagreements

when two or more items are close together. This also shows clustering. Ask for comments about items where people disagree with current position of an item.

Make comments about items you want to see others change their votes about.

Present discussions about items for review. As more people vote or change votes scales will reflect decreasing uncertainty and often more separation between options.

© 2009 Turoff 27

Policy Delphi

System Functions Participants Responses System Actions

State a policy issue to be examined. Ask for specific policy solutions

Add resolution options or specific policies

Request vote for Desirability and Feasibility scales of each solution

Plot two dimensional distribution of policy resolutions

Exploring desirable but infeasible solutions often important

Request comments especially about those showing disagreement

Request comments about policy resolutions. Indicate if comment is pro, con, or neutral.

Request vote on comments for importance and validity

It might be considered important because others believe it to be true

A person may think a comment is important because others think it is valid.

Do same two dimensional plots and summarize discussions

© 2009 Turoff 28

Cross Impact Modeling I

System Functions Participants Responses System Actions

Use problem solving Delphi to produce a set of future unique events focused on a given situation

Evaluate those events for their relative importance to the future objective guiding the choice of events

Place the final most important events into a cross impact model

Ask each individual to answer:What are the probabilities of each

event occurring in some future time frame?

Tell them for each event that they should assume it will or will not occur and ask them to express any changes in the probabilities of the other events due to that certain knowledge about the future.

Show them the expected outcome of their model which will have differences from their predictions. Allow them to vary initial probabilities to see how the future changes.

Allow them to go back and modify some of their estimates

Create the cross impact model using the approach by Turoff (logistic, Fermi Dirac equations). This provides a scale changes from nonlinear probabilities (0 to 1) to a linear influence factor between each pair of events (plus to minus infinity).

When participants are satisfied with their individual model utilize the internal linear influencing factors (Cij) to create a group model.

© 2009 Turoff 29

Cross Impact Modeling IICreating Scenarios

System Functions Participant Responses System Actions

Analyze the internal parameters to show people which of the relationships between which events show the most disagreement among the group.

Ask for comments on these combinations form those who have inconsistent or extreme views.

Show these comments and others and allow those who wish to change some of their original estimations.

Create a model of interacting scenarios by voting on where to stop the integration of the events in process that can turn all the events into one scenario

When no more changes are being made produce. Use Interpretive Structural modeling to generate a set of macro scenarios collecting individual events that are tightly coupled into a set of scenarios that interact.

Requires human monitor to know when to trigger the scenario creation part

© 2009 Turoff 30

Some appropriate words of wisdom

© 2009 Turoff 31

From “The people, yes” by Carl Sandburg

The white man drew a small circle in the sandand told the red man, “This is what the Indian,knows,” and drawing a big circle around thesmall one, “This is what the white man knows.”The Indian took the stick and swept an

immense ring around both circles: This is where thewhite man and the red man know nothing.”(The foresight paradox!)

A seer upon perceiving a flood should bethe first to climb a tree – Kahlil Gibran

© 2009 Turoff 32

Three slides on recommendations in higher education follow

© 2009 Turoff 33

Recommendation for Higher Education I

Major effort to provide online discussion systems for all regular face to face courses Do not emphasize any integration with

administrative services Avoid separate administration services just support

of the hardware and software Seek ability to easily add communication learning

features For example question & answer discussion where

student cannot see ongoing answers and follow on discussion until they have answered the question

Instructors must have control of setting up conferences and membership Ability to bring in guest lectures Ability to set up team conferences Ability to merge different classes for certain projects

and exercises

© 2009 Turoff 34

Recommendation for Higher Education II

Set up prizes and recognition for best online learning exercises in different areas

Provide some guidance, seminars, and workshops for teaching online

Provide some added equipment (cheap) for an instructor to able to record mp3 audio during his real class and be able to put it and slides online that evening With separate slides and audio students can view

printout while listening to lecture off line and/or when traveling

Allow those instructors willing to accept online students to his or her face-to-face class (so sick or traveling students do not have to miss anything)

Allow face to face students to enroll in some online courses

Set a goal of no difference between face-to-face and online student courses, instructors, and or requirements

Offer prizes for student projects to develop improved learning tools for use in what ever is the system supporting the online discussion spaces

© 2009 Turoff 35

Recommendations for Higher Education III

Allow universities to arranging cross registrations to share courses where a given campus does not offer useful courses

Develop missing courses for local pre college schools that can be brought in online for areas where there are missing qualified teachers

Don’t ignore adult education, literacy programs, and skill training programs and online distribution

Consider at some point reducing face to face courses to 1/2 the time in class and put more emphasis on utilizing online time More discussion doable on line then ever can occur

in most face-to-face classes. Start marketing internationally at some point Ask now what are the consequences of not moving

forward in this direction as other countries begin to introduce online courses for Romania

The time to lay out an action plan is now

© 2009 Turoff 36

The following slides contain references for those that want to

learn more

© 2009 Turoff 37

Papers on my website Complete copy of original reference book on the Delphi Method

The Delphi Method: Techniques and Applications by Harold Linstone and Murray Turoff, 1975.

Turoff, M., Hiltz, S. R., Li, Z., Wang, Y., Cho, H., Yao, X., (2004),Online Collaborative Learning Enhancement through the Delphi Method, Proceedings of the OZCHI 2004 Conference, November 22-24, University of Wollongong, Australia

Murray Turoff, Starr Roxanne Hiltz, Michael Bieber, Ajaz Rana (1998), Collaborative Discourse Structures in Computer Mediated Group Communications, 1998

Turoff, Murray, and Starr Roxanne Hiltz (1996), Computer Based Delphi Processes,a version will appear as an INVITED BOOK CHAPTER for Michael Adler and Erio Ziglio, editors., Gazing Into the Oracle: The Delphi Method and Its Application to Social Policy and Public Health, London, Kingsley Publishers (in press).

Turoff, Murray (1997) Alternative Futures for Distance Learning: The Force and the Darkside, The material in this paper was utilized for an Invited Keynote Presentation at the UNESCO / OPEN UNIVERSITY International Colloquium, April 27-29: Virtual Learning Environments and the Role of the Teacher, Open University, Milton Keynes. It also forms the basis of a planned talk at the Third International ALN (Asynchronous Learning Networks) meeting in NY City, October, 1997. (Presentation Overheads for ALN conference, October, 1997, NYU)

Turoff, Murray (1995), Software Design and the Future of the Virtual Classroom, Journal of Information Technology for Teacher Education, Vol 4, No. 2, 1995

Turoff, Murray (1999), An End to Student Segregation: No More Separation Between Distance Learning and Regular Courses. A summary of the invited plenary for the Telelearning 99 meeting in Montreal, Canada, November, 1999. (Also: ppt presentation used in talk.)

© 2009 Turoff 38

Other Papers 1

Turoff, M., Hiltz, S.R.: The Future of Professional Communities of Practice. In: Weinhardt, C., Luckner, S., Stößer, J. (eds.) WeB 2008. LNBIP, vol. 22, pp. 144-158. Springer-Verlag, Berlin Heidelberg (2009)

Xiang, Y, Turoff, M., and Chumer, M. Designing a group support System to Review and Practice Emergency Plans in Virtual Teams, Proceedings of the 6th International ISCRAM Conference, Gothenburg, Sweden, May 2009 (http://iscram.org)

White, Connie, Murray Turoff, Bartel Van de Walle, A Dynamic Delphi Process Utilizing a Modified Thurstone Scaling Method: Collaborative Judgment in Emergency Response, Proceedings of ISCRAM 2007, 4th International Conference on Information Systems for Crisis Response and Management, Delft, the Netherlands, May 13-16, Brussels University Press

Plotnick, Linda, Elizabeth Avey Gomez, Connie White, Furthering Development of a Unified Emergency Scale Using Thurstone's Law of comparative Judgment: A progress Report, Proceedings of ISCRAM 2007, 4th International Conference on Information Systems for Crisis Response and Management, Delft, the Netherlands, May 13-16, Brussels University Press

Turoff, Murray, Starr Roxanne Hiltz, Xiang Yao, Zheng Li, Yuanqiong Wang, and Hee-Kyung Cho, Online Collaborative Learning Enhancement Through the Delphi Method, Turkish Online Journal of Distance Education-TOJDE April 2006 ISSN 1302-6488 Volume: 7 Number: 2 Article: 6, Publisher: Anadolu University, Eskisehir, Turkey, http://tojde.anadolu.edu.tr/index.htm

Hee-Kyung Cho, Murray Turoff, and Starr Roxanne Hiltz, The Impact of Delphi Communication on Small and Medium Sized Asynchronous Groups: Preliminary Results, HICSS 36, January 2003, IEEE Computer Society Press.

Turoff, M., Hiltz, S. R., Li, Z., Wang, Y., Cho, H. "The Delphi Process as a Collaborative Learning Method." In (edited by J. C. Moore) Elements of Quality Online Education: Into the Mainstream: Wisdom from the Sloan Consortium, 121-134. Needham, MA: Sloan-C, September 2004

Banuls, V., and Turoff, M., Scenario Construction via Cross-Impact, Draft under review 2009.

© 2009 Turoff 39

Other Papers 2 Cho, H.K. & Turoff, M., “Delphi Structure and Group Size in Asynchronous

Computer-Mediated Communications,” Proceedings of the Americas Conference on Information Systems, Tampa, August 2003.

Wang, Y., Li, Z., Turoff, M. and Hiltz, S.R. (2003). Using a social decision support system toolkit to evaluate achieved course objectives. Proceedings of the Americas Conference on Information Systems, Tampa, August. (Nominated as a “best paper.”)

Turoff, Murray and S. R. Hiltz, (1995), Computer Based Delphi Processes, in Michael Adler and Erio Ziglio, editors., Gazing Into the Oracle: The Delphi Method and Its Application to Social Policy and Public Health, London, Kingsley Publishers, pp. 56-88.

Worrell, W., Hiltz, S. R., Turoff, M. and Fjermestad, J. (1995) An experiment in collaborative learning using a game and a computer-mediated conference in accounting games. Proceedings of the 28th Annual Hawaii International Conference on System Sciences, Vol. IV, pp. 63-71. Los Alamitos, CA: IEEE Computer Society Press, 1995.

Hsu, Enrico Y. P., Hiltz, S. R., and Turoff, Murray (1992). Computer-Mediated Conferencing System as Applied to a Business Curriculum: A Research Update. In V. S. Jacob and H. Pirkul, eds., The Impact of Information Technology on Business Schools: Research, Teaching and Administration, Proceedings of the 20th Annual North American Conference of the International Business School Computer Users Group, pp. 214- 227. Awarded "Best Paper- Teaching.“

Hiltz, S.R. and Turoff, M., The Network Nation: Human Communication via Computer, 1978, revised edition reprinted 1993 by MIT Press