Eric van Heck

Experimental Results of Online Information Aggregation Markets for Sales Forecasting

Eric van Heck

Rotterdam School of Management

Erasmus University

evanheck@rsm.nl

www.rsm.nl/evanheck

INSEAD Presentation

Fontainebleau, 20 June 2008

© Eric van Heck, 2008.

Menu

1. What are information aggregation markets (or also called prediction markets)?

2. State-of-the-art in practice

� IOWA Political Markets

� Hollywood Stock Exchange

3. State-of-the-art in theory

� Project 1 with Mathijs van der Vlis

4. State-of-the-art in theory and practice

� Project 2 with Annie Yang et al. and anonymous company

5. Conclusions

IntroductionS

toc

kh

olm

v.v

.

How many passengers can travel

with the Silja Symphony?

He

lsin

ki –

Sto

ck

ho

lm v

.v.

Aggregation and AveragingS

toc

kh

olm

v.v

.

� Bo: 2,700

� Heli: 2,640

� Virpi: 3,005

� Ari: 3,050

� Pekka: 2,502

� Mika: 2,600

He

lsin

ki –

Sto

ck

ho

lm v

.v.

� Jyrki: 2,845

� Szymon: 2,777

� Ralph: 2,799

� Esko: 3,592

Aggregation and AveragingS

toc

kh

olm

v.v

.

� Bo: 2,700

� Heli: 2,640

� Virpi: 3,005

� Ari: 3,050

� Pekka: 2,502

� Mika: 2,600

He

lsin

ki –

Sto

ck

ho

lm v

.v.

� Jyrki: 2,845

� Szymon: 2,777

� Ralph: 2,799

� Esko: 3,592

� Average: 2,851

Aggregation and AveragingS

toc

kh

olm

v.v

.

� Bo: 2,700

� Heli: 2,640

� Virpi: 3,005

� Ari: 3,050

� Pekka: 2,502

� Mika: 2,600

He

lsin

ki –

Sto

ck

ho

lm v

.v.

� Jyrki: 2,845

� Szymon: 2,777

� Ralph: 2,799

� Esko: 3,592

� Average: 2,851

� Correct answer: 2,852

Aggregation and AveragingS

toc

kh

olm

v.v

.

� Bo: 2,700

� Heli: 2,640

� Virpi: 3,005

� Ari: 3,050

� Pekka: 2,502

� Mika: 2,600

He

lsin

ki –

Sto

ck

ho

lm v

.v.

� Jyrki: 2,845

� Szymon: 2,777

� Ralph: 2,799

� Esko: 3,592

� Average: 2,851

� Correct answer: 2,852

� The average is a very good predictor – wisdom of crowds.

� Jyrki is closest to the correct answer!

What are information markets?

1. A group of people that buy and sell stocks.

2. Stocks represent the potential outcome of the subject to be forecasted (number of Silja passengers, future demand of mobile telephones, winner soccer game, etc).

3. Market mechanism is a double auction.

4. Market price of a particular stock represent the probability that that potential outcome will happen – for example: stock Italy (in the game Italy – NL) is 0,80 cent (range 0 – 100 cents) = probability that Italy wins is 80%.

5. The market aggregates information by the aggregation of the individual beliefs of the players.

State-of-the-art in practice

Some applications in practice:

� IOWA Political Markets

� Hollywood Stock Exchange

� Internal Information Markets for example by HP,

Google, and external Information Markets such as

NewsFutures, Foresight Exchange.

IOWA Political Markets

• IOWA political markets perform better than

polling results

• Presidential election markets perform

better than (lower profile) congressional,

state, or local elections

Lessons Learned (Berg et al, 1996, 2000)

state, or local elections

• Markets with more volume near the

election perform better

• Markets with fewer contracts (i.e. fewer

candidates or parties) predict better

Hollywood Stock Exchange

Trading in MovieStocks

Trading in StarBonds

• Prices of securities in Oscar, Emmy, and

Grammy awards correlate well with actual

award outcome frequencies, and prices of

movie stocks accurately predict real box

office results (Pennock, 2001).

Lessons Learned

office results (Pennock, 2001).

Hype Cycle for Emerging Technologies 2006

Market Characteristics

Trader Characteristics

Incentive Mechanism

Market Information/Signals

Trading Mechanism

Contract Type (binary, spread, index)

Trader Anonymity

Liquidity/Market Size Selling short/portfolios

Market Efficiency

Information Cascades/Market Bubbles

Transaction Costs

Prediction Metric (last trading price, avg price)

Frequency of information update

State-of-the-art in theory

Trader Characteristics

Characteristics of the to-be-predicted event

Biases/Bounded Rationality

Trading Experience/Knowledge

TraderType

Information Source

Risk Attitude

Private Information

Time Scope

Aggregate Certainty

Information Availability/Costs

Inherent Predictability Wealth

Cheating/Collaboration/Manipulation

Trader Demographics

Homogeneity/Heterogeneity

• Mechanism Design Theory (Hayek 1945)

Markets are an appropriate mechanism for the purpose of efficient information aggregation and decision making due to the incentives for information discovery.

• Double Auction Theory (Plott and Sunder, 1982, 1988)

Prediction markets have the ability to aggregate dispersed private information held by individuals as the double auction mechanism has the ability to disseminate private information among traders.

Main Theories

information held by individuals as the double auction mechanism has the ability to disseminate private information among traders.

• Rational Expectation Theory (Lucas 1972, Grossman 1981)

The price observed in a prediction market is a sufficient statistic for all information available to traders

• The Wisdom of Crowds (Surowiecki 2004)

Small and large groups of people seem to do better at decision making than individuals.

Mechanism Design Theory

Project 1 - with Mathijs van der Vlis :

What is the impact of the number of traders,

the distribution of wisdom, and monetary incentives

to the outcome of information markets?

Hypotheses1. Number of traders (Surowiecki, 2004)

More traders will increase the level of aggregation and the level of prediction accuracy

2. Distribution of wisdom (Anderson and Holt, 1997; Hansen, Smith, and Strober, 2001; Hanson and Oprea, 2004)

Uneven distribution among traders will increase the level of aggregation and the level of prediction accuracy

3. Monetary incentives (Servan-Schreiber, Wolfers, Pennock, and Galebach, 2004)

Monetary incentives will not increase the level of aggregation and the level of prediction accuracy

128 Laboratory Experiments to forecast future demand

of mobile telephones

Results Experiments (N = 128)

Hypotheses1. Number of traders (Surowiecki, 2004)

More traders will increase the level of aggregation and level of prediction accuracy

2. Distribution of wisdom (Anderson and Holt, 1997; Hansen, Smith, and Strober, 2001; Hanson and Oprea, 2004)

No

Yes

Uneven distribution among traders will increase the level of aggregation and the level of prediction accuracy

3. Monetary incentives (Servan-Schreiber, Wolfers, Pennock, and Galebach, 2004)

Monetary incentives will not increase the level of aggregation and the level of prediction accuracy

Yes

No

Yes

Yes

• Results indicate that even in the presence of a small number of traders there tends to be aggregation, while only in the presence of a large number of traders are accurate predictions generated.

• When wisdom is unequally distributed there is aggregation (wise people lead markets), yet the markets

Lessons Learned

aggregation (wise people lead markets), yet the markets do not produce more accurate predictions (wise people can potentially mislead markets).

• Monetary incentives impact neither the level of aggregation nor the level of accuracy.

State-of-the-art in theory and practiceS

toc

kh

olm

v.v

.

Some applications:

� Internal Information Markets for example at a

financial company

He

lsin

ki –

Sto

ck

ho

lm v

.v.

Project 2 - with Annie Yang, Maarten Colijn, Willem

Verbeke, Mathijs van der Vlis and anonymous

company

What is the performance of information markets in

forecasting the overall sales of a product

over several regions in the Netherlands?over several regions in the Netherlands?

• Market Size – Number of Traders (Surowiecki 2004, Hansen 2003)

H1a: A prediction market with more traders is likely to aggregate sooner and more significantly.

H1b: A prediction market with more traders is likely to forecast more accurately.

Hypotheses

• Monetary Incentives (Servan-Schreiber et al. 2004)

H2: An offer of monetary incentives does not affect the activeness of traders’ participation in a prediction market.

• Time Horizon (Berg et al. 2003)

H3: A prediction market forecasts more accurately in a short run than in a long run.

Trading Web Page

De

sc

rip

tio

n o

f P

red

icti

on

Ma

rke

ts1st Prediction Market 2nd Prediction Market

Subject to be predicted Annual sales of a financial product Periodical sales of a financial product

Contracts Spread contracts (in million euro) Spread contracts (in million euro)

Description of traders Regional sales managers Regional sales managers

Number of stocks 10 9

Number of traders 34 34

Number of active traders 31 18

De

sc

rip

tio

n o

f P

red

icti

on

Ma

rke

ts

Number of active traders 31 18

Number of very active traders

8 3

Total number of bids (incl. demand and sell)

604 461

Total number of completed bids (buy and sell)

368 275

Time of markets 24 hrs / 7 days 24 hrs / 7 days

Market duration 12 calendar days (Feb 2007) 12 calendar days (June 2007)

Historical Stock Prices in 1st Prediction Market

50

60

70

80

Stock

Pric

e (in

point)

110-120

121-130

131-140

141-150

151-160

161-170

Market forecast

Actual sales

i.e. 133

Aggregation and Forecasting Results

0

10

20

30

40

1st 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th

Trade Day

Stock

Pric

e (in

point)

161-170

171-180

181-190

191-200

201-210

Trading Day

Top-down

forecast

Historical Stock Prices in 2nd Prediction Market

80

100

120

Stock

Pric

e (in po

int)

19 - 22

22 - 25

25 - 28

28 - 31

31 - 34

Market forecast

Actual sales

Top-down

forecast

Aggregation and Forecasting Results

0

20

40

60

1st 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th

Trade Day

Stock

Pric

e (in po

int)

34 - 37

37 - 40

40 - 43

43 - 46

Trading Day

i.e. 28.6

1st Prediction Market 2nd Prediction Market

% error % error

Actual results 133 28.6

Forecasting Accuracy

Prediction market forecast 141-150 +6% - 13% 19-22 -23% - 34%

Top-down forecast 150-160 +13% - 20% 27 -6%

• Market Size – Number of Traders (Surowiecki 2004, Hansen 2003)

H1a: A prediction market with more traders is likely to aggregate sooner and more significantly.

H1b: A prediction market with more traders is likely to forecast more accurately.

Hypotheses

No

Yes

• Monetary Incentives (Servan-Schreiber et al. 2004)

H2: An offer of monetary incentives does not affect the activeness of traders’ participation in a prediction market.

• Time Horizon (Berg et al. 2003)

H3: A prediction market forecasts more accurately in a short run than in a long run.

Yes

No

1. Market size, in terms of the number of traders, does not necessarily influence market aggregations but the accuracy of predictions. A thicker market is more likely to forecast accurately.

2. Monetary incentives are not effective to motivate traders to trade in internal prediction markets – time for trading is a constraint.

Lessons Learned

is a constraint.

3. Markets predict more accurately in a long run than in a short run. Interesting because the impact of the worldwide mortgage crises was predicted very well

4. Traders are sensitive to the prices of contracts, learning from signals and constantly updating their beliefs. However, this yields that traders could be easily misled, particularly in a thin market.

ConclusionsS

toc

kh

olm

v.v

.

1. “Information Aggregation” is a Key Critical

Component for Firms - online markets can

improve the information aggregation capability of

a firm!

2. Several issues need to be solved for example:

He

lsin

ki –

Sto

ck

ho

lm v

.v.

2. Several issues need to be solved for example:

� details of the market design

� incentive structure of players

3. Do you want to know more: please join!