Embed Size (px)
DESCRIPTION
Prediction and Change Detection. Mark Steyvers Scott Brown Mike Yi University of California, Irvine. This work is supported by a grant from the US Air Force Office of Scientific Research (AFOSR grant number FA9550-04-1-0317). Overview. - PowerPoint PPT Presentation
Citation preview
Prediction and Change Detection
Mark Steyvers Scott Brown Mike Yi
University of California, Irvine
This work is supported by a grant from the US Air Force Office of Scientific Research (AFOSR grant number FA9550-04-1-0317)
Overview
• Prediction in non-stationary time-series data
– statistical properties changing over time
– Example: stock market, traffic, weather
• Accurate prediction requires detection of change
• How well can people predict future outcomes?
• What are the individual differences?
Previous Work
• Much work on perception of stationary random sequences:
– Gambler’s fallacy
– Hot hand belief (e.g. Gilovich et al.)
• Shows how people often perceive changes in arguably stationary sequences: overfitting
Our Approach
• Non-stationary random sequences:
– Distribution changes over time at random points
• Allows for perception of
– too little structure: underfitting
– too much structure: overfitting
Basic Task
• Given a sequence of random numbers, predict the next one
Experiment 1
1
2
3
4
5
6
7
8
9
10
11
12
12 PossibleLocations
• Where next blue square will arrive on right side?
Experiment 1
• 15 blocks of 100 trials
• 21 subjects: all get same sequence
• Window shows history of 30 trials
• Each trial is subject initiated
• Points are given for correct or near-to-correct predictions.
0
5
10
0
5
10
Subject 4
40 50 60 70 80 90 100 110 120 1300
5
10
Time
Subject 12
Sequence Generation
• Locations are drawn from a binomial distribution of size 11, with probability of success θ drawn from [0,1].
• Each time step carries a 10% chance that θ will be changed to a new random value in [0,1]
• Example sequence:
Time
θ=.12 θ=.95 θ=.46 θ=.42 θ=.92 θ=.36
Optimal Strategy
• Optimal strategy: detect change points for θ then identify the mode within each section
• Bayesian model formalizes this strategy(Steyvers & Brown, NIPS, in press)
0
5
10
Optimal Bayesian Solution
0
5
10
Subject 4
40 50 60 70 80 90 100 110 120 1300
5
10
Time
Subject 12
= observed sequence
Optimal Bayesian Solution= prediction
0
5
10
Optimal Bayesian Solution
0
5
10
Subject 4
40 50 60 70 80 90 100 110 120 1300
5
10
Time
Subject 12
Subject 4 – change detection too slow0
5
10
Optimal Bayesian Solution
0
5
10
Subject 4
40 50 60 70 80 90 100 110 120 1300
5
10
Time
Subject 12
Subject 12 – change detection too fast
(sequence from block 5)
Tradeoffs
• Detecting the change too slowly will result in lower accuracy and less variability in predictions than an optimal observer.
• Detecting the change very quickly will result in false detections, leading to lower accuracy and higher variability in predictions.
0 0.5 1 1.5 2
1.2
1.4
1.6
1.8
2
2.2
2.4
Mean Absolute Movement
Me
an
Abs
olu
te T
ask
Err
or
12
3
4
56
7
8
910
11
12
13
14
15
16
17
1819
2021
OPTIMAL SOLUTION
Average Error vs. Movement
= subject
Relatively many changes
Relatively few changes
A simple model
1. Make new prediction some fraction α of the way between old prediction and recent outcome.
2. Fraction α is a linear function of the error made on last trial
3. Two free parameters: A, B
A<B bigger jumps with higher error
A=B constant smoothing
1 (1 )t t tp p y
t t tError y p
α
0
1
A
B
BA
Sweeping the parameter space
0 0.5 1 1.5 2
1.2
1.4
1.6
1.8
2
2.2
2.4
Mean Absolute Movement
Me
an
Abs
olu
te T
ask
Err
or
12
3
4
56
7
8
910
11
12
13
14
15
16
17
1819
2021
OPTIMAL SOLUTION
= subject
= model
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
B parameter
A p
ara
me
ter
1
2
34
56
7
89
10
11
12
13
14
15
16
17
18
19
2021
Best fitting parameters for individual subjects
t t tError y p
α
0
1A=B
A<B
A p
aram
eter
B parameter
α ≈ constant(bad strategy– no
jumps)
Jumps with large errors: good strategy
Effect of A and B parameters
0 0.5 1 1.5 2
1.2
1.4
1.6
1.8
2
2.2
2.4
Mean Absolute Movement
Me
an
Abs
olu
te T
ask
Err
or
12
3
4
56
7
8
910
11
12
13
14
15
16
17
1819
2021
OPTIMAL SOLUTION
= subject
= model A ≈ B
= model A << B
Model misses some trends in data…
0
2
4
6
8
10
12 = observed sequence
= prediction
False perception of motion: if successive blocks go up, then extrapolate the trend
(subject 12, block 3)
= observed data
= prediction
Experiment 2: two-dimensional prediction
• Touch screen monitor
• 1500 trials • Self-paced • Same sequence
for all subjects
0.5 1 1.5 2 2.5 3 3.52.7
2.8
2.9
3
3.1
3.2
3.3
3.4
3.5
Mean Absolute Movement
Me
an
Abs
olu
te T
ask
Err
or
12
34
5
6
7
8 9
OPTIMAL SOLUTION
Average Error vs. Movement
= subject
Average Error vs. Movement
0.5 1 1.5 2 2.5 3 3.52.7
2.8
2.9
3
3.1
3.2
3.3
3.4
3.5
Mean Absolute Movement
Me
an
Abs
olu
te T
ask
Err
or
12
34
5
6
7
8 9
OPTIMAL SOLUTION
= subject
= model
Conclusion
• Individual differences
– Overfitters: hypotheses too complex
– Underfitters: hypotheses too simple
• Relation to perception of real-world phenomena?
• Relation to personality characteristics?
Best fitting parameters for individual subjects
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1
B parameter
A p
ara
me
ter
12
3
4
5
6
7
8
9
t t tError y p
α
0
1A=B
A<B
Responses across subjects
= observed sequence
= #subjects with that prediction
(sequence from block 5)
60 70 80 90 100 110 120 130
0
2
4
6
8
10
12
Time
Responses across subjects
40 50 60 70 80 90 100 110 120 130
0
2
4
6
8
10
12
= observed sequence
= #subjects with that prediction
(sequence from block 5)
