Embed Size (px)
Citation preview
© 2010 The MITRE Corporation. All rights reserved.
Prediction Markets:Do They Improve Risk Management?
Jon SchulerMITRE [email protected] September 2010
© 2010 The MITRE Corporation. All rights reserved.
■ Introduced an InklingTM prediction market to the USAF■ Generic questions:
– “Will the New England Patriots sign Terrell Owens by the start of the 2010 NFL season? “
– “Will both BP and U.S. Government sources officially report no more oil is leaking from the Deepwater Horizon well by July 1, 2010?
■ Acquisition-program specific questions:– “Will Predator platforms adopt the Program2 Rev B specs?”– “Will any of the load-balancing, multi-path TRILL boxes tested
at last month's UNH Interop lab appear in an USAF Advanced Technology Demonstration by Spring 2011?”
– “Which of following Spectrum bands will Program2 be approved for use by Jan 2011”
What we did:
Page 2
© 2010 The MITRE Corporation. All rights reserved.
“What’s a prediction market good for?”
Page 3
Improve dialogue withinacquisition team
Lets people offer “reprisal-free” feedback
Foster diversity of viewpointsinto analysis
Etc. etc etc.
© 2010 The MITRE Corporation. All rights reserved.
■ Pre-existing risk-management process already in place– MITRE provides on-site support to the USAF– Established process that identifies, enunciates, quantifies, and
mitigates acquisition risks– 14 acquisition programs; various levels of activity
■ 2+ years data logged in RiskNav software– Front-end: web-based interface
■ Enter/modify assessments. Provide summary display– Back-end: Microsoft Access database– Primary use: characterize current state– Mining DB change-logs can extract a historical record
Can we clinically measure that?
Page 4
© 2010 The MITRE Corporation. All rights reserved.
Page 5
Prediction Market Analysis Framework
People
PredictionMarket
Acquisition ProgramRisk Management
Infer Influence
RiskNavanalysis
Prediction Marketanalysis
© 2010 The MITRE Corporation. All rights reserved.
Page 6
‘Treatment Effects’ we would expect to see
■ Is there an increase in the overall database activity?■ Is there an increase in the rate of newly identified risks?■ Are new risks identified earlier from their event-horizon?■ Do risks get mitigated or closed more quickly?
PredictionMarket
Acquisition ProgramRisk Management
RiskNavanalysis
© 2010 The MITRE Corporation. All rights reserved.
■ Special Note:– This doesn’t analyze the prediction market itself; we just
assume that one took place beginning June 2010– This analyzes project risk-management activity, comparing 2
months before with 2 months after the prediction market began
Event timeline under analysis
Page 7
April-May June July-Aug
2010
Base Period Test Period
Inkling outreach/invitationsto USAF
1 June 2010
© 2010 The MITRE Corporation. All rights reserved.
Database activity over time
Page 8
0 100 200 300 400 500 600 700 8000
100
200
300RiskNav activity for Program1 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
200
400
600
30-d
ay s
um
0
500
1000
1500
-Jul-2008-Aug-2008-Sep-2008-O
ct-2008-N
ov-2008-D
ec-2008-Jan-2009-Feb-2009-M
ar-2009-Apr-2009-M
ay-2009-Jun-2009-Jul-2009-Aug-2009-Sep-2009-O
ct-2009-N
ov-2009-D
ec-2009-Jan-2010-Feb-2010-M
ar-2010-Apr-2010-M
ay-2010-Jun-2010-Jul-2010-Aug-2010
30-d
ay b
oots
trap
© 2010 The MITRE Corporation. All rights reserved.
Database activity over time
Page 9
0 100 200 300 400 500 600 700 8000
10
20
RiskNav activity for Program5 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
20
40
60
30-d
ay s
um
0
100
-Jul-2008-Aug-2008-Sep-2008-O
ct-2008-N
ov-2008-D
ec-2008-Jan-2009-Feb-2009-M
ar-2009-Apr-2009-M
ay-2009-Jun-2009-Jul-2009-Aug-2009-Sep-2009-O
ct-2009-N
ov-2009-D
ec-2009-Jan-2010-Feb-2010-M
ar-2010-Apr-2010-M
ay-2010-Jun-2010-Jul-2010-Aug-2010
30-d
ay b
oots
trap
© 2010 The MITRE Corporation. All rights reserved.
■ April-May updates: 246 (Baseline)■ July-Aug updates: 381 (Test period)
Program1 update activity
Page 10
April-May July-AugJune
0 100 200 300 400 500 600 700 8000
100
200
300RiskNav activity for Program1 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
200
400
600
30-d
ay s
um
0
500
1000
1500
-Jul-2008-A
ug-2008-S
ep-2008-O
ct-2008-N
ov-2008-D
ec-2008-Jan-2009-Feb-2009-M
ar-2009-A
pr-2009-M
ay-2009-Jun-2009-Jul-2009-A
ug-2009-S
ep-2009-O
ct-2009-N
ov-2009-D
ec-2009-Jan-2010-Feb-2010-M
ar-2010-A
pr-2010-M
ay-2010-Jun-2010-Jul-2010-A
ug-2010
30-d
ay b
oots
trap
0 50 100 1500
10
20
30
40
50
60
RiskNav activity for Program1 from 01-Apr-2010 to 31-Aug-2010
upda
tes
© 2010 The MITRE Corporation. All rights reserved.
■ April-May updates: 246 (Baseline)■ July-Aug updates: 381 (Test period)
Program1 update activity
Page 11
O.K….I can see an increase in activity but is that increase significant?
April-May July-AugJune0 50 100 150
0
10
20
30
40
50
60
RiskNav activity for Program1 from 01-Apr-2010 to 31-Aug-2010
upda
tes
© 2010 The MITRE Corporation. All rights reserved.
■ Pick 62 days at random (with replacement) from baseline period– Tally up the total number of database updates observed in re-sample
Bootstrap re-sampling of baseline (Apr-May) data
Page 12
April-May July-AugJune0 50 100 150
0
10
20
30
40
50
60
RiskNav activity for Program1 from 01-Apr-2010 to 31-Aug-2010
upda
tes
© 2010 The MITRE Corporation. All rights reserved.
■ Pick 62 days at random (with replacement) from baseline period– Tally up the total number of database updates observed in re-sample
■ Repeat a few thousand times; build empirical distribution
■ 1-sided hypothesis test– If we see 377 or more RiskNav updates in 62 days, we can ascribe this
increase to the Prediction Market…at the 5% chance this could have otherwise occurred ‘naturally’
Bootstrap re-sampling of baseline (Apr-May) data
Page 13
0 100 200 300 400 500 6000
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1bootstrap distribution of #RiskNav updates over a 62-day period
50 100 150 200 250 300 350 400 450 500 5500
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.05 p-value occurs after 377 updates in 62 days
246 baseline
updates381 test period
updates
© 2010 The MITRE Corporation. All rights reserved.
■ Corresponding bootstrap analysis: how many new risks must we see in a 62-day period?
■ Concluding answer: need to see 17+ new risks
Examine the rate of newly identified risks
Page 14
0 5 10 15 20 25 30 350
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1bootstrap distribution of #RiskNav updates over a 62-day period
0 5 10 15 20 25 300
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.05 p-value occurs after 17 updates in 62 days
0 100 200 300 400 500 600 700 8000
10
20
New Risks for Program1 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
10
20
30-d
ay s
um
0
50
100
-Jul-2008
-Aug-2008
-Sep-2008
-Oct-2008
-Nov-2008
-Dec-2008
-Jan-2009
-Feb-2009
-Mar-2009
-Apr-2009
-May-2009
-Jun-2009
-Jul-2009
-Aug-2009
-Sep-2009
-Oct-2009
-Nov-2009
-Dec-2009
-Jan-2010
-Feb-2010
-Mar-2010
-Apr-2010
-May-2010
-Jun-2010
-Jul-2010
-Aug-2010
30-d
ay b
oots
trap
© 2010 The MITRE Corporation. All rights reserved.
■ April-May new risks: 11 (Baseline)■ July-Aug new risks: 17 (Test period)
Program1 new risks
Page 15
O.K….I can see an increase in activity, and it is statistically anomalous
April-May July-AugJune0 50 100 150
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
5.5RiskNav activity for Program1 from 01-Apr-2010 to 31-Aug-2010
upda
tes
© 2010 The MITRE Corporation. All rights reserved.
Risk Management Update New # Observed Updates # Observed New RisksProcess Maturity Program Activity Risks Baseline Test .05 Level p-value Baseline Test .05 Level p-valueSelf-Generating Program1 Y Y 246 381 376 0.044 11 17 17 0.047
Program2 * Y 258 117 377 0.991 6 8 6 0.003
Program3 ** 162 145 246 0.635 3 0 3 0.640Stimulus Driven Program4 Y Y 100 222 173 0.003 0 5 0 0.000
Program5 24 8 52 0.839 1 0 3 0.646Program6 Y 8 21 20 0.027 3 0 0 0.000Program7 63 56 132 0.541 2 1 7 0.738
Program8 Y Y 10 30 24 0.012 0 2 0 0.000Initiating Program9 0 0 0 0.000 0 0 0
Program10 Y 0 20 0 0.000 0 3 0 0.000
Program11 46 1 117 0.967 3 0 7 0.870Inactive Program12 0 0 0 0 0 0
Program13 0 0 0 0 0 0
Program14 0 0 0 0 0 0
Update & New Risk activity for all programs
Page 16
© 2010 The MITRE Corporation. All rights reserved.
■ 4 types of risk-management cultures observed in the 14 programs– Self-generating - routine updates on a
consistent basis with no evident stimulus
– Stimulus Driven – periodic updates as the result of external events (e.g. PMRs, risk meetings, risk team interactions)
– Initiating – programs initializing a risk management process or use of the tool
– Inactive – programs no longer actively managing risks
Let’s break this down into something simpler
Page 17
0 100 200 300 400 500 600 700 8000
50
100
150
200
250
300
RiskNav activity for Program1 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
5
10
15
20
RiskNav activity for Program5 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
5
10
15
20
25
30
35
40
45
50
RiskNav activity for Program11 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
50
100
150
RiskNav activity for Program13 from 09-Jun-2008 to 01-Sep-2010
daily
© 2010 The MITRE Corporation. All rights reserved.
■ Look at the #days between database entries■ Kruskal-Wallace rank-based 1-way ANOVA tells demarks
programs by significantly different median inter-arrival times
You can distinguish programs rigorously…
Page 18
0
50
100
150
200
Pro
gram
1
Pro
gram
12
Pro
gram
2
Pro
gram
9
Pro
gram
5
Pro
gram
6
Pro
gram
11
Pro
gram
4
Pro
gram
3
Pro
gram
13
Pro
gram
7
Pro
gram
14
Pro
gram
8
inte
r-arri
val t
ime
(day
s)
0 100 200 300 400 500 600 700 800 900
Program8
Program14
Program7Program13
Program3
Program4
Program11
Program6
Program5
Program9
Program2
Program12
Program1
Click on the group you want to test
3 groups have mean ranks significantly different from BACN JUON
© 2010 The MITRE Corporation. All rights reserved.
■ Treatment effect concentrated in programs with self-generating risk dialogues– *Program2: just coming on-line to
RiskNav– **Program3: Experienced program
cuts in Aug.■ Treatment effect in stimulus driven
is observed but discounted – Sufficiently large variability in
sparse updates
Actionable summary
Page 19
Risk Management Update NewProcess Maturity Program Activity RisksSelf-Generating Program1 Y Y
Program2 * YProgram3 **
Stimulus-Driven Program4 Y YProgram5Program6 YProgram7Program8 Y Y
Initiating Program9Program10 YProgram11
Inactive Program12Program13Program14
© 2010 The MITRE Corporation. All rights reserved.
– Program transitioning into RiskNav tool– Abnormally high activity during baseline period.– Don’t see a treatment effect during test period in update activity– Do see a treatment effect for new risks
Program2 in more detail
Page 20
0 100 200 300 400 500 600 700 8000
20
40
60RiskNav activity for Program2 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
100
200
30-d
ay s
um
0
200
400
-Jul-2008-A
ug-2008-S
ep-2008-O
ct-2008-N
ov-2008-D
ec-2008-Jan-2009-Feb-2009-M
ar-2009-A
pr-2009-M
ay-2009-Jun-2009-Jul-2009-A
ug-2009-S
ep-2009-O
ct-2009-N
ov-2009-D
ec-2009-Jan-2010-Feb-2010-M
ar-2010-A
pr-2010-M
ay-2010-Jun-2010-Jul-2010-A
ug-2010
30-d
ay b
oots
trap
© 2010 The MITRE Corporation. All rights reserved.
– Mission fundamentally re-evaluated during test period■ Program cuts August 2010
– Treatment effect is observed comparing {July} vs. {April-May}
Program3 in more detail
Page 21
0 100 200 300 400 500 600 700 8000
50
100
150RiskNav activity for Program3 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
200400600800
30-d
ay s
um
0
500
1000
-Jul-2008-A
ug-2008-S
ep-2008-O
ct-2008-N
ov-2008-D
ec-2008-Jan-2009-Feb-2009-M
ar-2009-A
pr-2009-M
ay-2009-Jun-2009-Jul-2009-A
ug-2009-S
ep-2009-O
ct-2009-N
ov-2009-D
ec-2009-Jan-2010-Feb-2010-M
ar-2010-A
pr-2010-M
ay-2010-Jun-2010-Jul-2010-A
ug-2010
30-d
ay b
oots
trap
© 2010 The MITRE Corporation. All rights reserved.
– 100+ update day occurred in June– Had that stimulus-day occurred in May or July, would have
come to different conclusions– Need longer observation time to declare/deny treatment effect
Program4 in more detail
Page 22
0 100 200 300 400 500 600 700 8000
50
100
150RiskNav activity for Program4 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
200
400
30-d
ay s
um
0
500
1000
-Jul-2008-A
ug-2008-S
ep-2008-O
ct-2008-N
ov-2008-D
ec-2008-Jan-2009-Feb-2009-M
ar-2009-A
pr-2009-M
ay-2009-Jun-2009-Jul-2009-A
ug-2009-S
ep-2009-O
ct-2009-N
ov-2009-D
ec-2009-Jan-2010-Feb-2010-M
ar-2010-A
pr-2010-M
ay-2010-Jun-2010-Jul-2010-A
ug-2010
30-d
ay b
oots
trap
© 2010 The MITRE Corporation. All rights reserved.
Page 23
‘Treatment Effects’ we would expect to see
■ Is there an increase in the overall database activity?■ Is there an increase the rate of newly identified risks?■ Are new risks identified earlier from their event-horizon?■ Do risks get mitigated or closed more quickly?
PredictionMarket
Acquisition ProgramRisk Management
RiskNavanalysis
© 2010 The MITRE Corporation. All rights reserved.
■ Subset of new risks specified a lead time:■ Wilcoxon rank-sum test: Do two populations have equally
large values?– Insufficient data to draw any conclusion
Risk lead time
Page 24
Program #newRisks (base) #newRisks (test) medianImpactDays (base) medianImpactDays (test) Wilcoxon p-valueProgram1 10 13 109 82 0.120853576
Program12 0 0
Program2 3 1 476 42 0.5
Program9 0 0
IProgram10 0 3
Program5 1 1 24 101 1
Program6 0 0
Program11 3 6 112 175 0.547619048
Program4 0 7
Program3 1 0
Program13 0 0
Program7 2 1 161 95 1
Program14 0 0
Program8 0 4
© 2010 The MITRE Corporation. All rights reserved.
■ Constructed Kaplan-Mier survival curves for new risks■ Log-Rank test determines if one population survives longer
than another– Insufficient data to declare closure times are shorter
Risk closures
Page 25
0 20 40 60 80 100 1200
0.2
0.4
0.6
0.8
1
LogRank test for difference in Kaplan-Meier survival function for Program1 between base and test period
Estim
ated
sur
viva
l fun
ctio
ns
Time
Treatment 1Treatment 2Censored
© 2010 The MITRE Corporation. All rights reserved.
■ Seasonal adjustments– Insufficient RiskNav histories to make meaningful adjustments– Traditionally, July-August is a period of reduced work activity
■ We observe an increase in risk-management activity despite this
■ Connection with the Hawthorne effect– This analysis doesn’t focus on specific questions or derived
information – Improvements may be result of general employee feedback in a
prediction market
■ This work was exploratory research, not a randomized clinical trial– We cannot statistically rule out some other (non-prediction market)
effect going on here■ However
– This provides strong direction for future research inquiries– Analogous to passing ‘Phase-0’ trials in FDA approval process
Possible concerns with this analysis
Page 26
© 2010 The MITRE Corporation. All rights reserved.
■ After introducing a prediction market to the USAF…– We observe enhanced risk management practices on programs
with self-generating risk identification and mitigation activity– We do not conclusively observe this effect on programs with
stimulus-driven risk management processes■ There was not enough data to evaluate whether prediction
markets would foster earlier risk identification or faster closure of risks
Conclusions
Page 27
© 2010 The MITRE Corporation. All rights reserved.
Program Analysis DataUpdate Activity and New Risks
Page 28
© 2010 The MITRE Corporation. All rights reserved.
Program1
Page 29
Update Activity New Risks Identified
# Observed Base 246# Observed Test 381# at .05 Level 376p-value .044
# Observed Base 11# Observed Test 17# at .05 Level 17p-value .047
0 100 200 300 400 500 600 700 8000
100
200
300RiskNav activity for Program1 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
200
400
600
30-d
ay s
um
0
500
1000
1500
-Jul-2008-A
ug-2008-S
ep-2008-O
ct-2008-N
ov-2008-D
ec-2008-Jan-2009-Feb-2009-M
ar-2009-A
pr-2009-M
ay-2009-Jun-2009-Jul-2009-A
ug-2009-S
ep-2009-O
ct-2009-N
ov-2009-D
ec-2009-Jan-2010-Feb-2010-M
ar-2010-A
pr-2010-M
ay-2010-Jun-2010-Jul-2010-A
ug-2010
30-d
ay b
oots
trap
0 100 200 300 400 500 600 700 8000
10
20
New Risks for Program1 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
10
20
30-d
ay s
um
0
50
100
-Jul-2008
-Aug-2008
-Sep-2008
-Oct-2008
-Nov-2008
-Dec-2008
-Jan-2009
-Feb-2009
-Mar-2009
-Apr-2009
-May-2009
-Jun-2009
-Jul-2009
-Aug-2009
-Sep-2009
-Oct-2009
-Nov-2009
-Dec-2009
-Jan-2010
-Feb-2010
-Mar-2010
-Apr-2010
-May-2010
-Jun-2010
-Jul-2010
-Aug-2010
30-d
ay b
oots
trap
© 2010 The MITRE Corporation. All rights reserved.
Program2
Page 30
Update Activity New Risks Identified
# Observed Base 258# Observed Test 177# at .05 Level 377p-value .991
# Observed Base 3# Observed Test 8# at .05 Level 6p-value .003
0 100 200 300 400 500 600 700 8000
20
40
60RiskNav activity for Program2 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
100
200
30-d
ay s
um
0
200
400
-Jul-2008-Aug-2008
-Sep-2008-O
ct-2008
-Nov-2008
-Dec-2008
-Jan-2009-Feb-2009
-Mar-2009
-Apr-2009
-May-2009
-Jun-2009-Jul-2009-Aug-2009
-Sep-2009-O
ct-2009
-Nov-2009
-Dec-2009
-Jan-2010-Feb-2010
-Mar-2010
-Apr-2010
-May-2010
-Jun-2010-Jul-2010-Aug-2010
30-d
ay b
oots
trap
0 100 200 300 400 500 600 700 80002468
New Risks for Program2 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
10
20
30-d
ay s
um
0
50-Jul-2008
-Aug-2008
-Sep-2008
-Oct-2008
-Nov-2008
-Dec-2008
-Jan-2009
-Feb-2009
-Mar-2009
-Apr-2009
-May-2009
-Jun-2009
-Jul-2009
-Aug-2009
-Sep-2009
-Oct-2009
-Nov-2009
-Dec-2009
-Jan-2010
-Feb-2010
-Mar-2010
-Apr-2010
-May-2010
-Jun-2010
-Jul-2010
-Aug-2010
30-d
ay b
oots
trap
© 2010 The MITRE Corporation. All rights reserved.
Program4
Page 31
Update Activity New Risks Identified
# Observed Base 100# Observed Test 222# at .05 Level 173p-value .003
# Observed Base 0# Observed Test 5# at .05 Level 0p-value .000
0 100 200 300 400 500 600 700 8000
50
100
150RiskNav activity for Program4 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
200
400
30-d
ay s
um
0
500
1000
-Jul-2008-A
ug-2008-S
ep-2008-O
ct-2008-N
ov-2008-D
ec-2008-Jan-2009-Feb-2009-M
ar-2009-A
pr-2009-M
ay-2009-Jun-2009-Jul-2009-A
ug-2009-S
ep-2009-O
ct-2009-N
ov-2009-D
ec-2009-Jan-2010-Feb-2010-M
ar-2010-A
pr-2010-M
ay-2010-Jun-2010-Jul-2010-A
ug-2010
30-d
ay b
oots
trap
0 100 200 300 400 500 600 700 8000
5
10
New Risks for Program4 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
10
20
30-d
ay s
um
0
20
40
60-Jul-2008
-Aug-2008
-Sep-2008
-Oct-2008
-Nov-2008
-Dec-2008
-Jan-2009
-Feb-2009
-Mar-2009
-Apr-2009
-May-2009
-Jun-2009
-Jul-2009
-Aug-2009
-Sep-2009
-Oct-2009
-Nov-2009
-Dec-2009
-Jan-2010
-Feb-2010
-Mar-2010
-Apr-2010
-May-2010
-Jun-2010
-Jul-2010
-Aug-2010
30-d
ay b
oots
trap
© 2010 The MITRE Corporation. All rights reserved.
Program5
Page 32
Update Activity New Risks Identified
# Observed Base 24# Observed Test 8# at .05 Level 52p-value .839
# Observed Base 1# Observed Test 0# at .05 Level 3p-value ..636
0 100 200 300 400 500 600 700 8000
10
20
RiskNav activity for Program5 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
20
40
60
30-d
ay s
um
0
50
100
150
-Jul-2008-A
ug-2008-S
ep-2008-O
ct-2008-N
ov-2008-D
ec-2008-Jan-2009-Feb-2009-M
ar-2009-A
pr-2009-M
ay-2009-Jun-2009-Jul-2009-A
ug-2009-S
ep-2009-O
ct-2009-N
ov-2009-D
ec-2009-Jan-2010-Feb-2010-M
ar-2010-A
pr-2010-M
ay-2010-Jun-2010-Jul-2010-A
ug-2010
30-d
ay b
oots
trap
0 100 200 300 400 500 600 700 8000
1
2
3
New Risks for Program5 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
2
4
30-d
ay s
um
0
5
10-Jul-2008
-Aug-2008
-Sep-2008
-Oct-2008
-Nov-2008
-Dec-2008
-Jan-2009
-Feb-2009
-Mar-2009
-Apr-2009
-May-2009
-Jun-2009
-Jul-2009
-Aug-2009
-Sep-2009
-Oct-2009
-Nov-2009
-Dec-2009
-Jan-2010
-Feb-2010
-Mar-2010
-Apr-2010
-May-2010
-Jun-2010
-Jul-2010
-Aug-2010
30-d
ay b
oots
trap
© 2010 The MITRE Corporation. All rights reserved.
Program6
Page 33
Update Activity New Risks Identified
# Observed Base 8# Observed Test 21# at .05 Level 20p-value .027
# Observed Base 0# Observed Test 0# at .05 Level 0p-value
0 100 200 300 400 500 600 700 8000
10
20
30RiskNav activity for Program6 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
50
100
30-d
ay s
um
0
100
200
-Jul-2008
-Aug-2008-Sep-2008
-Oct-2008
-Nov-2008
-Dec-2008
-Jan-2009-Feb-2009
-Mar-2009
-Apr-2009-M
ay-2009
-Jun-2009-Jul-2009
-Aug-2009
-Sep-2009-O
ct-2009
-Nov-2009
-Dec-2009
-Jan-2010-Feb-2010-M
ar-2010
-Apr-2010-M
ay-2010
-Jun-2010
-Jul-2010-Aug-2010
30-d
ay b
oots
trap
0 100 200 300 400 500 600 700 8000
2
4
New Risks for Program6 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
5
10
30-d
ay s
um
0
10
20-Jul-2008
-Aug-2008
-Sep-2008
-Oct-2008
-Nov-2008
-Dec-2008
-Jan-2009
-Feb-2009
-Mar-2009
-Apr-2009
-May-2009
-Jun-2009
-Jul-2009
-Aug-2009
-Sep-2009
-Oct-2009
-Nov-2009
-Dec-2009
-Jan-2010
-Feb-2010
-Mar-2010
-Apr-2010
-May-2010
-Jun-2010
-Jul-2010
-Aug-2010
30-d
ay b
oots
trap
© 2010 The MITRE Corporation. All rights reserved.
Program7
Page 34
Update Activity New Risks Identified
# Observed Base 63# Observed Test 56# at .05 Level 132p-value .541
# Observed Base 3# Observed Test 1# at .05 Level 7p-value .747
0 100 200 300 400 500 600 700 8000
20
40
RiskNav activity for Program7 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
20
40
60
30-d
ay s
um
0
100
200
300
-Jul-2008-Aug-2008
-Sep-2008-O
ct-2008
-Nov-2008
-Dec-2008
-Jan-2009-Feb-2009
-Mar-2009
-Apr-2009
-May-2009
-Jun-2009-Jul-2009-Aug-2009
-Sep-2009-O
ct-2009
-Nov-2009
-Dec-2009
-Jan-2010-Feb-2010
-Mar-2010
-Apr-2010
-May-2010
-Jun-2010-Jul-2010-Aug-2010
30-d
ay b
oots
trap
0 100 200 300 400 500 600 700 8000
2
4
New Risks for Program7 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
2
4
30-d
ay s
um
0
10
20-Jul-2008
-Aug-2008
-Sep-2008
-Oct-2008
-Nov-2008
-Dec-2008
-Jan-2009
-Feb-2009
-Mar-2009
-Apr-2009
-May-2009
-Jun-2009
-Jul-2009
-Aug-2009
-Sep-2009
-Oct-2009
-Nov-2009
-Dec-2009
-Jan-2010
-Feb-2010
-Mar-2010
-Apr-2010
-May-2010
-Jun-2010
-Jul-2010
-Aug-2010
30-d
ay b
oots
trap
© 2010 The MITRE Corporation. All rights reserved.
Program8
Page 35
Update Activity New Risks Identified
# Observed Base 10# Observed Test 30# at .05 Level 24p-value .012
# Observed Base 0# Observed Test 2# at .05 Level 0p-value .000
0 100 200 300 400 500 600 700 8000
20
40
60
RiskNav activity for Program8 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
50
100
30-d
ay s
um
0
200
400
-Jul-2008-Aug-2008
-Sep-2008-O
ct-2008
-Nov-2008
-Dec-2008
-Jan-2009-Feb-2009
-Mar-2009
-Apr-2009
-May-2009
-Jun-2009-Jul-2009-Aug-2009
-Sep-2009-O
ct-2009
-Nov-2009
-Dec-2009
-Jan-2010-Feb-2010
-Mar-2010
-Apr-2010
-May-2010
-Jun-2010-Jul-2010-Aug-2010
30-d
ay b
oots
trap
0 100 200 300 400 500 600 700 8000
5
10
New Risks for Program8 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
10
20
30-d
ay s
um
020406080
-Jul-2008
-Aug-2008
-Sep-2008
-Oct-2008
-Nov-2008
-Dec-2008
-Jan-2009
-Feb-2009
-Mar-2009
-Apr-2009
-May-2009
-Jun-2009
-Jul-2009
-Aug-2009
-Sep-2009
-Oct-2009
-Nov-2009
-Dec-2009
-Jan-2010
-Feb-2010
-Mar-2010
-Apr-2010
-May-2010
-Jun-2010
-Jul-2010
-Aug-2010
30-d
ay b
oots
trap
© 2010 The MITRE Corporation. All rights reserved.
Program9
Page 36
Update Activity New Risks Identified
# Observed Base 0# Observed Test 0# at .05 Level 0p-value
# Observed Base 0# Observed Test 0# at .05 Level 0p-value
0 100 200 300 400 500 600 700 8000
5
10
RiskNav activity for Program9 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
10
20
30-d
ay s
um
0
20
40
60
-Jul-2008-Aug-2008
-Sep-2008-O
ct-2008
-Nov-2008
-Dec-2008
-Jan-2009-Feb-2009
-Mar-2009
-Apr-2009
-May-2009
-Jun-2009-Jul-2009-Aug-2009
-Sep-2009-O
ct-2009
-Nov-2009
-Dec-2009
-Jan-2010-Feb-2010
-Mar-2010
-Apr-2010
-May-2010
-Jun-2010-Jul-2010-Aug-2010
30-d
ay b
oots
trap
0 100 200 300 400 500 600 700 8000
2
4
New Risks for Program9 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
2
4
30-d
ay s
um
0
10
20-Jul-2008
-Aug-2008
-Sep-2008
-Oct-2008
-Nov-2008
-Dec-2008
-Jan-2009
-Feb-2009
-Mar-2009
-Apr-2009
-May-2009
-Jun-2009
-Jul-2009
-Aug-2009
-Sep-2009
-Oct-2009
-Nov-2009
-Dec-2009
-Jan-2010
-Feb-2010
-Mar-2010
-Apr-2010
-May-2010
-Jun-2010
-Jul-2010
-Aug-2010
30-d
ay b
oots
trap
© 2010 The MITRE Corporation. All rights reserved.
Program10
Page 37
Update Activity New Risks Identified
# Observed Base 0# Observed Test 20# at .05 Level 0p-value .000
# Observed Base 0# Observed Test 3# at .05 Level 0p-value .000
0 100 200 300 400 500 600 700 8000
10
20
RiskNav activity for Program10 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
10
20
30-d
ay s
um
0
50
-Jul-2008-Aug-2008
-Sep-2008-O
ct-2008
-Nov-2008
-Dec-2008
-Jan-2009-Feb-2009
-Mar-2009
-Apr-2009
-May-2009
-Jun-2009-Jul-2009-Aug-2009
-Sep-2009-O
ct-2009
-Nov-2009
-Dec-2009
-Jan-2010-Feb-2010
-Mar-2010
-Apr-2010
-May-2010
-Jun-2010-Jul-2010-Aug-2010
30-d
ay b
oots
trap
0 100 200 300 400 500 600 700 8000
1
2
3
New Risks for Program10 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
1
2
3
30-d
ay s
um
0
5
10
15-Jul-2008
-Aug-2008
-Sep-2008
-Oct-2008
-Nov-2008
-Dec-2008
-Jan-2009
-Feb-2009
-Mar-2009
-Apr-2009
-May-2009
-Jun-2009
-Jul-2009
-Aug-2009
-Sep-2009
-Oct-2009
-Nov-2009
-Dec-2009
-Jan-2010
-Feb-2010
-Mar-2010
-Apr-2010
-May-2010
-Jun-2010
-Jul-2010
-Aug-2010
30-d
ay b
oots
trap
© 2010 The MITRE Corporation. All rights reserved.
Program11
Page 38
Update Activity New Risks Identified
# Observed Base 49# Observed Test 1# at .05 Level 117p-value .967
# Observed Base 3# Observed Test 0# at .05 Level 7p-value .870
0 100 200 300 400 500 600 700 8000
20
40
RiskNav activity for Program11 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
50
100
30-d
ay s
um
0
100
200
300
-Jul-2008-Aug-2008
-Sep-2008-O
ct-2008
-Nov-2008
-Dec-2008
-Jan-2009-Feb-2009
-Mar-2009
-Apr-2009
-May-2009
-Jun-2009-Jul-2009-Aug-2009
-Sep-2009-O
ct-2009
-Nov-2009
-Dec-2009
-Jan-2010-Feb-2010
-Mar-2010
-Apr-2010
-May-2010
-Jun-2010-Jul-2010-Aug-2010
30-d
ay b
oots
trap
0 100 200 300 400 500 600 700 8000
1
2
3
New Risks for Program11 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
2
4
6
30-d
ay s
um
0
10
20
-Jul-2008
-Aug-2008
-Sep-2008
-Oct-2008
-Nov-2008
-Dec-2008
-Jan-2009
-Feb-2009
-Mar-2009
-Apr-2009
-May-2009
-Jun-2009
-Jul-2009
-Aug-2009
-Sep-2009
-Oct-2009
-Nov-2009
-Dec-2009
-Jan-2010
-Feb-2010
-Mar-2010
-Apr-2010
-May-2010
-Jun-2010
-Jul-2010
-Aug-2010
30-d
ay b
oots
trap
© 2010 The MITRE Corporation. All rights reserved.
Program12
Page 39
Update Activity New Risks Identified
# Observed Base 0# Observed Test 0# at .05 Level 0p-value
# Observed Base 0# Observed Test 0# at .05 Level 0p-value
0 100 200 300 400 500 600 700 8000
20406080
RiskNav activity for Program12 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
50
100
30-d
ay s
um
0
200
400
-Jul-2008-Aug-2008
-Sep-2008-O
ct-2008
-Nov-2008
-Dec-2008
-Jan-2009-Feb-2009
-Mar-2009
-Apr-2009
-May-2009
-Jun-2009-Jul-2009-Aug-2009
-Sep-2009-O
ct-2009
-Nov-2009
-Dec-2009
-Jan-2010-Feb-2010
-Mar-2010
-Apr-2010
-May-2010
-Jun-2010-Jul-2010-Aug-2010
30-d
ay b
oots
trap
0 100 200 300 400 500 600 700 8000
2
4
6
New Risks for Program12 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
2
4
6
30-d
ay s
um
0
10
20
30-Jul-2008
-Aug-2008
-Sep-2008
-Oct-2008
-Nov-2008
-Dec-2008
-Jan-2009
-Feb-2009
-Mar-2009
-Apr-2009
-May-2009
-Jun-2009
-Jul-2009
-Aug-2009
-Sep-2009
-Oct-2009
-Nov-2009
-Dec-2009
-Jan-2010
-Feb-2010
-Mar-2010
-Apr-2010
-May-2010
-Jun-2010
-Jul-2010
-Aug-2010
30-d
ay b
oots
trap
© 2010 The MITRE Corporation. All rights reserved.
Program13
Page 40
Update Activity New Risks Identified
# Observed Base 0# Observed Test 0# at .05 Level 0p-value
# Observed Base 0# Observed Test 0# at .05 Level 0p-value
0 100 200 300 400 500 600 700 8000
50
100
150RiskNav activity for Program13 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
200
400
30-d
ay s
um
0200400600800
-Jul-2008-Aug-2008
-Sep-2008-O
ct-2008
-Nov-2008
-Dec-2008
-Jan-2009-Feb-2009
-Mar-2009
-Apr-2009
-May-2009
-Jun-2009-Jul-2009-Aug-2009
-Sep-2009-O
ct-2009
-Nov-2009
-Dec-2009
-Jan-2010-Feb-2010
-Mar-2010
-Apr-2010
-May-2010
-Jun-2010-Jul-2010-Aug-2010
30-d
ay b
oots
trap
0 100 200 300 400 500 600 700 8000
10
20
New Risks for Program13 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
20
40
30-d
ay s
um
0
50-Jul-2008
-Aug-2008
-Sep-2008
-Oct-2008
-Nov-2008
-Dec-2008
-Jan-2009
-Feb-2009
-Mar-2009
-Apr-2009
-May-2009
-Jun-2009
-Jul-2009
-Aug-2009
-Sep-2009
-Oct-2009
-Nov-2009
-Dec-2009
-Jan-2010
-Feb-2010
-Mar-2010
-Apr-2010
-May-2010
-Jun-2010
-Jul-2010
-Aug-2010
30-d
ay b
oots
trap
© 2010 The MITRE Corporation. All rights reserved.
Program14
Page 41
Update Activity New Risks Identified
# Observed Base 0# Observed Test 0# at .05 Level 0p-value
# Observed Base 0# Observed Test 0# at .05 Level 0p-value
0 100 200 300 400 500 600 700 8000
5
10
15
RiskNav activity for Program14 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
5
10
15
30-d
ay s
um
0
50
-Jul-2008-Aug-2008
-Sep-2008-O
ct-2008
-Nov-2008
-Dec-2008
-Jan-2009-Feb-2009
-Mar-2009
-Apr-2009
-May-2009
-Jun-2009-Jul-2009-Aug-2009
-Sep-2009-O
ct-2009
-Nov-2009
-Dec-2009
-Jan-2010-Feb-2010
-Mar-2010
-Apr-2010
-May-2010
-Jun-2010-Jul-2010-Aug-2010
30-d
ay b
oots
trap
0 100 200 300 400 500 600 700 8000
2
4
New Risks for Program14 from 09-Jun-2008 to 01-Sep-2010
daily
0 100 200 300 400 500 600 700 8000
1
2
30-d
ay s
um
0
5
10
-Jul-2008
-Aug-2008
-Sep-2008
-Oct-2008
-Nov-2008
-Dec-2008
-Jan-2009
-Feb-2009
-Mar-2009
-Apr-2009
-May-2009
-Jun-2009
-Jul-2009
-Aug-2009
-Sep-2009
-Oct-2009
-Nov-2009
-Dec-2009
-Jan-2010
-Feb-2010
-Mar-2010
-Apr-2010
-May-2010
-Jun-2010
-Jul-2010
-Aug-2010
30-d
ay b
oots
trap
