Upload
murray-cantor
View
62
Download
0
Embed Size (px)
Citation preview
Sa
fe a
rea
– n
o g
rap
hic
s h
ere
Sa
fe a
rea
– n
o g
rap
hics h
ere
Steer with Predictive AnalyticsLess Surprises, More Insight
Walker Royce Murray CantorChief Software Economist Distinguished EngineerIBM Software Group IBM Software Group
@weroyceusibmcom @murraycantor
[email protected] [email protected]
1
Sa
fe a
rea
– n
o g
rap
hic
s h
ere
Sa
fe a
rea
– n
o g
rap
hics h
ere
2
Deliver business critical software product release within 12 months
Challenge
SCHEDULE
6 MONTHS 12 MONTHS 18 MONTHS
Sa
fe a
rea
– n
o g
rap
hic
s h
ere
Sa
fe a
rea
– n
o g
rap
hics h
ere
Jerry’s probable outcome…Reality
3
LATE SCRAP & REWORK
SCHEDULE
6 MONTHS 12 MONTHS 18 MONTHS
0%
100%
PR
OG
RE
SS
INTEGRATION BEGINS
Sa
fe a
rea
– n
o g
rap
hic
s h
ere
Sa
fe a
rea
– n
o g
rap
hics h
ere
4
Program parameters are also delivery predictionsCOST, SCHEDULE, EFFORT, QUALITY…
6 MONTHS 12 MONTHS 18 MONTHS
PR
OB
AB
ILIT
Y
Insight
SCHEDULE
Sa
fe a
rea
– n
o g
rap
hic
s h
ere
Sa
fe a
rea
– n
o g
rap
hics h
ere
5
There is roughly a 50% chance of making the date
6 MONTHS 12 MONTHS 18 MONTHS
PR
OB
AB
ILIT
Y
Coin Flip
SCHEDULE
Sa
fe a
rea
– n
o g
rap
hic
s h
ere
Sa
fe a
rea
– n
o g
rap
hics h
ere
6
Move out the date to improve likelihood of shipping?
6 MONTHS 12 MONTHS 18 MONTHS
PR
OB
AB
ILIT
Y
Option 1Absolutely Absolutely NOT!NOT!
SCHEDULE
Sa
fe a
rea
– n
o g
rap
hic
s h
ere
Sa
fe a
rea
– n
o g
rap
hics h
ere
7
Decrease time estimate by sacrificing quality or content?
6 MONTHS 12 MONTHS 18 MONTHS
PR
OB
AB
ILIT
Y
Option 2Absolutely Absolutely NOT!NOT!
SCHEDULE
Sa
fe a
rea
– n
o g
rap
hic
s h
ere
Sa
fe a
rea
– n
o g
rap
hics h
ere
8
Reduce the variance to improve likelihood of shipping
Optimize
6 MONTHS 12 MONTHS 18 MONTHS
PR
OB
AB
ILIT
Y
SCHEDULE
Sa
fe a
rea
– n
o g
rap
hic
s h
ere
Sa
fe a
rea
– n
o g
rap
hics h
ere
9
Measure validated learning | Reduce the variance
Uncertainty in stakeholder satisfaction
Optimize
Uncertainty in Plans, Scope and Design
Sa
fe a
rea
– n
o g
rap
hic
s h
ere
Sa
fe a
rea
– n
o g
rap
hics h
ere
10
Economic Governance
The moral of this story…
Variance Reduction
Smarter Analytics
Sa
fe a
rea
– n
o g
rap
hic
s h
ere
Sa
fe a
rea
– n
o g
rap
hics h
ere
A (Fictional) Case Study The Setting: Product planning for KillerApp V2 for US
Treasury property auctions
– KillerApp v1 met with unprecedented success
– Since its release and success, many state governments have indicated interest in variants specific to their local laws and needs
The Opportunity
– Sales reports that they can close $1B in new business with a promise for features supporting state government auctions, including mobile, analytics, and social media support
The Catch
– An aggressive startup has plans to release a similar capacity in August, 2014
• The first to market will have a huge competitive advantage
Sa
fe a
rea
– n
o g
rap
hic
s h
ere
Sa
fe a
rea
– n
o g
rap
hics h
ere
KillerApp’s Product Development Team Structure
Alice, Team Lead
Peter, Product Manager
Joe, Team Lead
12
• The server team develops using hybrid process• Agile with a target date
• They must manage technical risks carefully to meet challenging delivery target
• Is their planned scope of work appropriate to their timeline, resources?
Sa
fe a
rea
– n
o g
rap
hic
s h
ere
Sa
fe a
rea
– n
o g
rap
hics h
ere
The server team has identified work they would like to complete, the task owners, and preliminary estimates
Analytics capability
Story 1Mobile support
Story 2
Owner: John
Task 1.1
[1w,2w,3w]
Owner:John
Task 1.2
[2w,5w,7w]
Owner: Alice
Task 1.3
[3w,7w,11w]
Owner: John
Task 2.1
[1w,1.5w,3w]
Owner: Claire
Task 2.2
[2d,1w,6w]
Owner: Claire
Task 2.3
[1w,2w,3w]
The team needs to finish its work by the integration milestone.Is it a good bet that it will?
Task 1.1.7
2d
5d
6w
Sa
fe a
rea
– n
o g
rap
hic
s h
ere
Sa
fe a
rea
– n
o g
rap
hics h
ere
Is the server team making a good “bet”? The conversation begins...
Where did these estimates come from?
– Should we bet the business on this data?
– Did tampering occur?
User expertise
– One developer’s experience
– Multiple developers’ experiences
Learning from prior history
– This team’s prior history (Machine Learning)
– Models that run over prior projects (e.g., COCOMO, SLIM, SEER)
Peter, Product Manager
Alice, Team Lead
Predictive analytics facilitate critical conversations between stakeholders.To have value, they must be trustworthy and explainable.
Sa
fe a
rea
– n
o g
rap
hic
s h
ere
Sa
fe a
rea
– n
o g
rap
hics h
ere
Is the server team making a good “bet”? The conversation continues...
Does it reflect realistic expectations?
– What is causing the variance (uncertainty)? What actions can we take to reduce it?
– Did you account for new work being identified in flight?
Variance is coming from
– Estimating large-scale pieces of work
– Uncertainty around reusability of components
15
Peter, Product Manager
Alice, Team Lead
Predictive analytics have value only if the data have value.They are not magic. They require hard work to provide value.
Sa
fe a
rea
– n
o g
rap
hic
s h
ere
Sa
fe a
rea
– n
o g
rap
hics h
ere
Once we trust the data, we can ask what it says about the outcome. But there are many different possible schedules,
leading to different outcomes Valid schedules come from
– Task dependencies
– Implied orderings due to owners
Actual schedules depend on when owners finish tasks (uncertain)
– NB: This is not critical path analysis
Schedule 1
John
Claire
AliceTask 1.1[1w,2w,3w]
Task 1.2[2w,5w,7w]
Task 2.1[1w,1.5w,3w]
Task 2.3[1w,2w,3w]
Task 2.2[2d,1w,6w]
Task 1.3[3w,7w,11w]
Schedule 2
John
Claire
AliceTask 1.1[1w,2w,3w]
Task 1.2[2w,5w,7w]
Task 2.1[1w,1.5w,3w]Task 2.3[1w,2w,3w]
Task 2.2[2d,1w,6w]
Task 1.3[3w,7w,11w]
Sa
fe a
rea
– n
o g
rap
hic
s h
ere
Sa
fe a
rea
– n
o g
rap
hics h
ere
There are many different possible outcomes, depending on the actual schedule and the time the tasks take
Schedule 1
John
Claire
AliceTask 1.1[1w,2w,3w]
Task 1.2[2w,5w,7w]
Task 2.1[1w,1.5w,3w]
Task 2.3[1w,2w,3w]
Task 2.2[2d,1w,6w]
Task 1.3[3w,7w,11w]
Schedule 2
John
Claire
AliceTask 1.1[1w,2w,3w]
Task 1.2[2w,5w,7w]
Task 2.1[1w,1.5w,3w]Task 2.3[1w,2w,3w]
Task 2.2[2d,1w,6w]
Task 1.3[3w,7w,11w]
What are the possible outcomes?How likely are they?
Is this set of work a good bet?
Sa
fe a
rea
– n
o g
rap
hic
s h
ere
Sa
fe a
rea
– n
o g
rap
hics h
ere
Is the server team likely to finish their planned work by 2/13?
[3w,7w,11w]Task 1.3
Alice
John
Claire
[2w,5w,7w]Task 1.2
[1w,1.5w,3w]Task 2.1
[2w,5w,7w]Task 2.2
[1w,2w,3w]Task 1.1
[1w,2w,3w]Task 2.3
Sim 1 8d
Task 2.1
10d
Task 2.3
10d
Task 1.1
10d
Task 2.2
35d
Task 1.3
25d
Task 1.260d
Sim 2 5d
Task 2.1
5d
Task 2.3
5d
Task 1.1
14d
Task 2.2
15d
Task 1.3
10d
Task 1.254d
Sim 3 15d
Task 2.1
15d
Task 2.3
15d
Task 1.1
35d
Task 2.2
55d
Task 1.3
35d
Task 1.2170d
10/1 1/1512/1
38% 62%
For each random variable (task duration), sample a value from its
distribution
Different values will be sampled in each simulation, leading to different outcomes
Sa
fe a
rea
– n
o g
rap
hic
s h
ere
Sa
fe a
rea
– n
o g
rap
hics h
ere
Server team has been working for a month. Will they deliver on time?
2/13Delivery
8/1Start
9/1Now
Pre
dic
ted
Del
ive
ryD
ate
Ra
ng
e
Date of Prediction
2/13
1/1
3/1
2/1
4/1• Best case: 1/1• Most likely: 2/11• Worst case: 2/28
• On-time delivery: 87%
• Best case: ?• Most likely: ?• Worst case: ?
• On-time delivery: ?
19
We have new learning: actuals, evidence of how the team is actually performing
Sa
fe a
rea
– n
o g
rap
hic
s h
ere
Sa
fe a
rea
– n
o g
rap
hics h
ere
Data mining can help us learn from the past to predict the future
Data mining (aka machine learning, statistical inference) is about learning correlations between inputs and outcomes from sample data (actuals)
– Classification prediction seeks to learn a model that predicts the classification of artifacts
– Numeric prediction seeks to learn a model that predicts a numeric value for an artifact of interest
20
Sa
fe a
rea
– n
o g
rap
hic
s h
ere
Sa
fe a
rea
– n
o g
rap
hics h
ere
Techniques for Learning From Actuals1. Leverage actual distribution
- The actuals form a distribution
- We can sample from this distribution during Monte Carlo simulation
2. Predicting based on relationships between estimates and actuals (regression)
– If estimates and actuals are strongly correlated, regression analysis can be useful in predicting actuals from estimates
3. Predicting based on learned relationships between multiple variables and actuals
4. Bayesian Networks
- After 50 tasks complete, we have new evidence:
• DB tasks: effort = [11d, 16d, 42d], type = 20%
• Analytics tasks: effort = [4d, 16d, 20d], type = 75%
• Mobile tasks: effort = [1d, 2d, 4d], type = 5%
- We can update the conditional probabilities to reflect this learned evidence
21P
rob
abil
ity
Actual Task Effort
Node 0n: 4591; %: 100.0Predicted: 5.5 E8
Node 0n: 4591; %: 100.0Predicted: 5.5 E8
Effort
Node 1n: 1911; %: 41.6
Predicted: 5.0 E8
Node 1n: 1911; %: 41.6
Predicted: 5.0 E8
Node 2n: 2680; %: 58.4
Predicted: 5.9 E8
Node 2n: 2680; %: 58.4
Predicted: 5.9 E8
Attribute: OwnerImprovement: 1.8 E15
Alice; David John; Claire
Node 3n: 832; %: 31.1
Predicted: 5.6 E8
Node 3n: 832; %: 31.1
Predicted: 5.6 E8
Node 4n: 1848; %: 68.9
Predicted: 6.1 E8
Node 4n: 1848; %: 68.9
Predicted: 6.1 E8
Attribute: OwnerImprovement: 2.7 E14
Claire John
Task duration
Task duration
Task owner
Task owner
Task type
Task type
Task expected
effort
Task expected
effort
Qualification for task
Qualification for task
Sa
fe a
rea
– n
o g
rap
hic
s h
ere
Sa
fe a
rea
– n
o g
rap
hics h
ere
Server team has been working for a month. It is not going well for them.
It is no longer a good bet that the server team will deliver the committed scope on time
2/13Delivery
8/1Start
9/1Now
Pred
icte
d D
eliv
ery
Dat
e Ra
nge
Date of Prediction
2/13
1/1
3/1
2/1
4/1 • Best case: 1/1• Most likely: 2/11• Worst case: 2/28
• On-time delivery: 87%
• Best case: 1/17• Most likely: 2/23• Worst case: 4/7
• On-time delivery: 43%
Sa
fe a
rea
– n
o g
rap
hic
s h
ere
Sa
fe a
rea
– n
o g
rap
hics h
ere
Lessons for us all: Making better bets
Eliminate some of the long, red tail
– Understand and remediate sources of uncertainty that are putting the deadline at risk
Change the scope
– Drop or postpone work
– Never negotiate estimates!
Change the resources available
– Beware Brooks’ Law!
Change the time
– Negotiate a later deadline
Make changes, rerun Monte Carlo simulation,check results
– Repeat until the outcome is acceptable10/1 12/1812/1
87% 13%
Task 1.1.7
2d
5d
6w
• Break down the task further• Do some work now to eliminate variance-inducing
questions