Upload
pamela-stroud
View
14
Download
1
Embed Size (px)
Citation preview
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 1
Copy r ight © 2016, SAS Insti tute I nc . All r i ghts r eserved.
CHANGING THE PARADIGM FOR
BUSINESS FORECASTING
FORESIGHT PRACTITIONER CONFERENCE 2016
Copy r ight © 2016, SAS Insti tute I nc . All r i ghts r eserved.
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 2
Copy r ight © 2016, SAS Insti tute I nc . All r i ghts r eserved.
Paradigms organize our perceptions…
…and make them understandable
Copy r ight © 2016, SAS Insti tute I nc . All r i ghts r eserved.
Normal Science
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 3
Copy r ight © 2016, SAS Insti tute I nc . All r i ghts r eserved.
THE GEOCENTRIC
PARADIGM
Copy r ight © 2016, SAS Insti tute I nc . All r i ghts r eserved.
EPICYCLES ACCOUNT FOR
RETROGRADE MOTION
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 4
Copy r ight © 2016, SAS Insti tute I nc . All r i ghts r eserved.
The Current Paradigm for
Business Forecasting
Copy r ight © 2016, SAS Insti tute I nc . All r i ghts r eserved.
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 5
Copy r ight © 2016, SAS Insti tute I nc . All r i ghts r eserved.
CHARACTERISTICS OF THE
“OFFENSIVE” PARADIGM
• More is better
• More data
• More computational power
• More complex forecasting models incorporating more
variables
• More elaborate collaborative processes
Copy r ight © 2016, SAS Insti tute I nc . All r i ghts r eserved.
The Paradigm Limits
What You See
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 6
Copy r ight © 2016, SAS Insti tute I nc . All r i ghts r eserved.
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 7
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
Anomalies:
The Beginning of a Crisis
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
The Crisis
in Business Forecasting
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 8
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
HIGH ON COMPLEXITY
Paul Goodwin, “High on Complexity, Low on Evidence: Are
Advanced Forecasting Methods Always as Good as They
Seem?” Foresight, Fall 2011.
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
HIGH ON COMPLEXITY
• Analytical Network Process
• Seasonal Hybrid Procedure
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 9
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
Is Complexity Bad?
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
Kesten Green and Scott Armstrong, “Simple versus
Complex Forecasting: The Evidence.” Journal of
Business Research 68 (2015)
SIMPLE VS. COMPLEX
FORECASTING
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 10
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
Simple Forecasting ≡
Processes that are understandable to forecast users
(Forecasting Simplicity Questionnaire at simple-forecasting.com)
SIMPLE VS. COMPLEX
FORECASTING
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
SIMPLE VS. COMPLEX
FORECASTING
• Review of 32 papers, reporting on 97 comparative studies
None of the papers provides a balance of evidence that
complexity improves forecast accuracy.
Remarkably, no matter what type of forecasting method is
used, complexity harms accuracy.
…the need for complexity has not arisen.
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 11
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
SIMPLE VS. COMPLEX
FORECASTING
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
Implications for the Offensive
Paradigm
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 12
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
Changing the Paradigm for
Business Forecasting
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
Why the Attraction for the
Offensive Paradigm?
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 13
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
WHY THE ATTRACTION?
• Forecasters’ clients may be reassured by
incomprehensibility
• Resistance to simple methods
• Complexity is often persuasive
• Researchers are rewarded for publishing in highly ranked
journals which favor complexity
• Forecasters can use complex methods to provide
forecasts that support decision makers’ plans
• Can add complexity to a model to better fit the history
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
“BEST FIT” MODEL SELECTION
Confusing “fit to history” with “appropriateness for forecasting”
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 14
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
The New Paradigm
for Business Forecasting
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
The “Defensive” Paradigm
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 15
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
Role of the Naïve Model
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
The Objective
To generate forecasts as
accurate as can reasonably be
expected…and to do this as
efficiently as possible
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 16
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
Your Job:
Forecast Heads or Tails in the
daily flip of a fair coin
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
The Means
By identifying and eliminating
forecasting process waste
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 17
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
THE 52%
• Series of enlightening disturbing articles published in
Foresight by Steve Morlidge
Studied 8 supply chain companies
300,000 real life forecasts
52% were less accurate than the no-change forecast!
“Forecast Errors and Their Avoidability” (Summer 2013)
“Do Forecasting Methods Reduce Avoidable Error” (Winter 2014)
“Forecast Quality in the Supply Chain” (Spring 2014)
“Using Relative Error Metrics to Improve Forecast Quality in the Supply Chain” (Summer
2014)
“Measuring the Quality of Intermittent Demand Forecasts (Spring 2015)
“A Better Way to Assess the Quality of Demand Forecasts” (Summer 2015)
“Using Error Analysis to Improve Forecast Performance (Spring 2016)
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
Forecast Value Added
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 18
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
Forecast Value Added ≡
The change in a forecasting performance metric
that can be attributed to a particular step or
participant in the forecasting process
DEFINITION OF
FORECAST VALUE ADDED
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
Theil’s U =
RMSE / RMSE of naïve model
• The closer Theil’s U is to 0, the better the model
• Theil’s U < 1.0 indicates value added
• Theil’s U > 1.0 indicates making the forecast worse
RELATIVE ERROR METRICS
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 19
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
Relative Absolute Error (RAE) =
| forecast error | / | naïve forecast error |
• RAE closer to 0 is better
• RAE < 1 means positive FVA “adding value”
• RAE > 1 means negative FVA
RELATIVE ERROR METRICS
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
RAE ~ 0.5 is “best case” forecast error you can
expect to achieve
RAE > 0.5 is “avoidable error”
RELATIVE ERROR METRICS
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 20
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
TYPICAL BUSINESS
FORECASTING PROCESS
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
FAILINGS OF TRADITIONAL
METRICS
• Dozens of forecasting performance metrics available
Some flavor of MAPE is the most commonly used
• Traditional metrics like MAD or MAPE tell you the size of
your forecast error
• But the traditional metrics by themselves are not
sufficient for properly evaluating performance:
They do not account for underlying “forecastability”
They do not indicate what error you should be able to achieve
They do not measure the efficiency of your process
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 21
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
WHAT IS FVA ANALYSIS?
• The application of traditional scientific method to
forecasting
H0: Your forecasting process has no effect
• FVA Analysis attempts to determine whether
forecasting process steps and participants are
improving the forecast – or just making it worse
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
NAÏVE FORECAST AS A PLACEBO
• Naïve forecast serves as the placebo in
evaluating forecasting process performance
Provides a reference standard for comparisons
Is the forecasting process “adding value” by
performing better than the placebo?
Analogy: Evaluating a new drug by comparing
to a control group (receiving a placebo)
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 22
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
FVA ANALYSIS: SIMPLE EXAMPLE
• Consider a very simple forecasting process:
Override Forecast
Statistical Forecast
Forecasting Software
Data
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
FVA ANALYSIS: SIMPLE EXAMPLE
FVA Analysis compares the performance of the statistical forecast to the performance of the analyst’s override forecast
FVA Analysis also compares both to a “naïve” forecast
Override Forecast
Naive Forecast
Naive Model
DataForecasting
SoftwareStatistical Forecast
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 23
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
FVA “STAIRSTEP” REPORT
• Can report on an individual time series, or for an
aggregation of many (or all) time series
If you are doing better than a naïve forecast, your process is
“adding value”
If you are doing worse than a naïve forecast, then you are
simply wasting time and resources
Process
Step
Forecast
Accuracy
FVA vs.
Naïve
FVA vs.
Statistical
Naïve
Forecast60% - -
Statistical
Forecast65% 5% -
Analyst
Override62% 2% -3%
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
ACADEMIC RESEARCH
Source: Robert Fildes and Paul Goodwin, “Good and Bad Judgment in
Forecasting.” Foresight, Fall 2007.
Improvement in Accuracy by Size of
Adjustment (at one company)
-5%
0%
5%
10%
15%
20%
25%
Quartile1 Quartile2 Quartile3 Quartile4
Size of Override
% I
mp
rovem
en
t Positive Override
Negative Override
Studied 60,000 forecasts at four supply chain companies
75% of statistical forecasts were manually adjusted
Large adjustments tended to be beneficial
Small adjustments did not significantly improve accuracy and sometimes made the forecast worse
Downward adjustments were more likely to improve the forecast than upward adjustments
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 24
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
Identifying Waste in the
Forecasting Process
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
Identify and eliminate worst practices
Research Agenda Under the
Defensive Paradigm
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 25
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
The Confessions
Worst Practices
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
The Aphorisms
for the new
Defensive Paradigm
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 26
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
APHORISM 1
Forecasting is a Huge Waste of
Management Time
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
APHORISM 2
Accuracy is Limited More by the
Nature of the Behavior Being
Forecast than by the Specific
Method Being Used to Forecast It
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 27
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
Coefficient of Variation (CV) =
Standard Deviation / Mean
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
Volatility (CV)
Forecast A
ccuracy
Reducing volatility will likely result in better forecasts
Comet Chart
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 28
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
• 36 months of data for 6000 items
• 87% of items had both MAPE and CV > 50%
0.0%
20.0%
40.0%
60.0%
80.0%
100.0%
120.0%
140.0%
160.0%
180.0%
200.0%
0.0% 20.0% 40.0% 60.0% 80.0% 100.0% 120.0% 140.0% 160.0% 180.0% 200.0%
Fo
recast
Err
or
(MA
PE
)
Volatility (CV)
Comet Chart
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
APHORISM 3
Organizational Policies and
Politics Can Have a Significant
Impact on Forecasting
Effectiveness
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 29
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
APHORISM 4
You May Not Control the Accuracy
Achieved, But You Can Control
the Process Used and the
Resources You Invest
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
• Determine what level of accuracy is reasonable
to expect
• Achieve this accuracy with the least cost in time
and resources
• Automate wherever possible
Corollary: Do not squander resources in pursuit of
unrealistic accuracy goals
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 30
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
APHORISM 5
The Surest Way to Get a Better
Forecast Is to Make the Demand
Forecastable
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
Identify inherent volatility and artificial volatility
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 31
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
Corollary: Any knucklehead can forecast a straight line
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
APHORISM 6
Minimize the Organization’s
Reliance on Forecasting
10/6/2016
Copy r igh t © 2015, SAS Inst i t u t e Inc. Al l r ights reserv ed . 32
Company Confidential - For Internal Use Only
Copyright © 2013, SAS Insti tute Inc. Al l rights reserved.
APHORISM 7
Just stop doing the stupid $#!+