Getting the Most out of Statistical Forecasting!

Author: Ryan Rickard, Senior Consultant Published: September 2017

Delivering Strategic, Implementation, Enhancement, Migration/Upgrade and Outsourced Support Services

across SAP’s Execution and Supply Chain Planning Suite Including and Not Limited to:

ERP ECC & S/4 HANA, SCM APO, IBP on HANA, SCIC/Control Tower, SNC, EIS (SmartOps),

S&OP Powered by HANA and Ariba

US-based Platinum Level Supply Chain Consultants With Deep Expertise in Both the Technical Tools and

Functional Business Processes

Delivering Projects and Services Across 20+ Different Countries in North and South America, Europe and Asia

Since Our Inception 15+ Years Ago

Founded in 2001, SCMO2 Specializes in High-End Supply Chain Consulting Work Focused on The Implementation and Better Use of SAP Applications, Including ERP ECC & S/4, SCM APO & IBP on HANA, Ariba, Among Others

Featured in Publications and Regularly Present at SAP Conferences

Globally, like SAP Insider, SAPPHIRE NOW and ASUG Annual Conference.

Partnered with SAP’s Supply Chain Group to Deliver Informative Sessions

on Latest Tools and Functionality, like SAP Integrated Business Planning.

Partnered with SAP Insider to Deliver Multi-Day “Bootcamp” Seminars.

Company Statistics

About SCMO2

Forecasting is a Core Competency

We already offer programs specific to Demand Planning and S&OP

Upcoming Events for SCMO2

SCMO2 and SAPinsider IBP Bootcamp: www.ibpbootcamp.com/SCMO2

SCMO2 presenting at Fall Focus (ASUG):

http://focus.asug.com/

Questions throughout today’s Webinar? Feel free to click on the Q&A.

An SCMO2 panelist will answer questions throughout the Webinar. We will address any outstanding questions at the end of the session.

Q&A

Meet Ryan Rickard

Session Leader

Ryan Rickard – Senior Consultant• 17 years’ Experience in Supply Chain Planning, Including Working as a

Planner, IT Resource, and as a Business Process Re-design Lead• Demand Planning and Statistical Forecasting Specialist in APO-DP and

IBP-Demand • Frequent Speaker at Many Premier Supply Chain Events

Contact Info:Ryan Rickard, Sr. Consultant

rrickard@scmo2.com

(770) 639-7285

Follow SCMO2:www.scmo2.com

www.facebook.com/SCMO2/

www.twitter.com/BreatheInSCMO2

Getting the Most out of Statistical Forecasting!A multi-series webinar to explain “How to Effectively Analyze & Model your Demand”

Session 1 Variability MattersCalculating Variability & Segmenting to help drive the process

Session 2 How Much is Enough?How much Historical Data is Enough? How frequent to Run (Stat) & React?

Session 3 Super Model ForecastingThe Optimal Level of Aggregation

Weeks vs. Months can make a Difference

Session 4 FVA: The New FrontierUnderstanding how Forecast Value Add can enhance your forecasting value

Webinar Series

Session 2 – How Much is Enough Typically, the more data you have the better

If you have Weekly and Monthly data, analyze both. Look for patterns weekly that are masked when aggregated Monthly.

The frequency of generating Stat should align to your business process. Running Stat more frequently allows your Supply Chain to react the fastest.

Considering both Variability & Historical Period counts is important to assigning appropriate models

Session 1, 2 and 3 RecapSession 1 – Variability Matters All products are not the same. Their DNA and patterns are different.

Calculating Variability can be done using the Coefficient of Variation methodology – in Excel or APO/IBP

Zeros matter in the CoV calculation, and when counting periods with historical values.

Variability correlates to Forecastability

Session 3 – Super Model Forecasting Patterns and periods of history are different at each level of aggregation

What’s most important for your business?

– The right Product/Customer forecast or Product/Location forecast?

Running Stat at the same level that you measure Forecast Accuracy will give the best understanding of Stat performance

For Seasonal Items, consider CoV2 (CoV of Forecast Error)

Session 4

FVA: The New FrontierUnderstanding how Forecast Value Add can enhance your Forecast Value

FORE CASTING

Forecasting Deep Dive

Forecasting is a key part of achieving effective planning results.It’s difficult to have an effective supply chain with poor forecasting.

FVA is a Forecasting Deep Dive

Forecast Accuracy

Reduced Inventory levels

Improved Customer Service Levels metrics

Forecast Bias

Improved Budgeting and Financial Reconciliation

Reduced Excess & Obsolete Inventory (% of Sales)

Forecasting Metrics:

Expected Outcomes:

Typical Forecasting Metrics & Expected Outcomes

Now we have another new metric…FVA!

Introduction to FVA

FVA is a metric that allows you to evaluate the performance of each step, each level, and even each participant/planner in the forecasting process

It expresses the results of doing something versus doing nothing

FVA can be either positive or negative, indicating whether your efforts are adding value to the forecast, or making it worse, and to what degree

Measuring FVA allows you to identify waste and inefficiency in the forecasting process

– When FVA is negative, then the process activity is making the forecast worse and should be eliminated

A Simple Example…

Let’s assume that you are forecasting amaterial and making inputs or adjustmentsat the customer level

You have generated a Statistical forecast (at either the material or material/customer level)

You have manual overrides to the Statistical forecast at various levels

Suppose your Statistical forecast achieved a MAPE (or error) of 30%

And, assume that your Consensus Demand (which included overrides) achieved a MAPE of 25%

This would indicate that the extra analysis and adjustment to the Statistical forecast actually made the forecast better

Why Use FVA?

Traditional forecasting metrics tell you about the Size, Direction or Tendencies of the Forecast Error

– Forecast Accuracy measures the absolute Forecast Error (aka MAPE)

– Forecast Bias tells you about the Forecast Tendency(the tendency to be too high, or too low)

But neither tell if you have been improving or hurting the process

FVA is an analysis that can help you determine: How efficient you are at forecasting If the adjustments or changes are actually adding value

over time Which inputs are adding value and how much

Where do we add value during the

forecasting process?

The Big Question is…

That’s the Key!

Where do we add value during the forecasting process?

StatisticalForecast

NaïveForecast

JudgmentForecast

Simplest Forecast: “Tomorrow will be like

today.”

System Forecast:Generated using tools like SAP APO, SAP IBP

Adjusted Forecast: Incorporates additional

market/economic information (a.k.a

“Sales”, “Marketing”)

Forecast Accuracy

+5%(value added to Naive)

-3%(value subtracted from Stat)

+2%(value added to Naïve)

60% 65% 62%

Forecast Value-Added Analysis (FVA)

A naïve forecast is something simple to compute. It requires minimum effort and manipulation to prepare.

Naïve Forecast

If you were asked to forecast the weather, the easiest and most predictable method would be to consider yesterday’s weather (last

month’s weather, or same month last year’s weather).

The Naïve forecast becomes your measuring stick

– It’s the baseline measurement that all other forecasts are compared to

– If you can’t beat the “simplest” forecast, then you need to eliminate the steps or waste and use the Naïve

Goal or Purpose of a Naïve Forecast

Methods for Creating a Naïve Forecast

Several commonly used methods to creating a naïve forecast are:

1. Sales History or “Seasonal Random Walk” – uses the history from the same period a year ago as the forecast for the same period this year

If you sold 50 units last June and 75 pieces last July, your future forecast would be 50 in July and 75 in August

2. “Random Walk” or last value – uses the last known actual value as the future forecast (in every period)

If you sold 15 units last month your future forecast in every time period would be 15 units. If your history drops to 10 units this month, then your future forecast shifts to 10 units.

3. Moving Average, or other very simple statistical formula

Requires very minimal effort

Doesn’t require many data points

Duration, or time periods, can be easily adjusted

Typically smooths outliers and seasonal patterns

4. Blend or Average of all 3

Naïve Examples1

2 3

Which is best to create the Naïve? As you can see, you can get different naïve forecasts based on the method chosen

Which is best? It depends on the nature of the pattern you are trying to, or wish to forecast.

One suggestion is to take a blend, or average of several. Or use one that is simple!

0

10

20

30

40

50

60

70

80

90

100

Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May

Seasonal Random Walk Random Walk Moving Average Average

Naïve Forecasts Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May

Seasonal Random Walk 78 40 50 80 42 38 90 42 50 63 52 40

Random Walk 40 40 40 40 40 40 40 40 40 40 40 40

Moving Average 58 58 58 58 58 58 58 58 58 58 58 58

Average 59 46 49 59 47 45 63 47 49 54 50 46

FVA Steps

Create a Naïve Forecast

Compare the Statistical Forecast to the Naïve forecast

Compare other “input” Forecasts to the Naïve forecast

Compare the final Consensus Demand to the Naïve Forecast

Review Results– By Material Groupings

– By Materials

– By Customer Groupings

– By Customer

Considerations:Which Lag or Snapshot do you want to Measure at

(M, M-1, M-2, M-3)?

Which Levels do you need to Store data and Measure a?. Create Naïve Forecasts for each.

Where to Build FVA? Where to Store? Where to Analyze and Report?

How to Display the Results?

First, do NO harm!

The easiest way to make the forecast better…is to STOP making it WORSE.

Take the Forecasting Hippocratic Oath

Dr. Gregory House

“The Flaw of Averages”

An average may hide as much information as it reveals

“Slice N Dice”

The FVA tool allows us to drilldown into the granular

details of our forecasts

We can break down the forecasting results by Month,

Family, Product, Customer, Location, etc

We can also break down the various forecasting inputs

(Stat, Sales, Marketing, Demand Planning, Consensus)

at the various levels

Let’s look at some REAL

LIFE examples!

May the Force be With You!

Stat forecasting is not beating the Naïve forecast. Are we using the proper Stat Forecast Model? Are you using the right amount of Historical Data points? Is the history cleaned?

FVA: Slice & Dice

Stat forecasting is adding value, but the Consensus Demand adjustments are hurting the accuracy. How can we improve? How should we use additional market and customer information to improve the forecast?

An Interlude on MetricsPhilosophy of Metrics

Forecast Accuracy Calculation

Displaying Results

Other Calculation Considerations

Monitor performance

Learn where we need to improve

Learn if we need to improve

“Why Do We Track a Metric?”

No metric is perfect

All metrics are wrong, some metrics are useful

The Philosophy of Metrics

Metrics are full of numbers

Perfection is the enemy of the good

Philosopher -Voltaire

1694-1778

The question is, “Is the metric better than using our gut to make decisions”

Philosopher – Adam Smith

1723-1790

Can the metric help me improve the forecast?

Philosopher -Voltaire

1503-1556

Numbers are…• Irrefutable• Objective• Lend Weight to Analysis &

Understanding

Notes:• Forecast Accuracy usually considers a “lagged” or snapshot of the forecast

(i.e. M-1, M-2, M-3 snapshot)• Error is typically calculated for each Product/Location, then summed to

get the total Error• This calculation weights higher volume SKUs more heavily • The Forecast Accuracy % result can be negative

Be mindful of the “Flaw of Averages”

n

i

i

n

i

i

ndActualDema

AbsError

MAPEcuracyForecastAc

1

11

Definition of Forecast Accuracy

Compared to the Naïve, each “input” forecast is tracked as a %

Displaying FVA Results

Compared to Naïve FVA - M6 2017 FVA - M7 2017 FVA - M8 2017

Statistical Forecast -16.5% -7.4% -8.5%

Sales Mgr Forecast -36.9% -19.1% -30.0%

Marketing Forecast -9.5% -17.5% -9.2%

Demand Planning Forecast -6.5% 1.1% -7.6%

Consensus Demand 2.0% -12.9% 1.7%

The graph of FVA% is a little “confusing.”

Will Management Understand this?

Displaying Forecast Accuracy Results

Understanding and comparing Forecast Accuracy is often much easier!

That’s much better!

Forecast Accuracy 2017-M06 2017-M07 2017-M08

Naïve Forecast 84.5% 81.2% 79.6%

Statistical Forecast 68.0% 73.8% 71.1%

Sales Mgr Forecast 47.5% 62.1% 49.6%

Marketing Forecast 75.0% 63.7% 70.4%

Demand Planning Forecast 78.0% 82.3% 72.0%

Consensus Demand 86.4% 68.3% 81.3%

Timing of Forecasts Misses

Should Forecast Accuracy go Negative?

What’s the Naïve Forecast for a New Product?

What Lag should we use?

What about forecast inputs made a multiple levels?

What if we don’t utilize Statistical Forecasting today?

Other FA & FVA Calculation Considerations

Typical Forecasting Scenario:

– A Promotion is forecasted for the 1st week of June

– Most of the promotional orders arrive early and are shipped in May.

– The rest of the orders arrive and ship in early June

Forecast Accuracy Doesn’t Consider Timing

You are a victim of the “double-ding”

MonthStatistical

Forecast

Promotion

Uplift

Total

ForecastActuals Error MAPE

Forecast

Accuracy %

May 50,000 50,000 75,000 25,000 33.3% 66.7%

June 50,000 30,000 80,000 55,000 25,000 45.5% 54.5%

Some companies stop Forecast Accuracy at 0%

– Why? Because it is hard to explain negatives to Executives

It’s harder to understand how negative accuracies impact the Overall results

So, should we allow the FA % to go negative? Yes!

The magnitude of the miss is very important!

– Especially for New Products and Promotions

Forecast Accuracy < 0%

ProductsTotal

ForecastActuals Error

A 4,200 2,000 2,200

B 10,000 2,000 8,000

C 70,000 2,000 68,000

(Wrong)

Fcst Accy %

0

0

0

(Correct)

Fcst Accy %

-10.0%

-300.0%

-3300.0%

There’s BAD, and then there is REALLY BAD!

The difference between 3x and 30x IS something to sneeze at!

What is the Naïve for a New Product?

If there is no Prior History,then the Naïve Forecast issimply 0

Naïve for New Products

Can we copy the History

from a similar item as the

Naive

Do we just use the initial

ForecastThe Average of other New Products launches

Zero

Wait for

History

Remember…the Naïve is just a reference forecast• It’s not the actual forecast or what we

would actually order from the factory/supplier

If the Naïve forecast is 0, we should always be able to beat it with either a Statistical Forecast or a Judgment Forecast right?

What do we do if we’re adding value at lag M-1, but not at lag M?

What Lag?

What lag should we use to measure Forecast Value Add and Forecast Accuracy?

Considerations:

– What are your average Product Lead Times?

– Are some items Manufactured and other Purchased?

– When do you take your forecast snapshot (end of month, or beginning of month)?

Select something Simple and Consistent (at least to begin) for all Products and Groupings

“Oh, I know!” Use FVA to compare the results at M-1 vs

M lags Drill down to determine which Levels and

Inputs made the forecast worse as we got closer

If we have forecast inputs or adjustments at a detailed level (i.e. Product/Customer), then what do we use as the Naïve forecast?– How do we know if detailed forecasts impact the aggregate?

Forecast Inputs & Changes at Multiple Levels

An Example for illustrative

purposes!

Material Customer Jun Jul Aug Jun Jul Aug Jun Jul Aug Jun Jul Aug

Retail1 1000 500 2000 1000 500 2000 950 525 1900

Retail2 20 25 20 20 25 20 22 20 25

Retail3 200 200 400 100 200 300 400 212 305 375

Dist1 30 30 30 20 20 20 50 50 50 30 45 38

Dist2 2 2 2 2 2 2 1 3 2

Web 50 50 50 200 50 250 50 48 125 55

Club 0 0 0 0 1 4

Comm1 100 100 100 250 350 100 100 325 98 110

Comm2 80 80 80 80 80 80 110 80 75

1482 987 2682 270 320 20 1752 1307 2702 1698 1202 2584

Statistical Forecast Adjustment to Stat Forecast Total Forecast Actuals

FIT001

If we have forecast inputs or adjustments at a detailed level (i.e. Product/Customer), then what do we use as the Naïve forecast?

– We use the Naïve at each of those Customer levels

What is the Naïve for Levels with Adjustments?

1. We’ll start with the Adjustments made at the customer level

2. We only consider the Actuals for the Customers with Adjustments

3. We pull the Naïve forecast at the Customers with Adjustments

4. Finally, we’ll also need the Stat forecast and the Final Judgment forecast (Stat+Adjustments) so we can track the FVA

Material Customer Jun Jul Aug

Retail1

Retail2

Retail3 100

Dist1 20 20 20

Dist2

Web 200

Club

Comm1 250

Comm2

270 320 20

FIT001

Adjustments Only Adjustment to Stat Forecast

Jun Jul Aug

305

30 45 38

125

325

355 475 38

Actuals

Jun Jul Aug

300

50 50 50

250

350

400 600 50

Total Forecast (Adj)

Jun Jul Aug

175

27 32 28

45

95

122 252 28

Naïve (Adjustments)

Calculating Naïve FA for Detailed Level Changes

Jun Jul Aug

175

27 32 28

45

95

122 252 28

Naïve (Adjustments)

Jun Jul Aug

305

30 45 38

125

325

355 475 38

Actuals

Jun Jul Aug

130

3 13 10

80

230

233 223 10

ABS Error (Naïve Adjustments)

Now that we have the detailed Naïve information, we’ll calculate the Naïve Forecast Accuracy for each Product/Customer

FA%

Naïve

57.4%

77.0%

36.0%

29.2%

Material Customer

Retail1

Retail2

Retail3

Dist1

Dist2

Web

Club

Comm1

Comm2

Adjustments Only

FIT001

The Naïve forecasts for the Customer specific Adjustments

The Actuals for the Customers with Adjustments

The Naïve Error & Forecast Accuracy for each Customer with Adjustments

Next we’ll calculate the Forecast Accuracy of the Stat and Adjustments to Stat

Calculating Input FA for Detailed Level Changes

Adjustments made for each Customer

Actuals for the Customers with Adjustments

Now we can calculate the Forecast Accuracy for the Stat and Total Forecast at the Customer level

Material Customer Jun Jul Aug

Retail1

Retail2

Retail3 100

Dist1 20 20 20

Dist2

Web 200

Club

Comm1 250

Comm2

270 320 20

Adjustments Only Adjustment to Stat Forecast

FIT001

Jun Jul Aug

305

30 45 38

125

325

355 475 38

Actuals

Jun Jul Aug

200

30 30 30

50

100

130 280 30

Statistical Forecast (Adj)

Jun Jul Aug

300

50 50 50

250

350

400 600 50

Total Forecast (Adj)

Stat Stat+Adj

65.6% 98.4%

79.6% 67.3%

40.0% 0.0%

30.8% 92.3%

FA%

Now we can compare the Forecast Accuracy % of the Naïve, Stat, and Adjusted (Judgment) Forecasts

Comparison of FA’s for the Detailed Changes

Material Customer Jun Jul Aug

Retail1

Retail2

Retail3 100

Dist1 20 20 20

Dist2

Web 200

Club

Comm1 250

Comm2

270 320 20

Adjustments Only Adjustment to Stat Forecast

FIT001

Jun Jul Aug

305

30 45 38

125

325

355 475 38

Actuals

Naïve Stat Stat+Adj

57.4% 65.6% 98.4%

77.0% 79.6% 67.3%

36.0% 40.0% 0.0%

29.2% 30.8% 92.3%

FA%

Stat-Naïve Total-Naïve

8.2% 41.0%

2.7% -9.7%

4.0% -36.0%

1.5% 63.1%

FVA

Now All the data is in place, and we’re ready for the FVA

For these Customers, the Stat forecast is better than the Adjustments which were made. The manual adjustments took away 9% and 36% points of value.

For these Customers, the Stat forecast is better than the Naïve, but the Adjustments to Stat added lots of value.

What if we don’t utilize a Statistical Forecast today?

Or, what if we don’t do it (Stat) well?

Client Example:– Created 4 “reference” Stat key figures (these didn’t impact the

business forecast in any way

– Used 4 different modeling scenarios

Crostons – Product/Monthly

Seasonal Linear Regression – Product/Monthly

Pick Best/Composite (between the 2 above) – Product/Monthly

Pick Best/Composite (between the 2 above) – Product/Customer

– Provided a good comparison when Stat Forecast wasn’t created

– Provided a good comparison when the current Stat wasn’t adding value

No Stat? Bad Stat?

Can you create a Simple Model?Maybe even an Average in Excel

Try different levels and time buckets

Does your business set Forecast Accuracy targets for Demand Planning, Sales, and Marketing?

Instead basing performance off of Forecast Accuracy…Why Not FVA?– Instead of Measuring Forecasting Success via FA, maybe FVA is better?

– For example, Demand Planners don’t control Inventory Planning or Product Availability. Nor do they control Customer Orders or Shipments.

– They can control the Accuracy of the Stat, and they can determine which forecast inputs are adding or hampering value. And they can drive communication and forecast adjustments to minimize the harm.

A More Modern Measurement Technique

Forecast Accuracy

Forecast Bias

Forecasting Metrics:

Reduction of Customers– One business went from forecasting at 37 Key Customers, down to 6

As a result, the FA% went up by 10+% at 90 day lag Hit a 7 month consecutive high and achieved up to 80% FA

– Removed & Combined Customers to those where they could actually add value

Shifted Focus from C & D products to A’s and B’s– Put the C & D products on “auto-pilot” (aka Stat Forecast)

In some cases, they chose to NOT forecast the C & D products at all

– Provided more time to focus on improving the A’s and B’s

Recognition that the Naïve forecast is valuable Recognition that generating a Statistical Forecast is extremely useful Recognition that aggregate forecasting (Product) is better than the sum of the

detailed forecasts Recognition that certain Sales input (aka Promotions) was never valuable

– Quit taking Promotional insight /uplift from certain Salespeople

Reallocation of Resources (New Statistical Forecasting Team)– After realizing that Stat and Naïve were valuable, shifted focus away from “most” inputs

and overrides and focused on optimizing the Statistical Forecasting Process (Segmentation, Levels of Aggregation, etc.)

Leadership recognition that Forecast Accuracy targets weren’t realistic…Forecastability!

Realization that changing the forecast in the Short-Term wasn’t adding value– Shifted focus to the periods (lead time) where an updated forecast really impacted

planning capabilities

How has FVA been Proven Useful

Quote from a Supply Chain Vice President who’s team embraced FVA…

FVA…A Paradigm Shift

“With FVA, you realize that perhaps the only reasonable goals for forecasting performance are to beat a naïve model and continuously improve the process.

Improvements can be reflected through a reduction in forecast errors, or a reduction in forecasting process (minimizing the use of company resources in forecasting).

If good automated software can give you usable forecasts with little or no management intervention, why not rely on the software and invest management time in other areas that have the potential to bring more value to the organization?

Let your production people produce and let your sales people sell – don’t encumber them with forecasting unless you truly must and it add value.

You want to eliminate waste and streamline your process for generating forecasts as accurately and efficiently as possible.”

FVA is a new forecasting metric which allows you to “Deep Dive” and “Slice n Dice” to understand value

A Naïve forecast is simple and easy, and it becomes the baseline measuring stick to base all other input comparisons against

First, “Do no Harm!” The easiest way to improve the forecast is to STOP making it WORSE.

No Metrics is Perfect, but Numbers are Irrefutable

Be mindful of the Flaw of Averages…dig into the weeds to understand detailed Value Add

Forecast Accuracy can go Negative

The Naïve for a new product is 0

Consider FVA as a new metric to measure forecasting importance to help eliminate waste in the process

Summary of Key Points – Session 4

Questions?

Contact Info:Ryan Rickard, Sr. Consultant

rrickard@scmo2.com

(770) 639-7285

Follow SCMO2:www.scmo2.com

www.facebook.com/SCMO2/

www.twitter.com/BreatheInSCMO2

So that we can improve our presentation going forward, please take a couple of minutes to give us your feedback

We have 4 brief questions that we’d appreciate your response on as you exit the Webinar

Survey

Getting the Most out of Statistical Forecasting!

Author: Ryan Rickard, Senior Consultant Published: September 2017