Using Key Metrics to Supercharge Your Demand Management and S&OP Process

1 © 2014 Steelwedge Software, Inc. Confidential.

Single Line of Sight: Plan, Perform, Profit

Measuring Forecast Accuracy

Improved S&OP through Analysis of Forecast Accuracy


Today’s Presenter

Background

Rick Blair VP, Solutions Design and Analytics

Steelwedge Software Inc.

3825 Hopyard Rd

Pleasanton, CA 94588

Tel : (925) 460-1700

[email protected]

• Over 25 years of experience in supply chain consulting and

operations management.

• Works directly with customers and team members to optimize solution

design, drive improvements and apply best practice methodologies.

• Leverages supply chain management practitioner experience in various

roles across S&OP, demand and supply planning


Why measure forecast accuracy?

Examples of forecast accuracy metrics

How is MAPE calculated?

Beyond a number: Key Considerations

Accuracy measures in action

Agenda



• What is Forecast Accuracy?

• A measure of deviation between plan and actual

• Less deviation = greater accuracy

• What motivates an organization to track accuracy?

• Hold individuals and groups accountable

• Benchmark against other companies

• Manage inventory levels

• Improve management of business


Quick Poll #1

Which reason is the most important?



• While each reason has its merit, I suggest that #3 and #4 are most

critical

1. Hold individuals and groups accountable

2. Benchmark against other companies

3. Manage inventory levels

4. Improve management of business

5. Something else


Manage Inventory Levels

• If demand fluctuates substantially from period-to-period, should

you carry more safety stock?

• Conventional supply chain planning says ‘Yes’

Carry buffer inventory to handle demand fluctuations

• But…what if demand variability is predictable?

– Buy or make to forecast

– Reduce safety stock to cover just unpredictable variability

High forecast accuracy translates into reduced inventory requirement


Improve Management of Business

• Questions worth asking

• What are we trying to accomplish?

• What are we trying to improve?

• If we measure forecast accuracy…

– At a high level such as Family or Business Unit

– Using most recent forecast values

• Our accuracy metric looks much better, so why not make ourselves

look good?

Are you trying to look good or improve the business?


More Reasons to Improve Forecast Accuracy

Benefits of Better Forecasting:

Improved customer service levels

Increased sales

Reduced inventory carrying costs

Improved cash flow projections

Production smoothing (level

loading)

Reduced employee costs

Increased ROI

Balancing Supply and Demand

0

10

20

30

40

50

60

70

80

90

100

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

Time

Qu

an

tity

Demand

Supply

Excess

Inventory Lost

Opportunity







Agenda


Accuracy Metrics

• MAD = Mean Absolute Deviation

• RMSE = Root Mean Square Error

• MAPE = Mean Absolute Percent Error

• WMAPE = Weighted Mean Absolute Percent Error

• MAD/Mean = Mean Absolute Deviation/Average

• Yields same result as WMAPE

• Bias = Tendency to over or under forecast

• % of forecasts over actual

• % of forecasts under actual


Which accuracy metric does your company use?

Quick Poll #2


Accuracy Metrics

• MAD = Mean Absolute Deviation

• RMSE = Root Mean Square Error

• MAPE = Mean Absolute Percent Error

• WMAPE = Weighted Mean Absolute Percent Error

• MAD/Mean = Mean Absolute Deviation/Average

• Yields same result as WMAPE

• Bias = Tendency to over or under forecast

MAPE & WMAPE are the most widely

used forecast accuracy measures







Agenda



• Mean Absolute Percent Error = MAPE = ∑│PE│/N

• N = number of periods for which we have PE values

• │PE│ = absolute value of the PE (Percent Error)

• Weighted MAPE = average of individual MAPEs weighted by

actual shipments

= ∑item MAPE * │(item actual / total actual)│

Let’s consider an example…


MAPE Example

1. Error: The difference between Forecast and Actual

2. Absolute Error: Convert negatives to positives

3. Absolute Percent Error = Absolute Error / Actual

Error

SKU 1

Actual 100

Forecast 90

Error (10)

Absolute Error

SKU 1

Actual 100

Forecast 90

Error (10)

Abs Error 10

Absolute Percent Error

SKU 1

Actual 100

Forecast 90

Error (10)

Abs Error 10

APE 10.0%


MAPE Example

• Mean Absolute Percent Error (MAPE)

• Average of APEs for multiple items or multiple periods or both

MAPE Calculation

SKU 1 2 3 4 5 6 7 8 9 10 Family A

Actual 100 90 8 1,000 1 90 10 11 50 76 1,436

Forecast 90 100 10 970 3 80 13 10 50 80 1,406

Error (10) 10 2 (30) 2 (10) 3 (1) - 4 (30)

Abs Error 10 10 2 30 2 10 3 1 - 4 30

APE 10.0% 11.1% 25.0% 3.0% 200.0% 11.1% 30.0% 9.1% 0.0% 5.3% 2.1%

MAPE 30.5%

MAPE = 30.5% = Average of 10 SKU APE values

MAPE for Family A: At Family level = 2.1% At SKU level = 30.5%


MAPE and WMAPE Calculations

SKU 1 2 3 4 5 6 7 8 9 10 Family A

Actual 100 90 8 1,000 1 90 10 11 50 76 1,436

Forecast 90 100 10 970 3 80 13 10 50 80 1,406

Error (10) 10 2 (30) 2 (10) 3 (1) - 4 (30)

Abs Error 10 10 2 30 2 10 3 1 - 4 30

APE 10.0% 11.1% 25.0% 3.0% 200.0% 11.1% 30.0% 9.1% 0.0% 5.3% 2.1%

Wtd MAPE 0.7% 0.7% 0.1% 2.1% 0.1% 0.7% 0.2% 0.1% 0.0% 0.3%

MAPE 30.5%

WMAPE 5.0%

WMAPE Example

• Weighted Mean Absolute Percent Error (WMAPE or WAPE)

• Average of individual MAPEs weighted by actual shipments

WMAPE = 5.0% = Weighted Average of 10 SKU APE values

WMAPE for Family A: At Family level = 2.1% At SKU level = 5.0%



SKU 1 2 3 4 5 6 7 8 9 10 Family A

Actual 100 90 8 1,000 2 90 10 11 50 76 1,437

Forecast 90 100 10 970 3 80 13 10 50 80 1,406

Error (10) 10 2 (30) 1 (10) 3 (1) - 4 (31)

Abs Error 10 10 2 30 1 10 3 1 - 4 31

APE 10.0% 11.1% 25.0% 3.0% 50.0% 11.1% 30.0% 9.1% 0.0% 5.3% 2.2%

Wtd MAPE 0.7% 0.7% 0.1% 2.1% 0.1% 0.7% 0.2% 0.1% 0.0% 0.3%

MAPE 15.5%

WMAPE 4.9%

MAPE & WMAPE: Impact of Small Changes

• What if SKU5 Actual was 2 (not 1)?

MAPE changes from 30.5% to 15.5% WMAPE changes from 5.0% to 4.9%

A one unit change can have a huge impact on MAPE.

WMAPE is less sensitive.



SKU 1 2 3 4 5 6 7 8 9 10 Family A

Actual 100 90 8 1,000 - 90 10 11 50 76 1,435

Forecast 90 100 10 970 3 80 13 10 50 80 1,406

Error (10) 10 2 (30) 3 (10) 3 (1) - 4 (29)

Abs Error 10 10 2 30 3 10 3 1 - 4 29

APE 10.0% 11.1% 25.0% 3.0% 0.0% 11.1% 30.0% 9.1% 0.0% 5.3% 2.0%

Wtd MAPE 0.7% 0.7% 0.1% 2.1% 0.0% 0.7% 0.2% 0.1% 0.0% 0.3%

MAPE 10.5%

WMAPE 4.9%

MAPE & WMAPE: Zero Actual Values

• What if SKU5 Actual was 0 (not 1)?

MAPE changes from 30.5% to 10.5% WMAPE changes from 5.0% to 4.9%

Pay close attention to instances where Actual = 0. Dividing by

zero causes an error, so the MAPE formula may assign 0%.

This can be misleading since 0% should represent zero

deviation between forecast and actual.







Agenda


Key Considerations

• Error vs Accuracy

• Metrics measure degree of error

• Accuracy can be defined as (1 – MAPE) or (1 – WMAPE)

– If MAPE = 25%, then Forecast Accuracy = 75%



Notice that Forecast Accuracy is not negative

Loss of visibility to error magnitude if MAPE > 100%


Key Considerations

• Aggregation Level

• Higher aggregation levels usually yield lower MAPE and WMAPE

values

– Variation is dampened as peaks and valleys get smashed together

• Ask: What are we trying to improve?

– Measure accuracy at level where you can affect change

– You may decide to measure accuracy at multiple levels

– For example, Product Mix (SKU) & Product Line (Family)


Key Considerations

• Time Buckets

• Examples: Month, Quarter, Rolling 3 Months

• The bigger the time bucket, the lower the MAPE

MAPE Calculation

Month Jan Feb March Q1

Actual 105 92 75 272

Forecast 95 100 82 277

Error (10) 8 7 5

Abs Error 10 8 7 5

APE 9.5% 8.7% 9.3% 1.8%

MAPE 9.2%Monthly deviations are

eliminated in Q1 MAPE value


Key Considerations

• “Lag” or “Offset”

• Use the forecast at time of decision

– May be production or raw material lead time

– Measure accuracy of forecast that manufacturing could actually use

• The most recent forecast provides little to no value in making business

decisions

– What can we do with a forecast for February provided in February?

In this example, the forecast from Month 2 of prior quarter -1 is compared to actual results from prior quarter. For example, forecast created in May for Q3 (July, August and September) is compared to actual results from Q3 (July, August and September).

Previous Quarter - 1 Previous Quarter Current Quarter

month 1 2 3 1 2 3 1 2 3

"as of date" Quarter to be measured


Key Considerations

Forecast Period:

As Of: Jan-13 Feb-13 Mar-13 Apr-13 May-13 Jun-13 Jul-13 Aug-13 Sep-13 Oct-13 Nov-13 Dec-13

Oct-12 124 140 90 120 120 85 120 120 100 120 100 120

Nov-12 110 130 100 100 100 100 100 100 90 110 100 100

Dec-12 120 125 100 100 115 90 85 90 95 100 110 110

Jan-13 90 95 100 130 120 125 100 120 115 90 95 80

Feb-13 100 105 110 120 100 100 110 100 90 80 80

Mar-13 120 95 100 120 120 100 120 100 120 115

Apr-13 100 95 115 123 85 90 95 100 110

May-13 85 103 113 102 85 90 95 100

Jun-13 100 96 88 118 120 100 120

Jul-13 111 112 115 112 85 90

Aug-13 120 102 115 120 85

Sep-13 100 102 115 97

Oct-13 95 106 135

Nov-13 95 100

Dec-13 115

Actual 95 105 120 105 90 95 101 130 95 95 100 120

MAPE

Offset Jan-10 Feb-10 Mar-10 Apr-10 May-10 Jun-10 Jul-10 Aug-10 Sep-10 Oct-10 Nov-10 Dec-10

0 5% 5% 0% 5% 6% 5% 10% 8% 5% 0% 5% 4%

1 26% 10% 13% 10% 6% 8% 5% 14% 7% 7% 6% 17%

2 16% 19% 17% 5% 11% 21% 12% 32% 21% 21% 15% 13%

3 31% 24% 17% 24% 33% 26% 22% 22% 24% 18% 20% 19%

In this example, monthly MAPE values are shown over time with

various offsets. Tracking MAPE trends is a great way to see

improvement. Multiple offsets allow insight into how accuracy may

improve as better information is available nearer to current period.


Key Considerations

• Units vs Revenue

• Accuracy measures need not be limited to Units

• $ may be more meaningful if Average Selling Prices vary considerably

– Use WMAPE and weight by $







Agenda


MAPE & Bias Analysis

Actual and Forecast values MAPE by Item

Bias

MAPE and WMAPE

Excel Slicers


Targeted MAPE Analysis

MAPE by Functional Group

WMAPE by Functional Group


Q&A
