Session 6: A guide to choosing forecast models Demand Forecasting and Planning in Crisis 30-31 July, Shanghai Joseph Ogrodowczyk, Ph.D

Session 6: A guide to choosing forecast models

Demand Forecasting and

Planning in Crisis

30-31 July, Shanghai

Joseph Ogrodowczyk, Ph.D.

Session 6 Joseph Ogrodowczyk, Ph.D.

Demand Forecasting and Planning in Crisis 30-31 July, Shanghai 2

A guide to choosing forecasting models Session agenda

Judgment modeling: Using expert knowledge to provide forecasts

Mixed methods: Constructing forecasting frameworks from expert knowledge

Quantitative modeling: Using statistical techniques to provide forecasts

Criteria for combining or adjusting forecasts



A guide to choosing forecasting models

Review The last session explained some short run forecasting

methods These techniques are used to quickly obtain forecasts

How do we produce more accurate forecasts for longer time horizons?

We need more sophisticated models What are the different types of forecasting models? What are the criteria for choosing which model to use in a

particular instance?




Judgment: Experts are providing forecasts Mixed: Applying forecasting guidelines to statistically

produced forecasts Quantitative: Using statistical techniques to generate

forecasts

(Armstrong and Green 2009)




Judgment model types: Experts (in forecasting or a product line) are producing the forecasts via … Unaided judgment Expert forecasting Decomposition Conjoint analysis Intentions / Expectations Role playing / Simulated interaction Structured analogies




Judgment models Unaided judgment: Forecasts made without the use of

formal forecasting methods Conditions of use

Large changes are not expected Forecasts are not used for policy analysis Highly predictable/repetitive atmosphere

Advantages: Quick to produce forecasts Inexpensive if only a few forecasters are needed Accuracy can be improved when forecaster obtains rapid feedback




Judgment models Expert forecasting: Experts are asked to provide forecasts

Conditions of use can vary Easy access to experts Motivated and knowledgeable experts Need for confidentiality Low dispersal of knowledge Limited time for forecast production



A guide to choosing forecasting models Judgment models

Expert forecasting Nominal group technique: A one-round survey for forecasts in which

experts possess similar information Estimate-talk-estimate: A three-round survey where experts possess

different information. Between forecast estimations, the participants are asked to have a discussion

Delphi method: At least two survey rounds with results of the previous round summarized for participants

Prediction markets: Incentive-based arrangements that use markets to aggregate, in the form of prices, information that is dispersed among participants.




Judgment models Decomposition: Breaking down the estimation task into a

set of components to produce a target forecast Method is to ask experts to forecast parts of the whole and then

aggregate for the whole forecast Conditions of use can vary

Forecasting a highly complex system Forecasting in an unfamiliar metric/market/customer Higher confidence in component forecasts than in the target

forecast Decomposition can be additive or multiplicative




Judgment models Conjoint analysis: Survey method based on characteristics

of a product Decomposition used pieces of the target forecast. Conjoint

uses characteristics of the forecasted item Sometimes used to construct a forecast manually but often

used as the basis for a regression type analysis (mixed method)

Conditions of use: Large changes in demand expected To be used for policy analysis Survey users of the forecasted item instead of experts with market /

forecasting knowledge




Judgment models Conjoint analysis

Popular in marketing Consumers are asked to rank / assign a value trade-offs between

multi-dimensional alternatives Automobiles, soft drinks, computers, checking accounts, hotel

accommodations, etc. Results can be plausible

Major drawback is the potential hypothetical nature of the survey Consumers are not deciding between actual goods and services




Judgment models Intentions / expectations: Survey method based on intended

future behavior Conjoint asked consumers for preferences on specific item

characteristics. Intentions asked consumers for anticipated future behavior

Conditions of use Large changes in demand are expected Relatively little conflicts among forecasters Forecasts are not used for policy analysis (a need to choose

between different courses of action) Disadvantages

Research shows that intentions are biased as measures for prediction

Research is inconclusive about how best to measure intentions




Role playing / Simulated interaction: Customers are asked to act out prospective interactions in a realistic manner

Conditions of use Important as a tool in forecasting within conflicts (threats of

striking workers, jury reactions, assessing outcomes from different strategies)

Small number of parties interacting Need to predict in situations involving large changes

Advantages Decisions are often difficult to forecast if they are the result of a

series of actions Role playing can be used to simulate the actions and

reactions between the parties




Structured analogies: Surveying experts to compare analogous situations and using the outcomes of the analogies as the forecast for target

Conditions of use Large changes are expected Difference among forecasters Similar cases exist

Methodology Describe the target situation Select experts Identify and describe analogies

Ask the experts to describe as many analogies as they can without considering the extent of the similarity to the target situation




Judgment models Structured analogies (Green and Armstrong 2007) Methodology

Rate similarity Ask the experts to list similarities and differences between their

analogies and the target situation, and then to rate the similarity of each analogy to the target

Ask them to match their analogies’ outcomes with target outcomes Derive forecasts

Set up the rule system (criteria) for choosing the analogy to use for the target forecast before interviewing experts

Many rules are reasonable For example, one could select the analogy that the expert rated as

most similar to the target and adopt the outcome implied by that analogy as the forecast







Mixed modeling : Forecasting using statistical models created from expert rules The development of statistical techniques is guided by

expert’s rules Rules: Criteria, inputs, and other variables that experts use in

producing forecasts using judgment methods

Mixed model types Quantitative analogies Expert systems Rule-based forecasting Bootstrapping



A guide to choosing forecasting models Mixed model types

Quantitative analogies: Data from analogous situations are used as input to derive the target forecast Sample rule: When data are not available, use data from a

similar situation Conditions of use

Not a good knowledge of the relationships in the data Cross sectional data (Data across multiple units for the same

time period) Not used for policy analysis

Advantages Statistical forecasting methodologies can be used on input data Allows for increased observations for a data-poor series such

as new products




Mixed model types Expert systems: Statistical models are designed to

represent the rules used by experts in the forecasting process Rules are based on knowledge of target area Models are based directly on the rules Information on rules can be found in research papers, surveys,

and interviews Expert systems should be easy to use, incorporate the best

available knowledge, and reveal the reasoning behind the recommendations they make



A guide to choosing forecasting models Mixed model types

Rule-based: A system to develop and apply weights for combining extrapolations Time series extrapolation: Univariate time series forecasting

methods Rules result in each extrapolation method being assigned a

weight based on trends, seasonality, and historical data The compilation forecast is the sum of the weighted

extrapolation methods Knowledge for rules can be obtained through expert judgment,

empirical research, and theory Guidelines for rules

Give separate consideration to level and trend Use different models for short- and long-run forecasts Damp the trend as the forecast horizon increases




Mixed model types Bootstrapping: Translating experts’ rules into a quantitative

model Difference from expert system

Based on inference about experts rules Requires repeated sampling Specific model design used in the translating (regression)

Model is produced by regressing the forecasts produced upon the information that the expert used

Guidelines for bootstrapping Use experts who differ Use simple analysis to represent behavior Note: Quantifying the variables used by the experts can greatly

affect the validity of the model







Quantitative modeling: Using statistical techniques to provide forecasts Forecasts are produced by statistical models modeling the behavior

generating the data series (historical data or external variables) Experts suggest appropriate variables Stepwise regression, state-space models, and Bayesian techniques

are statistical tools that can be applied either to univariate or multivariate models

Quantitative model types Extrapolation/neural networks Statistical regression Segmentation Index




Quantitative model types Extrapolation: Statistical models using only historical

information to produce forecasts Univariate times series, Holt-Winters (exponential smoothing),

Box-Jenkins and ARIMA (ARMA), autoregressive, linear trend (trend using only time), simple (single) regression

From session 4, Naïve model and moving average models (without and with confidence intervals)

Assumes that all the necessary information for forecasting is contained in the historical data

Can also be used for cross-sectional data To estimate the probability of a new hire lasting more than a year,

analyze the percent of the previous 50 applicants lasting more than a year




Quantitative model types Neural net: Models using complex, interdependent, variable

relationships to produce forecasts Inspired by the behavior of biological neurons Often a “black box” for understanding the relationships

Conditions of use Best for quarterly or monthly data Discontinuous series Several-period lag between forecasting and forecasted periods

Advantages Does not need to fully understand the relationship of

explanatory variables Estimates nonlinear functions well




Quantitative model types Statistical regression (Econometrics): Using statistical

methods to estimate the relationships of variables based on theory, prior studies, and expert knowledge (Augmented) Dickey-Fuller, vector autoregressive (VAR), error

correction models (ECM), multiple regression Parameter estimates (elasticities, measures of influence on

dependent variable by independent variable) can be obtained through using least squares or maximum likelihood

Models use theory and expert knowledge to select the explanatory variables

Dependent variable is the forecasted item and independent (explanatory, causal) variables are those which “explain” the behavior of the dependent variable




Quantitative model types Segmentation: Forecasting a heterogeneous whole through

forecasting of parts of the whole separately When the dependent variable responds in different ways to the

independent variables Forecasts for the parts will be created from separate econometric

models because of the difference in causal effects Airline tickets: Business class and recreational coach class

customers respond differently to price changes Better to forecast each type of passenger and then aggregate

Similar, but not the same, as bottom-up forecasting Bottom-up down not necessarily need multiple model types



A guide to choosing forecasting models Quantitative model types

Index: Forecasting the value of the dependent variable by adding values of the independent variables Improper linear models. Unit-weight is a special case when

variables are weighted evenly Explanatory variables can be subjective and assigned a 1 or 0

depending on if they are present or absent, respectively Explanatory variables can be quantitative data that have been

normalized (units mathematically removed) Weights can be chosen by experts Values of the dependent variable can be used to forecast the

probability of an event Example: Factors contributing to drug use in adolescents

Grades, parent relationship, self esteem, etc. (Bry et al. 1982)







Criteria for combining or adjusting forecasts Two main questions on combining

Are several methods producing useful forecasts? If so, how can they be combined?

Two main questions on adjusting Is there a need to adjust the forecast because of omitted

data? If so, how should the forecasts be adjusted?

At what level in the hierarchy, over what time horizon, and by how much (percents or quantities)?




Criteria for combining or adjusting forecasts Best practices for forecasting include a business

process for answering the questions on combining and adjusting. Tool is the demand management of the S&OP process

Tips for combining forecasts Use different data for different models Use equal weight unless there is strong evidence Use expert knowledge to vary the weights

Collect historical data on weight accuracy




Criteria for combining or adjusting forecasts Tips for adjusting forecasts

Adjusting forecasts can be necessary when… Recent events are not fully reflected in the data Experts possess reliable knowledge about future events Key variables were omitted from the models

To gain consensus for adjustment level, time horizon, and degree… Construct scenarios with representative events Ask experts to provide explanations of outcomes Be careful to avoid “boomerang” effect









A guide to choosing forecasting models References

Armstrong, J. Scott and Kesten Green. 2009. Selection Tree for Forecasting Methods. Forecasting Principles (April) http://www.forecastingprinciples.com (accessed May 2009).

Bry, Brenna H., P. McKeon, and R.J. Pandina. 1982. Extent of drug use as a function of a number of risk factors. Journal of Abnormal Psychology 9: 273-279., in Armstrong J. Scott, ed. 2001. Principles of Forecasting: A handbook for researchers and practitioners. Norwell, Mass.: Kluwer Academic Publishers.

Green, K.C. and J.S. Armstrong. 2007. Structured analogies for forecasting. International Journal of Forecasting 23: 365-376., in Armstrong J. Scott, ed. 2001. Principles of Forecasting: A handbook for researchers and practitioners. Norwell, Mass.: Kluwer Academic Publishers.

Documents

Session 6: A guide to choosing forecast models Demand Forecasting and Planning in Crisis 30-31 July, Shanghai Joseph Ogrodowczyk, Ph.D