Forecasting Revenues in Ancillary Markets - smawins.com New Market Forecasting 180114.pdftypes of timing delays. ... He holds an MBA in Strategic ... Traditional approaches to revenue

©2018 SMA, Inc. All Rights Reserved.

Forecasting Revenues

in Ancillary Markets

Ajay Patel and Eddie Solares

Technical Report

11 January 2018, V1

Forecasting Revenues in Ancillary Markets 1

Technical Report 11 January 2018, V1


Summary

Most companies use an expected value formula to forecast revenues from a sales or project opportunity

pipeline. The expected value is typically calculated as the aggregate of the estimated likelihood of the

opportunity being real P(go), times the likelihood of the company winning the opportunity P(win), times

the estimated annual project revenue. This approach is accurate when two conditions are met: 1) when

there are a large number of opportunities in the pipeline and 2) when timing of the project is well known.

However, when companies start to rely on new or ancillary markets for a larger share of the future rev-

enues, these two conditions of the traditional forecasting approach are rarely met for a variety of reasons.

Though new and ancillary markets may be a significant source of future revenues, companies typically

focus on few, large projects to optimize their investment (thus, any single outcome can significantly alter

the results from an expected value), and may not have the necessary insight to realistically estimate the

factors in the expected value formula, especially timing. This leads to the statistics behind the traditional

expected value formula breaking down and giving erroneous results. This could lead to dramatic sur-

prises in financial results from a win or loss of a single large project that was incorporated in the expected

value based forecast. Thus, traditional forecasting techniques are not adequate nor well suited to predict

future revenues for firms pursuing growth in new and ancillary markets. In this study, SMA’s analysis

shows that the traditional expected value approach systematically over-estimates forecasts by as much

as 60% in the near-term, and creates more than a 58% likelihood of a significant negative revenue

surprise.

The study shows that project timing is the largest source of forecast inaccuracy for a typical project pipe-

line. For the illustrative pipeline example used in the study, the small sample size of high value projects

typical of pursuing an ancillary market accounted for approximately 12.5% of the forecast inaccuracy,

whereas uncertainty in project timing is responsible for 50% (the remainder is from the inherent uncer-

tainty modeled by P(go) and P(win)). The study proposes a method to incorporate project timing

uncertainty as a fourth factor P(t) in the expected value formula. The proposed approach can be easily

implemented by the finance department, and as a component to monitoring business development

activities. If it were the case that a project decision and contract award was always as planned, then

within one standard deviation we would expect that the results of the traditional expected value formula

would be equal to the actual results that occur in real life. When you add the uncertainty that the contract

might not be awarded the year it is scheduled, a clearer picture emerges as to why the traditional

approach is not as robust as we would expect it to be. Using a Monte-Carlo method to simulate real life

scenarios, the study shows that by considering realistic delays we over-estimate near-term revenues by

20% to 60% using the traditional expected value formula. The more disconcerting result is that within

one standard deviation, the forecast can be as good as 10% or as bad as 150%. These are not the kind of

results the finance division of a company wants to be dealing with when planning company revenues.

So if timing is an issue how can we address it without completely scrapping the traditional approach?

In the study, we develop a unique approach to the traditional expected value approach where we

introduce a matrix that incorporates a timing delay on a year-by-year basis. The matrix is designed to

only take in one additional timing probability to simplify the approach. Running the same Monte-Carlo

simulation with this added matrix, we get a new expected value formula that improves accuracy by as

much as 57% and reduces the likelihood of any surprise by 30% within one standard deviation. This

added fix to the traditional approach can be modified and tailored to suit historical data and fit multiple

types of timing delays. It also is a relatively easy fix to the traditional approach and does not overly

complicate an already uncertain process that has been known to be difficult to pin down.




Contents

1. Introduction ....................................................................................................................................................... 3

2. How Forecasts are Traditionally Calculated .................................................................................................. 4

3. The Challenge of Ancillary Markets ............................................................................................................... 5

4. Study Design Overview .................................................................................................................................... 7

5. Static Pipeline Design for International Opportunities in A&D .................................................................. 8

6. Creating Realistic Futures (i.e. Simulations) ................................................................................................ 13

7. Timing: The Hidden Third Parameter .......................................................................................................... 16

8. Obstacles in Forecasting ................................................................................................................................. 19

9. The New Expected Value Formula ............................................................................................................... 20

10. Expected Value vs. New Expected Value Analysis ................................................................................... 23

11. Conclusions and Recommendations ........................................................................................................... 25

Appendix A .......................................................................................................................................................... 26

Appendix B .......................................................................................................................................................... 27

About the Authors

Ajay Patel is President and CEO of SMA with over 30 years of strategy consulting, business development,

operations, program management, and systems engineering experience. He holds an MBA in Strategic

Planning and Finance from USC and a BS Physics from John Hopkins University. Eddie Solares is a

management consulting analyst at SMA with research experience focused on statistical analysis of large

datasets using programming languages. He holds a MS in Physics from UCLA and a BS in Astrophysics

from UCSC.




1. Introduction

Most companies today rely on revenues from ancillary or adjacent markets as a part of their core growth

strategy. Revenues from these pursuits have been typically difficult to forecast, largely because of mis-

understood nuances of those markets especially customer buying behaviors and processes. To a new

entrant, and even established competitors, these markets can appear to have murky decision mecha-

nisms, informal influence networks, and significant uncertainties with customer budgets and needs.

This has been particularly true for project-driven industries pursuing international opportunities, such

as Aerospace and Defense, Engineering and Construction, and public sector consultancies. These pursuits

drive critical decisions on allocation of resources, especially for large projects that the company is relying

on to “make their numbers.”

Our experience working with clients across many industry sectors has been that revenue forecasting in

these markets has been persistently challenging, even for companies that have been in ancillary markets

for many years. The factors that are considered include estimates of the value and scope of the oppor-

tunity, the uncertainty of the project (or purchase) moving forward, the firm’s competitive position and

timing of the award. Traditional approaches to revenue forecasting clearly do not work for these types

of markets. The most widely used approach is an expected value calculation for each opportunity that

discounts the annual revenue estimate by a probability of the project moving forward and a probability

of winning the contract. The reason why this traditional approach fails in ancillary markets is that the

opportunities typically tend to be large in revenue and few in number, thus the uncertainties and out-

come of any individual opportunity can easily affect the overall revenue forecast and financial result.

To compensate for these uncertainties and the possibility of a surprise from a single project outcome

(win or loss), experienced firms typically use a rule-based or ad hoc decision as to whether to include

or not include a specific opportunity in the forecast. Though this approach compensates for overall

uncertainty, it also creates a systemic bias and relies heavily on individual judgement resulting in a

process where the forecasting accuracy is not predictable. See Appendix A for more details.

In this paper, we demonstrate through statistical simulations that the traditional forecasting approach

does not accurately predict realistic scenarios. The traditional approach systematically over-estimates

forecasts by 22–54% in the near-term, and creates more than a 58% likelihood of a significant negative

revenue surprise. We address the shortcomings of the traditional approach with an easily implemented

new expected value formula that improves accuracy by as much as 57% and reduces the likelihood of a

surprise by 30%. The new formula accounts for the following attributes: 1) the probability of the project

being real, 2) the probability of being awarded the project and 3) the probability of the contract being

awarded at the planned start year. We use an example of an aerospace and defense firm pursuing

international projects for illustration; but the approach can be implemented to any ancillary markets or

industry sector. Note that this paper applies only to revenue forecasting from new opportunities, and

does not address revenues from existing backlog.




2. How Forecasts are Traditionally Calculated

The traditional approach to revenue forecasting is to use the expected value formula. The expected

value is an anticipated value for a given sales opportunity. The general expected value (EV) formula in

revenue forecasting is given by

EV𝑗[𝑡] = P𝑗 × R𝑗[𝑡] (1)

where P is the probability of opportunity j, and R is revenue of the opportunity at some time t.

Unfortunately, if either the opportunity itself is not certain or the confidence of winning is wrong, then

the forecast can be significantly misrepresented. For this reason, most companies uncouple these two

factors into individual probabilities: the probability of the opportunity occurring and the probability

that the opportunity is won. In this manner, the EV formula then becomes a realistic revenue

forecasting method. The modified formula uses these two probability variables as a compounded

probability:

P𝑗 = P𝑗(go) × P𝑗(win|go = 1)

The probability P𝑗(win|go = 1) is the conditional probability of winning given that the opportunity is real

(i.e. the project or purchase actually proceeds; whereas P𝑗(go) is the independent probability of the

project or purchase occurring. This enables an intuitive approach to estimating the confidence of each

opportunity in the revenue forecast. Taking all the opportunities in a pipeline, the sum of all the EV

revenues gives the total forecasted revenue. Therefore, the total revenue forecast F from an opportunity

j (of a total of n opportunities in the sales pipeline) for a given year i can be written as

F𝑖 = ∑ EV𝑗[𝑡𝑖] = ∑ 𝑛𝑗=1 P𝑗

𝑛𝑗=1 (go) × P𝑗(win|go = 1) × R𝑗[𝑡𝑖] (2)

where EV is the expected value, t is time, P are the probabilities and R is the revenue for the

opportunity. The traditional expected value method is a simple calculation, only requires two

probability estimates, and is intuitive for management and finance departments. However, it is a

statistical approach and requires a large number of opportunities for the statistics to give accurate

results which stems from the mathematical law of large numbers.




3. The Challenge of Ancillary Markets

When pursuing sales opportunities in ancillary markets, there are sources of uncertainty that are hard

to manage. This is in comparison to core markets where the company has years of experience under-

standing buying processes and has experiential knowledge that can be applied to make reasonable

judgements on the viability of a project and the firms competitive position to win, namely P(go) and

P(win|go = 1). When pursing and competing in ancillary markets, as in Figure 1 for example, lack of

selling experience, lack of deep customer intimacy and lack of access to influence networks make it

difficult to ascertain project viability and competitive position.

Figure 1: Classification scheme tailored to revenue planning for new market opportunities. Although category E is

included for completeness, it is excluded in study once opportunities are classified.

Category E Dropped

No financial impact within the first five

years

Category BRevenue

Category D Revenue

Category ARevenue

Category C Revenue

Medium Outlook

Good P(go) probability

and decent P(win)

probability

4-5 Year Revenue

Meaningful financial

impact starting on the

fourth year

Good Outlook

High P(go) probability

and good P(win)

probability

High Outlook

High P(go) and

P(win), most likely

won opportunity




Furthermore, some markets have inherent uncertainties with regard to customer needs, budgets, project

timing, and other factors that further complicate an already complex situation. This is particularly true

when pursuing international public-sector projects.

As an example, aerospace and defense firms have been pursing growth in international markets for the

past five years as a key complement to their core domestic business. These are typically large projects,

but few in number where winning or losing a handful of contracts can have an appreciable impact on

revenues. Companies in many industry sectors have faced similar obstacles on their path to globalization:

inability to make progress within the formal or imputed buying process, blind spots that arise from

pre-existing biases, influence levers that are obscured or concealed, and general difficulty discerning

“signal from noise” as an outsider looking in.

As these firms rely more on international sales, these obstacles make forecasting revenue with reasonable

certainty difficult. The probabilities P(go) and P(win), the program value and the start year of these

opportunities, are much less certain for international opportunities. On top of that, they are relatively

few in number but each with a much larger potential value creating the potential for significant revenue

surprises from a single loss or win. Therefore, in order to create a realistic pipeline we suggest that

companies include these factors explicitly as a way to categorize and prioritize opportunities in their

sales pipeline. An example screening process of a firm pursing international opportunities as an ancillary

market is illustrated in Figure 1. In this example, five categories to group each individual opportunity

are created to tackle the problem. The categories defined are generically referred to A, B, C, D, and E,

each has distinct attributes of timing, program viability and competitive position. We use this structure

to model traditional and alternate forecasting methodologies. Our models are limited to a five-year

horizon, so opportunities in category E are dropped from this study due to its lack of impact in the

revenue planning horizon.




4. Study Design Overview

Our study systematically tests the traditional approaches of forecasting revenue at a detailed level to

gain insight into the sources of error and develop improved techniques. For our example in Figure 1 of

penetrating foreign markets, the following is an outline of our study design:

1. A static sample pipeline is created as a baseline to test the accuracy of different forecasting techniques.

The pipeline consists of 1,052 opportunities (1,000 domestic-market and 52 international-market

opportunities). This sample size choice was to simulate the small number of opportunities in an

ancillary market, compared to the large number of core-market opportunities. For each opportunity,

the P(go) and P(win) probabilities are assigned according to pipeline category shown in Figure 1.

2. The accuracy of the traditional forecasting method is tested by creating 1,000 future scenarios of the

pipeline using a Monte-Carlo technique. Each future scenario is a statistical simulation of each

opportunity going forward as a real program or purchase, and whether the company wins or loses

the competition for the opportunity. These binary outcomes are tested against the expected value

calculation. The difference between the sum of the expected values and the sum of the binary

revenues is a measure of the accuracy of the forecasting technique.

3. We analyze the sources of variability to better understand the accuracy of the traditional forecasting

technique. We demonstrate that most significant source of forecasting errors (and systemic revenue

misses) is with the estimate of the program or purchase planned start year versus actual start year.

4. Finally, an alternative forecasting method is developed to improve forecasting accuracy by consider-

ing a third probability parameter that takes into account the predicted program start year versus the

actual start year of the program.




5. Static Pipeline Design for International Opportunities in A&D

The first step in our outline in Section 4 requires us to design the pipeline for our international market

example. A pipeline of new business opportunities is created in order to test the different forecasting

method approaches. A typical pipeline for a public sector contractor that is relying on international

opportunities as part of their core revenues is emulated.

To create a methodical way to categorize new market opportunities, a classification scheme based on

five categories that allows the differentiation and assignment of realistic variables for the probabilities

P(go) and P(win) is created. As stated Section 4, 1000 domestic opportunities (250 in each category

A/B/C/D) and 52 international opportunities (13 in each category A/B/C/D) are created. We randomly

assign a revenue value of $100 million, $500 million, or $750 million for all 1,000 domestic opportunities

giving a total domestic revenue of $212,500 million.

Similarly, revenue values of $500 million, $750 million, or $1,500 million for each of the 52 international

opportunities is assigned at random totaling $41,750 million in international revenues. In this model,

the international pipeline is 16% of the total. The distribution of these revenues can be seen in Figure 2.

These values are kept static throughout the study.

Figure 2: Revenue distribution across 1,000 domestic and 52 international opportunities.

Domestic

International

Opportunity Value, $ million

Number of Opportunities

Total Revenue, $ million

Opportunity Value, $ million

Number of Opportunities

Total Revenue, $ million

$100 750 $75,000

$500 25 $12,500

$500 200 $100,000

$750 15 $11,250

$750 50 $37,500

$1,500 12 $18,000

Totals 1000 $212,500

Totals 52 $41,750

From Figure 2, we can also see that the international opportunities are fewer quantity and larger in rev-

enue value. In general, the number of opportunities in ancillary markets is usually much less than core

markets. This comes with the territory of exploring and penetrating new markets. The larger revenue

reflects how companies prioritize opportunities; they generally seek new opportunities where the reve-

nue is worth the risk of entering ancillary markets.

Now that the revenue distribution is established, start years for each opportunity are assigned in order

to configure our pipeline. As Figure 3 shows, the 5-year pipeline is simulated with each opportunity

having a different start date. The opportunities in categories A, B, and C are apportioned to start in

different years based on a random seed such that 70% of the opportunities will start in 2018, 20% will

start in 2019 and 10% start in 2020.

Figure 3: Start year percentage distribution based on category for each opportunity.

Category Start Year Percentage Distribution

2018 2019 2020 2021 2022

A/B/C 70% 20% 10% 0% 0%

D 0% 0% 0% 70% 30%

For category D, based on our definition of the category in Figure 1, it is seen that this opportunity does

not have a meaningful impact in the first three years so the start year percentages do not begin until the

year 2021. 70% of the opportunities in category D are assigned to start in 2021 and 30% start in 2022. Note

that category E is excluded from the study since by design those opportunities have no significant




revenue impact in the first five years. Once start years are assigned to each opportunity, for both domes-

tic and international, they are kept static throughout the course of the study.

The last step to finalize the baseline pipeline is to distribute the recognition of revenue for each oppor-

tunity over the years that the program is executed. We assume that each program is executed over a 5--

year period and has a simple increasing profile defined in Figure 4. The revenue recognized in year 1 is 2/20th of the total, increasing to 6/20th in year 5, resulting in recognizing 100% of the revenue over 5 years.

The increasing program revenue profile is not necessarily typical across all types of programs or pur-

chases, but compensates for the burn-off of backlog resulting in an overall growth forecast for the firm.

Figure 4: Revenue distribution for compensation of burn-off backlog of program opportunities.

Year Revenue Distribution

Year 1 2/20 x Revenue





For visualization purposes, Figure 5 shows a graph of the revenue distribution by category for domestic

and international. Note that the total international revenue is about a quarter of the total domestic

revenue.

Figure 5: Total revenue for both domestic and international category.

By design, an even distribution split among categories is noted. Recall that of the 1,000 domestic oppor-

tunities, 250 were assigned to each category and for the 52 international opportunities, 13 were assigned

to each category. To view the ramp up in revenue the graph of the revenue distribution by domestic and

international we graph it in Figure 6 and by the categories in Figure 7. Note, that in alignment to the

model design, category D does not begin until after 2020.

$0

$50,000

$100,000

$150,000

$200,000

$250,000

Domestic International

Reve

nue

($ m

illio

n)

Domestic and International

A B C D




Figure 6: Revenue distribution per year for domestic and international opportunities.

Figure 7: Revenue distribution per year based on category distribution.

Now that the category, revenue, and start year distributed across all five years of our planning horizon

are created, probabilities are assigned for each opportunity. A systematic way of assigning the probability

of the opportunity occurring P(go) and the probability of winning the opportunity P(win) needs to be

created. Recall that in Figure 1, categories with qualitative definitions were classified among opportuni-

ties. In order to quantify the P(go) and P(win), a percentage range is assigned to each category for both

probabilities. The model must also take into account that ancillary markets have much smaller probabil-

ities due to the inherent risks a company takes in doing business in unknown markets.

$0

$10,000

$20,000

$30,000

$40,000

$50,000

$60,000

$70,000

$80,000

$90,000

2018 2019 2020 2021 2022

Reve

nue

($ m

illio

n)


$0

$10,000

$20,000

$30,000

$40,000

$50,000

$60,000

$70,000

$80,000

$90,000

2018 2019 2020 2021 2022

Reve

nue

($M

M)

A B C D




To assign probabilities based on our category distribution, probability ranges are assigned according to

Figure 8. These probabilities are connected to the attributes of each pipeline category as defined in Figure

1. A random sampling based on these ranges is done and assigned a probability within the range. Con-

sistent with how the parameters are established, the probabilities of P(go) and P(win) are kept static

throughout the duration of the study.

Figure 8: Probability ranges for each category based on type of opportunity and category of opportunity. Category E

opportunities are removed from study but included for completeness.

Category Probability (Go) Probability (Win)

Do

mesti

c

A 90% – 100% 80% – 90%

B 80% – 100% 70% – 80%

C 70% – 100% 50% – 70%

D 50% – 100% 50% – 70%

E < 50% < 50%

Inte

rna

tio

nal A 70% – 100% 80% – 90%

B 60% – 100% 70% – 80%

C 50% – 100% 50% – 70%

D 30% – 100% 50% – 70%

E < 30% < 50%

Figure 9 column headings reflect all static variables that have been created so far in our pipeline. The

expected value is calculated in Figure 9 using the formula EV = P(go) × P(win) × Revenue. This is done

for all 1,000 domestic opportunities and all 52 international opportunities. These values are all kept static

for the remainder of the simulation in order to keep the relative comparisons free of any statistical bias.

Note that the expected value is less than the total pipeline, which makes sense since not all opportunities

will result in revenue. The reader is reminded that the aggregate sum of all expected values of the op-

portunities should equal the realistic revenue stream given a large enough sample size.

Figure 10 graphs the expected revenue from all opportunities over our 5-year planning horizon. As

described above, this is the aggregate of the expected revenue from each opportunity estimated by

multiplying the recognized revenue in each year by the two probabilities. Now that the static pipeline

is fixed, statistical simulations can be conducted and results tested to see whether or not the expected

value is an accurate measure for forecasting revenue. Forecasted revenue will be used interchangeably

with the expected value (EV).

Figure 9: Sample of static pipeline created showing opportunity number, whether domestic D or international I, cate-

gory, planned start year, two probabilities assigned to them, total opportunity revenue and expected value (EV) calcu-

lated from traditional formula.

Opportunity Dom/Int Category Year P(go) P(win) Revenue $ million EV, $ million

1001 D A 2018 90.52% 83.52% $100 $75.61

1002 D A 2018 99.19% 82.45% $500 $408.92

1003 D A 2018 94.45% 88.53% $500 $418.12

1004 D A 2018 91.32% 87.78% $500 $400.82

1005 D A 2018 95.72% 83.73% $100 $80.15

1006 D A 2019 91.86% 81.50% $500 $374.33

1007 D A 2019 94.44% 84.01% $100 $79.33

1008 D A 2020 99.84% 88.56% $100 $88.42

… … … … … … … …




Figure 10: Sum of EV throughout five years for both international and domestic.

$0

$5,000

$10,000

$15,000

$20,000

$25,000

$30,000

$35,000

$40,000

$45,000

$50,000

2018 2019 2020 2021 2022

Reve

nue

($ m

illio

n)





6. Creating Realistic Futures (i.e. Simulations)

Moving on to Step 2 of the study approach (see Section 4), testing of the traditional forecasting method-

ology is done to see how well the expectation value formula predicts future revenues. Future scenarios

are created by sampling from the probability distributions defined in the previous section; each scenario

represents the collection of outcomes of all opportunities determined from the sampling. For each op-

portunity, a random number is generated representing the outcome for the variable P(go) to determine

the outcome pertinent to whether the project will proceed or not. If the random number is equal to or is

less than the defined P(go), then the project will be awarded (i.e. it is assigned a value of 1) and if is

greater than P(go) then the project was not real (i.e. it is assigned a value of 0). We do the same for P(win).

To avoid any statistical correlations, different random samples for each probability are used. As an exam-

ple, if a random sample of s = 0.78 is generated and compared to a probability of P(go) = 0.92, it is seen

see that 0.78 < 0.92 and thus falls within the probability, so a binary value of B(go) = 1 is assigned. If,

however, a random sample of s = 0.96 is generated then it is greater that P(go) = 0.92 and thus assigned

the binary value of B(go) = 0. The actual scenario value (SV) is then calculated by SV = B(go) × B(win) ×

Revenue. This is done for all 1,000 domestic opportunities and all 52 opportunities.

From Figure 11, it is seen that if either or both opportunities get a binary value of 0, the entire scenario

value is $0.00. This scenario value simulates an actual real-life scenario where the opportunity is either

won or lost.

Figure 11: Pipeline with random sample drawn which assigns binary 0 or 1 to B(go) and B(win) to calculate Scenario

Value (SV).

Opportunity Dom/Int Category Year P(go) P(win) B(go) B(win) Scenario Value, $ million

1245 D A 2018 90.27% 89.63% 1 1 $500

1246 D A 2018 96.59% 87.38% 1 1 $100

1247 D A 2019 90.77% 86.47% 1 1 $500

1248 D A 2018 98.09% 88.52% 1 1 $750

1249 D A 2019 93.50% 80.41% 1 1 $100

1250 D A 2018 98.91% 84.69% 1 0 $0

1251 D B 2018 91.35% 74.05% 1 1 $100

1252 D B 2018 82.11% 74.91% 0 1 $0

1253 D B 2019 97.67% 77.04% 1 1 $100

1254 D B 2018 91.45% 79.65% 1 0 $0

1255 D B 2018 85.74% 70.23% 1 0 $0

1256 D B 2018 99.29% 77.34% 1 1 $100

… … … … … … … … …

From the law of large numbers, given a large enough sample size, it is expected that the sum of all the

expected values (EV) equals the sum of all scenario values (SV), that is SV – EV = 0. The sum of all of the

EV and the sum of all SV in our pipeline is taken and the difference between them for each of the five

years give us our percent difference. That is, the percent difference can be written as

% Diff = Σ(SV − EV) = ∑[B(go) × B(win) - P(go) × P(win)] × R (3)

where R is the revenue. This is done for both the domestic and the international opportunities. This pro-

cess counts as one simulation that produces one value for the percent difference. In order to generate a

standard deviation and determine the accuracy of our results 1,000 simulations are done.




Once the 1,000 runs are completed, the average of these is taken along with the corresponding standard

deviation of each. The mean value is calculated and an error bar of one standard deviation is assigned

for each of the five. The process for running one of these simulations can be quite long so the Python

programming language is used to quickly run 1,000 of these simulations.1

Running and plotting the percent differences between the expected value (EV) and the scenario value

(SV) for all 1,000 runs and taking the average percent difference gives us the results of Figure 12. Fore-

cast for core domestic markets is spot on and there is very little mean error between the differences. This

means that SV = EV on average and there is very little variation within the error bars. The standard

deviation, or error bar, is less than 5% for the domestic market and is relatively small which reflects the

high number of opportunities.

Figure 12: Percent difference between the expected value (EV) and the scenario value (SV) with one standard deviation

as the error bar.

From a mathematical standpoint, this reflects the law of large numbers since there is a large sample size

of 1,000 opportunities the results tend to not deviate as much since the EV approaches SV as the sample

size nears infinity. The mean percent difference for the international ancillary market is also relatively

small. It is still slightly larger than for the domestic opportunities, but the difference is within 5% points

of the 0% difference. However, the error bar is considerably larger than the domestic one with values

greater than 15%. This is reflective of the sample size since there are fewer opportunities, 52 to be exact,

and each individual opportunity has a more significant impact on the revenue. A point should be made

to state that the size of the error bar is what indicates whether the traditional approach has flaws—

namely, if any single future scenario could result in a significant revenue forecast miss. This is mostly

1 Programming languages are useful to do routine calculations in a quick and efficient way. Since the simulation is

run 1,000 times, it is much easier for a computer to do this job as opposed to manually recalculating the results

one by one. High-level programming languages are useful for these types of calculations as they are designed for

general purpose programming. The programming language used in this study is Python, but there are other high-

level languages such as C++ and Java that can similarly be used.

-0.3

-0.25

-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

0.25

2017 2018 2019 2020 2021 2022 2023

Perc

ent

Diff

ere

nce (1

-E

V/S

)

Domestic International Combined




driven by the wide range of possible outcomes for smaller number but larger in value of the international

opportunities. However, from these results two things can be incorrectly assumed:

1. The traditional expected value represented in Equation 1 is a good way to forecast revenue;

2. Having a larger sample size reduces the standard deviation of your results.

We will next show that the first assumption is incorrect since there is a critical hidden parameter that is

assumed to be true that is not considered. We will also address the second assumption in our recommen-

dations for an improved forecasting approach. It will be seen that the true error bars of the traditional

forecasting method defined by Equation 1 will be large and the flaws of these results will be evident once

the hidden parameter is introduced.




7. Timing: The Hidden Third Parameter

The results for the traditional expected value seem acceptable notwithstanding the possible wide range

of outcomes, but is missing an important aspect of forecasting. The timing of the opportunity is critical

to forecasting revenues and is the additional hidden parameter that failed to be taken into account. This

source of uncertainty will now be considered in the next step of the analysis (see Section 4). In the prior

analysis, it was assumed that the planned start year was the actual start year of the opportunity. From

experience, this is not realistic especially in public sector driven markets and in ancillary markets where

there is less budgetary and decision transparency. It is also known that forecasters have a significant

systemic bias for optimism. There is a tendency for companies to over-estimate their revenue in fore-

casting models for new markets and timing plays an important factor. Thus, it is crucial to estimate the

probability for the planned start year to be the actual start year. This introduces a new statistical param-

eter when running the future scenarios that was not considered before.

The question now becomes “how is uncertainty in timing simulated?” when running the future scenarios.

For each opportunity, when the dice is rolled for each future scenario value, not only is the probability

of whether or not the program happens and if it is won simulated but also when it occurs. To do this a

probability is associated with each start year and the remaining probabilities are pushed out to later

years. That is, there is a certain probability it starts in the planned year and if it does not fall under this

probability the project gets pushed out a year or two down the line. Using the timing probability matrix

in Figure 13, probabilities are assigned for each of the five years of forecasting revenues. From the

matrix, it is seen that for domestic opportunities with a planned year of 2021 there is a 65% chance it

actually starts in 2021, a 20% chance that it starts in 2022, and a 15% chance that it occurs in 2023 or

later. Since only five years revenue is considered in the forecasting horizon, this means that there is a

15% chance that this opportunity is not included in the model since it is pushed out past our five-year

range. This is similar to treatment of category E opportunities that do not have a financial impact within

the immediate five years that the revenue is forecasted.

Figure 13: Probability matrix of planned year versus actual start year.

Planned Year Probability Domestic Actual Year

2018 2019 2020 2021 2022 ≥ 2023

2018 80% 80% 15% 5%

2019 75% 75% 20% 5%

2020 70% 70% 20% 10%

2021 65% 65% 20% 15%

2022 60% 60% 40%

Planned Year Probability International Actual Year

2018 2019 2020 2021 2022 ≥ 2023

2018 80% 60% 25% 15%

2019 75% 55% 20% 10% 5% 1000%

2020 70% 50% 15% 5% 30%

2021 65% 45% 10% 45%

2022 60% 40% 60%

For this model, inferences were made based on reasonable assumptions from core and ancillary markets.

For core domestic markets, businesses are usually well informed on the timing and scope of the oppor-

tunity start dates and so high probabilities are assigned for the planned year. However, as the program

lies further out in the timeline, there tends to be less certainty as the opportunity may not be mature




enough to have a well-planned start year. For international ancillary markets, there is less certainty in the

assumption that the planned start year is the actual start year. Whether that is an attribute of the market

or lack of intimacy and knowledge of the market, higher uncertainty is generally the case. Similar to the

domestic case, the probability for international opportunities also degrades as planned years moves

further out in the timeline. It is noted that since international opportunities planned start years are less

certain than domestic ones, more international opportunities than domestic opportunities will pushed

out beyond the five-year scope of our revenue forecast.

To provide some backup to these numbers, a study done on 18 major defense acquisition programs by

SMA looked into the delay for both the requests for proposals and the program awards in recent

contracts.2 For the Request for Proposal (RFP) delays, there was a 78% probability it was awarded the

same year it was planned, 17% within two years, and 6% within the three years.

Similarly, for the program award delays, there was a 72% probability it was awarded within same year

it was planned, 17% within the two years, 6% within three years, and 6% within four years. Taking the

average of these delays gives roughly 75% probability that there is a delay within the year it was planned,

17% chance of delay within two years, 6% chance of delay with three years, and a 3% chance of delay

within four years. These represent domestic program awards within the United States and the proba-

bilities assigned to the probability matrix in Figure 13 are roughly within the same order of magnitude

for the domestic opportunities. This is a good indicator that the numbers used are accurate representa-

tions of opportunity awards in core domestic markets. For international markets, it is well known these

probabilities are less certain, so probabilities assigned are smaller than their domestic counterparts.3

When running the scenario simulation, similar calculations as before are done. The reader is reminded

that the static values in the pipeline did not change. The only change is that this iteration of the 1,000

scenarios is run with the added probability that the planned year is the actual start year. The percent

difference formula is thus modified as follows

% Diff = ∑[B(𝑡) × B(go) × B(win) - P(go) × P(win)] × R (4)

where B(𝑡) is the binary value assigned depending on whether or not the planned year is the start year.

This value is assigned similarly to the other binary values using the probability values from Figure 13.

Using Equation 4, the results are graphed and the percent differences are shown in Figure 14. There are

some dramatic differences from this graph than that of Figure 12. The only similarity between them is

that the mean percentage difference is less for domestic opportunities than the international ones and

that as the years progress they get closer to 0% difference with smaller standard deviations.

2 The study done was a SMA funded project that analyzed the request for proposal (RFP) and award dates of

major defense acquisition programs (MDAP). 3 “Analysis of Major Defense Acquisition Programs (MDAP) Award Delays.” SMA,

www.smawins.com/Content/Files/SMA%20MDAP%20Award%20Delays%20171211.pdf.

http://www.smawins.com/Content/Files/SMA%20MDAP%20Award%20Delays%20171211.pdf




Figure 14: Calculation of percent difference between expected value (EV) and scenario value (SV) with additional prob-

ability of timing delays added to scenario calculations.

The percentage differences for both the domestic and international opportunities do not lie close to 0%

difference. This implies that there is a systemic bias (over-estimation) of approximately 20% to 60% in

the near-term forecast. In 2018 for the domestic case, the percentage difference is bigger than 20%. In the

latter years, one can see that the percentage difference starts to converge close to the 0% axis meaning

that the expected value results become closer to reality. The error bars on the domestic cases for the

standard deviation are relatively small so there is little variability as before.

As for the international case, there is a large percent difference of about 80% in 2018 with the years con-

verging to approximately 18% in the later years. The more disconcerting result is that within one stan-

dard deviation a large percentage difference of up to 150% may be possible. Putting this into context,

the revenue forecast can be as low as 10% or as high as 150%.

This is clearly a dramatically erratic result and tells us that timing is highly important when considering

forecasting revenues. Across the board, it is known that many companies have international revenue

forecasting models that give drastically inaccurate results. From doing these simulations, it is seen how

timing of the program start year is an attribute to factor into the expected value to develop a more

accurate forecast.

-1.8

-1.6

-1.4

-1.2

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

2017 2018 2019 2020 2021 2022 2023

Perc

ent

Diff

ere

nce (1

-E

V/S

)





8. Obstacles in Forecasting

When entering new markets there are three underlying issues that create obstacles to forecast future

revenues accurately:

1. Small Sample Sizes: The relatively small number of opportunities in new or ancillary markets is inherent

with the pursuit of growth in these markets. Small sample sizes do not lend themselves to simple

statistical methods that can be easily and intuitively incorporated in revenue forecasting.

2. Large Revenues: When entering a new or ancillary market, firms typically focus on projects with

relatively large revenues to balance the pay-off against investment risk. These pursuits require

significant investment of resources especially since the firm is not as familiar with the market and

may be in a less advantageous competitive position. The large project revenue means that each

opportunity is critical to the total forecasted revenue, thus outcomes can vary significantly on a

single win or loss.

3. Timing: One of the most significant sources of uncertainty is timing. This is particularly true in public

sector markets where projects are not necessarily driven by an urgent competitive situation and sub-

ject to government budgets and lengthy approvals. As seen from the results of Section 7, including a

probability estimate for the planned start year in our testing creates drastically different results in

estimating forecast accuracy.

Each of these can have a significant impact on forecasting revenue techniques, specifically the expected

value formula. For the first case, there exist statistical methods to handle small sample sizes. However,

these statistics are difficult to implement as part of a routine business planning function and are non-

intuitive. Secondly, being more selective about which opportunities to include or exclude from the fore-

cast based on the static values assigned to the opportunity such as revenue and probabilities can be done.

However, eliminating too many of these opportunities limits the sample size and introduces the problem

inherent in the first case. See Appendix A for more details.

Also, selectively excluding opportunities introduces additional biases since it involves non-Bayesian

estimation and is largely done either ad-hoc or with arbitrary rules. The third issue can be effectively

addressed by modifying the traditional expected value formula with a probability estimate associated

with the planned start date. This approach also helps mitigate the first two issues by help push out the

revenues even more than the traditional expected value. Introducing an estimate of the probability that

the actual start date is the planned start date is consistent with the Bayesian approach of the traditional

forecasting methodology. The probability estimates can be informed and validated from historical data

in the individual markets. Although each approach can improve the predictive capability of the model,

our research indicates that addressing the timing issue with a new probability can significantly improve

forecasts and is a simple enhancement to revenue planning process being used today at most firms. Our

analysis also shows that small sample size statistics typically associated with pursuits in ancillary markets

only account for a small percentage of forecast inaccuracy when compared to addressing the timing issue.

See Appendix B for more details.




9. The New Expected Value Formula

The introduction of an estimate of project timing uncertainty (i.e. the probability that the actual start year

is the planned year) will be added to improve the traditional expected value-based forecasting approach.

This requires the need to define a matrix that distributes the expected value based on probabilities of

planned start year versus actual start year. This can be simplified by using a single probability value

from which subsequent values in the matrix are derived.

Since the total revenue of the opportunity is already distributed throughout the five years, the probability

associated with timing to each of these five years is assigned and revenues are pushed with into the

later years based off the probabilities. This creates the matrix redistribution of revenue that is used to

create the New Expected Value (NEV) formula. Modifying the expected value formula from Equation 2,

the formula is written as

NEV[𝑡𝑖] = ∑ ( P(go) × P(win) × R[𝑡𝑖] × Mik[P(time), 𝑡𝑖])5𝑘=1 (5)

where 𝑖 is the planned year, 𝑘 is the actual year, P(time) is the probability that the planned year is the

start year and M[P(time), t ] is the matrix associated with the timing of the opportunity. This formula

redistributes out the revenue throughout the five years based on the individual timing probability.

Equation 5 is abstract so an example will be shown to illustrate the type of matrix used in the new ex-

pected value formula. Figure 15shows an example of how revenue for a project is distributed throughout

each year (for both the actual start year and the planned start year). This demonstrates the how the

expected value (EV) and the new expected value (NEV) differ. The yellow shaded section represents the

matrix M and the probability associated with it.

Figure 15: Example pipeline of how revenue is distributed based on expected start year 2018.

Opportunity (j) Actual Start Year (i) Total

Planned Year (k)* 2018 2019 2020 2021 2022 >2023

Rev Potential [$100M] [$150M] [$150M] [$150M] [$125M] [$0M] [$675M]

P(go) [90%] 90%

P(win|go = 1) [90%] 90%

2018 P(t)

[80%: $65M] (1 – P(t))*0.8 [16%: $13M]

(1 – P(t))*0.2 [4%: $3M]

[$81M]

2019

P(t) [80%: $97M]

(1 – P(t))*0.8 [16%: $19M]

(1 – P(t))*0.2 [4%: $5M]

[$123M]

2020

P(t) [80%: $97M]

(1 – P(t))*0.8 [16%: $19M]

(1 – P(t))*0.2 [4%: $5M]

[$123M]

2021

P(t) [80%: $97M]

(1 -P(t))*0.8 [16%: $19M]

(1 – P(t))*0.2 [4%: $5M]

[$119M]

2022

P(t) [80%: $81M]

(1 – P(t))*0.8 [20%: $20M]

[$101M]

[NEV] [$65M] [$110M] [$119M] [$121M] [$105M] [$25M] [$547M]

[EV] [$81M] [$122M] [$122M] [$122M] [$101M] [$0M] [$547M]

[X] example calculations

* For each planned start year, the matrix is displaced by one year down and one year to the right

This entire figure represents some opportunity 𝑗 that has a planned start year of 2018. The rows represent

the planned years and the columns represent the actual start year where P(𝑡) is the probability that the

planned year is the start year. Looking at the row for planned year 2018, it is seen that it has the values

P(𝑡), (1 − P(𝑡)) × 0.8, and (1 − P(𝑡)) × 0.2 for 2018, 2019, and 2020 respectively. When the terms are

added, the resulting total is 1 indicating that the total revenue assigned to planned year 2018 is distrib-

uted out three years.




Similarly, this occurs for all the years and some of the revenue is distributed to 2023 and beyond which

is not included in our total five year forecasting range. Starting from the top, the revenue for the oppor-

tunity, the two probabilities, and the matrix columns of the revenue are pushed out from the timing ma-

trix. If the planned year is the expected year, that is P(𝑡) = 1, then only the diagonal entries of the matrix

would remain giving the traditional EV. Also, if the opportunity does not start for another n years the

entries would all be displaced n years down and n years to the right.

The matrix shows that the new expected value is lower in revenue in earlier years. This is expected since

a probability is introduced that delays the expected start year versus the actual start year resulting in

revenues being pushed out to later years. The last column in the table provides the total revenue for all

the years. Although the sums at the bottom right are equal, the NEV has $25 million allocated to 2023

and later years, which are outside the range of the five-year forecast. Another attribute to note is that

the matrix is designed to only assume one probability P(𝑡) and the probability was used to create a dis-

crete drop in revenue across the entire matrix. This makes it so that a company only needs to assign one

additional probability rather than design an entire matrix for the NEV. The delay in timing is inherently

embedded in this matrix and is easy to implement for companies developing revenue forecasting in

ancillary markets. The discrete drops in delay are simple, but more complicated and possibly continuous

model designs can fit actual historical data or known timing delays.

As done before, the 1,000 simulations are run once again and the percent difference is plotted using the

modified version of the percent difference formula given as

% Diff = ∑[B(t) × B(go) × B(win) − NEV] (6)

where NEV was given by Equation 5. Running the simulations using Equation 6, the results are shown

in Figure 16. Similar to Figure 14, the domestic percent difference is more accurate than the

international percent difference.

Figure 16: Percent difference between NEV and scenario value.

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

2017 2018 2019 2020 2021 2022 2023

Per

cen

t D

iffe

ren

ce (

1 -

NEV

/S)





The domestic percent difference also lies close to the zero percent difference line with much smaller error

bars than the international case. Again, this is because the international case has a smaller sample size

and larger revenue for the opportunities, and thus every opportunity has a significant impact on the

total forecasted revenue. A single opportunity—whether it proceeds, whether the firm wins it and when

the project starts—can drive the company’s financial results. Note that the revenue values for each year

are pushed out as before and one sees a convergence toward a value close to the 0% difference axis. As

the years pass, the forecasted revenue gets closer to the actual value. Another thing to note is the small

bump at the year 2021, which is because category D opportunities have no financial impact until year 4,

which is inherent, our study design. The start year 2018 for international opportunities has a percent

difference that is much smaller than the traditional EV results in Figure 14, suggesting the results for the

NEV formula revenue forecasting are more accurate.




10. Expected Value vs. New Expected Value Analysis

The goal of any forecast model is to predict accurately future revenues, with accuracy measured by both

average error and the standard deviation of the error when compared to simulated outcomes. For this

study, this means that the percent difference mean is as close to zero as possible with a small standard

deviation represented by the error bar. Figure 17 shows the quantitative results of the EV and the NEV

percent differences when run against the scenario values. All five years are plotted with the percent

difference and plus/minus the standard deviation. Looking at the domestic values, for the EV the results

tend to converge to around the –4% percent difference meanwhile the new expected value it seems to

converge around the 0% difference.

Figure 17: Final Comparison for all five years for both domestic and international opportunities in both traditional

expected value (EV) and new expected value (NEV).

2018 2019 2020 2021 2022

EV Domestic –25 ± 7% –10 ± 4% –4 ± 4% –6 ± 4% –4 ± 3%

NEV Domestic –10 ± 6% –4 ± 4% –2 ± 4% –4 ± 4% 2 ± 3%

EV International –82 ± 71% –34 ± 31% –16 ± 20% –17 ± 19% –16 ± 18%

NEV International –36 ± 54% –17 ± 26% –10 ± 19% –13 ± 18% –8 ± 17%

EV Combined –54 ± 58% –22 ± 25% –10 ± 16% –11 ± 15% –10 ± 14%

NEV Combined –23 ± 41% –10 ± 20% –6 ± 14% –8 ± 14% –3 ± 13%

However, for the year 2018 the EV is around –25 ± 7% meanwhile the NEV sits at –10 ± 6%. This means

that the percent difference for the EV is less accurate than the percent difference of the NEV. This shows

that the NEV gives more accurate forecasts for the domestic case with the NEV method when compared

to the traditional EV. This is a good indication of a better formula but a check of the international case

shows a much clearer picture.

For planning international revenues, the traditional EV converges to about a difference of –15%. The

NEV converges to a difference of –8%. By 2022, the error bars are about equal but the NEV has about

half the percent difference than the EV indicating the NEV converges to a more accurate forecast

revenue. Furthermore, looking at the year 2018 the traditional EV has a difference of about –82 ± 71%

while the NEV has a difference of about –35 ± 54%. The forecasted revenue for the EV in the year 2018

is erratic with a range of –11% to –153% difference within one standard deviation. For the NEV there is

a range of +19% to –89% difference within one standard deviation. This range indicates that for the

NEV at worst there is an average percent difference of the EV, which is –82%, and at best there is an

underestimate of +19%. The result shows that there is a higher probability that the NEV will accurately

forecast the revenue and achieve a 0% difference within one standard deviation. The results of Figure

17 are graphed for visual comparison in Figure 18, Figure 19 and Figure 20. These results show that

using the NEV provides a significant increase in accuracy for simulated events. The NEV would only

require one additional probability estimate (associated with the project timing) and is a simple improve-

ment to the traditional EV. The model used employs a simple matrix for our percentage distribution of

timing. An examination of historical data on project delays for the firm’s specific markets can be used

to create a more precise and refined matrix to provide better results than the simulations conducted in

this study.




Figure 18: Comparison of traditional expected values (EV) and the new expected value (NEV) for domestic.

Figure 19: Comparison of traditional expected values (EV) and the new expected value (NEV) for international.

Figure 20: Comparison of traditional expected values (EV) and the new expected value (NEV) for domestic and

international combined.

-0.35

-0.3

-0.25

-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

2017 2018 2019 2020 2021 2022 2023

Perc

ent

Diff

ere

nce (1

-E

V/S

)

NEV EV

-1.8

-1.6

-1.4

-1.2

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

2017 2018 2019 2020 2021 2022 2023

Perc

ent

Diff

ere

nce (1

-E

V/S

)

NEV EV

-1.2

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

2017 2018 2019 2020 2021 2022 2023

Perc

ent

Diff

ere

nce (1

-E

V/S

)

NEV EV




11. Conclusions and Recommendations

When running through the simulations it was discovered that the new expected value formula had a

significantly better forecasting accuracy than the traditional expected value. The new expected value

formula gives results closer to the simulated value and thus represents a better forecasting method.

This paper used an example of an aerospace and defense firm pursuing growth through international

projects. However, the recommended enhanced forecasting method is not necessarily limited to

aerospace and defense or similar public-sector firms, but extends to any business trying to penetrate

ancillary markets characterized by fewer but larger value projects. The proposed method described in

this paper can be further improved by incorporating actual project delay trends in specific markets. The

delays are typically as much a result of government processes and governance as the unique situation

of the individual project, thus enabling us to predict likely project delays. Unique knowledge can be

used to further inform the likelihood of a project delay, similar to the experiential estimates of the

probabilities P(go) and P(win) by the project leadership.

A company’s revenue forecasting approach should consider 1) the probability of the project being

awarded (i.e. is it real?), 2) the probability of winning and 3) the probability of the contract being

awarded at the planned start year. By considering this last source of uncertainty, companies can

improve their forecasts by 57% and avoid the possibility of a negative surprise by as much as 30%.

Tailoring the approach described in this paper to trends in specific markets can further improve

forecast accuracy and provide key market insights.




Appendix A

A common approach to avoid strategic surprises and moderate the forecast is to exclude certain oppor-

tunities from the forecast such as those with very large potential values but low to medium probabilities

of win or occurrence. For example, a “billion dollar class” opportunity for a firm that is several hundred

million or less in annual revenues can easily skew the firm’s near term financial forecast. We use a

heuristic rule to test whether or not selectively eliminating specific opportunities from a forecast helps

increase forecasting accuracy. The procedures for defining the pipeline and conducting scenarios are

the same used in this paper following Section 7 with the timing parameter added. To test the forecast

accuracy of selectively excluding certain opportunities, we define a new probability called the total

probability as P(total) = P(win) x P(go). This probability is a representative proxy for an ad hoc rule that

is typically used by companies as a threshold. Setting a minimum total probability and excluding any

opportunity that falls below this threshold is a way of removing opportunities that the company has

little confidence of regardless of the value. The expectation is that by eliminating these opportunities,

we improve the forecast accuracy since low-confidence large-value projects will not swing the results.

The percentage difference defined by Equation 4 was determined for different thresholds and compared

to our enhanced approach in Figure 19. With a P(total) of 20% and a 30% it is seen that the difference in

results are negligible (within one standard deviation). However, when the threshold is increased to 40%

and above (that is only include medium and high-confidence opportunities in the revenue forecast, using

an ad-hoc selective threshold reduces the forecast accuracy significantly. This is because the international

opportunities have a small total sample size and eliminating more of these in the forecast increases the

susceptibility of the total revenue estimate to swing dramatically with a single project win or loss.

Figure 21: Percent difference calculation based on P(total) thresholds.

A threshold of P(total) of 40% removes nearly 50% of the total international opportunities thereby

reducing the of a Bayesian approach to forecasting (expected value statistics). This method of

systemically biasing forecasts can be done with a variety of heuristic rules including rules based on

P(win), P(go) and value or any combination of these. All the results lead to the same conclusion that

selectively removing opportunities to an already small sample of opportunities leads to poor statistics

and thus introduces additional inaccuracies in revenue forecasting.

-300.00%

-250.00%

-200.00%

-150.00%

-100.00%

-50.00%

0.00%

50.00%

2017 2018 2019 2020 2021 2022 2023

Per

cen

t D

iffe

ren

ce (

1 -

EV/S

V)

No Threshold P(total) = 20% P(total) = 30%

P(total) = 40% P(total) = 50% P(total) = 60%




Appendix B

The relatively small number of opportunities in the pipeline for ancillary markets contributes to forecast

inaccuracy, and perhaps more significantly the likelihood of a surprise since these opportunities are

typically of large value. To better understand how much small size statistics contributes to forecast

inaccuracy compared to timing uncertainty, we conducted simulations on our illustrative pipeline with

twice the number of international opportunities (i.e. 104 opportunities instead of 52, with a total value

of $83.5B instead of $41.75B compared to the domestic pipeline of 1,000 opportunities valued at $212.5B).

Figure 20 compares the forecast accuracy of the traditional approach (EV) with the proposed new

approach (NEV) and the simulated pipeline with twice the number of international opportunities (EV

Double). We see that the larger sample size only slightly improves forecast accuracy by less than 10%

compared to the traditional expected value formula (EV), whereas considering timing uncertainty im-

proves forecast accuracy by a factor of two in the first year. The small sample size of high value projects

typical of pursuing an ancillary market accounted for approximately 12.5% of the forecast inaccuracy,

whereas uncertainty in project timing is responsible for 50%. This indicates that our proposed new

expected value formula addresses the most significant source of forecast errors.

Figure 22: Standard expected value (EV) compared to doubling of the sample size (EV double) to the new expected

value (NEV).

-1.8

-1.6

-1.4

-1.2

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

2017 2018 2019 2020 2021 2022 2023

Perc

ent

Diff

ere

nce (1

-E

V/S

)

NEV EV EV Double

Documents

Forecasting Revenues in Ancillary Markets - smawins.com New Market Forecasting 180114.pdftypes of timing delays. ... He holds an MBA in Strategic ... Traditional approaches to revenue