© Copyright ISPE 2009. All rights reserved.www.ISPE.org

ISPEKnowledge Brief

by Marisa Ehinger

Fundamental

KB-0018-Nov09

Forecasting for Clinical Trials

IntroductionForecasting patient enrollment and drug supply logistics in the clinical trial industry has gained momentum in the past few years. One of the principle objectives in forecasting is to minimize risk and costs and it is no longer practical to mitigate risk by simply increasing costs. An optimum forecasting process would provide insight and data to support production planning, distribution strategies, supply overage predictions and shipment frequencies.

For sure, the need for more advanced forecasting has become an essential component of our business. A burgeoning marketplace offers a variety of clinical trial forecasting products and services. However, the current lack of standard terminology and different methodologies in both the discipline and marketplace may cause confusion when it comes to decision-making.

This Knowledge Brief will provide a high level overview and clarification of the basic concepts, terminology, methodologies, and benefits of forecasting for clinical trials. This

information can serve as a primer for understanding the discipline and the value of the information that forecasting for clinical trials provides. This basic understanding can help facilitate decisions regarding a company’s process and job functions and in the selection of the most appropriate forecasting system.

TerminologyA review of the industry terminology shows that the terms forecasting, simulation, predictions, and projections are used interchangeably. The following are general definitions of key terms:

• Forecasting – the prediction of something unknown or unplanned as in patient enrollment. This is a ubiquitous term in the industry used to describe many ways in which to derive a future value, whether it is an arbitrary value based on assumptions or a complex calculation using historical input.

• Simulation – a computer modeling technique which, using random variables, provides insight into future

Page 2 ISPE Knowledge Brief Forecasting for Clinical Trials

© Copyright ISPE 2009. All rights reserved.

activities or requirements. It can be a derived value based on predictive needs (the quantity of kits needed to supply forecasted patients) or an absolute value derived from a known planned event (a fixed patient visit schedule). The supplies calculation is performed in addition to (or based on) the simulation itself. One should be careful not to confuse this with Monte Carlo simulations, which generate a statistical reference to a model or process performance. This will be explained later in this brief.

• Predictions/Projections – this seems to be generally used as a verb to describe a process, as in “it will project the demand for the next month,” but could mean a forecast of unknown events or a derived value from planned events that are simulated.

In the industry, all of these terms are used interchangeably. To prevent any confusion or misunderstanding, be sure to confirm the intended meaning.

Forecasting Methodologies When one considers industry forecasting outside of clinical trials, the methodologies and algorithms are mature and well known. The most analogous industry to a clinical trial is the consumer goods industry. The randomness of patient enrollment can be compared to the random purchases by a customer in Wal-Mart in any part of the world during a specific season. There are whole companies, organizations, and niche industries devoted to predicting what, when, and where consumers will purchase. Relatively speaking, the clinical trial forecasting industry is gathering momentum and maturing methodologies needed to forecast patient enrollment and drug supply. To lend confidence to this momentum, many of the clinical trial forecasting systems are adapting accepted consumer goods practices, such as those described below.

Monte Carlo SimulationThis method is used to imitate the patient population of a clinical trial and to provide

metrics about the results. Multiple simulations are run, analogous to throwing the dice many times. The result of each simulation (each throw of the die) is captured. All of the simulations are then pulled together to provide metrics to the user about how confident they should be in the results of the resulting forecast. This method is useful when facing random events, especially chains of interdependent random events that are too complex to be modeled directly.

Time Series ForecastTime Series Forecast is used when historical values are associated with a date. Those dates are put in chronological order and a pattern develops. In consumer goods, this could be the buying pattern of electronics or seasonal items, such as coolers. In clinical trials, this relates to enrollment history, among other things. Looking at the historical pattern, there are several ways to compute a forecast from it. These methodologies allow for different and sometimes more accurate calculations from the history. For example, when looking at two years of enrollment history and knowing that the oldest six months no longer have any relevance to what is happening today; a forecasting algorithm could exclude that data and just analyze the relevant, more recent data.

Considerations in User Process and Vendor SelectionIt is important when discussing this process with a user of the forecast or a vendor under consideration that there is an understanding of how they look at history to develop a forecast. The more ways in which a system can analyze historical data and determine the best calculation to fit the historical pattern, the more opportunity there is for the forecast to be accurate. Different forecasting algorithms are better suited for different situations. It is important that you understand your situation and can match your forecasting needs with products and services that are offered in the market.

In addition to generating a forecast, it is helpful that a system provides statistics

to measure how confident the system is in its forecast. These statistics should provide the user with enough information about the forecasting results to allow the user to analyze and qualify the forecast for their uses. For example, if the forecast is 100 kits a month, it would be important to understand the variability of that value. Could the actual demand be as much as 200 kits and as low as 50 with high probability that it will be close to 100?

Because clinical trials can have both random events to forecast (patient at site enrollment) and time series data (historical discontinuation rates), some systems use a combination of the Monte Carlo simulation and the Time Series Forecast.

Challenges and ConcernsUsing any system to calculate information always comes back to Garbage In equals Garbage Out. If you do not have confidence in your data and assumptions, you cannot be confident in the system’s output. The assumptions and your confidence in them will vary significantly, especially before incorporating historical actual data; therefore, some sensitivity analysis will be required. However, there is still value in using a system across your studies in order to provide a standardized and objective approach to making predictions based on the best information available at the time. By using a standardized approach, one will have a common platform from which all forecasts are generated; a baseline from which to move away from and course correct as the assumptions turn to reality; and a system that will allow you to make accurate and easy changes to adapt quickly to ongoing challenges.

It is also important to have a decision-making process in place that ensures different users are analyzing output consistently and in a way that matches company priorities.

Some Basic Forecasting Axioms1

Armed with the above information and the following axioms, you can determine

ISPE Knowledge Brief Page 3Forecasting for Clinical Trials

© Copyright ISPE 2009. All rights reserved.

whether the assumptions used for your forecast and what you are forecasting will allow for a higher degree or lower degree of confidence:

• The past will repeat itself – this would assume that you have some pattern of the past that would be useful to predict the future. This can be the enrollment pattern of an entire study or more specific to the enrollment pattern at a site. It can be the discontinuation rate of patients or any other historical pattern you would like to predict. Additionally, one may be able to leverage historical actuals not just from the study being run, but based on prior studies for same or similar drug and/or medical condition.

• The shorter the forecast horizon the more accurate the forecast – it is much easier to predict an accurate patient enrollment for next month versus an accurate patient enrollment for the 24th month of the study.

• The forecast at the highest level is more accurate than at the lower level of data – managing the enrollment at a study level is much more accurate and predictable than forecasting the site enrollment for an average of .75 patients per month per site.

• Forecasts are seldom accurate – you must always have contingency plans, safety stock, and management oversight to protect yourself against these inaccuracies.

What Success Looks LikeUsing a system successfully means being able to enter data and parameters that fit your situations more than 80% of the time. The system should be at least quicker, easier, and more accurate

than your current method of forecasting. It is also important to be able to save forecasts and assumptions at a point in time so that in the future you can compare actual outcomes with the planned forecast. ConclusionThere is no forecasting system that will “do it all” for you every time. What it will do is give you a greater degree of confidence in your supply decisions, allow you to adjust to changes quicker with more accuracy, and perform “what if” scenarios for complex planning and greater confidence. You should be able to layer in more sophisticated planning more frequently. It should save you time over the life of the trial and provide a common platform for department planning and communication. Everyone can use a forecast system, but how frequent and complex your needs are will determine if you maintain your Excel spreadsheets or purchase a service or system.

References1. Reference for Business, www.

referenceforbusiness.com.

For Further InformationFor more detailed and related information, the following ISPE resources are available:

Recorded Webinars:• Investigational Products: Balancing

Risk and Costs to Optimize the Clinical Supply Chain – A Step Beyond Simulation.

• Investigational Products Series: Reduce Supply Availability Time by 70% – Remarkable Changes in Operations Management.

http://www.ispe.org/onlinelearning

Pharmaceutical Engineering Articles:• “Next-Generation Clinical Supply

Chain Management Systems,” by Douglas Meyer, Pharmaceutical Engineering, September/October 2002, Volume 22, Number 5, pp. 8-16.

• “Business Planning for Clinical Trial Materials,” by Robert A. Young, Pharmaceutical Engineering, May/June 1999, Volume 19, Number 2, pp. 25-35.

http://www.ispe.org/pharmaceuticalengineering

Journal of Pharmaceutical Innovation Article:• “Balancing Risk and Costs to

Optimize the Clinical Supply Chain – A Step Beyond Simulation,” Journal of Pharmaceutical Innovation, Volume 4, Issue 3 (2009), p. 96.

http://www.ispe.org/jpi

Investigational Products (IP) Community of Practice (COP):• Visit our IP COP on the ISPE Web

site for the most current and up-to-the-minute discussions on the topic discussed in this Knowledge Brief and other related topics.

http://www.ispe.org/communitiesof practice

About the AuthorMarisa Ehinger is a Director of Product Development at Almac Clinical Technologies. She was formerly a forecasting and supply chain consultant in the consumer goods industry working with middle and large tier manufacturers. She has been in the IT and software industry for more than 15 years and has been with Almac for more than seven years. •