A methodology for the evaluation of research and development projects and associated resource allocation

Comput & Elect Engng, Vol. 8, No 2, pp. 123--152, 1981 0045-79<)6/81/020123-30502.00/0 Printed in Great Britain © 1981 Pergamon Press Ltd

A M E T H O D O L O G Y F O R T H E E V A L U A T I O N O F R E S E A R C H A N D D E V E L O P M E N T P R O J E C T S A N D A S S O C I A T E D

R E S O U R C E A L L O C A T I O N

CRAIG MILLER

Department of Computer Science, University of Colorado, Boulder, CO 80304, U.S.A.

and

ANDREW P. SAGE

Department of Engineering, Science and Systems, University of Virginia, Charlottesville, VA 22901, U.S.A.

(Received 23 June 1980; received for publication 9 September 1980)

A~traet--This paper develops a methodology for the evaluation of research and development projects and the allocation of resources for the development of large scale technologies. The methodology is directed at the problem of selecting a research portfolio when the number of projects is large enough that enumeration of all the possibilities is impractical. In a series of successive screening stages, the number of candidate portfolios is reduced to a practical number. The initial screening stage is based on projected performance of the technology. Successive stages first screen market penetration by ignoring competition from other projects in the portfolio, and then screen by considering project competition. Thus succeeding stages in the methodology are successively finer but more costly to implement. Project screening techniques, which include elimination by aspects, ranking, and stochastic dominance are applied throughout the methodology to result in a heuristic and effective approach to evaluation, and associated resource allocation, in very large scale systems.

1. I N T R O D U C T I O N AND C L A S S I F I C A T I O N OF R&D R E S O U R C E ALLOCATION PROBLEMS

Research and development (R&D) is a major business. In the United States alone, annual research spending passed $35 billion in 1975. Decisions about how much and what kind of research should be done are important. The research being done now will have a profound effect on the competitive positions of corporations and nations in the decades to come. The importance of these decisions has focused a great deal of attention on the development of R&D allocation and management methodologies. This paper develops one approach to a specific class of R&D problems. It restricts itself to a specific class because the great variety of R&D programs and R&D management environments require a wide range of approaches.

The product of research is information. Managing R&D relies heavily on attaching a value to this product. This, unfortunately, is not easy. Information has no intrinsic value, though in use it can be invaluable. Neither is there a competitive market for information which would explicitly define its worth. This complicates the job of the R&D manager; even assuming (falsely) that the results of research can be guaranteed, it is necessary to decide how much to pay for something of uncertain value. In the real world, where the results are anything but certain, the question is much more difficult. The value of information is an abstract issue. When faced with the need to allocate and manage a research budget, the program manager wants concrete measures of program worth. So, R&D allocation methodologies give little or no consideration to abstract concepts and make some important simplifying assumptions. Generally, the assumption is that the value of information is the value of what it allows one to do. The value of an innovation which permits the construction of an advanced energy system is the worth of the advanced energy system relative to its conventional alternative. Also, the last stages before the marketing of a process are very different than the first stages of research. Basic research produces information which usually cannot be reliably valued by assessing the downstream products, as they are simply too far downstream for their benefits to be estimated with certainty. The general applicability of basic research is another stumbling block to research worth assessment. The information generated in fundamental studies shows up in many technologies, a great number of which may not have been conceived when the research started.

123

124 C. MILLER and A. P. S^OE

Different R&D allocation methodologies are appropriate for different stages of research. This difference is enhanced by the large difference between the cost of a typical basic research project and a typical demonstration project. Basic research often costs from $50,000 to $200,000 per project, while demonstrations usually run upward from $I million to over $I billion. The high degree of uncertainty with respect to outcome and value associated with basic research dictates that many parallel, generally low cost, projects be funded. A consequence of this is that methodologies for early stage allocation must handle a large number of very uncertain projects, while methodologies for later stage allocation must deal with a smaller number of less uncertain projects. It would seem that the second task is easier than the first. But the large sums of money involved in the later stages of development make it very critical to fund the best alternative. The decision is easier, but the penalty for error is much greater particularly since only one project can be funded as the cost of developing several approaches is too large.

Accepting that the value of research is measured by the value of the resulting products, the focus of R&D program evaluation becomes the estimation of the market performance of the innovations. Market penetration estimates and analysis of the expected return on investment are the principal measures of research worth. Conventional market methodologies fail, however, when the innovation is not marketed, or when it has no conventional competitor.

When an innovation has a conventional competitor, the potential market is that fraction of the conventional product's market in which the innovation enjoys an economic advantage, minus some fraction of conventional users who are unable to convert for economic, environmental, institutional, technical, or other reasons, plus some additional market which was unable to use the conventional product but is able to use the innovation. This is not an easy number to estimate; but consider how much more difficult it is to estimate the potential market when the product is wholly unlike anything else. It is very difficult to predict the number of potential users, and even more difficult to estimate how much people are willing to pay for the product and how quickly they will accept it.

The second problem with assessing research programs by the market potential of final products is that some products do not compete in a marketplace. Two situations where this commonly occurs are when an innovation is developed for internal use only and when the development cost is a large fraction of the sales potential. It is usually more difficult to associate R& D projects which are not market related with return on investment measures than projects which are closely related to well defined markets.

There are many ways of classifying the range of R&D resource allocation problems. It is, consequently, of interest to divide R&D evaluation problems into small groups with strong methodological implications, such that a few methodologies will deal with several groups. No classification scheme is inherently most proper, and many equally valid alternatives are possible. However, one scheme which seems particularly useful is to divide R&D evaluation problems on the basis of:

• The interactions between the products and • the information transfer between projects.

Research projects can interact through their products. Some projects have products which compete in the market place. The value of pursuing two research projects which lead to competing products is never greater than the sum of the value of each pursued alone and often quite a bit less, though never less than the value of the single most valuable product. Products may contribute to each other's value and, hence, to the value of the projects which lead to their development. A classical example of this is the development of the electric lightbulb and central power generation. Without central power, the lightbulb was not a particularly useful invention, and without the lightbulb the value of central power was diminished.

The other type of interaction used in this classification is information transfer. When projects share similar elements, the work done in one can assist in others. This situation occurs most commonly in projects which are very closely related or are in their very early stages so that they are not yet strongly differentiated. It is also possible that the information gained in one project will delay completion of another project beyond its originally scheduled date by

Evaluation of research and development projects 125

pointing out some previously unforeseen pitfall. This sort of interaction need not be considered separately from simple beneficial information transfer, if it is assumed that the same problems would have been encountered had the projects been carried out independently. It follows from this assumption that the net effect of information transfer can only be beneficial, or at worst, null. It cannot be harmful.

If the information is accurate and useful, then it eliminates the need for duplicate research, thus reducing the cost of the second project and possibly the time needed to complete it. If the information concerns some unanticipated pitfall, then the early warning may allow more effective treatment of the problem area or permit cost savings by abandonment of a project which would not have worked. These parameters, project interaction and information transfer, define six different types of R&D allocation problems if we assume that product competition and augmentation cannot occur simultaneously. Each of the different types of problems has some methodological implications which makes it difficult for one methodology to deal effectively with them all. The specific classes of problems we deal with in this research are those with no information transfer and either competing or independent products.

When products compete, the important issue is their relative economics. Assuming that both are better than a conventional alternative, the success of one product undermines the performance of the other. In the case where both products become available in the same market at the same time, only the one with the lowest perceived cost will sell. Only one project will be successful in terms of resulting in a useful product.

Another complication is that competing innovations may not all enter the same market at the same time. If one product enjoys a cost advantage over the alternatives in one region but not in others, several innovations may each capture a market share. If, however, the market is small and/or the number of manufacturers is small, only one alternative may be produced and a suboptimal technology used in at least some segment of a market. A related issue can arise when one innovation reaches the market before the others. Even if the market is homogeneous to the extent that the economic rank ordering of the competing new technologies is the same in all markets, some users can still end up with a suboptimal technology. If the technology which is marketed first is not the best, but is still better than the conventional alternative by enough of a margin to justify the switch, then it will sell to that segment of the market which is unaware of the forthcoming, superior technology, the segment which is aware but unwilling to wait for the better alternative, and the segment which is willing to purchase the immediate alternative for use until the improved version is available. The market shares of the competing alternatives will not reflect their relative economics because of the different time horizons associated with product introduction. The implication of this is that R&D managers must consider timing when considering the worth of projects. The value of a project completed some time in the future is usually not as great as the value of the same project completed now, except when market conditions on the lack of supporting technologies prohibit immediate adoption of the product.

Some products enhance the value of other products. There are many common examples, including the lightbulb and central power plant noted earlier in this section. The basic issue with augmenting technologies is evaluating the conditional and independent worth of the individual technologies. The value of each technology must be evaluated assuming that none of the others are successful, and then reevaluated for various outcomes of the other research projects. This is not an easy task. One very difficult complication is that the different augmenting technologies may compete in different markets. The lightbulb is a consumer product, but no consumer can buy a central power station. Market penetration for products is very difficult to estimate. Each decision maker will be wary of the performance of other competing products. A decision maker's actions under these circumstances are not easy to predict.

In the special case where two or more technologies are worthless without each other, but do have an estimable value when all technologies are adopted, the group of projects can be treated as one for management purposes. The problem is modeling the decision to implementthe technologies, but the R&D manager's decision is simplified to a binary all-or-none comparison. This assumes a degree of authority centralization which may not always be achievable.

In the situation where there are several competing technologies which augment a third, the

126 C. MILLER and A. P. SAGE

program worth analysis becomes especially complicated. There must be careful study of the market, the decision makers, and their preference structure, and the time horizon associated with product introductions.

When two projects share similar informational or hardware elements, the work on one project can sometimes be used to support the other. This simple information transfer can substantially complicate the R&D allocation and evaluation process. If simultaneous development of the two projects makes it possible to eliminate or reduce work in the common areas, the cost of doing both projects is not the sum of doing either without the other. Thus, it is not possible to use simple linear evaluation methods without modification to consider this depen- dence, and there can be some serious difficulties in modeling the project interaction.

2. THE R&D ALLOCATION METHODOLOGY

This paper presents the development of a methodology for the allocation of resources among technology development projects. The methodology is intended for application in the situation described in our introductory section. Specifically, the methodology can only be used without modification, in the specific form presented, when:

(1) Enough is known about the technologies under development to develop credible estimates of their performance.

(2) There is no transfer of information between projects. (3) The performance of each product does not depend on the performance of the others.

These criteria are generally met by large scale projects directed at the final stages of technology development and are not met by many basic research efforts.

In essence, resource allocation to technology development projects means selecting the combination of projects and associated funding levels which maximizes the worth of or return from the research program. An analysis with five important general steps, within the general three step framework of systems engineering[I-3], is used to make this determination. The approach involves:

Formulation Step 1. Goal: to identify research needs and constraints on research (Problem Definition). Step 2. Goal: to determine measures of program worth to provide a basis for comparing

groups of projects (Value System Design). Step 3. Goal: to identify candidate projects and alternate funding levels (System Syn-

thesis).

Analysis Step 4. Goal: to estimate the expected performance of candidate groups of projects at

alternative levels of funding (Systems Analysis).

Interpretation Step 5. Goal: to valuate, interpret, and select the group of projects and funding levels

which has the highest estimated net worth (Decision Making).

Many existing resource allocation methodologies, do not address all three steps of the general resource allocation problem, i.e. formulation, analysis, and interpretation of the effects of different funding allocations. Instead, they focus on one of the final two steps.

The methodology developed in this research is focused on the last two steps, analysis and interpretation, of the resource allocation process. The methodology can function with any measure of worth provided that the measure meets two requirements. Specifically, the measure must:

• be expressable as a scalar and • be comprehensive to the extent that a group of projects with a given worth will always be


preferred over a group of projects with a lower worth provided they both meet all resource and other constraints.

Whenever the term "worth" is used in this and subsequent discussions, it represents project worth as manifested in a quantity meeting these requirements. "Value" and "Utility" are used here as exact synonyms for "worth".

It is reasonable to begin evaluation of a research program by evaluation of individual candidate projects. Analysis of a single project can be viewed as a two-step process. First, the technical outcome of each project is assessed. Our object is to characterize the technology which the project will ultimately produce. Those parameters of the technology which enter into or otherwise impact the program evaluators "assessment of the technologies" worth should be the focus of the technical forecast. This step corresponds to the impact analysis step of the three step systems engineering process.

The second step is to relate the measures of technical outcome to the measures of project worth. For example, in evaluating a project which is directed toward producing a commercial product, the relationship between technical performance and future sales should be examined. This step corresponds to the interpretation step of systems engineering and involves elicitation and assignment of a utility to each event outcome.

Separation of the evaluation process into impact identification and analysis, and utility assessment, is practical bothin terms of the resources needed to make the evaluation and in terms of the quality of the results. The information and skills needed to forecast performance of an experimental technology are quite different than those needed to evaluate its long term worth. In the first case, engineering specialty area skills are most important, and only information about the technology under study is needed. In the second case, a broad spectrum of information is needed along with the skills to interpret it. The value of a technology is a complex function of its performance relative to other technologies serving the same functional use, the resources and preferences of the consumers, environmental considerations, and a long list of other factors. Clearly, this portion of the evaluation is a systems engineering function and involves inputs from a broad spectrum of disciplines.

Bifurcation of the evaluation process has the further benefit of reducing the opportunity for bias. If the engineering evaluation is done without reference to market characterizations or information about the performance of potentially competing technologies, it is less likely to be tailored to produce an unrealistically favorable level of performance.

After each p~'oject is examined separately, it is necessary to consider combinations of projects (portfolios) to determine which grouping maximizes the worth of the research program. If the projects are independentt, then the worth of a portfolio is the sum of the worth of the individual projects and the cost is the sum of the individual costs. Evaluation of a portfolio is trivial, and no further work is needed as a preface to the final stage of the analysis which is selection of the optimal portfolio, which is here taken to mean the portfolio which has the maximum assessed worth and meets cost and other constraints imposed on or specified by the research agency.

If the projects are not independent, then there must be a third analytical step in addition to the two discussed above. The third step is to characterize the relationship between the new technologies in their final market; that is, to develop a model of the impact of each project's technical success on the worth of the other projects. The next and final step in the resource allocation methodology is to select the best portfolio. In general, mathematical programming, decision analysis, and other techniques can be applied with relative ease when the candidate projects are independent. When the projects are not independent, it becomes more difficult to use them.

The four general steps for analysis and interpretation in our proposed resource allocation methodology are shown in Table 1. The tasks described in this table define the specific actions needed to evaluate a product whose worth is measured by its market performance. Analysis of the impacts of each project separately is accomplished in the first two steps of the proposed

tOne project is defined as being independent of another if the benefit to the researchers derived from pursuing one is the same whether the other is developed or not. A "group" o[ proiects consists of a set of projects which are independent of all projects outside the group.


Table 1. Four general steps for analysis and interpretation for R&D resource allocation

Step 1. Estimate the technical impact or performance of each product of each project as a function of resource allocation to each project.

Step 2. Identify and measure the market impact for each product and predict the market share of each over time as a function of its technical performance on the assumption that the technologies do not

compete with each other.

Step 3. Identify which products compete for each market segment. Determine how the characteristics of each segment affect the relative utility of each innovation.

Step 4. Evaluate each portfolio of projects and select the optimum or the one with the highest worth at one or more possible funding levels.

methodology. The last two steps deal with interpretation and evaluation of projects in groups. A specific focus of our research is on the development of techniques to accomplish the fourth step [4].

PROJECT WORTH

The worth of a project can be measured in many ways. The goals of the organization sponsoring the research are the primary determinant of the best measure of worth[5]. Often, however, these goals cannot be used, unmodified, because they cannot be measured. Either a proxy or surrogate for the goals, that is an objectives measure, must be used. The form of the measure of worth is of interest. Our discussion focuses on the separation of the analysis and interpretation steps of systems engineering. While the methodology is not wedded to any particular measure of worth, it is assumed that the measure meets the criteria set forth at the beginning of Section 2. Assuming also that the frequency distribution of the worth of an individual project can be calculated or estimated, the expected worth of the project can be defined as

W = wf(w) dw (I)

where

W = expected worth of theproject

[(w) = the probability density function (pdf) of worth

w = worth of the project.

This form is appropriate for projects directed at the development of a technology with a known number of applications. Project worth, w, is the product of the average worth of a single unit of the technology and the number of units put into use, assuming constant returns to scale. For technologies for use within the developing agency or directed at a fixed market, the number of applications is well known, and it is feasible to address project worth as a whole.

When the technology is destined for an uncertain market, the number of units resulting from a given level of funding can be very difficult to predict. In this case, it can be useful to separate worth into two components--the worth of a single unit and the number of units. Estimation of the first of these parameters is primarily a problem in forecasting the performance of a technology before it has been developed. Specific engineering disciplinary skills are the primary requirement. Estimating the number of units can involve a wide range of disciplinary skills [6].

The interface between technology performance and market performance estimation problems can be difficult to cope with. What is needed is a measure of technology performance that can be related to market performance. As is often the case, money is a convenient yardstick. The performance of a technology can be formulated in terms of the cost of delivering a specific service (e.g. transportation of a human being between two points or 12 MMBtu/hr of 600 ° steam at 100 psi). Cost is not directly related to either the worth of the technology or the number of units in use.


Market price must first be determined. Assuming that it is possible to develop some relationship between number of units in use and market price, it is fairly simple to determine the market price which maximizes return of investment. If return on investment (ROI) is either the measure of worth to the developer or can be related to it, then a technology cost based formulation is appropriate.

A worth construct based on cost is also appropriate for much publicly funded research and development, the objective of which is to make a service available to the public at a lower cost to the public or the producer of the technology than currently available. An alternative to eqn (1), based on cost as a measure of technology performance, is

w = ~(c)~,~(c)

where

w = worth of the project as in eqn (1) ~,~(c) = expected worth to the developer derived from one unit of the technology given cost to

the producer c n(c) = expected number of units of the technology which will be adopted given cost c.

Then we have for expected worth

d(n(c)w~(c)) w= f+_f n<c)w'<c)[ ~c ]dc = (+~ n(c)wdc)l(c) dc

J -o~

where

f(c) = the probability density function of technology cost.

The variables wu(c) and n(c) are expectation values taken over those parameters other than cost which have a major impact on worth and number of units adopted. Also we have

¢vu(c)= f_+f dXl f+_f dx2 . . . . . f+_f dxnwu(c, xl . . . . . x~).f(c, xt, x2 . . . . . x~)

where

wu = worth of one unit x I . . . . . .I n = variables with a major impact on worth of the unit to the producers

f(c, .I~ . . . . . xn) = joint probability density function of cost c and other determining factors

and

tz(c)= f_~dxl f+~ dx2 . . . . . f+: dXln(c, xt . . . . . xn)f(c, xl, x2 . . . . . Xn)

where the variables are defined as above.

4. EVALUATING ONE PORTFOLIO As the process followed in planning and R&D allocation moves to the final stages,

evaluation focuses on a smaller set of candidate projects. Demands for computational efficiency suggest that various details of the assessment process be approximated. Until approximations

130 C. MILLER and A. P. S^~E

are needed for the evaluation process, it appears best to concentrate on producing a more accurate evaluation rather than an approximate evaluation that can be done quickly. When evaluating one portfolio, it is possible to consider many of the intricacies of the technologies and their associated markets.

Our discussion of the process for portfolio evaluation focuses first on a single project and the general process of technology assessment. Then, we consider ways in which the worth of a portfolio can differ from the sum of the worth of the constituent projects of the portfolio.

4.1 Worth of one project The worth of one project to the developing organization can be a function of many

attributes. Clearly, the best procedure for assessment depends on the measures of worth applied. Any assessment of worth must include an estimate of how well the product will perform and how much it will cost. It is difficult to conceive of any realistic measure of worth that does not involve these factors. The assessment may stop there, if the technology is being developed for internal use or it may be carried through a forecast of market penetration if the technology is intended for use outside the developing organization. Many elements may be included in the assessment but two, technology performance and market penetration, are very common and very important and will be discussed in this section.

4.1.1 Technology performance The issue at hand is how to predict the performance of a technology that has not yet been

developed. It is generally not possible to predict, both precisely and accurately, how a technology will perform in advance of its development. At this stage, it is proper to regard cost and performance as uncertain parameters with associated probability distributions rather than certain values.

The simplest approach to assessing performance of a technology is to ask the individuals who know the technology best. However, it is not simple to develop a reliable estimate of an uncertain parameter through questioning. Practitioners of decision analysis, a technique which relies heavily on such assessments, have developed some general rules for questioning that may enhance the quality of the outcome. Some approaches are available in Raiffa[7], Keeney and Raifa[8], and other sources. Our discussion assumes that frequency distributions of performance and/or cost will be developed in a conversation between an elicitor and a respondent, generally expert with respect to the particular technology involved. Some general guidelines are:

1. The person interviewed should have specific knowledge of the aspect of the technology which is being assessed.

2. There should be an explicit understanding between the elicitor and the respondent as to what is being assessed. For example, specifying that a discussion is about the cost of a technology is not sufficient. Does the cost include capital, operating, maintenance, installation, over what time span, and to whom?

3. Some respondents may be uncomfortable thinking in terms of frequency destributions. They may tend to offer a single specific fixed estimate[9]. Their willingness to consider variations about the single estimate may be enhanced by postulating a higher or lower level of performance and asking them how it could be achieved. This broadens the discussion to more than their "best guess" scenario.

4. The elicitor should not begin by asking for a best guess. This will focus the discussion on a single value and limit consideration of the extremes. Instead, the elicitor should first attempt to define the limits of performance.

5. The elicitor should explore the extreme values suggested by the respondent by postulating scenarios that would extend the limits or by having the respondent do this. The ultimate limits achieved in this fashion are more realistic bounds than the ones that will be suggested initially as they are more nearly free of the central bias.

6. Graphical tools to illustrate the concepts of probability may be useful in helping the respondent make tradeoffs between ranges of outcomes.

7. Care must be taken to assure that the opinions of the respondent are unbiased. It is likely


that an expert on the performance of a new technology may be a strong advocate or opponent of the technology or perhaps even have a personal stake in the decision whether to continue or forego development. The possibility of bias should be investigated in a discussion at the beginning of the interview. The elicitor must be very careful to build a cooperative mood and not prejudice the expert against the assessment process by any overt suggestion of bias regarding technology performance. It should be evident that the process of eliciting assessments of performance is an exercise in applied systems engineering psychology. However, an understanding of the technology under review is vital in understanding the responses of the expert, and helping to develop various performance scenarios. A recent survey[10] discusses many of the issues in the elicitation process especially as they are affected by risks and hazards.

4.1.2 Market penetration Given some measure of the performance of a technology under development, the next step

in project worth assessment is to estimate the number of units of the new technology which will come into use over time. Market penetration estimation is very important in many financial decisions. Consequently, many penetration models have been developed. They can be roughly divided into three classes:

• empirical models which estimate long range penetration from the first few sales or historical analogy

• economics-based models which are based on the assumption that everyone who ought to adopt the innovation for economic reasons will do so, and

• behavioral models which attempt to simulate the decision to purchase and usually consider many attributes of the innovation and the purchaser.

The market potential is defined as the number of units of the new technology which could be adopted in a unit of time if every potential user preferred it to all alternatives. Essentially, the market potential is the total number of units of the new technology which would be necessary to satisfy the demands of equipment as well as growth within the new technology's functional uses. Market share is the fraction of the potential market which is actually captured. If the technology addresses a totally new use such that there is no conventional alternative, the market share is necessarily 100%. The unit of time used in a penetration analysis should be chosen on the basis of expected rate of penetration and the market life of the product.

• Empirical models The simplest market penetration models are empirical. They do not attempt any description

of the decision to purchase. Instead, they assume that penetration follows some functional form and attempt to fit a curve using a historical analogue or data on the growth of the innovation's market share through the first few per cent.

Historically, the growth in market share of a new technology has followed an "S" or sigmoid curve. That is, the growth in market share has been slow at first and near saturation (100% market share) and fastest when the market share was about 50%. A characteristic S curve is shown in Fig. 1.

There are many function forms which have a sigmoidal shape. Past work in assessing market penetration has generally centered on the form:

f = [3 e at

1 - f

where 13 and a are adjustable parameters and f = market share expressed in fraction of total market. This form evolves from solution of the nonlinear differential equation

d--t = a f(1 - f ) , f(O) = 1 + ~"


Linear plot

c O 4 .2

0 Time

tv2 = the time when the innovation has captured half of its market.

Fig. I. General form of substitution.

A characteristic of this form is that the logarithm of the ratio of the new technology's market share to the market share of all competing conventional technologies is lineaTly related to time as shown in Fig. 2.

This form was suggested by Fisher and Pry[ll]. The assumption is made here that the penetration of a new technology into a competitive market is analogous to the spread of diseases in a closed population.

Classic studies of epidemics show that disease spreads slowly when there are few sick people to infect the healthy ( f - 0) or few healthy people for the sick to infect ( .f- 1), while disease spreads most quickly when the healthy and infected populations are equal. Fisher and Pry reason that, analogously, the growth in a new product's market share is slowest when there are few units in use or when there are few remaining opportunities for adoption. The reason for the slow growth near saturation is obvious. Fisher and Pry explain the slow initial growth as being due to reluctance on the part of those who make adoption decisions to select a technology that they have not seen functioning. As the number of units of the new technology in use increases, familiarity with the innovation increases and a larger fraction of the decision makers are willing to adopt the new technology.

Fisher and Pry tested their model against historical data, but did not attempt to apply the construct to innovation~ which had not yet saturated their market. Lenz and Lanford[12] extended this approach by developing an algorithm to estimate a and fl from a few percent penetration.

Lenz and Lanford cite an earlier unpublished work by Fisher and Pry which concludes that only 2% penetration is required to prepare a forecast. No measurement of the accuracy of such predictions is included. An estimate by Lenz and Lanford of the curve parameters for the conversion of shipping from sail to steam made on the basis of 2% penetration

g

t~ o

tv2 = the time

8

Logarithmic plot of ratio

IO

I O -

OI

Time

when the innovation has captured half of its market.

Fig. 2. General form of substitution.


understates the correct time by 25%. This error is substantial but may be attributable in part to the small number of freighters, the low turnover rate, and the high cost, all of which strain the disease analogy.

• Economic based models Market penetration depends on two factors, the turnover rate and the decision to purchase.

If only a small fraction of the total product inventory is replaced each year, then penetration will proceed slowly. If a large percentage is replaced each year, then penetration can proceed quickly, but only if the innovation appeals to the person or group making the decision to purchase.

Because the turnover rate is a measurable (or easily estimable) quantity, economics-based penetration models have focused on the purchase decision. The basic reasoning is that the decision parameters are primarily economic. To a very large extent this is true, but for some products, in some sectors, it is not entirely so. Consumers, more so than business and industry, sometimes go counter to the economically preferred decision. For this reason, economic penetration models have usually dealt with nonconsumer oriented innovations.

In the early 1960s, Edwin Mansfield developed a deterministic and a stochastic model for the spread of industrial innovations[13]. The stochastic version is very elegant and very complex. It is probably the best description of market penetration to date, but to use it in a normative mode requires data that often does not exist and which quite possibly may be technically and practically impossible to gather. The deterministic model is simpler. Mansfield's hypothesis is that the proportion of 'hold-outs' at time t that introduce the innovation by time t + I is a function of (1) the proportion of the firms that have introduced it by time t, (2) the profitability of installing it, (3) the size investment required to install it, and (4) other unspecified variables [14]. Mansfield postulates a functional relationship between the characteristics of a technology and its market, and the rate of adoption. Mansfield addresses unspecified variables by postulating that they can be combined in a single measure, which he calls an "innovation index".

Mansfield calibrated his simple linear model for the logistics functions parameters for a number of new technologies in a number of industries. The results were generally good, showing the postulated invariance of the innovation index, but the number of verifying cases was small.

The Mansfield model may not in itself be practically useful, but it illustrates the general theme of economic market penetration models. They generally adopt a sigmoidal curve to describe penetration, as do the empirical models, but attempt to derive a function for equation parameters that permits prediction of them prior to the date of initial penetration. The economic models are ambitious, but are less well accepted than the empirical models.

• Behavioral models Like economics based models, behavioral models focus their attention on the decision to

purchase. Unlike the economics models, the behavioral approaches do not assume that the decision is primarily fiscal. Consequently, they are applicable in sectors where noneconomic factors play a major role.

A simple behavioral model was proposed by Nevers[14]. His work was based on similar proposals by Bain[15] and Haines [16]. The fundamental hypothesis is that the probability of a customer making a decision in favor of purchase is a function of the propensity to innovate and the propensity to imitate. Nevers postulated the following relationship:

where

P(T) = p + q Y(T) m

P(T) = probability of a purchase in time period T p = coefficient of innovation = P(0) q = coefficient of imitation

Y(T) = purchases of the innovation prior to period T m = potential market size.

CAEE Vol. 8, No. 2--D

134 C. MILLER and A. P. SAtE

The assumption resulting in the first term is that each member of the purchasing population has a likelihood of purchasing the innovation without seeing it in use, and that this probability does not vary with time. The second term is based on the assumption that the probability of a purchase due to imitation is directly proportional to the fraction of the potential market which has been captured.

The number of purchases in a time period, S(T), is the probability of an individual purchase times the size of the remaining market:

S(T) = P(T) . (m - Y(T)).

We combine this with the foregoing equation to obtain

S(T) =pm + ( q - p ) Y ( T ) qy2(T) m

An elaborate behavioral model was developed by General Electric as part of the National Solar Demonstration Program[17]. The methodology follows the prescriptions of paretian analysis which is useful when several decision makers or actors with different preference structures are involved in the purchase decision. The General Electric Model incorporates departures from the basic paretian model by changing the interpretation of the political weights and simulating the decision split propensity of a single actor. Rather than synthesizing a group decision, GE normalized the political weights and gave each actor absolute control over a share of the total number of purchase decisions proportional to their weight. The other change is the decision split capability. After deriving the worth to a single decision maker of the innovation and the conventional alternative, the innovation's percentage of the total purchase was calculated as a function of the ratio of the worths. This model appears capable of yielding a very good description of the penetration process, but data are not generally available to make it work in a normative or prescriptive mode.

4.2 Cost of one project Implicit in the discussion which has preceded this section is the assumption that the funding

organization wishes to select a limited number of projects. The reason for the assumption is obvious--few organizations (if any) have resources in excess of the possibilities for spending them. That such resource constraints exist make it necessary to consider not only the worth of a project, but also its cost in terms of money, people, equipment, and any other quantity which could be binding on the research program.

This section does not address the specific issues of how cost is assessed. Dean[18] has published a discussion of research cost estimation and associated uncertainty in U.S. Army research that appears to be practical and accurate. It is directed at defining the general types of costs that must be considered and discusses the treatment of cost in the design of a research program. Mansfield [19] has published an excellent general description of the cost of research. Procedures for actual cost assessment range from simple counting, when the project is very well defined, to a forecasting problem analogous to the one described in the preceding section when the scheduling and content of the project are not yet finalized or the cost of specific elements cannot be determined. Some relevant projects costs are:

Sunk costs In the idealized allocation problem, time conveniently begins when the allocation decision is

made. Project history is not considered except to the extent that it is used as a state variable to determine future worth and cost. Of course, organizations have a history of research and a stock of projects in progress. The costs of the projects underway from initiation to the point in time used for the allocation analysis are defined as sunk costs.


Fixed costs of the organization The fixed costs of the organization are those that do not vary with the size of the

organization or the size and makeup of the research program, and cannot be directly associated with one or more specific projects.

Variable costs of the organization The variable costs of the organization are similarly free of any association with individual

projects, but they do vary with the size of the organization and the research program. The distinction between fixed and variable costs can be vague. Some costs are fixed over ranges of organization and research program size, but do vary when size is changed over a large range. Also, some costs are fixed for the short-term, but can be varied over the long-term. The definition of which costs are fixed and which are variable is one which must be made when the planning horizon and planning period, and the range of variation in the size of the research program, have been determined.

Fixed costs of the project The fixed costs of a project are those that can be associated with an individual project and

do not vary with the level of effort associated with the project.

Variable costs o[ the projects The variable costs of a project are those costs that are directly attributable to one or more

projects and which do vary with the level of effort in the associated projects. Equipment purchase would be a variable cost if equipment requirements were different in different configurations of the project.

The total cost of a project is not easy either to define or to determine. The definition of total project cost is a function of the purpose for which detailed cost estimates are prepared. An evaluation of the overall worth of the organization should include all cost elements, but the fixed costs of the organization, which by definition are not attributable to an individual project, cannot be used in the resource allocation analysis. The interpretation of total costs is very important for two purposes: the determination of the optimum portfolio at a given total cost of the portfolio, and the determination of the optimum total portfolio cost.

Optimum port[olio at specified total costs The basic focus of this research is the development of a methodology for determining the

optimum portfolio of research and development projects within specified resource (cost) constraints. While the general discussion of methodology presented in this section is valid for a wide range of forms of constraints, actual implementation of a methodology requires complete knowledge of the way the constraints are formulated.

The existence of resource constraints determines to a large extent the definition of project cost. The cost of a project should be calculated in a way which is consistent with the constraints. If there is a limit on a particular resource on an annual basis, then demand should be formulated on an annual basis. If the only limit is total demand, then the time stream of demand for a project can be collapsed into a single number. Clearly, when there are multiple resource constraints, costs of a project may need to be calculated many different ways. This is also true when cost is included in the measure of portfolio worth, for example, if worth is defined as the benefit]cost ratio.

Both fixed and variable costs should be included in the estimate of total project costs. Constraints on sunk costs are not meaningful. The constraints on total cost should apply only to costs after the effective date of the allocation decision. The fixed and variable cost of the organization should be subtracted from the available resources unless these are normally covered by other resources.

Research projects are here considered to have a definite beginning and end. While an organization's total research program may be ongoing, individual projects are not. Thus, it is not really meaningful to speak of the worth per unit of resources used in a unit of time for an individual project. A measure of worth for an individual project should be based on aggregate expenditure over time. For the research program as a whole, it is reasonable to use unit time

136 C. MILLER and A. P. SA~E

measures provided that the level of resource utilization for the entire program does not vary substantially over time.

If we accept the use of an aggregate measure of cost, the issue becomes what cost elements to include in the measure. Clearly, for the purposes of assessing project worth, the measure should be comprehensive. The fixed and variable costs of any project must be included. The fixed and variable costs of the organization are not included. While they enter into formulation of cost constraints, these elements are not directly attributable to single projects, and are useless in making comparisons between projects and between portfolios at the same level of total cost. Traditionally, sunk costs should not be considered [20]. The past is unalterable; resources that have been spent cannot be regained. A project that is 90% complete will cost less to finish than a comparable one that has not yet started; but if the net return from the new project is greater than the return from the project underway, considering only the remaining expenditures, then completion of the older project is not the best course of action. However, the negative political implications of consistently ignoring sunk costs, with the possible consequence of frequent changes in the research program, are obvious. The decision maker would appear to be indecisive.

Determination of the optimal portfolio cost The fixed and variable costs of the organization do not enter into the comparison of

alternative portfolios except in determining cost constraints. However, they play an important role in determining how large a research program the organization should undertake. The theory surrounding the allocation of funds to research as a function of the costs and revenue requirements of the organization and the alternative options for investment is a large, complex, and heavily studied area that is related to but not an integral part of this paper.

It is necessary to insure that the marginal worth of the portfolio is not less than the marginal costs to the organization of the program. One of the cost items defined at the beginning of this section was the variable costs of the organization with respect to the size of the research program. The derivative of the variable costs with respect to research program size is the marginal cost to the organization of doing research over and above the direct costs associated with individual projects.

It is a well known principle that no project should be funded if its marginal contribution to the research program is less than its marginal cost. The principle is extended here to require that each project produce a return in excess of its internal costs sufficient to offset the increase in overall organizational costs associated with that project. Often it is very difficult to determine the true cost to an organization of doing a piece of research. Usually, a general overhead rate is established to cover the cost of administration of the project and production support, but this can rarely be expressed as a function of the volume of research.

4.3 Worth of one portfolio When the constituent projects are independent to the extent that the worth and costs of the

portfolio equal the sum of the worth and costs of the projects, evaluation is simple and mathematical programming techniques can be used to isolate the optimal package of projects quickly and inexpensively. When worth and cost are not additive in this fashion, evaluating a portfolio becomes substantially more difficult. The basic complication is that market penetration in a competitive environment generally cannot be calculated simply and accurately.

This section and the one which follows address the problem of assessing the worth of a group of projects when worth is not additive. There are many reasons why the worth of a group of projects may differ from the sum of the worths of its members. Among these are:

• The projects may produce technologies that enhance each other's worth so that the worth of the group is greater than the worth of the technologies alone.

• The projects may produce technologies that compete with each other so that the worth of the group is less than the worth of the individual projects.

Both of these types of interaction are considered in this section.


4.3.1 Enhanced product worth It is difficult to define general classes of ways in which one product can enhance the worth

of another. There are numerous examples of how the development of one specific product has made another more valuable. However, the degree of the enhancement in worth seems to cover a broad range depending upon the specific products under consideration.

4.3.2 Competition between projects This section discusses two simple models of competition between projects that can be

integrated into the portfolio evaluation process. The two models presented are intended for use in different situations. The first is applicable when the markets to which the products are directed are very homogenous. That is, all potential users of a technology have the same preferences. When this is true, each potential user will perceive the same cost and worth for a particular technology. The other model of competition is intended for applications where the end market is nonhomogeneous. Both models assume that a potential end user will always select the technology which meets demand at the lowest perceived cost. This assumption neglects differences in quality of service, unless a measure of the service quality can be rolled into the cost parameter making it more a measure of disutility than simple cost.

Homogeneous market If all of the potential .users of the competing technologies have precisely the same

preferences, then they all perceive the same relative costs for the technologies for the same quality of service. Thus, they will all select precisely the same one. In essence, such a competition is "winner take all". The problem in resolving the outcome of the competition is determining what is the best technology.

Perhaps the simplest model competition construct is based on the assumption that the project with the lowest expected cost will capture the entire market. However, this is not a very meaningful assumption. Unless the cost of the technologies are known ahead of time, in which case there is really no allocation problem to speak of, there is some probability that the technology with the lowest expected cost will turn out to have a higher actual cost when the research is done.

Example Two projects, each costing $300,000 are directed at the development of technologies to

supply the same service. The technology resulting from Project A has a 0.6 chance of supplying the service at

$1.00/unit and a 0.4 chance of supplying it at $2.00/unit. The technology resulting from Project B has chances of 0.3 and 0.7 of supplying the services

at $1.00 and $2.00 per unit, respectively. The worth to the developer of being able to provide the service at $1.00/unit is estimed to

be $2 x 106, while the value of having a supply at $2.00/unit is estimated to be $1 x 106. In practice, determining the return on investment of a product can be a complex task involving a wide range of factors.

The allocation decision is illustrated in the tree of Fig. 3. The expected worth of Project A is:

(0.6) x ($2 x 106) + (0.4) x ($1 x 106) - $300,000 = $1.3 × 106

The expected worth of Project B is:

(0.3) x ($2 x 106) + (0.7) x ($1 x 106) - $300,000 = $1.0 x 106

Given this information and the desire to use the expected worth criterion, Project A is the best choice if only one project can be developed as its expected worth is higher.

It is still possible, however, that Project B, assuming the outcomes, are independent, may produce a better technology. The probability that this will occur is the probability that Project


~ .=.0.~'~1 $2x10 s

~ ~ $2xlOS

Fig. 3.

A will deliver the service at $2.00/unit while Project B will deliver it at $1.00/unit. In this case, (0.3) x (0.4) = 0.12. The probability that the technology produced by Project B will be at least as good as the one developed by Project A is 0.12 + (0.3) x (0.6) + (0.7) x (0.4) = 0.58.

The consequence of uncertainties in the performance of technologies prior to their development is that there is some utility to developing suboptimal projects in addition to the best single choice. The marginal worth of funding an additional project is the difference between the expected worth of the best project and the expected worth of the portfolio which includes both projects. The additional cost of the second project must be included in this calculation.

This brings up the problem of estimating the marginal worth of an additional project to develop a single type of technology. One model of competition that can be used to prepare this estimate in the case of a homogeneous end market is based on the concept that additional projects can increase the probability that some version of the technology will be available at a lower cost thus decreasing the expected cost of the technology and increasing the benefit derived from its application. The example which follows illustrates this approach to estimating marginal worth.

Example Recall from the previous example that the expected net worth of pursuing Project A is

$1,300,000 while the expected net worth of Project B is $1,000,000. Developed alone, Projects A and B have probabilities of 0.6 and 0.3 respectively of

producing a technology to supply the target service at a unit cost of $1.00. The probability that either technology will be available at a cost of $1.00 is the sum of the probabilities that either technology will meet that cost minus the probability that both will. In this case, the probability that one or both technologies will cost $1.00 is:

(0.6) + (0.3) - (0.3) x (0.6) = 0.72.

The probability that neither technology will be available at $1.00 is 0.7(0.4), or 1 minus the probability that only one technology will, or 0.28.

These two numbers represent a relative frequency distribution describing the likelihood of the lowest cost available to the consumer.

Cost Probability that the cheapest technology will be at this cost

$1.00 0.72 $2.00 0.28"


A benefit to society is accrued regardless of which technology is adopted. The frequency distribution of lowest available cost can be used to estimate the expected worth of the portfolio which includes both technologies A and B. In this case:

(0.72) x ($2,000,000) + (0.28) x ($1,000,000) - $600,000 = $1,120,000.

In this case, the expected net worth of the portfolio containing both projects is lower than the expected net worth of either portfolio of one project. Funding both projects is not the preferred decision for this example.

For continuous frequency distributions over cost, x, of a technology, the cumulative distribution of lowest cost is calculated as follows:

F.,,. ( x ) = 1 - fi (1 - E ( x ) ) i=l

where

From(X) = the probability that the cheapest available technology in a market area will have a cost less than or equal to x

F~(x) = the probability that technology i will have a unit cost less than or equal to x in the market area

n = the number of technologies addressing the market area.

Nonhomogeneous markets If all of the individuals who need a particular technology do not have the same preferences,

then the perceived worth of the technologies addressing those needs will not generally be the same for each individual. Again, assuming that worth can be rolled into the economic measure, this means that the perceived cost of applying a single technology that addresses the need will be different for different potential users. When such diversity in perceived cost exists, it is likely that no one technology will capture the entire market. To estimate the market share of each technology, which is an integral part of assessing the worth of different portfolios, it is necessary to develop estimates of the range in perceived cost of delivered service using each technology.

One approach to estimating the cost of service begins with disaggregation of the cost into components. For example, the cost per million Btu of medium Btu gas from a small scale on-site gasification unit includes the cost of the gasification unit, the cost of the coal required to produce one MMBtu of output, the delivery cost, installation cost, and maintenance cost. Since there can be a wide variation in the cost of service due to factors other than the off-the-shelf cost of the technology, it is important to make the distinction between technology cost and service cost when the market is dishomogeneous.

If the preference functions of the potential users vary as well as the cost, these variations must also be assessed. Disaggregation of worth into components in a manner analogous to the cost disaggregation will be useful, but the general assessment procedure may be very different. If certain conditions prevail, it is possible to construct a cost frequency distribution for total cost of service from the distributions for each cost element. Basically, the necessary condition for this calculation to be possible is that the relationship for each technology between the user's perceived costs in any two cost elements be known. For example, if it can be determined that for all potential users, the perceived cost in one component (e.g. delivery cost) has no bearing on the perceived cost in another component (e.g. installation cost), then it is possible to construct the distribution of combined delivery and installation cost by Monte Carlo simulation. Similarly, if a mathematical relationship can be determined relating a user's perceived costs in one element to the perceived cost in another element, then these can also be combined into a single distribution. If enough is known about the relationship between cost elements, a single cost of service can be developed for each technology. Care should be taken in defining the cost elements to insure that the necessary conditions will prevail for performing this combination.

140 C. MILLER and A. P. SAOE

If distributions can be developed for total cost of service for each technology it is possible, under some circumstances, to calculate the fraction of all potential users who see each technology as the least costly means to meet their need. Assuming that each user will select the technology with the lowest perceived cost of service, this fraction is the technologies' market shares.

The necessary condition for estimating market share is analagous to the condition necessary for combining the distributions of the different cost elements into a total technology curve. That is, the relationship between the costs associated with the different technologies must be known. The calculation of market shares is possible if the cost of one technology infers nothing about the cost of the others, or if there is an exact relationship between the two costs which holds across all users.

The case where the costs are independent between technologies is illustrated in Fig. 4. In this case, there are two technologies, A and B, competing with a conventional technology C. The X axis is perceived technology cost. The Y axis is the probability that a randomly selected user of the service supplied by these technologies will see a price less than or equal to the corresponding value on the X axis. There is a separate curve for each technology. In this graph, 50% of the users perceive a cost of less than $3.00/MMBtu for coal gas obtained using technology A.

In the sample figure, no user perceives the gas from technology B as costing less than $2.50/MMBtu, while 22% of all users perceive technology A as capable of producing gas at less than this price. These 22% will certainly adopt technology A. All users perceive the cost of gas from technology B as less than $3.50. This is cheaper than any user's perception of the conventional technology. This implies that the entire market will go to technologies A and B.

There is a limitation to the use of this model of product competition for research products with highly uncertain performance. The cost distributions reflect only the variation in the perceived price of the products over the potential users at the time the purchase decision is made. The distributions do not include any measure of uncertainty in the price of the products due to uncertainty about their performance. In fact, it is at least very difficult to treat technical uncertainty in this construct. Thus, this approach is primarily useful for products near ready for adoption, so that market dishomogeneity is the dominant uncertainty.

5. EVALUATING MANY PORTFOLIOS

The problem of allocating resources to research and development concerns portfolios. First

1.2

I 0

0 8

06

O.4

0 2

0 I

0 I

Fraction of total market that perceives cost ~ - corresponding $ value

14

2 3 4 5

Cost of service, ~ MMBTU

Fig. 4. Service cost distributions.


there is the consideration of goals for the research or development and the formalization of these into distinct measures of worth with which to measure project attainment. Then, the focus turns to the individual project. How is each individual research project evaluated? When this question is answered, attention turns to evaluating a group of projects. An essential question is how the value of the group differs from the sum of the values of the individual projects, and how can this aggregate worth be estimated. Our last section discussed two ways to address this problem. With a technique developed to evaluate one portfolio, attention is turned to translating that technique into one that can be used in a search for the optimum portfolio.

Conceptually, it is simple to determine the optimal portfolio given a scalar objective function and a mechanism to evaluate it--the mechanism is simply applied to every possible combination of projects and the best selected. The problem with this approach is that the number of possible combinations is often too large to manage. If there are n competitive projects, there are 2 ~ portfolios which must be evaluated. This number includes n portfolios with only one project, and the null portfolio so the actual number of multiple project proposals that must be considered is 2 ~ - n - 1. Even at one second of computer time per portfolio, the running time exceeds an hour when there are 12 projects and a year when there are 25. The solution to this curse of dimensionality is to evaluate only those projects which are serious contenders for the optimum. The problem is to develop a heuristic algorithm or technique to accomplish this.

Because, more often than not, realities of a research program place program analysis beyond the reach of rigorous mathematical techniques, the methodology developed here is heuristic. It has the benefits of a heuristic methodology (wide applicability, easier for the program manager not trained as an expert in operations research to understand), but it also has the serious problem that it may, if inappropriately applied, yield a wrong answer. To reduce the risk of a wrong answer, and to build confidence in the results, the search should be extended as far as possible. The key to extending the search successfully over a very large number of portfolios is computational efficiency.

There are at least three ways to select the optimum portfolio from among a large number of alternatives:

1. Reduce the number of alternatives by some sort of screening process. 2. Constrain the evaluation technique so that it is compatible with nonenumerative opti-

mization techniques such as mathematical programming. 3. Improve the numerical efficiency of the evaluation process so that a larger number of

alternatives can be considered.

These procedures are, of course, neither mutually exclusive not exhaustive.

5.1 Limiting the number o[ alternative port[olios Three general classes of techniques for limiting the number of portfolios that must be

considered are discussed in this section. These are screening, ranking, and the use of stochastic dominance concepts.

5.1.1 Screening Perhaps the simplest and most effective way to reduce the number of portfolios that must be

considered is a quick screening procedure. The most effective screen is one which eliminates projects from consideration. The number of portfolios that must be considered is reduced by a factor of two for every project eliminated.

The use of screens of this sort for portfolio analysis implies the assumption that if a project does not produce sufficient return to justify its cost when developed alone, it will not produce a sufficient marginal return when included in a research portfolio to justify its cost. That is:

E(A) >_ E(P + A) - E(P) for all portfolios P

142

where

C. MILLER and A. P. SAGE

E(A) = the expected net worth of Project A developed alone E(P + A) = the expected net worth of a portfolio of Projects P

(not including A) and Project A E(P) = the expected net worth of the portfolio of Projects P.

This relation is valid in many cases, such as when worth is additive, or if either of the models of product competition presented in Section 4 pertain. However, the relation may not be valid if the projects augment each other's worth[21].

In addition to screening on a project level, portfolios can be eliminated from consideration prior to evaluation if they are in violation of constraints on the composition of a portfolio or if they can be determined a priori to be inferior to other portfolios. Constraints on the composition of a portfolio can take many forms. Constraints on total cost and other limited resources are common. It is also common for there to be constraints on the composition of the organization's research portfolio to limit concentration in too narrow an area. Constraints of this sort can require a minimum commitment to a range of subject areas or administrative divisions.

Comparison of the composition of a portfolio against a set of constraints is generally a trivial task, and there are certainly savings in doing this compared with the often complex procedure for complete portfolio evaluation with a large number of portfolios.

5.1.2 Ranking The simplest form of portfolio analysis occurs when the worth of the projects in a portfolio

is additive. In this case, the optimal portfolio at each funding level can be obtained from a simple benefit cost[20] ranking, provided that partial funding of marginal projects is possible without diminution of the return per unit investment. Even when partial funding is not allowed, the portfolio obtained from a benefit cost ranking is a good approximation of the optimum 'when the number of projects is high.

Whether a ranking procedure can be used when project worth is not additive depends on precisely how the worth of a portfolio deviates from the sum of the worth of the constituent projects. The general problem of determining that ranking can be used for a particular method of assessing portfolio worth can be shown through proof by induction. The problem can be stated as follows:

Suppose that we are given a set of projects A,, i = 1 to n, and the relationship that E(A3 < E(A;+I), i = 1 to n - 1, where E(A3 is the expected worth of project A,.

Ranking can be used to determine the optimal portfolio if conditions

E(P = AO <- E(P = A2)

and

E(P = X & Am) <- E(P = X & Am+O for any m

are satisfied, where

E(P = A3 is the expected worth of the portfolio containing only project A

E(P--X&Ai) is the expected net worth of the portfolio containing an arbitrary group of projects X and project Ai.

The first of the two conditions is given. The second can be difficult to prove if the worth of a portfolio is not a simple mathematical function of parameters of the individual projects. The forms of the models for product competition presented in Section 4 make it unlikely that


ranking techniques could be used with either. Specific proof or disproof of the validity of using ranking techniques requires specification of the relationship between technology cost and worth. This is a function of the specific types of technologies being studied among other factors.

Ranking within subsets of projects can be used to reduce the size of the evaluation process even when a single ranking of all projects is not feasible. Such simplification is possible if the projects can be divided into groups such that the worth of a portfolio is the sum of the worth of each group of projects contained in the portfolio. In essence, this means that the technologies addressed by the different groups do not interact in any way. Product interaction within groups is allowable.

For example, it may be possible to divide an appliance R&D portfolio into two groups, one containing n refrigerator projects, and one of m dishwasher projects. The worth of a portfolio containing projects of both sorts is the worth of the dishwasher projects plus the worth of the refrigerator project if:

1. the worth of having one appliance is neither augmented nor diminished by having the other

2. the two appliance types do not compete for adoption in any way.

The second assumption would seem to be approximately correct in this case as refrigerators and dishwashers do not compete for the same applications, but the validity of the assumption is limited by the possibility that the two appliances may compete for limited consumer capital.

When the portfolio can be divided into groups in this fashion, a separate analysis can be performed for each group. The output of each group analysis is the worth of each possible portfolio composed of projects within that group. These can then be ranked in order of ascending cost. This done, it is simple to eliminate projects which cost more than another but have lower gross worth. These portfolios are poorer in two principal aspects and cannot be the optimal choice at any level of funding.

If the remaining list of portfolios is large, it can be viewed as defining a curve relating worth to cost for each group. The derivative of these curves is the marginal return on investment for each group. In many cases, there will be diminishing marginal returns on investment. When marginal return on investment is monotonically decreasing with increasing cost, these curves can be useful in determining the optimal composition of the portfolio involving projects from all groups.

The total portfolio can be viewed as a portfolio of group portfolios. The selection of the portfolio from each group to be included can be viewed as an investment decision: what resources should be allocated to each group? Since our objecteve is to maximize the expected worth of the entire portfolio, the optimum at any level of funding is the one which just meets the cost constraint (assuming there are enough projects) and is composed of group portfolios with the same marginal return. That this portfolio is the optimum is shown in the following example.

Example Consider the case where there are two independent groups of projects A and B. It is

hypothesized that the total worth of a portfolio consisting of a group of projects taken from each of these groups is a maximum, at a particular level of funding, when the marginal return from the two groups are equal. In this region, the marginal worth curves appear in Fig. 5. The hypothesized optimal allocation of funds to A and B are Ca and Cb as shown in Fig. 5.

At the hypothesized optimum, the entire portfolio has a net worth of W. The cost is C total which is the sum of Ca and Cb, shown on the curves above. Consider any perturbation from this point. Assume that the allocation to group A is increased by an amount D, and the allocation to group B decreased by the corresponding amount to maintain the maximum permissible total portfolio cost.

The worth of the portfolio including the cost perturbations is given by

Wtotal~d = Wtotal+ fC°+°MWa(x)dx- f ca MWb(x)dx J C a J C a - D


Morglnol worth Morginol worth

Co Cb Cost of group A Cost of group B

Fig. 5.

where:

Wtotalperturbed - - - - - worth of the perturbed portfolio W t o t a I = worth of the original portfolio

Ca = original allocation to group A MWa(X) = the marginal worth of group A at

the level of allocation X D = the cost perturbation.

The first integral is the increase in portfolio worth due to the additional allocation of funds to group A. The second integral is the decrease in portfolio worth due to the reduction in the allocation of funds to group B. It is clear that the value of the first integral is smaller than the value of the second integral as the integrals span equal ranges and the marginal worth of A is lower over the range of its integral than the marginal worth of B is over the range of its integral.

Thus, any perturbation of the allocation from the one which equalizes marginal worth between independent groups decreases the worth of the whole portfolio.

When the conditions exist which permit the division of the projects into independent groups, when the number of portfolios is high enough to approximate a continuous range, and when marginal worth decreases monotonically with increased allocation of funds, the number of portfolios that must be evaluated can be reduced dramatically. The number of portfolios that must be considered is given by

%

2/, i=1

where

ng is the number of groups

mi is the number of projects in group i.

The process of ranking and screening portfolios to determine the curve of marginal returns and determining the optimal allocation between groups is relatively simple.

5.1.3 Stochastic dominance Consider the case where there are two projects A and B with certain outcomes resulting in a

net gain in worth of W^ and Wn respectively. If WA is greater than WB, then project A can be said to dominate project B. If the outcomes of projects are not certain, the concept of dominance is still useful. While the expected net worth of one project may be greater than the expected net worth of another, there is still the possibility that the actual return from the projects will be contrary to the expectations. Stochastic dominance is a term applied to the relationship between the returns from projects with uncertain outcomes when it can be shown


that regardless of which uncertain event occurs one project is always better than those stochastically dominated by it.

First order stochastic dominance occurs if the probability that the dominated project will have a net worth of less than a value X is at least as great as the probability that the dominating project will have a net worth less than X for all values of X, and this probability will be strictly greater for at least one value of X [22]. That is, project A is first order dominant over project B if

FB(x) >- Fa(X)VX

FB(x) > F^(x) for at least one x

where

Fs(x) = the probability that project B will have a net worth less than x

FA(X) = the probability that project A will have a net worth less than x.

Second order stochastic dominance is a less stringent criterion than first order dominance. Project A is second order dominant over project B if[23]

f_ (FA(x) -- FB(x)) dx < 0Vz

where all variables retain the meanings used above. In many cases, it is possible to use stochastic dominance relationships to reduce the number of portfolios which must be evaluated.

Example Consider four projects A, B, C, and D. There are 2 4 = 16 possible portfolios:

1. A&B 5. B&D 9. A&C&D 13. B only 2. A&C 6. C&D 10. B&C&D 14. C only 3. A&D 7. A&B&C 11. A&B&C&D 15. D only 4. B&C 8. A&B&D 12. A only 16. Null

If it is known that B dominates C which dominates D, then there are only eight portfolios which can potentially be the optimum:

1. A&B 5. B&C&D 2. A&B&C 6. A only 3. A&B&C&D 7. B only 4. B &C 8. Null

6. METHODOLOGY

We began our efforts with a brief introduction to the general phases of a systemic process directed at determining an effective allocation of resources for research and development. Subsequent sections described tools of use in the allocation process. Here we provide a summary discussion of the developed methodology. The steps of the procedure are shown in Fig. 6. The general theme of the procedure is screening of candidate projects and candidate portfolios using increasingly refined estimates of performance or worth until the number of remaining possibilities is small enough for a detailed assessment of worth.


V --I I I 1

I I I I I._

L-

IDEA GENERATION

I Idenufy goals of ] the research program I

i Determine measures of |

Iperformance that assess attain.- I Iment of organizational goals ]

Identify areas of ] ~ l ~ I t ' ~ ' ] ~1 Identify new unsatisfied demand technological options.

I i * I * ,-.---~ 6 1 4 6 I

Identify technologies that L - - - - - - - - . ~ A I q ~ Identify demand for new I address these demands i_._ __ __~ '~ '~_ ~_ _ _ ~ I technologies I

7 ormulate specific project plans 8 manc

P' s / / /

9 i

MARKET PERFORMANCE

SCREENING

12 Establish mtmmum market J~

performance of new technologies I

t3

I A,,es, market rformance [ of new technologies

Review standards and ] performance of new technologies I

I Ehmlnate inferior technologms.

_..I

Assess performance of new technologies

i 10 | Review standards and

I Nrformance of marginal technologies

i 11 [ Ehminate mfer:or

I technologies

TECHNICAL PERFORMANCE

SCREENING

_.1

16 ~ [ Define potentlaly optimal Y[ portfolios

I Prepare estimates of the worth of candidate portfolios

Identify final candidate portfolios

i9 ~ Assess worth of final

candidates

20 ~ Review evaluat,on procecmre ]

and make final recommendahons /

PORTFOLIO EVALUATION

L _

Fig. 6. General methodology for portfolio prioritization.


6.1 Goals of the research Any exercise in research planning must begin with a study of the goals of the research

program [24, 25]. In some cases, those planning the research are given the goals and a mandate to conduct the research. Some organizations have a very narrowly focused purpose. The goals may be focused on creation or development of a particular technology and may not allow the consideration of whether other technologies may address the same purpose more effectively or whether other avenues of research may better serve the needs of the sponsoring organization.

6.2 Measures of performance Once the goals of the research program have been fixed, it is necessary to determine how

attainment of these goals will be measured. This is not difficult to do in some cases but impossible to do in others without ambiguity. In many cases, the difficulty in assigning a measure of performance depends on how well the goals have been defined.

If there is a single goal to the program and it is very specific, e.g. to develop an energy efficient refrigerator, it is fairly simple to select a measure of performance. In this case, perhaps energy cost of operation per unit of volume at specific interior conditions under specific ambient conditions. If however, there is a single goal that is very broad, e.g. to improve the quality of life in America, there are no universally accepted measures of program worth. Step 1 is not complete. A goal such as this provides no direction to the research program and has no functional value other than on the first step toward more specific goals.

When the results of the first task do not lead to a convenient measure of performance, the difficulty often lies in assessment or interpretation of program goals. However, it is also possible that a definitive statement of program objective involves multiple and conflicting goals. When this is the case, it is necessary to study the tradeoff between the differing goals in order to derive a scalar objective function or to accept that the analysis requires multiple objective optimization and ranking [4].

In some cases, an aggregate measure of worth can be derived by linearly combining measures of the various goals. The scores for each goal can be added if weights measuring the selective importance of the various goals are assessed. The weights, given each of the measures, can be difficult to determine. Basically, the approach is to consider simple questions comparing two measures at a time. The questioning would progressively refine the estimate of the correspondence between the two measures and then move on to a comparison of another measure with one of the original two. This procedure is continued until all measures have been included. It is, of course, necessary to establish these performance measures prior to eliciting weights of the various goals or attributes [1].

The basic objective of the process is to determine weights that translate the independent measures into a common unit of utility. Some of the independent measures will be in physical units while others will be unitless subjective scores. For example, an increase in usable refrigerator space from 83 to 93 ft may not have the same value as an increase from 143 to 153ft. This nonlinearity can be resolved by using a weighting value that is a function of the value of the measure.

If there are several different groups or individuals involved in the decision, it is unlikely that all will perceive the same trade-off between measures. In very polarized cases, an increase in a measure may enhance perceived worth for one group while diminishing it for another; or one group may be insensitive to measures of great importance to another group.

Jacques Gros has suggested a way for combining the opinions of different groups [26]. He begins by assessing each group's perceptions of the worth of the measures.

He then develops weights which measure the relative importance of each group's opinion in the overall decision. This weight can be an indicator of the power (political, economic, or other) of the groups relative to each other. The overall relative worth of each measure is then taken to be the average of the relative worths assigned by each group multiplied by its associated weight. In essence, the opinion of all groups is taken to be a weighted linear combination of the opinions of each group. An obvious needed effort involves assessing group weights. In a corporate structure, various levels of corporate management may be able to assign weights, but


in the public sector it is difficult to assess either the actual or desirable level of importance of the various interest groups. There are a number of methods which may be used to estimate group values [1,22] and an appropriate one of these can be utilized here.

6.3 Steps 3-6: Generating ideas Steps 3-6 are directed at the same purpose; generating ideas for research and development.

Thus, they can be considered together. It is impossible to prescribe a sequence of steps that will inevitably lead to good ideas, but several approaches have been put forth with the intent of increasing the likelihood of finding good opportunities, and a great deal has been written about procedures now used with some degree of success. Souder and Ziegler[28] have written a brief but impressively broad survey of idea generating techniques. Geshka[29] and associates have done a survey of users of idea generating methods to determine levels of satisfaction with their use. Warfield[30] and Sage[l] also discuss a number of idea generating techniques.

Tasks 3-6 are shown as distinct elements of the methodology. In practice, it is likely that the process of idea generation will actually constitute a single task embodying elements of each of the separate tasks both in parallel and in series. The creative process would not seem to be sufficiently malleable to fit the structure shown. However, the bifurcation of the idea generating pha~e into two distinct paths illustrates the distinction between two sources for explanation of behavior, and the steps shown are a good way to begin the thinking process and to sustain it until ideas begin to flow.

The two approaches to idea generation shown in the methodology are correlated with two sources of invention commonly called demand pull and technology push[31]. Demand pull describes innovation arising from recognition of an unsatisfied demand. The innovator, noting a need, and considers ways to address that need until one or more viable approaches are found. Innovations arising from technology push are those that arise because changes in available or achievable technology make it possible to do something that was not possible before, or to do something in a new and better way. The force behind technology push may be a new technology, or it may be a new material, process, or application of existing technology. Conceivably, a new view of an existing technology can be a source of inspiration.

Quinn[32] suggests that such a survey should take the form of a three phase forecasting exercise. The first phase is economic. The goals of the phase are in large part directed at determining the level of research which the organization can support, but the idea generation phase is targeted at determining the economic factors that influence the need for and prognosis for acceptance of new products, technologies, and services. Quinn argues that an economic forecast is essential as the demand for new products is highest in expanding segments of the economy and lowest in those that are declining. The areas of economic growth are the principal markets for the research. Quinn's perspective is that of the private sector. In the public sector, the purpose of the economic forecast is to identify segments of society which are likely to be concentrations of demands for projects by virtue of their economic health.

The second of Quinn's forecasts is a sociological forecast. The objective is to identify changes in the society which effect the demand for new technologies and the environment for innovation. Specific areas of the forecast include [32].

• demographic structure--location and makeup of the future population • spending priorities--public vs private consumption, preference shifts within product

groups, acceptability of new technologies, emphasis on various areas of expenditure • role of government--activities of government which influence the demand for new

technologies through both direct purchase and regulations affecting the use of existing technologies.

• public and legal attitudes toward business---definition of monopoly, patent sanctity, public control of product, material, and service proces.

• international affairs--foreign markets, foreign political shifts, tariff barriers, international currency stability

• labor conditions--labor conditions favorable to or adverse to the introduction of automation, ability of the labor pool to respond to manufacture of new products


• education--number of trained scientists and engineers, level of consumer sophistication, products of university research.

The technical forecast consists of determination of the ongoing effort to stay abreast of the developments in the general scientific community and among competitors, and the much more intensive effort to stay abreast of the emerging technological needs of the customers. Again, the private sector orientation is evident, although adaptation to the public sector follows readily. This approach has utility in identifying opportunities for technology push, but is much more oriented at studying demand. Certainly it is easier for a research organization to address known needs, and the inscoping, problem oriented sort of thinking associated with this is easier than the outscoping effort that is needed to identify the potential in new technologies.

6.4 Step 7: Develop project plans The resource allocation decision cannot be based on a vague concept of work areas.

Specific, detailed work plans, with a schedule, a budget, and a forecast of other resource requirements must be prepared. Work plan preparation is a normal part of every business and government organization, and while there is certainly room for further development of technique, so much effort has been directed at the problem that many sound procedures have been developed to the point of step by step instructions.

Two practical approaches.to planning are suggested by Delembo[33] and Birnbrauer[34], who employ forms and checklists to direct the project planner's efforts. This is a common approach. The forms provide a written proven guideline to the planning process then also serve to standardize planning across divisions of a large organization, thereby simplifying inter- divisional coordination. The "forms" approach to project planning would seem to be more accurate when applied to a small, well defined work element rather than to a large, complex project. Most approaches to planning approach complex tasks by disaggregation into compact elements, and reaggregation after each element is planned independently. A vast amount has been written about scheduling and managing multistage projects. Graphical tools have proven to be very useful[35,36]. PERT and the Critical Path Method (CPM) are commonly recommended [37, 38]. The DELTA chart is a PERT-like tool with great flexibility, and is a very practical alternative [30].

6.5 Steps 8-11: Technical evaluation and screening Steps 8-11 should be regarded as a unit. In these steps, estimates of the performance of the

products of the research are developed and compared against minimum standards. Unsuccess- ful projects are eliminated from further consideration.

The technical evaluation process begins by developing standards of performance. The initial step in this process must be to characterize the product of the research. Product charac- terization should concentrate on precise definition since the objective is to define the market for a product well enough so that competing technologies can be identified.

The product may be a component of another technology. One complication is identifying the technical constraints that limit the use of the component in some application. Another is being sure to interpret the uses of the product broadly enough to encompass all realistic applications. Assessing the applications of any component is more difficult than assessing the applications of a stand alone product. Once the major potential applications of the products have been identified, competing technologies, if any, can be characterized. The objective is to establish a minimum standard of product performance and to eliminate obviously inferior projects. A simple standard to use is the level of performance of the technologies serving the same functional use. It is doubtful that a new technology which does not perform as well as the one currently in use will ultimately prove successful. A standard based on current technologies is a very conservative one. More stringent standards might require performance in excess of some improved version of existing technology or expected performance of other developing technologies. While a more stringent standard of this sort will eliminate more projects, simplifying the allocation process, the risk is greater that projects will be eliminated which ultimately would have been successful.

When a new technology serves a function which is not addressed by conventional ones, it is

CAEE Vol 8, No 2--E


necessary to base the technical screening standard on either a forecast of the performance of developing competing technologies, estimates of the requirements for successful interface to related technologies, or an assessment of the minimum level which will be acceptable in the final market place.

Product performance can rarely, if ever, be described or assessed by a single attribute. Even if product quality can be measured with a single attribute, in most types of products there is a cost/quality tradeoff and, thus, a minimum of two attributes of product worth. Selection of the level of performance to be used in the technical screening steps may be very difficult. The fundamental issue is how the new technology differs from its competition in ways that influence the acceptability of the technology.

It is certainly possible to use one set of measures of performance in the technical screening, another in the market performance screening, and a third in the final evaluation of portfolio worth. However, it would obviously reduce the data collection effort if the same measures of performance can be used in all of these steps. This can be done if the characteristics of the product that determine that its worth to the consumer can also be used to predict the worth of the project to the developer.

Once the markets have been identified, and technical screening measures selected, the problem at hand is assessing technology performance. This is perhaps, the most difficult element in the allocation process--predicting how well a technology will work before it has been invented. Estimates of performance need not be deterministic. If in fact, it was possible to determine with precision how well the technology would work, the allocation process would be a trivial exercise. The problem is one of eliciting an estimate of technology performance and the associated uncertainty, and integrating the differing opinions into a single estimate, preferably expressed in a standard form such as a frequency histogram.

The screening process should not be just a simple comparison of performance estimates with standards. First of all, the performance estimates have some uncertainty associated with them. Projects which narrowly fail to meet the screening criteria may fail not because the project is actually deficient but because of errors in the estimate. A further level of technical screening iteration may be necessary to accommodate information developed in the estimation process. If the performance of the new technologies is deemed to be much higher than originally expected, it may be reasonable to assume that the results of developing competitive technologies may also be better than expected. A higher threshold value would serve to increase the probability that any technology developed would compete successfully. Further, if performance is better than expected, more projects than desired may be left after the initial filtering. A more stringent standard would address this problem.

6.6 Market performance screening Steps 12-15 are analogous to Steps 8-11. The first screening was based on the performance

of the technology. A technology was retained if its performance, cost considerations included where relevant, was at least as good as that of competing technologies. The second screening phase addresses the question of how that technical advantage translates into acceptance of the product by the ultimate consumer and whether the degree of acceptance is sufficient to justify development. Evaluation of market performance is done for each technology independently, assuming that potentially competing projects within the organization are not developed. As there is one less level of competition to be considered, the independent evaluation is substantially simpler than general portfolio analysis.

Screening should be more of a market sensitive project worth filter than a pure market one. For example, a profit seeking organization might use return on investment as a screen. Two critical connections must be made to implement a market related screen. First, market performance must be related to the measure(s) of technical performance developed in Step 8. Second, the relationship between number of units of the technology adopted and return on investment must be established. The first issue was addressed briefly in the discussion of market penetration estimation earlier in this section concerning our methodology. As in the technical screening, the standard should be reviewed as should the estimate for any projects near the standard.


6.7 Step 16: Define potentially optimal portfolios As with the preceding eight steps, the objective of Step 16 is to reduce the number of

candidate portfolios. In this step, portfolios are eliminated in several ways. In general, it is much easier to estimate the resource requirements of a portfolio of projects than to estimate the worth. Thus, it is preferable to verify that resource requirements are within constraints before proceeding through the portfolio evaluation phase.

Depending on the mechanism which will be used to evaluate the worth of a portfolio, several techniques can be used to further screen portfolios. Ranking and the use of stochastic dominance relationships were two techniques discussed earlier. The final output of Task 16 is a generally small list of portfolios that are potentially optimal.

6.8 Step 17: Prepare estimates of the worth of candidate portfolios Our preceding sections have addressed the problem of evaluating the worth of a portfolio.

Basically, the appropriate model of portfolio worth is applied to each candidate, or, if the number of portfolios is large, a formula which is essentially equivalent to the model of worth is developed and substituted. If the process for evaluating portfolios is very time consuming and/or expensive, it may be possible to integrate yet another screening step into the evaluation process. It may be possible in some cases to infer that one portfolio is not consistent with organizational objectives on the basis of the evaluation of another portfolio. Care should be taken, however, to insure that the cost of such screening does not exceed the cost of simply proceeding to evaluate all of the candidates.

6.9 Step 18: Final screening Based on the results of Step 17, those portfolios which are obviously inferior should be

eliminated.

6.10 Step 19: Final portfolio evaluation The portfolios which survive the final screening in Step 18 are potentially the optimum.

Therefore, great care must be taken to assure that evaluation is done with the best possible accuracy and precision. Refinements at this stage fall into two general classes, improved execution of the original assessment process and inclusion of other factors.

The procedure used to arrive at the final evaluations should be reviewed. If the results are particularly sensitive to any factors, attempts should be made to refine the values used. It is also possible at this stage to include factors that could not be included in the initial analysis. The evaluation process will always be limited in scope due to the massive chore of sifting through the many possible portfolios. While the major factors influencing portfolio worth must be included, factors that effect worth in a minor way should initially be omitted. At this stage, the number of portfolios should be small enough that additional factors can be assessed, in a timely matter, for each portfolio.

Assessment of the potential market is not difficult unless the technology is of a totally new type. The historical market of the existing technology most comparable to the new one and historical turnover rates provide a good indication of the maximum rate at which a new technology will be adopted. Predicting the actual rate is more difficult, however, and the subject of estimating market penetration rates was discussed earlier. Several mechanisms for assessing the worth of portfolios were also discussed. These ranged from the case where the projects did not compete with each other for market share at all to the case where they competed for precisely the same market. In Step 19, project competition must, of course, be considered.

6.11 Step 20: Review and sensitivity analysis After completing the analysis, it is reasonable to ask if it was done right. After seeing the

research possibilities, are the original goals correct? Were the measures used really appropriate, and was the assessment procedure really the best? To what inputs are the results most sensitive? How can the inputs be improved? In essence, the entire analysis should be done over, at least as a Gedanken experiment and perhaps in actuality.


REFERENCES 1. A. P. Sage, Methodology for Large Scale Systems. McGraw-Hill, New York (1977). 2. A. P. Sage, (Editor) Systems Engineering--Methodology and Applications. IEEE Press (1977). 3. A. P. Sage, Systems engineering linkages, modeling and policy analysis. Policy Anal. ln[o. Sci. 2(3), 66--77,

April (1979). 4. C. C. White, llI and A. P. Sage, A multiple objective optimization based approach to choicemaking. IEEE Trans.

Systems, Man and Cybernetics 10(6), 315-326, June (1980). 5. I. S. Serui, Multi-objective research planning. Research Management, pp. 24-26, Sept. (1976). 6. R. G. Murdidi, Mathematical Models in Marketing. Scranton, PA: Intext Educational Publ. (1971). 7. H. Raiffa, Decision Analysis. pp. 256-264. Addison-Wesley, Reading, Massachusetts (1970). 8. R. Keeney and H. Raiffa Decisions With Multiple Objectives, Wiley, New York (1976). 9. A. Tversky and D. Kahneman, Judgment under uncertainty: heuristics and biases. Science 185, 1124-1131 Sept. (1974).

10. A. P. Sage and E. B. White, Methodologies for risk and hazard assessment--A survey and status report. IEEE Trans Systems, Man and Cybernetics 10(8), 425--446, Aug (1980).

I I. J. C. Fisher and R. H. Pry, A simple substitution model of technological change. TechnoL Forecasting and Social Change 3 (1971).

12. R. C. Lenz and H. W. Lanford, The substitution phenomenon. Business Horizons, pp. 63-68 Feb. (1972). 13. E. Mansfield, Technical change and the rate of limitations. Econometrica 29(4), Oct. (1961). 14, J. Nevers, Extension of a new product growth model. SIoan Management Rev. pp. 77-91. Winter (1972). 15, A. D. Bain, The Growth o/Television Ownership in the United Kingdom, A Lognormal Model, The University Press,

Cambridge (1964). 16. G. H. Haines A theory of market behavior after innovation. Management Sci. Oct. (1960). 17, General Electric Company, National solar demonstration program, technical volume. GE Document 755D54265 April

(1976). 18. B. V. Dean, Research project cost distributions and budget forecasting. Quantitative Decision Aiding Techniques

[or Research and Development, (Edited by M. J. Cetron, et aL), Gordon and Breach, New York (1972). 19. E. Mansfield, Econometric Studies of Industrial Research and Technological Innovation. Norton, New York (1968). 20. C. W. Haley and L. D. Schall, The Theory o[ Financial Decisions. McGraw-Hill, New York (1973). 21. W. Locander and R. Scammel, Screening new product ideas--A two phase approach. Research Management, pp.

14-18, March (1976). 22. R. K. Harvey and A. V. Cabot, A decision theory approach to capital budgeting under risk. Engng Econ.

20(1), 37-49 (1976). 23. D. W. Bunn, Screening methods in policy analysis. Socio-Economic Planning Sci. 12(3), 329-331 (1978). 24. M. J. Cooper, An evaluation system for project selection. Research Management, pp. 29-33, July (1978). 25. A. L. Froham, Critical mid-management functions for innovative R&D. Research Management, pp. 7-13, July (1976). 26. J. Gros, Power plant siting--A paretian environment approach. Nuclear Engng Design M(2), 251-292,

November (1975). 27. D. R. Farris and A. P. Sage, Introduction and survey of group decision making with application to worth assessment.

IEEE Trans Systems, Man, and Cybernetics SMC-5(3), 346-356, May (1975). 28. W. E. Souder and R. W. Ziegler, A review of creativity and problem solving techniques. Research Management, pp.

34-41, July (1977). 29. H. Geshka, G. R. Schaude and H. Schlichsupp, Modern techniques for solving problems. Chemical Engng, pp.

91-97, August (1973). 30. J. N. Warfield, Societal Systems. Wiley, New York (1976). 31. E. Mansfield and S. Wagner, Organization and strategic factors associated with probability of success in industrial

R&D. J. Business 48(2), 179-199, April (1975). 32. J. B. Quinn, Long-range planning of industrial research. Harvard Business Rev., pp. 88-102, July (1961). 33. W. E. Delembo, Project time estimates. Machine Design, pp. 66-69, August (1976). 34. H. Birnbrauer, Project planning made simple. Machine Design, pp. A,4 ~9, May (1977). 35. D. G. S. Davies, Research planning diagrams. R&D Management 1(1), 22-29, Nov. (1970). 36. R. C. Parket and A. J. P. Sabberwal, Controlling R&D projects by networks. R&D Management 1(3), 147-153, June

(1971). 37. H. Eisner, A generalized network approach to the planning and scheduling of a research program. J. Operations

Research Soc. 10, 110--121 (1962). 38. W. E. Souder, Projects selection, planning and control. Handbook Operations Res., Models and Appl. 2, (edited by J. J.

Moder and S. E. EImaghraby). Van Nostrand Reinhold, New York (1978).

Documents

A methodology for the evaluation of research and development projects and associated resource allocation