IEEE Energy2030 Atlanta, GA USA 17-18 November, 2008

A Framework for Portfolio Management of Renewable Hybrid Energy Sources

Tommer Ender1, Jonathan Murphy1, Comas Haynes2

1Aerospace Systems Design Laboratory

School of Aerospace Engineering Georgia Institute of Technology

Atlanta, GA 30332-0150

2Georgia Tech Center for Innovative Fuel Cell & Battery Technology Georgia Institute of Technology

Atlanta, Georgia 30332-0853

Abstract – Existing models of various energy assets do not consider social, economical, and environmental factors in addition to the technical. An energy systems modeling tool must address variability of ecological and socio-economic sensitivities in order to practically guide policy and budget related decisions. The aim of this effort is to produce an advisory and design tool geared toward aiding entities with robust planning and implementation of effective renewable energy solutions based on trusted models. A tool was developed that enables tradeoffs between various energy systems, based on neural network surrogate models of a publicly available power systems modeling tool. These surrogate models enable the higher-level decision making tool to manipulate surrogate representations of actual engineering models, as opposed to relying on qualitative or expert-driven estimations which are traditionally used in this regard. This research will present a decision-maker with the ability to determine which various renewable and non-renewable energy systems meet annual energy load requirements, acquisition and operation costs, and individual solution attributes.

I. INTRODUCTION

A. Motivation

Changes in lifestyle, especially regarding production and consumption, will eventually be forced on populations by ecological and economic pressures. Technical, societal and economic insights and preparedness will facilitate this requisite transition. The goal of this pilot research is to develop an initial tool for distributed hybrid renewable energy system design and implementation which sufficiently characterizes viable renewable energy candidate technologies, inclusive of conversion processes and storage media. Existing hybrid energy design tools allow the selection of systems based on economic objectives [1][2][3] or combinations of performance and economic objectives [4][5], but produce static results; any changes in assumptions, cost, or requirements require entirely new analysis, or require separate sensitivity analysis executions. In addition, current

models do not consider social, economical, and environmental factors in addition to the technical. However, in order to be practical for guiding policy and/or decision-makers investment in energy resources, an energy systems modeling tool has to address variability of ecological and socio-economic sensitivities. The aim of this effort is to produce an advisory and design tool geared toward aiding entities with rapid, dynamic planning of effective renewable energy solutions based on trusted models.

This paper introduces the development of a tool that enables tradeoffs between various energy systems, based on neural network surrogate models of a publicly available power systems modeling tool. These surrogate models enable a higher level decision making tool to manipulate real engineering models, as opposed to qualitative information based on expert estimation. This tool presents a decision-maker with the ability to determine which various renewable and non-renewable energy systems meet annual energy load requirements, acquisition and operation costs, and individual solution attributes. This allows the determination of the best energy portfolio purchasing strategy over several years, given a load growth profile, annual budgets, dynamic technical and cost assumptions, and variable importance weightings for technical and non-technical factors. This research is based on advanced design and systems engineering tools and methods developed at the Aerospace Systems Design Laboratory (ASDL) at Georgia Tech. Specific energy system subject matter expertise is drawn from the Georgia Tech Center for Innovative Fuel Cell & Battery Technology and the Georgia Tech Research Institute (GTRI).

To adequately design and plan energy systems with renewable sources requires addressing technical, cost, availability, environmental, and other factors in an integrated analysis approach. The ability to infuse qualitative elements was achieved through the use of various Quality Engineering Methods. First, a Quality Function Deployment (QFD) exercise is conducted to qualitatively map the impacts of engineering degrees of freedom, such as power derived from a particular renewable source, to customer desires, such as

ability to self sustain. An interactive version of the QFD process integrated in the proposed tool enables the engineer to quickly identify the key attributes of an integrated energy system that need to be addressed in a modeling and simulation environment. Next, those attributes are used in a Multi-Attribute Decision Making (MADM) environment to decide which integrated hybrid energy solution best meets the customer requirements. Ideally, this “best-in-class” solution is selected from a thoroughly exhaustive trade space evaluation. What is needed is a way to rapidly explore options based on proven modeling and simulation tools, without having to execute those tools every time an assumption is changed.

B. Case Study: Remote Area Electrification

In many developing countries, especially in rural or remote areas, connectivity to an electricity grid is often rudimentary or non-existent. For many of these areas the only potential method of electricity generation is through stand-alone sources, such as diesel generators. However, the logistics of transporting fuel to far-remote areas means that even this option is impractical and uneconomical for many areas of the world. Therefore, stand-alone energy generation based on renewable sources is appealing; for example, the use of solar photovoltaics is growing rapidly in developing countries for pumping water [6].

The proof-of-concept presented in this paper shows the application of systems engineering tools and methods to an energy systems analysis of a rural or remote area that does not have grid connectivity. Therefore, a problem will be explored in which an investor (or investment group) has a given amount of capital to purchase and operate a given portfolio of energy producing sources over a number of years. A given group of stakeholders also may have a list of requirements for this rural electrification, many of which may be qualitative in nature (i.e. not measurable by standard metrics). The case study will therefore show how the methods introduced in this paper can be used to guide investors and requirements developers as to which portfolio of energy systems to invest in over a given time frame.

C. Hybrid Energy Systems Design Considerations

The integration of a variety of energy generation and storage mechanisms is very complex. As shown in Fig. 1, there are many parameters which may be considered. For example many forms of energy generation units exist, such as wind turbines, solar photovoltaics, and fossil fuel based generators. The location of an energy system that uses energy generation based on natural elements will affect operation and performance. Additionally, given that renewable generation units rely on intermittent sources, such as wind and solar irradiance, a storage device may be considered to store energy generated during periods of low demand and then discharge that energy to supplement energy

produced during times of high demand (for example, storing energy produced by wind turbines at night in batteries and then discharging at mid-day to offset peaks in demand).

Power qualityEfficiencyEmissionsCostAvailabilityPrioritization of multiple load demands

Wind

Solar

Oil

Storage

Design Parameters

Location

• Hybrid power systems

• Energy storage and conversion

Systems Modeling

Wave

Multiple Objectives

Fig. 1. Energy Systems Design Considerations

An energy system will be designed to meet various objectives, the most obvious being availability of power to meet load demand. Capturing the interactions of such a hybrid energy system in light of multiple and competing objectives drives the need for the application of a systems engineering process which adequately captures requirements, contains a functional analysis which breaks requirements down into a set of functions that must be met in order to satisfy those requirements, and finally executes a synthesis routine which yields measurable results which can be compared against requirements [7].

D. Novelty of Approach

The application of methods described in this paper for energy systems modeling moves beyond the notion of individual component design. The approach uses elements from the field of systems-of-systems research, where each system is independently managed and operated. In this line of analysis, the capability of the integrated whole will produce results greater than sum of the individual components. An examination of whether the hybrid energy systems studied in this paper are considered a system or a system-of-systems is beyond the scope of this paper; the notion of systems-of-systems is briefly introduced because the analysis methods used in this study are born from this field of research.

Research methods conducted on capability-focused and inverse design for analysis of complex systems-of-systems [8][9] will be used to identify hybrid energy solutions that meet dynamic requirements. This includes enabling inter-system requirements tradeoff analyses. Surrogate models, which are bounded equation representations of more complex tools that offer negligible loss in fidelity, are created based on trusted modeling and simulation tools. These surrogate models (in this case neural networks) can be executed thousands of times in fractions of a second, enabling on-the-fly trade-offs that yield results that might not otherwise have been discovered with traditional means. Decision-makers are afforded a novel real-time, panoramic view of trade-offs and

parametric sensitivities via advanced visualization features. The result is the ability to conduct qualitative decision-making based on rapid manipulation of quantitative modeling and simulation.

II. ELEMENTS OF DECISION MAKING

This section will introduce the approach and methodology used to create the decision making environment discussed in this paper. These include the method used to identify critical engineering functions based on weighted requirements, through the method used to incorporate modeling and simulation into a higher level decision making environment.

A. Quality Function Deployment

Quality Function Deployment (QFD) is a formal technique for capturing the user’s requirements (voice of the customer) and mapping them to controllable product and process parameters or vehicle attributes (voice of the engineer) [10]. A traditional example of a QFD is shown in Fig. 2. The customer requirements are listed along the vertical column on the left hand side of the QFD, and the engineering attributes are listed across the top row. The impact of each engineering attribute on each requirement is mapped qualitatively on a scale of 0 (no relationship) to 9 (strong relationship). Because these engineering attributes may have adverse impacts on various customer requirements, when used as part of the process introduced in this paper, these qualitative mappings may be positive or negative.

Fig. 2. Quality Function Deployment

Each requirement is assigned an importance weighting by the user, which may be done objectively on an arbitrary scale of 0-10, 0-100, or any similar scale capturing the level of fidelity desired. The importance weighting of each engineering attribute is found by multiplying the requirements weightings vector by the impact vector of that engineering characteristic, given in Equation 1. These attribute weightings are then normalized across all of the engineering attributes as given in Equation 2, which may be used to guide a Multi-Attribute Decision Making process as described in the next section.

Attribute Weighting j = Req Weightingi * Attribute_to_Req Mappingi,j (1) Normalized Attribute Weighting j = Attribute Weighting j / Attribute Weightingi (2)

B. Multi-Attribute Decision Making

Most optimization techniques for design are poorly suited to handle multiple and/or conflicting objectives. The design of complex interacting systems requires holistic solutions that are valid in multiple dimensions, given that requirements can impact multiple design variables and measures of effectiveness may be conflicting. Beginning in the 1950’s and continuing all the way to the 1970’s, the U.S. Department of Defense invested heavily in the development of mathematical techniques for decision making in the presence of many attributes which are valid for a large number of complex system design processes. These are referred to as Multi-Attribute Decision Making (MADM) techniques [11].

The Technique for Ordered Preference by Similarity to Ideal Solution (TOPSIS) is one of many MADM tools available [12]. This uses a weighted series of criteria to identify the best and worst of each criterion and combines them into the theoretical best and worst points. Actual ranking is performed based on maximizing the normalized distance from the theoretical worst and minimizing the distance from the theoretical best. For the process used in this paper, these weighted series of criteria are identified through the QFD. The “points”, or designs evaluated through the TOPSIS process are created through interaction with modeling and simulation, which is introduced in the next section.

C. Surrogate Modeling

The primary enabler of rapid manipulation of complex modeling and simulation is through the use of surrogate models. Surrogate models, based on response surface methodology [13], are equation regression representations of more complex modeling and simulation tools. To create these, a Design of Experiments (DoE) is used to define the cases to execute using an M&S environment that would capture the most information from a design space with

minimal computational expenditure. The surrogate models are then created through regression of the DoE. Because surrogate models are equations, albeit with possibly complex functional form, they can be analyzed almost instantaneously using any standard desktop computer. Once these surrogate models are created, a design space can be explored by rapidly generating thousands of cases, each with negligible (but measurable) loss in fidelity from the original M&S environment. The process used to create the neural network based surrogate models in this study is similar to that described by one of the authors of this paper in a similar application to ballistic missile defense analysis [14].

D. Robust Design: Enabling “Quality”

The quality of a system, or its ability to meet requirements consistently, is jeopardized by uncertainty and risk. The evaluation of a design may not be driven solely by its capability to achieve specific mission requirements or remain within specific product constraints. Rather, a robust design process, or one that leads to a design that is least sensitive to influence of uncontrollable factors, is needed to balance mission capability with other system effectiveness attributes. Zang et al. [15] describe those design problems that have a nondeterministic formulation, including the field of robust design, as uncertainty-based design.

In the context of an energy system that incorporates renewable sources of power production, there is a certain amount of risk created by the intermittency of the energy source. For example wind is not always available to power wind turbines, and when it is available it usually changes velocity with an element of randomness which directly applies an element of randomness and uncertainty to the power output of that wind turbine. Elements of robust design are used in this study to quantify the uncertainty of achieving certain metrics due to varying factors uncontrollable by the designer and/or decision-maker.

III. PARAMETRIC MODELING AND SIMULATION

A. Simulation Environment

The authors have used HOMER as the modeling and simulation backbone behind the decision making tool set developed for this effort. HOMER is a design tool for grid-connected or off-grid power systems developed by and available freely through the U.S. Department of Energy’s National Renewable Energy Laboratory (NREL) [15]. Given a desired energy load profile, climate conditions such as wind patterns and available sunlight, and an array of energy sources, for example diesel generators, wind turbines, photovoltaic arrays, among many other options, HOMER determines the lower-cost energy solution, and provides sensitivities to changes in costs and resources. With its large database of components and performance models, HOMER

significantly simplifies the design process. However, its trade space analysis has a strong reliance on a computationally intensive combinatorial design process. Furthermore, HOMER selects systems based exclusively on the levelized cost of energy of the system, and cannot rank designs based on any other criteria. However, the authors decided that the logic within HOMER could be captured in a form usable by a higher level decision making tool to make decisions based on other technical and non-technical criteria, and to aid in longer-range energy portfolio planning.

A test case was developed in order to determine the feasibility of capturing HOMER’s capabilities in a tool with shorter runtimes and the ability to trade between various energy sources over a multi-year investment timeline. A notional scenario was created, using a sample load profile and sample wind and solar radiation data for a location in central Asia. The system configuration was that of a stand-alone renewable/fossil power system, with scalable/optional components. Components modeled included non-tracking photovoltaic (PV) arrays, a wind turbine with a fixed steady-state wind/power curve, a diesel generator modeled as a steady-state device, and lead-acid batteries.

Sensitivity variables were also controlled, including average solar insolation, average wind speed, the hub height of the wind turbine, the efficiency of the DC to AC inverter, and the required operating reserve. A separate set of runs included economic rather than technical sensitivities.

A Design of Experiments was run in which the sensitivity variables were varied over pre-determined ranges. Because of the nature of the HOMER tool as it was run, the system component sizes could not be varied independently in the DoE. Instead, for every point in the DoE, a full-factorial set of component sizes was evaluated, resulting in almost 73,000 simulation runs. The resulting system performance data was used to generate surrogate models of the system.

B. Parametric Hybrid Energy Systems Models

From the performance data collected from HOMER, single-hidden-layer neural network surrogate models were regressed that represented key performance variables as a function of the five sensitivity and four component size variables. These neural networks are equations that can be executed very quickly, enabling rapid exploration of the design space. They can be used to create dynamic multidimensional plots, to quickly assess the effects of changing constraints or assumptions, or for rapid optimization. The use of surrogate models enables a degree of insight into the problem that is not possible with the simulation code by itself, or with raw data output.

An example of a useful visualization can be seen in Fig. 3. The dark surface represents a boundary between a feasible region (acceptably small capacity shortage) and an infeasible region (unacceptable capacity shortage). A plot is shown with three axes, each axis representing the size of a system component (generator, wind turbine, or PV). In the

background, all sensitivity variables are held constant. From the left-most plot to the right-most plot, the size of the battery bank is gradually increased. By using these three plots (by dynamically adjusting the plot, in practice), an engineer can visualize the trade-off between photovoltaics, wind turbines, diesel generators, and batteries to achieve a desired low value of capacity shortage. It can be seen that with no batteries,

some size of diesel generator is required; but as the size of the battery bank increases, there emerges a trade-off between wind turbines and photovoltaics to enable a generator-free system. These plots were generated using the visualization capabilities of JMP software by SAS [17].

feasible region

infeasible region

feasible region

infeasible region

feasible region

infeasible region

With increased battery capacity, tradeoff region

between PV and wind turbines such

that no need for diesel generator!

Increased battery capacity

Battery capacity insufficient for a

“stand alone”renewable source

system

With even greater battery capacity, tradeoff region

retreats to lower values of PV/wind sources for all RE

systems

Fig. 3. Neural Network Surrogate Models

C. Uncertainty Quantification through Monte Carlo Analysis

Since the use of surrogate models enables modeling and simulation cases to be evaluated very quickly, Monte Carlo investigations comprising hundreds of thousands of runs can be conducted within several seconds on a standard desktop PC. This process enables the uncertainty quantification introduced in Section II D.

An example of a sensitivity study is shown in Fig. 4 and Fig. 5. A given number of variables are treated as noise variables in this study, meaning that operationally the decision maker has no control over their fluctuations. As an example, atmospheric data such as the available solar irradiance and wind, as well as the price of fuel are treated as noise variables. This means that although these variables may be known in a controlled M&S environment, they are not known exactly in an operational environment.

The noise variables of average wind speed, average annual solar irradiance, and fuel price are assigned distributions, and thousands of cases are run through a neural network surrogate model of levelized cost of energy (as shown in Fig. 4). The results can be plotted in a cumulative distribution function (as shown in Fig. 5), allowing the engineer to quickly gauge, for example, a 90% confidence upper bound on energy cost. Such methods enable the engineer to find robust solutions which offer high likelihood of success.

Fig. 4. Uncertainly Distributions on Noise Variables

Cost of Energy ($/kWhr)

Pro

babi

lity

90% Probability of Achieving

Cost Goal

Fig. 5. Monte Carlo Output

IV. PORTFOLIO MANAGEMENT

A. Linking Qualitative Requirements to Quantitative Analysis

Decision-makers may have higher-level requirements or objectives that are not directly represented by the technical performance characteristics that are produced by the simulation tool and its surrogates. If this is the case, the engineering outputs must be mapped to the higher-level objectives of the decision-maker.

This can be achieved through the use of QFD. As previously described, QFD uses a matrix of qualitative impact weightings to map technical attributes to top-level objectives. Since the impact weightings are subjective, the process of assigning them should be a collaborative exercise undertaken by experts in the field. A notional QFD used for this exercise is shown in Table 1.

Table 1. QFD and MADM Values

Cap

acity

Sh

orta

ge

Ren

ewab

le

Frac

tion

Die

sel F

uel

Use

d (L

)

Prod

uctio

n w

ind

Prod

uctio

n So

lar

Prod

uctio

n ge

nera

tor

Batte

ry

Thro

ughp

ut

Ease of Integration 5 0 0 -1 -9 -3 -1 -5Reliability of Equipment 5 0 0 -1 -3 -1 -5 -5Availability of Power 8 -9 0 0 0 0 0 0Technology Maturity 5 0 0 0 3 2 9 2Energy Independence 2 -9 9 -9 9 9 -9 5Environmentally Friendly 2 0 9 -9 9 9 -9 2

TOPSIS Weighted score 0.354 0.142 0.181 0.035 0.102 0.083 0.102

Quantitative M&S Metrics

Requirements Cus

tom

er

Wei

ghtin

gs

The QFD process results in relative importance weightings

assigned to each engineering characteristic, which can then be used to rank different possible system configurations using a MADM technique such as TOPSIS, as described previously. For any combination of importance weightings, and given a discrete set of alternative system designs, a “best” system will emerge; the inter-system tradeoffs are inherent to the QFD/MADM process.

Since the system is represented with fast-executing surrogates, it is easy to see how changes in system sensitivity variables (such as mean annual insolation) affect the choice of “best” system. Alternately, it is possible to select systems that consistently rank high, under a range of sensitivity conditions.

B. Interactive Tool

The interactive tool developed through this research is shown for two use case scenarios in Fig. 6 and Fig. 7 below. Note that high level requirements (such as ease of integration, energy independence, etc.) are shown with slide bars which control the importance weightings. These are the importance weightings which are translated through the QFD to drive the importance of the various simulation specific attributes (for example capacity shortage, diesel fuel used, etc.). The user

has the ability to control the desired load demand over time, as well as to limit the amount of investment dollars over time. Assumptions such as average insolation and average wind speed may be adjusted, as well as changes in equipment purchase and maintenance costs over the life of the project (not shown).

The user may change any of the inputs and re-evaluate, and in a few seconds the system will assess thousands of portfolio options through the use of surrogate models, decide which options best meet the weighted requirements through the MADM process, and select annual equipment purchases for the life of the project.

The rapid execution time of the tool, combined with the ease of adjusting requirements, budgets, and assumptions, allows a decision-maker to answer a multitude of questions without having to execute the original M&S through a number of different cases. This eliminates the lag created when the decision-maker redirects the engineering analyst. The amount of visual information available helps the decision-maker better understand the nature of the problem, and the use of adjustable non-technical requirements allows the decision-maker to treat these often un-quantified (yet still important) factors in a more formal and considered manner.

C. Case Study Findings for Notional Scenario

An example of use is shown in Fig. 6 (Scenario A). The user has specified a load growth profile, starting at 20 kW average in 2007 and growing to 46 kW average by 2011. The user has also specified a capital investment budget, ranging from $50,000/yr in 2007 to $230,000/yr in 2008. Most requirements are given equal weighting, but "energy independence" and "environmental friendliness" are set to zero, that is they do not factor into the decision. Under these conditions, the tool decides to purchase diesel generators, and thereafter to improve power quality with battery purchases. However, the fuel budget is large (note that in this implementation, operations costs are separate from the purchase budget).

In Fig. 7 (Scenario B), the requirements have been altered so that that "energy independence" and "environmental friendliness" are rated equally with other factors. All other assumptions and settings are kept the same. When the user selects these “greener” requirements weightings, those weightings are instantaneously sent through the QFD which reprioritizes the engineering characteristics, and in turn selects a new “best in class” through the MADM process evaluation of the surrogate model results. For this Scenario B shown in Fig. 7 with the “green” requirements weightings, the tool still purchases diesel generators early on, but once it has allowable budget it begins to purchase photovoltaics, batteries, and wind turbines. Noting the bottom-right plot in Fig. 7, after the first year, the diesel fuel consumption drops to almost nothing.

Fig. 6: Screenshot of Energy Systems Portfolio Tool (Scenario A)

Fig. 7: Screenshot of Energy Systems Portfolio Tool (Scenario B)

V. CONCLUSIONS

This study introduced an interactive tool for energy systems portfolio planning developed through a systems engineering process that enables real-time decision making through integration with rapid modeling and simulation. A structured process that combines elements of Quality Function Deployment, Multi-Attribute Decision Making, and surrogate modeling together enable qualitative decision making based on quantitative modeling and simulation based tools. An advisory and design tool was introduced that aids decision-makers with robust planning and implementation of effective renewable energy solutions.

A decision-maker can use the methodology and tool presented in this paper to determine which various renewable and non-renewable energy portfolio options can meet annual energy load requirements, acquisition and operation costs, and individual solution attributes. The aim of the tool development was to bridge the gap which currently affects existing models of various energy assets which do not consider social, economical, and environmental factors in addition to the technical. The QFD process was used to capture those elements that are qualitative in nature, and to drive those elements that are quantitative. Through the direct integration of the QFD within the interactive tool, an example was given how a user may change importance weightings on requirements, in addition to constraining other programmatic issues such as annual budget, and very quickly analyze thousands of portfolio options through the use of surrogate models and select the “best in class” through the MADM process.

ACKNOWLEDGEMENT

The authors wish to thank Mr. Daniel Brady for developing the initial portfolio management framework and to Dr. Tom Fuller of the Georgia Tech Research Institute for guidance and direction. This effort was funded by the Georgia Tech Research Institute.

REFERENCES

[1] Beyer, H.G., Langer, C. “A method for the identification of configurations of PV/wind hybrid systems for the reliable supply of small loads.” Solar Energy, vol. 57, No. 5, pp. 381-391, 1996.

[2] Habib, M.A., Said, S.A.M., El-Hadidy, M.A., Al-Zaharna, I. “Optimization procedure of a hybrid photovoltaic wind energy system.” Energy, vol. 24, pp. 919-929, 1999.

[3] Celik, A.N. “Techno-economic analysis of autonomous PV-wind hybrid energy systems using different sizing methods.” Energy Conversion and Management, vol. 44, pp. 1951-1968, 2003.

[4] Shi, J., Zhu, X., Cao, G. “Design and techno-economical optimization for stand-alone hybrid power systems with multi-objective evolutionary algorithms.” International Journal of Energy Research. 2007; 31:315-328.

[5] Anagnostopoulos, J.S., Papantonis, D.E. “Simulation and size optimization of a pumped-storage power plant for the recovery of wind-farms rejected energy.” Renewable Energy, 33 (2008) 1685-1694.

[6] Boyle, editor. Renewable Energy: Power for a Sustainable Future. Oxford: Oxford University Press, 1996.

[7] Systems Engineering Fundamentals, Defense Acquisition University, Defense Acquisition University Press. January 2001

[8] Ender, T.R., “A top-down, hierarchical, system-of-systems approach to the design of an air defense weapon”. PhD thesis, Georgia Institute of Technology, 2006.

[9] Biltgen, P. T., Ender, T. R., and Mavris, D. N., “Development of a collaborative capability-based tradeoff environment for complex system architectures,” presented at the 44th AIAA Aerospace Sciences Meeting and Exhibit, Reno, Nevada, Jan. 9-12, 2006. AIAA-2006-0728.

[10] Akao, Y.: “Development history of Quality Function Deployment”; In: Shigero Mizuno, Yoji Akao (Editors): “QFD, the customer-driven approach to quality planning and development”. Tokyo, 1994, pp. 339-351

[11] Yoon, K., Hwang, C.-R., Multiple Attribute Decision Making: An introduction. Iowa City: Sage Publications Inc., 1995.

[12] Hwang, C.-R., Yoon, K., Lecture notes in economics and mathematical systems, Springer-Verlag Berlin Heidelberg, New York, 1981.

[13] Myers, R. H. and Montgomery, D. C., Response Surface Methodology: Process and Product Optimization Using Designed Experiments. New York: John Wiley & Sons, Inc, 1995.

[14] Ender, T., Leurck, R., Weaver, B., Miceli, P., Blair, W.D., West, P., Mavris, D., “Systems-of-systems analysis of ballistic missile defense architecture effectiveness through surrogate modeling and simulation”, in Proc. of the 2nd IEEE International Systems Conference, Montreal, Canada, April 2008, pp 333-340.

[15] Zang, T. A., Hemsch, M. J., Hilburger, M. W., Kenny, S. P., Luckring, J., Maghami, P., Padula, S. L., and Stroud, W. J., “Needs and opportunities for uncertainty-based multidisciplinary design methods for aerospace vehicles,” Tech. Rep. TM-2002-211462, NASA Langley Research Center, Hampton, VA, 2002.

[16] National Renewable Energy Laboratory, U.S. Department of Energy, “HOMER – Analysis of Micropower System Options”. www.nrel.gov/homer

[17] SAS Institute. JMP 7.0. Software. 2007.