SIAG/OPT Views-and-NewsA Forum for the SIAM A tivity Group on OptimizationVolume 11 Number 2 August 2000ContentsCase StudiesOptimization of Engineering Systems Governed byDi�erential Equations: Editor's Introdu tionN. M. Alexandrov . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1Redu ed-Basis Output-Bound Methods for Ellip-ti Partial Di�erential EquationsA. T. Patera, D. V. Rovas and L. Ma hiels . . . . . . 4The Adjoint Approa h in a NutshellR. M. Lewis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9A Lagrange-Newton-Krylov-S hur Method for PDE-Constrained OptimizationG. Biros and O. Ghattas . . . . . . . . . . . . . . . . . . . . . . . 12Bulletin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18Comments from the Chair and EditorThomas Coleman and Juan Meza . . . . . . . . . . . . . . 18Case StudiesOptimization of EngineeringSystems Governed by Di�erentialEquations: Editor's Introdu tionNatalia AlexandrovMultidis iplinary Optimization Bran hAerospa e Systems Con epts & AnalysisNASA Langley Resear h Center, Hampton, Virginia23681-2199, USA n.alexandrov�lar .nasa.govMany engineering optimization problems fall intoone of three ategories: optimal design, parameter

estimation, and optimal ontrol. Their ommon fea-ture is that di�erential equations appear in the prob-lem formulation as onditions that must be satis-�ed for a physi al system to be realizable or on-stru tible. This issue of V&N is devoted to someaspe ts of problem solving in engineering optimiza-tion. A brief dis ussion of the multidis iplinary op-timization (or optimal design) problem will serve tointrodu e some terminology to the readers unfamil-iar with this area and to pla e the subje ts of theinvited arti les in the ontext of several a tive re-sear h dire tions.Multidis iplinary optimization (MDO) may beviewed as a olle tion of systemati approa hes tothe design optimization of omplex, oupled engi-neering systems (e.g., [1℄), where \multidis iplinary"refers to di�erent aspe ts of a design problem. Forinstan e, the design of aerospa e vehi les involvesaerodynami s, stru tural analysis, propulsion, and ontrol, among many other dis iplines.Be ause the total design problem is an amor-phous reature, not easily quanti�ed or even de-�ned (for reasons both so iologi al and te hni al),resear hers who develop methods for optimal design,�nd it onvenient to abstra t the MDO problem asa subset of the total design problem that an be for-mulated as the following (here simpli�ed) nonlinearprogram: minimizex f(x; u(x))subje t to h(x; u(x)) = 0g(x; u(x)) � 0; (1)where x is the ve tor of design variables and u(x) isde�ned via a blo k system of equations,A(x; u(x)) = 0B� A1(x; u1(x); : : : ; uN (x))...AN (x; u1(x); : : : ; uN (x)); 1CA = 0(2)

2 SIAG/OPT Views-and-Newsfor N blo ks. The blo ks represent the state equa-tions of the dis iplinary analyses (known as simu-lations if they are a urate and expensive) and theinterdis iplinary ouplings. The state equations nor-mally form a set of oupled di�erential equationsthat des ribe the behavior of the dis iplinary sub-systems. Given the design variables x, a dis iplinaryanalysis Ai((x; u1(x); : : : ; uN (x)) = 0 solves for thestate variables or dis iplinary responses ui(x). Solv-ing (2) leads to a full multidis iplinary analysis, orMDA, in whi h the oupled dis iplines give a phys-i ally onsistent|and thus meaningful|result. Forinstan e, in aerospa e MDO, a two-dis ipline prob-lem might represent the aeroelasti intera tion be-tween aerodynami s (A1) and stru tural analysis(A2) for a wing in steady-state ow. Then u1 andu2 may represent the ow �eld near the wing andthe deformed shape of the wing due to stru turalresponse and aerodynami loads, respe tively. Com-puting the ow �eld u1 requires the shape of the wing ontained in u2, while omputing the wing deforma-tion u2 requires the aerodynami loads, ontained inu1. Traditionally, at ea h iteration of optimization,the design variable ve tor x is passed to the MDAsystem. The oupled PDEs are then solved for thestate ve tor u, thus redu ing the dimension of (1) bymaking it a problem in x only. Optimizers will re og-nize this method as a variable redu tion or redu ed-spa e approa h.Among numerous diÆ ulties asso iated withsolving (1), the most pronoun ed one has to do withthe frequently prohibitive expense of repeated eval-uation of fun tions and derivatives via simulations.Several urrent interrelated resear h dire tions aimto over ome this diÆ ulty.The expense of implementing and exe uting astraightforward, onventional optimization approa hto (1) has|for a long time|motivated resear hersto propose alternative optimal design problem for-mulations and attendant optimization algorithms.Histori al notes and re ent e�orts in analysis anddevelopment of problem formulations for MDO anbe found in [2, 3, 4℄ and referen es therein.Simultaneous Analysis and Design (SAD orSAND) (e.g., [5, 6℄) is an example of an alterna-tive problem formulation. It is motivated by theexperien e whi h suggests that allowing infeasibility

with respe t to the MDA manifold far from solu-tions may redu e the ost of optimization. SANDtreats both the design variables and the state vari-ables as expli it optimization variables, while (2) istreated as expli it equality onstraints. The arti- le by Omar Ghattas and George Biros in this issuedis usses promising re ent advan es in using opti-mization methods for solving redu ed-spa e and full-spa e (SAND) formulations.A�ordable omputation of derivatives for any en-gineering design problem is an important ompo-nent of an optimization pro edure. The \adjointapproa h" is frequently used for omputing deriva-tives in problems governed by PDE. Mi hael Lewisexplains the adjoint approa h in this issue for read-ers unfamiliar with the te hnique. Interesting re entresults in omputing derivatives for optimal designproblems an also be found in, e.g., [7, 8℄ and refer-en es therein.Another general resear h dire tion addresses theexpense of using simulations in omputational op-timization by developing adaptive modeling te h-niques and optimization strategies for a systemati use of variable-�delity models.Models and approximations of varying degree of�delity have been used in engineering for a long time(e.g., see [9℄ for a review in the area of stru turaloptimization), with optimization pro edures largelybased on heuristi s. Due to improvements in the nu-meri al modeling te hniques and the in reased avail-ability of high-�delity analyses, optimization withvariable-�delity models and approximations has be- ome a subje t of mu h interest in the past few years(e.g., [10, 11, 12℄).A variety of interesting modeling te hniques|su h as redu ed-order models [13, 14℄ and spa e map-ping methods [15℄|have been under study in a num-ber of engineering areas. In this issue, Tony Pateraand o-authors des ribe some of the ex iting re entdevelopments in redu ed-order modeling at MIT.Variable-�delity models must be managed in op-timization pro edures in ways that minimize the ex-pense, yet guarantee onvergen e to high-�delity an-swers. Re ent model management methods in ludete hniques that use sensitivities (e.g., [16, 17℄) andte hniques that do not (e.g., [18℄). The latest resultsin model management with variable-resolution mod-els [19℄ and variable physi al �delity models [20℄ are

Volume 11 Number 2 August 2000 3promising.Optimization of engineering systems involvesmany other areas of resear h, su h as omputer s i-en e (for omputational frameworks and problem in-tegration), and other areas of optimization, su h asoptimization under un ertainty. Interested readersmay �nd papers from a variety of ontributing �eldsin [21℄. Last-minute news in some of the areas de-s ribed in this issue may be heard at the minisym-posia titled \Optimization of Engineering SystemsGoverned by Di�erential Equations", Parts I andII, at the First SIAM Conferen e on ComputationalS ien e and Engineering, Sept. 21{24, WashingtonDC. Finally, all optimizers are invited to ontributeto the new journal \Optimization and Engineering"published by Kluwer1.REFERENCES[1℄ J. Sobiesz zanski-Sobieski and R. T. Haftka. Multi-dis iplinary aerospa e design optimization: Surveyof re ent developments. Stru tural Optimization,14(1):1{23, August 1997.[2℄ N. M. Alexandrov and R. Mi hael Lewis. Analyti- al and omputational aspe ts of ollaborative op-timization. National Aeronauti s and Spa e Admin-istration, NASA/TM{210104{2000, April 2000.[3℄ |. Analyti al and omputational properties of dis-tributed approa hes to MDO. AIAA-2000-4718,2000.[4℄ |. Algorithmi perspe tives on problem formulationsin MDO. AIAA-2000-4719, 2000.[5℄ R. T. Haftka, Z. G�urdal, and M. P. Kamat. Elements ofstru tural optimization. Kluwer A ademi Publish-ers, Dordre ht, 1990.[6℄ R. M. Lewis. Pra ti al aspe ts of variable redu tion for-mulations and redu ed basis algorithms in multidis- iplinary design optimization. In N.M. Alexandrovand M. Y. Hussaini, editors, Multidis iplinary De-sign Optimization: State-of-the-Art, Philadelphia,1997. SIAM.[7℄ A. A. Giunta. Sensitivity analysis for oupled aero-stru tural systems. NASA-TM-1999-209367, Au-gust 1999.[8℄ J. Borggaard and J. A. Burns. Asymptoti ally onsis-tent gradients in optimal design. In N. M. Alexan-drov and M. Y. Hussaini, editors, Multidis iplinary

Design Optimization: State of the Art, pages 303{314. SIAM, 1997.[9℄ J.-F. M. Barthelemy and R. T. Haftka. Approximation on epts for optimum stru tural design|a review.Stru tural Optimization, 5:129{144, 1993.[10℄ S. Burgee, A.A. Giunta, V. Balabanov, B. Gross-man, W.H. Mason, R. Nardu i, R.T. Haftka,and L.T. Watson. A oarse-grained parallelvariable- omplexity multidis iplinary optimizationproblem. The International Journal of Super om-puting Appli ations and High Performan e Comput-ing, 10(4):269{299, Winter 1996.[11℄ A.A. Giunta. Air raft Multidis iplinary OptimizationUsing Design of Experiments Theory and ResponseSurfa e Modeling Methods. PhD thesis, Departmentof Aerospa e and O ean Engineering, Virginia Te h,Bla ksburg, VA, 1997.[12℄ J. C. Otto, M. Paras hivoiu, S. Ye�silyurt, and A. T. Pat-era. Bayesian-validated omputer-simulation surro-gates for optimization and design. In N. M. Alexan-drov and M. Y. Hussaini, editors, Multidis iplinaryDesign Optimization: State of the Art, pages 368{390. SIAM, 1997.[13℄ M. C. Romanowski. Redu ed order unsteady aerody-nami and aeroelasti models using Karhunen-Lo�eveeigenmodes. AIAA-96-3981-CP, 1996.[14℄ W. A. Silva. Identi� ation of linear and nonlinear aero-dynami impulse responses using digital �lter te h-niques. NASA-TM-112872, August 1997.[15℄ M. H. Bakr, J. W. Bandler, N. Georgieva, and K. Mad-sen. A hybrid aggressive spa e mapping algorithmfor em optimization. IEEE Trans. Mi hrowave The-ory Te h, 47, 1999.[16℄ N. M. Alexandrov. A trust region framework for manag-ing approximation models in engineering optimiza-tion. AIAA-96-4102, September 1996.[17℄ R. M. Lewis. A trust region framework for managingapproximation models in engineering optimization.AIAA Paper 96-4101, September 1996.[18℄ A. J. Booker, J. E. Dennis, Jr., P. D. Frank, D. B. Ser-a�ni, V. Tor zon, and M. W. Trosset. A rigorousframework for optimization of expensive fun tionsby surrogates. Stru tural Optimization, 17(1):1{13,1999.[19℄ N.M. Alexandrov, R.M. Lewis, C.R. Gumbert, L.L.Green, and P.A. Newman. Optimization withvariable-�delity models applied to wing design.AIAA-2000-0841, January 2000.1Journal information may be be found at http://fmad-www.lar .nasa.gov/mdob/users/natalia/opte/ (US site) or athttp://ssor.twi.tudelft.nl/ terlaky/OPTE/opte.html (European site).

4 SIAG/OPT Views-and-News[20℄ N. M. Alexandrov, R. M. Lewis, and E. J. Nielsen. Aero-dynami optimization with variable-�delity models.AIAA-2000-4886, September 2000.[21℄ N. M. Alexandrov and M. Y. Hussaini, editors. Multi-dis iplinary Design Optimization: State of the Art,Philadelphia, 1997. So iety for Industrial and Ap-plied Mathemati s.Redu ed-Basis Output-BoundMethods for Ellipti PartialDi�erential Equations1Anthony T. PateraDepartment of Me hani al Engineering, Room 3-264,Massa husetts Institute of Te hnology, Cambridge, MA,02139-4307, patera�mit.eduDimitrios V. RovasDepartment of Me hani al Engineering, Room 3-243,Massa husetts Institute of Te hnology, Cambridge, MA,02139-4307, rovas�mit.eduLu Ma hielsLawren e Livermore National Laboratory, L-561, 7000 EastAvenue, Livermore, CA 94550, ma hiels1�llnl.gov1. MotivationTo motivate and illustrate our methods we onsidera spe i� example: a thermal �n. The �n, shownin Figure 1, onsists of a entral \post" and four\sub�ns;" the �n ondu ts heat from a pres ribeduniform ux \sour e" at the root, �root, throughthe large-surfa e-area sub�ns to surrounding ow-ing air. The �n is hara terized by seven designparameters, or \inputs," � 2 D � IRP=7, where�i = ki; i = 1; : : : ; 4; �5 = Bi; �6 = L; and �7 = t.Here ki is the thermal ondu tivity of the ith sub-�n (normalized relative to the post ondu tivity); Biis the Biot number, a nondimensional heat transfer oeÆ ient re e ting onve tive transport to the airat the �n surfa es; and L and t are the length andthi kness of the sub�ns (normalized relative to thepost width). The performan e metri , or \output,"s 2 IR, is hosen to be the average temperature ofthe �n root normalized by the pres ribed heat ux

into the �n root. In order to optimize the �n de-sign, we must be able to evaluate s(�) repeatedlyand rapidly.Figure 1We an express our input-output relationship ass = `O(u(�)), where `O(v) is a ( ontinuous) lin-ear fun tional | `O(v) = R�root v | and u(�) isthe temperature distribution within the �n. (Thetemperature �eld is of ourse a fun tion of the spa-tial oordinate, x; we expli itly indi ate this depen-den e only as needed.) The temperature distribu-tion u(�) 2 Y satis�es the ellipti partial di�erentialequation des ribing heat ondu tion in the �n,a(u; v;�) = `(v);8v 2 Y ; (1)a(u; v;�) is the weak form of the Lapla ian, and`(v) re e ts the pres ribed heat ux at the root.Here Y is the appropriate Hilbert spa e with asso i-ated inner produ t (�; �)Y and indu ed norm k �kY 2.The bilinear form a(�; �;�) is symmetri , a(w; v;�) =a(v; w;�);8w; v 2 Y 2;8� 2 D; uniformly ontinu-ous, ja(w; v;�)j � kwkY kvkY ;8w; v 2 Y 2;8� 2 D;and oer ive, �kvk2Y � a(v; v;�);8v 2 Y;8� 2 D.Here � and are stri tly positive real onstants. Fi-nally, the form `(v) is a linear bounded fun tional;for our hoi e of s aling and output, `O(v) = `(v),whi h we will exploit to simplify the exposition.It an further be shown for our problem that a an be expressed asa(w; v;�) = QXq=1�q(�)aq(w; v);8w; v 2 Y 2;8� 2 D;(2)1The material presented in this arti le is an expository version of work performed in ollaboration with Professor Yvon Ma-day of University of Paris VI and reported in greater detail in referen es [1, 2, 3, 4℄. We also thank Professor Jaime Peraireof MIT, Professor Einar R�nquist of Norwegian University of S ien e and Te hnology, Mr. Roland Von Kaenel of EPFL,and Ms. Shidrati Ali of National University of Singapore{Singapore-MIT Allian e (SMA) for helpful omments. The work issupported by AFOSR, NASA Langley Resear h Center, and SMA.2Here Y = H1(), the spa e of fun tions that are square integrable and that have square integrable �rst (distributional)derivatives over the �n referen e domain . The inner produ t (w; v)Y may be hosen to be Rrw � rv + wv.

Volume 11 Number 2 August 2000 5for appropriately hosen fun tions �q:D ! IR andasso iated �-independent bilinear forms aq:Y �Y !IR, q = 1; : : : ; Q. Note that we pose our problem ona �xed �n referen e domain in order to ensure thatthe parametri dependen e on geometry | L and t| enters through a(�; �;�) and ultimately the �q(�).For our parti ular problem, Q = 15; if we freeze (�x)all parameters ex ept L and t (su h that Pe� = 2),Q = 8; if we freeze only L and t (su h that Pe� = 5),Q = 6.In the ontext of design, optimization, and on-trol, we require very rapid response and many out-put evaluations. Our goal is thus to onstru t anapproximation to u(�), ~u(�), and hen e approxima-tion to s(�), ~s(�) = `O(~u(�)), whi h is (i) erti�ablya urate, and (ii) very eÆ ient in the limit of manyevaluations. By the former we mean that the errorin our approximate output, js(�) � ~s(�)j, is guar-anteed to be less than a pres ribed toleran e "; bythe latter we mean that, following an initial �xedinvestment, the additional in remental ost to eval-uate ~s(�) for any new � 2 D is mu h less than thee�ort required to dire tly ompute s(�) = `O(u(�))by (say) standard �nite element approximation.2. Redu ed-Basis ApproximationRedu ed-basis methods (e.g., [5, 6, 7℄) area \parameter-spa e" version of weighted-residual(here Galerkin) approximation. To de�ne ourredu ed-basis pro edure, we �rst introdu e a sam-ple set in parameter spa e, SN = f�1; : : : ; �Ng, andasso iated redu ed-basis spa e WN = spanf�n �u(�n); n = 1; : : : ; Ng; where u(�n) satis�es (1) for� = �n 2 D (note �i refers to the ith omponent ofthe P{tuple �, whereas �n refers to the nth P{tuplein SN ). We then require our redu ed-basis approxi-mation to u(�) for any given �, uN (�) 2 WN � Y ,to satisfy a(uN (�); v;�) = `(v);8v 2WN ; (3)the redu ed-basis approximation to s(�) an subse-quently be evaluated as sN (�) = `O(uN (�)).It is a simple matter to show thatku(�)�uN (�)kY � r � minwN2WN ku(�)�wNkY ; (4)

whi h states that our approximation is optimal inthe Y norm. It an also be readily shown for ourparti ular problem thats(�) = sN (�) + a(eN (�); eN (�);�); (5)where eN = u� uN . It follows from (4),(5), and the ontinuity of a thatjs(�)� sN (�)j � 2� ( minwN2WN ku(�)� wNkY )2; (6)thus our output approximation is also optimal.We must, of ourse, also understand the extentto whi h the best wN in WN an, indeed, approx-imate the requisite temperature distribution. Theessential point is that, although WN learly doesnot have any approximation properties for generalfun tions in Y , simple interpolation arguments inparameter spa e suggest that WN should approx-imate well u(�) even for very modest N ; indeed,exponential onvergen e is obtained in N for suf-� iently smooth �-dependen e (e.g., [6, 7℄). It is forthis reason that, even in high-dimensional (large P )parameter spa es, redu ed-basis methods ontinueto perform well | indeed, thanks to (6), mu h bet-ter than ad ho , un ontrolled \non-state-spa e" �tsof (�; s(�)) input-output pairs.We now turn to the omputational issues. We�rst express the redu ed-basis approximation asuN (x;�) = NXj=1uNj (�)�j(x) = (uN (�))T �(x); (7)and hoose for test fun tions v = �i(x); i = 1; : : : ; N .We then insert these representations into (3) to yieldthe desired algebrai equations for uN (�) 2 IRN ,NXj=1 a(�j; �i;�)uNj = `(�i); i = 1; : : : ; N: (8)Equation (8) an be written in matrix form asA(�)uN (�) = L; (9)where A(�) 2 IRN�N is the SPD matrix with entriesAi;j(�) = a(�j ; �i;�); 1 � i; j � N , and L 2 IRN isthe \load" ve tor with entries Li = `(�i); 1 � i � N .

6 SIAG/OPT Views-and-NewsWe now invoke (2) to note thatAi;j(�) = a(�j ; �i;�) = QXq=1�q(�)aq(�j ; �i)= QXq=1 �q(�)Aqi;j ; (10)where the matri es Aq 2 IRN�N are given by Aqi;j =aq(�j ; �i); 1 � i; j � N; q = 1; : : : ; Q. The o�-line/on-line de omposition is now lear. In the o�-line stage, we onstru t the Aq; q = 1; : : : ; Q. In theon-line stage, for any given �, we �rst form A fromthe Aq a ording to (10); we next invert (9) to �nduN (�); and we then ompute sN (�) = `O(uN (�)) =`(uN (�)) = (uN (�))TL. As we shall see, N will typ-i ally be O(10) for our parti ular problem. Thus,as required, the in remental ost to evaluate sN (�)for any given new � is very small: O(N2Q) to formA(�); O(N3) to invert (the typi ally dense) A(�)system; and O(N) to evaluate sN(�) from uN (�).The above a priori results tell us only that weare doing as well as possible; it does not tell us howwell we are doing. Sin e the error in our output isnot known, the minimal number of basis fun tionsrequired to satisfy the desired error toleran e annot be as ertained. As a result, either too manyor too few fun tions are retained; the former resultsin omputational ineÆ ien y, the latter in una ept-able un ertainty. We thus need a posteriori errorbounds as well.3. Output BoundsTo begin, we assume that we may �nd a fun tiong(�):D ! IR+ and a symmetri ontinuous oer ivebilinear form a:Y � Y ! IR su h that kvk2Y � g(�)a(v; v) � a(v; v;�);8v 2 Y;8� 2 D;(11)for some real positive onstant ; for our thermal �nproblem we an readily �nd a g(�) and a(w; v) su hthat (11) is satis�ed. The pro edure is then simple:we �rst ompute e(�) 2 Y solution ofg(�)a(e(�); v) = R(v;�);8v 2 Y; (12)where R(v;�) � `(v)�a(uN ; v;�) is the residual; wethen evaluate our bounds assN� (�) = sN (�); sN+ (�) = sN (�) + �N (�); (13)

where �N (�), the bound gap, is given by�N (�) = g(�)a(e(�); e(�)): (14)The notion of output bounds is not restri ted toredu ed-basis approximations: it an also be appliedwithin the ontext of �nite element dis retization aswell as iterative solution strategies [8, 9℄.We an then show thatsN� (�) � s(�) � sN+ (�); 8N ; (15)we thus have a erti� ate of �delity for sN | it iswithin �N (�) of s(�). To prove the left inequality weappeal to (5) and the oer ivity of a. To demonstratethe right inequality we �rst note that R(eN (�);�) =`(eN (�))� a(uN (�); eN (�);�) = a(eN (�); eN (�);�),sin e `(eN (�)) = a(u; eN (�);�) from (1) for v =eN (�); we next hoose v = eN (�) in (12) to obtaing(�)a(e(�); eN (�)) = a(eN (�); eN (�);�); then fromthe right inequality of (11) we have�N (�) � g(�)a(e; e)= g(�)a(e� eN ; e� eN ) + 2a(eN ; eN )� g(�)a(eN ; eN )� g(�)a(e� eN ; e� eN ) + a(eN ; eN );from the left inequality of (11) we thus on lude that�N (�) � a(eN ; eN ); a omparison of (5) and (13)then ompletes the proof.We an now as ertain, through �N , the a u-ra y of our output predi tion, whi h will in turnpermit us to adaptively modify our approximationuntil the pres ribed error toleran e " is satis�ed.However, it is also riti al that �N (�) be a gooderror estimator; a poor estimator will en ourage usto unne essarily re�ne an approximation whi h is,in fa t, adequate. To prevent the latter the e�e tiv-ity �N (�) � �N (�)=js(�) � sN (�)j should be orderunity. For our problem it is simple to prove that�N (�) � = , independent of � and N ; in pra ti e,e�e tivities are typi ally less than 10, whi h is ade-quate given the rapid onvergen e of redu ed-basisapproximations.

Volume 11 Number 2 August 2000 7We now turn to the omputational issues. From(2) and (7), (12) an be re-written asa(e(�); v) =1g(�) `(v)� QXq=1 NXj=1�q(�)uNj (�)aq(�j ; v)!;8v 2 Y:We thus see from simple linear superposition thate(�) an be expressed ase(�) = 1g(�) (z0 + QXq=1 NXj=1�q(�)uNj (�)zqj );where z0 2 Y satis�es a(z0; v) = `(v);8v 2 Y;and zqj 2 Y; j = 1; : : : ; N; q = 1; : : : ; Q, satis�esa(zqj ; v) = �aq(�j; v);8v 2 Y: It then follows thatwe an express �N (�) of (14) as�N (�) = 1g(�)" a(z0; z0)| {z } 0 +2 QXq=1 NXj=1�q(�)uNj (�) a(z0; zqj )| {z }�qj +QXq=1 QXq0=1 NXj=1 NXj0=1�q(�)�q0(�)uNj (�)uNj0 (�) a(zqj ; zq0j0 )| {z }�qq0jj0 #;(16)sN+ (�) then dire tly follows from (13).The o�-line/on-line de omposition is now lear.In the o�-line stage we ompute z0 and zqj ; j =1; : : : ; N; q = 1; : : : ; Q, and then the inner produ ts 0;�qj , and �qq0jj0 de�ned in (16). In the on-line stage,for any given new �, and given sN (�) and uN (�) as omputed in the on-line stage of the output predi -tion pro ess (Se tion 2), we evaluate �N (�) as�N (�) = 1g(�)" 0 + 2 QXq=1 NXj=1�q(�)uNj (�)�qj +QXq=1 QXq0=1 NXj=1 NXj0=1�q(�)�q0(�)uNj (�)uNj0 (�)�qq0jj0#;and then evaluate sN+ (�) = sN (�) +�N (�). The in- remental ost to evaluate sN+ (�) for any given new� is very small: O(N2Q2).

4. Numeri al AlgorithmIn the simplest ase we take our �eld and output ap-proximations to be ~u(�) = uN (�) and ~s(�) = sN (�),respe tively, for some given N , and then ompute�N (�) to assess the error. However, we an im-prove upon this re ipe: we take ~u(�) = u ~N (�)and ~s(�) = s ~N (�), where u ~N (�) and s ~N (�) are theredu ed-basis approximations asso iated with a sub-spa e of WN , W ~N , in whi h we sele t only ~N ofour available basis fun tions. In pra ti e, we in ludein W ~N the basis fun tions orresponding to samplepoints �n losest to the new � of interest; we on-tinue to (say) double our spa e until � ~N (�) � "(and hen e js(�) � s ~N (�)j � "). If we satisfy our riterion for ~N � N the adaptive pro edure is en-tirely ontained within the on-line stage of the pro- edure, and the omplexity of this stage is redu edfrom O(N2Q+N3+N2Q2) to O( ~N2Q+ ~N3+ ~N2Q2).Note the riti al role that our error bound plays ine�e ting this e onomy.In pra ti e | to ensure that the �n; z0; zqjare a tually al ulable | we repla e the in�nite-dimensional spa e Y with a very high-dimensional\truth" spa e YT (e.g., a �nite element spa e as-so iated with a very �ne triangulation). It followsthat we obtain bounds not for s, but rather forsT = `O(uT ), where uT 2 YT satis�es a(uT ; v;�) =`(v);8v 2 YT . The essential point is that YT maybe hosen very onservatively | and hen e the dif-feren e between sT and s rendered arbitrarily small| sin e (i) the on-line work and storage are in fa tindependent of the dimension of YT , N , and (ii) theo�-line work will remain modest sin e N will typi- ally be quite small.5. Results and Dis ussionWe �rst demonstrate the a ura y of the redu ed-basis output predi tion and output bounds by on-sidering the ase Pe� = 5 in whi h L = 2:5and t = 0:25 are �xed; the remaining parametersk1; k2; k3; k4;Bi vary in De� � [0:1; 10℄4 � [0:01; 1℄.The sample points for SN are hosen randomly (uni-formly) over De�; the new value of � to whi h

8 SIAG/OPT Views-and-Newswe apply the redu ed-basis approximation is k1 =0:5; k2 = 1:0; k3 = 3:0; k4 = 9:0;Bi = 0:6 (similarresults are obtained at other points in De�). Wepresent in Table 1 the a tual error js(�) � sN (�)j;the estimated error �N (�) (our stri t upper boundfor js(�) � sN (�)j); and the e�e tivity �N (�) (theratio of the estimated and a tual errors). We ob-serve the high a ura y and rapid onvergen e ofthe redu ed-basis predi tion, even for this relativelyhigh-dimensional parameter spa e; and the verygood a ura y (low e�e tivity) of our error bound�N (�). The ombination of high a ura y and er-ti�able �delity permits us to pro eed with an ex-tremely low number of modes.N js� sN j �N �N10 4:68 � 10�3 1:43 � 10�2 3.0620 4:70 � 10�4 1:13 � 10�3 2.4030 3:04 � 10�4 1:04 � 10�3 3.4340 1:08 � 10�4 4:61 � 10�4 4.2750 2:47 � 10�5 6:89 � 10�5 2.78Table 1As regards omputational ost, in the limit of\in�nitely many" evaluations, the al ulation of ~s(�)to within 0.1% of sT is roughly 24 times faster thandire t al ulation of sT = `O(uT ); here uT is our un-derlying \truth" �nite element approximation. Thebreakeven point at whi h the redu ed-basis approxi-mation �rst be omes less expensive than dire t eval-uation of sT is roughly 250 evaluations. These arefair omparisons: our \truth" approximation hereis not overly �ne, and our solution strategy foruT 2 YT (an ILU-pre onditioned onjugate-gradientpro edure) is quite eÆ ient. The redu ed-basis ap-proa h is mu h faster simply be ause the dimensionof WN , N , is mu h smaller than the dimension ofYT , N (whi h more than ompensates for the lossof sparsity in A). For more diÆ ult problems thatrequire larger N , or that are not as amenable to fastsolution methods on YT , the relative eÆ ien y of theredu ed-basis approa h is even more dramati .The obvious advantage of the redu ed-basis ap-proa h within the design, optimization, and ontrolenvironment is the very rapid response. However,the \bla kbox" nature of the on-line omponent ofthe pro edure has other advantages. In parti ular,the on-line ode is simple, non-proprietary, and om-pletely de oupled from the (often ompli ated) o�-line \truth" ode. This is parti ularly important in

multidis iplinary design optimization, in whi h vari-ous models and approximations must be integrated.We lose this se tion with a more applied ex-ample. We now �x all parameters ex ept L and t,so that Pe� = 2; (L; t) vary in De� = [2:0; 3:0℄ �[0:1; 0:5℄. We hoose for our two outputs the vol-ume of the �n, V, and the root average tempera-ture, s. As our \design exer ise" we now onstru tthe a hievable set | all those (V; s) pairs asso i-ated with some (L; t) in De�; the result, based onmany evaluations of (V; s ~N+ ) for di�erent values of(L; t) 2 De�, is shown in Figure 2. We present theresults in terms of s ~N+ rather than s ~N to ensure thatthe a tual temperature sT will always be lower thanour predi tions (that is, onservative); and we hoose~N su h that s ~N+ is always within 0.1% of sT to en-sure that the design pro ess is not misled by ina - urate predi tions. Given the obvious preferen es oflower volume and lower temperature, the designerwill be most interested in the lower left boundary ofthe a hievable set | the Pareto eÆ ient frontier; al-though this boundary an of ourse be found without onstru ting the entire a hievable set, many evalua-tions of the outputs will still be required.

19 20 21 22 23 24 25 26 274

6

8

10

12

14

16

Figure 26. Generalizations and IssuesMany of the assumptions that we have introdu edare assumptions of onvenien e and exposition, notne essity. First, the output fun tional `O need notbe same as the inhomogeneity `; with the introdu -tion of an adjoint (or dual) problem [2℄, our results

Volume 11 Number 2 August 2000 9above extend to the more general ase. Se ond, thefun tion g(�) need not be known a priori: g(�) isrelated to an eigenvalue problem whi h an itselfbe readily approximated by a redu ed-basis spa e onstru ted as the span of appropriate eigenfun -tions (in theory we an now only prove asymptoti bounding properties as N ! 1, however in pra -ti e the redu ed-basis eigenvalue approximation on-verges very rapidly, and there is thus little loss of ertainty). Third, these same notions extend, withsome modi� ation, to non oer ive problems, whereg(�) is now in fa t the inf-sup stability parameter[3, 4℄. Finally, nonsymmetri operators are readilytreated, as are ertain lasses of nonlinearity in thestate variables (e.g., eigenvalue problems [1℄).Perhaps the most limiting assumption is (2),aÆne dependen e on the parameter fun tions. Insome ases (2) may indeed apply, but Q may berather large. In su h ases we an redu e the om-plexity and storage of the o�-line and on-line stagesfrom O(Q2) to O(Q) by introdu ing a redu ed-basisapproximation of the error equation (12) for a suit-ably hosen \staggered" sample set SMerr and asso i-ated redu ed-basis spa e onstru ted as the span ofappropriate error fun tions. These ideas also extendto the ase in whi h the parameter dependen e annot be expressed (or a urately approximated) as in(2); however we now need to at least partially aban-don the bla kbox nature of the on-line stage of om-putation, allowing evaluation (though not inversion)of the truth-approximation operator, as well as stor-age of some redu ed-basis ve tors of size N . Thesemethods are urrently under development. For moreinformation and a ess to intera tive examples, seethe web site http://augustine.mit.edu/.REFERENCES[1℄ L. Ma hiels, Y. Maday, I.B. Oliveira, A.T. Pat-era, and D.V. Rovas. Output bounds for redu ed-basis approximations of symmetri positive de�niteeigenvalue problems. C. R. A ad. S i. Paris, S�erieI, to appear.[2℄ Y. Maday, L. Ma hiels, A.T. Patera, and D.V.Rovas. Bla kbox redu ed-basis output bound meth-ods for shape optimization. In Pro eedings 12th

International Domain De omposition Conferen e,Chiba Japan, 2000, to appear.[3℄ D.V. Rovas. An overview of bla kbox redu ed-basisoutput bound methods for ellipti partial di�eren-tial equations. In Pro eedings 16th IMACS WorldCongress 2000, Lausanne Switzerland, 2000, to ap-pear.[4℄ Y. Maday, A.T. Patera, and D.V. Rovas. A bla k-box redu ed-basis output bound method for non- oer ive linear problems. MIT-FML Report 00-2-1,2000; also in the Coll�ege de Fran e Series, to appear.[5℄ A.K. Noor and J.M. Peters. Redu ed basis te hniquefor nonlinear analysis of stru tures. AIAA Journal,18(4):455-462, 1980.[6℄ J.P. Fink and W.C. Rheinboldt. On the error behav-ior of the redu ed basis te hnique in nonlinear �niteelement approximations. Z. Angew. Math. Me h.,63:21-28, 1983.[7℄ T.A. Pors hing. Estimation of the error in the redu edbasis method solution of nonlinear equations. Math-emati s of Computation, 45(172):487-496, 1985.[8℄ Y. Maday, A.T. Patera, and J. Peraire. A generalformulation for a posteriori bounds for output fun -tionals of partial di�erential equations; appli ationto the eigenvalue problem. C. R. A ad. S i. Paris,S�erie I, 328:823-829, 1999.[9℄ A.T. Patera and E.M. R�nquist. A general outputbound result: appli ation to dis retization and it-eration error estimation and ontrol. Math. ModelsMethods Appl. S i., to appear.The Adjoint Approa h in aNutshell1Robert Mi hael LewisICASE, Mail Stop 132C NASA Langley Resear h CenterHampton, Virginia 23681-2199bu karoo�i ase.edu1. Introdu tionOne often hears the term \adjoint approa h" in on-ne tion with the al ulation of �rst derivatives inthe optimization of systems governed by di�eren-tial equations. We give here a qui k introdu tion to1This resear h was supported by the National Aeronauti s and Spa e Administration under NASA Contra t No. NAS1-97046 while the author was in residen e at the Institute for Computer Appli ations in S ien e and Engineering (ICASE),NASA Langley Resear h Center, Hampton, VA 23681-2199.

10 SIAG/OPT Views-and-Newsthe adjoint approa h for those unfamiliar with themethod.As an example, onsider the di�erential equation�(a(x)u0(x))0 � u0(x) = q(x) x 2 (0; 1)u(0) = 0; u(1) = 0: (1)Given the oeÆ ient a(x), u(x) = u(a)(x) is the so-lution of (1). Now suppose we wish to �nd a(x) thatminimizes some fun tional of u(x), say, the least-squares fun tionalF (a) = 12 Z 10 (u(a)(x) � u�(x))2dx; (2)where u�(x) is some given target. Su h a fun -tional might arise in parameter estimation, where weare attempting to estimate a(x) from observationsu�(x), or optimal design, where we are attemptingto hoose a(x) in order to obtain some preferred be-havior u�(x). The question we onsider is how one omputes the derivative F 0(a) of F (a) with respe tto a for use in a derivative-based optimization te h-nique.Abstra tly, we have a nonlinear program of theform minimizea F (a) = f(a; u(a))subje t to C(a) = (a; u(a)) � 0; (3)where, given the design variables a, we ompute thestate variable u(a) by solving some manner of di�er-ential equation S(a; u(a)) = 0: (4)We an take advantage of the stru ture introdu edby (4) when omputing derivatives of F and C.2. A qui k and dirty al ulationWe pro eed formally. By the hain rule,F 0(a) = fa + fududa ; C 0(a) = a + ududa :Impli it di�erentiation of (4), yieldsduda = �S�1u Sa:2

Thus, F 0(a) = fa � fuS�1u Sa (5)C 0(a) = a � uS�1u Sa:3. The sensitivity equationsFormula (5) orresponds to what is known as thesensitivity equations approa h to omputing deriva-tives. In a omputational setting, suppose that thereare n design variables: a = (a1; � � � ; an),m state variables: u = (u1; � � � ; um),p onstraints: = ( 1; � � � ; p).Then the dimensions of the matri es in the produ t(5) are F 0(a)| {z }1�n = fa|{z}1�n � fu|{z}1�m S�1u|{z}m�m Sa|{z}m�n (6)C 0(a)| {z }p�n = a|{z}p�n � u|{z}p�m S�1u|{z}m�m Sa|{z}m�n :This al ulation of F 0(a) and C 0(a) entails mul-tipli ation of the n olumns of Sa by S�1u . That is,we must solve the linearized state equation (repre-sented by the linearized operator Su) for n di�erentright-hand sides. Thus, the ost of omputing F 0(a)via (6) grows with the number of inputs (indepen-dent variables) a. For those readers familiar withautomati di�erentiation, we note that this order of al ulations orresponds to the forward mode of au-tomati di�erentiation.2The appeal to the impli it fun tion theorem requires justi� ation in the general setting of the dependen e on a of u(a),sin e a may represent su h quantities as oeÆ ients in di�erential operators, boundary values, or the geometry of the domainon whi h the problem is posed. In pra ti e, it is sometimes diÆ ult and even impossible to verify that the obje tive and onstraints are truly di�erentiable.

Volume 11 Number 2 August 2000 114. The adjoint approa hFor a �nite-dimensional problem, we an transpose(6) to obtainrF (a)| {z }n�1 = raf| {z }n�1 � STa|{z}n�m S�Tu|{z}m�mruf| {z }m�1 (7)rC(a)| {z }n�p = ra |{z}n�p � STa|{z}n�m S�Tu|{z}m�mru |{z}m�pThis al ulation requires only one appli ation ofS�Tu to ompute F 0(a), and p appli ations of S�Tu to ompute C 0(a). After transposition of (6), the ostthus depends on the number of outputs. This orderof al ulations orresponds to the reverse mode ofautomati di�erentiation.The adjoint approa h is the in�nite-dimensionalanalog of this transposition. The in�nite-dimensional situation is more ompli ated, sin eF 0(a), fa, fu, et ., are now linear fun tionals on pos-sibly in�nite-dimensional spa es of fun tions. Nev-ertheless, the transposition in (7) is the basi ideaunderlying the adjoint approa h.The quantity � = �S�Tu ruf that arises in om-puting raF (a) is alled the ostate. One similarly omputes ostates for the olumns of raC(a).5. The model problemWe illustrate the adjoint approa h for the modelproblem (1){(2). It is onvenient to view (1) in theweak form, so that u 2 H10 [0; 1℄ andZ 10 [a(x)u0(x)v0(x) + u(x)v0(x)℄ dx (8)� Z 10 q(x)v(x) dx = 0for all v 2 H10 . Then S(a; u(a)) = 0 is de�ned by(8). Note that S takes its value in the spa e of lin-ear fun tionals a ting on v 2 H10 .The adjoint equation STu � = �fu is then givenby 3hSTu �; vi � h�; Suvi (9)= Z 10 [a(x)u0(x)�0(x) + u(x)�0(x)℄ dx

= h�fu; vi = � Z 10 (u(x)� u�(x))v(x) dxfor all v 2 H10 , where h�; �i represents the dual-ity pairing between linear fun tionals and the ar-guments they a t on. We re ognize (9) as the weakform of the adjoint problem�(a(x)�0(x))0 + �0(x) = �(u(x)� u�(x))�(0) = �(1) = 0:Finally, F 0(a) = STa �. We haveF 0(a)� = hSTa �; �i � h�; Sa�i= Z 10 �(x)u0(x)�0(x) dx:This gives a formula and a representation for the lin-ear fun tional F 0(a) in terms of integration againstthe fun tion �0(x)u0(x). However, the fun tion�0(x)u0(x) may or may not be suitable as a dire tionof steepest des ent, and may require further smooth-ing depending on the appli ation.6. Some ommentsNote that the adjoint approa h is more eÆ ient whenone has fewer dependent variables (i.e., outputs Fand C) than independent variables (design variablesa), while the sensitivity equations are more eÆ ientif the situation is reversed. If one has a large numberof both inputs and outputs, then one approa h or theother may be more eÆ ient relative to the other, buteither will be expensive in absolute terms.What about se ond derivatives? Unfortunately,there is no espe ially eÆ ient way to ompute se ondderivatives. The analyti al al ulation of a se ondderivative requires an appli ation of both S�1u andS�Tu .Finally, we allude to interesting problems that an arise in onne tion with the ostate � and theadjoint problem. The ostate � is a tually a linearfun tional, and as su h, it an \look" rather strangeif one tries to interpret it as a fun tion in the usualsense (for instan e, onsider the delta fun tion). For3For a linear map L : X ! Y between two linear spa es X and Y , the adjoint LT is a map LT : X ! Y between thedual spa es Y 0 and X 0 de�ned by hLT�; vi = h�; Lvi for all � 2 Y 0 and v 2 X. The appearan e of the dual spa es ofin�nite-dimensional spa es is what sometimes makes the interpretation of the transposition in (7) a bit subtle.

12 SIAG/OPT Views-and-Newssome problems, the nature of � an intera t withthe de�nition of the adjoint problem to make someapproa hes to deriving the adjoint problem breakdown. This in turn, has led some authors to on- lude that the adjoint approa h annot be appliedto their problem. This is, in fa t, not the ase, ifthe nature of � and the adjoint problem are prop-erly understood. For a dis ussion of these subtleties,see [4℄.7. Histori al noteTo the author's knowledge, the earliest uses in printof the te hnique we now all the adjoint approa happear in [3, 2℄. The adjoint approa h is sometimesattributed to Lions in [5℄, but what Lions uses is theKarush-Kuhn-Tu ker system of ne essary onditionsfor (3){(4) (absent onstraints) formulated asminimizea;u f(a; u)subje t to S(a; u) = 0: (10)Treatment of (4) as equality onstraints in the on-text of generalized redu ed gradient algorithms alsoled early on (e.g., [1℄ ) to te hniques that may beviewed as ante edents of the adjoint approa h. In-deed, the reader familiar with variable redu tionte hniques will note that the formulae (6) and (7) orrespond to formulae for the redu ed gradient of fwith respe t to the equality onstraints S(a; u) = 0in (10). REFERENCES[1℄ J. Abadie, Appli ation of the GRG algorithm to optimal ontrol problems, in Integer and Nonlinear Program-ming, J. Abadie, ed., North-Holland Elsevier, 1970.[2℄ G. Chavent, M. Dupuy, and P. Lemonnier, His-tory mat hing by use of optimal theory, So iety ofPetroleum Engineers Journal, (1975), pp. 74{86.[3℄ W. H. Chen, G. R. Gavalas, J. H. Seinfeld, andM. L. Wasserman, A new algorithm for automati history mat hing, So iety of Petroleum EngineersJournal, (1974), pp. 593{608.[4℄ R. M. Lewis, Numeri al omputation of sensitivitiesand the adjoint approa h, in Computational Meth-ods for Optimal Design and Control, J. Borggaard,J. Burns, E. Cli�, and S. S hre k, eds., Birkh�auser,1998, pp. 285{302. Also available as ICASE te hni alreport 97{61.

[5℄ J. L. Lions, Controle optimal de syst�emes gouvern�es pardes �equations aux d�eriv�ees partielles, Dunod, 1968.A Lagrange-Newton-Krylov-S hurMethod forPDE-Constrained OptimizationGeorge Biros and Omar GhattasMe hani s, Algorithms, and Computing LaboratoryDepartment of Civil & Environmental EngineeringCarnegie Mellon University, Pittsburgh, PA, USAEmail: fbiros,oghattasg� s. mu.eduURL: http://www. s. mu.edu/~fgbiros,oghattasg1. Introdu tionPDE- onstrained optimization is a frontier problemin omputational s ien e and engineering. All PDE- onstrained problems share the diÆ ulty that PDEsolution is just a subproblem asso iated with opti-mization. Thus, the optimization problem is oftensigni� antly more diÆ ult to solve than the simu-lation problem. We are parti ularly interested inlarge-s ale problems that require parallel omputingto make them tra table.To illustrate the main issues, let's onsider amodel problem of optimal distributed ontrol of aNavier-Stokes ow:minF(u; p; b) =12 Z(ru+ruT ) : (ru+ruT )d+ �2 Z b � b dsubje t to:��r � (ru+ruT ) + (ru)u+rp+ b = 0 in r � u = 0 in u = 0 on �Here, u is the uid velo ity �eld, p the pressure �eld,b the body for e ontrol fun tion, � a weighting pa-rameter, and � the inverse of the Reynolds number.The obje tive is to minimize the rate of dissipationof vis ous energy and a ost asso iated with a bodyfor e ontrol fun tion. The onstraints are the sta-tionary in ompressible Navier-Stokes equations withDiri hlet boundary onditions.

Volume 11 Number 2 August 2000 13We an form a Lagrangian fun tional, and re-quire its stationarity with respe t to the state (u,p)and optimization (b) variables and the Lagrangemultipliers. Taking variations and invoking the ap-propriate Green identities, we arrive at the following�rst-order ne essary onditions:Adjoint Equations:��r � (r�u +r�Tu ) + (ru)T�u � (r�u)u+r�p �r � (ru+ruT ) = 0 in r � �u = 0 in �u = 0 on �State Equations:��r � (ru+ruT ) + (ru)u+rp+ b = 0 in r � u = 0 in u = 0 on �Control Equations:� b+ �u = 0 in The state equations are just the original Navier-Stokes PDEs. The adjoint equations, whi h resultfrom stationarity with respe t to state variables, arethemselves PDEs, and are linear in the Lagrangemultipliers �u and �p. Finally, the ontrol equationsare (in this ase) algebrai .Thus we end up with a large, oupled, unstru -tured system of optimality onditions (or at leastbigger, more oupled, and less stru tured than seenby a Navier-Stokes solvers). How to go about solvingit? The usual way is to eliminate state variables andLagrange multipliers and, orrespondingly, the stateequations and adjoint equations; to redu e the sys-tem to a manageable one in just the ontrol (i.e. de- ision) variables b. Here's one way to do this: givenb at some iteration, we solve the state equations forthe state variables u; p. Knowing the state variablesthen permits us to solve the adjoint equations for theLagrange multipliers �u; �p. Finally, with the statesand multipliers known, we an update b by iterat-ing on the ontrol equation. The whole pro ess isrepeated until onvergen e. This elimination pro e-dure is termed a redu ed spa e method, in ontrastto a full spa e method, in whi h one solves for thestates, ontrols, and multipliers simultaneously.

Redu ed spa e methods are attra tive for severalreasons. Solving the subsets of equations in sequen eimparts some stru ture to the problem. State equa-tion solvers build on years of development of large-s ale parallel PDE solvers. Adjoint PDE solversdon't exa tly grow on trees|but the strong simi-larities between the state and adjoint operators sug-gest that an existing PDE solver for the state equa-tions an be modi�ed easily to handle the adjointsystem (at least on a good day). Finally, the on-trol equations are usually reasonably tame, at leastto evaluate. Another advantage of redu tion is thatthe full spa e system is often very ill- onditioned,whereas the three subsystems are typi ally better onditioned.On the other hand, the big disadvantage of re-du ed methods is the need to solve the state andadjoint equations at ea h iteration of the redu edsystem|a dire t onsequen e of the redu tion ontothe de ision variable spa e. So it's natural to goba k to the full spa e, and ask if it's possible to solvethe entire optimality system simultaneously, but re-tain the stru ture-indu ing, ondition-improving ad-vantages of redu ed spa e methods|while avoidingtheir disadvantages.In this arti le, we present su h a method. Thekey idea is to solve in the full spa e using a Newtonmethod, but pre ondition with a quasi-Newton re-du ed spa e method. The Karush-Kuhn-Tu ker sys-tem arising at ea h Newton iteration is solved usinga Krylov iterative method, and it is this system towhi h the pre onditioner is applied. We have foundthat the redu ed spa e pre onditioner is very e�e -tive in redu ing the number of Krylov iterations, andapplying it aptures the favorable stru ture of re-du ed methods. On the other hand, sin e the redu -tion is used just as a pre onditioner, we an heat onthe state and adjoint solves, repla ing them with ap-proximations whi h ould be their own pre ondition-ers. So we arrive at a method that ombines rapid onvergen e in the outer Newton iteration (typi allymesh-independent), with fast onvergen e of the in-ner Krylov iteration (whi h an be as good as mesh-independent). We don't even need to ompute se -ond derivatives|sin e a Krylov method is used tosolve the KKT system, we an apply the usual di-re tional di�eren ing tri k to approximate the La-grangian Hessian{ve tor produ t.

14 SIAG/OPT Views-and-NewsWhy the name Lagrange-Newton-Krylov-S hur?It is ommon in PDE-solver ir les to use the phraseNewton-Krylov-X to refer to Newton methods forsolving PDEs that employ Krylov linear solvers,with X as the pre onditioner for the Krylov method.Sin e Lagrange-Newton is sometimes used to de-s ribe a Newton method for solving the optimalitysystem (a.k.a. an SQP method), and sin e a redu edspa e method an be viewed as a S hur omplementmethod for the KKT system, we arrive at the on- atenation LNKS. It's a mouthful, but it preservesthe tie to modern PDE solvers, whose use of approxi-mate de ompositions as pre onditioners inspired thisapproa h [6℄. David Keyes suggested (a variation of)this name in his plenary talk at the 1999 ombinedSIAM Optimization/Annual meeting [5℄.In the remainder of this arti le, we give a briefoverview of the LNKS method and some sample re-sults for an optimal ow ontrol problem on a CrayT3E. Further details an be found in [2℄, and moreextensive dis ussion and results in forth oming arti- les that fo us on the inner Krylov iteration [3℄ andthe outer Newton iteration [4℄. We note �nally thatBattermann and Heinkens hloss have presented asomewhat di�erent method for pre onditioning KKTmatri es that also makes use of state and ontrolspa e de ompositions [1℄.2. Redu ed Spa e MethodsIn this se tion we dis uss redu ed spa e SQP meth-ods, on entrating on the dis rete form of a typi alPDE- onstrained optimization problem:minx f(x) subje t to (x) = 0;where x are the state and de ision variables, f isthe obje tive fun tion and are the dis retized stateequations. Using Lagrange multipliers �, we an de-�ne the Lagrangian fun tion byL(x;�) := f(x) + �T (x):The �rst order optimality onditions require that theLagrangian gradient vanish:( �xL��L ) = ( g +AT� ) = 0;

where g is the gradient of the f and A is the Ja o-bian matrix of the onstraints. A Newton step onthe optimality onditions (whi h, in the absen e ofinequality onstraints, is Sequential Quadrati Pro-gramming) is given by:" W ATA 0 #( px�+ ) = �( g ) ;where W is the Hessian of the Lagrangian fun -tion with respe t to the optimization variables, pxis the sear h dire tion in x, and �+ is the updatedLagrange multiplier. This system is known as theKarush-Kuhn-Tu ker (KKT) system, and its oef-� ient matrix as the KKT matrix. To exploit thestru ture of the state onstraints, we partition theoptimization variables into state variables xs andde ision variables xd. The partitioned KKT systembe omes:264 Wss Wsd ATsWds Wdd ATdAs Ad 0 3758><>: pspd�+ 9>=>; = �8><>: gsgd 9>=>; (1)This system is of dimension 2n+m, where n is thenumber of state variables and m the number of de i-sion variables. State-of-the-art algorithms for PDE- onstrained optimization exploit two fa ts. First,nobody wants to ompute se ond derivatives|it'shard enough onvin ing the PDE solver ommunityof the need for �rst derivatives. (No doubt this dif-� ulty will be mitigated by ontinuing advan es inautomati di�erentiation tools.) And se ond, every-body wants to use existing software for \inverting"the state Ja obian. Sin e this is the kernel step ina Newton-based PDE solver, there is a large bodyof work to draw from. For example, for ellipti PDEs, there exist optimal or nearly-optimal paral-lel algorithms (e.g. domain de omposition methodsor multigrid) that require algorithmi work that islinear or weakly superlinear in n, and s ale to thou-sands of pro essors and millions of variables.One way to exploit existing PDE-solvers is toeliminate the state and adjoint equations and vari-ables, and then solve an un onstrained optimizationproblem in the remaining de ision spa e (this is sim-ilar to the argument of the previous se tion, ex epthere we are linearizing �rst, then eliminating, as op-posed to vi e versa.) We refer to this as Newtonredu ed SQP (or N-RSQP), and it an be derived

Volume 11 Number 2 August 2000 15by blo k elimination on the KKT system: Given pd,solve the last blo k of equations (the state system)for ps; then solve the �rst (the adjoint system) to�nd �+, and �nally solve the middle (the de isionsystem) for pd. It is easy to verify that this blo kelimination is equivalent to the following blo k fa -torization of the KKT matrix:264 WssA�1s 0 IWdsA�1s I ATdA�TsI 0 0 375264 As Ad 00 Wz 00 Wyz ATs 375(2)where the redu ed Hessian matrix is de�ned byWz := ATdA�Ts WssA�1s Ad�ATdA�Ts Wsd ��WdsA�1s Ad +Wdd;and the \ ross-Hessian" byWyz :=Wsd �WssA�1s Ad:Note that these fa tors an be permuted to blo ktriangular form, so we an think of this as a blo kLU fa torization of the KKT matrix. It is lear thatthe only linear systems that need to be solved haveeither the state Ja obian As or its transpose as their oeÆ ient matrix|a \solved problem"|or else theredu ed HessianWz, whi h is dense and of dimensionof the de ision spa e. Thus, redu ed methods areparti ularly attra tive when the the de ision vari-ables are mu h fewer than the states.But two problems remain. First are the se -ond derivative terms. Se ond, and more problem-ati , is the need for m solutions of the (linearized)state equations for onstru tion of A�1s Ad in Wz.This is parti ularly troublesome for large-s ale 3Dproblems, where (linearized) PDE systems are usu-ally solved iteratively, and solution osts annot beamortized over multiple right hands as e�e tively aswith dire t solvers. When the simulation problemis an overnight run on a large parallel ma hine, thisrequirement e�e tively rules out the use of N-RSQP.A popular te hnique that addresses these twodiÆ ulties is a quasi-Newton RSQP (QN-RSQP)method that repla es the redu ed Hessian Wz witha quasi-Newton approximation Bz, and dis ards allother Hessian terms. This orresponds to the follow-

ing approximation of the KKT blo k fa tors:264 0 0 I0 I ATdA�TsI 0 0 375264 As Ad 00 Bz 00 0 ATs 375 (3)It is easy to verify that just two state solves per iter-ation are required (a tually one linearized state, andone adjoint), as opposed to the m of N-RSQP. Andwith Hessian terms either approximated or dropped,no se ond derivatives are needed. A measure ofthe su ess of QN-RSQP is its appli ation to nu-merous optimal ontrol, optimal design, and inverseproblems governed by PDEs from linear and nonlin-ear elasti ity, in ompressible and ompressible ow,heat ondu tion and onve tion, phase hanges, owthrough porous media, et . Of ourse, something hasto give, and that is the onvergen e rate: a redu tionfrom quadrati in the Newton ase to two-step su-perlinear. Moreover, the number of iterations takenby QN-RSQP depends on the onditioning of the re-du ed Hessian, and often in reases as the number ofde ision variables grows, rendering large-s ale prob-lems intra table. In the next se tion, we propose amethod that ombines the fast onvergen e of New-ton's method with the stru ture-exploiting proper-ties of redu ed methods.3. LNKS: Krylov solution of theKKT system with approximateQN-RSQP pre onditioningIn this se tion, we present a method for solving theKKT system (1). For optimization problems on-strained by 3D PDEs, sparse fa torization of theKKT matrix is not an option|su h methods arenot viable for As, let alone the entire matrix. In-stead, we use a Krylov iterative method, spe i� allythe quasi-minimum residual (QMR) method. How-ever, the varying s ales between Hessian and Ja o-bian terms in the KKTmatrix, and its inde�niteness,demand an e�e tive pre onditioner. This pre ondi-tioner must be apable of exploiting the stru ture ofthe state onstraints (spe i� ally that good pre on-ditioners exist for As), must be heap to apply, andmust be e�e tive in redu ing the number of Kryloviterations. The QN-RSQP method des ribed in the

16 SIAG/OPT Views-and-Newsprevious se tion �ts the bill. Applying the pre ondi-tioner amounts to solving with the QN-RSQP fa tor-ization (3), ex ept that state Ja obians are repla edby their approximations ~As:264 0 0 I0 I ATd ~As�TI 0 0 375264 ~As Ad 00 Bz 00 0 ~AsT 375 (4)Repla ing As with ~As is permissible, sin e QN-RSQP is being used as a pre onditioner. A good hoi e for ~As is, in turn, one of the available pre- onditioners for As|for many PDE operators, thereexist near-spe trally-equivalent pre onditioners thatare both heap to apply (typi ally linear or weaklysuperlinear in problem size) and e�e tive (resultingin iteration numbers that are independent of, or in- rease very slowly in, problem size).With (4) used as a pre onditioner, the pre on-ditioned KKT matrix ends up having the followingform:264 Is O(Es) 0~W Tyz ~As�1 O(Es) +WzB�1z O(Es)Wss ~As�1 ~WyzB�1z Is 375where Es := A�1s � ~As�1Is := As ~As�1~Wyz := Wsd �Wss ~As�1AdFor exa t state equation solution, Es = 0 and Is =I , and we see that the QN-RSQP pre onditioner lusters the spe trum of the KKT matrix, withall eigenvalues either unit or belonging to WzB�1z .Therefore, when ~As is a good pre onditioner for thestate Ja obian, and when Bz is a good approxima-tion of the redu ed Hessian (as it should be asymp-toti ally), we might expe t the QN-RSQP pre ondi-tioner (4) to be e�e tive in redu ing the number ofKrylov iterations (but note that the pre onditionedKKT matrix is non-normal, so a rigorous analysisrequires well- onditioned eigenve tors).How s alable is the method, with respe t to in- reasing problem size and number of pro essors? Fors alability, we require that the work in rease near-linearly with problem size (algorithmi s alability)

and that it parallelizes well (parallel s alability). Letus examine the major omponents:Formation of the KKT matrix{ve tor prod-u t. For PDE- onstrained optimization, the Hes-sian of the Lagrangian fun tion and the Ja obianof the onstraints are usually sparse with stru turedi tated by the mesh (parti ularly when the de i-sion variables are mesh-related). Thus, formationof the matrix-ve tor produ t at ea h QMR itera-tion is linear in both state and de ision variables,and parallelizes well due to a high omputation-to- ommuni ation ratio and minimal sequential bottle-ne ks.Appli ation of the QN-RSQP pre onditioner.The main work involved is appli ation of the stateJa obian pre onditioner ~As and its transpose, and\inversion" of the quasi-Newton approximation tothe redu ed Hessian, Bz. We an often make use ofs alable, parallel state Ja obian pre onditioners thatrequires O(n) work to apply (as in various domainde omposition pre onditioners for ellipti problems).Furthermore, whenBz is based on a limited-memoryquasi-Newton update (as in our implementation), itswork is also linear in the de ision variables, and theve tor operations are easily parallelized (or as eas-ily as ve tor inner produ ts an be). Therefore, we on lude that appli ation of the QN-RSQP pre on-ditioner requires linear work and parallelizes well.The Krylov (inner) iteration. As argued above,with an \optimal" state pre onditioner and a goodBz approximation, we an anti ipate that the num-ber of inner, Krylov iterations will be relatively in-sensitive to the problem size.The Lagrange-Newton (outer) iteration. Thenumber of outer, Newton iterations is often indepen-dent of problem size for PDE-type problems, and theproblems we have solved exhibit this type of behav-ior as well.This ombination of linear work per Krylov itera-tion, weak dependen e of Krylov iterations on prob-lem size, and independen e of Lagrange-Newton iter-ations on problem size suggest a method that s aleswell with in reasing problem size and number of pro- essors.How well does the LNKS method work in pra -ti e? Here, we quote a set of representative resultsfrom many we have obtained for up to 1.5 millionstate variables and 50,000 ontrol variables on up

Volume 11 Number 2 August 2000 17to 256 pro essors. The problem is optimal Navier-Stokes ow ontrol, similar to that of Se tion 1, ex- ept that the ontrols are boundary velo ities. Thespe i� problem is ontrol of 3D ow around a ylin-der at sub riti al onditions, with ontrols on thedownstream side of the ylinder. Approximationis by Galerkin �nite elements, both for state and ontrol variables. We have implemented the LNKSmethod on top of the PETS library for parallel solu-tion of PDEs from Argonne. The table shows resultsfor 64 and 128 pro essors of a Cray T3E for a roughlydoubling of problem size. Results for the QN-RSQPand LNKS algorithms are presented. In the tableLNKS-EX refers to exa t solution of the linearizedNavier-Stokes equation within the QN-RSQP pre- onditioner, whereas LNKS-PR refers to appli ation

of a blo k-Ja obi (with lo al ILU(0)) approximationof the linearized Navier-Stokes operator. LNKS-PR-TR uses a trun ated Newton method and avoidsfully onverging the KKT system for iterates thatare far from a solution.The results in the table re e t the independen eof Newton iterations on problem size, the mild de-penden e of KKT iterations on problem size, and theresulting reasonable s alability of the method. It isimportant to point out here that the Navier-Stokesdis rete operator is very ill- onditioned, and there isroom for improvement of its domain-de ompositionpre onditioner. The performan e of the QN-RSQPKKT pre onditioner would improve orrespondingly.A dramati a eleration of the LNKS algorithm isa hieved by trun ating the Krylov iterations.states ontrols pre onditioning Newton iter average KKT iter time (hours)389,440 QN-RSQP 189 | 46.36,549 LNKS-EX 6 19 27.4(64 pro s) LNKS-PR 6 2,153 15.7LNKS-PR-TR 13 238 3.8615,981 QN-RSQP 204 | 53.18,901 LNKS-EX 7 20 33.8(128 pro s) LNKS-PR 6 3,583 16.8LNKS-PR-TR 12 379 4.1More detailed results are given in [2, 3, 4℄. Thesereferen es also dis uss the important topi s of glob-alization and the details of the inexa tness in solv-ing the KKT system, whi h were not mentioned herefor reasons of spa e. Another issue is additional in-equality onstraints; we have re ently implementedwith Andreas W�a hter and Larry Biegler a parallelversion of their interior point method for treatingsu h onstraints, within the ontext of LNKS. Fi-nally, this summer we will be releasing a publi ly-available software library for parallel solution ofPDE- onstrained optimization problems, built ontop of the PETS system, and in luding LNKS andother methods.A knowledgmentsThis work is a part of the Teras ale Algo-rithms for Optimization of Simulations (TAOS)proje t at CMU, with support from NASA grant

NAG-1-2090 and NSF grant ECS-9732301 (underthe NSF/Sandia Life Cy le Engineering Program).Computing servi es on the Pittsburgh Super omput-ing Center's Cray T3E were provided under PSCgrant ASC-990003P. We thank Satish Balay, BillGropp, Lois M Innes, and Barry Smith of ArgonneNational Lab for their work in making PETS avail-able to the resear h ommunity. We also thankJonathan Shew huk of UC Berkeley for provid-ing the meshing and partitioning routines Pyramidand Sli e. Finally, we thank David Keyes of OldDominion University/ICASE/Lawren e Livermore,David Young of Boeing, and other members of theTAOS proje t|Ros oe Bartlett, Larry Biegler, IvanMal� evi� , Andreas W�a hter|for their useful om-ments. REFERENCES

18 SIAG/OPT Views-and-News[1℄ A. Battermann and M. Heinkens hloss, Pre ondi-tioners for Karush{Kuhn{Tu ker matri es arising inoptimal ontrol of distributed systems, Te h. Rep.TR96-34, Department of Computational and AppliedMathemati s, Ri e University, November 1996.[2℄ G. Biros and O. Ghattas, Parallel Newton-Krylovalgorithms for PDE{ onstrained optimization, inPro eedings of SC99, Portland, Oregon, 1999.Available fromhttp:// s. mu.edu/�oghattas/papers/s 99/s 99.ps.[3℄ , Lagrange-Newton-Krylov-S hur algorithms forPDE- onstrained optimization. Part I: KKT pre on-ditioners, 2000. In preparation.[4℄ , Lagrange-Newton-Krylov-S hur algorithms forPDE- onstrained optimization. Part II: Trun atedNewton solver, 2000. In preparation.[5℄ D. E. Keyes, Krylov, Lagrange, N℄ewton, and S hwarz:Combinations and Permutations, note = Plenary talkto joint session of the 1999 SIAM Annual Meetingand 1999 SIAM Conferen e on Optimization, address= Atlanta.[6℄ D. E. Keyes and W. D. Gropp, A omparison of do-main de omposition te hniques for ellipti partial dif-ferential equations and their parallel implementation,SIAM Journal on S ienti� and Statisti al Comput-ing, 8 (1987), pp. S166{S202.BulletinDon't forget to register for the �rst SIAM Con-feren e on Computational S ien e and Engineering,Sept 21-24 in Washington DC. It promises to be agreat onferen e and there are several good minisym-posium and ontributed sessions on optimization inthe program. In luding the ones mentioned in the�rst arti le of this newsletter, there are also sessionson Numeri al Optimization in Engineering, OptimalControl and Shape Optimization, Large S ale Non-linear Programming: Algorithms and Appli ations,

Optimizing Matrix Stability, Optimizing under Un- ertainty, and Unstru tured Mesh Generation andOptimization.Comments from the Chairand EditorWe would like to thank Natalia Alexandrov foragreeing to serve as the guest editor for this issue ofthe SIAG/OPT Views-and-News. She has ompileda very ni e set of arti les on engineering optimiza-tion.Our intention is to ontinue to produ e at leastone newsletter a year that is devoted to a spe i� area of optimization. We would like to invite othersto submit ideas for spe ial issues of the newsletter. Ifyou have a parti ular area of optimization that youthink would be of interest to the general optimiza-tion audien e please onta t either Tom Coleman ormyself with the topi and a list of suggested ontrib-utors.Thomas F. Coleman, SIAG/OPT ChairComputer S ien eCornell UniversityItha a, NY 14853 oleman�t . ornell.eduJuan C. Meza, EditorSandia National LaboratoriesP.O. Box 969, MS 9217Livermore, CA 94551meza� a.sandia.gov