Springer New York Berlin Heidelberg Hong Kong London Milan Paris Tokyo

Texts in Applied Mathematics 43

Editors J.E. Marsden

L. Sirovich M. Golubitsky

S.S. Antman

Advisors G.looss

P. Holmes D. Barkley

M. Dellnitz P. Newton

Texts in Applied Mathematics

I. SiTOvich: Introduction to Applied Mathematics. 2. Wiggins: Introduction to Applied Nonlinear Dynamical Systems and Chaos. 3. Hale/Kor;ak: Dynamics and Bifurcations. 4. Chorin/Marsden: A Mathematical Introduction to Fluid Mechanics, 3rd ed.

5. Hubbard/West: Differential Equations: A Dynamical Systems Approach: Ordinary Differential Equations.

6. Sontag: Mathematical Control Theory: Deterministic Finite Dimensional Systems, 2nd ed.

7. Perko: Differential Equations and Dynamical Systems, 3rd ed. 8. 5mborn: Hypergeometric Functions and Their Applications. 9. Pipkin: A Course on Integral Equations. 10. Hoppensteadt/Peskin: Modeling and Simulation in Medicine and the Life

Sciences, 2nd cd. II. Braun: Differential Equations and Their Applications, 4th ed. 12. Stoer/Bulirsch: Introduction to Numerical Analysis, 3rd ed. 13. Renardy/Rogers: An Introduction to Partial Differential Equations. 14. Banks: Growth and Diffusion Phenomena: Mathematical Frameworks and

Applications. 15. Brenner/Scott: The Mathematical Theory of Finite Element Methods, 2nd ed. 16. Van de Velde: Concurrent Scientific Computing. 17. Marsden/Ratiu: Introduction to Mechanics and Symmetry, 2nd ed. 18. Hubbard/West: Differential Equations: A Dynamical Systems Approach:

Higher-Dimensional Systems. 19. Kaplan/Glass: Understanding Nonlinear Dynamics. 20. Holmes: Introduction to Perturbation Methods. 21. Curtain/Zwart: An Introduction to Infinite-Dimensional Linear Systems

Theory. 22. Thomas: Numerical Partial Differential Equations: Finite Difference Methods. 23. Taylor: Partial Differential Equations: Basic Theory. 24. Merkin: Introduction to the Theory of Stability of Motion. 25. Naber: Topology, Geometry, and Gauge Fields: Foundations. 26. Polderman/Willems: Introduction to Mathematical Systems Theory: A

Behavioral Approach. 27. Reddy: Introductory Functional Analysis with Applications to Boundary-Value

Problems and Finite Elements. 28. Gustafson/Wilcox: Analytical and Computational Methods of Advanced

Engineering Mathematics. 29. Tveito/Winther: Introduction to Partial Differential Equations: A

Computational Approach. 30. Gasquet/Witomski: Fourier Analysis and Applications: Filtering, Numerical

Computation, Wavelets.

(continued after index)

Peter Deuflhard Andreas Hohmann

Numerical Analysis in Modern Scientific Computing

An Introduction

Second Edition

With 65 Illustrations

, Springer

Peter Deuflhard Konrad-Zuse-Zentrum (ZIB) Berlin-Dahlem, D-14195 Germany deuflhard@zib.de

Series Editors J.E. Marsden Control and Dynamical Systems 107-S1 California Institute of Technology Pasadena, CA 91125 USA marsden@cds.caltech.edu

M. Golubitsky Department of Mathematics University of Houston Houston, TX 77204-3476 USA

Andreas Hohmann AMS D2 Vodafone TPAI Dusseldorf, D-40547 Germany andreas.hohmann@d2vodafone.de

L. Sirovich Division of Applied Mathematics Brown University Providence, RI 02912 USA chico@camelot.mssm.edu

S.S. Antman Department of Mathematics and Institute for Physical Science

and Technology University of Maryland College Park, MD 20742-4015 USA ssa@math.umd.edu

Mathematics Subject Classification (2000): 65-XX, 6S-XX, 65-01, 65Fxx, 6SNxx

Library of Congress Cataloging-in-Publication Data Deuflhard. P. (Peter)

Numerical analysis in modern scientific computing: an introduction / Peter Deuflhard, Andreas Hohmann.-2nd ed.

p. cm. - (Texts in applied mathematics; 43) Rev. ed. of: Numerical analysis. 1995. Includes bibliographical references and index.

I. Numerical analysis-Data processing. I. Hohmann, Andreas. 1964- II. Deutlhard, P. (Peter). Numerische Mathematik I. English. III. Title. IV. Series. QA297 .D45 2003 519.4-dc21 2002030564

ISBN 978-1-4419-2990-7 ISBN 978-0-387-21584-6 (eBook) Printed on acid-free paper.

DOl 10.1007/978-0-387-21584-6

© 2003 Springer-Verlag New York, Inc. Softcover reprint of the hardcover 1 st edition 2003 All rights reserved. This work may not be translated or copied in whole or in part without the written per­mission of the publisher (Springer-Verlag New York, Inc., 175 Fifth Avenue, New York, NY 10010, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such. is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights.

9 8 7 6 5 4 3 2 I SPIN 10861791

www.springer-ny.com

Springer-Verlag New York Berlin Heidelberg A member (If' Berte/smannSpringer Sciellce+Business Media GmbH

Series Preface

Mathematics is playing an ever more important role in the physical and biological sciences, provoking a blurring of boundaries between scientific disciplines and a resurgence of interest in the modern as well as the clas­sical techniques of applied mathematics. This renewal of interest, both in research and teaching, has led to the establishment of the series Texts in Applied Mathematics (TAM).

The development of new courses is a natural consequence of a high level of excitement on the research frontier as newer techniques, such as numerical and symbolic computer systems, dynamical systems, and chaos, mix with and reinforce the traditional methods of applied mathematics. Thus, the purpose of this textbook series is to meet the current and future needs of these advances and to encourage the teaching of new courses.

TAM will publish textbooks suitable for use in advanced undergraduate and beginning graduate courses, and will complement the Applied Mathe­matical Sciences (AMS) series, which will focus on advanced textbooks and research-level monographs.

Pasadena, California Providence, Rhode Island Houston, Texas College Park, Maryland

J.E. Marsden L. Sirovich

M. Golubitsky S.S. Antman

Preface

For quite a number of years the rapid progress in the development of both computers and computing (algorithms) has stimulated a more and more de­tailed scientific and engineering modeling of reality. New branches of science and engineering, which had been considered rather closed until recently, have freshly opened up to mathematical modeling and to simulation on the computer. There is clear evidence that our present problem-solving ability does not only depend on the accessibility of the fastest computers (hard­ware), but even more on the availability of the most efficient algorithms (software) .

The construction and the mathematical understanding of numerical al­gorithms is the topic of the academic discipline Numerical Analysis. In this introductory textbook the subject is understood as part of the larger field Scientific Computing. This rather new interdisciplinary field influ­ences smart solutions in quite a number of industrial processes, from car production to biotechnology. At the same time it contributes immensely to investigations that are of general importance to our societies-such as the balanced economic and ecological use of primary energy, global climate change, or epidemiology.

The present book is predominantly addressed to students of mathematics, computer science, science, and engineering. In addition, it intends to reach computational scientists already on the job who wish to get acquainted with established modern concepts of Numerical Analysis and Scientific Computing on an elementary level via personal studies.

viii Preface

The field of Scientific Computing, situated at the confluence of mathe­matics, computer science, natural science, and engineering, has established itself in most teaching curricula, sometimes still under the traditional name Numerical Analysis. However, basic changes in the contents and the presen­tation have taken place in recent years, and this already at the introductory level: classical topics, which had been considered important for quite a time, have just dropped out, new ones have entered the stage. The guiding principle of this introductory textbook is to explain and exemplify essential concepts of modern Numerical Analysis for ordinary and partial differential equations using the simplest possible model problems. Nevertheless, read­ers are only assumed to have basic knowledge about topics typically taught in undergraduate Linear Algebra and Calculus courses. Further knowledge is definitely not required.

The primary aim of the book is to develop algorithmic feeling and think­ing. After all, the algorithmic approach has historically been one of the roots of today's mathematics. It is no mere coincidence that, besides con­temporary names, historical names like Gauss, Newton, and Chebyshev are found in numerous places all through the text. The orientation toward algorithms, however, should by no means be misunderstood. In fact, the most efficient algorithms often require a substantial amount of mathemat­ical theory, which will be developed in the book. As a rule, elementary mathematical arguments are preferred. In topics like interpolation or in­tegration we deliberately restrict ourselves to the one-dimensional case. Wherever meaningful, the reasoning appeals to geometric intuition-which also explains the quite large number of graphical representations. Notions like scalar product and orthogonality are used throughout-in the finite dimensional case as well as in infinite dimensions (functions). Despite the elementary presentation, the book contains a significant number of other­wise unpublished material. Some of our derivations of classical results differ significantly from traditional derivations-in many cases they are simpler and nevertheless more stringent. As an example we refer to our condition and error analysis, which requires only multidimensional differentiation as the main analytical prerequisite.

Compared to the first English edition, a polishing of the book as a whole has been performed. The essential new item is Section 5.5 on stochastic eigenvalue problems-a problem class that has gained increasing impor­tance and appeared to be well-suited for an elementary presentation within our conceptual frame. As a recent follow-up, there exists an advanced textbook on numerical ordinary differential equations [22].

Preface ix

Of course, any selection of material expresses the scientific taste of the authors. The first author founded the Zuse Institute Berlin (ZIB) as a re­search institute for Scientific Computing in 1986. He has given Numerical Analysis courses at the Technical University of Munich and the Univer­sity of Heidelberg, and is now teaching at the Free University of Berlin. Needless to say, he has presented his research results in numerous invited talks at international conferences and seminars at renowned universities and industry places all over the world. The second author originally got his mathematical training in pure mathematics and switched over to compu­tational mathematics later. He is presently working in the communication industry. We are confident that the combination of a senior and a junior author, of a pure and an applied mathematician, as well as a member of academia and a representative from industry has had a stimulating effect on our presentation.

At this point it is our pleasure to thank all those who have particularly helped us with the preparation of this book. The first author remembers with gratitude his early time as an assistant of Roland Bulirsch (Technical University of Munich, retired since 2001), in whose tradition his present views on Scientific Computing have been shaped. Of course, our book has significantly profited from intensive discussions with numerous colleagues, some of which we want to mention explicitly here: Ernst Hairer and Ger­hard Wanner (University of Geneva) for discussions on the general concept of the book; Folkmar Bornemann (Technical University of Munich) for the formulation of the error analysis, the different condition number con­cepts, and the definition of the stability indicator in Chapter 2; Wolfgang Dahmen (RWTH Aachen) for Chapter 7; and Dietrich Braess (Ruhr Uni­versity Bochum) for the recursive derivation of the Fast Fourier Transform in Section 7.2.

The first edition of this textbook, which already contained the bulk of material presented in this text, was translated by Florian Potra and Fried­mar Schulz-again many thanks to them. For this, the second edition, we cordially thank Rainer Roitzsch (ZIB), without whose deep knowledge about a rich variety of fiddly TEX questions this book could never have appeared. Our final thanks go to Erlinda Kornig and Sigrid Wacker for all kinds of assistance.

Berlin and Dusseldorf, March 2002

Peter Deufihard and Andreas Hohmann

Outline

This introductory textbook is, in the first place, addressed to students of mathematics, computer science, science, and engineering. In the second place, it is also addressed to computational scientists already on the job who wish to get acquainted with modern concepts of Numerical Analysis and Scientific Computing on an elementary level via personal studies.

The book is divided into nine chapters, including associated exercises, a software list, a reference list, and an index. The contents of the first five and of the last four chapters are each closely related.

In Chapter 1 we begin with Gaussian elimination for linear systems of equations as the classical prototype of an algorithm. Beyond the ele­mentary elimination technique we discuss pivoting strategies and iterative refinement as additional issues. Chapter 2 contains the indispensable error analysis based on the fundamental ideas of J. H. Wilkinson. The condition of a problem and the stability of an algorithm are presented in a unified framework, well separated and illustrated by simple examples. The quite unpopular "E-battle" in linearized error analysis is avoided~which leads to a drastic simplification of the presentation and to an improved under­standing. A stability indicator arises naturally, which allows a compact classification of numerical stability. On this basis we derive an algorithmic criterion to determine whether a given approximate solution of a linear system of equations is acceptable or not. In Chapter 3 we treat orthogo­nalization methods in the context of Gaussian linear least-squares problems and introduce the extremely useful calculus of pseudo-inverses. It is imme­diately applied in the following Chapter 4, where we present iterative

xu Outline

methods for systems of nonlinear equations (Newton method), nonlinear least-squares problems (Gauss-Newton method), and parameter-dependent problems (continuation methods) in close mutual connection. Special at­tention is paid to modern affine invariant convergence theory and iterative algorithms. Chapter 5 starts with a condition analysis of linear eigen­value problems for general matrices. From this analysis, interest is naturally drawn to the real symmetric case, for which we present the power method (direct and inverse) and the QR-algorithm in some detail. Into the same context fits the singular value decomposition for general matrices, which is of utmost importance in application problems. As an add-on in this second edition, we finally consider stochastic eigenvalue problems, which in recent years have played an increasing role, especially in cluster analysis.

The second closely related chapter sequence begins in Chapter 6 with an extensive theoretical treatment of three-term recurrences, which play a key role in the realization of orthogonal projections in function spaces. The condition of three-term recurrences is represented in terms of discrete Green's functions-thus paving the way toward mathematical structures in initial and boundary value problems for differential equations. The signif­icant recent spread of symbolic computing has renewed interest in special functions also within Numerical Analysis. Numerical algorithms for their fast summation via the corresponding three-term recurrences are exem­plified for spherical harmonics and for Bessel functions. In Chapter 7 we start with classical polynomial interpolation and approximation in the one­dimensional case. We then continue over Bezier techniques and splines up to methods that nowadays are of central importance in CAD (Computer­Aided Design) or CAGD (Computer-Aided Geometric Design), disciplines of computer graphics. Our presentation in Chapter 8 on iterative methods for the solution of large symmetric systems of linear equations benefits con­veniently from Chapter 6 (three-term recurrences) and Chapter 7 (minimax property of Chebyshev polynomials). The same is true for our treatment of the Lanczos algorithm for large symmetric eigenvalue problems.

Finally, Chapter 9 has deliberately gotten somewhat longer: it bears the main burden of presenting principles of the numerical solution of or­dinary and partial differential equations without any technicalities at the simplest possible problem type, which here is numerical quadrature. We start with the historical Newton-Cotes and Gauss-Christoffel quadrature. As a first adaptive algorithm, we introduce the classical Romberg quadra­ture, wherein, however, only the approximation order can be varied. The formulation of the quadrature problem as an initial value problem offers the opportunity to work out an adaptive Romberg algorithm with variable or­der and step-size control; this approach opens the possibility to discuss the principle of extrapolation methods, which playa key role in the numerical solution of ordinary differential equations. The alternative formulation of the quadrature problem as a boundary value problem is used for the deriva-

Outline xiii

tion of an adaptive multigrid quadrature; in this way we can deal with the adaptivity principle behind multigrid methods for partial differential equa­tions in isolated form-clearly separated from the principle of fast solution, which is often predominant in the context of partial differential equations.

Contents

Preface vii

Outline xi

1 Linear Systems 1 1.1 Solution of Triangular Systems. . . . . . . . 3 1.2 Gaussian Elimination . . . . . . . . . . . . . 4 1.3 Pivoting Strategies and Iterative Refinement 7 1.4 Cholesky Decomposition for Symmetric Positive Definite

Matrices 14 Exercises .... 16

2 Error Analysis 21 2.1 Sources of Errors . . . . . . . . . . . 22 2.2 Condition of Problems . . . . . . . . 24

2.2.1 Normwise Condition Analysis 26 2.2.2 Componentwise Condition Analysis 31

2.3 Stability of Algorithms . . 34 2.3.1 Stability Concepts 35 2.3.2 Forward Analysis . 37 2.3.3 Backward Analysis 42

2.4 Application to Linear Systems 44

XVI Contents

2.4.1 2.4.2 2.4.3

Exercises .

A Zoom into Solvability ........ . . . Backward Analysis of Gaussian Elimination Assessment of Approximate Solutions.

3 Linear Least-Squares Problems 3.1 Least-Squares Method of Gauss

3.1.1 Formulation of the Problem 3.1.2 Normal Equations ..... . 3.1.3 Condition ......... . 3.1.4 Solution of Normal Equations

3.2 Orthogonalization Methods .. . 3.2.1 Givens Rotations ... . 3.2.2 Householder Reflections

3.3 Generalized Inverses.

44 46 49 52

57 57 57 60 62 65 66 68 70 74

Exercises ................ 78

4 Nonlinear Systems and Least-Squares Problems 81 4.1 Fixed-Point Iterations. . . . . . . . . . . . . . . . 81 4.2 Newton Methods for Nonlinear Systems. . . . . . 86 4.3 Gauss-Newton Method for Nonlinear Least-Squares Prob-

lems ....................... 92 4.4 Nonlinear Systems Depending on Parameters.

4.4.1 Solution Structure 4.4.2

Exercises . Continuation Methods

99 100 102 113

5 Linear Eigenvalue Problems 119 5.1 Condition of General Eigenvalue Problems . . . . . 120 5.2 Power Method . . . . . . . . . . . . . . . . . . . . . 123 5.3 QR-Algorithm for Symmetric Eigenvalue Problems 126 5.4 Singular Value Decomposition . 132 5.5 Stochastic Eigenvalue Problems 137 Exercises ................ 148

6 Three-Term Recurrence Relations 6.1 Theoretical Background .....

6.1.1 Orthogonality and Three-Term Recurrence Rela-

151 153

tions . . . . . . . . . . . . . . . . . . . . . . 153 6.1.2 Homogeneous and Inhomogeneous Recurrence Re-

lations . . . . . . . 156 6.2 Numerical Aspects

6.2.1 Condition Number 6.2.2 Idea of the Miller Algorithm

6.3 Adjoint Summation . . . . .....

158 160 166 168

6.3.1 Summation of Dominant Solutions 6.3.2 Summation of Minimal Solutions

Exercises

Contents XVII

169 172 176

7 Interpolation and Approximation 179 7.1 Classical Polynomial Interpolation. . . . . . . . . . . 180

7.1.1 Uniqueness and Condition Number . . . . . . 180 7.1.2 Hermite Interpolation and Divided Differences 184 7.1.3 Approximation Error . . . . . . . . . . . . . . 192 7.1.4 Min-Max Property of Chebyshev Polynomials 193

7.2 Trigonometric Interpolation . . . . . . . . . . . . . . 197 7.3 Bezier Techniques . . . . . . . . . . . . . . . . . . . . 204

7.3.1 Bernstein Polynomials and Bezier Representation 205 7.3.2 De Casteljau Algorithm .. 211

7.4 Splines................ 218 7.4.1 Spline Spaces and B-Splines 219 7.4.2 Spline Interpolation. . . . . 226 7.4.3 Computation of Cubic Splines 230

Exercises . 233

8 Large Symmetric Systems of Equations and Eigenvalue Problems 237 8.1 Classical Iteration Methods 239 8.2 Chebyshev Acceleration ... 8.3 Method of Conjugate Gradients 8.4 Preconditioning. 8.5 Lanczos Methods Exercises ..... .

9 Definite Integrals

244 249 256 261 266

269 9.1 Quadrature Formulas. . . . . 270 9.2 Newton-Cotes Formulas. . . . 273 9.3 Gauss-Christoffel Quadrature 279

9.3.1 Construction of the Quadrature Formula 280 9.3.2 Computation of Nodes and Weights. . . 285

9.4 Classical Romberg Quadrature. . . . . . . . . . 287 9.4.1 Asymptotic Expansion of the Trapezoidal Sum 288 9.4.2 Idea of Extrapolation . . 290 9.4.3 Details of the Algorithm 295

9.5 Adaptive Romberg Quadrature 298 9.5.1 Principle of Adaptivity . 299 9.5.2 Estimation of the Approximation Error. 301 9.5.3 Derivation of the Algorithm 304

9.6 Hard Integration Problems . . . 310 9.7 Adaptive Multigrid Quadrature . . 313

xviii Contents

9.7.1 9.7.2

Exercises

References

Software

Index

Local Error Estimation and Refinement Rules . . Global Error Estimation and Details of the Algo­rithm .

314

318 321

325

331

333