Matrix Computations (3rd Ed.)CopyrightContentsPreface to the 3rd Ed.SoftwareSelected ReferencesCh1 Matrix Multiplication Problems1.1 Basic Algorithms & Notation1.1.1 Matrix Notation1.1.2 Matrix Operations1.1.3 Vector Notation1.1.4 Vector Operations1.1.5 Computation of Dot Products & Saxpys1.1.6 Matrix-Vector Multiplication & Gaxpy1.1.7 Partitioning Matrix into Rows & Columns1.1.8 Colon Notation1.1.9 Outer Product Update1.1.10 Matrix-Matrix Multiplication1.1.11 Scalar-Level Specifications1.1.12 Dot Product Formulation1.1.13 Saxpy Formulation1.1.14 Outer Product Formulation1.1.15 Notion of "Level"1.1.16 Note on Matrix Equations1.1.17 CompIex MatricesProblems

1.2 Exploiting Structure1.2.1 Band Matrices & x-0 Notation1.2.2 Diagonal Matrix Manipulation1.2.3 Triangular Matrix Multiplication1.2.4 Flops1.2.5 Colon Notation: Again1.2.6 Band Storage1.2.7 Symmetry1.2.8 Store by Diagonal1.2.9 Note on Overwriting & WorkspacesProblems

1.3 Block Matrices & Algorithms1.3.1 Block Matrix Notation1.3.2 Block Matrix Manipulation1.3.3 Submatrix Designation1.3.4 Block Matrix Times Vector1.3.5 Block Matrix Multiplication1.3.6 Complex Matrix Multiplication1.3.7 Divide & Conquer Matrix MultiplicationProblems

1.4 Vectorization & Re-Use Issues1.4.1 Pipelining Arithmetic Operations1.4.2 Vector Operations1.4.3 Vector Length Issue1.4.4 Stride Issue1.4.5 Thinking about Data Motion1.4.6 Vector Touch Issue1.4.7 Blocking & Re-Use1.4.8 Block Matrix Data StructuresProblems

Ch2 Matrix Analysis2.1 Basic Ideas from Linear Algebra2.1.1 Independence, Subspace, Basis & Dimension2.1.2 Range, Null Space & Rank2.1.3 Matrix Inverse2.1.4 Determinant2.1.5 DifferentiationProblems

2.2 Vector Norms2.2.1 Definitions2.2.2 Some Vector Norm Properties2.2.3 Absolute & Relative Error2.2.4 ConvergenceProblems

2.3 Matrix Norms2.3.1 Definitions2.3.2 Some Matrix Norm Properties2.3.3 Matrix 2-Norm2.3.4 Perturbations & InverseProblems

2.4 Finite Precision Matrix Computations2.4.1 Floating Point Numbers2.4.2 Model of Floating Point Arithmetic2.4.3 Cancellation2.4.4 Absolute Value Notation2.4.5 Rouadoff in Dot Products2.4.6 Alternative Ways to Quantify Roundoff Error2.4.7 Dot Product Accumulation2.4.8 Rowzdoff in other Basic Matrix Computations2.4.9 Forward & Backward Error Analyses2.4.10 Error in Strassen MultiplicationProblems

2.5 Orthogonality & SVD2.5.1 Orthogonality2.5.2 Norms & Orthogonal Transformations2.5.3 Singular Value Decomposition2.5.4 Thin SVD2.5.5 Rank Deficiency & SVD2.5.6 Unitary MatricesProblems

2.6 Projections & CS Decomposition2.6.1 Orthogonal Projections2.6.2 SVD-Related Projections2.6.3 Distance between Suhpaces2.6.4 CS DecompositionProblems

2.7 Sensitivity of Square Systems2.7.1 SVD Analysis2.7.2 Condition2.7.3 Determinants & Nearness to Singularity2.7.4 Rigorous Norm Bound2.7.5 Some Rigorous Componentwise BoundsProblems

Ch3 General Linear Systems3.1 Triangular Systems3.1.1 Forward Substitution3.1.2 Back Substitution3.1.3 Column Oriented Versions3.1.4 Multiple Right Hand Sides3.1.5 Level-3 Fraction3.1.6 Non-Square Triangular System Solving3.1.7 Unit Triangular Systems3.1.8 Algebra of Triangular MatricesProblems

3.2 LU Factorization3.2.1 Gauss Transformations3.2.2 Applying Gauss Transformations3.2.3 Roundoff Properties of Gauss Transforms3.2.4 Upper Triangularizing3.2.5 LU Factorization3.2.6 Some Practical Details3.2.7 Where is L?3.2.8 Solving Linear System3.2.9 Other Versions3.2.10 Block LU3.2.11 LU Factorization of Rectangular Matrix3.2.12 Note on FailureProblems

3.3 Roundoff Analysis of Gaussian Elimination3.3.1 Errors in LU Factorization3.3.2 Triangular Solving with Inexact TrianglesProblems

3.4 Pivoting3.4.1 Permutation Matrices3.4.2 Partial Pivoting: Basic Idea3.4.3 Partial Pivoting Details3.4.4 Where is L?3.4.5 Gaxpy Version3.4.6 Error Analysis3.4.7 Block Gaussian Elimination3.4.8 Complete Pivoting3.4.9 Comments on Complete Pivoting3.4.10 Avoidance of Pivoting3.4.11 Some ApplicationsProblems

3.5 Improving & Estimating Accuracy3.5.1 Residual Size vs Accuracy3.5.2 Scaling3.5.3 Iterative Improvement3.5.4 Condition EstimationProblems

Ch4 Special Linear Systems4.1 LDM & LDL Factorizations4.1.1 LDM Factorization4.1.2 Symmetry & LDL FactorizationProblems

4.2 Positive Definite Systems4.2.1 Positive Definiteness4.2.2 Unsymmetric Positive Definite Systems4.2.3 Symmetric Positive Definite Systems4.2.4 Gaxpy Cholesky4.2.5 Outer Product Cholesky4.2.6 Block Dot Product Cholesky4.2.7 Stability of Cholesky Process4.2.8 Semidefinite Case4.2.9 Symmetric Pivoting4.2.10 Polar Decomposition & Square RootProblems

4.3 Banded Systems4.3.1 Band LU Factorization4.3.2 Band Triangular System Solving4.3.3 Band Gaussian Elimination with Pivoting4.3.4 Hessenberg LU4.3.5 Band Cblesky4.3.6 Tkidiagonal System Solving4.3.7 Vectorization Issues4.3.8 Band Matrix Data StructuresProblems

4.4 Symmetric Indefinite Systems4.4.1 Parlett-Reid Algorithms4.4.2 Method of Aasen4.4.3 Pivoting in Aasen's Method4.4.4 Diagonal Pivoting Methods4.4.5 Stability & Efficiency4.4.6 Note on Equilibrium SystemsProblems

4.5 Block Systems4.5.1 Block Tridiagonal LU Factorization4.5.2 BIock Diagonal Dominance4.5.3 Block vs Band Solving4.5.4 Block Cyclic Reduction4.5.5 Kronecker Product SystemsProblems

4.6 Vandermonde Systems & FFT4.6.1 Polynomial Interpolation: V'a = f4.6.2 System Vz = b4.6.3 Stability4.6.4 Fast Fourier TransformProblems

4.7 Toeplitz & Related Systems4.7.1 Three Problems4.7.2 Solving Yule-Walker Equations4.7.3 General Right Hand Side Problem4.7.4 Computing the Inverse4.7.5 Stability Issues4.7.6 Unsymmetric Case4.7.7 Circulant SystemsProblems

Ch5 Orthogonalization & Least Squares5.1 Householder & Givens Matrices5.1.1 2-by-2 Preview5.1.2 Householder Reflections5.1.3 Computing Householder Vector5.1.4 Applying Householder Matrices5.1.5 Roundoff Properties5.1.6 Factored Form Representation5.1.7 Block Representation5.1.8 Givens Rotations5.1.9 Applying Givens Rotations5.1.10 Roundoff Properties5.1.11 Representing Products of Givens Rotations5.1.12 Error Propagation5.1.13 Fast Givens TransformationsProblems

5.2 QR Factorization5.2.1 Householder QR5.2.2 Block Householder QR Factorization5.2.3 Givens QR Methods5.2.4 Hessenberg QR via Givens5.2.5 Fast Givens QR5.2.6 Properties of QR Factorization5.2.7 Classical Gram-Schmidt5.2.8 Modified Gram-Schmidt5.2.9 Work & Accuracy5.2.10 Note on Complex QRProblems

5.3 Full Rank LS Problem5.3.1 Implications of Full Rank5.3.2 Method of Normal Equations5.3.3 LS Solution via QR Factorization5.3.4 Breakdown in Near-Rank Deficient Case5.3.5 Note on MGS Approach5.3.6 Fast Givens LS Solver5.3.7 Sensitivity of LS Problem5.3.8 Normal Equations vs QRProblems

5.4 Other Orthogonal Factorizations5.4.1 Rank Deficiency: QR with Column Pivoting5.4.2 Complete Orthogonal Decompositions5.4.3 Bidiagonalization5.4.4 R-Bidiagonalization5.4.5 SVD & its ComputationProblems

5.5 Rank Deficient LS Problem5.5.1 Minimum Norm Solution5.5.2 Complete Orthogonai Factorization & XLS5.5.3 SVD & LS Problem5.5.4 Pseudo-Inverse5.5.5 Some Sensitivity Issues5.5.6 QR with Column Pivoting & Basic Solutions5.5.7 Numerical Rank Determination with AII = QR5.5.8 Numerical Rank & SVD5.5.9 Some ComparisonsProblems

5.6 Weighting & Iterative Improvement5.6.1 CoIumn Weighting5.6.2 Row Weighting5.6.3 GeneraIized Least Squares5.6.4 Iterative ImprovementProblems

5.7 Square & Underdetermined Systems5.7.1 Using QR & SVD to Solve Square Systems5.7.2 Underdetermined Systems5.7.3 Perturbed Underdetermined SystemsProblems

Ch6 Parallel Matrix Computations6.1 Basic Concepts6.1.1 Distributed Memory System6.1.2 Communication6.1.3 Some Distributed Data Structures6.1.4 Gaxpy on Ring6.1.5 Cost of Communication6.1.6 Efficiency & Speed-Up6.1.7 Challenge of Load Balancing6.1.8 Tradeoffs6.1.9 Shared Memory Systems6.1.10 Shared Memory Gaxpy6.1.11 Memory Traffic Overhead6.1.12 Barrier Synchronization6.1.13 Dynamic SchedulingProblems

6.2 Matrix Multiplication6.2.1 Block Gaxpy Procedure6.2.2 TorusProblems

6.3 Factorizations6.3.1 Ring Cholesky6.3.2 Shared Memory CholeskyProblems

Ch7 Unsymmetric Eigenvalue Problem7.1 Properties & Decompositions7.1.1 Eigenvalues & Invariant Subspaces7.1.2 Decoupling7.1.3 Basic Unitary Decompositions7.1.4 Nonunitary Reductions7.1.5 Some Comments on Nonunitary Similarity7.1.6 Singular Values & EigenvaluesProblems

7.2 Perturbation Theory7.2.1 Eigenvalue Sensitivity7.2.2 Condition of Simple Eigenvalue7.2.3 Sensitivity of Repeated Eigenvalues7.2.4 Invariant Subspace Sensitivity7.2.5 Eigenvector SensitivityProblems

7.3 Power Iterations7.3.1 Power Method7.3.2 Orthogonal Iteration7.3.3 QR Iteration7.3.4 LR IterationsAppendixProblems

7.4 Hessenberg & Real Schur Forms7.4.1 Real Schur Decomposition7.4.2 Hessenberg QR Step7.4.3 Hessenberg Reduction7.4.4 Level-3 Aspects7.4.5 Important Hessenberg Matrix Properties7.4.6 Companion Matrix Form7.4.7 Hessenberg Reduction via Gauss TransformsProblems

7.5 Practical QR Algorithm7.5.1 Deflation7.5.2 Shifted QR Iteration7.5.3 Single Shift Strategy7.5.4 Double Shift Strategy7.5.5 Double Implicit Shift Strategy7.5.6 Overall Process7.5.7 BalancingProblems

7.6 Invariant Subspace Computations7.6.1 Selected Eigenvectors via Inverse Iteration7.6.2 Ordering Eigenvalues in Real Schur Form7.6.3 Block Diagonalization7.6.4 Eigenvector Bases7.6.5 Ascertaining Jordan Block StructuresProblems

7.7 QZ Method for Ax = lambdaBx7.7.1 Background7.7.2 Generalized Schur Decomposition7.7.3 Sensitivity Issues7.7.4 Hessenberg-Triangular Form7.7.5 Deflation7.7.6 QZ Step7.7.7 Overall QZ Process7.7.8 Generalized Invariant Subspace ComputationsProblems

Ch8 Symmetric Eigenvalue Problem8.1 Properties & Decompositions8.1.1 Eigenvalues & Eigenvectors8.1.2 Eigenvalue Sensitivity8.1.3 Invariant Subspaces8.1.4 Approximate Invariant Subspaces8.1.5 Law of InertiaProblems

8.2 Power Iterations8.2.1 Power Method8.2.2 Inverse Iteration8.2.3 Rayleigh Quotient Iteration8.2.4 Orthogonal Iteration8.2.5 QR IterationProblems

8.3 Symmetric QR Algorithm8.3.1 Reduction to Tridiagonal Form8.3.2 Properties of Tridiagonal Decomposition8.3.3 QR Iteration & Tridiagonal Matrices8.3.4 Explicit Single Shift QR Iteration8.3.5 Implicit Shift Version8.3.6 Orthogonal Iteration with Ritz AccelerationProblems

8.4 Jacobi Methods8.4.1 Jacobi Idea8.4.2 2-by-2 Symmetric Schur Decomposition8.4.3 Classical Jacobi Algorithm8.4.4 Cyclic-by-Row Algorithm8.4.5 Error Analysis8.4.6 Parallel Jacobi8.4.7 Ring Procedure8.4.8 Block Jacobi ProceduresProblems

8.5 Tridiagonal Methods8.5.1 Eigenvalues by Bisection8.5.2 Sturm Sequence Methods8.5.3 Eigensystems of Diagonal Plus Rank-1 Matrices8.5.4 Divide & Conquer Method8.5.5 Parallel ImplementationProblems

8.6 Computing SVD8.6.1 Perturbation Theory & Properties8.6.2 SVD Algorithm8.6.3 Jacobi SVD ProceduresProblems

8.7 Some Generalized Eigenvalue Problems8.7.1 Mathematical Background8.7.2 Methods for Symmetric-Definite Problem8.7.3 Generalized Singular Value ProblemProblems

Ch9 Lanczos Methods9.1 Derivation & Convergence Properties9.1.1 Krylov Subspaces9.1.2 Tridiagonalization9.1.3 Termination & Error Bounds9.1.4 Kaniel-Paige Convergence Theory9.1.5 Power Method vs Lanczos Method9.1.6 Convergence of Interior EigenvaluesProblems

9.2 Practical Lanczos Procedures9.2.1 Exact Arithmetic Implementation9.2.2 Roundoff Properties9.2.3 Lanczos with Complete Reorthogonalization9.2.4 Selective Orthogonalization9.2.5 Ghost Eigenvalue Problem9.2.6 Block Lanczos9.2.7 s-Step LanczosProblems

9.3 Applications to Ax = b & Least Squares9.3.1 Symmetric Positive Definite Systems9.3.2 Symmetric Indefinite Systems9.3.3 Bidiagonalization & SVD9.3.4 Least SquaresProblems

9.4 Arnoldi & Unsymmetric Lanczos9.4.1 Basic Arnoldi Iteration9.4.2 Arnoldi with Restarting9.4.3 Unsymmetric Lanczos Tridiagonalization9.4.4 Look-Ahead IdeaProblems

Ch10 Iterative Methods for Linear Systems10.1 Standard Iterations10.1.1 Jacobi & Gauss-Seidel Iterations10.1.2 Splittings & Convergence10.1.3 Practical Implementation of Gauss-Seidel10.1.4 Successive Over-Relaxation10.1.5 Chebyshev Semi-Iterative Method10.1.6 Symmetric SORProblems

10.2 Conjugate Gradient Method10.2.1 Steepest Descent10.2.2 General Search Directions10.2.3 A-Conjugate Search Directions10.2.4 Choosing Best Search Direction10.2.5 Lanczos Connection10.2.6 Some Practical Details10.2.7 Convergence PropertiesProblems

10.3 Preconditioned Conjugate Gradients10.3.1 Derivation10.3.2 Incomplete Cholesky Preconditioners10.3.3 Incomplete Block Preconditioners10.3.4 Domain Decomposition Ideas10.3.5 Polynomial Preconditioners10.3.6 Another PerspectiveProblems

10.4 Other Krylov Subspace Methods10.4.1 Normal Equation Approaches10.4.2 Note on Objective Functions10.4.3 Conjugate Residual Method10.4.4 GMRES10.4.5 Preconditioning10.4.6 Biconjugate Gradient Method10.4.7 QMR10.4.8 SummaryProblems

Ch11 Functions of Matrices11.1 Eigenvalue Met hods11.1.1 Definition11.1.2 Jordan Characterization11.1.3 Schur Decomposition Approach11.1.4 Block Schur ApproachProblems

11.2 Approximation Methods11.2.1 Jordan Analysis11.2.2 Schur Analysis11.2.3 Taylor Approximants11.2.4 Evaluating Matrix Polynomials11.2.5 Computing Powers of Matrix11.2.6 Integrating Matrix FunctionsProblems

11.3 Matrix Exponential11.3.1 Pade Approximation Method11.3.2 Perturbation Theory11.3.3 Some Stability Issues11.3.4 Eigenvalues & Pseudo-EigenvaluesProblems

Ch12 Special Topics12.1 Constrained Least Squares12.1.1 Problem LSQI12.1.2 LS Minimization over Sphere12.1.3 Ridge Regression12.1.4 Equality Constrained Least Squares12.1.5 Method of WeightingProblems

12.2 Subset Selection using SVD12.2.1 QR with Column Pivoting12.2.2 Using SVD12.2.3 More on Column Independence vs ResidualProblems

12.3 Total Least Squares12.3.1 Mathematical Background12.3.2 Computations for k = 1 Case12.3.3 Geometric InterpretationProblems

12.4 Computing Subspaces with SVD12.4.1 Rotation of Subspaces12.4.2 Intersection of Null Spaces12.4.3 Angles between Subspaces12.4.4 Intersection of SubspacesProblems

12.5 Updating Matrix Factorizations12.5.1 Rank-One Changes12.5.2 Appending or Deleting Column12.5.3 Appending or Deleting Row12.5.4 Hyperbolic Transformation Methods12.5.5 Updating ULV DecompositionProblems

12.6 Modified/Structured Eigenproblems12.6.1 Constrained Eigenvalue Problem12.6.2 Two Inverse Eigenvalue Problems12.6.3 Toeplitz Eigenproblem12.6.4 Orthogonal Matrix EigenproblemProblems

BibliographyIndex