A First Course in Applied Mathematics

A First Course in Applied Mathematics

Jorge Rebaza Department of Mathematics

Missouri State University Springfield, MO

WILEY A JOHN WILEY & SONS, INC., PUBLICATION

Copyright © 2012 by John Wiley & Sons, Inc. All rights reserved.

Published by John Wiley & Sons, Inc., Hoboken, New Jersey. Published simultaneously in Canada.

No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 750-4470, or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permission.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representation or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.

For general information on our other products and services please contact our Customer Care Department within the United States at (800) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002.

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print, however, may not be available in electronic formats. For more information about Wiley products, visit our web site at www.wiley.com.

Library of Congress Cataloging-in-Publication Data:

Rebaza, Jorge. A first course in applied mathematics / Jorge Rebaza.

p. cm. Includes bibliographical references and index.

ISBN 978-1-118-22962-0 1. Mathematical models. 2. Computer simulation. I. Title. TA342.R43 2012 510—dc23 2011043340

Printed in the United States of America.

10 9 8 7 6 5 4 3 2 1

http://www.copyright.comhttp://www.wiley.com/go/permissionhttp://www.wiley.com

To my parents:

Leoncio Rebaza Minano and

Santos Vasquez Paredes

CONTENTS

Preface xiii

1 Basics of Linear Algebra 1 1.1 Notation and Terminology 1 1.2 Vector and Matrix Norms 4 1.3 Dot Product and Orthogonality 8 1.4 Special Matrices 9

1.4.1 Diagonal and triangular matrices 9 1.4.2 Hessenberg matrices 10 1.4.3 Nonsingular and inverse matrices 11 1.4.4 Symmetric and positive definite matrices 12 1.4.5 Matrix exponential 14 1.4.6 Permutation matrices 15 1.4.7 Orthogonal matrices 17

1.5 Vector Spaces 21 1.6 Linear Independence and Basis 24 1.7 Orthogonalization and Direct Sums 31 1.8 Column Space, Row Space, and Null Space 34

1.8.1 Linear transformations 40 1.9 Orthogonal Projections 43 1.10 Eigenvalues and Eigenvectors 47

vii

Viii CONTENTS

1.11 Similarity 56 1.12 Bezier Curves and Postscript Fonts 59

1.12.1 Properties of Bezier curves 61 1.12.2 Composite Bezier curves 66

1.13 Final Remarks and Further Reading 68 Exercises 69

Ranking Web Pages 79 2.1 The Power Method 80 2.2 Stochastic, Irreducible, and Primitive Matrices 84 2.3 Google's PageRank Algorithm 92

2.3.1 The personalization vector 99 2.3.2 Speed of convergence and sparsity 100 2.3.3 Power method and reordering 105

2.4 Alternatives to the Power Method 106 2.4.1 Linear system formulation 107 2.4.2 Iterative aggregation/disaggregation (IAD) 111 2.4.3 IAD and linear systems 117

2.5 Final Remarks and Further Reading 120 Exercises 121

Matrix Factorizations 131 3.1 LU Factorization 132

3.1.1 The complex case 137 3.1.2 Solving several systems 137 3.1.3 The PA = LU factorization 139

3.2 QR Factorization 142 3.2.1 QR and Gram-Schmidt 143 3.2.2 The complex case 147 3.2.3 QR and similarity 148 3.2.4 The QR algorithm 149 3.2.5 QR and LU 151

3.3 Singular Value Decomposition (SVD) 155 3.3.1 The complex case 160 3.3.2 Low-rank approximations 161 3.3.3 SVD and spectral norm 164

3.4 Schur Factorization 166 3.4.1 The complex case 171 3.4.2 Schur factorization and invariant subspaces 172 3.4.3 Exchanging eigenblocks 177 3.4.4 Block diagonalization 180

CONTENTS JX

3.5 Information Retrieval 186 3.5.1 Query matching 187 3.5.2 Low-rank query matching 190 3.5.3 Term-term comparison 192

3.6 Partition of Simple Substitution Cryptograms 194 3.6.1 Rank-1 approximation 197 3.6.2 Rank-2 approximation 199

3.7 Final Remarks and Further Reading 203 Exercises 205

Least Squares 215 4.1 Projections and Normal Equations 215 4.2 Least Squares and QR Factorization 224 4.3 Lagrange Multipliers 228 4.4 Final Remarks and Further Reading 231

Exercises 231

Image Compression 235 5.1 Compressing with Discrete Cosine Transform 236

5.1.1 1 -D discrete cosine transform 236 5.1.2 2-D discrete cosine transform 242 5.1.3 Image compression and the human visual system 245 5.1.4 Basis functions and images 247 5.1.5 Low-pass filtering 250 5.1.6 Quantization 254 5.1.7 Compression of color images 257

5.2 Huffman Coding 260 5.2.1 Huffman coding and JPEG 262

5.3 Compression with SVD 267 5.3.1 Compressing grayscale images 268 5.3.2 Compressing color images 268

5.4 Final Remarks and Further Reading 269 Exercises 271

Ordinary Differential Equations 277 6.1 One-Dimensional Differential Equations 278

6.1.1 Existence and uniqueness 278 6.1.2 A simple population model 284 6.1.3 Emigration 285 6.1.4 Time-varying emigration 285 6.1.5 Competition 286

X CONTENTS

6.1.6 Spring systems 287 6.1.7 Undamped equations 293 6.1.8 Damped equations 299 6.1.9 RLC circuits 303

6.2 Linear Systems of Differential Equations 307 6.3 Solutions via Eigenvalues and Eigenvectors 307

6.3.1 Chains of generalized eigenvectors 311 6.4 Fundamental Matrix Solution 312

6.4.1 Nonhomogeneous systems 314 6.5 Final Remarks and Further Reading 316

Exercises 316

7 Dynamical Systems 325

7.1 Linear Dynamical Systems 326 327 335 337

7.2 Nonlinear Dynamical Systems 340 342

348

352

360

365

7.3 Predator-prey Models with Harvesting 374 376

376

379

380

382

7.4 Final Remarks and Further Reading 385 385

8 Mathematical Models 395

8.1 Optimization of a Waste Management System 396 8.1.1 Background 396 8.1.2 Description of the system 397 8.1.3 Development of the mathematical model 398 8.1.4 Building the objective function 399 8.1.5 Building the constraints 400 8.1.6 Numerical experiments 400

8.2 Grouping Problem in Networks 404

7.1.1 7.1.2 7.1.3

Dynamics in two dimensions Trace-determinant analysis Stable, unstable, and center subspaces

Nonlinear Dynamical Systems 7.2.1 7.2.2 7.2.3 7.2.4 7.2.5

Linearization around an equilibrium point Linearization around a periodic orbit Connecting orbits Chaos Bifurcations

Predator-prey Models with Harvesting 7.3.1 7.3.2 7.3.3 7.3.4 7.3.5

Boundedness of solutions Equilibrium point analysis Bifurcations Connecting orbits Other models

Final Remarks and Further Reading

CONTENTS Xi

405 406 407 409

8.3 American Cutaneous Leishmaniasis 410 410

413

414

416

418

8.4 Variable Population Interactions 420 420 421 425

References 431

Index 435

8.2.1 8.2.2 8.2.3 8.2.4

Background The TV-median approach The probabilistic approach Numerical experiments

American Cutaneous Leishmaniasis 8.3.1 8.3.2 8.3.3 8.3.4 8.3.5 Variabl 8.4.1 8.4.2 8.4.3

Background Development of the mathematical model Equilibria and periodic orbits Stability properties Numerical computations

le Population Interactions Model formulation Local stability of equilibria Bifurcations

PREFACE

Going back in history, mathematics originated as a practical science, as a tool to facilitate administration of harvest, computation of the calendar, collection of taxes, and so on. But even i early Greek society, the study of mathematics had one main goal: the understanding of humankind's purpose in the universe according to a rational scheme. Thus developed a mathematics investigated more in the spirit of understanding rather than only of utility, and this has been a central and successful focus of mathematics since then.

The constant development of new and more sophisticated technologies, in particular, the very fast progress of software and hardware technology, has contributed to a clear change in how mathematics should be studied and taught nowadays, and in how old and new mathematical theories can now be effectively and efficiently applied to the solution of current real-world problems.

Not everybody agrees on what Applied Mathematics means and which subjects or topics it includes. Differential equations may be applied mathematics for some (as it applies notions e.g. from analysis and linear algebra), while for others it is just another subject of pure mathematics, of course with several potential applications. However, we can make a first attempt to list topics:

(Pure) Mathematics: Topology, Abstract Algebra, Analysis, Linear Algebra.

Applied Mathematics: Dynamical Systems, Matrix Computations, Optimization, Financial Mathematics, Numerical Methods.

xiii

XJV PREFACE

Figure 0.1 Applied Mathematics.

On the other hand, several people would argue that all of the subjects above are just mathematics, and not exactly applied mathematics. They would ask for the applications of dynamical systems or of numerical analysis; in other words, they would ask for applications of the applications, or "real-world applications".

Here is when the terms Industrial Mathematics and Mathematical Modeling would probably help. Given a real-world ("industrial") problem, e.g., describe the motion of a particle in certain fluid or matter, or rank all web pages in terms of importance, we first try to describe this problem in mathematical terms (a process called mathematical modeling), arriving at a mathematical model (e.g., a set of differential equations or an eigenvalue/eigenvector problem). This mathematical problem is then solved using math-ematical tools (e.g. numerical analysis and linear algebra). Finally, this mathematical solution is analyzed together with the original problem and if necessary, the modeling step is modified and the process is repeated to obtain a final solution. Figure 0.1 illustrates this idea.

All the structure and ramifications of every application would not be possible without a solid theory supporting it. Theory is the indispensable and fundamental basis for any application. But theory could be better understood when at the same time corresponding methods or algorithms are used to solve a real-world problem. We could limit ourselves to studying the interesting and challenging questions of existence and uniqueness of solutions, but this is sterile from the point of view of applied mathematics. The beauty and importance of a mathematical concept may be better understood and appreciated when it is applied to a determined problem. For instance, the concept of an eigenvector and the power method to compute a dominant eigenvector are clearly illustrated when we study the mathematics behind the ranking of web pages by search engines like Google.

Through sound theory, a long collection of examples, and numerical computations, in this book we make an attempt to cover all the stages shown in Figure 0.1, and to illustrate how applied mathematics involves each component of the process toward the goal of solving real-world problems. However, only some selected topics in applied mathematics will be

PREFACE XV

considered, mainly because of limitations of space and time. Thus, for the applications we have in mind, we need to review some mathematical concepts and techniques, espe-cially those in linear algebra, matrix analysis, and differential equations. Some classical definitions and results from analysis will also be discussed and used. Some applications (postscript fonts, information retrieval, etc.) are presented at the end of a chapter as an immediate application of the theory just covered, while those applications that are discussed in more detail (ranking web pages, compression, etc.) will be presented in chapters of their own.

This book is intended for advanced undergraduate and beginning graduate students in mathematics, computer science, engineering, and sciences in general, with an interest in mathematical modeling, computational methods, and the applications of mathematics. It contains a little more material that could be covered in one semester. The instructor decides how much time should be devoted to those chapters or sections dealing mainly with a review of some material. It will mostly depend on the background level of the students. Several chapters and sections are independent from the rest, and although different instructors can choose different chapters or sections to cover, here are some guidelines.

Chapters 1, 3, and 4 could be covered sequentially. Within Chapter 1, one could cover in detail, briefly, or even skip some of the first sections before covering the section on Bezier curves or before going to Chapter 2 or 3. The first two sections of Chapter 2 are the additional background needed for the web page ranking problem studied in that chapter. The first two sections of Chapter 3 may be discussed briefly to allow some extra time to study sections three and four; in some cases, section 4 may be skipped; although it will be needed in the second section of Chapter 7. Some material covered in Chapter 1, and especially in Chapter 3, is used in the study of Chapter 4.

Chapter 5 is for the most part independent from the previous chapters, except when viewed as an application of the concepts of linear combination and basis of vector spaces presented in Chapter 1.

The first two sections of Chapter 6 are a review of basic material on differential equations, and more time should be spent instead on sections three and four. This provides a good starting point for the next chapter. In Chapter 7, the first section should be covered relatively quickly as a background for section two on nonlinear systems, which contains a little more advanced material. Section three in this chapter applies the concepts studied in the first two sections, to a concrete problem.

The final chapter is a collection of some mathematical models of a slightly different nature; in particular, the first two sections deal with applications of some basic discrete mathematics and optimization, and for the most part is therefore independent from the previous chapters.

Depending on the background of the students, among other factors, here are a few sequences of sections that could be followed:

• Chapter 1, Chapter 2, 3.1-3.3, 3.5, 5.1,5.3, Chapter 6, 7.1,7.2, 8.1,8.2

• 1.4, 1.7- 1.9, 1.12, Chapter 2, 3.2-3.5, Chapter 4, Chapter 5, 6.3, 6.4, Chapter 7

• 1.8, 1.9, 1.12, 2.3,2.4, 3.3-3.5, Chapter 4, 7.2,7.3, Chapter 8

XVi PREFACE

Besides two or more semesters of calculus, we are assuming the student has taken some basic linear algebra and an introductory differential equations course. Most of this material is reviewed in this book, and those sections can be used as a reference when studying new or more advanced topics. We also take care to introduce results that are typically part of a graduate course. An effort has been made to organize the book so that the transition from undergraduate to graduate material is as smooth as possible, and to make this text self-contained. Again, the main goal is to introduce real-world applications where each concept or theory just learned is used.

This book developed from lectures notes for the Applied Mathematics course at Missouri State University, with a typical audience consisting of some juniors and mostly seniors from mathematics and computer science, as well as first-year graduate students. It has been the textbook for this course since fall 2007. Although the content and order of sections covered may vary from semester to semester, typical sections covered are:

• 1.2,1.4.7,1.8,1.12, 2.1-2.3,3.1-3.3,3.5, 5.1,5.3, 6.3,6.4, 7.1,7.2.1,7.2.2, 7.2.5.

Selected topics from other sections, including 2.4,3.4,3.6,5.2,7.3, and 8.1- 8.4 are usually assigned as group projects, and students are required to turn in a paper and give a seminar presentation on the one topic assigned. Different groups get different topics, and group discussions in class are part of the course.

We remark that in general, theory and examples in this book will be illustrated with the help of MATLAB software package. Previous knowledge of MATLAB or programming is not required.

CHAPTER 1

BASICS OF LINEAR ALGEBRA

Undoubtedly, one of the subjects in mathematics that has become more indispensable than ever is linear algebra. Several application problems involve at some stage solving linear systems, the computation of eigenvalues and eigenvectors, linear transformations, bases of vector subspaces, and matrix factorizations, to mention a few. One very important characteristic of linear algebra is that as a first course, it requires only very basic pre-requisites so that it can be taught very early at undergraduate level; at the same time, mastering vector spaces, linear transformations and their natural extensions to function spaces is essential for researchers in any area of applied mathematics. Linear algebra has innumerable applications, including differential equations, least-square solutions and opti-mization, demography, electrical engineering, fractal geometry, communication networks, compression, search engines, social sciences, etc. In the next sections we briefly review the concepts of linear algebra that we will need later on.

1.1 NOTATION AND TERMINOLOGY

We start this section by defining an m x n matrix as a rectangular array of elements arranged in m rows and n columns, and we say the matrix is of order m x n. We usually denote the elements of a matrix A of order m x n a s a^, where i = 1 , . . . m, j = 1 , . . . , n, and we

A First Course in Applied Mathematics. By Jorge Rebaza 1 Copyright © 2012 John Wiley & Sons, Inc.

2 BASICS OF LINEAR ALGEBRA

write the matrix A as

A =

an &\2 * ' ' aln &2\ 0*22 " ' 0,2n

Q"ml &m2 * ' * ^rrn

Although the elements of a matrix can be real or complex numbers, here we will mostly consider the entries of a given matrix to be real unless otherwise stated. In most cases, we will take care of stating the complex version of some definitions and results.

Matrix addition and multiplication. Given two arbitrary matrices Amxn and Bmxn, we define the matrix

C = A + B

by adding the entries of A and B componentwise. That is, we have

C{j := CLij ~r Oij j

fori = 1 , . . . ,ra, j = 1 , . . . , n. This means that the addition of matrices is well defined only for matrices of the same order.

Now consider two arbitrary matrices Amxp and Bpxn. Then we define the product matrix &mxn — A • B as

V

k=l

where i = 1 , . . . , m, j = 1 , . . . , n. This means that to obtain the entry (i, j) of the product matrix, we multiply the i-th row of A with the j-th column of B entry-wise and add their products. Observe that for the product to be well-defined, the number of columns of A has to agree with the number of rows of B.

EXAMPLE 1.1

Let A B - 1 2 0 3 . Then, for instance, to obtain the entry

4 - 5 3 2 1 6

C32 of the product matrix C = AB, we multiply entrywise the third row of A with the second column of B: (1)(2) + (6)(3) = 20. Thus, we get

C = A-B = " 4

3 _ 1

- 5 ' 2 6 _

- 1 2 0 3 L J

= -4 - 7 -3 12 -1 20

NOTATION AND TERMINOLOGY 3

MATLAB command: A + B, A* B.

Given a matrix BpXn, we can denote its columns with 6 1 , . . . , 6n, where each b{ is a p-dimensional vector. We will write accordingly,

B = [h . . . bn).

In such a case, we can write a product of matrices as

AB = A[b! •■• bn] = [Ah . . . Abnl (1.1)

so that Ab\,..., A6n are the columns of the matrix AB.

Similarly, if we denote with CL\ , . . . , CLm the rows of a matrix AmXp, then

,4£ r ai

Q"m

5 = aiJ3 ]

amB

For an arbitrary matrix Amxn, we denote with AT its transpose matrix, that is, the matrix of order nxm, where the rows of A have been exchanged for columns and vice versa. For instance,

If A = -4 7 3 8

then A1 = 6 - 2

-4 3 7 8

Remark 1.1 If Amxn is a complex matrix, its adjoint matrix is denoted as A*, where A* = AT, that is, the conjugate transpose. For instance,

If A 3 - 4 + i -2i 5 + 2i then A* 3 2% - 4 - i 5 - 2%

• MATLAB command: A'

The sum and product of matrices satisfy the following properties (see Exercise 1.1):

(A + B)T = AT + BT, (AB)T = £ T A T . (1.2)

Definition 1.2 The trace 0/ a square matrix A of order n is defined as the sum of its diagonal elements, that is,

tr{A) = ̂ 2 an. (1.3)

4 BASICS OF LINEAR ALGEBRA

MATLAB command: trace(A)

A particular and very special case is that of matrices of order n x 1. Such a matrix is usually called an n-dimensional vector. That is, here we consider vectors as column-vectors, and we will use the notation

Xi

[Xi • • • Xn}1

for a typical vector. This substitutes the usual notation x = (x\,..., xn), which we reserve to denote a point, and at the same time this notation will allow us to perform matrix-vector operations in agreement with their dimensions. This also closely follows the notation used in MATLAB .

The first two vectors below are column vectors, whereas the third is a row vector.

9 4 3

[ 4 - 3 5]T, [1 8 5].

1.2 VECTOR AND MATRIX NORMS

It is always important and useful to have a notion for the "size" or magnitude of a vector or a matrix, just as we understand the magnitude of a real number by using its absolute value. In fact, a norm can be understood as the generalization of the absolute value function to a higher dimensional case. This is especially useful in numerical analysis for estimating the magnitude of the error when approximating the solution to a given problem.

Definition 1.3 A vector norm, denoted by \\-\\,isa real function that satisfies the following properties for arbitrary n-dimensional vectors x and y and for arbitrary real or complex a:

(i) \\x\\ > 0, (ii) \\x\\ = 0 if and only if x = 0, (Hi) \\ax\\ = \a\ \\x\\, (iv) \\x + y\\

VECTOR AND MATRIX NORMS 5

||#||i = ^2 \xi\ Sum norm i=l

1/2 v. I FjiirliHfian norm K'-'^J

/n y/* Ml 2 = ( Yl xl ) Euclidean norm

IÎ Hoo = max \x{\ Maximum norm

• MATLAB commands: norm(#, 1), norm(x) norm(x,inf)

EXAMPLE 1.2

Letx = [3 - 2 4 v/7]T.Then,

||a:||i = |3| + | - 2 | + |4| + |>/7 |« 11.6458. ||x||2 = V9 + 4 + 16 + 7 = 6. W 0 0 = m a x { | 3 | , | - 2 | , |4|, |A /7 |} = 4.

Remark 1.4 In general, for an arbitrary vector x £ Rn, the following inequalities (see Exercise 1.4) relate the three norms above:

Nloo < IMh < ||x||i. (1.5)

EXAMPLE 1.3

The unit ball in E n is the set {x e Mn : ||x|| < 1}. The geometrical shape of the unit ball varies according to what norm is used. For the case n = 2, the unit balls for the three norms in (1.4) are shown in Figure 1.1.

Note: For simplicity of notation, || • || will always denote the Euclidean norm || • H2 for vectors, unless otherwise stated.

We now introduce the notion of norms for a matrix, in some sense as a generalization of vector norms.

Definition 1.5 A matrix norm is a real function that for arbitrary matrices A and B and arbitrary real or complex a, satisfies

d) Mil > 0, (ii) \\A\\ = 0 if and only if A = 0, (Hi) \\aA\\ = |a|||j4||,

6 BASICS OF LINEAR ALGEBRA

- , . 5 | . _ .

1 h Jv

O . S h / \

° I \ / ^ O.S k \ / -J

r— r~*n

Co) 1'5| ' T ' 1

-i L J

o . s h J

o —W

O.S p -H

_ i L I 1 J

o -i - 1 O -I

Figure 1.1 Unit ball in (a) || • ||lf (b) || • ||2, (c) || • ||oo.

(iv) P + JB||

VECTOR AND MATRIX NORMS 7

Remark 1.7 There are two other ways of defining the Frobenius norm, which are useful for certain computations, as we illustrate later on. Let Amxn be an arbitrary matrix, and denote its columns with a i , . . . , an. Then,

(a) \\AfF = | |a i | | | + - . . + | |an | | l . (1.7)

(b) \\A\\% = tv(ATA). (1.8)

It seems natural to think that vector norms can be directly generalized to obtain matrix norms (after all, mxn matrices can be thought of as vectors of dimension m • n). However, not all vector norms directly become matrix norms (see Exercise 1.11).

A matrix norm can be induced or constructed directly from a vector norm by defining for 1 < p < oo

\\A\\P = max \\Ax\\p, (1.9) N I P = I

where the norms on the right-hand side represent the vector norm. This is the way to correctly extend or use a vector norm into obtaining a matrix norm.

Given a vector x of norm ||#||, when it is multiplied by A, we get the new vector Ax of norm || Ax\\. Thus, we can interpret the matrix norm (1.9) as a natural way to measure how much the vector x can be stretched or shrunk by A.

The definition of the p-norm of a matrix in (1.9) is not easy to implement or compute in general. Luckily enough, there are alternative ways to compute such a norm for some particular values of p.

EXAMPLE 1.5

It can be proved (see Exercise 1.12) that for p = 1 and p = oo, the p-norms in (1.9) can be directly computed through the corresponding definitions in (1.6).

■ EXAMPLE 1.6

For p = 2, the p-norm in (1.9) can be computed as the square root of the largest eigenvalue of the symmetric matrix AT A. That is,

||^4||2 = maxjv/X : A is an eigenvalue of AT A}. (1.10)

For instance, for the matrix of Example 1.4, we get || A\\2 ~ 5.9198.

Remark 1.8 It is possible to show that an equivalent way to define the matrix norm (1.9) is

||A||p = 8 u p J ! ^ . (1.11)

8 BASICS OF LINEAR ALGEBRA

Remark 1.9 In general, inequalities for matrix norms similar to the ones in (1.5) are not true. However, it is still true that

\\Ah

SPECIAL MATRICES

Following (1.15), we also see that for arbitrary AmXn, x e Rn, y G E m ,

< Ax, y >= (Ax)Ty = xTATy = < x, ATy > . (1.17)

Remark 1.11 The dot product introduced here is a particular case of the general inner product function studied in the context of inner product vector spaces, where the elements are not restricted to real n-dimensional vectors.

There is a special kind of vectors that is very useful in several instances in linear algebra and matrix computations. These are the so-called orthonormal vectors, which are orthogonal (perpendicular) to each other and they are unit, that is, x and y are orthonormal if

xTy = 0, and ||x|| = ||y|| = 1.

For example, the following set of vectors is orthonormal:

Vi _2_ o - VE. V2 = -7= ° -7= \/5 \ /5 ,

V3 = [0 - 1 0]T.

In fact, we can readily verify that

v[v2 = v[vs = v%v3 = 0, and ||vi|| = ||v2| *>3 1.

1.4 SPECIAL MATRICES

We will be using matrices at almost every place in this book, matrices of different kinds and properties. Here we list some of the most common types of matrices we will encounter.

1.4.1 Diagonal and triangular matrices

A square matrix A of order n is called diagonal if a^ = 0 for all i ^ j .

EXAMPLE 1.7

A = ' 4 0

0 1 0 0

0 0

- 9

MATLAB command: diag

A square matrix A of order n is called upper (resp. lower) triangular if a^ = 0 for all i > j , (resp. for all i < j).

10 BASICS OF LINEAR ALGEBRA

■ EXAMPLE 1.8

" 4 2 5 " 0 1 3

_ 0 0 9 _ , B =

' 4 0 0 " 5 1 0 7 1 9

The matrix A is upper triangular, and B is lower triangular.

• MATLAB commands: triu, tril

Remark 1.12 If the matrix A is rectangular, say order m x n, we say it is upper (resp. lower) trapezoidal ifaij = 0 for all i > j (resp. for all i < j).

A more general case of triangular matrices is that of block triangular matrices. For example, the following 5 x 5 matrix is block upper triangular

" - 8 1 4 - 2 0 0 0 0 0 0

0 9 8 5 5 7

- 9 - 8 0 0

3 1 5 1 4

- 6 J where for instance, the zero in the (2,1) entry of the matrix on the right represents the corresponding 2 x 2 zero block of the matrix on the left. In a similar way, a block diagonal matrix can be defined. We will encounter this type of matrices in Section 1.10 when we compute eigenvalues of a matrix and when compute special vector subspaces later in Chapter 7.

1.4.2 Hessenberg matrices

A matrix Anxn is called an upper Hessenberg matrix if a^ = 0 for alH > j + 1 . They take the form:

a n ^ 2 1

0 0

0

a i 2

^22

^32 0

0

a i 3

« 2 3

« 3 3

« 4 3

0

In other words, it is an upper triangular matrix with an additional subdiagonal below the main diagonal. A matrix A is called lower Hessenberg if AT is upper Hessenberg.

An A12 A13 0 A22 A23 0 0 A33

ain-i ain

Q

SPECIAL MATRICES 11

1.4.3 Nonsingular and inverse matrices

In the set of real numbers R, the number 1 is the multiplicative identity, meaning that a -1 = 1 • a = a, for any real number a. This is generalized to what is called the identity matrix of order n, denoted by / . It consists of a diagonal of ones, and every other entry is zero (see (1.21) below), with the property that

AI = IA = A,

for any matrix A.

• MATLAB command: eye(n).

A square matrix A of order n is called nonsingular if its determinant is not zero: det( A)^ 0. Otherwise it is called singular.

EXAMPLE 1.9

det " 4 2

1 1 7 1

9 0 3

= —48. Hence, the matrix is nonsingular.

• MATLAB command: det(A).

Remark 1.13 Two very useful properties of determinants are the following:

det(AB) = det A det B, (1.19)

det A" 1 = 1/detA

In one dimension, every real number a ^ 0 has a unique multiplicative inverse 1/a, and it is obvious that a ( £ ) = ( £ ) a = l. This has a natural generalization to matrices. A nonsingular matrix Anxn has a unique nx n inverse matrix, denoted by A~x with the property that

A A'1 =A~1A = L (1.20)

For this reason, nonsingular matrices are also called invertible.

• MATLAB command: inv(A).

12 BASICS OF LINEAR ALGEBRA

EXAMPLE 1.10

Let A = 4 0 9 1 - 1 0

-2 1 0 . Since det(A) = —9, the matrix is nonsingular and has

an inverse: A 1 = 0 -1 - 1 " 0 -2 -1

_ 1/9 4/9 4/9 _ , and we can ven

A A'1 =A~1A = I = " 1 0 0 0 1 0 0 0 1

(1.21)

Computing the inverse of a matrix is not an easy task. There is more than one analytical or exact way to do this, e.g., using the adjoint method, we have

A'1 = adjoint (A) /det (A).

However, this method is rarely used in practice, and we do not discuss it here. A second and more efficient method is through Gauss elimination (see Section 3.1). In general, the computation of the inverse of a matrix has to be done numerically, and great care has to be taken due to potentially large accumulation of errors for some matrices. Thus, in practice it is customary to avoid computing the inverse of a matrix explicitly, and some other options must be used. We will discuss these issues later on.

Remark 1.14 For the inverse of a product of nonsingular matrices, we have

(A1A2--.Ak)-1 =A^A-l_l---A^\ (1.22)

1.4.4 Symmetric and positive definite matrices

A square matrix A of order n is called symmetric if A = A7', that is, the matrix equals its transpose. If A = (a^), then we can also say A is symmetric if a^ = a^, for i,j = l , . . . , n .

EXAMPLE 1.11

The following matrices are symmetric:

A = 1 4 4 9

1 - 2 7 -2 3 5 7 5 6