View
2
Download
0
Category
Preview:
Citation preview
AMS526: Numerical Analysis I(Numerical Linear Algebra for
Computational and Data Sciences)Lecture 1: Course Overview;
Matrix Multiplication
Xiangmin Jiao
Stony Brook University
Xiangmin Jiao Numerical Analysis I 1 / 22
Outline
1 Course Overview
2 Matrix Notation and Basic Operations (MC �1.1)
3 Range, Null Space and Rank (NLA �1)
Xiangmin Jiao Numerical Analysis I 2 / 22
Course Description
What is numerical linear algebra?
I Solving linear algebra problems using e�cient algorithms on computers
Topics: Direct and iterative methods for solving simultaneous linearequations, least squares problems, computation of eigenvalues andeigenvectors, and singular value decomposition
Required textbooks[NLA] L. N. Trefethen and D. Bau III, Numerical Linear Algebra,SIAM, 1997.[MC] G. H. Golub and C. F. Van Loan, Matrix Computations, 4thEdition, Johns Hopkins University Press, 2012. ISBN 978-1421407944.
Additional supplementary material will be provided as needed
Course webpage:http://www.ams.sunysb.edu/~jiao/teaching/ams526
Xiangmin Jiao Numerical Analysis I 3 / 22
Prerequisite
This MUST NOT be your �rst course in linear algebra, or you willget lost
Prerequisite/Co-requisite:
I AMS 510 (linear algebra portion) or equivalent undergraduate-levellinear algebra course. Familiarity with following concepts is assumed:Vector spaces, Gaussian elimination, Gram-Schmidt orthogonalization,and eigenvalues/eigenvectors
I AMS 595 (co-requisite for students without programming experience)
To review fundamental concepts of linear algebra, see textbook such as
I Gilbert Strang, Linear Algebra and Its Applications, 4th Edition, BrooksCole, 2006.
Xiangmin Jiao Numerical Analysis I 4 / 22
Why Learn Numerical Linear Algebra?Foundation of scienti�c computations and data sciencesMany problems ultimately reduce to linear algebra concepts oralgorithms, either analytical or computationalExamples: Finite-element analysis, data �tting, PageRank (Google)
Focus: Fundamental concepts, e�ciency and stability of algorithms,and programmingNew focus: relevance to computational and data sciences
Xiangmin Jiao Numerical Analysis I 5 / 22
Course OutlineFundamentals (matrix notation and basic operations; vector spaces;algorithmic considerations; norms and condition numbers;decomposition of matrices)Linear systems (triangular systems; Gaussian elimination; accuracyand stability; Cholesky factorization; sparse linear systems)QR factorization and least squares (Gram-Schmidtorthogonalization; QR factorization with Householder re�ection;updating QR factorization with Givens rotation; stability; least squaresproblems; rank-revealing QR; SVD and low-rank approximations)Eigenvalue problems (eigenvalues and invariant spaces; classicaleigenvalue methods; QR algorithms; two-stage methods; Arnoldi andLanczos iterations)Iterative Methods for linear systems (basic iterative methods;conjugate gradient methods; minimal residual style methods;bi-Lanczos iterations; preconditioners)Special topics (multigrid methods; under-determined systems, etc., iftime permits)
Xiangmin Jiao Numerical Analysis I 6 / 22
Course Policy
Assignments (written or programming)
I Assignments are due in class one to two weeks after assignedI You can discuss course materials and homework problems with others,
but you must write your answers completely independentlyI Do NOT copy solutions from any source. Do NOT share your solutions
with others
Exams and tests
I All exams are closed-bookI However, one-page cheat sheet is allowed
Grading
I Assignments: 30%I Two midterm exams: 40%I Final exam: 30%
Xiangmin Jiao Numerical Analysis I 7 / 22
Outline
1 Course Overview
2 Matrix Notation and Basic Operations (MC �1.1)
3 Range, Null Space and Rank (NLA �1)
Xiangmin Jiao Numerical Analysis I 8 / 22
Matrices and Vectors
Denote vector space of all m-by-n real matrices by Rm×n.
A ∈ Rm×n ⇔ A = (aij) =
a11 a12 . . . a1na21 a22 . . . a2n...
.... . .
...am1 am2 . . . amn
Denote vector space of all real n-vectors by Rn, or Rn×1
x ∈ Rn ⇔ x =
x1x2...xn
Transposition (Rm×n → Rn×m): C = AT ⇒ cij = aji
Row vectors are transpose of column vectors and are in R1×n
Xiangmin Jiao Numerical Analysis I 9 / 22
Matrix Operations
Addition and subtraction (Rm×n × Rm×n → Rm×n):C = A± B ⇒ cij = aij ± bij
Scalar-matrix multiplication or scaling (R× Rm×n → Rm×n):C = αA⇒ cij = αaij
Matrix-matrix multiplication/product (Rm×p × Rp×n → Rm×n):C = AB ⇒ cij =
∑pk=1 aikbkj (denoted by A ∗ B in MATLAB)
Each operation also applies to vectors. In particular,
I Inner product is row vector times column vector, i.e., c = xT y
(it is called dot product in vector calculus and denoted as x · y)I Outer product is column vector times row vector, i.e., C = xyT
(it is a special case of Kronecker product and denoted as x ⊗ y)
Element-wise multiplication and division (Rm×n × Rm×n → Rm×n)
I C = A. ∗ B ⇒ cij = aijbijI C = A./B ⇒ cij = aij/bij , where bij 6= 0
Matrix inversion (A−1) and division (A/B and A\B) to be de�ned later
Xiangmin Jiao Numerical Analysis I 10 / 22
Notation of Matrices and VectorsMatrix notation
I Capital letters (e.g., A, B, ∆, etc.) for matricesI Corresponding lower case with subscript ij (e.g., aij , bij , δij) for (i , j)
entry; sometimes with notation [A]ij or A(i , j)
Vector notationI Lowercase letters (e.g., x , y , etc.) for vectorsI Corresponding lower case with subscript i for ith entry (e.g., xi , yi )
Lower-case letters for scalars (e.g., c , s, α, β, etc.)Some books suggest using boldface lowercase (e.g., x) for vectors,regular lowercase (e.g. c) for scalars, and boldface uppercase formatricesA matrix is a collection of column vectors or row vectors
A ∈ Rm×n ⇔ A = [c1|c2| . . . |cn] , ck ∈ Rm
A ∈ Rm×n ⇔ A =
rT1...rTm
, rk ∈ Rn
Xiangmin Jiao Numerical Analysis I 11 / 22
Complex Matrices
Occasionally, complex matrices are involved
Vector space of m-by-n complex matrices is designated by Cm×n
I Scaling, addition, multiplication of complex matrices correspondexactly to real case
I If A = B + iC ∈ Cm×n, then Re(A) = B, Im(A) = C , and conjugate ofA is A = (aij)
I Conjugate transpose is de�ned as AH = BT − iCT , or
G = AH ⇒ gij = aji
(also called adjoint, and denoted by A∗; (AB)∗ = B∗A∗)
Vector space of complex n-vectors is designated by Cn
I Inner product of complex n-vectors x and y is s = xHyI Outer product of complex n-vectors x and y is S = xyH
We will primarily focus on real matrices
Xiangmin Jiao Numerical Analysis I 12 / 22
Matrix-Vector Product
Matrix-vector product b = Ax is special case of matrix-matrix product
bi =n∑
j=1
aijxj
For A ∈ Rm×n, Ax is a mapping x 7−→ Ax from Rn to Rm
This map is linear, which means that for any x , y ∈ Rn and any α ∈ R
A(x + y) = Ax + Ay
A(αx) = αAx
Xiangmin Jiao Numerical Analysis I 13 / 22
Linear Combination
Let aj denote jth column of matrix A
I Alternative notation is colon notation: A(:, j) or a:,jI Use A(i , :) or ai,: to denote ith row of A
b is a linear combination of column vectors of A, i.e.,
b = Ax =n∑
j=1
xjaj = x1
a11a21...
am1
+ x2
a12a22...
am2
+ · · ·+ xn
a1ma2n...
amn
In summary, two di�erent views of matrix-vector products:
1 Scalar operations: bi =∑n
j=1aijxj : A acts on x to produce b
2 Vector operations: b =∑n
j=1xjaj : x acts on A to produce b
Xiangmin Jiao Numerical Analysis I 14 / 22
Matrix-Matrix Multiplication
Computes C = AB , where A ∈ Rm×r , B ∈ Rr×n, and C ∈ Rm×n
Element-wise, each entry of C is
cij =r∑
k=1
aikbkj
Column-wise, each column of C is
cj = Abj =r∑
k=1
bkjak ;
in other words, jth column of C is A times jth column of B
Xiangmin Jiao Numerical Analysis I 15 / 22
Outline
1 Course Overview
2 Matrix Notation and Basic Operations (MC �1.1)
3 Range, Null Space and Rank (NLA �1)
Xiangmin Jiao Numerical Analysis I 16 / 22
Terminology of Vector Space
Vector space is closed under addition and scalar multiplication, withzero vector as a member
Vector space spanned by a set of vectors {aj} is
span{a1, . . . , an} =
n∑
j=1
βjaj | βj ∈ R
I Space spanned by n-vectors is a subspace of Rn
I If {a1, . . . , an} is linearly independent, then the aj are the basis ofS = span{a1, . . . , an}, dimension of S is dim(S) = n, and each b ∈ S isunique linear combination of the aj
If S1 and S2 are two subspaces, then S1 ∩ S2 is a subspace, so isS1 + S2 = {b1 + b2 | b1 ∈ S1, b2 ∈ S2}
I Note: S1 + S2 is di�erent from S1 ∪ S2; latter may not be a subspace
Two subspaces S1 and S2 of Rn are complementary subspaces of eachother if S1 + S2 = Rn and S1 ∩ S2 = {0}
I In other words, dim(S1) + dim(S2) = n and S1 ∩ S2 = {0}Xiangmin Jiao Numerical Analysis I 17 / 22
Range and Null Space
De�nition
The range of a matrix A, written as range(A) or ran(A), is the set ofvectors that can be expressed as Ax for some x{y ∈ Rm|y = Ax for some x ∈ Rn}.
Theorem
range(A) is the space spanned by the columns of A.
Therefore, range of A is also called the column space of A.
De�nition
The null space of A ∈ Rm×n, written as null(A), is the set of vectors x thatsatisfy Ax = 0.
Entries of x ∈ null(A) give coe�cient of∑
xiai = 0.Note: The null space of A is in general not a complementary subspace ofrange(A).
Xiangmin Jiao Numerical Analysis I 18 / 22
Range and Null Space
De�nition
The range of a matrix A, written as range(A) or ran(A), is the set ofvectors that can be expressed as Ax for some x{y ∈ Rm|y = Ax for some x ∈ Rn}.
Theorem
range(A) is the space spanned by the columns of A.
Therefore, range of A is also called the column space of A.
De�nition
The null space of A ∈ Rm×n, written as null(A), is the set of vectors x thatsatisfy Ax = 0.
Entries of x ∈ null(A) give coe�cient of∑
xiai = 0.Note: The null space of A is in general not a complementary subspace ofrange(A).
Xiangmin Jiao Numerical Analysis I 18 / 22
Range and Null Space
De�nition
The range of a matrix A, written as range(A) or ran(A), is the set ofvectors that can be expressed as Ax for some x{y ∈ Rm|y = Ax for some x ∈ Rn}.
Theorem
range(A) is the space spanned by the columns of A.
Therefore, range of A is also called the column space of A.
De�nition
The null space of A ∈ Rm×n, written as null(A), is the set of vectors x thatsatisfy Ax = 0.
Entries of x ∈ null(A) give coe�cient of∑
xiai = 0.Note: The null space of A is in general not a complementary subspace ofrange(A).
Xiangmin Jiao Numerical Analysis I 18 / 22
Relationship Between Null and Range Space
For real matrices A ∈ Rm×n
I null(A) and range(AT ) are complementary subspacesI For symmetric matrices (A = AT ), null(A) and range(A) are
complementary subspaces
For complex matrices A ∈ Cm×n
I null(A) and range(AH) are complementary subspacesI For Hermitian matrices (A = AH), null(A) and range(A) are
complementary subspaces
Xiangmin Jiao Numerical Analysis I 19 / 22
Rank
De�nition
The column rank of a matrix is the dimension of its column space. The rowrank is the dimension of the space spanned by its rows.
Question: Can the column rank and the row rank be di�erent?
Answer: No. We will give a proof in later lectures.
We therefore simply say the rank of a matrix.
Question: Given A ∈ Rm×n, what is dim(null(A)) + rank(A) equal to?Answer: n.
A real matrix A is rank-1 if it can be written as A = uvT , where u andv are nonzero vectors
Xiangmin Jiao Numerical Analysis I 20 / 22
Rank
De�nition
The column rank of a matrix is the dimension of its column space. The rowrank is the dimension of the space spanned by its rows.
Question: Can the column rank and the row rank be di�erent?Answer: No. We will give a proof in later lectures.
We therefore simply say the rank of a matrix.
Question: Given A ∈ Rm×n, what is dim(null(A)) + rank(A) equal to?
Answer: n.
A real matrix A is rank-1 if it can be written as A = uvT , where u andv are nonzero vectors
Xiangmin Jiao Numerical Analysis I 20 / 22
Rank
De�nition
The column rank of a matrix is the dimension of its column space. The rowrank is the dimension of the space spanned by its rows.
Question: Can the column rank and the row rank be di�erent?Answer: No. We will give a proof in later lectures.
We therefore simply say the rank of a matrix.
Question: Given A ∈ Rm×n, what is dim(null(A)) + rank(A) equal to?Answer: n.
A real matrix A is rank-1 if it can be written as A = uvT , where u andv are nonzero vectors
Xiangmin Jiao Numerical Analysis I 20 / 22
Full Rank
De�nition
A matrix has full rank if it has the maximal possible rank, i.e., min{m, n}.
Otherwise, it is called rank de�cient.
Theorem
A matrix A ∈ Rm×n with m ≥ n has full rank if and only if it maps no twodistinct vectors to the same vector.
In other word, the linear mapping de�ned by Ax for x ∈ Rn is one-to-one.
Proof.
(⇒) Column vectors of A forms a basis of range(A), so every b ∈ range(A)has a unique linear expansion in terms of the columns of A.(⇐) If A does not have full rank, then its column vectors are lineardependent, so its vectors do not have a unique linear combination.
Xiangmin Jiao Numerical Analysis I 21 / 22
Full Rank
De�nition
A matrix has full rank if it has the maximal possible rank, i.e., min{m, n}.
Otherwise, it is called rank de�cient.
Theorem
A matrix A ∈ Rm×n with m ≥ n has full rank if and only if it maps no twodistinct vectors to the same vector.
In other word, the linear mapping de�ned by Ax for x ∈ Rn is one-to-one.
Proof.
(⇒) Column vectors of A forms a basis of range(A), so every b ∈ range(A)has a unique linear expansion in terms of the columns of A.(⇐) If A does not have full rank, then its column vectors are lineardependent, so its vectors do not have a unique linear combination.
Xiangmin Jiao Numerical Analysis I 21 / 22
Full Rank
De�nition
A matrix has full rank if it has the maximal possible rank, i.e., min{m, n}.
Otherwise, it is called rank de�cient.
Theorem
A matrix A ∈ Rm×n with m ≥ n has full rank if and only if it maps no twodistinct vectors to the same vector.
In other word, the linear mapping de�ned by Ax for x ∈ Rn is one-to-one.
Proof.
(⇒) Column vectors of A forms a basis of range(A), so every b ∈ range(A)has a unique linear expansion in terms of the columns of A.(⇐) If A does not have full rank, then its column vectors are lineardependent, so its vectors do not have a unique linear combination.
Xiangmin Jiao Numerical Analysis I 21 / 22
Full Rank vs. Non-singularity
If A ∈ Rn×n and AX = I , then X is the inverse of A, denoted by A−1
I (AB)−1 = B−1A−1
I (A−1)T = (AT )−1 = A−T
I (A + UV T )−1 = A−1 − A−1U(I + V TA−1U)−1V TA−1
for U,V ∈ Rn×k (Sherman-Morrison-Woodbury formula)
If A−1 exists, A is nonsingular. A is square and has full rank.
In A is nonsingular, linear system Ax = b results in x = A−1b, and itis the inverse problem of matrix-vector multiplication
If A ∈ Rm×n where n 6= m and A has full rank, what is the inverseproblem of matrix-vector multiplication?
If A is rank de�cient, what is the inverse problem of matrix-vectormultiplication?
We will need progressively more advanced linear algebra concepts toanswer these questions in later part of this semester.
Xiangmin Jiao Numerical Analysis I 22 / 22
Recommended