Upload
lyanh
View
255
Download
3
Embed Size (px)
Citation preview
AMS526: Numerical Analysis I(Numerical Linear Algebra)
Lecture 1: Course Overview;Matrix Multiplication
Xiangmin Jiao
Stony Brook University
Xiangmin Jiao Numerical Analysis I 1 / 22
Outline
1 Course Overview
2 The Language of Matrix Computations
3 Algorithms and Efficiency
Xiangmin Jiao Numerical Analysis I 2 / 22
Course Description
What is numerical linear algebra?I Solving linear algebra problems using efficient algorithms on computers
Topics: Direct and iterative methods for solving simultaneous linearequations. Matrix factorization, conditioning, stability, sparsity, andefficiency. Computation of eigenvalues and eigenvectors. Singularvalue decompositionRequired textbook (also an excellent reference book)
I G. H. Golub and C. F. Van Loan, Matrix Computations, 4th Edition,Johns Hopkins University Press, 2012. ISBN 978-1421407944.
Supplementary textbook (a very insightful book)I L. N. Trefethen and D. Bau III, Numerical Linear Algebra, SIAM, 1997.
Xiangmin Jiao Numerical Analysis I 3 / 22
Prerequisite
Prerequisite/Co-requisite:I AMS 510 (linear algebra portion) or equivalent undergraduate-level
linear algebra course. Familiarity with following concepts is assumed:Vector spaces, Gaussian elimination, Gram-Schmidt orthogonalization,and eigenvalues/eigenvectors
I AMS 595 (co-requisite for students without programming experience)I This MUST NOT be your first course in linear algebra, or you will get
lost
To review fundamental concepts of linear algebra, see textbook such asI Gilbert Strang, Linear Algebra and Its Applications, 4th Edition, Brooks
Cole, 2006.
Xiangmin Jiao Numerical Analysis I 4 / 22
Why Learn Numerical Linear Algebra?Numerical linear algebra is foundation of scientific computationsMany problems ultimately reduce to linear algebra concepts oralgorithms, either analytical or computationalExamples: Finite-element analysis, data fitting, PageRank (Google)
Focus of this course: Fundamental concepts, efficiency and stability ofalgorithms, and programming
Xiangmin Jiao Numerical Analysis I 5 / 22
Course Outline
Basic linear algebra concepts and algorithms (1.5 weeks)Gaussian elimination and its variants (2 weeks)Sensitivity and stability (1.5 weeks)Linear least squares problem (1.5 weeks)Eigenvalues, eigenvectors, and SVD (4 weeks)Iterative methods for linear systems (2 weeks)Course webpage:http://www.ams.sunysb.edu/~jiao/teaching/ams526_fall14
Note: Course schedule online is tentative and is subject to change.
Xiangmin Jiao Numerical Analysis I 6 / 22
Course Policy
Assignments (written or programming)I Assignments are due in class one to two weeks after assignedI You can discuss course materials and homework problems with others,
but you must write your answers completely independentlyI Do NOT copy solutions from any source. Do NOT share your solutions
with others
Exams and testsI All exams are closed-bookI However, one-page cheat sheet is allowed
GradingI Assignments: 30%I Two midterm exams: 40%I Final exam: 30%
Xiangmin Jiao Numerical Analysis I 7 / 22
Outline
1 Course Overview
2 The Language of Matrix Computations
3 Algorithms and Efficiency
Xiangmin Jiao Numerical Analysis I 8 / 22
Matrices and Vectors
Denote vector space of all m-by-n real matrices by Rm⇥n.
A 2 Rm⇥n , A = (aij) =
2
6664
a11
a12
. . . a1n
a21
a22
. . . a2n
......
. . ....
am1
am2
. . . amn
3
7775
Denote vector space of all real n-vectors by Rn, or Rn⇥1
x 2 Rn , x =
2
6664
x1
x2
...xn
3
7775
Transposition (Rm⇥n ! Rn⇥m): C = AT ) cij = aji
Row vectors are transpose of column vectors and are in R1⇥n
Xiangmin Jiao Numerical Analysis I 9 / 22
Matrix OperationsAddition and subtraction (Rm⇥n ⇥ Rm⇥n ! Rm⇥n):C = A ± B ) cij = aij ± bij
Scalar-matrix multiplication or scaling (R⇥ Rm⇥n ! Rm⇥n):C = ↵A) cij = ↵aij
Matrix-matrix multiplication/product (Rm⇥p ⇥ Rp⇥n ! Rm⇥n):C = A B ) cij =
Ppk=1
aikbkj (denoted by A ⇤ B in MATLAB)Each operation also applies to vectors. In particular,
I Inner product is row vector times column vector, i.e., c = xT y(it is called dot product in vector calculus and denoted as x · y)
I Outer product is column vector times row vector, i.e., C = xyT
(it is a special case of Kronecker product and denoted as x ⌦ y)
Elementwise multiplication and division (Rm⇥n ⇥ Rm⇥n ! Rm⇥n)I C = A. ⇤ B ) cij = aijbijI C = A./B ) cij = aij/bij , where bij 6= 0
Matrix inversion (A�1) and division (A/B and A\B) to be defined later
Xiangmin Jiao Numerical Analysis I 10 / 22
Notation of Matrices and VectorsMatrix notation
I Capital letters (e.g., A, B, �, etc.) for matricesI Corresponding lower case with subscript ij (e.g., aij , bij , �ij) for (i , j)
entry; sometimes with notation [A]ij or A(i , j)
Vector notationI Lowercase letters (e.g., x , y , etc.) for vectorsI Corresponding lower case with subscript i for ith entry (e.g., xi , yi )
Lower-case letters for scalars (e.g., c , s, ↵, �, etc.)Some suggest using boldface lowercase (e.g., x) for vectors, regularlowercase (e.g. c) for scalars, and boldface uppercase for matricesA matrix is a collection of column vectors or row vectors
A 2 Rm⇥n , A = [c1
|c2
| . . . |cn] , ck 2 Rm
A 2 Rm⇥n , A =
2
64r
T1
...r
Tm
3
75 , rk 2 Rn
Xiangmin Jiao Numerical Analysis I 11 / 22
Complex Matrices
Occasionally, complex matrices are involvedVector space of m-by-n complex matrices is designated by Cm⇥n
I Scaling, addition, multiplication of complex matrices correspondexactly to real case
I If A = B + iC 2 Cm⇥n, then Re(A) = B, Im(A) = C , and conjugate ofA is A = (aij)
I Conjugate transpose is defined as AH = BT � iCT , or
G = AH ) gij = aji
(also called adjoint, and denoted by A⇤; (AB)⇤ = B⇤A⇤)
Vector space of complex n-vectors is designated by Cn
I Inner product of complex n-vectors x and y is s = xHyI Outer product of complex n-vectors x and y is S = xyH
We will primarily focus on real matrices
Xiangmin Jiao Numerical Analysis I 12 / 22
Matrix-Vector Product
Matrix-vector product b = Ax is special case of matrix-matrix product
bi =nX
j=1
aijxj
For A 2 Rm⇥n, Ax is a mapping x 7�! Ax from Rn to Rm
This map is linear, which means that for any x , y 2 Rn and any ↵ 2 R
A(x + y) = Ax + AyA(↵x) = ↵Ax
Xiangmin Jiao Numerical Analysis I 13 / 22
Linear Combination
Let aj denote jth column of matrix AI Alternative notation is colon notation: A(:, j) or a:,jI Use A(i , :) or ai,: to denote ith row of A
b is a linear combination of column vectors of A, i.e.,
b = Ax =nX
j=1
xjaj = x1
2
6664
a11
a21
...am1
3
7775+ x
2
2
6664
a12
a22
...am2
3
7775+ · · ·+ xn
2
6664
a1m
a2n...
amn
3
7775
In summary, two different views of matrix-vector products:1 Scalar operations: bi =
Pnj=1 aijxj : A acts on x to produce b
2 Vector operations: b =Pn
j=1 xjaj : x acts on A to produce b
Xiangmin Jiao Numerical Analysis I 14 / 22
Matrix-Matrix Multiplication
Computes C = AB, where A 2 Rm⇥r , B 2 Rr⇥n, and C 2 Rm⇥n
Element-wise, each entry of C is
cij =rX
k=1
aikbkj
Column-wise, each column of C is
cj = Abj =rX
k=1
bkjak ;
in other words, jth column of C is A times jth column of B
Xiangmin Jiao Numerical Analysis I 15 / 22
Outline
1 Course Overview
2 The Language of Matrix Computations
3 Algorithms and Efficiency
Xiangmin Jiao Numerical Analysis I 16 / 22
Algorithms for Matrix-Vector Multiplication
Suppose A 2 Rm⇥n and x 2 Rn
MATLAB-style code for b = Ax :
Row oriented
for i = 1 : mb(i) = 0for j = 1 : n
b(i) = b(i)+A(i , j)⇤x(j)end
end
Column oriented
b(:) = 0for j = 1 : n
for i = 1 : mb(i) = b(i)+A(i , j)⇤x(j)
end
end
Number of operations is O(mn), but big-Oh notation is insufficientNumber of operations is 2mn: coefficient of leading-order term isimportant for comparison
Xiangmin Jiao Numerical Analysis I 17 / 22
Flop Count
It is important to assess efficiency of algorithms. But how?I We could implement different algorithms and do direct comparison, but
implementation details can affect true performanceI We could estimate cost of all operations, but it is very tediousI Relatively simple and effective approach is to estimate amount of
floating-point operations, or “flops”, and focus on asymptotic analysisas sizes of matrices approach infinity
IdealizationI Count each operation +,�, ⇤, /, and p as one flopI This estimation is crude, as it omits data movement in memory, which
is non-negligible on modern computer architectures (e.g., different looporders can affect cache performance)
Matrix-vector product requires about 2mn flopsSuppose m = n, it takes quadratic time in n, or O(n2)
Xiangmin Jiao Numerical Analysis I 18 / 22
Algorithms for Saxpy: Scalar a x plus y
Saxpy computes ax + y and updates y
y = ax + y ) yi = axi + yi
Suppose x , y 2 Rn and a 2 R
MATLAB-style code
for i = 1 : ny(i) = y(i) + a ⇤ x(i)
end
Pseudo-code
for i = 1 : nyi yi + a xi
Number of flops is 2nPseudo-code cannot run on any computer, but are human readableand straightforward to convert into real codes in any programminglanguage (e.g., C, FORTRAN, MATLAB, etc.)We use pseudo-code on slides for conciseness
Xiangmin Jiao Numerical Analysis I 19 / 22
Gaxpy: Generalized saxpy
Computes y = y + Ax , where A 2 Rm⇥n and x 2 Rn
Row oriented
for i = 1 : mfor j = 1 : n
yi = yi + aijxj
Column oriented
for j = 1 : nfor i = 1 : m
yi = yi + aijxj
Inner loop of column-oriented algorithm can be converted toy = y + xja:,jNumber of flops is 2mn
Xiangmin Jiao Numerical Analysis I 20 / 22
Matrix Multiplication Update
Computes C = C + AB, where A 2 Rm⇥r , B 2 Rr⇥n, and C 2 Rm⇥n
cij = cij +rX
k=1
aikbkj
for i = 1 : mfor j = 1 : n
for k = 1 : rcij = cij + aikbkj
Number of flops is 2mnrIn BLAS (Basic Linear Algebra Subroutines), functions are groupedinto level-1, 2, and 3, depending on whether complexity is linear,quadratic, or cubic
Xiangmin Jiao Numerical Analysis I 21 / 22
Six Variants of AlgorithmsThere are six variants depending on permutation of i , j , and k :
ijk, jik , ikj , jki , kij , kji
Inner product: cij = cij + ai ,:bj
for i = 1 : mfor j = 1 : n
cij = cij + ai ,:bj
Saxpy: computes as cj = cj + Abj
for j = 1 : ncj = cj + Abj
Outer product: computes as C = C +Pr
k=1
akb:,k
for k = 1 : rC = C + akb:,k
Xiangmin Jiao Numerical Analysis I 22 / 22