Math 408A: Linear Algebra Review

Outline Linear Algebra Review Block Structured Matrices Gaussian Elimination Matrices Gauss-Jordan Elimination (Pivoting)

Linear Algebra Review

Block Structured Matrices

Gaussian Elimination Matrices

Gauss-Jordan Elimination (Pivoting)

Matrices in Rm×n

A ∈ Rm×n

a11 a12 . . . a1na21 a22 . . . a2n...

.... . .

...am1 am2 . . . amn

Matrices in Rm×n

A ∈ Rm×n

a11 a12 . . . a1na21 a22 . . . a2n...

.... . .

...am1 am2 . . . amn

a•1 a•2 . . . a•n]

columns

Matrices in Rm×n

A ∈ Rm×n

a11 a12 . . . a1na21 a22 . . . a2n...

.... . .

...am1 am2 . . . amn

a•1 a•2 . . . a•n]

a1•a2•...

columns rows

Matrices in Rm×n

A ∈ Rm×n

a11 a12 . . . a1na21 a22 . . . a2n...

.... . .

...am1 am2 . . . amn

a•1 a•2 . . . a•n]

a1•a2•...

columns rows

a11 a21 . . . am1

a12 a22 . . . am2...

.... . .

...a1n a2n . . . amn

Matrices in Rm×n

A ∈ Rm×n

a11 a12 . . . a1na21 a22 . . . a2n...

.... . .

...am1 am2 . . . amn

a•1 a•2 . . . a•n]

a1•a2•...

columns rows

a11 a21 . . . am1

a12 a22 . . . am2...

.... . .

...a1n a2n . . . amn

aT•1

aT•2...

aT•n

Matrices in Rm×n

A ∈ Rm×n

a11 a12 . . . a1na21 a22 . . . a2n...

.... . .

...am1 am2 . . . amn

a•1 a•2 . . . a•n]

a1•a2•...

columns rows

a11 a21 . . . am1

a12 a22 . . . am2...

.... . .

...a1n a2n . . . amn

aT•1

aT•2...

aT•n

aT1• aT2• . . . aTm•

Matrix Vector Multiplication

A column space view of matrix vector multiplication.

a11 a12 . . . a1na21 a22 . . . a2n...

.... . .

...am1 am2 . . . amn

x1x2...xn

a11 a12 . . . a1na21 a22 . . . a2n...

.... . .

...am1 am2 . . . amn

x1x2...xn

a11a21...

a11 a12 . . . a1na21 a22 . . . a2n...

.... . .

...am1 am2 . . . amn

x1x2...xn

a11a21...

a12a22...

a11 a12 . . . a1na21 a22 . . . a2n...

.... . .

...am1 am2 . . . amn

x1x2...xn

a11a21...

a12a22...

+ · · ·+ xn

a1na2n...

a11 a12 . . . a1na21 a22 . . . a2n...

.... . .

...am1 am2 . . . amn

x1x2...xn

a11a21...

a12a22...

+ · · ·+ xn

a1na2n...

= x1 a•1 + x2 a•2 + · · · + xn a•n

a11 a12 . . . a1na21 a22 . . . a2n...

.... . .

...am1 am2 . . . amn

x1x2...xn

a11a21...

a12a22...

+ · · ·+ xn

a1na2n...

= x1 a•1 + x2 a•2 + · · · + xn a•n

A linear combination of the columns.

The Range of a Matrix

Let A ∈ Rm×n (an m × n matrix having real entries).

Range of A

Ran(A) = {Ax |x ∈ Rn }

Range of A

Ran(A) = {Ax |x ∈ Rn }

Ran(A) = the linear span of the columns of A

A Couple of Special Subspaces

Let v1, . . . , vk ∈ Rn.

◮ The linear span of v1, . . . , vk :

Span(v1, . . . , vk) = {ξ1v1 + ξ2v2 + · · ·+ ξkvk |ξ1, . . . , ξk ∈ R}

Let v1, . . . , vk ∈ Rn.

SoRan(A) = Span(a•1, . . . , a•n).

Let v1, . . . , vk ∈ Rn.

SoRan(A) = Span(a•1, . . . , a•n).

Dot product of two vectors xT y =∑n

i=1 xiyi .

Let v1, . . . , vk ∈ Rn.

SoRan(A) = Span(a•1, . . . , a•n).

Dot product of two vectors xT y =∑n

i=1 xiyi .

◮ The subspace orthogonal to v1, . . . , vk :

{v1, . . . , vk}⊥ =

z ∈ Rn∣

∣zT vi = 0, i = 1, . . . , k

A row space view of matrix vector multiplication.

a11 a12 . . . a1na21 a22 . . . a2n...

.... . .

...am1 am2 . . . amn

x1x2...xn

aT1•x

aT2•x...

aTm•x

i=1 a1ixi∑n

i=1 a2ixi...

i=1 amixi

The dot product of x with the rows of A.

The Null Space of a Matrix

Null Space of A

Nul(A) = {x ∈ Rn |Ax = 0}

Null Space of A

Nul(A) = {x ∈ Rn |Ax = 0}

Nul(A) = subspace orthogonal to the rows of A

Null Space of A

Nul(A) = {x ∈ Rn |Ax = 0}

= Span(a1•, a2•, . . . , am•)⊥

Null Space of A

Nul(A) = {x ∈ Rn |Ax = 0}

= Span(a1•, a2•, . . . , am•)⊥

= Ran(AT )⊥

Null Space of A

Nul(A) = {x ∈ Rn |Ax = 0}

= Span(a1•, a2•, . . . , am•)⊥

= Ran(AT )⊥

Fundamental Theorem of the Alternative:

Nul(A) = Ran(AT )⊥ Ran(A) = Nul(AT )⊥

Matrix-matrix multiplication

For matrices A ∈ Rm×n and A ∈ R

n×k the product AB ∈ Rm×k is

defined by(AB)ij = aTi•b•j .

Matrix-matrix multiplication

For matrices A ∈ Rm×n and A ∈ R

n×k the product AB ∈ Rm×k is

defined by(AB)ij = aTi•b•j .

Equivalently

AB = [AB•1 AB•2 . . . AB•k ]

3 −4 1 1 0 02 2 0 0 1 0

−1 0 0 0 0 10 0 0 2 1 40 0 0 1 0 3

3 −4 1 1 0 02 2 0 0 1 0

−1 0 0 0 0 10 0 0 2 1 40 0 0 1 0 3

3 −4 1 1 0 02 2 0 0 1 0

−1 0 0 0 0 1

0 0 0 2 1 40 0 0 1 0 3

3 −4 1 1 0 02 2 0 0 1 0

−1 0 0 0 0 10 0 0 2 1 40 0 0 1 0 3

3 −4 1 1 0 02 2 0 0 1 0

−1 0 0 0 0 1

0 0 0 2 1 40 0 0 1 0 3

B I3×3

02×3 C

3 −4 1 1 0 02 2 0 0 1 0

−1 0 0 0 0 10 0 0 2 1 40 0 0 1 0 3

3 −4 1 1 0 02 2 0 0 1 0

−1 0 0 0 0 1

0 0 0 2 1 40 0 0 1 0 3

B I3×3

02×3 C

3 −4 12 2 0−1 0 0

2 1 41 0 3

Multiplication of Block Structured Matrices

Consider the matrix product AM, where

3 −4 1 1 0 02 2 0 0 1 0

−1 0 0 0 0 10 0 0 2 1 40 0 0 1 0 3

and M =

1 20 4

−1 −12 −14 3

−2 0

3 −4 1 1 0 02 2 0 0 1 0

−1 0 0 0 0 10 0 0 2 1 40 0 0 1 0 3

and M =

1 20 4

−1 −12 −14 3

−2 0

Can we exploit the structure of A?

3 −4 1 1 0 02 2 0 0 1 0

−1 0 0 0 0 10 0 0 2 1 40 0 0 1 0 3

and M =

1 20 4

−1 −12 −14 3

−2 0

3 −4 1 1 0 02 2 0 0 1 0

−1 0 0 0 0 10 0 0 2 1 40 0 0 1 0 3

and M =

1 20 4

−1 −12 −14 3

−2 0

B I3×3

02×3 C

3 −4 1 1 0 02 2 0 0 1 0

−1 0 0 0 0 10 0 0 2 1 40 0 0 1 0 3

and M =

1 20 4

−1 −12 −14 3

−2 0

B I3×3

02×3 C

so take M =

3 −4 1 1 0 02 2 0 0 1 0

−1 0 0 0 0 10 0 0 2 1 40 0 0 1 0 3

and M =

1 20 4

−1 −12 −14 3

−2 0

B I3×3

02×3 C

so take M =

where X =

1 20 4

−1 −1

, and Y =

2 −14 3

−2 0

B I3×3

02×3 C

B I3×3

02×3 C

BX + Y

B I3×3

02×3 C

BX + Y

2 −112 12

−1 −2

2 −14 3

−2 0

0 1−4 −1

B I3×3

02×3 C

BX + Y

2 −112 12

−1 −2

2 −14 3

−2 0

0 1−4 −1

4 −126 15

−3 −20 1

−4 −1

Solving Systems of Linear equations

Let A ∈ Rm×n and b ∈ R

m.Find all solutions x ∈ R

n to the system Ax = b.

The set of solutions is either.

The set of solutions is either empty,.

The set of solutions is either empty, a single point,.

The set of solutions is either empty, a single point, or an infiniteset.

If a solution x0 ∈ Rn exists, then the set of solutions is given by

x0 + Nul(A) .

Gaussian Elimination and the 3 Elementary Row

Operations

We solve the system Ax = b by transforming the augmented matrix

[A | b]

into upper echelon form using the three elementary row operations.

Operations

[A | b]

This process is called Gaussian elimination.

Operations

[A | b]

The three elementary row operations.

Operations

[A | b]

1. Interchange any two rows.

Operations

[A | b]

2. Multiply any row by a non-zero constant.

Operations

[A | b]

3. Replace any row by itself plus a multiple of any other row.

Operations

[A | b]

3. Replace any row by itself plus a multiple of any other row.

These elementary row operations can be interpreted as multiplyingthe augmented matrix on the left by a special matrix.

Exchange and Permutation Matrices

An exchange matrix is given by permuting any two columns of theidentity.

Multiplying any 4× n matrix on the left by the exchange matrix

1 0 0 00 0 0 10 0 1 00 1 0 0

will exchange the second and fourth rows of the matrix.

1 0 0 00 0 0 10 0 1 00 1 0 0

(multiplication of a m × 4 matrix on the right by this exchangesthe second and fourth columns.)

1 0 0 00 0 0 10 0 1 00 1 0 0

(multiplication of a m × 4 matrix on the right by this exchangesthe second and fourth columns.)A permutation matrix is obtained by permuting the columns of theidentity matrix.

Notes on Matrix Multiplication

Let A = [aij ]m×n ∈ Rm×n.

Left Multiplication of A:

Left Multiplication of A:When multiplying A on the left by an m ×m matrix M, it is oftenuseful to think of this as an action on the rows of A.

For example, left multiplication by a permutation matrix permutesthe rows of the matrix.

However, mechanically, left multiplication corresponds to matrixvector multiplication on the columns.

MA = [Ma•1 Ma•2 · · · Ma•n]

Right Multiplication of A:

Right Multiplication of A:When multiplying A on the right by an n × n matrix N, it is oftenuseful to think of this as an action on the columns of A.

For example, right multiplication by a permutation matrixpermutes the columns of the matrix.

However, mechanically, rightmultiplication corresponds toleft matrix vector multiplicationon the rows.

a1•N

a2•N...

am•N

The key step in Gaussian elimination is to transform a vector ofthe form

where a ∈ Rk , 0 6= α ∈ R, and b ∈ R

n−k−1, into one of the form

The key step in Gaussian elimination is to transform a vector ofthe form

where a ∈ Rk , 0 6= α ∈ R, and b ∈ R

n−k−1, into one of the form

This can be accomplished by left matrix multiplication as follows.

a ∈ Rk , 0 6= α ∈ R, and b ∈ R

n−k−1

Ik×k 0 00 1 00 −α−1b I(n−k−1)×(n−k−1)

a ∈ Rk , 0 6= α ∈ R, and b ∈ R

n−k−1

Ik×k 0 00 1 00 −α−1b I(n−k−1)×(n−k−1)

a ∈ Rk , 0 6= α ∈ R, and b ∈ R

n−k−1

Ik×k 0 00 1 00 −α−1b I(n−k−1)×(n−k−1)

a ∈ Rk , 0 6= α ∈ R, and b ∈ R

n−k−1

Ik×k 0 00 1 00 −α−1b I(n−k−1)×(n−k−1)

a ∈ Rk , 0 6= α ∈ R, and b ∈ R

n−k−1

Ik×k 0 00 1 00 −α−1b I(n−k−1)×(n−k−1)

The matrix

Ik×k 0 00 1 00 −α−1b I(n−k−1)×(n−k−1)

is called a Gaussian elimination matrix.

The matrix

Ik×k 0 00 1 00 −α−1b I(n−k−1)×(n−k−1)

This matrix is invertible with inverse

Ik×k 0 00 1 00 α−1b I(n−k−1)×(n−k−1)

The matrix

Ik×k 0 00 1 00 −α−1b I(n−k−1)×(n−k−1)

This matrix is invertible with inverse

Ik×k 0 00 1 00 α−1b I(n−k−1)×(n−k−1)

Note that a Gaussian elimination matrix and its inverse are bothlower triangular matrices.

Matrix Sub-Algebras

Lower (upper) triangular matrices in Rn×n are said to form a

sub-algebra of Rn×n.

Matrix Sub-Algebras

A subset S of Rn×n is said to be a sub-algebra of Rn×n if

Matrix Sub-Algebras

◮ S is a subspace of Rn×n,

Matrix Sub-Algebras

◮ S is closed wrt matrix multiplication, and

Matrix Sub-Algebras

◮ S is closed wrt matrix multiplication, and

◮ if M ∈ S is invertible, then M−1 ∈ S .

Gaussian Elimination in Practice

Transformation to echelon(upper triangular) form.

1 1 22 4 2

−1 1 3

1 1 22 4 2

−1 1 3

Eliminate the first column witha Gaussian elimination matrix.

1 1 22 4 2

−1 1 3

1 0 0−2 1 01 0 1

1 1 22 4 2

−1 1 3

1 0 0−2 1 01 0 1

1 1 22 4 2

−1 1 3

1 1 22 4 2

−1 1 3

1 0 0−2 1 01 0 1

1 1 22 4 2

−1 1 3

1 1 22 4 2

−1 1 3

1 0 0−2 1 01 0 1

1 1 22 4 2

−1 1 3

1 1 22 4 2

−1 1 3

1 0 0−2 1 01 0 1

1 1 22 4 2

−1 1 3

1 1 22 4 2

−1 1 3

1 0 0−2 1 01 0 1

1 1 22 4 2

−1 1 3

1 1 22 4 2

−1 1 3

1 0 0−2 1 01 0 1

1 1 22 4 2

−1 1 3

1 1 20

1 1 22 4 2

−1 1 3

1 0 0−2 1 01 0 1

1 1 22 4 2

−1 1 3

1 1 20 2

1 1 22 4 2

−1 1 3

1 0 0−2 1 01 0 1

1 1 22 4 2

−1 1 3

1 1 20 2 − 2

1 1 22 4 2

−1 1 3

1 0 0−2 1 01 0 1

1 1 22 4 2

−1 1 3

1 1 20 2 − 20 2 5

Now do Gaussian eliminiation on the second column.

1 1 20 2 −20 2 5

1 0 00 1 00 −1 1

1 1 20 2 −20 2 5

1 0 00 1 00 −1 1

1 1 20 2 −20 2 5

1 0 00 1 00 −1 1

1 1 20 2 −20 2 5

1 0 00 1 00 −1 1

1 1 20 2 −20 2 5

1 0 00 1 00 −1 1

1 1 20 2 −20 2 5

1 0 00 1 00 −1 1

1 1 20 2 −20 2 5

1 1 200

1 1 20 2 −20 2 5

1 0 00 1 00 −1 1

1 1 20 2 −20 2 5

1 1 20 20

1 1 20 2 −20 2 5

1 0 00 1 00 −1 1

1 1 20 2 −20 2 5

1 1 20 2 − 20

1 1 20 2 −20 2 5

1 0 00 1 00 −1 1

1 1 20 2 −20 2 5

1 1 20 2 − 20 0 7

Gauss-Jordan Elimination, or Pivot Matrices

What happens in the following multiplication?

Ik×k −α−1a 00 α−1 00 −α−1b I(n−k−1)×(n−k−1)

What is the inverse of this matrix?

Ik×k −α−1a 00 α−1 00 −α−1b I(n−k−1)×(n−k−1)

What is the inverse of this matrix?

Ik×k a 00 α 00 b I(n−k−1)×(n−k−1)

Math 408A: Linear Algebra Review

Documents

Algebra Summer Math Packet

EDS-405A/408A Series Hardware Installation Guidestore.express-inc.com/pdf/EDS-405A-408A-installationguide.pdf · - 2 - Overview The Moxa EtherDevice™ EDS-405A/408A series, which

MATH 103 COLLEGE ALGEBRA MATH 88 COLLEGE ......MATH 103 – COLLEGE ALGEBRA MATH 88 – COLLEGE ALGEBRA COMPANION 4 Credits, CRN: 61062 & 61217 / 2 Credits, CRN: 61216 MW/F, 5:30 pm

Algebra Math Notes Abstract Algebra - MIT ESP Math Notes • Study Guide Abstract Algebra Table of Contents Rings ..... 3

Math 408A Line Search Methods - University of Washington

Summer Math Practice 2020: 7 Pre-Algebra Entering 8 Algebra 1 · 2020-06-01 · Summer Math Practice 2020: 7th Pre-Algebra Entering 8th Algebra 1 Dear Students and Parents, This math

Math 090 Beginning Algebra

Math Explorations Algebra I

Math Review Packet for Pre-Algebra to Algebra 1

Math Workbook Algebra - Free Worksheets and Math

ACT 1-on-1 Curriculum Math: Pre-algebra & Algebra

Math 152, Intermediate Algebra Practice Problems #13jkrause/Web152/Pprobs13-15.pdf · Math 152, Intermediate Algebra Practice Problems #13 ... Math 152, Intermediate Algebra Practice

Algebra & Math Power

MATH 1314 College Algebra

MATH 101B: ALGEBRA II PART A: HOMOLOGICAL ALGEBRA

Math Review 2 Algebra

Math 324: Linear Algebra

Algebra Math

Math /Algebra Talks Mental Math Strategies Math /Algebra Talks: Mental Math Strategies

Hanlonmath – Math help online, math tutorials : algebra