Lecture 12 orhogonality - 6.1 6.2 6.3

6

6.1

© 2012 Pearson Education, Inc.

Orthogonality

INNER PRODUCT, LENGTH,

AND ORTHOGONALITY

Slide 6.1- 2© 2012 Pearson Education, Inc.

INNER PRODUCT

If u and v are vectors in Rn, then we regard uand v as nx1 matrices

The number uTv is called the inner product of u and v, and it is written as u∙v

aka dot product

Scalar

Slide 6.1- 3© 2012 Pearson Education, Inc.

INNER PRODUCT

If 𝐮 =

𝑢1

𝑢2

⋮𝑢𝑛

and 𝐯 =

𝑣1

𝑣2

⋮𝑣𝑛

then the inner product of u and v is

𝑢1 𝑢2 ⋯ 𝑢𝑛

𝑣1

𝑣2

⋮𝑣𝑛

= 𝑢1𝑣1 + 𝑢2𝑣2 + ⋯+ 𝑢𝑛𝑣𝑛

Inner Product - Example

Find u∙v & v∙u, for:

𝐮 =2−5−1

, 𝐯 =323

Slide 6.1- 4© 2012 Pearson Education, Inc.

INNER PRODUCT - Properties

Theorem 1: Let u, v, and w be vectors in Rn, and let c

be a scalar. Then

a. u•v = v•u

b. (u + v)•w = u•w + v•w

c. (cu)•v = c(u•v) = u•(cv)

d. u•u ≥0, and =0 iff u = 0

Properties (b) and (c) can be combined several

times to produce the following useful rule:

(c1u1 + c2u2 + … + cnun)•v = c1(u1•v) + … + cp(up•v)

Slide 6.1- 6© 2012 Pearson Education, Inc.

THE LENGTH OF A VECTOR

Definition: The length (or norm) of v:

𝐯 = 𝐯 ∙ 𝐯 = 𝑣12 + 𝑣2

2 + ⋯+ 𝑣𝑛2

||cv|| = |c| ||v||

Unit vector has length 1

Any vector divided by its length is a unit vector in same direction normalizing

Slide 6.1- 7© 2012 Pearson Education, Inc.

Normalization – Example 1

Example 1: Let v = (1, -2, 2, 0). Find a unit vector

u in the same direction as v.

Slide 6.1- 8© 2012 Pearson Education, Inc.

Normalization – Example 2

Example 2: Let W be a subspace of R2 spanned

by x = 2/31

. Find a unit vector z that is a basis for

W.

Slide 6.1- 9© 2012 Pearson Education, Inc.

DISTANCE IN Rn

Definition: For u and v in Rn, the distance between u

and v, written as dist (u, v), is the length of the vector

u-v. i.e.,

dist(u, v) = ||u – v||

Example: Find the distance between u=(7,1) & v=(3,2)

Orthogonal Vectors

Definition: 2 vectors u and v are orthogonal

if u•v = 0.

Theorem 6-2: Pythagorean Theorem: 2

vectors u and v are orthogonal iff

𝐮 + 𝐯 2 = 𝐮 2 + 𝐯 2

Slide 6.1- 10© 2012 Pearson Education, Inc.

Slide 6.1- 11© 2012 Pearson Education, Inc.

Orthogonal Complement

If a vector z is orthogonal to every vector in a

subspace W of Rn, then z is said to be

orthogonal to W.

The set of all vectors z that are orthogonal to W

is called the orthogonal complement of W and

is denoted by W┴ (and read as “W

perpendicular” or simply “W perp”).

ORTHOGONAL COMPLEMENTS

Theorem 3: Let A be an mxn matrix. The orthogonal

complement of the row space of A is the null space of

A, and the orthogonal complement of the column

space of A is the null space of AT:

Proof: ai = ith row of A

Row(A) = Span{rows of A}

Nul(A) = x:Ax = 0

ai • x = 0 for every i

x is orthogonal to each row of A

(Row ) Nul A A

(Col ) Nul TA A

Slide 6.1- 13© 2012 Pearson Education, Inc.

ANGLES & Inner Products

u•v = ||u|| ||v|| cosϑ

Slide 6.1- 14© 2012 Pearson Education, Inc.

ANGLES & Inner Products - Example

Find the angle between 110

,011

6

6.1

© 2012 Pearson Education, Inc.

Orthogonal Sets

Slide 6.2- 16© 2012 Pearson Education, Inc.

ORTHOGONAL SETS

A set of vectors {u1,…,up} in Rn is said to be an

orthogonal set if each pair of distinct vectors

from the set is orthogonal, i.e, if

ui •uj = 0 if i ≠ j.

Orthogonal Sets - Example

Show that S={u1,u2,u3} is an orthogonal set.

𝐮1 =311

, 𝐮2 =−121

, 𝐮1 =−1/2−27/2

Slide 6.1- 17© 2012 Pearson Education, Inc.

Theorem 6-4 – Orthogonal Set as a Basis

Theorem 4: If S = {u1, …, up} is an

orthogonal set of nonzero vectors in Rn ,

then S is linearly independent and hence is

a basis for the subspace spanned by S.

Slide 6.1- 18© 2012 Pearson Education, Inc.

Slide 6.2- 19© 2012 Pearson Education, Inc.

ORTHOGONAL Basis

Definition: An orthogonal basis for a subspace W

of Rn is a basis for W that is also an orthogonal set.

Theorem 5: Let {u1,…,up} be an orthogonal basis for

a subspace W of Rn. For each y in W, the weights in

the linear combination

are given by y = c1u1+ … + cpup

𝑐𝑗 =𝐲∙𝐮𝒋

𝐮𝒋∙𝐮𝒋(𝑗 = 1,… , 𝑝)

Orthogonal Basis – Example 2

Same S as in Example 1:

𝐮1 =311

, 𝐮2 =−121

, 𝐮1 =−1/2−27/2

write 𝐲 =61−8

as a linear combination of the

vectors in S.

Slide 6.1- 20© 2012 Pearson Education, Inc.

Slide 6.2- 21© 2012 Pearson Education, Inc.

AN ORTHOGONAL PROJECTION

Decompose y into 2 vectors:

One in direction of u ŷ = αu

One orthogonal to u z = y - ŷ

Orthogonal Projections

ŷ is the projection of y onto u

If L is subspace spanned by u

ŷ is the projection of y onto L

𝐲 = proj𝐿𝐲 =𝐲 ∙ 𝐮

𝐮 ∙ 𝐮𝐮

Slide 6.1- 22© 2012 Pearson Education, Inc.

Slide 6.2- 23© 2012 Pearson Education, Inc.

ORTHOGONAL PROJECTION - Example

Example 1: 𝐲 =76

, 𝐮 =42

a) Find the orthogonal projection of y onto u.

b) Write y as the sum of two orthogonal vectors,

one in Span {u} and one orthogonal to u.

ORTHOGONAL PROJECTION - Example

7 8 1

6 4 2

yy ˆ(y y)

Slide 6.2- 25© 2012 Pearson Education, Inc.

AN ORTHOGONAL PROJECTION

Check: ŷ•z = 0?

Distance from y to L?

Distance from y to u?

Slide 6.2- 26© 2012 Pearson Education, Inc.

ORTHONORMAL SETS

A set {u1,…,up} is an orthonormal set if it is an

orthogonal set of unit vectors (||u||=1).

If W is the subspace spanned by such a set, then

{u1,…,up} is an orthonormal basis for W, since

the set is automatically linearly independent, by

Theorem 4.

The simplest example of an orthonormal set is the

standard basis {e1,…,en} for Rn.

Slide 6.2- 27© 2012 Pearson Education, Inc.

ORTHONORMAL SETS - Example

Example 2: Show that {v1, v2, v3} is an

orthonormal basis of R3, where

1

3 / 11

v 1/ 11

1/ 11

2

1/ 6

v 2 / 6

1/ 6

3

1/ 66

v 4 / 66

7 / 66

Slide 6.2- 28© 2012 Pearson Education, Inc.

ORTHONORMAL SETS

When the vectors in an orthogonal set of

nonzero vectors are normalized to have unit

length, the new vectors will still be orthogonal,

and hence the new set will be an orthonormal

set.

Slide 6.2- 29© 2012 Pearson Education, Inc.

ORTHONORMAL SETS

Theorem 6: An mxn matrix U has orthonormal columns if and only if UTU = I

Slide 6.2- 30© 2012 Pearson Education, Inc.

ORTHONORMAL SETS

Theorem 7: Let U be an mxn matrix with

orthonormal columns, and let x and y be in Rn.

Then

a. ||Ux|| = ||x||

b. (Ux)•(Uy) = x•y

c. (Ux)•(Uy) = 0 iff x•y=0

Properties (a) and (c) say that the linear

mapping x Ux preserves lengths and

orthogonality.

Orthonormal Sets - Example

𝑈 =

1 2 2 3

1 2 2/30 1/3

, 𝐱 = 23

Slide 6.1- 31© 2012 Pearson Education, Inc.

Orthogonal Matrix

If U is square and U-1=UT

Called “Orthogonal Matrix”

Really should be Orthonormal matrix

Orthonormal columns

Orthonormal rows as well

Slide 6.1- 32© 2012 Pearson Education, Inc.

6

6.1

© 2012 Pearson Education, Inc.

Orthogonal Projections

Slide 6.3- 34© 2012 Pearson Education, Inc.

ORTHOGONAL PROJECTIONS

Extend ideas of previous section from R2 to Rn

Given a vector y and a subspace W in Rn, there

is a vector ŷ in W such that

ŷ is the unique vector in W for which y - ŷ is

orthogonal to W

ŷ is the unique vector in W closest to y. See the

following figure. y

Slide 6.3- 35© 2012 Pearson Education, Inc.

THE ORTHOGONAL DECOMPOSITION

THEOREM

Theorem 8: Let W be a subspace of Rn . Then

each y in Rn can be written uniquely in the form

y = ŷ + z

where ŷ is in W and z is in W┴ .

If {u1,…,up} is any orthogonal basis of W, then

𝐲 =𝐲 ∙ 𝐮1

𝐮1 ∙ 𝐮1𝐮1 + ⋯+

𝐲 ∙ 𝐮𝑝

𝐮𝑝 ∙ 𝐮𝑝𝐮𝑝

and

z = y - ŷ

Slide 6.3- 36© 2012 Pearson Education, Inc.

THE ORTHOGONAL DECOMPOSITION

THEOREM

The vector ŷ is called the orthogonal

projection of y onto W and often is written as

projWy. See the following figure.

Slide 6.3- 37© 2012 Pearson Education, Inc.

THE ORTHOGONAL DECOMPOSITION

THEOREM - Example

Example 1: Write y as the sum of a vector in W =

span{u1,u2}, and a vector orthogonal to W.

𝐮1 =25−1

, 𝐮2 =−211

, 𝐲 =123

Slide 6.3- 38© 2012 Pearson Education, Inc.

PROPERTIES OF ORTHOGONAL

PROJECTIONS

If {u1,…,up} is an orthogonal basis for W and if y

happens to be in W, then the formula for projWy

is exactly the same as the representation of y

given in Theorem 5 in Section 6.2.

If y is in W = Span {u1,…,up}, projwy = y

Slide 6.3- 39© 2012 Pearson Education, Inc.

THE BEST APPROXIMATION THEOREM

Theorem 9: Let W be a subspace of Rn, let y be any

vector in Rn, and let ŷ be the orthogonal projection of

y onto W. Then ŷ is the closest point in W to y, in the

sense that

||y – ŷ|| < ||y – v||

for all v in W distinct from ŷ.

The Best Approximation Theorem

ŷ in Theorem 9 is called the best

approximation to y by elements of W.

The distance from y to v, given by ||y – v||,

can be regarded as the “error” of using v in

place of y.

Theorem 9 says that this error is minimized

when v=ŷ

Note – does not depend of basis of W used

Slide 6.1- 40© 2012 Pearson Education, Inc.

Slide 6.3- 41© 2012 Pearson Education, Inc.

THE BEST APPROXIMATION THEOREM

Slide 6.3- 42© 2012 Pearson Education, Inc.

Best Approximation - Example

Example 2: Find the distance from y to W = Span

{u1,u2} for:

𝐮1 =5−21

, 𝐮2 =12−1

, 𝐲 =−1−510