6

6.5

© 2012 Pearson Education, Inc.

Orthogonality and Least Squares

LEAST-SQUARES PROBLEMS

Slide 6.5- 2 © 2012 Pearson Education, Inc.

LEAST-SQUARES PROBLEMS

Definition: If A is and b is in , a least-squares solution of is an in such that

for all x in .

The most important aspect of the least-squares problem is that no matter what x we select, the vector Ax will necessarily be in the column space, Col A.

So we seek an x that makes Ax the closest point in Col A to b. See the figure on the next slide.

m nx bA x̂ˆb x b xA A

Slide 6.5- 3 © 2012 Pearson Education, Inc.

LEAST-SQUARES PROBLEMS

Solution of the General Least-Squares Problem

Given A and b, apply the Best Approximation Theorem to the subspace Col A.

Let Col b̂ proj bA

Slide 6.5- 4 © 2012 Pearson Education, Inc.

SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM

Because is in the column space A, the equation is consistent, and there is an in such that

----(1)

Since is the closest point in Col A to b, a vector is a least-squares solution of if and only if satisfies (1).

Such an in is a list of weights that will build out of the columns of A. See the figure on the next slide.

b̂ ˆx bA x̂

ˆx̂ bA

b̂ x̂x bA x̂

x̂ n b̂

Slide 6.5- 5 © 2012 Pearson Education, Inc.

SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM

Suppose satisfies . By the Orthogonal Decomposition Theorem, the

projection has the property that is orthogonal to Col A, so is orthogonal to each column of A.

If aj is any column of A, then , and .

x̂ ˆx̂ bA

b̂ ˆb bˆb xA

ˆa (b x) 0j A ˆa (b x)T

j A

Slide 6.5- 6 © 2012 Pearson Education, Inc.

SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM

Since each is a row of AT, ----(2) Thus

These calculations show that each least-squares solution of satisfies the equation

----(3) The matrix equation (3) represents a system of

equations called the normal equations for . A solution of (3) is often denoted by .

aTjˆ(b x) 0TA A

ˆb x 0

x̂ b

T T

T T

A A A

A A A

x bA x bT TA A A

x bA x̂

Slide 6.5- 7 © 2012 Pearson Education, Inc.

SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM

Theorem 13: The set of least-squares solutions of coincides with the nonempty set of solutions of the normal equation .

Proof: The set of least-squares solutions is nonempty and each least-squares solution satisfies the normal equations.

Conversely, suppose satisfies . Then satisfies (2), which shows that is

orthogonal to the rows of AT and hence is orthogonal to the columns of A.

Since the columns of A span Col A, the vector is orthogonal to all of Col A.

x bA

x bT TA A A

x̂

x̂ x̂ bT TA A Ax̂ ˆb xA

ˆb xA

Slide 6.5- 8 © 2012 Pearson Education, Inc.

SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM

Hence the equation

is a decomposition of b into the sum of a vector in Col A and a vector orthogonal to Col A.

By the uniqueness of the orthogonal decomposition, must be the orthogonal projection of b onto Col A.

That is, and is a least-squares solution.

ˆ ˆb x (b x)A A

x̂A

ˆx̂ bA x̂

Slide 6.5- 9 © 2012 Pearson Education, Inc.

SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM Example 1: Find a least-squares solution of the

inconsistent system for

Solution: To use normal equations (3), compute:

x bA 4 0 2

0 2 ,b 0

1 1 11

A

4 04 0 1 17 1

0 20 2 1 1 5

1 1

TA A

Slide 6.5- 10 © 2012 Pearson Education, Inc.

SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM

Then the equation becomes

24 0 1 19

b 00 2 1 11

11

TA

x bT TA A A

1

2

17 1 19

1 5 11

x

x

Slide 6.5- 11 © 2012 Pearson Education, Inc.

SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM

Row operations can be used to solve the system on the previous slide, but since ATA is invertible and , it is probably faster to compute

and then solve as

2 2

15 11

( )1 1784

TA A

x bT TA A A1x̂ ( ) b

5 1 19 84 11 1=

1 17 11 168 284 84

T TA A A

Slide 6.5- 12 © 2012 Pearson Education, Inc.

SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM

Theorem 14: Let A be an matrix. The following statements are logically equivalent:

a. The equation has a unique least-squares solution for each b in .

b. The columns of A are linearly independent.c. The matrix ATA is invertible.When these statements are true, the least-squares solution is given by

----(4)a. When a least-squares solution is used to produce

as an approximation to b, the distance from b to is called the least-squares error of this approximation.

m n

x bA m

x̂1x̂ ( ) bT TA A Ax̂

x̂Ax̂A

Slide 6.5- 13 © 2012 Pearson Education, Inc.

ALTERNATIVE CALCULATIONS OF LEAST-SQUARES SOLUTIONS

Example 2: Find a least-squares solution of for

Solution: Because the columns a1 and a2 of A are orthogonal, the orthogonal projection of b onto Col A is given by

----(5)

x bA 1 6 1

1 2 2,b

1 1 1

1 7 6

A

1 21 2 1 2

1 1 2 2

b a b a 8 45b̂ a a a a

a a a a 4 90

Slide 6.5- 14 © 2012 Pearson Education, Inc.

ALTERNATIVE CALCULATIONS OF LEAST-SQUARES SOLUTIONS

Now that is known, we can solve . But this is trivial, since we already know weights to

place on the columns of A to produce . It is clear from (5) that

2 3 1

2 1 1

2 1/ 2 5 / 2

2 7 / 2 11 / 2

b̂ ˆx̂ bA

b̂

8 / 4 2ˆ

45 / 90 1/ 2x

Slide 6.5- 15 © 2012 Pearson Education, Inc.

ALTERNATIVE CALCULATIONS OF LEAST-SQUARES SOLUTIONS

Theorem 15: Given an matrix A with linearly independent columns, let be a QR factorization of A. Then, for each b in , the equation has a unique least-squares solution, given by

----(6)

Proof: Let .

Then

m nA QRm x bA

1x̂ bTR Q

1x̂ bTR Q

1ˆ ˆx x b bT TA QR QRR Q QQ

Slide 6.5- 16 © 2012 Pearson Education, Inc.

ALTERNATIVE CALCULATIONS OF LEAST-SQUARES SOLUTIONS

The columns of Q form an orthonormal basis for Col A.

Hence, by Theorem 10, QQTb is the orthogonal projection of b onto Col A.

Then , which shows that is a least-squares solution of .

The uniqueness of follows from Theorem 14.

b̂

ˆx̂ bA x̂x bA

x̂