5 - 1
3.8 Inner Product Spaces3.8 Inner Product Spaces
Euclidean Euclidean nn-space:-space:
Rn was defined to be the set of all ordered n-tuples of real
numbers. When Rn is combined with the standard operations of
vector addition, scalar multiplication, vector lengthvector length, and the dot dot
productproduct, the resulting vector space is called Euclidean Euclidean nn-space-space.
1 1 2 2u v n nu v u v u v L
The dot product of two vectors is defined to be
The definitions of the vector length and the dot product are needed to provide the metric conceptthe metric concept for the vector space.
5 - 2
(1)
(2)
(3)
(4) and if and only if
〉〈〉〈 uvvu ,,
〉〈〉〈〉〈 wuvuwvu ,,, 〉〈〉〈 vuvu ,, cc
0, 〉〈 vv 0, 〉〈 vv 0v
Axioms of inner product:Axioms of inner product:
Let u, v, and w be vectors in a vector space V, and let c be
any scalar. An inner product on An inner product on VV is a function that is a function that
associates a real number <u, v> with each pair of vectors associates a real number <u, v> with each pair of vectors
u and vu and v and satisfies the following axioms.
5 - 3
Note:
V
Rn
space for vectorproduct inner general,
)for product inner Euclidean (productdot
vu
vu
Note:
A vector space V with an inner product is called an inner
product space.
, ,V Vector space:
Inner product space: , , , ,V
5 - 4
Ex: (A different inner product for Rn)
Show that the function defines an inner product on R2, where and .
2211 2, vuvu 〉〈 vu),( ),( 2121 vvuu vu
Sol: 〉〈〉〈 uvvu ,22, )( 22112211 uvuvvuvua
〉〈〉〈
〉〈
wuvu
wvu
,,
)2()2(
22
)(2)(,
22112211
22221111
222111
wuwuvuvu
wuvuwuvu
wvuwvu),( )( 21 wwb w
〉〈〉〈 vuvu ,)(2)()2(, )( 22112211 cvcuvcuvuvuccc
02, )( 22
21 vvd 〉〈 vv
)0(0020, 212
22
1 vvv vvvv〉〈
5 - 5
Ex: (A function that is not an inner product)
Show that the following function is not an inner product on R3.
332211 2 vuvuvu 〉〈 vu
Sol:
Let )1,2,1(v
06)1)(1()2)(2(2)1)(1(,Then vv
Axiom 4 is not satisfied.
Thus this function is not an inner product on R3.
5 - 6
Ex: (A function that is not an inner product)
Show that the following function is not an inner product on R3.
332211 2 vuvuvu 〉〈 vu
Sol:
Let )1,2,1(v
06)1)(1()2)(2(2)1)(1(,Then vv
Axiom 4 is not satisfied.
Thus this function is not an inner product on R3.
5 - 7
Norm (length) of u:
〉〈 uuu ,|||| 〉〈 uuu ,|||| 2
(5) If A and B are two matrices, an inner product can be <A,B>=Tr(A†B), where † is the transpose complex conjugate of the matrix and Tr means the trace. Therefore
For a norm, there are many possibilities.
5 - 8
For a norm, there are many possibilities.
There is an example in criminal law in which the distinctions between some of these norms has very practical consequences. If you’re caught selling drugs in New York there is a longer sentence if your sale is within 1000 feet of a school. If you are an attorney defending someone accused of this crime, which of the norms would you argue for? The legislators didn’t know linear algebra, so they didn’t specify which norm they intended. The prosecuting attorney argued for norm #1, “as the crow flies.” The defense argued that “crows don’t sell drugs” and humans move along city streets, so norm #2 is more appropriate.
The New York Court of Appeals decided that the Pythagorean norm (#1) is the appropriate one and they rejected the use of the pedestrian norm that the defendant advocated (#2).
5 - 9
DistanceDistance between u and v:
vuvuvuvu ,||||),(d
AngleAngle between two nonzero vectors u and v:
0,||||||||
,cos
vu
vu 〉〈
u and v are orthogonal if .
0, 〉〈 vu
OrthogonalOrthogonal: )( vu
Note:Note: If , then v is called a unit vectorunit vector. 1|||| v
0
1
v
v gNormalizin
v
v (the unit vector in the direction of v)
5 - 10
Properties of norm:Properties of norm:
(1)
(2) if and only if
(3)
Properties of distance:Properties of distance:
(1)
(2) if and only if
(3)
0|||| u
0|||| u 0u
|||||||||| uu cc
0),( vud
0),( vud vu
),(),( uvvu dd
5 - 11
Ex: (Finding inner product)
)(in spolynomial be24)(,21)(Let 222 xPxxxqxxp
0 0 1 1, n np q a b a b a b L is an inner product
( ) , ?a p q 〈 〉 ( ) || || ?b q ( ) ( , ) ?c d p q
Sol:Sol:( ) , (1)(4) (0)( 2) ( 2)(1) 2a p q 〈 〉
2 2 2( ) || || , 4 ( 2) 1 21b q q q 〈 〉2
2 2 2
( ) 3 2 3
( , ) || || ,
( 3) 2 ( 3) 22
c p q x x
d p q p q p q p q
Q
2: (1,0, 2) (4, 2,1) {1, , }Note p and q with basis x x
5 - 12
ThmThm 3.21 3.21:
Let u and v be vectors in an inner product space V.
(1) Cauchy-Schwarz inequalityCauchy-Schwarz inequality:
(2) Triangle inequalityTriangle inequality:
(3) Pythagorean theoremPythagorean theorem :
u and v are orthogonal if and only if
||u v|| ||u|| ||v||
222 |||||||||||| vuvu
| u , v | ||u|| ||v||〈 〉
5 - 13
NoteNote:We can solve for this coefficient by noting that because is orthogonal to a scalar multiple of , it must be orthogonal to itself. Therefore, the consequent fact that the dot product is zero, giving that
Orthogonal projectionsOrthogonal projections in inner product spaces:
Let u and v be two vectors in an inner product space V,
such that . Then the orthogonal projection of u onto
v is given by
0v
,proj ,
,
v
u vu u v v
v vv
u
s
sr
s( Proj )v v s rr r r
g
sProjv v rr r
proj ,s v v s s rr r r r
sr
sr
5 - 14
Ex: (Finding an orthogonal projection in R3)
Use the Euclidean inner product in R3 to find the
orthogonal projection of u=(6, 2, 4) onto v=(1, 2, 0).
Sol:u , v (6)(1) (2)(2) (4)(0) 10 Q
2 2 2v , v 1 2 0 5
proj 10v 5
u vu v (1 , 2 , 0) (2 , 4 , 0)
v v
5 - 15
ThmThm 3.22 3.22: (Orthogonal projection and distance)
Let v and s be two vectors in an inner product space V,
such that . Then 0s
proj,
( , ) ( , ) , ,s
v sd v v d v cs c
s s
r
r rr r r r
r r
5 - 16
3.93.9 Orthonormal Bases: Gram-Schmidt ProcessOrthonormal Bases: Gram-Schmidt Process OrthogonalOrthogonal:
A set S of vectors in an inner product space V is called an
orthogonal setorthogonal set if every pair of vectors in the set is
orthogonal.
OrthonormalOrthonormal:
An orthogonal set in which each vector is a unit vectorunit vector is
called orthonormal.
1 2
1v , v , , v v , v
0n i j
i jS V
i j
1 2v , v , , v v , v 0n i jS V L ji
5 - 17
The standard basis is orthonormal.
Ex: (An orthonormal basis for )
In , with the inner product
)(3 xP
0 0 1 1 2 2,p q a b a b a b
21 2 3{1, , } { , , }B x x v v v
3( )P x
Sol:Sol: ,001 21 xx v ,00 2
2 xx v ,00 23 xx v
1 2
1 3
2 3
, (1)(0) (0)(1) (0)(0) 0,
, (1)(0) (0)(0) (0)(1) 0,
, (0)(0) (1)(0) (0)(1) 0
v v
v v
v v
Then 1110000
,1001100
,1000011
333
222
111
v,vv
v,vv
v,vv
Thus, B is an orthonormal basis for .)(3 xP
5 - 18
Thm 3.23:Thm 3.23: (Orthogonal sets are linearly independentOrthogonal sets are linearly independent)
If is an orthogonal setorthogonal set of nonzero vectors
in an inner product space V, then S is linearly independentlinearly independent.
1 2v , v , , vnS L
Pf:
S is an orthogonal set of nonzero vectors
0and0i.e. iiji ji v,vv,v
1 1 2 2
1 1 2 2
Let v v v 0
v v v , v 0, v 0n n
n n i i
c c c
c c c i
L
L
1 1 2 2 v , v v , v v , v v , v
0i i i i i n n ic c c c
L L
v , v 0 0 is linearly independent.i i ic i S Q
5 - 19
Ex: (Using orthogonality to test for a basis)
Show that the following set is a basis for .4R
)}1,1,2,1(,)1,2,0,1(,)1,0,0,1(,)2,2,3,2{(4321
S
vvvv
Sol:
: nonzero vectors
02262
02402
02002
41
31
21
vv
vv
vv
4321 ,,, vvvv
01201
01001
01001
43
42
32
vv
vv
vv
.orthogonal is S4for basis a is RS
5 - 20
Thm 3.24:Thm 3.24: (Coordinates relative to an orthonormal basis)
If is an orthonormal basisorthonormal basis for an inner
product space V, then the coordinate representation of a vector
w with respect to B is
1 2{v , v , , v }nB L
1 2{v , v , , v }nB Q L is orthonormal
ji
jiji
0
1, vv
Vw
1 1 2 2w v v vn nk k k L (unique representation)
Pf:Pf:
is a basis for V1 2{v , v , , v }nB L
1 1 2 2[w] w , v v w , v v w , v vB n n L
5 - 21
1 1 2 2
1 1
w , v ( v v v ) , v
v , v v , v v , vi n n i
i i i i n n i
i
k k k
k k k
k i
L
L L
1 1 2 2w w, v v w, v v w, v vn n L
1
2
w, v
w, vw
w, v
B
n
M
5 - 22
Ex: (Representing vectors relative to an orthonormal basis)
Find the coordinates of w = (5, -5, 2) relative to the following
orthonormal basis for .
3 34 41 2 3 5 5 5 5{ , , } {( , , 0) ,( , , 0) , (0 , 0 , 1)}B v v v
3R
Sol:
2)1,0,0()2,5,5(,
7)0,,()2,5,5(,
1)0,,()2,5,5(,
33
53
54
22
54
53
11
vwvw
vwvw
vwvw
2
7
1
][ Bw
5 - 23
Gram-Schmidt orthonormalization process:Gram-Schmidt orthonormalization process:
is a basis for an inner product space V 1 2{u , u , , u }nB L
11Let uv })({1 1vw span
}),({2 21 vvw span
1 2' {v , v , , v }nB L
1 2
1 2
vv v'' { , , , }
v v vn
n
B L
is an orthogonal basis.
is an orthonormal basis.
1
1
W1
v , vv u proj u u v
v , vn
nn i
n n n n ii i i
M2
3 1 3 23 3 W 3 3 1 2
1 1 2 2
u , v u , vv u proj u u v v
v , v v , v
111
122222 〉〈
〉〈proj
1v
v,v
v,uuuuv W
5 - 24
Sol: )0,1,1(11 uv
)2,0,0()0,2
1,
2
1(
2/1
2/1)0,1,1(
2
1)2,1,0(
222
231
11
1333
vvv
vuv
vv
vuuv
Ex: (Applying the Gram-Schmidt orthonormalization process)
Apply the Gram-Schmidt process to the following basis.
)}2,1,0(,)0,2,1(,)0,1,1{(321
B
uuu
)0,2
1,
2
1()0,1,1(
2
3)0,2,1(1
11
1222
vvv
vuuv
5 - 25
}2) 0, (0, 0), , 2
1 ,
2
1( 0), 1, (1,{},,{' 321
vvvB
Orthogonal basis
}1) 0, (0, 0), , 2
1 ,
2
1( 0), ,
2
1 ,
2
1({},,{''
3
3
2
2
v
v
v
v
v
v
1
1B
Orthonormal basis
5 - 26
Ex: Find an orthonormal basis for the solution spacesolution space of the
homogeneous system of linear equations.
0622
07
4321
421
xxxx
xxx
Sol:
08210
01201
06212
07011 .. EJG
1
21 2
3
4
2 2 1
2 8 2 8
1 0
0 1
x s t
x s ts t sv tv
x s
x t
r r
5 - 27
Thus one basis for the solution space is
)}1,0,8,1(,)0,1,2,2{(},{ 21 vvB
1 ,2 ,4 ,3
0 1, 2, ,2 9
181 0, 8, 1,
,
,
0 1, 2, ,2
111
1222
11
vvv
vuuv
uv
1,2,4,3 0,1,2,2' B (orthogonal basis)
30
1,
30
2,
30
4,
30
3 , 0,
3
1,
3
2,
3
2''B
(orthonormal basis)
5 - 28
3.10 Mathematical Models and Least-Squares Analysis3.10 Mathematical Models and Least-Squares Analysis
Let WW be a subspacesubspace of an inner product space VV.
(a) A vector u in V is said to orthogonal to W, if u is
orthogonal to every vector in W.
(b) The set of all vectors in V that are orthogonal to W is
called the orthogonal complement of W.
(read “ perp”)
} ,0,|{ WVW wwvv
W W
Orthogonal complementOrthogonal complement of W:
5 - 29
Direct sumDirect sum:
Let and be two subspaces of . If each vector
can be uniquely written as a sum of a vector from
and a vector from , , then is the
direct sum of and , and you can write
1W 2W nR nRx
1W1w
2W2w21 wwx nR
21 WWRn
ThmThm 3.25 3.25: (Properties of orthogonal subspaces) Let W be a subspace of Rn. Then the following properties
are true.
(1)
(2)
(3)
nWW )dim()dim( WWRn
WW )(
1W 2W
5 - 30
Thm 3.26Thm 3.26: (Projection onto a subspace)
If is an orthonormal basis for the
subspace W of V, and for , then
1 2{u , u , , u }tL
Vv
vv W
proj
||
proj 1 1 2 2v v , u u v , u u v , u uW t t L
proj v, 0 iWu i Q
5 - 31
Ex: (Projection onto a subspace)
3 ,1 ,1 ,0 ,0 ,2 ,1 ,3 ,0 21 vww
Find the projection of the vector v onto the subspace W.
:, 21 ww
Sol:
an orthogonal basis for W
:0,0,1),10
1,
10
3,0( , ,
2
2
1
121
w
w
w
wuu
an orthonormal basis for W
}),({ 21 wwspanW
5 - 32
Scientists are often presented with a system that has no solution and they must find an answer that is as close as possible to being an answer.
Suppose that we have a coin to use in flipping and this coin hassome proportion m of heads to total flips.
Because of randomness, we do not find the exact proportion with this sample
The vector of experimental data {16, 34, 51} is not in the subspace of solutions.
Fitting by Least-SquaresFitting by Least-Squares
5 - 33
However, we want to find the m that most nearly works. An orthogonal projection of the data vector into the line subspace gives our best guess.
The estimate (m = 7110/12600 ~ 0.56) is a bit high than 1/2 but not much, so probably the penny is fair enough. The line with the slope m= 0.56 is called the line of best fitline of best fit for this data.
Minimizing the distance between the given vector and the vector used as the left-hand side minimizes the total of these vertical lengths. We say that the line has been obtained through fitting by fitting by least-squaresleast-squares.
5 - 34
The different denominations of U.S. money have different averagetimes in circulation
The linear system with equations has no solution, but we can use orthogonal projection to find a best approximation.
1 1 1.5
1 5 2
1 10 3
1 20 5
1 50 9
1 100 20
bAx v
m
tr r
5 - 35
The method on the Projection into a Subspace says that coefficients b and m so that the linear combination of the columns of AA is as close as possible to the vector are the entries of
Some calculation gives an intercept of b b = 1= 1..0505 and a slope of m m = 0= 0..1818.
vr
1( )T Tbx A A A v
m
r r
5 - 36
Thm 3.27Thm 3.27: (Orthogonal projection and distance)
Let SS be a subspace of an inner product space V, and .
Then for all ,
Vv
s S proj vSs
proj||v v|| ||v ||S s
o projr ||v v|| min ||v ||S s
( is the best approximation to v from SS)proj vS
5 - 37
Pf:Pf:proj projv (v v) ( v )S Ss s
proj proj(v v) ( v )S S s
By the Pythagorean theorem
proj proj2 2 2||v || ||v v|| || v ||S Ss s
proj projv v 0S Ss s
proj2 2||v || ||v v||Ss
proj||v v|| ||v ||S s
5 - 38
For a given mnmn matrix [AAmnmn]: The spaces NS(A) and RS(A) are orthogonal
complements of each other within Rn. This means that any vector from NS(A) is
orthogonal to any vector from CS(AT), and the vectors in these two spaces span
Rn, i.e., ( ) ( )n TR NS A CS A
.
Fundamental subspaces of a matrixFundamental subspaces of a matrix
5 - 39
Thm 3.28Thm 3.28:
If A is an m×n matrix, then
(1)
(2)
(3)
(4)
( ( )) ( )
( ( )) ( )
CS A NS A
NS A CS A
( ( )) ( )
( ( )) ( )
CS A NS A
NS A CS A
( ) ( ) ( ) ( ( ))T m mCS A NS A R CS A NS A R
( ) ( ) ( ) ( ( ))T n T nCS A NS A R CS A CS A R
5 - 40
Ex: (Fundamental subspaces)
Find the four fundamental subspaces of the matrix.
000
000
100
021
A (reduced row-echelon form)
Sol: span 4( ) 1,0,0,0 0,1,0,0 is a subspace of CS A R
span 3( ) 1,2,0 0,0,1 is a subspace of CS A RS A R
span 3( ) 2,1,0 is a subspace of NS A R
5 - 41
span 4( ) 0,0,1,0 0,0,0,1 is a subspace of NS A R
1 0 0 0 1 0 0 0
2 0 0 0 0 1 0 0
0 1 0 0 0 0 0 0
A R
Check:
)())(( ANSACS
)())(( ANSACS
4)()( RANSACS T 3)()( RANSACS T
ts
5 - 42
Ex:
Let W is a subspace of R4 and .
(a) Find a basis for W
(b) Find a basis for the orthogonal complement of W.
)1 0, 0, 0,( ),0 1, 2, 1,( 21 ww
Sol:
1 2
1 0 1 0
2 0 0 1
1 0 0 0
0 1 0 0
w w
A R
(reduced row-echelon form)
}),({ 21 wwspanW
5 - 43
( )
1,2,1,0 , 0,0,0,1
a W CS A
is a basis for W
1
2
3
4
( )
2 2 1
1 2 1 0 1 0
0 0 0 1 0 1
0 0 0
2,1,0,0 1,0,1,0 is a basis for
b W CS A NS A
x s t
x sA s t
x t
x
W
Q
Notes:
4
4
(2)
)dim()dim()dim( (1)
RWW
RWW
5 - 44
Least-squares problem:Least-squares problem:
(A system of linear equations)
(1) When the system is consistent, we can use the Gaussian
elimination with back-substitution to solve for x
A x b11 mnnm
(2) When the system is consistent, how to find the “best possible”
solution of the system. That is, the value of x for which the
difference between Ax and b is small.
5 - 45
Least-squares solutionLeast-squares solution:
Given a system Ax = b of m linear equations in n unknowns,
the least squares problem is to find a vector x in Rn that
minimizes with respect to the Euclidean inner
product on Rn.
Such a vector is called a least-squares solutionleast-squares solution of Ax = b.
bx A
5 - 46
x
x ( ) ( is a subspace of )
( )
nm n
m
A M R
A CS A CS A R
Let W CS A
Q
1ˆ
x̂ b
ˆ(b x) ( Pr b)
ˆ( x) ( )
ˆb x ( ( )) ( )
ˆ(b x) 0
ˆi.e. x b ( ),
W
T T
W
A Proj
A b oj W
b A CS A
A CS A NS A
A A
A A A o x A A br A
(This is the solution of the normal system associated
with Ax = b)
5 - 47
Note: Note: The problem of finding the least-squares solutionthe least-squares solution of
is equal to the problem of finding an exact solution of the an exact solution of the
associated normal systemassociated normal system .
bx A
bx AAA ˆ
Thm:Thm:
If A is an m×n matrix with linearly independent column vectors,
then for every m×1 matrix b, the linear system Ax = b has a unique
least-squares solution. This solution is given by
Moreover, if W is the column space of A, then the orthogonal
projection of b on W is
1x ( ) bLS A A A
1b x bproj [ ( ) ]W AA A A A
5 - 48
Ex: (Solving the normal equations)
Find the least squares solution of the following system
and find the orthogonal projection of b on the column space of A.
5 - 49
Sol:
the associated normal system
5 - 50
the least squares solution of Ax = b
53
32
xLS
the orthogonal projection of b on the column space of A
165
3 8( ) 63
2 176
1 1
proj 1 2
1 3
b xCS A LSA