Upload
gasser-alaa-eldien
View
220
Download
0
Embed Size (px)
Citation preview
7/28/2019 linear algebra course part 3
1/65
Chapter 3
Vector Spaces and Subspaces
Po-Ning Chen, Professor
Department of Electrical Engineering
National Chiao Tung University
Hsin Chu, Taiwan 30010, R.O.C.
7/28/2019 linear algebra course part 3
2/65
3.1 Spaces of vectors 3-1
What is a Space?
Answer: An area that is free to occupy.
What is the Space of vectors?
Answer: An area that is free to occupy by vectors.
Example. v = v1v2 can occupy a 2-dimensional plane. What is a vector space?
Answer: A vector space is not just a space occupied by vectors. It is a
mathematical structure formed by vectors and rules (axioms) to handlethese vectors. In other words, this term is reserved specifically for this use in
mathematics!
7/28/2019 linear algebra course part 3
3/65
3.1 Spaces of vectors 3-2
Definition (Vector space): A vector space is a space of vectors in V andscalars in F (named field) with operations:
vector addition: v + w scalar-to-vector multiplication: av
scalar addition: a + b
that satisfy the following axioms:
1. Commutativity of addition: v + w = w + v
2. Associativity of addition: u + (v + w) = (u + v) + w
3. Identity element of addition: There exists a unique zero vector 0 such that
v + 0 = v for all vector v in the space
4. Inverse elements of addition: For every v, there exists a unique v such that
v + (v) = 0
5. Identity element of scalar multiplication: There exists a unique scalar 1 such
that 1v = v
7/28/2019 linear algebra course part 3
4/65
3.1 Spaces of vectors 3-3
6. Compatibility of scalar multiplication with scalar multiplication:
a(bv) = (ab)v
7. Distributivity of scalar multiplication with respect to vector addition:a(v + w) = av + aw
8. Distributivity of scalar multiplication with respect to scalar addition:
(a + b)v = av + bv
Example of a vector space.
1. V = n and F = (with usual real addition and multiplication).
2. V =The set of all real-valued function f(t) and F = .
Note: We only need scalar and vector additions and scalar-to-vector
multiplication. Hence, {f(t)} satisfies such demand.
3. V =The set of all 2 2 matrices and F = (with usual matrix addition and
scalar-to-matrix multiplication).
7/28/2019 linear algebra course part 3
5/65
3.1 Spaces of vectors 3-4
Some important notes:
A vector is not necessary of the form v1...
vn.
A vector (in its most general form) is an element of a vector space.
Example of a vector space.
1. V = n and F = . vector=
v1...vn
.2. V =The set of all real-valued function f(t) and F = . vector=f(t).
3. V =The set of all 2 2 matrices and F = . vector=
v1,1 v1,2v2,1 v2,2
.
In fact, it makes no difference in placing a (column) vector in 4 into v1,1 v1,2v2,1 v2,2as long as the operation defined is term-wisely based.
7/28/2019 linear algebra course part 3
6/65
3.1 Sub-vector space of a vector space 3-5
In terminology, we usually term sub(-vector)space of a (vector) space for sim-
plicity.
Definition (Subspace): A sub(-vector)space S of a (vector) space V sat-isfies
1. that all the vectors it contains belong to the vector space, and
2. the eight axioms/rules.
Equivalently,
v S and w S v + w S, and
v S and a F av S
The two equivalent conditions imply the validity of Conditions 1. and 2.
Exercise. Prove that the zero vector should belong to subspace S of the three-
dimensional real vector space 3 with field .
Proof: av should belong to S for any a F = . The proof is completed by
taking a = 0. 2
7/28/2019 linear algebra course part 3
7/65
3.1 Sub-vector space of a vector space 3-6
Examples of subspace of 3:
A single origin point. (A space always contains zero point.)
Any line passes through the origin.
Any plane passes through the origin.
Choose any two vectors v and w, the set that consists of all the linear combi-
nation of the two vectors, i.e., av + bw.
This is exactly equivalent to the two equivalent definitions of the subspace.I.e.,
v S and w S v + w S, and
v S and a F av S This will be useful in proving the
legitimacy of a subspace!
7/28/2019 linear algebra course part 3
8/65
3.1 Column space 3-7
Following the last example on the previous slide:
S =
b 3 : av + bw = b for some a, b
=
b 3 :
v w a
b
= b for some a, b
=
b 3 : Ax = b for some x 2
where A = v w.Every subspace of a space n can be of the form:
b m : Amnxn1 = bm1 for some x
n
Since it is a linear combination of the column vectors ofA, it is also termedcolumn space (denoted by C(A)).
So, given A, a subspace is then defined (through the old friend Ax = b).
If Amm invertible,S
=V
= m
. Example. A = I. If Amm non-invertible, S V =
m. Example. A = all-zero matrix.
7/28/2019 linear algebra course part 3
9/65
3.1 Column space 3-8
Ax = b is solvable if, and only if, b C(A).
C(A) is sometimes viewed as the subspace spanned by column vectors of A.
7/28/2019 linear algebra course part 3
10/65
3.1 Column space 3-9
Examples of C(A):
If Ann invertible, C(A) = n.
If An1, C(A) is a line in n space.
If A =
a a a
nn, C(A) is a line in n space.
Exercise: Does there exist an A such that
C(A) = b 3 : Ax = b for some x 2is an empty set?
(Hint: What is the element that must be contained in all subspaces? The subspace
that only consists of this element is denoted by Z, and is called zero subspace.)
7/28/2019 linear algebra course part 3
11/65
3.1 Column space 3-10
Exercise: If C(Ann) = n, is it possible that Ax = b has no solution for some
b n?
(Hint:C
(A
) = b n : Annxn1 = bn1 for some x n is the set of bsuch that Ax = b admits solutions.)Note from the textbook:
We can also find the subspace of a subspace S.
In textbook, it is denoted by SS.
We can certainly have SSS, the subspace of the subspace SS of a subspace S
of a spaceV
.
7/28/2019 linear algebra course part 3
12/65
3.2 Nullspace of a matrix A 3-11
Before introducing the nullspace, we give a formal definition of the columnspace.
Definition (Column space): A column space of a matrix A, denoted byC(A), consists of all the vectors that are linear combinations of column vectors of
A. It can always be represented as:
C(A) =
b m : Amnxn1 = bm1 for some x
n
Definition (Nullspace): A nullspace of a matrix A, denoted by N(A), consistsof all the vectors that nullify the linear combinations of column vectors ofA. It
can always be represented as:
N(A) = x n : Amnxn1 = 0m1 N(A) is a vector space.
N(A) is a sub-space ofn.
However, nullspace N(A) is not necessarily (often, not) a sub-space of columnspace C(A).
7/28/2019 linear algebra course part 3
13/65
3.2 Determination of nullspace 3-12
How to easily determine the nullspace ofA?
Recall that in linear equations Ax = b, x is called the solutions.
Specially when b = 0, x = 0 is called the special solutions.
Answer: Method of special solution.
The subspace of3 can only be:
A single point 0 Determined by no special solutions
A line passing through 0 Determined by one special solutions
A plane passing through 0 Determined by two special solutions
3 itself Determined by three special solutions
7/28/2019 linear algebra course part 3
14/65
3.2 Determination of nullspace 3-13
Example. A =
1 2 2
0 2 0
N(A) =x 3 :
1 2 20 2 0
x1x2x3
= 0 The second row of A indicates x2 = 0.
The first row ofA (i.e., x1 +2x3 = 0) indicates x =20
1
is a special solution.
We then know (by intuition) that N(A) is a line passing through 0
00 and
2
01.2
Question is Can we always rely on our intuition?
Is there a systematic method to determine N(A)?
7/28/2019 linear algebra course part 3
15/65
3.2 Determination of nullspace 3-14
Example (continued). The solution ofAx = 0 does not change by multiplying aproper matrix such as
EAx = E0 = 0.
So we do forward and backward eliminations (with no row exchange) and
pivot normalization to obtain:
1 0 20 1 0x1x
2x3 = 0
Recall that this is called reduced row echelon form.
We then have x1 + 2x3 = 0x2 = 0
We conclude that N(A) should be the intersection of two non-parallel planes. 2
What will happen if we need to perform row exchanges during forward elimination!
7/28/2019 linear algebra course part 3
16/65
3.2 Determination of nullspace 3-15
Example. A =
0 2 0
1 2 2
N(A) =x 3 :
0 2 01 2 2
x1x2x3
= 0 The first row ofA indicates x2 = 0.
The second row of A (i.e., x1 + 2x3 = 0) indicates x =20
1
is a specialsolution.
We then know (by intuition) that N(A) is a line passing through 000
and 201
.2
Question is Can we always rely on our intuition?
Is there a systematic method to determine N(A)?
7/28/2019 linear algebra course part 3
17/65
3.2 Determination of nullspace 3-16
Example (continued). The solution ofAx = 0 does not change by multiplying aproper matrix such as
EAx = E0 = 0.
So we do forward and backward eliminations (with row exchange) and
pivot normalization to obtain:
1 0 20 1 0x1x2x3 = 0
Recall that this is called reduced row echelon form.
We then have x1 + 2x3 = 0x2 = 0
We conclude that N(A) should be the intersection of two non-parallel planes. 2
What will happen ifN(A) is a plane!
7/28/2019 linear algebra course part 3
18/65
3.2 Determination of nullspace 3-17
Example. A =
1 1 2 3
2 2 8 10
3 3 1 0 1 3
Lets begin to work on forward elimination on the first column. We then obtain:
1 1 2 3
0 0 4 4
0 0 4 4
1st pivot
We continue to work on the second row
(by choosing the left-most non-zero entry as the next pivot)!
1 1 2 3
0 0 4 4
0 0 0 0 1st pivot
2nd pivot
Since we only have two pivots, the nullspace N(A) should be an intersectionof two 4 hyperplanes, which can be represented as linear combinations of two
special solutions.
7/28/2019 linear algebra course part 3
19/65
3.2 Determination of nullspace 3-18
Proceed with the back elimination:
1 1 0 10 0 4 4
0 0 0 0 1st pivot
2nd pivot
and pivot normalization:
1 1 0 10 0 1 1
0 0 0 0 1st pivot
2nd pivot
So we know thatx1, x3 pivot variables
x2, x4 free variables (free to choose its values for special solutions)
We then assign (x2, x4) = (1, 0) and (x2, x4) = (0, 1) and obtain
x1 =
1
1
0
0
and x2 =
1
0
1
1
Then, N(A) consists of all linear combinations of the above two vectors. 2
7/28/2019 linear algebra course part 3
20/65
3.2 Determination of nullspace 3-19
Remark.
If we have three free variables, we may assign (1, 0, 0), (0, 1, 0) and (0, 0, 1) for
special solutions.
Exercise 1. How about four free variables ?
Exercise 2. How about zero free variables ?
7/28/2019 linear algebra course part 3
21/65
3.2 Determination of nullspace 3-20
In MATLAB:
The special solutions for N(A) can be determined by:
null(A); % Take A from the previous example.
Note that the special solutions are not unique!
The result is:answer =
-0.4059 -0.6597
0.7623 0.1373
0.3564 -0.5225
-0.3564 0.5225
Recall that the reduced row echelon form can be obtained by
rref(A)
The result is:answer =
1 1 0 1
0 0 1 1
0 0 0 0
7/28/2019 linear algebra course part 3
22/65
3.2 Echelon matrix 3-21
It is very useful to find the (row) echelon matrix of a matrix A.
Row echelon form: A form of matrices satisfying the below two conditions.
Every all-zero row is below the other non-all-zero rows.
The leading coefficient (the first nonzero number from the left, also calledthe pivot) of a non-all-zero row is always strictly to the right of the leading
coefficient of the row above it.
(row) Reduced echelon form: A form of matrices in row echelonform with the leading coefficient being one and also being the only non-zero
entry in the same column.
Example. A U =
p x x x x x x
0 p x x x x x0 0 0 0 0 p x
0 0 0 0 0 0 0
row echelon form
R =
1 0 x x x 0 x
0 1 x x x 0 x0 0 0 0 0 1 x
0 0 0 0 0 0 0
row reduced echelon form
In the above example, N(A) = N(U) = N(R) is a 4-dimensional hyperplane
(in a 7-dimensional space) since there are four free variables.
7/28/2019 linear algebra course part 3
23/65
3.2 Echelon matrix 3-22
If Ann is invertible, then R = I.
Exercise. Is the all-zero matrix in the row echelon form?
7/28/2019 linear algebra course part 3
24/65
3.2 Echelon matrix 3-23
Example (Problem 20). Suppose column 1 + column 3 + column 5 = 0 in a 4by 5 matrix with four pivots. Which column is sure to have no pivot (and which
variable is free)? What is the special solution? What is the nullspace?
Answer:
Since column 1 + column 3 + column 5 = 0, column 5 has no pivot.
It is not possible for columns 1 and 3 having no pivot. Check the row reduced
echelon matrix R under the assumption that column 1 (or column 3) has no
pivot. A contradiction will be obtained!
Since column 5 has no pivot, the fifth variable is free.
Since there are four pivots, column 5 has no pivot, and column 1 + column 3
+ column 5 = 0, the row reduced echelon matrix is
R =
1 0 0 0 1
0 1 0 0 0
0 0 1 0 1
0 0 0 1 0
7/28/2019 linear algebra course part 3
25/65
3.2 Echelon matrix 3-24
The special solution is of the form x =
x1x2x3
x41
satisfying Rx = 0.
Hence, x =
1
0
10
1
.
The nullspace contains all multiples ofx =
1
01
0
1
(a line in 5). 2
7/28/2019 linear algebra course part 3
26/65
3.2 Echelon matrix 3-25
Example (Problem 21). Construct a matrix whose nullspace consists of all com-
binations of (2, 2, 1, 0) and (3, 1, 0, 1).
Answer:
Two special solutions two free variables and two pivots.
R is an m 4 matrix, where m 2. Take m = 2 for simplicity.
R is of the form:
R = 1 0 r1,3 r1,40 1 r2,3 r2,4 The two special solutions (Rx = 0) give:
R = 1 0 2 3
0 1 2 12
The answer is not unique ! For example,
R = 1 0 2 3
0 1 2 10 0 0 0
7/28/2019 linear algebra course part 3
27/65
3.2 Echelon matrix 3-26
Example (Problem 22). Construct a matrix whose nullspace consists of all mul-
tiples of (4, 3, 2, 1).
Answer:
One special solution one free variables and three pivots.
R is a m 4 matrix, where m 3. Take m = 3 for simplicity.
R is of the form:
R =1 0 0 r1,40 1 0 r2,4
0 0 1 r3,4
The special solution (Rx = 0) give:
R =1 0 0 40 1 0 3
0 0 1 2
2
7/28/2019 linear algebra course part 3
28/65
3.2 Echelon matrix 3-27
Example (Problem 23). Construct a matrix whose column space contains (1, 1, 5)
and (0, 3, 1) and whose nullspace contains (1, 1, 2).
Answer:
Linear combination of column vectors c1a1 + c1a2 + c3a3 can be made equal
to (1, 1, 5) and (0, 3, 1). So, let c1 = 1 and c2 = c3 = 0 for the first solution,
and let c1 = c3 = 0 and c2 = 1 for the second solution. We then obtain
A =1 0 a1,31 3 a2,3
5 1 a3,3
.
Solving A1
12 = 0 gives
A =
1 0 0.5
1 3 2
5 1 3
.
2
7/28/2019 linear algebra course part 3
29/65
3.2 Echelon matrix 3-28 What is the relationship between column space and nullspace of a symmetric
matrix A?
Answer: Their vectors are perpendicular to each other. I.e.,
a N(A) and b C(A) a b = 0.
Proof:
a N(A) implies Aa = 0.
b C(A) implies Ax = b for some x. Hence, by AT = A, we obtain
a b = bTa = xTATa = xTAa = xT0 = 0.
2
IfAmn for n = m, then column space ofA is a vector space ofm-dimensional
vectors, but nullspace of A is a vector space ofn-dimensional vectors.
So, we can talk about the relation ofcolumn space and nullspace only when
A is a square matrix. The text then also introduces row space.
7/28/2019 linear algebra course part 3
30/65
3.2 Echelon matrix 3-29
Definition (Row space): A row space of a matrix A, denoted by R(A),consists of all the vectors that are linear combinations of row vectors ofA. It can
always be represented as:
R(A) =
b n : ATx = bn1 for some x m
What is the relationship between row space and nullspace of a matrix A?
Answer: Their vectors are perpendicular to each other. I.e.,
a N(A) and b R(A) a b = 0.
Proof:
a N(A) implies Aa = 0.
b R(A) implies AT
x = b for some x. Hence, we obtain
a b = bTa = xTAa = xT0 = 0.
2
For a symmetric matrix, row space = column space.The row space can be defined in terms of reduced row echelon matrix R.
7/28/2019 linear algebra course part 3
31/65
3.2 Echelon matrix 3-30Exercise. What is the relationship between column space of A and nullspace of
AT?
7/28/2019 linear algebra course part 3
32/65
3.3 The rank and row reduced form 3-31Now we turn to another important quantitative index of matrix rank.
Definition (Rank): The rank of a matrix A is the number of non-zero pivots.
How to determine the rank?
Answer: By elimination,
A upper triangular U Row reduced echelon R.
The rank r is
the dimension of the row space
The non-pivot rows should be linear combinations of the pivot rows!
the dimension of the column space
7/28/2019 linear algebra course part 3
33/65
3.3 The rank and row reduced form 3-32
First note that
C(A) =
b m : Amnxn1 = bm1 for some x
n
and a m : Emmbm1 = am1 for some b C(A)
has the same dimension ifE is invertible.
So, C(A) and C(R) has the same dimension as R = EA for some invertible E.
Apparently, the non-pivot columns ofR are linear combinations of pivot columns(cf., slide 3-21).
the number of independent rows in A
the number of independent columns in A
The dimension of the nullspace ofA is (n r). I.e., (n r) = the number offree variables = the number of special solutions (for Ax = 0).
Definition (Independence): A set of vectors {v1, . . . , vm} is said to be linearly in-dependent if the linear combination of them (a1v1 + + amvm) equals 0 only when the
coefficients are zero (a1 = a2 = = am = 0).
7/28/2019 linear algebra course part 3
34/65
3.3 Pivot columns 3-33 How to locate the r pivot columns in A?
Answer: R =rref(A).
The r pivots columns ofR = EA and A are exactly at the same positions.
By R = EA, an interesting observation is that the first r columns of E1 is
exactly the r pivot columns ofA!
This is because
the r pivot columns ofR form an r r identity matrix.
Example. A = 1 3 0 2 10 0 1 4 31 3 1 6 4
R = 1 3 0 2 10 0 1 4 30 0 0 0 0
A = E1R =
d1,1 d1,2 d1,3d2,1 d2,2 d2,3
d3,1 d3,2 d3,3
1 3 0 2 1
0 0 1 4 3
0 0 0 0 0
=
d1,1 d1,2
d2,1 d2,2
d3,1 d3,2
2
7/28/2019 linear algebra course part 3
35/65
3.3 Pivot columns 3-34 Can we find the special solutions directly using the fact that
the r pivot columns ofR form an r r identity matrix.
Answer: Of course.
For a matrix Amn, there are (n r) special solutions.
These special solutions satisfies
Rx1 = 0, Rx2 = 0, Rxnr = 0.
Equivalently,
Rmn
x1 x2 xnr
n(nr)
= RmnNn(nr) = 0m(nr).
By reordering the columns ofR, it can be of the form:
R =
Irr Fr(nr)
0(nr)r 0(nr)(nr)
Nn(nr) =
Fr(nr)
I(nr)(nr)
So, the special solutions just to collect the free-variable columns,
negate them, and add an identity matrix.
7/28/2019 linear algebra course part 3
36/65
3.3 Pivot columns 3-35
Example. A =
1 3 0 2 1
0 0 1 4 3
1 3 1 6 4
R =
1 3 0 2 10 0 1 4 3
0 0 0 0 0
N =
1 0 0
0 1 0
0 0 1
Put identity matrix
=
3 2 1
1 0 0
0 4 3
0 1 0
0 0 1
Put F
linear independent pivot column of A
2nd column of A is a linear combination of 1st column of A
linear independent pivot column of A
4th column of A is a linear combination of 1st & 3rd columns of A
5th column of A is a linear combination of 1st & 3rd columns of A
2
Example. A = 1 2 3 R = 1 2 3 N =
1 0
0 1
Put identity matrix=
2 31 0
0 1
Put F2
7/28/2019 linear algebra course part 3
37/65
3.3 Pivot columns 3-36In summary:
The pivot columns are not linear combination of earlier columns.
The free columns are linear combination of earlier pivot columns.
Final question:
How to find the matrix E such that R = rref(A) = EA?
Answer: Forward and backward eliminations (with possibly row exchanges)
on matrix Amn Imm
m(n+m)
will yield
Emm Amn Immm(n+m)
=
Rmn Emmm(n+m)
.
Example. A =
1 3 0 2 10 0 1 4 3
1 3 1 6 4
R = rref1 3 0 2 1 1 0 00 0 1 4 3 0 1 01 3 1 6 4 0 0 1
= 1 3 0 2 1 0 1 10 0 1 4 3 0 1 00 0 0 0 0 1 1 1
2
7/28/2019 linear algebra course part 3
38/65
3.3 Pivot columns 3-37
Example. A =
0 1 0
1 0 0
.
R = rref0 1 0 1 01 0 0 0 1 = 1 0 0 0 10 1 0 1 0 2
7/28/2019 linear algebra course part 3
39/65
3.3 Rank revisited 3-38 The matrix A can be decomposed into sum of outer products of r pairs of
vectors.
I.e., A = ri=1 uivTi How to prove this? Or how to determine these {ui, vi}ri=1 pairs?Answer: (Denote E = E1 for convenience.)
Amn = EmmRmn =e1 e2 emmm
rT1...
rT
r
0T...
0T
mn
... 1st pivot row...
. . . rth pivot row... all-zero row
...
... all-zero row
=
e1 e2 em
mm
rT10T
0T...
0T
0T
0T
mn
+ +
0T
...
0T
rTr0T...0T
mn
7/28/2019 linear algebra course part 3
40/65
3.3 Rank revisited 3-39
=
e1 em
rT10T
0
T
...
0T
0T
0T
+ +
e1 em
0T...
0
T
rTr0T...
0T
= e1rT1 + + err
Tr
2
The determination of E and R (with A = ER) gives the outer-product
decomposition ofA.
7/28/2019 linear algebra course part 3
41/65
3.3 Rank revisited 3-40
Exercise 1 (Problem 25): Show that every m-by-n matrix of rank r reduces to
(m by r) times (r by n). Also, show that the column of the m by r matrix equal
the pivot columns ofA.
Hint: Amn =
e1 er
mr
rT1...rTr
rn
Exercise 2 (Problem 16): If Amn = eAr
T
A and Bnk = eBr
T
B are two rank-1matrices, prove that the rank ofAB is 1 if, and only if, rTAeB = 0.
Exercise 3: If A =r
i=1 eirTi is a rank-r matrix, and B =
rj=1 ejr
Tj is a rank-r
matrix, show that
AB = Ae1 Aer =EAB
R
What is the condition under which AB remains rank r? Answer: EAB has rank r,
or Ae1, , Aer are linearly independent.
Exercise 4: Prove that rank(AB) rank(B).
Exercise 5: Prove that rank(AB) rank(A). Hint: rank(A) =rank(AT).
7/28/2019 linear algebra course part 3
42/65
3.4 The complete solution of Ax = b 3-41 Recall how we completely solve Amnxn1 = 0m1.
(Here, complete means that we wish to find all solutions.)
Answer:
R = rref(A) =
Irr Fr(nr)
0(mr)r 0(mr)(nr)
Nn(nr) =
Fr(nr)
I(nr)(nr)
Then, every solution x for Ax = 0 is of the form
x(null)n1 = Nn(nr)v(nr)1 for any v
nr.
How to completely solve Amnxn1 = bm1?
Answer: Identify one particular solution x(p) to Amnx(p)n1 = bm1.
Then, every solution x for Ax = b is of the form
x(p) + x(null).
7/28/2019 linear algebra course part 3
43/65
3.4 The complete solution of Ax = b 3-42Proof:
- Suppose that w satisfies Aw = b but cannot be expressed as x(p) + x(null).
- Then, A(w x(p)) = Aw Ax(p) = b b = 0. Hence, (w x(p)) N(A).
- This implies w = x(p) + x(null) for some x(null) N(A); a contradiction is
accordingly obtained. 2
Exercise: When will there be only one solution to Ax = b?
Answer: When N(A) consists of only 0, and x(p) exists!Note that x(p) exists if, and only if, b C(A).
7/28/2019 linear algebra course part 3
44/65
3.4 The complete solution of Ax = b 3-43 How to find (one of) x(p)?
Answer:
R d
= rref(
A b
) =
Irr Fr(nr)d1...
dr0(mr)r 0(mr)(nr) 0(mr)1
.
It is the solution of Ax = b (or Rx = EAx = Eb = d) by setting all freevariables equal to zeros, and the remaining variables equal to d1, d2, . . . , dr.
Example (Problem 3). A = 1 3 3
2 6 91 3 3
and b = 1
55. Find x(p).
rref(
A b
) =
1 3 0 d1 = 2
0 0 1 d2 = 1
0 0 0 0
. Then, x(p) =
d10
d2
.... free variable = 0 .
7/28/2019 linear algebra course part 3
45/65
3.4 The complete solution of Ax = b 3-44Based on what we learn about rank, we can summarize the solutions ofAmnxn1 =
bm1 as follows.
A Ax = b N(A)
r = m r = n square and invertible 1 solution {0n1}r = m r < n rectangular (smaller height) solutions combination of (n r) linearly
independent n-by-1 vectors
r < m r = n rectangular (smaller width) 0 or 1 solution {0n1}
r < m r < n not full rank 0 or solutions combination of (n r) linearly
independent n-by-1 vectorsIt may be easier to memorize this by the following equivalent table.
A Ax = b N(A)
r = m r = n square and invertible 1 solution combination of (n r) = 0 linearly
C(A) =
m
independent n-by-1 vectorsr = m r < n rectangular (smaller height) solutions combination of(n r) > 0 linearly
C(A) = m independent n-by-1 vectors
r < m r = n rectangular (smaller width) 0 or 1 solution combination of(n r) = 0 linearly
C(A) m independent n-by-1 vectors
r < m r < n not full rank 0 or solutions combination of(n r) > 0 linearlyC(A) m independent n-by-1 vectors
7/28/2019 linear algebra course part 3
46/65
3.4 Denote for convenience
E = E1
3-45A Ax = b C(A)
(a) r = m r = n square and invertible 1 solution m
(b) r = m r < n rectangular (smaller height) solutions m
(c) r < m r = n rectangular (smaller width) 0 or 1 solution combination ofr linearly
independent m-by-1 vectors(d) r < m r < n not full rank 0 or solutions combination ofr linearly
independent m-by-1 vectors
C(A) = b m : Amnxn1 = bm1 for some x n=
b m : Emm
Irr Fr(nr)
0(mr)r 0(mr)(nr)
xn1 = bm1 for some x
n
= b m : Emm
xr1 + Fr(nr)x(nr)10
(mr)1 = bm1 for some x n
=
b r : Errxr1 = br1 for some x
r
(a)b r : Errxr1 + ErrFr(nr)x(nr)1 = br1 for some x
n
(b)
b m : Emrxr1 = bm1 for some x r (c)b m : Emrxr1 + EmrFr(nr)x(nr)1 = bm1 for some x
n
(d)
7/28/2019 linear algebra course part 3
47/65
3.4 Denote for convenience
E = E1
3-46
C(A) =
b m : Emrxr1 = bm1 for some x
r
(a)
b m : Emrxr1 = bm1 for some x r (b)b m : Emrxr1 = bm1 for some x
r (c)b m : Emrxr1 = bm1 for some x
r
(d)
7/28/2019 linear algebra course part 3
48/65
3.4 Denote for convenience
E = E1
3-47
Example. Find the condition on b under which Ax = b has solutions.
Answer:
R d
= rref(
A b
) =
Irr Fr(nr)
d1
...dr
0(mr)r 0(mr)(nr)
dr+1...
dm
.
We will have (m r) conditions, i.e.,
dr+1(b) = 0...
dm(b) = 0
Example. Find the condition on b under which
1 2 3 52 4 8 12
3 6 7 13
x = b has solutions.
Answer: rref(A b) = 1 2 0 2 4b1 32b2
0 0 1 1 b1 + 12b20 0 0 0 5b1 + b2 + b3
d3(b) = 5b1 + b2 + b3 = 0.
7/28/2019 linear algebra course part 3
49/65
3.4 Column space revisited 3-48The above example gives an alternative way to define the column space ofA.
C(A) =
b m : Amnxn1 = bm1 for some x n
= b m : Emrxr1 = bm1 for some x r=
b m :
dr+1(b) = 0...
dm(b) = 0
Recall that by R = EA, an interesting observation is that r pivot columns ofE = E1 is exactly the r pivot columns ofA! Hence, the above Emr are exactly
the r pivot columns ofA.
Example. Find the column space ofA34 = 1 2 3 5
2 4 8 12
3 6 7 13.Answer:
C(A) =
b 3 :
1 3
2 8
3 7
x21 = b31 for some x 2
=
b 3 : 5b1 + b2 + b3 = 0
2
7/28/2019 linear algebra course part 3
50/65
3.5 Independence, basis and dimension3-49
From the previous section, we learn that the dimension of column spaceC(Amn) is rank r, i.e., it is a linear combination ofr linearly independentm 1 vectors.
These linearly independent vectors span the space C(A). So they can bethe basis of the vector space C(A).
Definition (Basis): Independent vectors that span the space are called thebasis of the space.
Note: Span = All vectors in the space can be represented as a uniquelinear combinations of the basis. (Please note again unique is an important
key word here!) Uniqueness can be proved by independence as can be seen
by the exercise on the bottom of this slide.
Example. The all-zero vector 0 is always dependent on other vectors. Why?
Exercise (Uniqueness). Suppose {v1, . . . , vn} is a basis, and a vector v = a1v1 +
+ anvn = b1v1 + + bnvn. Show that ai = bi for 1 i n. (Hint: See p. 172
of the text.)
7/28/2019 linear algebra course part 3
51/65
3.5 Independence, basis and dimension3-50
Example. Prove that if the columns of A are linearly independent, then Ax = 0has a unique solution x = 0.
Proof: Suppose Ax = mi=1 xiai = 0 for some non-zero x. Then the columns
of A are not linearly independent according to the definition in slide 3-32, whichcontradicts to the assumption that the columns of A are linearly independent! 2
How to examine whether a group of vectors are independent or not?
Answer: Place them as column vectors of a matrix A.
They are independent if, and only if, Ax = 0 does not have non-zerosolution.
They are independent if, and only if, N(A) = N(R) = {0}.
They are independent if, and only if, no free variables.
They are independent if, and only if, Amn has full column rank, i.e.,r = n.
7/28/2019 linear algebra course part 3
52/65
3.5 Column space and row space ofAmn with rank r3-51
After the introduction of independence, we can re-define column space and row
space as follows.
Column space = linear combinations ofr independent columns.
These r independent columns are the basis of the column space.
The dimension ofC(A) is r.
Row space = linear combinations ofr independent rows.
These r independent rows are the basis of the row space. The dimension ofR(A) = C(AT) is r.
Null space = linear combinations ofn r independent vectors, each of which
is perpendicular to r independent rows ofA.
These n r independent vectors are the basis of the null space.
The dimension ofN(A) is n r.
7/28/2019 linear algebra course part 3
53/65
3.5 Column space and row space ofAmn and Rmn3-52
Column space = linear combinations ofr independent columns.
It is possible that C(A) = C(R).
However, they have the same rank.
Row space = linear combinations ofr independent rows.
R(A) = R(R). In fact, R(EA) = R(A) for invertible E.
Null space = linear combinations ofn r independent vectors, each of which
is independent ofr independent rows. N(A) = N(R). In fact, N(EA) = N(A) for invertible E.
Tip: C(AE) = C(A) for invertible E.
7/28/2019 linear algebra course part 3
54/65
3.5 Basis3-53
The number of basis for a vector space is unique.
Proof: Suppose w1, . . . , wn and v1, . . . , vm are two bases of the same vec-
tor space, and suppose n > m. By definition of basis, w1, . . . , wn can be
represented as linear combinations ofv1, . . . , vm. Hence,
W =
w1 wn
=
Va1 Van
= V Amn
where
V = v1 vm and A = a1 an .Then, m < n implies that Ax = 0 has non-zero solution (at least n mfree variables). Consequently, Wx = V Ax = V0 = 0, i.e., a non-zero linear
combination w1, . . . , wn equal 0, which implies wn is linearly dependent onthe other vectors w1, . . . , wn1. A contradiction to the linear independence
property of a basis is obtained.
The number of basis is also called the dimension () of the space.
The number of basis is sometimes referred as the degree of freedom (
) of the space.
7/28/2019 linear algebra course part 3
55/65
3.5 Basis3-54
This is an extension to the dimension or degree of freedom notion of the
Euclidean space.
Example. Find a basis of the vector space of polynomial equations of order
m, amxm + am1xm1 + + a0, where each ai .
Answer: In this vector space, V is a set of polynomial equations of order m
and scalar field F = . One of the basis can be
{1, x , x2, . . . , xm}.
All the vectors (i.e., polynomials) can be represented as a linear combination
of the basis.
Back to the Euclidean space, how to find the basis for a set of vectors?
Answer:Approach 1) Put them as the rows of a matrix A. The r pivot rows of
R = rref(A) are the answers.
Approach 2) Put them as the columns of a matrix A. Determine the r
pivot columns through R. Then, the r pivot columns ofA (not R) are the
answers.
7/28/2019 linear algebra course part 3
56/65
3.5 Extension to matrix space and function space3-55
Extension to Matrix space and Function space is only for your reference, and
is excluded from our focus (specifically is excluded from the exam).
It is however an important concept in the area of communications. E.g., to find
a basis of a group of communication signals.
Exercise. Can 0 be a part of the basis? Hint: 0 is linearly dependent on anyvector, including itself.
Definition (Independence): A set of vectors (v1, . . . , vm) is said to be lin-early independent if the linear combination of them (a1v1 + + amvm) equals
0 only when the coefficients are zero (a1 = a2 = = am = 0).
We may then modify the definition of basis as:
Definition (Basis): Independent (non-zero) vectors that span the space arecalled the basis of the space.
Note: Span = All vectors in the space can be represented as a unique linearcombinations of the basis. (Please note again unique is an important key word
here!)
7/28/2019 linear algebra course part 3
57/65
3.5 Extension to matrix space and function space3-56
Exercise. What is the basis for null space {0}? What is the dimension of this nullspace?
Hint 1: Put A = 0, and find the rank r of A.Hint 2: If we choose 0 to be the basis of such a null space, can all vectors in thespace {0} be represented as a unique linear combinations of the basis?
Answer: The dimension is 0 and no basis exists.
Example (Challenge). What are all the matrices that have the column space
spanned by two vectors v1 =
1
2
0
and v2 =
2
3
0
?
Answer: All 3 n matrices with rank r = 2 and with all columns are linear
combination of v1 and v2. I.e.,
v1 v2 c1,1 c1,2 c1,n
c2,1 c2,2 c2,n
with n 2 and at
least two columns ofC are not equal.
Important notion: All the columns ofA must lie in C(A), and the r pivot columnsform a basis.
7/28/2019 linear algebra course part 3
58/65
3.5 Extension to matrix space and function space3-57
Example (Challenge). What are all the matrices that have the null space spanned
by two vectors v1 =
1
2
0
and v2 =
2
3
0
?
Answer: All m 3 matrices with rank r = 1 and with all rows are linear combi-
nation of the vector w =
0
0
1
(because wTv1 = w
Tv2 = 0). I.e.,
c1,1
...
cm,1
wT with
m 1 and at least two rows ofC are unequal.Important notion: All the rows ofA must lie in the dual space ofN(A), i.e., row
space R(A). Also, (n r) = 2 is the dimension ofN(Amn).
Example (Challenge). Can the matrices in each of the above two examples forma vector space?
7/28/2019 linear algebra course part 3
59/65
3.6 Dimensions of the four subspaces3-58
What are the four subspaces?
Column space C(A)
Nullspace N(A)
Row space R(A) = C(AT)
Left nullspace N(AT): Solution y to ATy = 0
Equivalently, solution y to yTon the left of A
A = 0T.
It is therefore named left nullspace.
7/28/2019 linear algebra course part 3
60/65
3.6 Determination of the left nullspace3-59
Exercise (Review of key idea 5). The last (m r) rows ofE are a basis of the leftnullspace of A.
Recall that the basis (special solutions, i.e., Fr(nr)I(nr)(nr)) for thenullspace of A can be obtained through
R = rref(A) =
Irr Fr(nr)
0(mr)r 0(mr)(nr).
This exercise tells you that the special solutions of the left nullspace isgiven by Emm.
7/28/2019 linear algebra course part 3
61/65
3.6 Determination of the left nullspace3-60
Proof:
That EmmAmn =
Erm
E(mr)m
Amn = Rmn and the last (m r) rows of
R are all zeros implyE(mr)mAmn = 0(mr)n.
Hence, ATnmETm(mr) = 0n(mr).
Observe that the left nullspace requires AT
nmym1 = 0n1, and the columns ofEm(mr) are linearly independent (because E is invertible), and the dimensionof the left null space is (m r). The proof is then completed.
2
Recall from slide 3-48 that by R = EA, an interesting observation is that r pivotcolumns ofE = E1 is exactly the r pivot columns ofA! So these r columns are
a basis of the column space ofA.
Now we further show that the last (m r) rows of E are a basis of the left
nullspace of A.
7/28/2019 linear algebra course part 3
62/65
3.6 Column space and left nullspace (cf. slides 3-47 and 3-48) 3-61Example. Find the condition on b under which Ax = b has solutions.
Answer:
R d
= rref(
A b
) =
Irr Fr(nr)
d1...dr
0(mr)r 0(mr)(nr)
dr+1...
dm
.
We will have (m r) conditions, i.e.,
dr+1(b) = 0...
dm(b) = 0
Example. Find the condition on b under which
1 2 3 52 4 8 123 6 7 13
x = b has solutions.Answer: rref(A b) =
1 2 0 2 4b1 32
b20 0 1 1 b
1+ 1
2b
20 0 0 0 5b1 + b2 + b3 d3(b) = 5b1 + b2 + b3 = 0.
7/28/2019 linear algebra course part 3
63/65
3.6 Column space and left nullspace (cf. slides 3-47 and 3-48) 3-62The above example gives an alternative way to define the column space ofA.
C(A) =
b m : Amnxn1 = EmmRmnxn1 = bm1 for some x n
= b m : Emrxr1 = bm1 for some x r=
b m :
dr+1(b) = 0...
dm(b) = 0
Recall that by R = EA, an interesting observation is that r pivot columns ofE = E1 is exactly the r pivot columns ofA! Hence, the above Emr are exactly
the r pivot columns ofA.
Example. Find the column space of1 2 3 5
2 4 8 1 2
3 6 7 1 3.Answer:
C(A) =
b
3 :
1 3
2 8
3 7
x21 = b31 for some x
2
=
b 3 : 5b1 + b2 + b3 = 0
7/28/2019 linear algebra course part 3
64/65
3.6 Column space and left nullspace (cf. slides 3-47 and 3-48) 3-63
C(A) =
b m :
dr+1(b) = 0...
dm(b) = 0
gives the (m r) basis of the left
nullspace because the vectors in C(A) are orthogonal to those in N(AT).
Example. Find the left nullspace of
1 2 3 5
2 4 8 1 2
3 6 7 1 3
whose column space is written
as
C(A) =
b 3 : 5b1 + b2 + b3 = 0
Answer:
N(AT) =
y 3 : y = c
511
for some c 25 1 1 is exactly the last row ofE (cf. slide 3-59).
7/28/2019 linear algebra course part 3
65/65
3.6 Column spaces and row spaces 3-64
Tip. Amn = BmrCrn (and the rank of all matrices is r). Then,
C(Amn) = C(Bmr) and R(Amn) = R(Crn).
If the rank of A is not r, then we can only have
C(Amn) C(Bmr) and R(Amn) R(Crn).
For example, for 1 1
0 0 A
= 1 0
0 1 B
1 1
0 0 C
, then C(A) C(B). They are equal
when A and B have the same rank.
Exercise. Give Amn = BmrCrn. Then, the rank of A is no greater than r.
Example. Amn = EmmRmn, where R = rref(A). Then,
C(Amn) = C(Emm) = C(Amr) = C(Emr) and R(Amn) = R(Rmn),
where Amr =
Emr consists of the r-pivot columns ofA.