Lecture 10Dimensions, Independence, Basis and Complete Solution of Linear Systems

Shang-Hua Teng

Linear Independence

Linear Combination

in

v

,...,v,vv

n

1i i

21

is vectorsofset a of

Linear Independence

vectorsofset A 21 n,...,v,vv

is linearly independent if only if none of them can be expressed as a linear combination of the others

Examples

1

1

0

,

4

2

2

,

1

0

1

1

1

2

2

1

1

0

1,,

Linear Independence and Null Space

Theorem/Definition

vectorsofset A 21 n,...,v,vvis linearly independent if and only1v1+2v2+…+nvn=0 only happens when all ’s are zero

The columns of a matrix A are linearly independent when only solution to Ax=0 is x = 0

2D and 3Dv

w

u

v

How do we determine a set of vectors are independent?

Make them the columns of a matrix

Elimination

Computing their null space

Permute Rows and Continuing Elimination (permute columns)

011121

131111

021111

110011

A

Theorem

If Ax = 0 has more have more unknown than equations (m > n: more columns than rows), then it has nonzero solutions.

There must be free variables.

Echelon Matrices

*

*

*

*

000000

**0000

*****0

******

A

Free variables

Reduced Row Echelon Matrix R

1

0

0

0

000000

*10000

*0**10

*0**01

A

Free variables

Computing the Reduced Row Echelon Matrix

• Elimination to Echelon Matrix

E1PA = U

• Divide the row of pivots by the pivots

• Upward Elimination

E2E1PA = R

Example: Gauss-Jordan Method for Matrix Inverse

• [A I]

• E1[A I] = [U, I]

• In its reduced Echelon Matrix

• A-1 [A I] = [I A-1]

A Close Look at Reduced Echelon Matrix

• The last equation of R x = 0 is redundant

0 = 0

• Rank of A is the number of pivots rank(A).

00000

34100

12031

R

What is the Rank of Outer Product

n

n

vv

u

u

1

1

Rank and Reduced Row Echelon Matrix

0000000

1

0

0

0

000000

*10000

*0**10

*0**01

A

Free variables• Theorem/Definition

• Rank(A) = number of independent rows

• Rank(A) = number of independent columns

Dimension of the Column Space and Null Space

• The dimensions of the column space of A is equal to Rank(A).

• The dimension of the null space of A is equal to the number of free variables which is n – Rank(A)

• A is an m by n matrix

Rank and Reduced Row Echelon Matrix

0000000

1

0

0

0

000000

*10000

*0**10

*0**01

A

Free variablesFree Columns

Pivot columns

The Pivot columns are not combinations of earlier columns

Reduced Echelon and Null Space Matrix

• Nullspace Matrix• Special Solutions

0

0

0

00000

34100

12031

5

4

3

2

1

x

x

x

x

x

Rx

100

010

340

001

123

N

100

010

001

100

010

340

001

123

Null Space Matrix

• Ax=0 has n-Rank(A) free variables and special solutions

• The Nullspace matrix has n-Rank(A) columns• The columns of the nullspace matrix are

independent• The dimension of the Null space is n – rank(A)

Complete Solution of Ax = 0• After column permutation, we can write

rows zeros

rowspivot

00 m-r

rFIR

r pivot columns n-r free columns

• Nullspace matrix

m-r

r

I

FN

Pivot variables

Free variables

• Moreover: RN = [0]

Complete Solution to Ax = b• A is an m by n matrix, and b is an n-place vector

– Unique solution– Infinitely many solution– No solution

Suppose Ax = b has more then one solution, say x1, x2 then A x1 = bA x2 = b

So A (x1 - x2 ) = 0

(x1 - x2 ) is in nullspace(A)

Complete Solution to Ax = b

Suppose we found a particular solution xp to Ax = b i.e, A xp = b

Let F be the indexes of free variables of Ax = 0 Let xF be the column vector of free variablesLet N be the nullspace matrix of A

Then

defines the complete set of solutions to Ax = b

FNx xp

Example: Complete Solution to Ax = b

7

6

1

6131

4100

2031

4

3

2

1

x

x

x

x

7

6

1

6131

4100

2031

Augmented matrix [A b]

Elimination to obtain [R d]

0

6

1

0000

4100

2031

Set free variables to 0 to find a particular solution(1,0,6,0)T

Compute the nullspace matrix

10

40

01

23

Complete solution is

1

4

0

2

0

0

1

3

0

6

0

1

10

40

01

23

0

6

0

1

424

2 xxx

xx

Full Rank Matrix

• Suppose A is an m by n matrix. Then

• A is full column if rank(A) = n– columns of A are independent

• A is full row rank if rank(A) = m– Rows of A are independent

),min()(rank nmA

Full Column Rank Matrix

• Columns are independent

• All columns of A are pivot columns

• There are non free variables or special solutions

• The nullspace N(A) contains only the zero vector

• If Ax=b has a solution (it might not) then it has only one solution

0

IR

n by n

m-n rows of zeros

Full Row Rank Matrix

• Rows are independent

• All rows of A have pivots, R has no zero rows

• Ax=b has a solution for every right hand side b

• The column space is the whole space Rm

• There are n-m special solutions in the null space of A

FIR

The Whole Picture• Rank(A) = m = n Ax=b has unique solution IR

FIR

0

IR

00

FIR

• Rank(A) = m < n Ax=b has n-m dimensional solution

• Rank(A) = n < m Ax=b has 0 or 1 solution

• Rank(A) < n, Rank(A) < m Ax=b has 0 or n-rank(A) dimensions

Basis and Dimension of a Vector Space

• A basis for a vector space is a sequence of vectors that – The vectors are linearly independent– The vectors span the space: every vector in the

vector can be expressed as a linear combination of these vectors

Basis for 2D and n-D

• (1,0), (0,1)

• (1 1), (-1 –2)

• The vectors v1,v2,…vn are basis for Rn if and only if they are columns of an n by n invertible matrix

Column and Row Subspace

• C(A): the space spanned by columns of A– Subspace in m dimensions

– The pivot columns of A are a basis for its column space

• Row space: the space spanned by rows of A– Subspace in n dimensions

– The row space of A is the same as the column space of AT, C(AT)

– The pivot rows of A are a basis for its row space

– The pivot rows of its Echolon matrix R are a basis for its row space

Important Property I: Uniqueness of Combination

• The vectors v1,v2,…vn are basis for a vector space V, then for every vector v in V, there is a unique way to write v as a combination of v1,v2,…vn .

v = a1 v1+ a2 v2+…+ an vn

v = b1 v1+ b2 v2+…+ bn vn

• So: 0=(a1 - b1) v1 + (a2 -b2 )v2+…+ (an -bn )vn

Important Property II: Dimension and Size of Basis

• If a vector space V has two set of bases– v1,v2,…vm . V = [v1,v2,…vm ]– w1,w2,…wn . W= [w1,w2,…wn ].

• then m = n– Proof: assume n > m, write W = VA– A is m by n, so Ax = 0 has a non-zero solution– So VAx = 0 and Wx = 0

• The dimension of a vector space is the number of vectors in every basis– Dimension of a vector space is well defined

Dimensions of the Four SubspacesFundamental Theorem of Linear

Algebra, Part I• Row space: C(AT) – dimension = rank(A)

• Column space: C(A)– dimension = rank(A)

• Nullspace: N(A) – dimension = n-rank(A)

• Left Nullspace: N(AT) – dimension = m –rank(A)

•