Systems of linear equationsfaculty.arts.ubc.ca/pschrimpf/526/lec02slides.pdf · Systems of linear equations Paul Schrimpf Cardinality review Introduction and deﬁnitions Gaussian

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview

Introductionand definitions

Gaussianelimination

Existence ofsolutions

Uniqueness ofsolutions

Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Systems of linear equations

Paul Schrimpf

UBCEconomics 526

September 10, 2013

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

More cardinality

• Let A and B be sets, the Cartesion product of A and Bis

A × B := {(a,b) : a ∈ A,b ∈ B}

QuestionIf A and B are countable is A × B countable?

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

More cardinality

TheoremIf A and B are countable, then so is A × B.

Proof.• A and B are countable, so we can write A = {a1,a2, ...}

and B = {b1,b2, ...}.• Consider

(a1,b1) (a1,b2) (a1,b3) · · ·(a2,b1) (a2,b2) (a2,b3) · · ·(a3,b1) (a3,b2) (a3,b3) · · ·

... . . .

• Count along the diagonals

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

More cardinality

QuestionIf A1,A2,A3, ... are each countable is ∪∞

n=1An countable?

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

More cardinality

TheoremIf {An} are countable, then so is ∪∞

n=1An.

Proof.• An is countable, so we can write An = {a1n,a2n, ...}• Consider

a11 a12 a13 · · ·a21 a22 a23 · · ·a31 a32 a33 · · ·

... . . .

• Count along the diagonals

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

..1 Introduction and definitions

..2 Gaussian elimination

..3 Existence of solutions

..4 Uniqueness of solutions

..5 Set of solutions

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Systems of linear equations

• Example:

5x1 − 7x2 =9−8x1 + x2 =0.

• In general:

a11x1 + a12x2 + ...+ a1nxn =b1

a21x1 + a22x2 + ...+ a2nxn =b2......

am1x1 + am2x2 + ...+ amnxn =bm,

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Coefficient matrix

• Coefficient matrix Aa11 · · · a1n...

...am1 · · · amn

x1

...xn

=

b1...

bm

Ax =b

• Augmented coefficient matrix A

A =

a11 · · · a1n b1...

......

am1 · · · amn bm

=(Ab

)

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Example: Markov model ofemployment

• Let st =

{1 if employed at time t0 if unemployed at time t

• Random process described by P(st |st−1, st−2, ...)

• Markovian: P(st |st−1, st−2, ...) = P(st |st−1)• Probability of being employed tomorrow only depends

on whether you’re employed today and not the moredistant past

• Stationary distribution: q stationary if whenP(st = 1) = q(1) and P(st = 0) = q(0) today, then alsohave P(st+1 = 1) = q(1) and P(st+1 = 0) = q(0)tomorrow

q(s) =∑

s0∈{0,1}P(s|s0)q(s0)

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.


• Stationary distribution satisfies:

q(1) =P(1|1)q(1) + P(1|0)q(0)q(0) =P(0|1)q(1) + P(0|0)q(0)

1 =q(1) + q(0)

system of linear equations for unknowns q(1) and q(2)• Coefficient matrix and augmented coefficient matrix = ?

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Questions to be answered:

Given a system of linear equations:..1 Does any solution exist?..2 How many solutions?..3 How can a solution be computed?

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Equation/row operations

• Familiar with solving equations by:• Substitution• Elimination

• Use equation (row) operations that preserve set ofsolutions

..1 Multiply an equation by a non-zero constant,

..2 Add a multiple of one equation to another, and

..3 Interchange two equations.

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Row echelon form

• Row echelon form: each row begins with more zerosthan the row above it or the row is all zeros

• Examples: (a11 a120 a22

)(

a11 a12 a130 0 a23

)• Once a system is in row echelon form it is easy to solve

by back substitution

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Gaussian elimination

• Systematic way to transform system of equations to rowechelon form

..1 Identify the first column to contain any non-zeroelements, call this column c∗.

..2 Interchange rows so that a nonzero entry appears at thetop of column c∗.

..3 Add a multiple of the first row to each of the rows belowso that the entries in column c∗ below the first row arezero.

..4 Repeat 1-2 on the submatrix consisting of the lowerright part of the original matrix below the first row and tothe right of column c∗. Stop if this submatrix has nocolumns or has no rows.

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Gaussian elimination: example

x1 + x2 + 3x3 = 02x1 + 3x2 + 7x3 = 93x1 + 5x2 + 13x3 = 1

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Transform the following system into row echelon form:

x + 2y − z =24y + z =5

−2x − 4y + 2z =1.

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Existence of solutions

• rank of A is the number of nonzero rows in its rowechelon form

• Is rank well-defined?

LemmaThe rank of a matrix A is always less than or equal to thenumber of columns of A and less than or equal to thenumber of rows of A.

LemmaLet A be a coefficient matrix and A be an augmentedcoefficient matrix. Then rankA ≤ rankA.

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Existence of solutions

Theorem (Existence of solutions)A system of linear equations with coefficient matrix A andaugmented coefficient matrix A has a solution (perhapsmore than one) if and only if rankA = rankA.

Proof.See notes.

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Consider the system:

4y + z =5x + 2y − z =2−8y − 2z =− 10.

• What if b3 = −10?

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Multiple solutions means infinitesolutions

LemmaSuppose x1 and x2 are two distinct solutions to the systemof equations Ax = b. Then the system of equations has(uncountably) infinitely many solutions.

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Solution existence for any b

Theorem (Solution existence)A system of linear equations with coefficient matrix A willhave a solution for any choice of b1, ..., bm if and only ifrankA is equal to the number of rows of A.

CorollaryFor any system of equations with more equations thanvariables, (i.e. an overdetermined system) there exists achoice of b such that no solutions exist.

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Uniqueness

Theorem (Solution uniqueness)Any system of equations with coefficient matrix A has atmost one solution for any b1, ..., bm if and only if rankAequals the number of columns of A.

CorollaryIf rankA is less than the number of columns of A then eitherno solutions exists or multiple solutions exists.

CorollaryA is nonsingular (always has a unique solution) if and only ifA has an equal number of columns and rows (A is square)and has rank equal to its number of columns (or rows).

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Summary

rankA = rankA rankA < rankArankA = columns(A) rankA < n

0 X1 X∞ X

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Sets of solutions

−2x + y = 1 x − y − z = 0−2x + y + 3z = 1 −2x − y + z = 2

-10

-5

0

5

10

15

-6 -4 -2 0 2 4 6

y

x

-2

0

2

4

6

8

10

12

x

-1

0

1

2

3

4

5

6

y

0

1

2

3

4

5

6

7

z

-10

-5

0

5

10

-10

-5

0

5

10-20

-10

0

10

20

30

z

x

y

z

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

DefinitionThe set S ⊆ Rn is called a linear subspace if it is closedunder (i) scalar multiplication and (ii) addition in other words,if

(i) for every (x1, ..., xn) ∈ S and a ∈ R, we have(ax1,ax2, ..., axn) ∈ S, and

(ii) for every (x1, ..., xn) ∈ S and (y1, ..., yn) ∈ S, we have(x1 + y1, ..., xn + yn) ∈ S

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

DefinitionA set of vectors in Rn, {xj = (x j

1, ..., xjn)}J

j=1, is linearlyindependent if the only solution to

J∑j=1

cjxj = 0

is c1 = c2 = ... = cJ = 0.

DefinitionThe dimension of a linear subspace S ⊆ Rn is the cardinalityof the largest set of linearly independent elements in S.

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Theorem (Rouche-Capelli)A system of linear equations with n variables has a solutionif and only if the rank of its coefficient matrix, A, is equal tothe rank of its augmented matrix, A. If a solution exists andrankA is equal to its number of columns, the solution isunique. If a solution exists and rankA is less than its numberof columns, there are infinite solutions. In this case the setof solutions is of the form1

{s + x∗ ∈ Rn : s ∈ S and Ax∗ = b}

where S is the linear subspace of dimension n − rankAdefined by S = {s ∈ Rn : As = 0} and x∗ is any singlesolution to Ax = b.

1A set of this form is called an affine subspace. It is a linear subspacethat has been shifted so that it no longer necessarily contains the origin.

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.


• Employment st ∈ {u,e}.• Markovian: P(st |st−1, st−2, ...) = P(st |st−1)

• Stationary distribution: (πe, πu) such that if st = e withprobability πe and u with probability πu, then st+1 hassame distribution, i.e.

P(e|e)πe + P(e|u)πu =πe

P(u|e)πe + P(u|u)πu =πu

Systems oflinear

equations

Paul Schrimpf

Cardinalityreview


Gaussianelimination



Set ofsolutions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.


• Stationary distribution is a solution to:

(pee − 1)πe + peuπu =0pueπe + (puu − 1)πe =0

πe + πu =1.

• Questions:..1 Does any solution exist?..2 How many solutions exist?..3 How can a solution be computed?

Documents

Systems of linear equationsfaculty.arts.ubc.ca/pschrimpf/526/lec02slides.pdf · Systems of linear equations Paul Schrimpf Cardinality review Introduction and deﬁnitions Gaussian