Chapter 10 LU Decomposition. L and U Matrices zLower Triangular Matrix zUpper Triangular Matrix

Chapter 10Chapter 10

LU DecompositionLU Decomposition

L and U MatricesL and U Matrices

Lower Triangular Matrix

Upper Triangular Matrix

44

3433

242322

13131211

u000

uu00

uuu0

uuuu

U

44344241

333231

2221

11

llll

0lll

00ll

000l

L

Another method for solving matrix equations Idea behind the LU Decomposition - start with

We know (because we did it in Gauss Elimination) we can write

4

3

2

1

4

3

2

1

44

3433

242322

14131211

d

d

d

d

x

x

x

x

u

uu

uuu

uuuu

d x U


x A b

LU DecompositionLU DecompositionAssume there exists [L]

Such that

This implies

b x A d x U L

b d L & AU L

44344241

333231

2221

11

llll

lll

ll

l

L

The Steps of LU DecompositionThe Steps of LU Decomposition

x B

L U A

L U


LU Decomposition * Based on Gauss elimination

*More efficient

Decomposition Methods (not unique)

* Doolittle decomposition lii = 1 * Crout decomposition uii = 1 (omitted) * Cholesky decomposition (for symmetric

matrices) uii = lii


Three Basic Steps

(1) Factor (decompose) [A] into [L] and [U]

(2) given {b}, determine {d} from [L]{d} = {b}

(3) using [U]{x} = {d} and back-substitution, solve for {x}

Advantage: Once we have [L] and [U], we can use many different {b}’s without repeating the decomposition process

LU decomposition / factorization [ A ] { x } = [ L ] [ U ] { x } = { b }Forward substitution [ L ] { d } = { b }Back substitution

[ U ] { x } = { d }Forward substitutions are more efficient

than elimination


1

4

23

5

F14

F23F12

F24

F45

H1

F35

F25

V2

V1

Simple TrussSimple Truss

W

Exampe: Forces in a Simple TrussExampe: Forces in a Simple Truss

0

0

0

0

0

0

0

100

0

0

F

F

F

F

F

F

F

V

H

V

1coscos0000000

0sinsin0000000

100cos0cos0000

000sin0sin0000

0cos00100000

0sin00000100

00coscos101000

00sinsin000000

00000cos1010

00000sin0001

45

35

25

24

23

14

12

3

1

1

[A] depends on geometry only; but {b} varies with applied load

LU FactorizationLU Factorization

Gauss elimination [A]{x}={b} elimination steps need to be repeated for

each different {b}

LU factorization / decomposition

[ A ] = [ L ] [ U ] does not involve the RHS {b} !!!

Doolittle LU DecompositionDoolittle LU Decomposition

Doolittle Algorithm

Get 1’s on diagonal of [L] (lii =1)

Operate on rows and columns sequentially, narrowing down to single element

Identical to LU decomposition based on Gauss elimination (see pp238-239), but different approach


44

3433

242322

14131211

434241

3231

21

44434241

34333231

24232221

14131211

u000

uu00

uuu0

uuuu

1lll

01ll

001l

0001

aaaa

aaaa

aaaa

aaaa

A

st21 21 11 31 31 11 41 41 111 column: / ; / ; /l a u l a u l a u

1414131312121111st au ;au ;au ;au :row1

uululululululululul

uululuululululul

uuluuluulul

uuuu

A

44344324421441334323421341224212411141

34243214313323321331223212311131

2414212313212212211121

14131211


nd31 12 32 22 32 41 12 42 22 42

32 32 31 12 22

42 42 41 12 22

2 column: ;

( ) /

( ) /

l u l u a l u l u a

l a l u u

l a l u u

nd21 12 22 22 21 13 23 23 21 14 24 24

22 22 21 12

23 23 21 13

24 24 21 14

2 : ; ;

row l u u a l u u a l u u a

u a l u

u a l u

u a l u

uululululululululul

uululuululululul

uuluuluulul

uuuu

A

44344324421441334323421341224212411141

34243214313323321331223212311131

2414212313212212211121

14131211


rd43 43 41 13 42 23 333 column: ( ) /l a l u l u u

31 13 32 23 33 33 31 14 32 24 34 34

33 33 31 13 32 23

34 34 31 14 32 24

3 : ;

rd row l u l u u a l u l u u a

u a l u l u

u a l u l u

th44 44 41 14 42 24 43 344 row: u a l u l u l u

uululululululululul

uululuululululul

uuluuluulul

uuuu

A

44344324421441334323421341224212411141

34243214313323321331223212311131

2414212313212212211121

14131211

Algorithm of Doolittle Decomposition

1, 1,

,1 ,1 1,1

[ ] [ ][ ] ( by matrix)

Staring the first row of , , for 1,2,......, ;

then the first column of , / , for 2,......, ;

Then alternatively determine the 2nd row o

i i

j j

A L U N N

U u a i N

L l a u j N

2, 2, 2,1 1,

,2 ,2 ,1 1,2 2,2

1th

, , , ,1

th

f ,

for 2,3,......, ;

and 2nd column of ;

( ) / , for 3,......, ; then

and n row of , for ,..., ;

and n column of

i i i

j j j

n

n i n i n k k ik

U

u a l u i N

L

l a l u u j N

U u a l u i n N

1

, , , , ,1

, / ,

for 1,..., ; ..........until row of [ ].

n

j n j n j k k n n nk

th

L l a l u u

j n N N U

Forward SubstitutionForward Substitution

4

3

2

1

444343242141

333232131

222121

111

4

3

2

1

44434241

333231

2221

11

b

b

b

b

dldldldl

dldldl

dldl

dl

d

d

d

d

llll

0lll

00ll

000l

d L

4434324214144

3323213133

2212122

1111

ldldldlbd

ldldlbd

ldlbd

lbd

/)(

/)(

/)(

/Very efficient for large matrices !

Once [L] is formed, we can use forward substitution instead of forward elimination for different {b}’s

Back SubstitutionBack Substitution

4

3

2

1

444

434333

424323222

414313212111

4

3

2

1

44

3433

242322

14131211

d

d

d

d

xu

xuxu

xuxuxu

xuxuxuxu

x

x

x

x

u000

uu00

uuu0

uuuu

x U

1141431321211

2242432322

3343433

4444

uxuxuxudx

uxuxudx

uxudx

udx

/)(

/)(

/)(

/Identical to

Gauss elimination

Forward SubstitutionForward Substitution

3314(2)6114dd6d1d

2(1/2)d2d

011d1d

1d

3214

23

12

1

b

1

2

1

1

d

d

d

d

11416

011/20

0011

0001

d L

4

3

2

1

33

2

0

1

d

Example:

Back-SubstitutionBack-Substitution

13/70x3x21x

8/35x2x

4/352x4x

33/707033/x

431

32

43

4

33/70

4/35

8/35

13/70

x}{

33

2

0

1

x

x

x

x

70000

4100

0420

3201

x U

4

3

2

1

Forward and Back SubstitutionsForward and Back Substitutions

Forward-substitution

Back-substitution (identical to Gauss elimination)

li

1jjijii n,1,2,i for dlbd

1232n1ni for u

xud

x

adx

ii

n

1ijjiji

i

nnnn

,,,,,

/

Error in textbook (p. 165)

Forward and Back SubstitutionsForward and Back Substitutions

Example: Forward and Back SubstitutionsExample: Forward and Back Substitutions» A=[1 0 2 3; -1 2 2 -3; 0 1 1 4; 6 2 2 4]; » b=[1 -1 2 1]';» [L,U] = LU_factor(A);L = 1.0000 0 0 0 -1.0000 1.0000 0 0 0 0.5000 1.0000 0 6.0000 1.0000 14.0000 1.0000U = 1 0 2 3 0 2 4 0 0 0 -1 4 0 0 0 -70

» x=LU_solve(L,U,b)x = -0.1857 0.2286 -0.1143 0.4714

Forward and back substitution

(without pivoting) MATLAB M-file

LU_factor

Pivoting in LU DecompositionPivoting in LU Decomposition

Still need pivoting in LU decomposition

Messes up order of [L]

What to do?

Need to pivot both [L] and a permutation matrix [P]

Initialize [P] as identity matrix and pivot when [A] is pivoted Also pivot [L]

LU Decomposition with PivotingLU Decomposition with Pivoting

Permutation matrix [ P ] - permutation of identity matrix [ I ]Permutation matrix performs “bookkeeping”

associated with the row exchangesPermuted matrix [ P ] [ A ]LU factorization of the permuted matrix

[ P ] [ A ] = [ L ] [ U ]Solution [ L ] [ U ] {x} = [ P ] {b}

Permutation MatrixPermutation MatrixBookkeeping for row exchangesExample: [ P1] interchanges row 1 and 3

Multiple permutations [ P ]

44434241

14131211

24232221

34333231

44434241

34333231

24232221

14131211

aaaa

aaaa

aaaa

aaaa

aaaa

aaaa

aaaa

aaaa

1000

0001

0010

0100

14131211

44434241

24232221

34333231

44434241

34333231

24232221

14131211

aaaa

aaaa

aaaa

aaaa

aaaa

aaaa

aaaa

aaaa

0001

1000

0010

0100

14226

24110

13221

13201

bA


1x4x2x2x6

2x4xx

1x3x2x2x

1 x3x2x

4321

432

4321

431

1

1

1

1

P

1

1

1

1

L

4226

4110

3221

3201

A

Start with

No need to consider {b} in decomposition

Forward EliminationForward Elimination

Interchange rows 1 & 4

Gauss elimination of first column

Save fi1 in the first column of [L]

0001

0100

0010

1000

P

11/6

10

11/6

1

L

7/35/31/30

4110

7/37/37/30

4226

A

1/6f

0f

1/6f

3201

4110

3221

4226

A

41

31

21


No interchange required

Gauss elimination of second column

Save fi2 in the second column of [L]

0001

0100

0010

1000

P

11/71/6

13/70

11/6

1

L

2200

5000

7/37/37/30

4226

A

1/7f

3/7f

7/35/31/30

4110

7/37/37/30

4226

A

42

32


Interchange rows 3 &4

Partial pivoting for [L] and [P]

0100

0001

0010

1000

P

13/70

11/71/6

11/6

1

L

5000

2200

7/37/37/30

4226

U

2200

5000

7/37/37/30

4226

A


Gauss elimination of third column

Save fi3 in third column of [L]

0100

0001

0010

1000

P

103/70

11/71/6

11/6

1

L

5000

2200

7/37/37/30

4226

U

0f

5000

2200

7/37/37/30

4226

A

43


0100

0001

0010

1000

P

4226

4110

3221

3201

A ;

2

1

1

1

b P

5000

2200

7/37/37/30

4226

U

103/70

011/71/6

0011/6

0001

L

Gauss elimination with partial pivoting


A P

4110

3201

3221

4226

5000

2200

7/37/37/30

4226

103/70

011/71/6

0011/6

0001

U L

33/14

5/7

5/6

1

d

d

d

d

2

1

1

1

d

d

d

d

103/70

011/71/6

0011/6

0001

d L

4

3

2

1

4

3

2

1

33/70

4/35

8/35

13/70

x

x

x

x

33/14

5/7

5/6

1

x

x

x

x

5000

2200

7/37/37/30

4226

x U

4

3

2

1

4

3

2

1

Forward substitution

Back substitution


partial pivoting

LU Decomposition with PivotingLU Decomposition with Pivoting» A=[1 0 2 3; -1 2 2 -3; 0 1 1 4; 6 2 2 4];» b=[1 -1 2 1]';» [L,U,P]=LU_pivot(A);L = 1.0000 0 0 0 -0.1667 1.0000 0 0 0.1667 -0.1429 1.0000 0 0 0.4286 0 1.0000U = 6.0000 2.0000 2.0000 4.0000 0 2.3333 2.3333 -2.3333 0 0 2.0000 2.0000 0 0 0 5.0000T1 = 6 2 2 4 -1 2 2 -3 1 0 2 3 0 1 1 4T2 = 6 2 2 4 -1 2 2 -3 1 0 2 3 0 1 1 4

T1 = [L][U]

T2 = [P][A]

Verify T1= T2

LU Decomposition with PivotingLU Decomposition with Pivoting» A=[1 0 2 3; -1 2 2 -3; 0 1 1 4; 6 2 2 4]; » b=[1; -1; 2; 1];» [L,U,P]=LU_pivot(A);» Pb=P*bPb = 1 -1 1 2» x=LU_Solve(L,U,Pb)d = 1.0000 -0.8333 0 0d = 1.0000 -0.8333 0.7143 0d = 1.0000 -0.8333 0.7143 2.3571

x = 0 0 -0.1143 0.4714x = 0 0.2286 -0.1143 0.4714x = -0.1857 0.2286 -0.1143 0.4714x = -0.1857 0.2286 -0.1143 0.4714

Forward Substitution Back Substitution

LU Decomposition

[L][U]{x} = [P]{b}

MATLAB’s MethodsMATLAB’s Methods

LU factorization [L,U] = lu (A)

-- returns [L] and [U] [L, U, P] = lu (A)

-- returns also the permutation matrix [P]Cholesky factorization: R = chol(A)

-- [A] = [R][R]Determinant: det (A)

MATLAB functionMATLAB function lulu» A=[1 0 2 3; -1 2 2 -3; 0 1 1 4; 6 2 2 4];

» b=[1; -1; 2; 1];» [L,U]=lu(A)L = 0.1667 -0.1429 1.0000 0 -0.1667 1.0000 0 0 0 0.4286 0 1.0000 1.0000 0 0 0U = 6.0000 2.0000 2.0000 4.0000 0 2.3333 2.3333 -2.3333 0 0 2.0000 2.0000 0 0 0 5.0000» L*Uans = 1 0 2 3 -1 2 2 -3 0 1 1 4 6 2 2 4

» d=L\b

d =

1.0000

-0.8333

0.7143

2.3571

» x=U\d

x =

-0.1857

0.2286

-0.1143

0.4714

Forward and Back

SubstitutionsLU Decomposition without Pivoting

Cholesky LU FactorizationCholesky LU Factorization

If [A] is symmetric and positive definite, it is convenient to use Cholesky decomposition.

[A] = [L][L]T= [U]T[U]

Cholesky factorization can be used for either symmetric or non-symmetric matrices

No pivoting or scaling needed if [A] is symmetric and positive definite (all eigenvalues are positive)

If [A] is not positive definite, the procedure may encounter the square root of a negative number


For a general non-symmetric matrix

For symmetric and positive definite matrices

44

3433

242322

14131211

44434241

333231

2221

11

44434241

34333231

24232221

14131211

d000

ud00

uud0

uuud

dlll

0dll

00dl

000d

aaaa

aaaa

aaaa

aaaa

A

44

3433

242322

14131211

44342414

332313

2212

11

44342414

34332313

24232212

14131211

u000

uu00

uuu0

uuuu

uuuu

0uuu

00uu

000u

aaaa

aaaa

aaaa

aaaa

A

Cholesky LU DecompositionCholesky LU Decomposition

11 11 12 12 11 13 13 11 14 14 111 column / row: u a ; u a /u ; u a /u ; u a /ust

221214242422121323232122222

2422241214232223121322222

212

nd

uuu au uuuau uau

auuuu auuuu auu :row / column2

/)(;/)(;

;;

33232413143434223

2133333

3434332423141333233

223

213

rd

uuuuuau uuau

auuuuuu auuu :row / column3

/)(;

;

234

224

214444444

244

234

224

214

th uuuau auuuu :row4

uuuuuuuuuuuuuuuu

uuuuuuuuuuuuuuu

uuuuuuuuuuuu

uuuuuuu

A

244

234

224

214333423241314222412141411

333423241314233

223

213222312131311

2224121422231213222

2121211

141113111211211

Example: Cholesky LUExample: Cholesky LU

13/3; u43/12; u23/6; u39u :column/row1 14131211st

012)1)( 2u 112)4 9u 125u

2uuuu 9uuuu 5uu :column/row2

24232

22

2224121422231213222

212

nd

/)((;/)((;)(

;;

2210410u 21421u

0uuuuuu 21uuu :column/row3

3422

33

343324231413233

223

213

rd

/)))(())(((;)()(

;

12016u 6uuuu :row4 22244

244

234

224

214

th )()()(

uuuuuuuuuuuuuuuu

uuuuuuuuuuuuuuu

uuuuuuuuuuuu

uuuuuuu

6023

021912

2956

31269

A

244

234

224

214333423241314222412141411

333423241314233

223

213222312131311

2224121422231213222

2121211

141113111211211


[A] = [U]T[U] = [U] [U]Recurrence relations

n,1,i j for u

uuau

uau

ii

1i

1kkjkiij

ij

1i

1k

2kiiiii

Script file for Cholesky DecompositionScript file for Cholesky Decomposition

Symmetric Matrix L = UCompute only the upper triangular elements

» A=[9 -6 12 -3; -6 5 -9 2; 12 -9 21 0; -3 2 0 6]A = 9 -6 12 -3 -6 5 -9 2 12 -9 21 0 -3 2 0 6» [L,U] = Cholesky(A)L = 3 0 0 0 -2 1 0 0 4 -1 2 0 -1 0 2 1 U = 3 -2 4 -1 0 1 -1 0 0 0 2 2 0 0 0 1

Symmetric

[L] = [U]

6023

021912

2956

31269

A

MATLAB Function: MATLAB Function: cholchol» A=[9 -6 12 -3; -6 5 -9 2; 12 -9 21 0; -3 2 0 6];» b=[24; -19; 51; 11];» U = chol(A)U = 3 -2 4 -1 0 1 -1 0 0 0 2 2 0 0 0 1» U'*Uans = 9 -6 12 -3 -6 5 -9 2 12 -9 21 0 -3 2 0 6» d = U'\bd = 8 -3 8 3» x = U\dx = 1 -2 1 3

Forward substitution

Back substitution

Documents

Chapter 10 LU Decomposition. L and U Matrices zLower Triangular Matrix zUpper Triangular Matrix