1

Linear Triangular System

bLx L – lower triangular matrix, nonsingular

2212122

1111

2

1

2

1

2221

11

/)(

/

0

axabx

abx

b

b

x

x

aa

a

Lx=bL: nxn nonsingular lower triangularb: known vector

b(1) = b(1)/L(1,1)For i=2:n b(i) = (b(i)-L(i,1:i-1)b(1:i-1))/L(i,i)end

Forward substitution, row version

2

Triangular System

Column version (column sweep method): As soon as a variable is solved, its effect can be subtracted from subsequent equations

16

1

7

13

5

2

89

05

3

5

2

6

897

051

002

3

2

1

3

2

1

x

x

x

x

x

x Lx = b

for j=1:n-1 b(j) = b(j)/L(j,j) b(j+1:n) = b(j+1:n)-b(j)L(j+1:n,j)endb(n) = b(n)/L(n,n)

Forward substitution, column version

Column version is more amenable to parallel computing

3

Triangular System: Parallel

L b

As soon as x_i (or a few x_i variables) is computed, the value is passed downward to neighboring cpus;As soon as a cpu receives x_i value, it passes the value downward to neighboring cpus;Then local b vector is updated.

Disadvantage: load imbalance, about 50% cpus are active on averageRemedy: cyclic or block cyclic distribution of rows.

L b

block Block cyclic

4

Triangular System: Inversion

113

12

12

111

23

1

0

0

AAAX

AX

AA

AA

AA

A – NxN lower triangularDivide A into equal blocks

Can inverse A recursively:Inverse A1;Inverse A2;Compute X by matrix multiplication

Matrix multiplication

5

Triangular System: Inversion1

4 2

-2 3 1

7 2 -3 3

-1 4 3 0 -1

0 1 2 5 -2 1

6 -1 3 0 0 1 -2

4 1 5 -3 2 0 0 1

1

4 1/2

-2 3 1

7 2 -3 1/3

-1 4 3 0 -1

0 1 2 5 -2 1

6 -1 3 0 0 1 -1/2

4 1 5 -3 2 0 0 1

1

-2 1/2

-2 3 1

7 2 1 1/3

-1 4 3 0 -1

0 1 2 5 -2 1

6 -1 3 0 0 1 -1/2

4 1 5 -3 2 0 0 1

First phase: invert diagonal elements of A2nd phase: compute 2x2 diagonal blocks of A^(-1)…K-th phase: compute diagonal 2^(k-1) x 2^(k-1) blocks of A^(-1)

Essentially matrix multiplications;K-th phase: N/2^(k-1) pairs of 2^(k-2)x2^(k-2) matrix multiplications

Can do in parallel on P=K^3 processors

6

Gaussian Elimination

476

953

21

21

xx

xx

143

953

2

21

x

xx

30

53

12

01

76

53

Ax = b

A = LU, L – unit lower triangular U – upper triangular

Ax = b LUx = b Ly = b, Ux = y

Especially with multiple rhs or solve same equations (same coefficient matrix) many times

7

A = LUA – nxn matrixA(1:k,1:k) nonsingular for k=1:n-1

(kij) versionFor k=1:n-1 for i=k+1:n A(i,k) = A(i,k)/A(k,k) for j=k+1:n A(i,j) = A(i,j) – A(i,k)A(k,j) end endend

orFor k=1:n-1 A(k+1:n,k) = A(k+1:n,k)/A(k,k) A(k+1:n,k+1:n) = A(k+1:n,k+1:n)- A(k+1:n,k)A(k,k+1:n)end

LU Factorization

After factorization, L is in strictly lower triangular part of A, U is in upper triangular part of A (including diagonal)

A(k,k) is the pivot

8

Factorization Breakdown

If A(k,k)=0 at any stage breakdown, LU factorization may not exist even if A is nonsingular

Theorem:Assume A is nxn matrix. (1) A has an LU factorization if A(1:k,1:k) is non-singular for all k=1:n-1.(2) If the LU factorization exists and A is non-singular, then the LU factorization is unique.

Avoid method breakdown pivotingPivoting is also necessary to improve accuracy. Small pivot increased errorsMake sure no large entries appear in L or U. Use large pivots.

9

Block LU Factorization

22

1211

2221

11

2221

1211

0

0

U

UU

LL

L

AA

AAA

A – nxn matrix, n = r*NA11 – rxr matrix, A22 – (n-r)x(n-r) matrix, A12 – rx(n-r) matrix, A21 – (n-r)xr

A11 = L11*U11A12 = L11*U12 U12A21 = L21*U11 L21A22 = L21*U12+L22*U22 A22-L21*U12 = A’ = L22*U22

LU factorization iteratively

10

Block LU Factorization

A – nxn matrixA(1:k,1:k) is non-singular for k=1:n-11<= r <= nUpon completion, A(i,j) overwritten by L(i,j) for i>j; A(i,j) overwritten by U(i,j) for i<=j

s = 1While s <= n q = min(n,s+r-1) Use scalar algorithm to LU-factorize A(s:q,s:q) into L and U Solve LZ = A(s:q,q+1:n) for Z and overwrite A(s:q,q+1:n) with Z Solve WU = A(q+1:n,s:q) for W and overwrite A(q+1:n,s:q) with W A(q+1:n,q+1:n) = A(q+1:n,q+1:n) – WZ s = q+1End

Matrix multiplication accounts for significant fraction of operations

11

Permutation MatrixPermutation matrix: identity matrix with its rows re-ordered.

0010

0100

0001

1000

P

p = [4 1 3 2] encodes permutation matrix P

p(k) is the column index of the “1” in k-th row

PA: row-permuted version of AAP: column-permuted version of A

0001

0100

0010

1000

E

Interchange permutation matrix: identity matrix with two rows swapped

Row 1 and 4 swapped

EA: swap rows 1 and 4 of AAE: swap columns 1 and 4 of A

12

Permutation Matrix

A permutation matrix can be expressed as a series of row interchanges:

12EEEP nIf E_{k} is the interchange permutation matrix with rows k and p(k) interchanged,Then P can be encoded by vector p(1:n).

If x(1:n) is a vector, then Px can be computed using p(1:n)

For k=1:n swap x(k) and x(p(k))End

p(1:n) vector is useful for pivoting

13

Partial PivotingPivoting is crucial to preventing breakdown and improving accuracy

Partial pivoting: choose largest element in a column (or row) and interchange rows (columns)

12186

242

10173

ASwap rows 1 and 3

10173

242

12186

1680

220

12186

Swap rows 2 and 3

220

1680

12186

600

1680

1286

14

LU Factorization with Row Partial Pivoting

A – nxn matrixAfter factorization, strictly lower triangular part of A contains L; upper triangular part contains U; vector p(1:n-1) contains permutation operations in partial pivoting

Algorithm F2:For k=1:n-1 Determine s with k<=s<=n s.t. |A(s,k)| is largest among |A(k:n,k)| swap A(k,1:n) and A(s,1:n) p(k) = s if A(k,k) != 0 A(k+1:n,k) = A(k+1:n,k)/A(k,k) A(k+1:n,k+1:n) = A(k+1:n,k+1:n)-A(k+1:n,k)A(k,k+1:n) endend

15

How to Use Factorized ASolve Ax = bUsing LU factorization of row partial pivoting

Need to swap elements of b according to partial pivoting information in p(1:n-1)

Assume A is LU factorized with row partial pivoting using algorithm F2:

For k=1:n-1 swap b(k) and b(p(k))EndSolve Ly = bSolve Ux = y

L - unit lower triangular matrix whose lower triangular part is the same as that of A; U - upper triangular part of A (including diagonal)

16

LU Factorization With Row Partial Pivoting

A – nxn matrixAfter factorization, strictly lower triangular part of A contains multipliers; upper triangular part contains U; vector p(1:n-1) contains permutation operations in partial pivoting

Algorithm F1:For k=1:n-1 Determine s with k<=s<=n s.t. |A(s,k)| is largest among |A(k:n,k)| swap A(k,k:n) and A(s,k:n) // only difference with F2 p(k) = s if A(k,k) != 0 A(k+1:n,k) = A(k+1:n,k)/A(k,k) A(k+1:n,k+1:n) = A(k+1:n,k+1:n)-A(k+1:n,k)A(k,k+1:n) endend

17

How to Use Factorized ASolve Ax = bUsing LU factorization of partial pivoting

Need to swap elements of b according to partial pivoting information in p(1:n-1)Need to multiply appropriate coefficients information in lower triangular part of A

Assume A is LU factorized with partial pivoting using algorithm F1:

For k=1:n-1 swap b(k) and b(p(k)) b(k+1:n) = b(k+1:n) – b(k)A(k+1:n,k)EndSolve Ux = b

U - upper triangular part of A (including diagonal)

18

Column Partial PivotingColumn partial pivoting: search row k for the largest element, exchange that columnwith column k.

A – nxn matrixAfter factorization, strictly lower triangular part of A contains L; upper triangular part contains U; vector p(1:n-1) contains permutation operations in partial pivoting

Algorithm G:For k=1:n-1 Determine s with k<=s<=n s.t. |A(k,s)| is largest among |A(k,k:n)| swap A(1:n,k) and A(1:n,s) p(k) = s if A(k,k) != 0 A(k+1:n,k) = A(k+1:n,k)/A(k,k) A(k+1:n,k+1:n) = A(k+1:n,k+1:n)-A(k+1:n,k)A(k,k+1:n) endend

19

How to Use Factorized ASolve Ax = bUsing LU factorization with column partial pivoting

Need to swap elements of x according to partial pivoting information in p(1:n-1)

Assume A is LU factorized with column partial pivoting using algorithm G:

Solve Ly = bSolve Ux = yFor k=n-1:-1:1 swap x(k) and x(p(k))end

L - unit lower triangular matrix whose lower triangular part is the same as that of A; U - upper triangular part of A (including diagonal)

20

Complete Pivoting

Complete pivoting: the largest element in submatrix A(k:n,k:n) is permuted into (k,k) as the pivot

Need a row interchange and a column interchange

A – nxn matrixp(1:n-1) – vector encoding row interchangesq(1:n-1) – vector encoding column interchanges

After factorization, lower triangular part of A contains L, upper triangular part of A contains U (including diagonal)

21

LU Factorization with Complete Pivoting

LU factorization with complete pivoting

For k=1:n-1 Determine s (k<=s<=n) and t (k<=t<=n) s.t. |A(s,t)| is largest among |A(i,j)| for i=k:n, j=k:n swap A(k,1:n) and A(s,1:n) swap A(1:n,k) and A(1:n,t) p(k) = s q(k) = t if A(k,k) != 0 A(k+1:n,k) = A(k+1:n)/A(k,k) A(k+1:n,k+1:n) = A(k+1:n,k+1:n)-A(k+1:n,k)A(k,k+1:n) endend

22

How to Use Factorized ASolve Ax = b By LU factorization with complete pivoting

Suppose A is LU factorized with complete pivoting, p(1:n-1) and q(1:n-1) are permutation encoding vectors

for k=1:n-1 swap b(k) and b(p(k))EndSolve Ly = b for ySolve Ux = y for xFor k=n-1:-1:1 swap x(k) and x(q(k))End

L and U are lower and upper triangular parts of factorized A

23

Parallelization of Gaussian Elimination

A(k,k)

Row-wise 1D block decomposition

At step k, the processor holding the pivot sends row k: A(k,k:n) to bottom neighboring processor;At each processor, forward data immediately to bottom neighbor upon receiving data from top processor; then update its own data; then wait for data from top neighbor

Disadvantage: load imbalanceRemedy: row-wise block cyclic distribution

24

Parallelization with Partial Pivoting

Row-wise block/block-cyclic decomposition

Gaussian elimination with column partial pivotingMore difficult with row partial pivoting

Pivoting search on the processor holding row k, no communication among processors;

Column index of the new pivot element together with row k: A(k:n) need to be sent out;

On each processor, upon receiving data from top neighbor, forward immediately to bottom neighbor, and swap column k and new pivot column of own data; update own data; wait data from top neighbor;