Ax=bZack

10/4/2013

Iteration method

Ax=b 𝑣1, 𝑣2… 𝑥𝑘 = 𝑉𝑘𝑦𝑘

Lanczos

• Given (𝐴, 𝑏) standard orthonormal base 𝑣1, 𝑣2… 𝑥𝑘 = 𝑉𝑘𝑦𝑘

• To generate a standard orthonormal base:• Step 1, set v1

• Step 2, if we have k orthonormal vectors 𝑉𝑘 = (𝑣1, 𝑣2…𝑣𝑘) , generate 𝑣𝑘+1

Lanczos

• Step 1: 𝑣1 =𝑏

𝑏, 𝛽 is a parameter to

uniformization vector v• 𝛽1𝑣1 = 𝑏

• Step 2: If we have 𝑉𝑘 = (𝑣1, 𝑣2…𝑣𝑘)• 𝛽𝑘+1𝑣𝑘+1 = 𝐴𝑣𝑘 + 𝑉𝑘𝑧

• Where z is a vector

Lanczos

• 𝛽𝑘+1𝑣𝑘+1 = 𝐴𝑣𝑘 + 𝑉𝑘𝑧

• 𝑣𝑘𝑇𝑣𝑘+1 = 0 𝑣𝑘

𝑇𝐴𝑣𝑘 + 𝑣𝑘𝑇𝑉𝑘𝑧 = 0

𝑒𝑘𝑇𝑧 = −𝑣𝑘

𝑇𝐴𝑣𝑘

• 𝑣𝑖𝑇𝑣𝑘+1 = 0 𝑖 < 𝑘𝑣𝑖

𝑇𝐴𝑣𝑘 + 𝑣𝑖𝑇𝑉𝑘𝑧 = 0

𝑒𝑖𝑇𝑧 = 𝑣𝑖

𝑇𝐴𝑣𝑘

Lanczos

• Notice that we have:𝛽𝑖+1𝑣𝑖+1 = 𝐴𝑣𝑖 + 𝑉𝑖𝑧

𝑣𝑘𝑇𝛽𝑖+1𝑣𝑖+1 = 𝑣𝑘

𝑇𝐴𝑣𝑖 + 𝑣𝑘𝑇𝑉𝑖𝑧

if i==k-1

𝛽𝑘 = 𝑣𝑘𝑇𝐴𝑣𝑖

else 0 = 𝑣𝑘𝑇𝐴𝑣𝑖

• 𝑒𝑖𝑇𝑧 = 𝑣𝑖

𝑇𝐴𝑣𝑘 = 𝑣𝑘𝑇𝐴𝑣𝑖 =

−𝛽𝑘 (𝑖 = 𝑘 − 1)0 (𝑖 < 𝑘 − 1)

Lanczos

• 𝛽𝑘+1𝑣𝑘+1 = 𝐴𝑣𝑘 − 𝑣𝑘𝑇𝐴𝑣𝑘 𝑣𝑘 − 𝛽𝑘𝑣𝑘−1

define 𝛼𝑘 = 𝑣𝑘𝑇𝐴𝑣𝑘

𝐴𝑣𝑘 = 𝛽𝑘+1𝑣𝑘+1 + 𝛼𝑘𝑣𝑘 + 𝛽𝑘𝑣𝑘−1 𝐴𝑉𝑘 = 𝑉𝑘+1𝐻𝑘

where, 𝐻𝑘 =

𝛼1 𝛽2𝛽2 𝛼2 𝛽3

⋱ ⋱ ⋱𝛽𝑘 𝛼𝑘

𝛽𝑘+1

Lanczos

• We can also define a tridiagonal matrix 𝑇𝑘 :

𝑇𝑘 =

𝛼1 𝛽2𝛽2 𝛼2 𝛽3

⋱ ⋱ ⋱𝛽𝑘 𝛼𝑘

;

𝐴𝑉𝑘 = 𝑉𝑘𝑇𝑘 + 𝛽𝑘+1𝑣𝑘+1𝑒𝑘𝑇

• Define 𝑥𝑘 = 𝑉𝑘𝑦𝑘 is approximation of x.

• residual vector

𝑟𝑘 ≡ 𝑏 − 𝐴𝑥𝑘= 𝛽1𝑣1 − 𝐴𝑉𝑘𝑦𝑘= 𝑉𝑘+1 𝛽1𝑒1 −𝐻𝑘𝑦𝑘= 𝑉𝑘+1𝑡𝑘+1

where 𝑡𝑘+1 = 𝛽1𝑒1 −𝐻𝑘𝑦𝑘

• minimize function 𝑓𝑘(𝑦𝑘)

• 𝑓𝑘(𝑦𝑘) = 𝑟𝑘𝑇𝐵𝑟𝑘

= 𝑏 − 𝐴𝑥𝑘𝑇𝐵 𝑏 − 𝐴𝑥𝑘

= 𝑏 − 𝐴𝑉𝑘𝑦𝑘𝑇𝐵 𝑏 − 𝐴𝑉𝑘𝑦𝑘

• 𝑓𝑘 𝑦𝑘 has a stationary value at 𝑦𝑘 if

𝑉𝑘𝑇𝐴𝐵𝐴𝑉𝑘𝑦𝑘 = 𝑉𝑘

𝑇𝐴𝐵𝑏

• Case (a)

𝐵 = 𝐴−1 (𝐴− 𝑖𝑓 𝐴 𝑖𝑠 𝑠𝑖𝑛𝑔𝑢𝑙𝑎𝑟)

• Case (b)

𝐵 = 𝐼

Lanczos

• 𝑉𝑘𝑇𝐴𝐵𝐴𝑉𝑘𝑦𝑘 = 𝑉𝑘

𝑇𝐴𝐵𝑏

𝑉𝑘𝑇𝐴𝑉𝑘𝑦𝑘 = 𝑉𝑘

𝑇𝑏

𝑉𝑘𝑇(𝑉𝑘𝑇𝑘 + 𝛽𝑘+1𝑣𝑘+1𝑒𝑘

𝑇)𝑦𝑘 = 𝑉𝑘𝑇𝛽1𝑣1

𝑇𝑘𝑦𝑘 = 𝛽1𝑒1

• 𝑇𝑘𝑦𝑘 = 𝛽1𝑒1

• Use LDLT decomposition, 𝑇𝑘 = 𝐿𝑘𝐷𝑘𝐿𝑘𝑇 . 𝐿𝑘 is a

bidiagonal lower matrix, 𝐷𝑘 is a diagonal matrix.

• 𝐿𝑘𝐷𝑘𝐿𝑘𝑇𝑦𝑘 = 𝛽1𝑒1

• Define 𝑧𝑘 = 𝐿𝑘𝑇𝑦𝑘 , 𝐿𝑘𝐷𝑘𝑧𝑘 = 𝛽1𝑒1

• Define 𝑊𝑘𝑇 = 𝐿𝑘

−1𝑉𝑘𝑇

• 𝑥𝑘 = 𝑉𝑘𝑦𝑘 = 𝑊𝑘𝐿𝑘𝑇𝑦𝑘 = 𝑊𝑘𝑧𝑘

1 1 1

0 1

T Tk k k

k T Tk k k

L W VL

w v

1

T T T

k k k kw w v

1 1

1 1 1 1 1

2

1 1

0 0

0 101

k k T TTk k k k k k kk k k k

k k

k k k k k

L DL L D L dT L D L

dd d d

1

2

1

k k k

k k k k

d

d d

1 1 1 11 1

1 11

=0 01

k k k kk k

k k kk k k k kk k

L D L Dz zL D z e

d d d

1 1 0k k k k kd d

1

1 1 1 1

k

k k k k k k k k k k k k

k

zx W z W w W z w x w

This method is named CG

• if A is an indefinite symmetric matrix, LDLT

decomposition can still be tried, often success, but it does not always exist, and can no longer be relied upon numerically.

• Use orthogonal factorization instead.

• 𝑇𝑘 = 𝐿𝑘𝑄𝑘 , 𝑄𝑘 is orthonormal matrix, 𝐿𝑘 is low matrix

• Define 𝑄𝑖,𝑖+1 is identity matrix except the elements𝑞𝑖𝑖 = −𝑞𝑖+1,𝑖+1 = 𝑐𝑖𝑞𝑖,𝑖+1 = 𝑞𝑖+1,𝑖 = 𝑠𝑖 (𝑠𝑖

2 + 𝑐𝑖2 = 1)

• 𝑄𝑖,𝑖+1 is orthonormal matrix, and 𝑄𝑘 :

is also orthonormal matrix.

1,2 2,3 2, 1 1,

T

k k k k kQ Q Q Q Q

• Prove that

𝐿𝑘 = 𝑇𝑘𝑄𝑘𝑇 can be a lower tridiagonal matrix with reasonable 𝑐𝑖 , 𝑠𝑖

• Proof: Assume that 𝐿𝑘 = 𝑇𝑘𝑄𝑘

𝑇 is a low tridiagonal matrix:

1

2 2

3 3 3k

k k k

L

• 𝑇𝑘+1𝑄𝑘𝑄𝑘,𝑘−1

1

1 1

1

1

k k

k k k

k k k k

T Q

c s

s c

1 1

1 1 1 1 1 1

1 1

= =k k k

k k k k k k

k k k k k k k k k k

T Q L

c s c s

s c s c s c s c

• we want the matrix below to be a lower matrix:

1 1

1 1 1 1 1

=k

k k k k k k k k

k k k k k k k k k

L

c s s c

s c c s c s c

1

1

=0

=

k k k k

k k k k k

s c

c s

22

1kk k kk

k

c

1kk

k

s

So 𝐿𝑘 = 𝑇𝑘𝑄𝑘𝑇 can be a lower tridiagonal matrix

• 𝐿𝑘𝑧𝑘 = 𝛽1𝑒1

• define 𝑧𝑘 , 𝐿𝑘𝑧𝑘 = 𝛽1𝑒1

𝑥𝑘𝐿 = 𝑊𝑘𝑧𝑘 = 𝑥𝑘

𝐶 − ζ𝑘+1𝑤𝑘+1

• This method is named SYMMLQ

1 2 1

T

k k k k kW w w w w V Q

1 2 1k k k k kz Q y

• If we set B=I

𝑉𝑘𝑇𝐴𝐵𝐴𝑉𝑘𝑦𝑘 = 𝑉𝑘

𝑇𝐴𝐵𝑏

𝑉𝑘𝑇𝐴2𝑉𝑘𝑦𝑘 = 𝑉𝑘

𝑇𝐴𝑏

𝑉𝑘𝑇𝐴2𝑉𝑘 = 𝑇𝑘

2 + 𝛽𝑘2𝑒𝑘𝑒𝑘

𝑇

𝑉𝑘𝑇𝐴𝑏 = 𝑉𝑘

𝑇𝐴𝛽1𝑣1 = 𝛽1𝑇𝑘𝑒1

• 𝑇𝑘2 + 𝛽𝑘

2𝑒𝑘𝑒𝑘𝑇 = 𝐿𝑘 𝐿𝑘

𝑇 + 𝛽𝑘2𝑒𝑘𝑒𝑘

𝑇 = 𝐿𝑘𝐿𝑘𝑇

• 𝑉𝑘𝑇𝐴2𝑉𝑘𝑦𝑘 = 𝑉𝑘

𝑇𝐴𝑏

𝐿𝑘𝐿𝑘𝑇𝑦𝑘 = 𝛽1𝑇𝑘𝑒1

𝐿𝑘𝐿𝑘𝑇𝑦𝑘 = 𝛽1𝐿𝑘𝑄𝑘𝑒1

𝐿𝑘𝐿𝑘𝑇𝑦𝑘 = 𝛽1𝐿𝑘𝐷𝑘𝑄𝑘𝑒1

𝐿𝑘𝑇𝑦𝑘 = 𝛽1𝐷𝑘𝑄𝑘𝑒1

• 𝐿𝑘𝑇𝑦𝑘 = 𝛽1𝐷𝑘𝑄𝑘𝑒1 = (𝜏1 …𝜏𝑘)

𝑇= 𝑡𝑘

𝜏1 = 𝛽1𝑐1 , 𝜏𝑖 = 𝛽1𝑠1𝑠2… 𝑠𝑖−1𝑐𝑖

𝑥𝑘𝑀 = 𝑉𝑘𝑦𝑘 = 𝑉𝑘𝐿𝑘

−𝑇𝐿𝑘𝑇𝑦𝑘 = 𝑀𝑘𝑡𝑘

where 𝑀𝑘 = 𝑉𝑘𝐿𝑘−𝑇

• This method is named MINRES

• CG• A must be positive definite matrix

• MINRES• Any symmetric A, 𝑟𝑘

𝑀 decreases monotonically

• Risk of cancellation error when A is indefinite

• SYMMLQ• QR factor, Any symmetric A, except Ax=b must be consistent, Very little cancellation error, method of choice for indefinite consistent system

Unsymmetric and rectangular iteration• Set 𝛼, 𝛽 as normalized parameters of 𝑢, 𝑣 , 𝛽1𝑢1 =𝑏, 𝛼1𝑣1 = 𝐴𝑇𝑢1

follow this iteration:

generate 2 orthonormal base

𝑈𝑘 = 𝑢1 𝑢2 …𝑢𝑘 𝑉𝑘 = 𝑣1 𝑣2 …𝑣𝑘

1 1

1 1 1 1

k k k k k

T

k k k k k

u Av u

v A u v

Unsymmetric and rectangular iteration• PROOF:

assume that we have orthonormal vectors

𝑢1 𝑢2 …𝑢𝑘 , 𝑣1 𝑣2…𝑣𝑘generate 𝑢𝑘+1 , 𝑣𝑘+1 as:

1 1

1 1 1 1

k k k k k

T

k k k k k

u Av u

v A u v

Unsymmetric and rectangular iteration

1

T T T T

k i i k i k i iv v v A u v v

1 1

T T T

i k k i k i k ku u u Av u u

When i=k

1 1 1+ 0T T T T T T

k k k k k k k k k k k k k k k k ku u u Av u u v v v v u u

When i<k

1 1 1= + =0T T T T T T

i k k i k i k k k i i k i i i k ku u u Av u u v v v v u u

So 1 1k ku u u 1 1k kv v v , we can also prove that

Unsymmetric and rectangular iteration• 𝐴𝑉𝑘 = 𝑈𝑘+1𝐵𝑘 = 𝑈𝑘𝐿𝑘 + 𝛽𝑘+1𝑢𝑘+1𝑒𝑘

𝑇

• 𝐴𝑇𝑈𝑘+1 = 𝑉𝑘𝐵𝑘𝑇 + 𝛼𝑘+1𝑣𝑘+1𝑒𝑘+1

𝑇 = 𝑉𝑘+1𝐿𝑘+1𝑇

• 𝐿𝑘 =

𝛼1𝛽2 𝛼2

⋱ ⋱𝛽𝑘 𝛼𝑘

𝐵𝑘 =

𝛼1𝛽2 𝛼2

⋱ ⋱𝛽𝑘 𝛼𝑘

𝛽𝑘+1

Direct method: LUSOL

• Factor:

:

:

/j ij

T

i

T

T

l A A

u A

L L l

UU

u

A A lu

Direct method: LUSOL

• Pivot strategies

1. Preserve sparsity

a Markowitz strategies is used to select potential pivot 𝐴𝑖𝑗 . Pivot should have a low Markowitz merit function

𝑀𝑖𝑗 ≡ 𝑟𝑖 − 1 𝑐𝑗 − 1

where 𝑟𝑖 and 𝑐𝑗 are number of nonzero element in row 𝑖 and column 𝑗.

• The sparsest columns and rows are searched in turn: columns of length 1,rows of length 1, then columns of length 2, rows of length 2, and so on).

Direct method: LUSOL

2. Preserve stability.

check (i,j) given by Markowitz strategies with stability test:

Strategy Name Stability test

Threshold Partial Pivoting TPP 𝑙 ∞ ≤ 𝐿𝑡𝑜𝑙

Threshold Rook Pivoting TRP 𝑙 ∞𝑎𝑛𝑑 𝑢/𝐴𝑖𝑗 ∞≤ 𝐿𝑡𝑜𝑙

Threshold Complete Pivoting TCP 𝐴𝑚𝑎𝑥 ≤ 𝐿𝑡𝑜𝑙 ∗ 𝐴𝑖𝑗

Name Stability test

TPP 𝑙 ∞ ≤ 𝐿𝑡𝑜𝑙

TRP 𝑙 ∞𝑎𝑛𝑑 𝑢/𝐴𝑖𝑗 ∞≤ 𝐿𝑡𝑜𝑙

TCP 𝐴𝑚𝑎𝑥 ≤ 𝐿𝑡𝑜𝑙 ∗ 𝐴𝑖𝑗

Example, Ltol=3.0

Thank you