Iterative Methods

Berlin ChenDepartment of Computer Science & Information Engineering

National Taiwan Normal University

Reference:1. Applied Numerical Methods with MATLAB for Engineers, Chapter 12 & Teaching material

Chapter Objectives

• Understanding the difference between the Gauss-Seidel and Jacobi methods

• Knowing how to assess diagonal dominance and knowing what it means

• Recognizing how relaxation can be used to improve convergence of iterative methods

• Understanding how to solve systems of nonlinear equations with successive substitution and Newton-Raphson

NM – Berlin Chen 2

[A] {x} = {b}

Gauss-Seidel Method• The Gauss-Seidel method is the most commonly used

iterative method for solving linear algebraic equations [A]{x}={b}

• The method solves each equation in a system for a particular variable, and then uses that value in later equations to solve later variables

• For a 3x3 system with nonzero elements along the diagonal, for example, the jth iteration values are found from the j-1th iteration using:

NM – Berlin Chen 3

x1j b1 a12x2

j1 a13x3j1

a11

x2j b2 a21x1

j a23x3j1

a22

x3j b3 a31x1

j a32x2j

a33

3333232131

2323222121

1313212111

bxaxaxabxaxaxabxaxaxa

Gauss-Seidel Method: Convergence

• The convergence of an iterative method can be calculated by determining the relative percent change of each element in {x}. For example, for the i th element in the j th iteration,

• The method is ended when all elements have converged to a set tolerance

NM – Berlin Chen 4

a,i xij xi

j1

xij 100%

Gauss-Seidel Method: An Example (1/2)

NM – Berlin Chen 5

Example 12.1

Gauss-Seidel Method: An Example (2/2)

NM – Berlin Chen 6

Example 12.1

Tactic of Gauss-Seidel Method

• As each new x value is computed for the Gauss-Seidel method, it is immediately used in the next equation to determine another x value

• Thus, if the solution is converging, the best available estimates will be employed for the Gauss-Seidel method

NM – Berlin Chen 7

Jacobi Iteration

• The Jacobi iteration is similar to the Gauss-Seidel method, except the j-1th information is used to update all variables in the j th iteration:– Gauss-Seidel– Jacobi

NM – Berlin Chen 8

Diagonal Dominance

• The Gauss-Seidel method may diverge, but if the system is diagonally dominant, it will definitely converge

• Diagonal dominance means:

NM – Berlin Chen 9

aii aijj1ji

n

MATLAB Program

NM – Berlin Chen 10

Relaxation

• To enhance convergence, an iterative program can introduce relaxation where the value at a particular iteration is made up of a combination of the old value and the newly calculated value:

– where is a weighting factor that is assigned a value between 0 and 2

• 0<<1: underrelaxation• =1: no relaxation• 1<≤2: overrelaxation

– The choice of a proper value for is highly problem-specific and is often determined empirically

NM – Berlin Chen 11

xinew xi

new 1 xiold

Gauss-Seidel with Relaxation: An Example (1/3)

NM – Berlin Chen 12

Example 12.2

Gauss-Seidel with Relaxation: An Example (2/3)

NM – Berlin Chen 13

Example 12.2

Gauss-Seidel with Relaxation: An Example (3/3)

NM – Berlin Chen 14

Example 12.2

Nonlinear Systems

• Nonlinear systems can also be solved using the same strategy as the Gauss-Seidel method – Solve each system for one of the unknowns and update each

unknown using information from the previous iteration

• This is called successive substitution

NM – Berlin Chen 15

573

102212

2121

xxx

xxx

Successive Substitution: An Example (1/2)

NM – Berlin Chen 16

Example 12.3

573

102212

2121

xxx

xxx

Successive Substitution: An Example (2/2)

NM – Berlin Chen 17

Example 12.3

1. For successive substitution, convergence often depends on the manner in whichthe equations are formulated

2. Divergence also can occur if the initial guesses are insufficiently close to the truesolution

Newton-Raphson (1/3)

• Nonlinear systems may also be solved using the Newton-Raphson method for multiple variables

• For a two-variable system, the Taylor series approximation and resulting Newton-Raphson equations are:

NM – Berlin Chen 18

2

,2,21,2

1

,2,11,1,21,2

2

,1,21,2

1

,1,11,1,11,1

xf

xxxf

xxff

xf

xxxf

xxff

iii

iiiii

iii

iiiii

0

0

1,2

1,1

i

i

f

f

ii

ii

iii

ii

ii

ii

iii

ii

xxf

xxf

fxxf

xxf

xxf

xxf

fxxf

xxf

,22

,2,1

1

,1,11,2

2

,11,1

1

,1

,22

,1,1

1

,1,11,2

2

,11,1

1

,1

ii

ii

iiiii

xfxfxx

fxfxxxfxf

1

11

) of expansion seriesTaylor order -(first0

:equation singlea ofroot thefor Raphon-Newton 6,Chapter in Recall

Newton-Raphson (2/3)

– After algebraic manipulation

NM – Berlin Chen 19

1

,2

2

,1

2

,2

1

,1

1

,2,1

1

,1,2

,21,2

1

,2

2

,1

2

,2

1

,1

2

,1,2

2

,2,1

,11,1

xf

xf

xf

xf

xf

fxf

fxx

xf

xf

xf

xf

xf

fxf

fxx

iiii

ii

ii

ii

iiii

ii

ii

ii

Referred to as the determinant of the Jacobian of the system

2

,2

1

,2

2

,1

1

,1

xf

xf

xf

xf

Jii

ii

Referred to as the Jacobian of the system

Newton-Raphson (3/3)

• Matrix notion of Newton-Raphson– Also can be generalized to n simultaneous equations

NM – Berlin Chen 20

fJxx

xJfxJ

ii

ii

11

1

Newton-Raphson: An Example

NM – Berlin Chen 21

0573,

010,

573

10

21

2212212

2121211

21

2212

2121

xxxxxxf

xxxxxf

xxxx

xxx

Example 12.4

MATLAB Program

NM – Berlin Chen 22