Numerical Algorithms

Numerical Algorithms

.Matrix Multiplication

.Gaussian Elimination

.Jacobi Iteration

.Gauss-Seidel Relaxation

Numerical Algorithms

Matrix addition

Numerical Algorithms

Matrix Multiplication

1

0

l

kijijij bac

Numerical Algorithms

Matrix-Vector Multiplication

Implementing Matrix Multiplication

for(i=0 ; i<n ; i++) for(j=0 ; j<n ; j++){ c[i][j] = 0; for(k=0 ; k<n ; k++) c[i][j] = c[i][j] + a[i][k] * b[k][j];}

Sequential Code O(n3)

Implementing Matrix Multiplication

Partitioning into Submatrices

for(p=0 ; p<s ; p++) for(q=0 ; q<s ; q++){ Cp,q = 0; for(r=0 ; r<m ; r++) Cp,q = Cp,q + Ap,r * Br,q; }

Implementing Matrix Multiplication

Implementing Matrix Multiplication

Implementing Matrix Multiplication

Implementing Matrix Multiplication

Analysis

communication

)()(

)()2(

22

22

datastartupdatastartupcomm

datastartupdatastartupcomm

ttntntt

broadcast

ttnnttnt

computation

nt

additionsnandtionsmultiplican

comp 2

Implementing Matrix Multiplication

O(n2) with n2 processorsO(log n) with n3 processors

Implementing Matrix Multiplication

submatricess=n/m

communication

)}()(2{ 22datastartupdatastartupcomm tmtmttst

computation

)()()2( 2323 nmOsmOmmstcomp

Recursive Implementation

mat_mult(App, Bpp, s) { if( s==1) C=A*B; else{ s = s/2; P0 = mat_mult(App, Bpp, s); P1 = mat_mult(Apq, Bqp, s); P2 = mat_mult(App, Bpq, s); P3 = mat_mult(Apq, Bqq, s); P4 = mat_mult(Aqp, Bpp, s); P5 = mat_mult(Aqq, Bqp, s); P6 = mat_mult(Aqp, Bpq, s); P7 = mat_mult(Aqq, Bqq, s); Cpp = P0 + P1; Cpq = P2 + P3; Cqp = P4 + P5; Cqq = P6 + P7; } return(C);}

Mesh Implementation

Connon's Algorithm

1. initially processor Pij has element Aij and Bij2. Elements are moved from their initial position to an "aligned" position. The complete ith row of A is shifted i places left and the complete jth column of B is shifted j places downward. this has the effect of placing the elements aij+1 and the element bi+jj in processor Pij, as illusrated in figure 10.10. These elements are pair of those required in the accumulation of cij3. Each processor, P1j, multiplies its elements.4. The ith row of A is shifted one place right, and the jth column of B is shifted one place downward. this has the effect of bringing together the adjacent elements of A and B, which will also be required in the accumulation, as illustrated in Figure 10.11.5. Each processor, Pij, multiplies the elements brought to it and adds the result to the accumulation sum.6. Step 4 and 5 are repeated until the final result is obtained

Mesh Implementation

Mesh Implementation

Analysis

communication

))(1(2 2datastartupcomm tmtst

computation

O(sm2)

nmsmmstcomp233 22)2)(1(

Two dimensional pipeline--- Systolic array

recv(&a, Pi,j-1);recv(&b, Pi-1,j);c=c+a*b;send(&a, Pi,j+1);send(&b, Pi+1,j);

Two dimensional pipeline--- Systolic array

Solving a System of Linear Equations

11,11,022,011,000,0

11,11,122,111,100,1

11,11,122,111,100,1

...............

................

....

....

.....

nnnn

nnnn

nnnnnnnn

bxaxaxaxa

bxaxaxaxa

bxaxaxaxa

Ax=bDense matrixSparse matrix

Solving a System of Linear Equations

Gaussian Elimination

0)(

ii

jiiijiji a

aaaa

Solving a System of Linear Equations

for(i=0 ; i<n-1 ; i++) for(j=i+1 ; j<n ; j++){ m = a[j][i]/a[i][i]; for(k=i ; k<n ; k++) a[j][k] = a[j][k] - a[i][k] * m; b[j] = b[j] - b[i] * m;

O(n3)

Solving a System of Linear Equations

communication

2

0

))1((n

idatastartupcomm tintt O(n2)

Solving a System of Linear Equations

computation

)(32

)2)(3()2(2 2

1

1

nOnn

jntn

jcomp

Solving a System of Linear Equations

Pipeline configuration

Solving a System of Linear Equations

Iterative Methods

Jacobi Iteration

ij

kjiji

ii

ki xab

ax ][

1 1

Iterative Methods

022

2

22

2

y

f

x

f

)],(),(2),([1

)],(),(2),([1

222

2

222

2

yxfyxfyxfy

f

yxfyxfyxfx

f

Iterative Methods

0)],(4),(),(),(),([1

2

yxfyxfyxfyxfyxf

4

)],(),(),(),([),(

yxfyxfyxfyxfyxf

4

)],(),(),(),([),(

1111

yxfyxfyxfyxfyxf

kkkkk

Relationship with a General System of Linear Equations

411 niiini

i

xxxxx

Iterative Methods

Iterative Methods

Iterative Methods

Iterative Methods

4

)],(),(),(),([),(

11

yxfyxfyxfyxfyxf

kkkkk

Gauss-Seidel Relaxation

Iterative Methods

Iterative Methods

Red-Black Ordering

Iterative Methods

forall(i=0 ; i<n ; i++) forall(j=1 ; j<n ; j++) if((i+j)%2 == 0) f[i][j] = 0.25*(f[i-1][j]+f[i][j-1]+f[i+1][j]+f[i][j+1]);

forall(i=1 ; i<n ; i++) forall(j=1 ; j<n ; j++) if((i+j)%2 !=0 ) f[i][j] = 0.25*(f[i-1][j]+f[i][j-1]+f[i+1][j]+f[i][j+1]);

Iterative Methods

High-Order Difference Methods

)]2,(),2()2,(),2(

),(16),(16),(16),(16[60

1

1111

1111

yxfyxfyxfyxf

yxfyxfyxfyxf

kkkk

kkkk

Iterative Methods

Multigrid Method

10,)1(][ 1

1

1

ki

j

kiiji

ii

ki xxab

ax