Parallel Algorithms on Networks of Processors

Parallel Algorithms on Networks of Processors

Roy (Hutch) Pargas, PhD Computer Science (UNC Chapel Hill)School of Computing, Clemson University

pargas@clemson.edu

2

OutlineWhat are parallel algorithms? Why use them?

Challenges for parallel algorithm designers

Choosing a network

Partitioning the data

Designing the algorithm

Example

Recurrences (binary tree)

Analysis (Speedup and Efficiency)

Summary and Conclusions

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

3

Why Parallel Computation?

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

4

Why Parallel Computation?

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

5

Why Parallel Computation?

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

6

Why Parallel Computation?

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

7

Why Parallel Computation?

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

8

Why Parallel Computation?

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

9

New ProcessorsFaster and Cheaper

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

January2011

10

Partition the Data

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

11

Organize the Processors

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

12

Build a Network

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

13

Build a Network

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

14

Choosing a Network

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

15

Choosing a Network

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

16

Choosing a Network

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

17

Choosing a Network

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

18

Choosing a Network

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

19

Are There Really Any Multiprocessing

Systems in Use Today?

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

20

Are There Really Any Multiprocessing

Systems in Use Today?

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

HamburgJune 2011Top 500

21

SupercomputersNEC/HP Tsubame

(Japan)

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

1.192 petaflops ≈ 1.28 quadrillion floating point

operations per sec

73,278 Xeon cores

Infiniband grid network

22

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

SupercomputersDawning Nebulae

(China)1.27 petaflops ≈ 1.36

quadrillion floating point operations per sec

9280 Intel 6-core Xeon processors = 55,680 cores

Infiniband grid network

23

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

SupercomputersCray Jaguar (USA)

1.75 petaflops ≈ 1.876 quadrillion floating point

operations per sec

224,256 AMD cores

3D torus network

24

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

SupercomputersNUDT Tianhe-1A

(China)2.566 petaflops ≈ 2.75

quadrillion floating point operations per sec

14336 CPUs

Undisclosed proprietary network

25

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

SupercomputersFujitsu “K” (Japan)

K = “kei” = Japanese for 10 quadrillion

8.162 petaflops ≈ 9 quadrillion floating point operations per

sec

68,544 8-core SPARC64 processors = 548,352

cores

3-dimensional torus network called Tofu

26

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

TOP500

Top 500 Computers in the World

27

Where Does that Leave Us?

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

28

Where Does that Leave Us?

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

In a wonderful playground of mathematical algorithmic design where imagination and creativity are key!

29

Where Does that Leave Us?

In a wonderful playground of mathematical algorithmic design where imagination and creativity are key!

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

30

Where Does that Leave Us?

In a wonderful playground of mathematical algorithmic design where imagination and creativity are key!

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

31

Where Does that Leave Us?

In a wonderful playground of mathematical algorithmic design where imagination and creativity are key!

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

32

Challenges for Parallel Algorithm

DesignersChoosing a network

Partitioning the problem

Designing the parallel algorithm

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

33

So Let’s Try It:

Choosing a network

Partitioning the problem

Designing the parallel algorithm

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

34

So Let’s Try It:Elliptic Partial Diff

EqnsChoosing a network

Partitioning the problem

Designing the parallel algorithm

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

35

Elliptic PDEs

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Problems involving second-order elliptic partial differential equations are equilibrium problems. Given a region R bounded by a curve C and that the unknown function z satisfies Laplace’s or Poisson’s equation in R, the objective is to approximate the value of z at any point in R. The method of finite differences is an often used numerical method for solving this problem. The basic strategy is to approximate the differential equation by a difference equation and to solve the difference equation.

36

Designing the Algorithm

Why Solve Linear Recurrences?

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

37

Designing the Algorithm

Why Solve Linear Recurrences?

The problem: Solving PDEs using the Method of Finite Differences leads to

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

38

Designing the Algorithm

Why Solve Linear Recurrences?

The problem: Solving PDEs using the Method of Finite Differences leads to

Solving Block Tridiagonal Systems which leads to

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

39

Designing the Algorithm

Why Solve Linear Recurrences?

The problem: Solving PDEs using the Method of Finite Differences leads to

Solving Block Tridiagonal Systems which leads to

Solving Tridiagonal Systems which leads to

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

40

Designing the Algorithm

Why Solve Linear Recurrences?

The problem: Solving PDEs using the Method of Finite Differences leads to

Solving Block Tridiagonal Systems which leads to

Solving Tridiagonal Systems which leads to

Solving Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

41

Designing the Algorithm

Why Solve Linear Recurrences?

The problem: Solving PDEs using the Method of Finite Differences leads to

Solving Block Tridiagonal Systems which leads to

Solving Tridiagonal Systems which leads to

Solving Linear Recurrences many many many times

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

42

Designing the Algorithm

Why Solve Linear Recurrences?

The problem: Solving PDEs using the Method of Finite Differences leads to

Solving Block Tridiagonal Systems which leads to

Solving Tridiagonal Systems which leads to

Solving Linear Recurrences many many many times

Why Use a Binary Tree? Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

43

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Key Idea: Successfully solving the original pde problem depends upon solving recurrences quickly and efficiently.

44

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Consider the following set of n equations:x0 = a0

x1 = a1 + b1 x0

x2 = a2 + b2 x1

...

xn-1 = an-1 + bn-1 xn-2

45

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Consider the following set of n equations:x0 = a0

x1 = a1 + b1 x0

x2 = a2 + b2 x1

...

xn-1 = an-1 + bn-1 xn-2

Can we solve for xi in parallel?

46

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

For uniformity:x0 = a0 + b0 x-1 b0=0, x-1=dummy variablex1 = a1 + b1 x0

x2 = a2 + b2 x1

...

xn-1 = an-1 + bn-1 xn-2

47

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

For uniformity:x0 = a0 + b0 x-1

x1 = a1 + b1 x0

x2 = a2 + b2 x1

...

xn-1 = an-1 + bn-1 xn-2

Observe, ifxi = a + b xj

xj = a’ + b’ xk

Thenxi = (a + ba’) +bb’ xk

= a” + b” xk

48

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Notation changex0 = a0 + b0 x-1 C0,-1 = (a0,b0)x1 = a1 + b1 x0 C1,0 = (a1,b1)x2 = a2 + b2 x1 C2,1 = (a2,b2)...

xn-1 = an-1 + bn-1 xn-2 Cn-1,n-2 = (an-1,bn-1)

Observe, ifxi = a + b xj

xj = a’ + b’ xk

Thenxi = (a + ba’) +bb’ xk

= a” + b” xk

49

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Notation changex0 = a0 + b0 x-1 C0,-1 = (a0,b0)x1 = a1 + b1 x0 C1,0 = (a1,b1)x2 = a2 + b2 x1 C2,1 = (a2,b2)...

xn-1 = an-1 + bn-1 xn-2 Cn-1,n-2 = (an-1,bn-1)

Observe, ifxi = a + b xj Ci,j = (a,b)xj = a’ + b’ xk Cj,k = (a’,b’)

Thenxi = (a + ba’) +bb’ xk Ci,j Cj,k =

= a” + b” xk Ci,k = (a+ba’,bb’)

50

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

To summarize:xi = a + b xj Ci,j = (a,b)xj = a’ + b’ xk Cj,k = (a’,b’)

Thenxi = (a + ba’) +bb’ xk Ci,j Cj,k =

= a” + b” xk Ci,k = (a+ba’,bb’)

51

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

To summarize:xi = a + b xj Ci,j = (a,b)xj = a’ + b’ xk Cj,k = (a’,b’)

Thenxi = (a + ba’) +bb’ xk Ci,j Cj,k =

= a” + b” xk Ci,k = (a+ba’,bb’)

One last observation:

52

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

To summarize:xi = a + b xj Ci,j = (a,b)xj = a’ + b’ xk Cj,k = (a’,b’)

Thenxi = (a + ba’) +bb’ xk Ci,j Cj,k =

= a” + b” xk Ci,k = (a+ba’,bb’)

One last observation:If any variable is expressed in terms of

the dummy variable x-1 (e.g., x0 = a0 + b0 x-1) then that variable is solved.

53

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

To summarize:xi = a + b xj Ci,j = (a,b)xj = a’ + b’ xk Cj,k = (a’,b’)

Thenxi = (a + ba’) +bb’ xk Ci,j Cj,k =

= a” + b” xk Ci,k = (a+ba’,bb’)

One last observation:If any variable is expressed in terms of

the dummy variable x-1 (e.g., x0 = a0 + b0 x-1) then that variable is solved. So Ci,-1 = (a,b)

54

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

To summarize:xi = a + b xj Ci,j = (a,b)xj = a’ + b’ xk Cj,k = (a’,b’)

Thenxi = (a + ba’) +bb’ xk Ci,j Cj,k =

= a” + b” xk Ci,k = (a+ba’,bb’)

One last observation:If any variable is expressed in terms of

the dummy variable x-1 (e.g., x0 = a0 + b0 x-1) then that variable is solved. So Ci,-1 = (a,b) means that b=0

55

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

To summarize:xi = a + b xj Ci,j = (a,b)xj = a’ + b’ xk Cj,k = (a’,b’)

Thenxi = (a + ba’) +bb’ xk Ci,j Cj,k =

= a” + b” xk Ci,k = (a+ba’,bb’)

One last observation:If any variable is expressed in terms of

the dummy variable x-1 (e.g., x0 = a0 + b0 x-1) then that variable is solved. So Ci,-1 = (a,b) means that b=0 and that xi = a + b x-1 = a + 0 x-1 = a

56

C0,-

1

C1,

0

C2,

1

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6 Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Linear Recurrences

57

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

C0,-1C1,0C2,1C3,2C4,3

C5,4C6,5C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

58

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

C0,-1C1,0C2,1C3,2C4,3

C5,4C6,5C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

59

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

C0,-1C1,0C2,1C3,2C4,3

C5,4C6,5C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C7,5 C5,3C3,1 C1,-1

60

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

C0,-1C1,0C2,1C3,2C4,3

C5,4C6,5C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C7,5 C5,3C3,1 C1,-1

C7,

3

C3,-

1

61

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

C0,-1C1,0C2,1C3,2C4,3

C5,4C6,5C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C7,5 C5,3C3,1 C1,-1

C7,

3

C3,-

1

C7,3 C3,-1

62

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

C0,-1C1,0C2,1C3,2C4,3

C5,4C6,5C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C7,5 C5,3C3,1 C1,-1

C7,

3

C3,-

1

C7,3 C3,-1

C7,-

1

63

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

C0,-1C1,0C2,1C3,2C4,3

C5,4C6,5C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C7,5 C5,3C3,1 C1,-1

C7,

3

C3,-

1

C7,3 C3,-1

C7,-

1

C7,-

1C7,-1

64

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

C0,-1C1,0C2,1C3,2C4,3

C5,4C6,5C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C7,5 C5,3C3,1 C1,-1

C7,

3

C3,-

1

C7,3 C3,-1

C7,-

1

C7,-

1C7,-1

Solved variable

65

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

C0,-1C1,0C2,1C3,2C4,3

C5,4C6,5C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C7,5 C5,3C3,1 C1,-1

C7,

3

C3,-

1

C7,3 C3,-1

C7,-

1

C7,-

1C7,-1

Solved variables

66

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

C0,-1C1,0C2,1C3,2C4,3

C5,4C6,5C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C7,5 C5,3C3,1 C1,-1

C7,

3

C3,-

1

C7,3 C3,-1

C7,-

1

C7,-

1C7,-1

How do we solve for the other variables?

67

C6,5

C7,-1

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

C0,-1C1,0C2,1C3,2C4,3

C5,4C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C7,5 C5,3C3,1 C1,-1

C7,

3

C3,-

1

C7,3 C3,-1

C7,-

1

C7,-

1

In the downsweep!

68

C6,5

C7,-1

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

C0,-1C1,0C2,1C3,2C4,3

C5,4C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C7,5 C5,3C3,1 C1,-1

C7,

3

C3,-

1

C7,3 C3,-1

C7,-

1

(x7,x-

1)

69

C6,5

(x7,x-1)

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

C0,-1C1,0C2,1C3,2C4,3

C5,4C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C7,5 C5,3C3,1 C1,-1

C7,

3

C3,-

1

C7,3 C3,-1

C7,-

1

(x7,x-

1)

70

C6,5

(x7,x-1)

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

C0,-1C1,0C2,1C3,2C4,3

C5,4C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C7,5 C5,3C3,1 C1,-1

C7,

3

C3,-

1

C7,3 C3,-1

x3

(x7,x-

1)

71

C6,5

(x7,x-1)

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

C0,-1C1,0C2,1C3,2C4,3

C5,4C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C7,5 C5,3C3,1 C1,-1

C7,

3

C3,-

1

x3

(x7,x-

1)

(x7,x3) (x3,x-1)

72

C6,5

(x7,x-1)

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

C0,-1C1,0C2,1C3,2C4,3

C5,4C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C3,1 C1,-1

x3

(x7,x-

1)

(x7,x3) (x3,x-1)

x5C7,5 C5,3

x1

73

C6,5

(x7,x-1)

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

C0,-1C1,0C2,1C3,2C4,3

C5,4C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

x3

(x7,x-

1)

(x7,x3) (x3,x-1)

x5 x1

(x7,x5) (x5,x3) (x3,x1) (x1,x-1)

74

(x7,x5)

C6,5

(x7,x-1)

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

C0,-1C1,0C2,1C3,2C4,3

C5,4C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

x3

(x7,x-

1)

(x7,x3) (x3,x-1)

x5 x1

(x5,x3) (x3,x1) (x1,x-1)

x6 x4 x3 x1

75

(x7,x5)

(x7,x-1)

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

x3

(x7,x-

1)

(x7,x3) (x3,x-1)

x5 x1

(x5,x3) (x3,x1) (x1,x-1)

x6 x4 x3 x1

(x7,x6) (x6,x5) (x5,x4)(x0,x-1)(x1,x0)(x4,x3) (x3,x2) (x2,x1)

76

(x7,x5)

(x7,x-1)

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

x3

(x7,x-

1)

(x7,x3) (x3,x-1)

x5 x1

(x5,x3) (x3,x1) (x1,x-1)

x6 x4 x3 x1

(x7,x6) (x6,x5) (x5,x4)(x0,x-1)(x1,x0)(x4,x3) (x3,x2) (x2,x1)

(x7,x6) (x0,x-

1)(x6,x5) (x5,x4) (x4,x3) (x3,x2) (x2,x1) (x1,x0)

77

(x7,x5)

(x7,x-1)

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

x3

(x7,x-

1)

(x7,x3) (x3,x-1)

x5 x1

(x5,x3) (x3,x1) (x1,x-1)

x6 x4 x3 x1

(x7,x6) (x6,x5) (x5,x4)(x0,x-1)(x1,x0)(x4,x3) (x3,x2) (x2,x1)

(x7,x6) (x0,x-

1)(x6,x5) (x5,x4) (x4,x3) (x3,x2) (x2,x1) (x1,x0)

Leaves contain solutions!

78

(x7,x5)

(x7,x-1)

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

x3

(x7,x-

1)

(x7,x3) (x3,x-1)

x5 x1

(x5,x3) (x3,x1) (x1,x-1)

x6 x4 x3 x1

(x7,x6) (x6,x5) (x5,x4)(x0,x-1)(x1,x0)(x4,x3) (x3,x2) (x2,x1)

(x7,x6) (x0,x-

1)(x6,x5) (x5,x4) (x4,x3) (x3,x2) (x2,x1) (x1,x0)

But we can do better!

79

(x7,x5)

(x7,x-1)

Linear Recurrences

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

x3

(x7,x-

1)

(x7,x3) (x3,x-1)

x5 x1

(x5,x3) (x3,x1) (x1,x-1)

x6 x4 x3 x1

(x7,x6) (x6,x5) (x5,x4)(x0,x-1)(x1,x0)(x4,x3) (x3,x2) (x2,x1)

(x7,x6) (x0,x-

1)(x6,x5) (x5,x4) (x4,x3) (x3,x2) (x2,x1) (x1,x0)

Pipelining the solution

80

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Linear RecurrencesPipelining the solution

81

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Linear RecurrencesPipelining the solution

82

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Linear RecurrencesPipelining the solution

83

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Linear RecurrencesPipelining the solution

84

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Linear RecurrencesPipelining the solution

85

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Linear RecurrencesPipelining the solution

86

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Linear RecurrencesPipelining the solution

87

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Linear RecurrencesPipelining the solution

88

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Linear RecurrencesPipelining the solution

89

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Linear RecurrencesPipelining the solution

90

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Linear RecurrencesPipelining the solution

91

AnalysisT1 = Time on one processor

Tn = Time on n processors

S = Speedup = T1/Tn (ideal: S = n)

E = Efficiency = S/n (ideal: E = 1)

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

92

Speedup and Efficiency

Single PassAssume 1 floating point operation requires 1 time unit

T1 = (n−1) (1 mult + 1 add) = 2n−2 time units

n leaves 2n processors T2n = (log2n)( 2 mults + 1 add) // upsweep

+ (log2n)(1 mult + 1 add) // downsweep= 5 log2n time units

S = Speedup = T1/T2n = (2n−2)/(5 log2n)

E = Efficiency = S/2n = (2n−2)/[ (5 log2n) (2n) ]

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

93

Speedup

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

0 200 400 600 800 1000 12000

0.02

0.04

0.06

0.08

0.1

0.12

0.14

Series1

Speedup and Efficiency

Single Pass

94

Speedup Efficiency

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

0 200 400 600 800 1000 12000

0.02

0.04

0.06

0.08

0.1

0.12

0.14

Series1

0 50 100 150 200 250 300 3500

0.01

0.02

0.03

0.04

0.05

0.06

Series1

Speedup and Efficiency

Single Pass

95

With Pipelining. Assume kn equations for large k

T1 = (kn−1) (1 mult + 1 add) = 2kn−2 time units

n leaves 2n processors T2n = (log2n)( 2 mults + 1 add) // pipefill up

+ (k – 2 log 2n) (5) // pipeline on k−2log 2n waves

+ (log2n)( 1 mult + 1 add) // pipedrain

down= 5(k – log2n) time units

S = Speedup = T1/T2n = (2kn−2)/[5(k – log2n )]

E = Efficiency = S/2n = (2kn−2)/[5(k – log2n ) (2n)]

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Speedup and Efficiency

With Pipelining

96

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Speedup and Efficiency

With PipeliningSpeedup

0 200 400 600 800 1000 12000

50

100

150

200

250

300

350

400

450

Series1

97

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Speedup and Efficiency

With PipeliningSpeedup Efficiency

0 200 400 600 800 1000 12000

50

100

150

200

250

300

350

400

450

Series1

0 200 400 600 800 1000 12000.1999

0.2000

0.2001

0.2002

0.2003

Series1

98

Technique Can Work For

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Second-order linear recurrencesx0 = a0

x1 = a1 + b1 x0

x2 = a2 + b2 x1 + c2 x0

...

xn-1 = an-1 + bn-1 xn-2 + cn-1 xn-3

Higher order linear recurrences

99

Technique Can Work For

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Quotients of linear recurrencesx0 = a0

xi = (ai + bi xi-1)/(ci + di xi-1) i=1,2,…, n-1

Other recurrences

100

Summary and Conclusions

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

101

Summary and Conclusions

Chip technology is going to get even better/faster/cheaper for the foreseeable future.

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

102

Summary and Conclusions

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Chip technology is going to get even better/faster/cheaper for the foreseeable future. (Ignore the naysaying pundits!)

103

Summary and Conclusions

Chip technology is going to get even better/faster/cheaper for the foreseeable future. (Ignore the naysaying pundits!)

More massively parallel processing systems are going to be built and will become even better/faster/cheaper.

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

104

Summary and Conclusions

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Chip technology is going to get even better/faster/cheaper for the foreseeable future. (Ignore the naysaying pundits!)

More massively parallel processing systems are going to be built and will become even better/faster/cheaper.

The challenging world of parallel algorithmic design beckons and awaits creative minds.

105

Summary and Conclusions

Chip technology is going to get even better/faster/cheaper for the foreseeable future. (Ignore the naysaying pundits!)

More massively parallel processing systems are going to be built and will become even better/faster/cheaper.

The challenging world of parallel algorithmic design beckons and awaits creative minds. Yes, this means you!

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

106

Where in the World is Clemson

University?

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

We are here!

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

107

Thank you for your kind attention!

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Questions?

108

Extra Slides

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

109

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Links

1. Fujitsu K Computer (K = “kei” = Japanese word for 10 quadrillion)

http://www.fujitsu.com/global/about/tech/k/http://en.wikipedia.org/wiki/K_computer

2. NUDT “Tianhe-1A” Computerhttp://blog.zorinaq.com/?e=36http://en.wikipedia.org/wiki/Tianhe-I

3. Cray Jaguarhttp://en.wikipedia.org/wiki/Jaguar_(computer)http://www.nccs.gov/jaguar/

4. Dawning Nebulaehttp://en.wikipedia.org/wiki/Dawning_Information_Industryhttp://www.theregister.co.uk/2010/05/31/top_500_supers_jun2010/

5. NEC/HP Tsubame 2.0 http://en.wikipedia.org/wiki/TOP500

110

Hypercube

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

0-Degree

111

Hypercube

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

0-Degree

112

Hypercube

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

0-Degree

113

Hypercube

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

0-Degree

114

Hypercube

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

0-Degree

1-Degree

0 1

115

Hypercube

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

0-Degree

1-Degree

0 1

0 1

116

Hypercube

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

0-Degree

1-Degree

0 1

0 1

0 1

117

Hypercube

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

0-Degree

1-Degree

0 1

0 1

0 1

118

Hypercube

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

0-Degree

1-Degree

0 1

00

01

10

11

2-Degree

119

Hypercube

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

3-Degree

0-Degree

1-Degree

0 1

00

01

10

11

2-Degree

120

Hypercube

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

3-Degree

4-Degree

0-Degree

1-Degree

0 1

00

01

10

11

2-Degree

121

Hypercube

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

3-Degree

4-Degree

5-Degree

122

All-to-All Communication

(Hypercube)

Roy Pargas, Clemson University pargas@clemson.edu

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

Problem

Motivation