Parallel Algorithms & Distributed Computing

Parallel Algorithms & Distributed Computing

Matt StimmelMatt White

The Goal: 1 Gigapixel

How?

How?

The Plan

Imhotep, How are we going to make all these bricks?

The Plan

We’ll use the Hebrew Slaves!

The Process

Hebrew Slave

+

Benford 6100 Brick Mold

=

Brick

Re-evaluation

Yeah, this is pretty slow.

Re-evaluation

We’ll use MORE Hebrew Slaves!

The Revised Process

+ =

+ =

+ =

+ =

+ =

+ =

Lots of Hebrew Slaves

Lots of Benford 6100 Brick Molds

Enough bricks for a pyramid!

Thus Parallel Algorithms were born

In computers

Job

Result

Process

Faster!

JobSub-task

Sub-task

Sub-task

Sub-task

Sub-result

Sub-result

Sub-result

Sub-result

Result

Process

Process

Process

Process

What happens here?

JobSub-task

Sub-task

Sub-task

Sub-task

Sub-result

Sub-result

Sub-result

Sub-result

Result

Process

Process

Process

Process

Ah Ha!

JobSub-task

Sub-task

Sub-task

Sub-task

Sub-result

Sub-result

Sub-result

Sub-result

Result

SplitCombine

Process

Process

Process

Process

Sequential!

JobSub-task

Sub-task

Sub-task

Sub-task

Sub-result

Sub-result

Sub-result

Sub-result

Result

SplitCombine

Process

Process

Process

Process

So, Really, we have…

Job

Result

SplitCombine

Sub-task

Sub-task

Sub-task

Sub-task

Sub-result

Sub-result

Sub-result

Sub-result

Process

Process

Process

Process

Sequential Parallel Sequential

Amdahl's Law

Speedup = 1

SequentialParallel(1 – Parallel) +

Amdahl's Law

Speedup = 1

SequentialParallel(1 – Parallel) +

Or, as a 5th grader would say, “How many times faster it goes”

Amdahl's Lawhttp://en.wikipedia.org/wiki/File:AmdahlsLaw.svg

Why Distributed Computing?

Process

Process

Process

Process

=

=

=

=

-

The Plan

Imhotep, we can’t afford all these Hebrew Slaves!

The Plan

Well the surrounding

countries have slaves they’re not

using…

The Revised Process: Cheap

+ =

+ =

+ =

+ =

+ =

Slaves from Egypt

Egypt’s Benford 6100 Brick Molds

Enough bricks for a pyramid!

Slaves from Mars Space Brickmakers

Slaves from ChickenLand

Avian Brickmakers

Robot Slaves Brick Factory

Distributed Computing

JobSub-task

Sub-task

Sub-task

Sub-task

Sub-result

Sub-result

Sub-result

Sub-result

Result

SplitCombine

Process

Process

Process

Process

Internet

Internet

Internet

Internet

Internet

Internet

Internet

Internet

Why Distributed Computing?

Process

Process

Process

Process

=

=

=

=

Internet

Internet

Internet

Internet

-

Not My

Design

The Big Picture

Client

FractalGenImageCombiner

Storage

Control

Data

Server

ServerFunctions:•Manages Connections to Clients•Allows the user to input task parameters. •Divides job into sub-tasks•Displays information about currently running job

Challenges:•Job division• Must divide into a perfect square number of Jobs

•Resolution• Problems occur when the resolution isn’t an integer multiple of the number of

jobs.•Networking• Treated client connections as a Finite State Machine for file transfers. Didn’t work.

• About 1 in 100 transfers failed.

Server

Client

ClientFunctions:•Manages Connection to server•Receives server-generated command line for job•Executes FractalGen on cue•Informs server upon completion

Challenges:•Maintaining a graceful disconnect•Aforementioned file transfer

FractalGen

FractalGenFunctions:•Render the fractal• Specified by the command line

•Save the fractal to the disk

Challenges:•How do you draw a Mandelbrot fractal anyway?• Z = Z2 + C

•Command Line Parsing•Generating large images:• Limit is approx. 8000x8000 due to graphics hardware

•Shader Implementation• Downloaded, then made attempts at optimization.• Iteration count

• Shader Model 3 gives a greater instruction cound allowing form more iterations.• Shader Model 2 has greater compatibility, but cannot render as many iterations.

• Client computer must have a GPU capable of running at least DirectX 9 or else FractalGen will not work.

Image Combiner

ImageCombinerFunctions:•Merge images back together

Challenges:•Finding .bmp files•Memory Allocation• Estimates demanded efficient heap usage• A lot of pointers, a lot of pointer math

•Working with Bitmaps• Multiple color modes – how many bits is a pixel?• “Upside Down” errors

•Running over a network

Client

FractalGen ImageCombiner

•C#•.NET•Multithreaded

•C++• OOP•Memory Allocation

•C#•.NET•XNA•HLSL•Shader Model 3.0

•C#•.NET•Multithreaded

Server

Overview of Technologies Used

How is the task divided into separate jobs?

Breaking up Jobs: Method A

Breaking up Jobs: Method B

Overlays

Method A: Since B didn’t work

We now send a square to each client

Client 1 Client 2

Client 3 Client 4

The clients return separate images of the fractal.

Run the image combiner.

Final Output

The Big Picture

Client

FractalGenImageCombiner

Storage

Control

Data

Server

Science

Science, Step 1

Using a parallel implementation and multiple processors, we will be able to increase performance over the same implementation on a single processor.

Hypothesis:

Science, Step 2

Run the system, splitting job into varying numbers of sub-tasks and varying the number of processors available.

Experiment:

Science, Step 3

Run the system using one client.Control:

Playwrighting, Step 1

Major Dramatic Question:

IsSequential

Parallel≥ 1?

Data

8000x 8000Number of

ClientsNumber of

Jobs Time 1 Time 2 Time 3 Time 4 Avg. Time Speedup "Percent Faster"

Serial 0 0 25 23 24 23 23.75 3.4245614 242.46%

1 Job, 1Client 1 1 82 81 81 81.333333 1 0.00%

4 Jobs, 1 Client 1 4 90 89 94 91 0.8937729 -10.62%

16 Jobs, 1 Client 1 16 105 106 105 103 104.75 0.7764519 -22.35%

100 Jobs, 1 Client 1 100 182 187 187 185.33333 0.4388489 -56.12%

1000 Jobs, 1 Client 1 1000 -100.00%

4 Jobs, 4 Clients 4 4 58 61 53 57.333333 1.4186047 41.86%

16 Jobs, 4 Clients 4 16 60 56 56 54 56.5 1.439528 43.95%

100 Jobs, 4 Clients 4 100 67 68 66 67 1.2139303 21.39%

1000 Jobs, 4 Clients 4 1000 306 306 0.2657952 -73.42%

Data

0 1 4 16 100 10000

50

100

150

200

250

300

350

Average Time by Number of Jobs

014

Number of Jobs

Time

Data

4 16 100 10000

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

Number of Jobs

Speedup

Useful

JobSub-task

Sub-task

Sub-task

Sub-task

Sub-result

Sub-result

Sub-result

Sub-result

Result

SplitCombine

Process

Process

Process

Process