8/11/2019 02 1 Lecture

http://slidepdf.com/reader/full/02-1-lecture 1/24

Advanced Topics in Algorithms and Data Structures

 Anoverviewofthelecture2

• Modelsofparallelcomputation

• CharacteristicsofSIMDmodels

• DesignissuefornetworkSIMDmodels

• Themeshandthehypercube

architectures

• ClassificationofthePRAMmodel

• MatrixmultiplicationontheEREWPRAM

8/11/2019 02 1 Lecture

http://slidepdf.com/reader/full/02-1-lecture 2/24

Advanced Topics in Algorithms and Data Structures

Modelsofparallelcomputation

Parallelcomputationalmodelscanbe

broadlyclassifiedintotwocategories,

• SingleInstructionMultipleData(SIMD)

•MultipleInstructionMultipleData(MIMD)

8/11/2019 02 1 Lecture

http://slidepdf.com/reader/full/02-1-lecture 3/24

Advanced Topics in Algorithms and Data Structures

Modelsofparallelcomputation

• SIMDmodelsareusedforsolving

problemswhichhaveregularstructures.

WewillmainlystudySIMDmodelsinthiscourse.

•MIMDmodelsaremoregeneralandused

forsolvingproblemswhichlackregularstructures.

8/11/2019 02 1 Lecture

http://slidepdf.com/reader/full/02-1-lecture 4/24

Advanced Topics in Algorithms and Data Structures

SIMDmodels

 AnN  -processorSIMDcomputerhasthe

followingcharacteristics:

• Eachprocessorcanstorebothprogramanddatainitslocalmemory.

• Eachprocessorstoresanidenticalcopy

ofthesameprograminitslocalmemory.

8/11/2019 02 1 Lecture

http://slidepdf.com/reader/full/02-1-lecture 5/24

Advanced Topics in Algorithms and Data Structures

SIMDmodels

• Ateachclockcycle,eachprocessor

executesthesameinstructionfromthis

program.However,thedataaredifferentindifferentprocessors.

• Theprocessorscommunicateamong

themselveseitherthroughaninterconnectionnetworkorthrougha

sharedmemory.

8/11/2019 02 1 Lecture

http://slidepdf.com/reader/full/02-1-lecture 6/24

Advanced Topics in Algorithms and Data Structures

Designissuesfornetwork

SIMDmodels• AnetworkSIMDmodelisagraph.The

nodesofthegrapharetheprocessors

andtheedgesarethelinksbetweentheprocessors.

• Sinceeachprocessorsolvesonlyasmall

partoftheoverallproblem,itisnecessarythatprocessorscommunicatewitheach

otherwhilesolvingtheoverallproblem.

8/11/2019 02 1 Lecture

http://slidepdf.com/reader/full/02-1-lecture 7/24Advanced Topics in Algorithms and Data Structures

Designissuesfornetwork

SIMDmodels• ThemaindesignissuesfornetworkSIMD

modelsarecommunicationdiameter,

bisectionwidth,andscalability.•Wewilldiscusstwomostpopularnetwork

models,meshandhypercubeinthis

lecture.

8/11/2019 02 1 Lecture

http://slidepdf.com/reader/full/02-1-lecture 8/24Advanced Topics in Algorithms and Data Structures

Communicationdiameter

•Communicationdiameteristhediameter

ofthegraphthatrepresentsthenetwork

model.Thediameterofagraphisthelongestdistancebetweenapairofnodes.

• Ifthediameterforamodelisd ,thelower

boundforanycomputationonthatmodelisΩ(d ).

8/11/2019 02 1 Lecture

http://slidepdf.com/reader/full/02-1-lecture 9/24Advanced Topics in Algorithms and Data Structures

Communicationdiameter

• Thedatacanbedistributedinsuchaway

thatthetwofurthestnodesmayneedto

communicate.

8/11/2019 02 1 Lecture

http://slidepdf.com/reader/full/02-1-lecture 10/24Advanced Topics in Algorithms and Data Structures

Communicationdiameter

CommunicationbetweentwofurthestnodestakesΩ(d )timesteps.

8/11/2019 02 1 Lecture

http://slidepdf.com/reader/full/02-1-lecture 11/24Advanced Topics in Algorithms and Data Structures

Bisectionwidth

• Thebisectionwidthofanetworkmodelis

thenumberoflinkstoberemovedto

decomposethegraphintotwoequalparts.• Ifthebisectionwidthislarge,more

informationcanbeexchangedbetween

thetwohalvesofthegraphandhenceproblemscanbesolvedfaster.

8/11/2019 02 1 Lecture

http://slidepdf.com/reader/full/02-1-lecture 12/24Advanced Topics in Algorithms and Data Structures

Dividingthegraphintotwoparts.

Bisectionwidth

8/11/2019 02 1 Lecture

http://slidepdf.com/reader/full/02-1-lecture 13/24Advanced Topics in Algorithms and Data Structures

Scalability

• Anetworkmodelmustbescalablesothat

moreprocessorscanbeeasilyadded

whennewresourcesareavailable.• Themodelshouldberegularsothateach

processorhasasmallnumberoflinks

incidentonit.

8/11/2019 02 1 Lecture

http://slidepdf.com/reader/full/02-1-lecture 14/24Advanced Topics in Algorithms and Data Structures

Scalability

• Ifthenumberoflinksislargeforeach

processor,itisdifficulttoaddnew

processorsastoomanynewlinkshavetobeadded.

• Ifwewanttokeepthediametersmall,we

needmorelinksperprocessor.Ifwewantourmodeltobescalable,weneedless

linksperprocessor.

8/11/2019 02 1 Lecture

http://slidepdf.com/reader/full/02-1-lecture 15/24Advanced Topics in Algorithms and Data Structures

DiameterandScalability

• Thebestmodelintermsofdiameteristhe

completegraph.Thediameteris1.

However,ifweneedtoaddanewnodetoann-processormachine,weneedn-1

newlinks.

8/11/2019 02 1 Lecture

http://slidepdf.com/reader/full/02-1-lecture 16/24Advanced Topics in Algorithms and Data Structures

DiameterandScalability

• Thebestmodelintermsofscalabilityis

thelineararray.Weneedtoaddonlyone

linkforanewprocessor.However,thediameterisn foramachinewith n

processors.

8/11/2019 02 1 Lecture

http://slidepdf.com/reader/full/02-1-lecture 17/24Advanced Topics in Algorithms and Data Structures

Themesharchitecture

• Eachinternalprocessorofa2-dimensional

meshisconnectedto4neighbors.

•Whenwecombinetwodifferentmeshes,onlytheprocessorsontheboundaryneed

extralinks.Henceitishighlyscalable.

8/11/2019 02 1 Lecture

http://slidepdf.com/reader/full/02-1-lecture 18/24Advanced Topics in Algorithms and Data Structures

• Boththediameterandbisectionwidthofan

n-processor,2-dimensionalmeshis

 A4x4mesh

Themesharchitecture

( )O n

8/11/2019 02 1 Lecture

http://slidepdf.com/reader/full/02-1-lecture 19/24Advanced Topics in Algorithms and Data Structures

Hypercubesof0,1,2and3dimensions

Thehypercubearchitecture

8/11/2019 02 1 Lecture

http://slidepdf.com/reader/full/02-1-lecture 20/24Advanced Topics in Algorithms and Data Structures

• Eachnodeofad -dimensionalhypercube

isnumberedusingd bits.Hence,there

are2d processorsinad -dimensionalhypercube.

• Twonodesareconnectedbyadirectlink

iftheirnumbersdifferonlybyonebit.

Thehypercubearchitecture

8/11/2019 02 1 Lecture

http://slidepdf.com/reader/full/02-1-lecture 21/24Advanced Topics in Algorithms and Data Structures

• Thediameterofad -dimensional

hypercubeisd asweneedtoflipatmostd 

bits(traversed links)toreachoneprocessorfromanother.

• Thebisectionwidthofad -dimensional

hypercubeis2d-1

.

Thehypercubearchitecture

8/11/2019 02 1 Lecture

http://slidepdf.com/reader/full/02-1-lecture 22/24

Advanced Topics in Algorithms and Data Structures

• Thehypercubeisahighlyscalable

architecture.Twod -dimensional

hypercubescanbeeasilycombinedtoformad+1-dimensionalhypercube.

• Thehypercubehasseveralvariantslike

butterfly,shuffle-exchangenetworkandcube-connectedcycles.

Thehypercubearchitecture

8/11/2019 02 1 Lecture

http://slidepdf.com/reader/full/02-1-lecture 23/24

Advanced Topics in Algorithms and Data Structures

 Addingnnumbersinsteps

 Addingnnumbersonthemesh

n

8/11/2019 02 1 Lecture

http://slidepdf.com/reader/full/02-1-lecture 24/24

Ad d T i i Al ith d D t St t

Adding n numbers in log n steps

 Addingnnumbersonthe

hypercube