Early days of message-passing computing: transputers, occam

and all that

Tony Hey Chief Data Scientist

STFC Rutherford Appleton Laboratory Harwell Campus, UK

•  In1981Iwasonsabba,calatCaltech–asatheore,calpar,clephysicist–andGeoffreyFoxandIwenttoacolloquiumbyCarverMead…

•  CarverdemonstratedthattherewerenoengineeringobstaclestochipsgeDngsmallerandfasterforthenext20years

•  IwentbacktotheUKandbuiltmessage-passingmachinesusingtheInmosTransputer

•  GeoffreyFoxcollaboratedwithChuckSeitzinbuildingahypercubemessage-passingmachinethatwasusableforscien,ficapplica,ons

TheBeginnings

•  Designedandbuiltintheearly1980’sbyGeoffreyFoxandChuckSeitzandtheirteamsinPhysicsandCS

•  Processorsatnodesofhypercube;messagepassingbetweennodes

•  Experimentedwithparallelizingawholesetofscien,ficapplica,ons

•  Developed‘CrystallineOS’–CrOS–whichwasreallyalibraryofcommunica,onrou,nes

•  Demonstratedadvantagesofvirtualaddresses,virtualcommunica,onchannelsandkernel-likesupportateachnode

TheCaltechCosmicCube

•  Exploiteddataparallelismofregularproblemsby‘domaindecomposi,on’

•  Forhighefficiency,needforlightweightkernelsonnodesthatallowedforlowlatencymessagestartup,mes

•  Laidthefounda,onsforparallelprogrammingmethodologyandparallelperformanceanalysisthatares,llrelevanttoday

•  Irregularproblemsweremoredifficult…

Lessonslearnt

•  TheIntelPersonalSupercomputer–  iPSC1hadOSFMachOSoneachnodewhichhadveryhighlatencyforini,a,ngcommunica,ons

–  iPSC2releasedsoonaZerwardswithNX‘DistributedProcess’environmentbasedonCaltech’s‘Reac,veKernel’OS

•  TheTransputerSupernodemachine–  BasedontheInmosT800transputerthatcombinedCPU,FPU,memoryandcommunica,onchannelsonchip

– Na,veprogramminglanguagewas‘occam’,arealiza,onofasimplifiedversionofHoare’sCSP

–  EU‘Supernode’project:MachinesmanufacturedbyTelMatandParsys

TheParallelCompu7ngLandscape(1)

•  Manyothervendorsofparallelmessage-passingmachines:– nCUBE– MeikoCS-1andCS-2– Suprenum– Parsytec–  IBMSPseries– …

Ø Eachvendorhadproprietarymessagepassingsystem

TheParallelCompu7ngLandscape(2)

•  ThePARMACSmacrosfromtheArgonneteam–  RustyLusketal.‘Mark1’

•  Thep4parallelprogrammingsystem–  RustyLusketal.‘Mark2’

•  TheParallelVirtualMachinePVM–  VaidySunderam,AlGeistandothers–  Supportedmessagepassingacrossheterogeneousdistributedsystems

•  ThePARMACSmessagepassinglibraries– DevelopedbyRolfHempelandothersintheEU‘PPPE’project

PortableMessagePassingInterfaces?

•  In1991GeoffreyFoxandKenKennedystartedacommunityprocesstowardsadataparallelFortranstandard–  ThisbecametheHighPerformanceFortraneffortandtypifiedthe‘heroic’compilerschoolofparallelprogramming

•  However,whatwasclearlyneededwasalowerlevelstandardforportabilityofmessagepassingprogramsacrossdifferentparallelcomputers–  TheUSwereusingp4andExpress–  TheEUwereusingPARMACSinthePPPEandRAPSprojects–  PVMwaswidelyusedforprogrammingnetworksofworksta,onsbutnotop,mizedformorecloselycoupledparallelmachines

TheOriginsofMPI(1)

•  WorkshoponStandardsforMessagePassinginaDistributedMemoryEnvironment– Williamsburg,Virginia,April1992– OrganizedbyJackDongarraandDavidWalker– SponsoredbyCRPCandKenKennedyurgedac,on

•  Insummerof1992,IcontactedJackDongarraaboutstar,ngsuchastandardiza,onac,vity– DidnotwantUSandEuropetodiverge– Co-wroteafirstdraZofanMPIstandardwithJackDongarra,RolfHempelandDavidWalkerinOctober1992,nowknownasMPI-0

TheOriginsofMPI(2)

OrganizedBOFsessionatSuperCompu,ng92inMinneapolis

•  MPI-0documentservedasacatalyst•  MarcSnirofIBMemailedmetosay‘hewashappytohavebeenplagiarized’

•  IhavenoideawhyweleZtheobviouscollec,vecommunica,onsrou,nesoutofMPI-0

•  RustyLuskandBillGroppfromArgonnevolunteeredtoproduceanopensourceimplementa,onoftheevolvingMPIstandard

•  AndtheEUPPPEprojectpaidforthebeer…

TheOriginsofMPI(3)

•  FollowedproceduresofHPFForum– Setambi,ousgoalofagreeingastandardwithinoneyear

•  Metevery6weeksinDallasairporthotel–  IsentIanGlendinningfrommygroupinSouthamptonfundedbyEUPPPEproject

•  InmyopinionMPI-1succeededbecause:– Argonneproducedanopensourceimplementa,on– Excep,onaltechnicalleadershipfrompeoplelikeMarcSnirfromIBMandJimCowniefromMeiko

–  Itwasneededandhadthesupportofthecommunity

TheMPIProcess

•  AdventofMPImeantthatitwaspossibletoassemblesuiteofMessage-Passingbenchmarksforperformanceanalysisofmachinesandapplica,ons

•  EUGenesisprojectdefined3levelsofbenchmarks–  Low-level,KernelsandasetofCompactApplica,onsimplementedwithPARMACSlibraries

•  Interna,onalParkbenchGroup–  CombinedGenesismethodologywithLinearAlgebraandNASParallelBenchmarksimplementedwithMPI-1

Ø Butthemarke,ngcommunitypreferredJack’sTop500Benchmark…

Parkbench:PortableDMMessage-PassingKernelsandBenchmarks

•  ManythankstoJackDongarra,RolfHempelandDavidWalker

•  Auseful‘aidememoire’wasthear,clebyDongarra,Fagg,HempelandWalkerintheEncyclopediaofElectronicsandElectricalEngineering(Wiley)

Acknowledgements

JackDongarra,RolfHempel,TonyHey,DavidWalker‘ADraZStandardforMessagePassingonDistributedMemoryComputers’ProceedingsoftheFiZhECMWFWorkshopontheUseofParallelProcessorsinMeteorology:“ParallelSupercompu,nginAtmosphericScience”Editors:Geerd-RHoffmannandTuomoKaurannePublishedbyWorldScien,fic,1993

MPI-0Reference