Moving Data Over Networks - nersc.gov · PDF fileMoving Data Over Networks Network-Based Data...

MovingDataOverNetworksNetwork-BasedDataTransferatNERSC

EliDart,NetworkEngineerESnetScienceEngagementLawrenceBerkeleyNaAonalLaboratory

NERSCUsersGroupTraining

Berkeley,CA

February24,2016

Outline

•  Context•  ScienceDMZoverview

•  DataTransferNodes•  HandofftoShreyasCholia

2 – ESnet Science Engagement (engage@es.net) - 2/24/17

ScienceNetworksforScience

•  TheglobalResearch&EducaAon(R&E)networkecosystemiscomprisedofhundredsofinternaAonal,naAonal,regionalandlocal-scalenetworks–eachindependentlyownedandoperated.

•  ThesenetworksarepartofandconnectedtotheInternet,butareengineeredspecificallyforhigh-performancescienAficapplicaAons

February24,2017 33 – ESnet Science Engagement (engage@es.net) - 2/24/17

•  DatatransfersbetweenresourcesconnectedtoR&Enetworkscandomuchbe\erthandatatransferswhichusethecommodityInternet–  Terabytesarenoproblem–  Petabytesarefeasible

•  Justneedtomakesurewedoacoupleofthings–  LongdistanceporAonsworkwellingeneral–  Large-scalecompuAngcentersworkwellingeneral–  LocalconfiguraAonisreallyimportant

•  NERSChashigh-performancedataresources–  Fastnetworks–  Fastsystemsandfilesystems

•  ThistalkwilldescribewhatyoucandotointerfacewithNERSCeffecAvely

Effec4veHighPerformanceDataTransfer

•  NetworksareanessenAalpartofdata-intensivescience–  Connectdatasourcestodataanalysis–  Connectcollaboratorstoeachother–  Enablemachine-consumableinterfacestodataandanalysisresources(e.g.portals),automaAon,scale

•  PerformanceiscriAcal–  ExponenAaldatagrowth–  Constanthumanfactors–  Datamovementanddataanalysismustkeepup

•  EffecAveuseofwidearea(long-haul)networksbyscienAstshashistoricallybeendifficult

•  Someofthisisforyoursystemadministrator–  Pointyoursysadmintoh\p://fasterdata.es.net/formoreinfo–  Feelfreetofollowupwithmelater–engage@es.net

Mo4va4on

TheCentralRoleoftheNetwork

•  Theverystructureofmodernscienceassumessciencenetworksexist:highperformance,featurerich,globalscope

•  Whatis“TheNetwork”anyway?–  “TheNetwork”isthesetofdevicesandapplicaAonsinvolvedintheuseofaremoteresource•  Thisisnotaboutsupercomputerinterconnects•  Thisisaboutdataflowfromexperimenttoanalysis,betweenfaciliAes,etc.

–  Userinterfacesfor“TheNetwork”–portal,datatransfertool,workflowengine–  Therefore,serversandapplicaAonsmustalsobeconsidered

•  Whatisimportant?Orderedlist:1.  Correctness2.  Consistency3.  Performance

TCP–UbiquitousandFragile

•  NetworksprovideconnecAvitybetweenapplicaAonsrunningonhosts–  FromanapplicaAon’sperspecAve,theinterfaceto“theotherend”isasocket

–  HostoperaAngsystemkernelprovidessocketinterface,kernelimplementsTCPwheretheapplicaAoncan’tsee

–  CommunicaAonisbetweenapplicaAons–mostlyoverTCP

•  TCP–thefragileworkhorse–  TCPis(forverygoodreasons)Amid–packetlossisinterpretedascongesAon

–  Likeitornot,TCPisusedforthevastmajorityofdatatransferapplicaAons(morethan95%ofESnettrafficisTCP)

–  PacketlossinconjuncAonwithlatencyisaperformancekiller

A small amount of packet loss makes a huge difference in TCP performance

MetroArea

Local(LAN)

Regional

ConAnental

InternaAonal

Measured (TCP Reno) Measured (HTCP) Theoretical (TCP Reno) Measured (no loss)

With loss, high performance beyond metro distances is essentially impossible

WorkingWithTCPInPrac4ce

•  FareasiertosupportTCPthantofixTCP–  PeoplehavebeentryingtofixTCPforyears–limitedsuccess–  Likeitornotwe’restuckwithTCPinthegeneralcase

•  PragmaAcallyspeaking,wemustaccommodateTCP–  SufficientbandwidthtoavoidcongesAon–  Zeropacketloss– Verifiableinfrastructure

•  Networksarecomplex•  Mustbeabletolocateproblemsquickly•  Smallfootprintisahugewin–smallnumberofdevicessothatproblemisolaAonistractable

PuIngASolu4onTogether

•  EffecAvesupportforTCP-baseddatatransfer–  Designforcorrect,consistent,high-performanceoperaAon–  DesignforeaseoftroubleshooAng

•  EasyadopAoniscriAcal–  LargelaboratoriesanduniversiAeshaveextensiveITdeployments–  DrasAcchangeisprohibiAvelydifficult

•  Cybersecurity–defensiblewithoutcompromisingperformance

•  BorrowideasfromtradiAonalnetworksecurity–  TradiAonalDMZ

•  Separateenclaveatnetworkperimeter(“DemilitarizedZone”)•  SpecificlocaAonforexternal-facingservices•  CleanseparaAonfrominternalnetwork

–  Dothesamethingforscience–ScienceDMZ

DedicatedSystemsforData

Transfer

NetworkArchitecture

PerformanceTesAng&

Measurement

DataTransferNode•  Highperformance•  Configuredspecifically

fordatatransfer•  Propertools

ScienceDMZ•  Dedicatednetwork

locaAonforhigh-speeddataresources

•  Appropriatesecurity•  Easytodeploy-noneed

toredesignthewholenetwork

perfSONAR•  EnablesfaultisolaAon•  VerifycorrectoperaAon•  WidelydeployedinESnet

andothernetworks,aswellassitesandfaciliAes

TheScienceDMZDesignPa\ern

AbstractorPrototypeDeployment

•  (ThissecAonisforyoursystemadministrator–sendthemtome,useengage@es.net)

•  Add-ontoexisAngnetworkinfrastructure–  Allthatisrequiredisaportontheborderrouter–  Smallfootprint,pre-producAoncommitment

•  Easytoexperimentwithcomponentsandtechnologies–  DTNprototyping–  perfSONARtesAng

•  LimitedscopemakessecuritypolicyexcepAonseasy–  Onlyallowtrafficfrompartners–  Add-ontoproducAoninfrastructure–lowerriskthanrebuildingexisAnginfrastructure

ScienceDMZDesignPaNern(Abstract)

Border Router

Science DMZSwitch/Router

Enterprise Border Router/Firewall

Site / CampusLAN

High performanceData Transfer Node

with high-speed storage

Per-service security policy control points

Clean, High-bandwidth

WAN path

Site / Campus access to Science

DMZ resources

perfSONAR

LocalAndWideAreaDataFlows

Border Router

Science DMZSwitch/Router

Site / CampusLAN

High performanceData Transfer Node

with high-speed storage

Per-service security policy control points

Clean, High-bandwidth

WAN path

Site / Campus access to Science

DMZ resources

perfSONAR

High Latency WAN Path

Low Latency LAN Path

perfSONAR

ModularArchitecture–Mul4pleScienceDMZs

Dark Fiber

DarkFiber

Border Router

Science DMZSwitch/Routers

Site / CampusLAN

Project A DTN(building A)

Per-project securitypolicy

perfSONAR

Facility B DTN(building B)

Cluster DTN(building C)

perfSONARperfSONAR

Cluster(building C)

SupercomputerCenterDeployment

•  High-performancenetworkingisassumedinthisenvironment–  Dataflowsbetweensystems,betweensystemsandstorage,widearea,etc.–  GlobalfilesystemoqenAesresourcestogether

•  PorAonsofthismaynotrunoverEthernet(e.g.IB)•  ImplicaAonsforDataTransferNodes

•  “ScienceDMZ”maynotlooklikeadiscreteenAtyhere–  BytheAmeyougetthroughinterconnecAngalltheresources,youendupwithmostofthenetworkintheScienceDMZ

–  Thisisasitshouldbe–thepointisappropriatedeploymentoftools,configuraAon,policycontrol,etc.

•  Officenetworkscanlooklikeanaqerthought,buttheyaren’t–  Deployedwithappropriatesecuritycontrols–  Officeinfrastructureneednotbesizedforsciencetraffic

HPCCenter

Routed

Border Router

Core Switch/Router

Firewall

Offices

perfSONAR

Supercomputer

Parallel Filesystem

Front endswitch

Data Transfer Nodes

Front endswitch

HPCCenterDataPath

Routed

Border Router

Core Switch/Router

Firewall

Offices

perfSONAR

Supercomputer

Parallel Filesystem

Front endswitch

Data Transfer Nodes

Front endswitch

High Latency WAN Path

Low Latency LAN Path

CommonThreads

•  Twocommonthreadsexistinalltheseexamples

•  AccommodaAonofTCP–  WideareaporAonofdatatransferstraversespurpose-builtpath–  Highperformancedevicesthatdon’tdroppackets

•  Abilitytotestandverify–  Whenproblemsarise(andtheyalwayswill),theycanbesolvediftheinfrastructureisbuiltcorrectly

–  Smalldevicecountmakesiteasiertofindissues–  MulApletestandmeasurementhostsprovidemulApleviewsofthedatapath•  perfSONARnodesatthesiteandintheWAN•  perfSONARnodesattheremotesite

DedicatedSystems–DataTransferNode

•  TheDTNisdedicatedtodatatransfer•  Setupspecificallyforhigh-performancedatamovement

–  Systeminternals(BIOS,firmware,interrupts,etc.)–  Networkstack–  Storage(globalfilesystem,Fibrechannel,localRAID,etc.)–  Highperformancetools–  Noextraneoussoqware

•  Limita.onofscopeandfunc.onispowerful–  NoconflictswithconfiguraAonforothertasks–  SmallapplicaAonsetmakescybersecurityeasier–keypoint

DataTransferToolsForDTNs

•  Parallelismisimportant–  ItisoqeneasiertoachieveagivenperformancelevelwithfourparallelconnecAonsthanoneconnecAon

–  Severaltoolsofferparalleltransfers,includingGlobus/GridFTP

•  LatencyinteracAoniscriAcal–  WideareadatatransfershavemuchhigherlatencythanLANtransfers–  ManytoolsandprotocolsassumeaLAN

•  WorkflowintegraAonisimportant

•  Keytools:GlobusOnline,HPN-SSH•  ESnettestDTNs:h\p://fasterdata.es.net/performance-tesAng/DTNs/

DataTransferToolComparison•  InaddiAontothenetwork,usingtherightdatatransfertooliscriAcal•  DatatransfertestfromBerkeley,CAtoArgonne,IL(nearChicago).RTT=53ms,networkcapacity=10Gbps.

Tool Throughput SCP: 140Mbps HPNpatchedSCP: 1.2GbpsFTP 1.4Gbps GridFTP,4streams 5.4Gbps GridFTP,8streams 6.6Gbps

•  NERSCDTNshavebothHPN-SSHandGlobus•  Keypoint–yourlocalDTNandnetworkconnecAonsignificantlyaffectyourabilitytomovedatainandoutofNERSC

8.9 Gbps

10.7 Gbps

7.5 Gbps

6.3 Gbps

7.6 Gbps 9.8 Gbps

11.4 Gbps

9.2 Gbps

21.9 Gbps

15.6 Gbps

26.1 Gbps

2.8 Gbps

alcf#dtn_miraALCF

nersc#dtnNERSC

olcf#dtn_atlasOLCF

ncsa#BlueWatersNCSA

Data set: L380Files: 19260Directories: 211Other files: 0Total bytes: 4442781786482 (4.4T bytes)Smallest file: 0 bytes (0 bytes)Largest file: 11313896248 bytes (11G bytes)Size distribution:

1 - 10 bytes: 7 files10 - 100 bytes: 1 files100 - 1K bytes: 59 files1K - 10K bytes: 3170 files10K - 100K bytes: 1560 files100K - 1M bytes: 2817 files1M - 10M bytes: 3901 files10M - 100M bytes: 3800 files100M - 1G bytes: 2295 files1G - 10G bytes: 1647 files10G - 100G bytes: 3 files

October 2016L380 Data Set

PerformanceBetweenCompu4ngFacili4es

23 – ESnet Science Engagement (engage@es.net) - 2/24/17

HandofftoShreyasCholia

•  Thanks!

Moving Data Over Networks - nersc.gov · PDF fileMoving Data Over Networks Network-Based Data...

Documents

Python in a Parallel Environment - nersc.gov · Python in a Parallel Environment Dave Grote ... first statement in any Warp input file. ... – More hacking is needed • Method based

Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov

Moving beyond colour - · PDF fileMoving beyond colour ... fats and oils refining, designed to remove ... soybean, palm, rapeseed, sunflower, palm kernel and coconut oil,

Quality and testing of PE release - nersc.gov · Zhengji Zhao! NERSC ! Cray QBR Meeting! July 30, 2014, Oakland, CA Quality and testing of PE release

File Transfer Best Practices - nersc.gov

Moving Interface Problems: Methods & Applications Tutorial ... · PDF fileMoving Interface Problems: Methods & Applications Tutorial Lecture II ... Numerical Method. ... equations

023 h1507 Special Report - ma · PDF filemoving into the top spot on the list. Marriott International moves into the second slot by adding 39,000 rooms,

Git + Docker tutorial - nersc.gov

Electricity Generation Due to Vibration of Moving Vehicles ... · PDF fileMoving Vehicles Using Piezoelectric Effect ... Electricity Generation Due to Vibration of Moving Vehicles

Exascale Programming Challenges - nersc.gov · Exascale Programming Challenges Sponsored by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing

MenaITech Coprorate Profile 2018 · PDF fileMoving HR from transactional function to transformational ... transactional technology usage, ... HR, while aligning your

Moving to a Smart Distribution Grid through Automatic ... · PDF fileMoving to a Smart Distribution Grid through Automatic Dynamic Loading of Substation Distribution Transformers by

Moving walls - Syston · PDF filemoving walls Albert Street, Syston, Leicester LE7 2JB, U.K. - Telephone: (0116) 260 8841 - Fax: (0116) 264 0846 ... CONCERTINA| FOLDING SYSTEMS

An Introduction to Grid Technologies at NERSC June 24, 2004 David Turner NERSC User Services Group dpturner@nersc.gov 510-486-4027

Moving Forward - ISCC System · PDF fileMoving Forward on Social Issues ... Grievance mechanism ... MSPO, the Free and Fair Labor in Palm Oil Principles,

The ACTS Toolkit (What can it do for you?) Osni Marques and Tony Drummond ( LBNL/NERSC ) acts-support@nersc.gov

MOVING ON FROM - World Meteorological · PDF fileMOVING ON FROM MERCURY: ... †Guidance in WMO CIMO Commission for Instruments flyer ... • Field‐swappable units and sensors

MOVING LAB EQUIPMENT Business - · PDF filemoving laboratory equipment laboratory relocation requires expertise in managing the complexities associated with moving scientific instruments

NASCIO 2015 CISO Toolkit Moving Forward: Leadership ... · PDF fileMoving Forward: Leadership Toolkit for State CISOs TABLE OF CONTENTS ... In 2006, the National ... Hackers have their

Moving towards Green Economies and Green Jobs - iges.or.jp · PDF fileMoving towards Green Economies and Green Jobs - a developing country perspective UNEP International Resource Panel