62
The Science DMZ Eli Dart, Network Engineer ESnet Science Engagement Lawrence Berkeley Na;onal Laboratory Moving My Data at High Speeds over the Network TNC16 Prague, CZ June 12, 2016

The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

  • Upload
    others

  • View
    6

  • Download
    0

Embed Size (px)

Citation preview

Page 1: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

The  Science  DMZ  

Eli  Dart,  Network  Engineer  ESnet  Science  Engagement  Lawrence  Berkeley  Na;onal  Laboratory  

Moving  My  Data  at  High  Speeds  over  the  Network    

TNC16  

Prague,  CZ  

June  12,  2016  

Page 2: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Overview  –  XXX    

•  Science  DMZ  Mo;va;on  and  Introduc;on  

•  Science  DMZ  Architecture  

•  Science  DMZ  Security  

•  Wrap  Up  

2 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 3: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

•  Networks  are  an  essen;al  part  of  data-­‐intensive  science  –  Connect  data  sources  to  data  analysis  –  Connect  collaborators  to  each  other  –  Enable  machine-­‐consumable  interfaces  to  data  and  analysis  resources  (e.g.  portals),  automa;on,  scale  

•  Performance  is  cri;cal  –  Exponen;al  data  growth  –  Constant  human  factors  –  Data  movement  and  data  analysis  must  keep  up  

•  Effec;ve  use  of  wide  area  (long-­‐haul)  networks  by  scien;sts  has  historically  been  difficult  –  Especially  for  large-­‐scale  data  movement  

 

Mo,va,on  

3 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 4: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

The  Central  Role  of  the  Network  

•  The  very  structure  of  modern  science  assumes  science  networks  exist:  high  performance,  feature  rich,  global  scope  

•  What  is  “The  Network”  anyway?  –  “The  Network”  is  the  set  of  devices  and  applica;ons  involved  in  the  use  of  a  remote  resource  •  This  is  not  about  supercomputer  interconnects  •  This  is  about  data  flow  from  experiment  to  analysis,  between  facili;es,  etc.  

–  User  interfaces  for  “The  Network”  –  portal,  data  transfer  tool,  workflow  engine  –  Therefore,  servers  and  applica;ons  must  also  be  considered  

•  What  is  important?    Ordered  list:  1.  Correctness  2.  Consistency  3.  Performance  

4 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 5: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

TCP  –  Ubiquitous  and  Fragile  

•  Networks  provide  connec;vity  between  hosts  –  how  do  hosts  see  the  network?  –  From  an  applica;on’s  perspec;ve,  the  interface  to  “the  other  end”  is  a  socket  

–  Communica;on  is  between  applica;ons  –  mostly  over  TCP  

•  TCP  –  the  fragile  workhorse  –  TCP  is  (for  very  good  reasons)  ;mid  –  packet  loss  is  interpreted  as  conges;on  

–  Like  it  or  not,  TCP  is  used  for  the  vast  majority  of  data  transfer  applica;ons  (more  than  95%  of  ESnet  traffic  is  TCP)  

–  Packet  loss  in  conjunc;on  with  latency  is  a  performance  killer  

5 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 6: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

A small amount of packet loss makes a huge difference in TCP performance

Metro  Area  

Local  (LAN)  

Regional  

Con;nental  

Interna;onal  

Measured (TCP Reno) Measured (HTCP) Theoretical (TCP Reno) Measured (no loss)

With loss, high performance beyond metro distances is essentially impossible

6 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 7: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Working  With  TCP  In  Prac,ce  

•  Far  easier  to  support  TCP  than  to  fix  TCP  –  People  have  been  trying  to  fix  TCP  for  years  –  limited  success  –  Like  it  or  not  we’re  stuck  with  TCP  in  the  general  case  

•  Pragma;cally  speaking,  we  must  accommodate  TCP  –  Sufficient  bandwidth  to  avoid  conges;on  –  Zero  packet  loss  –  Verifiable  infrastructure  

•  Networks  are  complex  •  Must  be  able  to  locate  problems  quickly  •  Small  footprint  is  a  huge  win  –  small  number  of  devices  so  that  problem  isola;on  is  tractable  

•  What  if  I  don’t  use  TCP?  –  TCP  benefits  are  significant,  but  are  not  the  only  reason  for  Science  DMZ  –  Architecture,  cost,  opera;onal  benefits  

7 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 8: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

PuDng  A  Solu,on  Together  

•  Effec;ve  support  for  TCP-­‐based  data  transfer  –  Design  for  correct,  consistent,  high-­‐performance  opera;on  –  Design  for  ease  of  troubleshoo;ng  

•  Easy  adop;on  is  cri;cal  –  Large  laboratories  and  universi;es  have  extensive  IT  deployments  –  Dras;c  change  is  prohibi;vely  difficult  

•  Cybersecurity  –  defensible  without  compromising  performance  

•  Borrow  ideas  from  tradi;onal  network  security  –  Tradi;onal  DMZ  

•  Separate  enclave  at  network  perimeter  (“Demilitarized  Zone”)  •  Specific  loca;on  for  external-­‐facing  services  •  Clean  separa;on  from  internal  network  

–  Do  the  same  thing  for  science  –  Science  DMZ  

8 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 9: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Dedicated  Systems  for  Data  

Transfer  

Network  Architecture  

Performance  Tes;ng  &  

Measurement  

Data  Transfer  Node  •  High  performance  •  Configured  specifically  

for  data  transfer  •  Proper  tools  

Science  DMZ  •  Dedicated  network  

loca;on  for  high-­‐speed  data  resources  

•  Appropriate  security  •  Easy  to  deploy  -­‐  no  need  

to  redesign  the  whole  network  

perfSONAR                          •  Enables  fault  isola;on  •  Verify  correct  opera;on  •  Widely  deployed  in  ESnet  

and  other  networks,  as  well  as  sites  and  facili;es  

The  Science  DMZ  Design  Pakern  

9 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 10: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Abstract  or  Prototype  Deployment  

•  Add-­‐on  to  exis;ng  network  infrastructure  –  All  that  is  required  is  a  port  on  the  border  router  –  Small  footprint,  pre-­‐produc;on  commitment  

•  Easy  to  experiment  with  components  and  technologies  –  DTN  prototyping  –  perfSONAR  tes;ng  

•  Limited  scope  makes  security  policy  excep;ons  easy  –  Only  allow  traffic  from  partners  –  Add-­‐on  to  produc;on  infrastructure  –  lower  risk  than  rebuilding  exis;ng  infrastructure  

10 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 11: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Science  DMZ  Design  PaLern  (Abstract)  

10GE

10GE

10GE

10GE

10G

Border Router

WAN

Science DMZSwitch/Router

Enterprise Border Router/Firewall

Site / CampusLAN

High performanceData Transfer Node

with high-speed storage

Per-service security policy control points

Clean, High-bandwidth

WAN path

Site / Campus access to Science

DMZ resources

perfSONAR

perfSONAR

perfSONAR

11 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 12: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Local  And  Wide  Area  Data  Flows  

10GE

10GE

10GE

10GE

10G

Border Router

WAN

Science DMZSwitch/Router

Enterprise Border Router/Firewall

Site / CampusLAN

High performanceData Transfer Node

with high-speed storage

Per-service security policy control points

Clean, High-bandwidth

WAN path

Site / Campus access to Science

DMZ resources

perfSONAR

perfSONAR

High Latency WAN Path

Low Latency LAN Path

perfSONAR

12 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 13: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Modular  Architecture  –  Mul,ple  Science  DMZs  

Dark Fiber

DarkFiber

10GE

DarkFiber

10GE

10G

Border Router

WAN

Science DMZSwitch/Routers

Enterprise Border Router/Firewall

Site / CampusLAN

Project A DTN(building A)

Per-project securitypolicy

perfSONAR

perfSONAR

Facility B DTN(building B)

Cluster DTN(building C)

perfSONARperfSONAR

Cluster(building C)

13 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 14: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Supercomputer  Center  Deployment  

•  High-­‐performance  networking  is  assumed  in  this  environment  –  Data  flows  between  systems,  between  systems  and  storage,  wide  area,  etc.  –  Global  filesystem  omen  ;es  resources  together  

•  Por;ons  of  this  may  not  run  over  Ethernet  (e.g.  IB)  •  Implica;ons  for  Data  Transfer  Nodes  

•  “Science  DMZ”  may  not  look  like  a  discrete  en;ty  here  –  By  the  ;me  you  get  through  interconnec;ng  all  the  resources,  you  end  up  with  most  of  the  network  in  the  Science  DMZ  

–  This  is  as  it  should  be  –  the  point  is  appropriate  deployment  of  tools,  configura;on,  policy  control,  etc.  

•  Office  networks  can  look  like  an  amerthought,  but  they  aren’t  –  Deployed  with  appropriate  security  controls  –  Office  infrastructure  need  not  be  sized  for  science  traffic  

14 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 15: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

HPC  Center  

©  2014,  Energy  Sciences  Network  15 – ESnet Science Engagement ([email protected]) - 06.07.16

Routed

Border Router

WAN

Core Switch/Router

Firewall

Offices

perfSONAR

perfSONAR

perfSONAR

Supercomputer

Parallel Filesystem

Front endswitch

Data Transfer Nodes

Front endswitch

Page 16: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

HPC  Center  Data  Path  

©  2014,  Energy  Sciences  Network  16 – ESnet Science Engagement ([email protected]) - 06.07.16

Routed

Border Router

WAN

Core Switch/Router

Firewall

Offices

perfSONAR

perfSONAR

perfSONAR

Supercomputer

Parallel Filesystem

Front endswitch

Data Transfer Nodes

Front endswitch

High Latency WAN Path

Low Latency LAN Path

Page 17: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Common  Threads  

•  Two  common  threads  exist  in  all  these  examples  

•  Accommoda;on  of  TCP  –  Wide  area  por;on  of  data  transfers  traverses  purpose-­‐built  path  –  High  performance  devices  that  don’t  drop  packets  

•  Ability  to  test  and  verify  –  When  problems  arise  (and  they  always  will),  they  can  be  solved  if  the  infrastructure  is  built  correctly  

–  Small  device  count  makes  it  easier  to  find  issues  –  Mul;ple  test  and  measurement  hosts  provide  mul;ple  views  of  the  data  path  •  perfSONAR  nodes  at  the  site  and  in  the  WAN  •  perfSONAR  nodes  at  the  remote  site  

17 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 18: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

But  What  If  I  Don’t  Use  TCP?  •  Some  sites  use  non-­‐TCP  tools/protocols  

–  Open  source  (e.g.  UDT)  –  Commercial  (e.g.  Aspera)  

•  Does  this  mean  we  don’t  need  a  Science  DMZ?  –  The  short  answer  is  no…a  Science  DMZ  is  s;ll  very  valuable  –  There  are  many  different  reasons  

•  Tension  between  security  and  performance  •  Offload  bandwidth  hogs  from  enterprise  network  •  Cost  savings  –  consolidate  high-­‐performance  services,  reduce  device  count  •  Flexibility  of  provisioning,  policy  applica;on,  enforcement  •  Flexibility  of  technology  adop;on  

•  Flexibility  offered  by  Science  DMZ  is  cri;cal  –  Decouple  enterprise  network  (stability  is  key)  from  science  infrastructure  –  How  fast  can  you  adapt?    How  fast  must  you  adapt?  

18 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 19: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Overview  

•  Science  DMZ  Mo;va;on  and  Introduc;on  

•  Science  DMZ  Architecture  

•  Science  DMZ  Security  

•  Wrap  Up  

 

19 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 20: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

•  Goal  –  disentangle  security  policy  and  enforcement  for  science  flows  from  security  for  business  systems  

•  Ra;onale  –  Science  data  traffic  is  simple  from  a  security  perspec;ve  –  Narrow  applica;on  set  on  Science  DMZ  

•  Data  transfer,  data  streaming  packages  •  No  printers,  document  readers,  web  browsers,  building  control  systems,  financial  databases,  staff  desktops,  etc.    

–  Security  controls  that  are  typically  implemented  to  protect  business  resources  omen  cause  performance  problems  

•  Separa;on  allows  each  to  be  op;mized  

Science  DMZ  Security  

20 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 21: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Performance  Is  A  Core  Requirement  

•  Core  informa;on  security  principles  –  Confiden;ality,  Integrity,  Availability  (CIA)  –  Omen,  CIA  and  risk  mi;ga;on  result  in  poor  performance  

•  In  data-­‐intensive  science,  performance  is  an  addi;onal  core  mission  requirement:  CIA  à  PICA  –  CIA  principles  are  important,  but  if  performance  is  compromised  the  science  mission  fails    

–  Not  about  “how  much”  security  you  have,  but  how  the  security  is  implemented  

–  Need  a  way  to  appropriately  secure  systems  without  performance  compromises  

21 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 22: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Placement  Outside  the  Firewall  

•  The  Science  DMZ  resources  are  placed  outside  the  enterprise  firewall  for  performance  reasons  –  The  meaning  of  this  is  specific  –  Science  DMZ  traffic  does  not  traverse  the  firewall  data  plane  

–  Packet  filtering  is  great  –  just  don’t  do  it  with  an  enterprise  firewall  

•  Lots  of  heartburn  over  this,  especially  from  the  perspec;ve  of  a  conven;onal  firewall  manager  –  Lots  of  organiza;onal  policy  direc;ves  manda;ng  firewalls  –  Firewalls  are  designed  to  protect  converged  enterprise  networks  –  Why  would  you  put  cri;cal  assets  outside  the  firewall???  

•  The  answer  is  that  enterprise  firewalls  are  typically  a  poor  fit  for  high-­‐performance  science  applica;ons  

22 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 23: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Typical  Firewall  Internals  

•  Typical  firewalls  are  composed  of  a  set  of  processors  which  inspect  traffic  in  parallel  –  Traffic  distributed  among  processors  such  that  all  traffic  for  a  par;cular  connec;on  goes  to  the  same  processor  

–  Simplifies  state  management  –  Paralleliza;on  scales  deep  analysis  

•  Excellent  fit  for  enterprise  traffic  profile  –  High  connec;on  count,  low  per-­‐connec;on  data  rate  –  Complex  protocols  with  embedded  threats  

•  Each  processor  is  a  frac;on  of  firewall  link  speed  –  Significant  limita;on  for  data-­‐intensive  science  applica;ons  –  Overload  causes  packet  loss  –  performance  crashes  

23 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 24: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Firewall  Capabili,es  and  Science  Traffic  

•  Commercial  firewalls  have  a  lot  of  sophis;ca;on  in  an  enterprise  serng  –  Applica;on  layer  protocol  analysis  (HTTP,  POP,  MSRPC,  etc.)  –  Built-­‐in  VPN  servers  –  User  awareness  

•  Data-­‐intensive  science  flows  typically  don’t  match  this  profile  –  Common  case  –  data  on  filesystem  A  needs  to  be  on  filesystem  Z  

•  Data  transfer  tool  verifies  creden;als  over  an  encrypted  channel  •  Then  open  a  socket  or  set  of  sockets,  and  send  data  un;l  done  (1TB,  10TB,  100TB,  …)  

–  One  workflow  can  use  10%  to  50%  or  more  of  a  10G  network  link  

•  Do  we  have  to  use  a  commercial  firewall?  

24 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 25: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Firewalls  As  Access  Lists  

• When  you  ask  a  firewall  administrator  to  allow  data  transfers  through  the  firewall,  what  do  they  ask  for?  –  IP  address  of  your  host  –  IP  address  of  the  remote  host  –  Port  range  –  That  looks  like  an  ACL  to  me!  

•  No  special  config  for  advanced  protocol  analysis  –  just  address/port  

•  Router  ACLs  are  beker  than  firewalls  at  address/port  filtering  –  ACL  capabili;es  are  typically  built  into  the  router  –  Router  ACLs  typically  do  not  drop  traffic  permiked  by  policy    

25 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 26: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Security  Without  Enterprise  Firewalls  

•  Data  intensive  science  traffic  interacts  poorly  with  enterprise  firewalls  

•  Does  this  mean  we  ignore  security?    NO!  – We  must  protect  our  systems  – We  just  need  to  find  a  way  to  do  security  that  does  not  prevent  us  from  gerng  the  science  done  

•  Key  point  –  security  policies  and  mechanisms  that  protect  the  Science  DMZ  should  be  implemented  so  that  they  do  not  compromise  performance  

•  Traffic  permiked  by  policy  should  not  experience  performance  impact  as  a  result  of  the  applica;on  of  policy  

26 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 27: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Systems  View  Of  Science  Infrastructure  

•  Security  is  a  component,  not  a  gatekeeper  

•  Think  about  the  workflows  •  Think  about  the  interfaces  to  data  (tools,  applica;ons)  

–  How  do  collaborators  access  data?  –  How  could  they  access  data  if  the  architecture  were  different?  

•  Think  about  costs/benefits  –  What  is  a  new  cancer  breakthrough  worth?  –  $30k  for  a  few  DTNs  –  what  is  that  in  context?  

•  Think  about  risks  –  What  risks  do  specific  technologies  mi;gate?  –  What  are  opportunity  costs  of  poor  performance?  

27 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 28: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Overview  

•  Science  DMZ  Mo;va;on  and  Introduc;on  

•  Science  DMZ  Architecture  

•  Science  DMZ  Security  

•  Wrap  Up  

28 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 29: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Context:  Science  DMZ  Adop,on  •  DOE  Na;onal  Laboratories  

–  Both  large  and  small  sites  –  HPC  centers,  LHC  sites,  experimental  facili;es  

•  NSF  CC-­‐NIE  and  CC*IIE  programs  leverage  Science  DMZ  –  $40M  and  coun;ng  (CC*DNI  awards  coming  soon,  es;mate  addi;onal  $18M  to  $20M)  –  Significant  investments  across  the  US  university  complex,  ~130  awards  –  Big  shoutout  to  Kevin  Thompson  and  the  NSF  –  these  programs  are  cri;cally  important  

•  Na;onal  Ins;tutes  of  Health  –  100G  network  infrastructure  refresh  

•  US  Department  of  Agriculture  –  Agricultural  Research  Service  is  building  a  new  science  network  based  on  the  Science  DMZ  model  –  hkps://www.xo.gov/index?s=opportunity&mode=form&tab=core&id=a7f291f4216b5a24c1177a5684e1809b  

•  Other  US  agencies  looking  at  Science  DMZ  model  –  NASA  –  NOAA  

•  Australian  Research  Data  Storage  Infrastructure  (RDSI)  –  Science  DMZs  at  major  sites,  connected  by  a  high  speed  network  –  hkps://www.rdsi.edu.au/dashnet  –  hkps://www.rdsi.edu.au/dashnet-­‐deployment-­‐rdsi-­‐nodes-­‐begins  

 29 – ESnet Science Engagement ([email protected]) - 06.07.16

©  2015,  Energy  Sciences  Network  

Page 30: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Strategic  Impacts  

•  What  does  this  mean?  –  We  are  in  the  midst  of  a  significant  cyberinfrastructure  upgrade  –  Enterprise  networks  need  not  be  unduly  perturbed  J  

•  Significantly  enhanced  capabili;es  compared  to  3  years  ago  –  Terabyte-­‐scale  data  movement  is  much  easier  –  Petabyte-­‐scale  data  movement  possible  outside  the  LHC  experiments  –  Widely-­‐deployed  tools  are  much  beker  (e.g.  Globus)  

•  Metcalfe’s  Law  of  Network  U;lity  –  Value  propor;onal  to  the  square  of  the  number  of  DMZs?  n  log(n)?  –  Cyberinfrastructure  value  increases  as  we  all  upgrade  

30 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 31: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Why  Build  A  Science  DMZ?  

•  Data  set  scale  –  Detector  output  increasing  

•  1Hz  à  10Hz  à  100Hz  à  1kHz  …  à  1MHz      –  HPC  scale  increasing  

•  Increased  model  resolu;on  à  increased  data  size  •  Increased  HPC  capability  means  addi;onal  problems  can  now  be  solved  

–  Sequencers,  Mass  Spectrometers,  …  •  Data  placement  

–  Move  compute  to  the  data?    –  Sure,  if  you  can…otherwise  you  need  to  move  the  data  

•  Without  a  Science  DMZ,  this  stuff  is  hard  –  Can  you  assume  nobody  at  your  ins;tu;on  will  do  this  kind  of  work?  –  If  this  kind  of  work  can’t  be  done,  what  does  that  mean  in  5  years?  

31 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 32: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

•  The  Science  DMZ  design  pakern  provides  a  flexible  model  for  suppor;ng  high-­‐performance  data  transfers  and  workflows  

•  Key  elements:  –  Accommoda;on  of  TCP  

•  Sufficient  bandwidth  to  avoid  conges;on  •  Loss-­‐free  IP  service  

–  Loca;on  –  near  the  site  perimeter  if  possible  –  Test  and  measurement  –  Dedicated  systems  –  Appropriate  security  

•  Science  DMZ  gives  flexibility,  scaling,  incremental  provisioning  for  advanced  services  

Wrapup  

32 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 33: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Links  

–  ESnet  fasterdata  knowledge  base  •  hkp://fasterdata.es.net/  

–  Science  DMZ  paper  •  hkp://www.es.net/assets/pubs_presos/sc13sciDMZ-­‐final.pdf  

–  Science  DMZ  email  list  •  hkps://gab.es.net/mailman/lis;nfo/sciencedmz  

–  perfSONAR  •  hkp://fasterdata.es.net/performance-­‐tes;ng/perfsonar/  •  hkp://www.perfsonar.net      

 

33 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 34: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Thanks!  

Energy  Sciences  Network  (ESnet)  Lawrence  Berkeley  Na;onal  Laboratory  

hkp://fasterdata.es.net/  

hkp://my.es.net/  

hkp://www.es.net/  

Page 35: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Extra  Slides  –  DTN  Cluster  for  HPC  Cluster  

35 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 36: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

HPC/Cluster  Environment  

•  Common  cluster  architectural  elements  –  Head/Login  nodes  

•  Primary  user  access  •  SSH  typically  required  for  access  (for  security  reasons)  •  Job  submission  tools  •  Small-­‐scale  test  jobs  

–  Compute  nodes  •  Run  user  jobs  •  Typically  not  accessible  from  outside  

–  Central  filesystem  •  Input  data  sets  •  Results  of  simula;on/analysis  •  Available  on  compute  nodes,  login  nodes,  etc.  

06.07.16  36  

Page 37: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Abstract  Cluster  

10GE10GE

Border Router

WAN

Firewall

Enterprise

perfSONAR

perfSONAR

10GE10GE

Filesystem

HEADCluster

Head/LoginNodes

Cluster compute nodes

HEAD

06.07.16  37  

Page 38: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Limita,ons  of  Abstract  Design  –  no  DTNs  

•  Abstract  cluster  has  limited  data  capabili;es  

•  All  data  transfers  must  traverse  Head/Login  nodes  –  Firewall  in  the  path  –  Configura;on  for  data  transfer  tools  conflated  with  cluster  configura;on  –  User  interac;ve  jobs  keep  CPUs  busy  

•  Solu;on  –  add  DTNs  –  Connect  DTNs  to  Science  DMZ  –  Mount  central  filesystem  on  DTNs  –  Only  permit  data  transfer  tools  on  DTNs  

06.07.16  38  

Page 39: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Abstract  Cluster  With  Science  DMZ  Data  Access  

10GE10GE

10GE10GE

10GE

10GE

Border Router

WAN

Science DMZSwitch/Router

Firewall

Enterprise

perfSONAR

perfSONAR

10GE10GE

10GE

10GE

10GE10GE

DTN

DTN

Filesystem

HEAD

“Sealed” DTNs(Globus only, no

shell access)

ClusterHead/Login

Nodes

DTN

DTN

Cluster compute nodes

HEAD

perfSONAR

06.07.16  39  

Page 40: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Cluster  Data  Paths  

10GE10GE

10GE10GE

10GE

10GE

Border Router

WAN

Science DMZSwitch/Router

Firewall

Enterprise

perfSONAR

perfSONAR

10GE10GE

10GE

10GE

10GE10GE

DTN

DTN

Filesystem

HEAD

“Sealed” DTNs(Globus only, no

shell access)

ClusterHead/Login

Nodes

DTN

DTN

Cluster compute nodes

HEAD

perfSONAR

Data Transfer Path

User Login/Shell Access Path

Compute Data Access Path

06.07.16  40  

Page 41: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Cluster  Security  Controls  

10GE10GE

10GE10GE

10GE

10GE

Border Router

WAN

Science DMZSwitch/Router

Firewall

Enterprise

perfSONAR

perfSONAR

10GE10GE

10GE

10GE

10GE10GE

DTN

DTN

Filesystem

HEAD

“Sealed” DTNs(Globus only, no

shell access)

ClusterHead/Login

Nodes

DTN

DTN

Cluster compute nodes

DTN Security Controls

HEAD

Filesystem Security Controls

perfSONAR

06.07.16  41  

Page 42: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Cluster  With  DTNs  –  Full  Picture  

10GE10GE

10GE10GE

10GE

10GE

Border Router

WAN

Science DMZSwitch/Router

Firewall

Enterprise

perfSONAR

perfSONAR

10GE10GE

10GE

10GE

10GE10GE

DTN

DTN

Filesystem

HEAD

“Sealed” DTNs(Globus only, no

shell access)

ClusterHead/Login

Nodes

DTN

DTN

Cluster compute nodes

DTN Security Controls

HEAD

Filesystem Security Controls

perfSONAR

Data Transfer Path

User Login/Shell Access Path

Compute Data Access Path

06.07.16  42  

Page 43: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Let’s  Talk  About  Risk  

•  What  part  of  the  cluster  poses  the  greatest  security  risk?  •  Greatest  risk:  Head/Login  nodes  

–  Users  have  shell  access  (via  SSH  –  the  firewall  can’t  see!)  –  Users  have  compiler  access,  can  build/run  arbitrary  code  –  Programma;c  access  to  filesystem  

•  Second  greatest  risk:  Compute  Nodes  –  Users  can  run  arbitrary  code  –  Programma;c  access  to  filesystem  

•  Lowest  risk:  DTNs  –  No  user  shell  access  –  No  user  programma;c  filesystem  access  –  Data  transfer  applica;on  (e.g.  Globus)  is  the  only  method  of  interac;on  

06.07.16  43  

Page 44: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

DTN  Cluster  Detail  

10GE10GE

10GE10GE

10GE

10GE

Border Router

WAN

Science DMZSwitch/Router

Firewall

Enterprise

perfSONAR

perfSONAR

10GE10GE

10GE

10GE

10GE10GE

DTN

DTN

Filesystem

HEAD

“Sealed” DTNs(Globus only, no

shell access)

ClusterHead/Login

Nodes

DTN

DTN

Cluster compute nodes

HEAD

perfSONAR

Configure as DTN Cluster

06.07.16  44  

Page 45: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

DTN  Cluster  Design  

•  Configure  all  four  DTNs  as  a  single  Globus  endpoint  –  Globus  has  docs  on  how  to  do  this  –  hkps://support.globus.org/entries/71011547-­‐How-­‐do-­‐I-­‐add-­‐mul;ple-­‐I-­‐O-­‐nodes-­‐to-­‐a-­‐Globus-­‐endpoint-­‐  

•  Recent  op;ons  for  increased  performance  –  Use  addi;onal  parallel  connec;ons  –  Distribute  transfers  across  mul;ple  DTNs  (Globus  I/O  Nodes)  –  Cri;cal  –  only  do  this  when  all  DTNs  in  the  endpoint  mount  the  same  shared  filesystem  

•  Use  the  Globus  CLI  command  endpoint-modify –  Use  the  -­‐-­‐network-­‐use  op;on  –  Adjusts  concurrency  and  parallelism  –  More  info  at  globus.org  (hkp://dev.globus.org/cli/reference/endpoint-­‐modify/)  

06.07.16  45  

Page 46: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Extra  Slides  –  Output  Queue  Discussion  

46 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 47: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Mul,ple  Ingress  Flows,  Common  Egress  

Background traffic or

competing bursts

DTN traffic with wire-speed

bursts

10GE

10GE

10GE

Hosts  will  typically  send  packets  at  the  speed  of  their  interface  (1G,  10G,  etc.)  

•  Instantaneous  rate,  not  average  rate  •  If  TCP  has  window  available  and  data  to  send,  

host  sends  un;l  there  is  either  no  data  or  no  window  

Hosts  moving  big  data  (e.g.  DTNs)  can  send  large  bursts  of  back-­‐to-­‐back  packets  

•  This  is  true  even  if  the  average  rate  as  measured  over  seconds  is  slower  (e.g.  4Gbps)  

•  On  microsecond  ;me  scales,  there  is  omen  conges;on  

•  Router  or  switch  must  queue  packets  or  drop  them  

 47 – ESnet Science Engagement ([email protected]) - 06.07.16

©  2015,  Energy  Sciences  Network  

Page 48: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Router  and  Switch  Output  Queues  

•  Interface  output  queue  allows  the  router  or  switch  to  avoid  causing  packet  loss  in  cases  of  momentary  conges;on  

•  In  network  devices,  queue  depth  (or  ‘buffer’)  is  omen  a  func;on  of  cost  –  Cheap,  fixed-­‐config  LAN  switches  (especially  in  the  10G  space)  typically  have  inadequate  buffering.    Imagine  a  10G  ‘data  center’  switch  as  the  guilty  party  

–  Cut-­‐through  or  low-­‐latency  Ethernet  switches  typically  have  inadequate  buffering  (the  whole  point  is  to  avoid  queuing!)  

•  Expensive,  chassis-­‐based  devices  are  more  likely  to  have  deep  enough  queues  –  Juniper  MX  and  Alcatel-­‐Lucent  7750  used  in  ESnet  backbone  –  Other  vendors  make  such  devices  as  well  -­‐  details  are  important  –  Thx  to  Jim:  hkp://people.ucsc.edu/~warner/buffer.html    –  This  expense  is  one  driver  for  the  Science  DMZ  architecture  –  only  deploy  the  expensive  features  where  necessary  

 

48 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 49: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Output  Queue  Drops  –  Common  Loca,ons  

10GE

1GE

10GE

1GE

10GE

1GE1GE

1GE

10GE

Site Border RouterSite Core Switch/Router

32+ cluster nodes

Wiring closet switch

Common locations of output queue drops for traffic

outbound toward the WAN

WAN

Department Core Switch

1GE1GE

1GE

WorkstationsDepartment

cluster switch

Department uplink to site core constrained by

budget or legacy equipment

Cluster data

transfer node

Common location of output queue drops for traffic inbound

from the WAN

Inbound data path

Outbound data path

Outbound data path

49 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 50: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Extra  Slides  –  Globus  Security  Map  

06.07.16  50  

Page 51: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Security  Footprint  of  a  Globus  Transfer  

Amazon AWS

100GE

10GE10GE

100GE

10GE

10GE100GE

DATA

TCP ports50000-51000

Lab1 Science DMZ

Lab1 Border Router

ESnet 100GEESnet Router

Lab2 Border Router

Lab2 Science DMZ

Lab1 DTN

DTN DTN

OrchestrationOrchestration

Lab2 DTN

ESnet Router

Lab1 DTN security

filters

Lab2 DTN security

filters

TCP ports 443,2811, 7512

TCP ports 443,2811, 7512

Logical data path

Physical data path

Logical control path

Physical control path

Lab1 DTN security filters Lab2 DTN security filters

06.07.16  51  

Page 52: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Security  Footprint  of  a  Globus  DTN  

06.07.16  52  

10GE

Amazon AWS

100GE

10GE

10GE

100GE

DATA

TCP ports50000-51000 Science DMZ

Site / Campus Border Router

World

DTN

DTN

Orchestration

Remote DTNs

DTN securityfilters

TCP ports 443,2811, 7512

DTN

DATA

Local DTN

Logical data path

Physical data path

Logical control path

Physical control path

Page 53: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Extra  Slides  –  Firewall  Internals  

53 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 54: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Thought  Experiment  

•  We’re  going  to  do  a  thought  experiment  

•  Consider  a  network  between  three  buildings  –  A,  B,  and  C  •  This  is  supposedly  a  10Gbps  network  end  to  end  (look  at  the  links  on  the  buildings)  

•  Building  A  houses  the  border  router  –  not  much  goes  on  there  except  the  external  connec;vity  

•  Lots  of  work  happens  in  building  B  –  so  much  that  the  processing  is  done  with  mul;ple  processors  to  spread  the  load  in  an  affordable  way,  and  results  are  aggregated  amer  

•  Building  C  is  where  we  branch  out  to  other  buildings  •  Every  link  between  buildings  is  10Gbps  –  this  is  a  10Gbps  network,  right???  

54 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 55: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

No,onal  10G  Network  Between  Buildings  

WAN

perfSONAR Building A

10GE 10GE

Building B

Building C

1G1G

1G1G

1G 1G1G

1G

1G1G

1G1G1G 1G1G 1G1G 1G1G

1G

10GE

Building Layout

To O

ther

Bui

ldin

gs

10GE

10GE

10GE

55 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 56: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Clearly  Not  A  10Gbps  Network  

•  If  you  look  at  the  inside  of  Building  B,  it  is  obvious  from  a  network  engineering  perspec;ve  that  this  is  not  a  10Gbps  network  –  Clearly  the  maximum  per-­‐flow  data  rate  is  1Gbps,  not  10Gbps  –  However,  if  you  convert  the  buildings  into  network  elements  while  keeping  their  internals  intact,  you  get  routers  and  firewalls  

–  What  firewall  did  the  organiza;on  buy?    What’s  inside  it?  –  Those  likle  1G  “switches”  are  firewall  processors  

•  This  parallel  firewall  architecture  has  been  in  use  for  years  –  Slower  processors  are  cheaper  –  Typically  fine  for  a  commodity  traffic  load  –  Therefore,  this  design  is  cost  compe;;ve  and  common  

56 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 57: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

No,onal  10G  Network  Between  Devices  

WAN

perfSONAR Border Router

10GE 10GE

Firewall

Internal Router

1G1G

1G1G

1G 1G1G

1G

1G1G

1G1G1G 1G1G 1G1G 1G1G

1G

10GE

Device Layout

To O

ther

Bui

ldin

gs

10GE

10GE

10GE

57 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 58: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

No,onal  Network  Logical  Diagram  

10GE

10GE

10GE

10GE

10GE10GE

Border Router

WAN

Internal Router

Border Firewall

perfSONAR

58 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 59: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

06.07.16  59  

Page 60: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

What  Is  A  Firewall?  •  Marketplace  view  

–  Specific  security  appliance,  with  “Firewall”  printed  on  the  side  –  Lots  of  protocol  awareness,  intelligence  –  Applica;on  awareness  –  User  awareness  (VPN,  specific  access  controls,  etc.)  –  Designed  for  large  concurrent  user  count,  low  per-­‐user  bandwidth  (enterprise  traffic)  

•  IT  Organiza;on  view  –  “Firewall”  appliance,  purchased  from  the  commercial  marketplace  –  The  place  in  the  network  where  security  policy  gets  applied  –  Owned  by  the  security  group,  not  by  the  networking  group  –  Primary  risk  mi;ga;on  mechanism  

•  NIST  view  (Publica;on  800-­‐41  rev.  1,  Sep.  2009)  –  “Firewalls  are  devices  or  programs  that  control  the  flow  of  network  traffic  between  networks  or  hosts  that  employ  differing  security  postures”  

–  This  is  very  general,  and  does  not  match  marketplace  view  or  IT  org.  view  

60 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 61: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

NIST  Sees  Two  Firewalls,  IT  Shop  Sees  One  

10GE

10GE

10GE

10GE

10G

Border Router

WAN

Science DMZSwitch/Router

Enterprise Border Router/Firewall

Site / CampusLAN

High performanceData Transfer Node

with high-speed storage

Per-service security policy control points

Clean, High-bandwidth

WAN path

Site / Campus access to Science

DMZ resources

perfSONAR

perfSONAR

perfSONAR

Stateful  

Stateless  

61 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network  

Page 62: The$Science$DMZ$...The$Science$DMZ$ Eli$Dart,$Network$Engineer$ ESnetScience$Engagement Lawrence$Berkeley$Naonal$Laboratory$ Moving$My$DataatHigh$Speeds$over$the$Network$$ Overview

Stateful  Inspec,on  For  Science  DMZ  Traffic?  

•  Science  DMZ  traffic  profile  –  Small  number  of  connec;ons  or  flows  –  Large  per-­‐connec;on  data  rate  (Gigabit  scale  or  higher)  –  Large  per-­‐connec;on  data  volume  (Terabyte  scale  or  higher)  

•  Stateless  firewall  –  Address/port  filtering  (which  systems  use  which  service)  –  TCP  connec;on  ini;a;on  direc;on  (ACK  flag)  

•  Stateful  firewall  adds  –  TCP  sequence  number  tracking  (but  Linux  stack  is  as  good  or  beker  compared  to  firewall  TCP  mi;ga;ons)  

–  Protocol/app  analysis  (but  not  for  the  apps  used  in  DMZ)  –  DoS  protec;on  (but  the  Science  DMZ  assets  are  filtered  already)  

62 – ESnet Science Engagement ([email protected]) - 06.07.16 ©  2015,  Energy  Sciences  Network