Network Architecture

Gary Buhrmaster

ST&E Readiness ReviewMay 14th, 2007

Work supported by U. S. Department of Energy contract DE-AC03-76SF00515

Network Philosophy

Support getting the science done (safely) The science is the thing

Simplicity (where possible) Limit vendors, technologies used Leverage existing SCCS staff expertise

Redundancy (where appropriate) SCCS is not staffed for 24/7 coverage “Throwing smart (dedicated) people at issues”

works as long as you do not throw them too often

Overview

SLAC administers globally routed network space of 134.79.0.0/16 “SLAC” address space Visitor and RAS subnets IPv6 (test) subnet

A number of internal private subnets for control systems, isolated systems, batch farms Accelerator, SSRL, IR2, SCCS

Overview

Hardware Vendors: Cisco, Nokia ~300 Layer 2 (capable) devices ~50 Layer 3 (capable) devices ~20 Enforcement (firewall/filter) devices Many devices are categorized as more than one

swouters/frankenrouters (not all swouters are used as L2/3) what is an infiniband “switch” (it has routing in it…)

Misc. appliances (WLSE (HP), EndRun) ~15 support systems (logging, monitoring, etc.)

Sun/Dell – systems managed by the systems group

Overview

Physical instantiation ~70 buildings

Some buildings have numerous switches (some none) klystron gallery, computer center, SSRL

~200 VLANS Switched network design Some buildings have multiple subnets/vlans Some vlans are in multiple buildings, some in only one

Some in only one switch router to router connections, span monitoring…

Some internally used by devices

Staffing

Network Engineering Manage/Configure/Monitor network devices Five FTEs

Network Research Primarily research activities

But operationally focused (not just blue sky), which is leveraged to support SLAC and HEP/BES activities (especially WAN performance issues)

Staffing (outside of Network group) Network Operations

Reports to SCCS Operations Physical installation/support Five FTEs

Netops also coordinate with CEF staff and contractors for some installations (cable pullers, bulk fiber installation and termination, etc.)

Staffing (outside of Network group) Security group

Responsible for overall security policies and approvals

Apply approved policies to the Cisco enforcement devices

Windows group Apply approved policies to the Checkpoint

enforcement devices Systems group

Maintain the Unix network support systems

SLAC Speak

IFZ – Internet Free Zone At least some part of every network is blocked

from offsite network access Printers, Batch nodes, Network devices, “problematic”

devices (i.e. SBCs/IOCs)

SFZ – SLAC Free Zone Some special networks (controls) are accessible

only from their local networks IR2, MCC

SLAC Speak

RouterBlock Layer 3 forward and uRPF blocking (advertise

the /32 addresses into routing table to null route device at the router(s))

EPN – “Extremely Private Network” Elevated level protections (the “PII” place)

EPN(1) (original design), EPN2 (revised design)

CANDO – Computer And Network Database in Oracle (?)

Database of record for IP addresses/systems

……………………………………………………

Big (dense) Picture

Border

MCCIR2 SSRL BSDEPN

VPN “Special”NetMgmt

Farm Netrsch

Infra Campus

Core

“Internet”IPv6

visitor

But still simplified

Drill down (Layer 3 view)

Network segmentation Enclaves

SLAC, accelerator…. Functional/Physical

research yard, visitor network, decnet Performance/Availability

batch farm, network research

IPv6 Network

Dipping a toe in the (IPv6) water its cold and lonely there

External to SLAC network One web server

was originally proposed to be named VVVVVV

ESnetBAMANrtr-ipv6

IPv6 Network

WWW

Visitor (& RAS) Network

External to SLAC network (no trust) Wireless access is only on visitor network Client only support (block servers)

ESnet

BAMAN

Visitor Network

Border Network

Border enforcement device is a filtering router ACLs block ports <1024 (except to allowed hosts), and

various special ports (X, netbus, backoriface, …)

ESnet

Stanford

CENICInternet2

BAMANBorderrouter

Infrastructure Services

Centrally administered servers Windows/Unix infrastructure services

Unix & Windows infrastructure – DNS, Kerberos, AFS, AD, file servers, web services, email, ….

IFZ and where possible Most exceptions to port < 1024 filters are to these

servers (web, email, kerberos)

SLAC Network“Nethub/IFZ/IFZ-Lite”

B050(2nd floor)

Campus

Most staff/engineers/scientists are connected to one of the “PUB” networks Legacy workgroup allocations (based on “yellow

cable”) have changed to physical location allocations (trying to avoid flat earth operations)

Campus

Campus Distribution

Access (many buildings)

Farm

Batch resources for scientific discovery Most resources are IFZ

Exceptions for external data transfer systems, and scientific login systems

Many resources are (policy (i.e. netgroup)) limited to be used only from other batch systems

Different Availability/Performance needs

SLAC Network

“Farm” networks

batch systems

Campus

BaBar / IR2

IR2 has four subnets one public general purpose subnet, one IFZ

subnet (local compute farm), one SFZ subnet (dedicated SBCs and detector subsystems) with EPICs gateway, and isolated device control

Intention is that these networks/systems can operate independently from SCCS

mcc

Farm

Accelerator (MCC)

Accelerator network has four subnets One public general purpose subnet (slclavc), two “slac free”

subnets (leb, slcc) for control systems, and one isolated subnet (pep)

Use of multi-homed controls systems (VMS) for access to isolated networks devices

Intention is that these networks/systems can operate independently from SCCS

IR2

SLAC Network

Network Management

Network monitoring and configuration management (BAM - Backup and Monitoring) SNMP (via acls on network devices) only respond

to requests from the management network hosts ACLs protect appliances/APs (bastion hosts) Systems are limited access

SLAC Network Network Management and monitoring networks

Network Research

Network Research activities Isolated to allow local experimentation

ex: tsunami multicast

Systems are maintained the same as other systems on site

Systems are limited login, sponsored users

SLAC Network Research network

SSRL

SSRL manages their own network equipment and configurations, including their own firewall implementations to protect their control and experimental systems A later presentation will discuss SSRL

BSD (EPN(1))

EPN(1) Air Gap possibility Extensive filtering Users access PeopleSoft via Citrix More details in later presentation

rtr-bsdnet

bsd-epnbsd

SLAC net

bsd-dmz

EPN2

Revised approach based on new PS arch Multiple DMZ nets (web servers), Backend nets

(app servers, DBs) In realty, collapsed firewalls

Details in later presentation

SLAC Network

DMZsBackend

VPN

VPN (GRE/IPSEC) only to official servers Windows PPTP/L2TP VPN server Discouraged (use Citrix where possible) Firewall/filters

Block RPC, NFS, CIFS except to approved servers, & NetBus, BackOriface, etc.

SLAC Network VPN Servers

“Special” subnet(letts)

A few networks specially protected due to inability to maintain the systems, or certified configurations Ex: GLAST Clean Room, PCD, HVAC Group responsible for equipment purchase, SCCS

maintains the devices/configurations

SLAC Network

SLAC Network

SLAC Network

Procedures/Policies

Device connection policy Devices need to be in CANDO

Network equipment Users are not to install switches/routers/hubs

Wireless No wireless on the SLAC networks Devices installed/coordinated by SCCS

Network protections

Dedicated subnet for network management Network devices are IFZ SNMP restricted to network management subnet

SSH on all but a few legacy devices Finally got funding to upgrade the last few

Disable ports not allocated on switches No devices on native .1q vlan WLSE used for rogue access point detection

Network protections

Restricted physical access to “core” devices (Building 050 OmniLock door access)

Routing/switching best practices no ip unreachable, BGP passwords, schedule

allocate, no source route, …. Strong working relationship with upstreams

Network Intrusion Detection

Primarily log and netflow based Central logging and analysis

“Significant” events cause paging Netflow detects many scanners (and P2P)

Collected for both internal and external traffic “scanning” detection catches (SMTP) bots in “real time”

And the occasional “special” user Extremely useful for incident analysis

Discussion?

Obligatory final slide to avoid “End of slide show” artifact