University of WashingtonComputing & Communications UW Medicine Networking Update Terry Gray Associate Vice President, IT Infrastructure University of Washington

University of Washington Computing & Communications

UW Medicine Networking Update

Terry GrayAssociate Vice President, IT Infrastructure

University of Washington

16 April 2004


Key Elements of the Partnership

Changed: C&C now responsible for... In-building network implementation and

operational support for med ctrs, clinics Med center network design “for real”

Not Changed: C&C still responsible for... Network backbone, routers Regional and Internet connectivity SoM and Health Sciences networking


Why the Partnership Makes Sense Consistency, interoperability, manageability Leverage C&C networking expertise Clinical/research hi-performance network needs 24x7 Network Operations Center (NOC) Advanced network management tools Avoid design/build organizational conflicts Beyond the network...

hope to share distributed system architecture and network computing expertise


Near-term Progress and Plans Created “Top 10” list --now up to Top 20 :) Agreement on standard maintenance window Static addressing work-around (sDHCP) FDDI, VLAN elimination Subnet splits/upgrades (1500 computers) Equipment upgrades Router consolidation, dedicated subnets, separate med

center backbone Equipment, outlet location database updates Initial wireless deployment NetVersant and Cisco external studies


The Challenge

Create a network computing environment

– with excellent security

– excellent supportability

– that users find reliable and responsive


Context: A Perfect Storm Increased dependency on network apps Decreased tolerance for outages Decades of deferred maintenance... Inadequate infrastructure investment Some old/unfortunate design decisions Some fragile applications Fragmented host management Increasingly hostile security environment Increasing legal/regulatory liability Increasing importance of research/clinical leverage


Context: Some Numbers

UW Total(incl UWMedicine)

HealthSciences(incl SoM)

MedicalCenters

Subnets 1022 52 145

Devices 75,000 >8,000 10,000


Network Device Growth

Note: Most dips reflect lower summer use; last one is a measurement anomaly


Network Traffic Growth (linear)


Network Traffic Growth (log)


System Elements

Environmentals (Power, A/C, Physical Security) Network Client Workstations Servers Applications Personnel, Procedures, Policy, and Architecture

Failures at one level can trigger problems at another level; need Total System perspective


Systemic Network Problems(some of these go back decades)

Old infrastructure (e.g cat 3 wire) Non-supportable technologies (e.g. FDDI) Non-supportable (non-geographic) topology Expensive shortcuts (e.g. cat5 mis-terminated) Security based on individual IP addresses Subnets with clients and critical servers Documentation deficiency

Contact database Device location database Critical device registry


Systemic General Problems Ever-increasing system complexity, dependencies Ever-increasing threats, liabilities Departmental autonomy Un-controlled hosts Un-reliable power and A/C in equipment rooms No net-oriented application procurement standards

Are HA and DRBR expectations realistic? Are backup plans workable?


Key Operational Objectives

• simplicity– lower cost– higher MTBF (modulo redundancy)– lower MTTR (quicker diagnosis)

• consistency– deterministic outlet behavior (Network Utility Model)– connection transparency (open/deterministic Internet)– easier problem diagnosis

• These objectives conflict with other goals


Design Tradeoffs

Networks = Connectivity; Security = Isolation Fault Zone size vs. Economy/Simplicity Reliability vs. Complexity Prevention vs. (Fast) Remediation Security vs. Supportability vs. Functionality

Differences in NetSec approaches relate to: Balancing priorities (security vs. ops vs. function) Local technical and institutional feasibility


Tradeoff Examples• Defense-in-depth conjecture (for N layers)

– Security: MTTE (exploit) N**2

– Functionality: MTTI (innovation) N**2

– Supportability: MTTR (repair) N**2

• Perimeter Protection Paradox (for D devices)– Firewall value/efficiency D– Firewall effectiveness 1 / D

• Border blocking criteria– Threat can’t reasonably be addressed at edge– Won’t harm network (performance, stateless block)– Widespread consensus to do it

• Security by IP address


Network Security Chronology• 1990: Five anti-interoperable networks• 1994: Nebula shows network utility model viable• 1998: Defined border blocking policy• 2000: Published Network Security Credo• 2000: Added source address spoof filters• 2000: Proposed med ctr network zone• 2000: Proposed server sanctuaries• 2001: Ban clear-text passwords on C&C systems• 2001: Proposed pervasive host firewalls• 2001: Developed logical firewall solution• 2002: Developed Project-172 solution• 2003: Slammer, Blaster… death of the Internet• 2003: Developed flex-net architecture


Next-Gen Network Architecture Parallel networks; more redundancy Supportable (geographic) topology Med center subnets = separate backbone zone Perimeter, sanctuary, and end-point defense Higher performance High-availability strategies

Workstations spread across independent nets Redundant routers Dual-homed servers


Success Metrics Tom’s

Nobody gets hurt Nobody goes to jail

Steve’s Four Nines or bust! High ROI (Return On Investment)

Terry’s Low ROI (Risk Of Interruption) Low MTTR (Quick to Fix) High predictability (No surprises)


Lessons Net reliability & host security are inextricably linked Five 9s is hard (unless we only attach phones?) $ for $, best security investment is central host management Nebula existence proof: security in an open network Watch out for unfair cost shifting The cost of static IP configuration is very high Controlling net access is hard --hublets, wireless Even host firewalls don’t guarantee safety Perimeter firewalls may increase user confusion, MTTR It only takes one compromise inside to defeat a firewall Next-generation threats: firewalls won’t help Even so… defense-in-depth is a Good Thing


Questions? Comments?


Network Security Addendum


Recent Events• attacks

– slammer (Jan 2003)– blaster (Aug 2003)– sobig (Sep 2003)– mydoom (Feb 2004)– witty (Mar 2004)

• impact– demise of the open/transparent/deterministic Internet– demise of the network utility model– demise of the unmanaged/autonomous PC– demise of reliable email


Seven Security Axioms1. Network security is maximized

when we assume there is no such thing.2. Large security perimeters mean large vulnerability zones.3. Firewalls are such a good idea,

every computer should have one. Seriously.4. Remote access is fraught with peril, just like local access.5. One person's security perimeter is another's broken network.6. Private networks won't help (Limits of isolation).7. Network security is about psychology as well as technology.


Network Security Credo

• Focus first on the edge(Perimeter Protection Paradox)

• Add defense-in-depth as needed

• Keep it simple (e.g. Network Utility Model)

• But not too simple (e.g. offer some policy choice)

• Avoid – one-size-fits-all policies– cost-shifting from “guilty” to “innocent”– confusing users and techs (“broken by design”)


Preserving the Net Utility Model• What is it?• Why important?• Incompatible with perimeter security?• Too late to save?• NUM-preserving perimeter defense

– Logical Firewalls– Project 172

• Foiled by static IP addressing…– Requires all hosts be reconfigured


Conflicting Perspectives• System administrator view

– some prefer local control/responsibility– some prefer central/big-perimeter defense– some underestimate cost impact on others

• User view– want just enough openness to run apps– prefer “unlisted numbers”?

• Network operator view– concerned about increased support costs and repair times

due to growing complexity and unpredictability– concerned about loss of network functionality


Generic Security Toolkit• host choice: truly thin clients; species diversity • host configuration management• conventional firewalls• logical firewalls• private addressing (e.g. project 172)• IDS, IPS, ADS• vulnerability scanning, anti-virus tools• QoS (to protect critical traffic types)• isolated networks (physical, VLAN, VPN)• non-technical: policies, education, staff


Lines of Defense

• network isolation for critical services• host integrity (Make the OS net-safe)• host perimeter (integral ACLs/firewalling)• cluster/lab perimeter (sanctuary, FW, LFW)• network zone perimeter (P172, FW)• real-time attack detection and containment• user education


Perimeter Firewalls• increase time-to-infection• increase time-to-repair • provide defense-in-depth• may look like a broken network to users• are defeated by a single hacked host • are defeated by tunneling/encryption• often give a false sense of security• encourage backdoors• may be a performance bottleneck• may inhibit legitimate activities, innovation• create a vulnerability zone that is hard to protect:

– vpns, laptops, wifi, usb drives, social engr attacks– the more you depend on perimeter defense, the more you must invest

in defending the perimeter


Operational Impact by firewall type

• host -- best case; user interaction w/FW possible

• cluster -- no impact on net diagnosis “beyond”

• logical -- low impact on basic net diagnosis

• subnet -- impacts almost all diagnosis

• zone -- impacts inter-zone diagnosis• border --impacts inter-enterprise diagnosis

NB: cost of maintaining firewall config depends on who is doing it, and how many rules/exceptions there are.


Limits of Isolation: attack gateways

hosts connected to two different networks can become attack gateways between the two

example: home PCs with VPN connection to protected network

safer remote access: SSH, SSL, K5, RDP, SSL VPNs


Med Center Zone Perimeter• purpose

– time to defend against zero-day events– protect the otherwise unprotected– defense-in-depth– reduced annoyance/noise traffic– DOS attack mitigation

• options– conventional inline firewall– private addressing + NAT or proxies– both


Protecting Non-fixable Devices

FDA-approved devices, printers, etc protection options (besides zone perimeter):

private addressing individual firewall, VPN, or NAT box ($25 - $2500)

--depending on performance requirements cluster/lab perimeter firewalls logical firewalls


NOC view of Firewall ApproachesEPFW = End-Point FirewallLFW = Logical Firewall w/masquerading NATSFW = Subnet FirewallBZFW = Border or Zone FirewallP172 = Project 172-phase III (Private addresses with NAT)

IDEAL EPFW LFW P172 SFW BZFW

Policy Enforcement Point? Host Host Subnet Zone Subnet Zone

Requires host reconfigure? No Yes Yes Yes No No

Requires network reconfig? No No No No Yes Yes

Destroys E2E transparency? No No No No Yes Yes Assured NOC access to switches? Yes Yes Yes Yes No* No* User sees why app failed? Yes Yes No No No No NOC-Predictable semantics? Yes No No Yes No No Inherent "unlisted number"? - No Yes Yes No No "unlisted number" possible? Yes Yes Yes Yes Yes Yes Adverse impact on internal network troubleshooting: Low Low Med Med High Low Adverse impact on external network troubleshooting: Low Low Med Med High High

Size of vulnerability zone: Small Small Med Large Med Large

* Can be mitigated by proper access lists and/or OOB connectivity


Network Security Trends

High

Low password guessingself-replicating code

password crackingexploiting known vulnerabilities

disabling auditsback doors

hijacking sessions

sniffers

packet spoofing

automated probes/scans

denial of service

www attacks

Att

ack

S

oph

isti

cati

on“stealth” /

advanced scanning techniques

burglaries

DDOS attacks

Source:

1980 1985 1990 1995 2000

Blendedattacks


Impact of Recent Security Events• more perimeter firewalls (demise of open Internet, NUM)• more VPNs• more tunneling (“firewall friendly” apps)• more encryption (thanks to RIAA)• more collateral damage (from attacks & remedies)• worse MTTR (complexity, broken tools)• constrained innovation (e.g. p2p, voip)• cost shifted from “guilty” to “innocent”• pressure to fix computer security problems in network• pressure for private nets• pressure to make network topology match org boundaries• blaster: triggered more perimeter defense, but showed

weakness of conventional perimeter defense

Documents

University of WashingtonComputing & Communications UW Medicine Networking Update Terry Gray Associate Vice President, IT Infrastructure University of Washington