1

DEMYSTIFYING CONFIGURATION CHALLENGES AND TRADE-OFFS IN NETWORK-BASED ISP SERVICES

Theophilus Benson*, Aditya Akella*, Aman Shaikh+

*University of Wisconsin, Madison+ ATT Labs Research

2

Adoption of Network-based Services

ISP Core

3

Adoption of Network-based Services

“By 2015, annual global IP traffic will reach 966 exabytes” [Cisco’11]

Customers adopting new applications ISPs upgrade and purchase new equipment Best effort ineffective for some applications

2010 2011 2012 2013 2014 20150E+00

2E+05

4E+05

6E+05

8E+05

1E+06

Time (in Years)

Size

of I

nter

net T

raffi

c(In

PB

)

4

Adoption of Network-based Services

Services are a crucial part of the Internet’s ecosystem

ISP Core

Host-based and network-based services “AT&T to spend $1 billion to ramp up enterprise services” Recover cost and improve application performance

5

Goals

Vision: Improve service integration/upgrades Simplify service management

Understand impediments Service configuration files Complexity of configuration

EdgeEdgeISP Core

6

Configuration is a crucial component

Configuration determines Customer functionality ISP interactions

Configuration is complex Most time consuming task [Feamster ‘05 ] Most error-prone

> 50% customer problems due to configuration errors [Yankee Group ‘04]

OSPF

BGP

7

Configuring a Service

PECEPE CE

acz

ISP Core

Control Plane

Data plane

acz

acz

8

Contributions

Analyzed 2.5 years of configurations Show how complexity evolves over time

Worsens over time Highlight the location of complexity

Complexity exists at the edge Identify the cause of complexity

Due to provisioning of new customers

PEPE

ISP CoreCE

EdgeCustomersEdge

Customers

9

Contributions

Identified potential ways to mitigate complexity Showed the impact of design choices on complexity

Vendor

Com

plex

ity

#1 #2 Routing Design

Com

plex

ity#1 #2 #3

10

Outline

Motivation Background Models and Data-Set Understanding Complexity Mitigating Complexity Conclusion

12

Configuring the Provider’s Edge

Complexity is due to: Dependent commands

PECEPE CE

ISP Core

Ip vrf blueRd 23234:100223Route-target import 1000:1Route-target export 1000:1!Interface ethernet1Ip address 128.105.82.66/30Ip vrf forwarding blueServices-policy output policy1!Policy-map policy1Policy 100 20 confirm-action transmit

VRF

Interface

Policy-map

a

13

Configuring the Control-Plane Core

PECEPE CE

Router bgp 65000Neighbor 129.168.6.6Neighbor 129.168.2.2!

Ip vrf blueRd 23234:100223Route-target import 1000:1Route-target export 1000:1!Router bgp 65000Neighbor 129.168.2.1Neighbor 129.168.2.1

ISP Core

acz

acz

acz

Complexity is due to: Dependent commands Maintaining consistency

14

Configuring the Data-Plane Core

PECEPE CE

acz

acz

Interface gigethernet1Ip address 128.105.82.66/30!Router ospf 2Network 128.105.82.0/24!

Ip vrf blueRd 23234:100223Route-target import 1000:1Route-target export 1000:1!Interface gigethernet1Ip address 128.105.82.65/30!

ISP Corea

Complexity is due to: Dependent commands Maintaining consistency

acz

15

Models and Data-Set

Requirements for Data Models

Quantify complexity of configuration Capture dependencies between commands Capture consistency across devices

Use complexity metrics [Benson ‘09] Motivated by software engineering techniques

Abstract away low level details Abstract groups of commands stanzas

16

Ip vrf blueRd 23234:100223!Interface ethernet1Ip address 128.105.82.66Ip vrf forwarding blueServices-policy input policy2!

17

Data Models

Referential Graph [Benson ‘09] Syntactic dependencies operators must track Network graph of dependent stanzas Metric: size of graph Larger graph more dependencies

Templates [Benson ‘09] Clone detection used to capture uniformity

VRF

Interface

Policy-map

18

Data-Set

Service % of Routers ( PE + Core)VPN 48%VPLS 27%VoIP 5%DDoS Prev. 31%Virtual Wire 25%

Diversity allows for a comprehensive study

Collected data from tier-1 ISP for 2.5 years 5 services: VPN, VPLS, VoIP, DDoS Prev., Virtual Wire Daily snapshots of router configuration files Metadata (per router): vendor, role and location

19

Understanding Complexity Provider Edge (PE) Complexity Control-Plane Core Complexity Data-Plane Core Complexity

PEPE

ISP CoreCE

20

Understanding Complexity Provider Edge (PE) Complexity Control-Plane Core Complexity Data-Plane Core Complexity

PEPEISP Core CE

21

PE Complexity over Time

Growth is due to worsening complexity New devices have less dependencies

Over time, configuration tasks become tricky

Dec 08

Dec 10

0E+0

2E+5

4E+5

6E+5

8E+5

1E+6

DDoS V-Wire VplsVPN VoIP

Time (in Months)

Tota

l # o

f Ref

. Lin

ks

Dec 08

Dec 10

0

500

1000

1500

2000

2500

DDoS V-Wire VplsVPN VoIP

Time (in Months)

Max

PE

Gra

ph S

ize

22

Understanding PE Complexity Which stanza contributes to VPLS growth?

Edge EdgeISP Core

VRF

Interface

Policy-map

Dec 08

Dec 10

0%20%40%60%80%

100%

Interface CoS VRF

Time (in Months)

Perc

enta

ge o

f Sta

nzas

Dec 08

Dec 10

0%20%40%60%80%

100%

Interface CoS VRF

Time (in Months)

Perc

enta

ge o

f Sta

nzas

VRF

23

Understanding PE Complexity Which stanza contributes to VPN growth?

Complexity caused by customer provisioning

Edge EdgeISP Core

VRF

Interface

Policy-map

Dec 08

Dec 10

0%20%40%60%80%

100%

Interface CoS VRF

Time (in Months)

Perc

enta

ge o

f Sta

nzas

Dec 08

Dec 10

0%20%40%60%80%

100%

Interface CoS VRF

Time (in Months)

Perc

enta

ge o

f Sta

nzas

26

Configuration Reuse over Time

Specialization leads to added complexity

• Reuse and specialization exists• Configuration overlap reduces over time

– Reduction due to specialized usage of service

0% 20% 40% 60% 80% 100%0

0.20.40.60.8

1

VPN 2008

Percentage of Reuse

CD

F of

Cus

tom

ers

0% 20% 40% 60% 80% 100%0

0.20.40.60.8

1

VPN 2008 VPN 2010

Percentage of Reuse

CD

F of

Cus

tom

ers

71%

62%88%

27

Understanding Complexity

Data-Plane Core: service-agnostic and simple Control-Plane Core: distinct across services

Growing number of adjacencies with PEs PE is the most complex

PEPE

ISP CoreCE

28

Mitigating Complexity Vendor Selection

Cost

Func

tiona

lity

Complexity

Time

Com

plex

ity

TimeC

ompl

exity

29

How to Compare Vendors

Different vendors different languages Language impacts complexity Difference in structure of functionality

Comparing vendor languages Configurations representing same policy Same customer same policy on all PEs

PECEPE CE

ISP CoreVendor1 Vendor2

30

Vendor Selection

Graph for vendor1 is consistently larger Vendor1 requires more stanzas for same policies Operators need to track more dependencies

Choice of vendor can reduce PE complexity

1 2 3 4 5 6 70.000.501.001.502.002.503.003.504.004.50

Anonymized Customer ID

Rat

io o

f Ref

eren

tial g

raph

(Ven

dor1

/Ven

dor2

)

1 2 3 4 5 6 70.000.501.001.502.002.503.003.504.004.50

Anonymized Customer ID

Rat

io o

f Ref

eren

tial g

raph

(Ven

dor1

/Ven

dor2

)

31

Conclusion

Studied the factors that impede services Complexity grows over time

Modifications become time consuming Most complexity lies in configuring customers

Varying requirements and specialized configuration Framework to systematically consider complexity

Choice of vendor can reduce complexity

32

Thank You

Theophilus Benson (tbenson@cs.wisc.edu)

"Complex systems are built out of a myriad of simple components"

33

34

35

Related Works

Customer usage patterns Traffic matrixes [Garimella’07,Qiu’09,] Control plane state [Raghunath’05]

Techniques for optimizing services Improve hardware usage [Kim’08,Raghunath’07]

Very little on configuration Modeling Class-of-Service [Sung’08] Root cause analysis [Mahimkar’10, Turner’10] Complementary to understanding complexity

PECEPE CEISP Core

36

Future Works

Automated simplification of configuration Utilizing insights on design choices

New abstraction layer Policy language for simpler configuration Translates language to configuration

Correlate complexity with manageability

37

38

Customer Provisioning

Customer provisioning causes complexity

Provisioning should slightly increase complexity Similar functionality for all customers

Small change in configurations Modular configuration languages

Reuse of configurations between customers

Dec 08

Dec 10

0E+0

2E+5

4E+5

6E+5

8E+5

1E+6

DDoS V-Wire VplsVPN VoIP

Time (in Months)

Tota

l # o

f Ref

. Lin

ks

39

Similarities Across Customers

Customers are not created equally VPLS: 2 types of customers VPN: Multiple types of customers

Groups of customers use services differently

Differences in requirements introduces complexity

0 5 10 15 20 250

0.2

0.4

0.6

0.8

1

VPLS VPN

Size of Referential Graph

CD

F of

Cus

tom

ers