Don’t Mind the Gapvarghese/NETWORK_VERIFICATION... · 2020-06-27 · Don’t Mind the Gap: Bridging Network-wide Objectives and Device-level Conﬁgurations Ryan Beckett (Princeton,

Don’t Mind the Gap: Bridging Network-wide Objectives and Device-level Configurations

Ryan Beckett (Princeton, MSR) Ratul Mahajan (MSR) Todd Millstein (UCLA) Jitu Padhye (MSR) David Walker (Princeton)

2

50-80% of outages from configuration changes

-Juniper 2008

~60% of network downtime is caused by human error

-Yankee group 2002

Configuring Networks is Error-Prone

3

YouTube/Pakistan incident: Could something similar whackyour site?

Configuring BGP properly is key to avoidance, 'Net registry officialsays

By Carolyn Duffy Marsan Network World | Mar 10, 2008 1:00 AM PT

In light of Pakistan Telecom/YouTube incident, Internet registry official explains how you can avoid havingyour web site victimized by such an attack.

When Pakistan Telecom blocked YouTube's traffic one Sunday evening in February, the ISP created aninternational incident that wreaked havoc on the popular video site for more than two hours.

RIPE NCC, the European registry for Internet addresses, has conducted an analysis of what happenedduring Pakistan Telecom's hijacking of YouTube's traffic and the steps that YouTube took to stop theattack.

We posed some questions to RIPE NCC's Chief Scientist Daniel Karrenberg about the YouTube incident.Here's what he had to say:

How frequently do hijacking incidents like the Pakistan Telecom/YouTube incident happen?

Misconfigurations of iBGP (internal BGP, the protocol used between the routers in the sameAutonomous System) happen regularly and are usually the result of an error. One suchmisconfiguration caused the Pakistan Telecom/YouTube incident. It appears that the PakistanTelecom/YouTube incident was not an "attack" as some have labeled it, but a configuration error. (SeeColumnist Johna Till Johnson's take on the topic.)

What is significant about the YouTube incident?

Sign In | Register

�


4












Sign In | Register

�

←

5












Sign In | Register

�

←

2/5/2016 China routing snafu briefly mangles interweb • The Register

http://www.theregister.co.uk/2010/04/09/china_bgp_interweb_snafu/ 1/4

More like this

China Network SecurityChina routing snafu briefly mangles interwebCockup, not conspiracy

Bad routing information sourced from China has disrupted the internet for the second time in a fortnight.

Global BGP (Border Gateway Routing) lookup tables sucked in data from a small ISP called IDC ChinaTelecommunication, apparently accidentally broadcast by stateowned carrier ChinaTelecommunications, IDG reports. ISPs including AT&T, France Telcom, Level3, Deutsche Telekom,Qwest and Telefonica accepted illthought out traffic routes as a result of the incident.

BGP is a core routing protocol which maps options for the best available routes for traffic to flow acrossthe net. Several routing options are normally included. The China BGP incident is the internet routingequivalent of TomTom publishing routes via Shanghai for motorists looking for alternative routesbetween London and Paris.

IDC China Telecommunication published illconceived routes for between 32,000 and 37,000 networks about 10 per cent of the net instead of the normal 40 or so routes, and this information was taken asviable routing options by many service providers for about 20 minutes early on Thursday morning (UStime) after China Telecommunications republished it and before the mixup was resolved. Routers inAsia would have been more likely to adopt the false routes as potentially viable, but effects of theincident were recorded all over the world.

BGPmon.net, a BGP monitoring service, has a detailed technical writeup of the snafu, which itdescribed as a prefix hijack, here.

Although it seems they [IDC China Telecommunication] have leaked a whole table, onlyabout 10 per cent of these prefixes propagated outside of the Chinese network. Theseinclude prefixes for popular websites such as dell.com, cnn.com, www.amazon.de,www.rapidshare.com and www.geocities.jp.A large number of networks impacted this morning were actually Chinese networks. Theseinclude some popular Chinese website such as www.joy.cn , www.pconline.com.cn ,www.huanqiu.com, www.tianya.cn and www.chinaz.com

A cockup is suspected, rather than a conspiracy, at least by BGPmon.net.

Given the large number of prefixes and short interval I don’t believe this is an intentionalhijack. Most likely it’s because of configuration issue, i.e. fat fingers. But again, this is justspeculation.

The practical consequences of the screwup are still being assessed but it could have resulted indropped connections or, worse, traffic routed through unknown systems in China. The mess providesone of the clearest illustrations of the security shortcomings of BGP, a somewhat obscure butnonetheless important network protocol.

The China BGP global routing represents a rare but not unprecedented mixup in global internet trafficmanagement. For example, just two weeks ago bad routing data resulted in the redirection of Chileaninternet traffic through a DNS (Domain Name System) server in China, as explained in a detailed postmortem by internet monitoring firm Renesys here. Bad BGP routing information from Pakistan caused

,

Networks Broadband

9 Apr 2010 at 12:24, John LeydenAbout the FT

The surest investmentyou’ll make this year.

Subscribe & save 33%

0:15

The surest investmentyou'll make this yearThe FT's comprehensive coverage ofglobal business provides the insightand analysis you need to stay onestep ahead in 2016 and beyond.

Viewing

Most read

German Chancellor fireshydrogen plasma with thepush of a button

Who would code a selfdestruct feature into theirown web browser? Oh,hello, Apple

Who wants a quadcore4.2GHz, 64GB, 5TB SSD

RAID 10 … laptop?

5 0

DATA CENTER SOFTWARE NETWORKS SECURITY INFRASTRUCTURE DEVOPS BUSINESS HARDWARE SCIENCE BOOTNOTES FORUMS

Log in Sign up Cash'n'Carrion Whitepapers The Channel The Next Platform

6












Sign In | Register

�

←

2/5/2016 China routing snafu briefly mangles interweb • The Register

http://www.theregister.co.uk/2010/04/09/china_bgp_interweb_snafu/ 1/4

More like this

China Network SecurityChina routing snafu briefly mangles interwebCockup, not conspiracy

Bad routing information sourced from China has disrupted the internet for the second time in a fortnight.

Global BGP (Border Gateway Routing) lookup tables sucked in data from a small ISP called IDC ChinaTelecommunication, apparently accidentally broadcast by stateowned carrier ChinaTelecommunications, IDG reports. ISPs including AT&T, France Telcom, Level3, Deutsche Telekom,Qwest and Telefonica accepted illthought out traffic routes as a result of the incident.

BGP is a core routing protocol which maps options for the best available routes for traffic to flow acrossthe net. Several routing options are normally included. The China BGP incident is the internet routingequivalent of TomTom publishing routes via Shanghai for motorists looking for alternative routesbetween London and Paris.

IDC China Telecommunication published illconceived routes for between 32,000 and 37,000 networks about 10 per cent of the net instead of the normal 40 or so routes, and this information was taken asviable routing options by many service providers for about 20 minutes early on Thursday morning (UStime) after China Telecommunications republished it and before the mixup was resolved. Routers inAsia would have been more likely to adopt the false routes as potentially viable, but effects of theincident were recorded all over the world.

BGPmon.net, a BGP monitoring service, has a detailed technical writeup of the snafu, which itdescribed as a prefix hijack, here.

Although it seems they [IDC China Telecommunication] have leaked a whole table, onlyabout 10 per cent of these prefixes propagated outside of the Chinese network. Theseinclude prefixes for popular websites such as dell.com, cnn.com, www.amazon.de,www.rapidshare.com and www.geocities.jp.A large number of networks impacted this morning were actually Chinese networks. Theseinclude some popular Chinese website such as www.joy.cn , www.pconline.com.cn ,www.huanqiu.com, www.tianya.cn and www.chinaz.com

A cockup is suspected, rather than a conspiracy, at least by BGPmon.net.

Given the large number of prefixes and short interval I don’t believe this is an intentionalhijack. Most likely it’s because of configuration issue, i.e. fat fingers. But again, this is justspeculation.

The practical consequences of the screwup are still being assessed but it could have resulted indropped connections or, worse, traffic routed through unknown systems in China. The mess providesone of the clearest illustrations of the security shortcomings of BGP, a somewhat obscure butnonetheless important network protocol.

The China BGP global routing represents a rare but not unprecedented mixup in global internet trafficmanagement. For example, just two weeks ago bad routing data resulted in the redirection of Chileaninternet traffic through a DNS (Domain Name System) server in China, as explained in a detailed postmortem by internet monitoring firm Renesys here. Bad BGP routing information from Pakistan caused

,

Networks Broadband

9 Apr 2010 at 12:24, John LeydenAbout the FT

The surest investmentyou’ll make this year.

Subscribe & save 33%

0:15

The surest investmentyou'll make this yearThe FT's comprehensive coverage ofglobal business provides the insightand analysis you need to stay onestep ahead in 2016 and beyond.

Viewing

Most read

German Chancellor fireshydrogen plasma with thepush of a button

Who would code a selfdestruct feature into theirown web browser? Oh,hello, Apple

Who wants a quadcore4.2GHz, 64GB, 5TB SSD

RAID 10 … laptop?

5 0

DATA CENTER SOFTWARE NETWORKS SECURITY INFRASTRUCTURE DEVOPS BUSINESS HARDWARE SCIENCE BOOTNOTES FORUMS

Log in Sign up Cash'n'Carrion Whitepapers The Channel The Next Platform

2/5/2016 Internet-Wide Catastrophe—Last Year - Dyn Research | The New Home Of Renesys

http://research.dyn.com/2005/12/internetwide-nearcatastrophela/ 1/5

± ¬ Search �

2 DECEMBER 24, 2005 ç COMMENTS (0) - VIEWS: 3038

1 ENGINEERING TODD UNDERWOOD

Internet‐WideCatastrophe—Last Year

One year ago today TTNet in Turkey (AS9121) pretended to be the entireInternet. And unfortunately for the rest of the Internet, many largenetwork providers believed them (or at least believed them in part). Asfar as anyone knows, it was a mistake, not a malicious act. But theconsequences were far from benign: for several hours a large number ofInternet users were unable to reach a large number of Internet sites.Twelve months later we can take a look at what happened, and whetherwe’ve learned much in the intervening time.

Early Christmas Eve morning 2004, TTNet (AS9121) started announcing

The NewThreat:Targeted

Internet Traffic Misdirection

NOVEMBER 19, 2013

EgyptLeaves theInternet

JANUARY 27, 2011

InternetTouchesHalf MillionRoutes:Outages

Possible Next Week

AUGUST 13, 2014

Authors ArchivesPopular

HOME TOPICS PRESENTATIONS ABOUT OUTAGES DYNCONTENTHUB

« Previous Story Next Story »

Fundamental Tradeoff?

7

Distributed Centralized

Distributed

Centralized

ControlMechanism

Configuration


8


Distributed

Centralized

ControlMechanism

OSPF

RIP

BGP

ScalabilityRobustnessComplexity

Configuration


9


Distributed

Centralized

ControlMechanism

OSPF

RIP

BGP


Configuration

100,000s oflines of config


10


Distributed

Centralized

ControlMechanism

OSPF

RIP

BGP

SDNScalabilityRobustnessComplexity


Configuration


11

Ideal


Distributed

Centralized

ControlMechanism

OSPF

RIP

BGP

SDNScalabilityRobustnessComplexity


Configuration

Propane Overview

12

Propane

Topology

BGP ConfigsCompiler

0,0

2,1

1,1 0,1

Propane System

13

AS 1AS 2

AS 3

Uniform abstractions for intra- and inter-domain routing

I) Language for expressing network-wide objectives with:

Path constraints and preferences in case of failures

14

Propane System

2) Compiler for a purely distributed implementation

Policy

15

Propane System

Generate BGP configs for each router

Compiler guarantees policy-compliance for all failures

2) Compiler for a purely distributed implementation

BGP

BGP

BGP

BGP

Example: A DC network with traditional configs

16

GlobalPrefixes

LocalPrefixes

GP1 GP2 LP1 LP2

Peer2

YX

DC

A B E F

HG

Peer1

Y

C D

A B

G H

E F

Goals• Local prefixes reachable only internally

• Global prefixes reachable externally

• Aggregate global prefixes as GP

• Prefer leaving through Peer1 over Peer2

• Prevent transit traffic between peers

Configuration Attempt

• Don’t export from G, H to external

• Aggregate externally as GP







GlobalPrefixes

LocalPrefixes

GP1 GP2 LP1 LP2

Peer2

YX

DC

A B E F

HG

Peer1

Y

C D

A B

G H

E F

17

GP

18

GlobalPrefixes

LocalPrefixes

GP1 GP2 LP1 LP2

Peer2

YX

DC

A B E F

HG

Peer1

Y

C D

A B

G H

E F

GP










19




• X,Y block routes through each other







GlobalPrefixes

LocalPrefixes

GP1 GP2 LP1 LP2

Peer2

YX

DC

A B E F

HG

Peer1

Y

C D

A B

G H

E F

20

GlobalPrefixes

LocalPrefixes

GP1 GP2 LP1 LP2

Peer2

YX

DC

A B E F

HG

Peer1

Y

C D

A B

G H

E F

GP GP











21

GlobalPrefixes

LocalPrefixes

GP1 GP2 LP1 LP2

Peer2

YX

DC

A B E F

HG

Peer1

Y

C D

A B

G H

E F







GP1 traffic dropped GP





22Aggregation-Induced Black Hole!


GlobalPrefixes

LocalPrefixes

GP1 GP2 LP1 LP2

Peer2

YX

DC

A B E F

HG

Peer1

Y

C D

A B

G H

E F

GP










GP1 traffic dropped

23







GlobalPrefixes

LocalPrefixes

GP1 GP2 LP1 LP2

Peer2

YX

DC

A B E F

HG

Peer1

Y

C D

A B

G H

E F

24

Example: A DC network with Propane

define Destination = {GP1 => end(A) GP2 => end(B) LP1 => end(E) LP2 => end(F) true => exit(Peer1 >> Peer2)}

GlobalPrefixes

LocalPrefixes

GP1 GP2 LP1 LP2

Peer2

YX

DC

A B E F

HG

Peer1

Y

C D

A B

G H

E F

25


define Locality = {LP1 | LP2 => internal}


GlobalPrefixes

LocalPrefixes

GP1 GP2 LP1 LP2

Peer2

YX

DC

A B E F

HG

Peer1

Y

C D

A B

G H

E F

26



define transit(X,Y) = enter(X|Y) and exit(X|Y)


GlobalPrefixes

LocalPrefixes

GP1 GP2 LP1 LP2

Peer2

YX

DC

A B E F

HG

Peer1

Y

C D

A B

G H

E F

27




define NoTransit = {true => !transit(Peer1,Peer2)}


GlobalPrefixes

LocalPrefixes

GP1 GP2 LP1 LP2

Peer2

YX

DC

A B E F

HG

Peer1

Y

C D

A B

G H

E F

28




define NoTransit = {true => !transit(Peer1,Peer2)}

define Main = Destination & Locality & NoTransit & agg(GP, in -> out)


GlobalPrefixes

LocalPrefixes

GP1 GP2 LP1 LP2

Peer2

YX

DC

A B E F

HG

Peer1

Y

C D

A B

G H

E F

Compilation

Propane Compiler

Topology

BGP Configs

29

Compilation

Propane Compiler

Topology

BGP Configs

30

0 1 2

0 1

State Machines

Constraints on Policy

Compilation

Propane Compiler

Topology

BGP Configs

31

0,0

2,1

1,1 0,10 1 2

0 1

Product GraphState Machines

Jointly analyzewith topology

Compilation: A simple Example

32

Y

A

C

D

EB

X

end(Y) & (path(A,C,D) >> any)


33

Y

A

C

D

EB

X

Convert to Regex

XACDY >> (Σ*)Y


Reversed Automata from Policies

34

Policy:

1. XACDY

2. (Σ*)Y

Y

A

C

D

EB

X

More preferred paths

Less preferred paths


35

0 1 2 3 4 5Y D C A X

0Y

Y

A

C

D

EB

X

Σ

1

Policy:

1. XACDY

2. (Σ*)Y


36

0 1 2 3 4 5Y D C A X

Reversed automatatracks BGP message flow

Y

A

C

D

EB

X

0Y

Σ

1

Policy:

1. XACDY

2. (Σ*)Y

Constructing the Product Graph (PG)

37

0 1 2 3 4 5Y D C A X

0Y

Σ

1

{1,2}

(D,2,1)

(C,3,1)

(A,4,1)

(X, -, 1) (X,5,1)

(Y,1,1)

(-,0,0)

(D, -,1)

(E, -,1)

(B, -,1)

{2}

(C, -,1)

(A, -,1)

YA

C

D

EB

X


38

0 1 2 3 4 5Y D C A X

0Y

Σ

1

(X,5,1)

TopologyLocation

AutomataStatesY

A

C

D

EB

X


39

0 1 2 3 4 5Y D C A X

0Y

Σ

1

(X,5,1)

TopologyLocation

AutomataStatesY

A

C

D

EB

X

{1,2}

Path preferences


40

0 1 2 3 4 5Y D C A X

0Y

Σ

1

(Y,1,1)

(-,0,0)

YA

C

D

EB

X


41

0 1 2 3 4 5Y D C A X

0Y

Σ

1

(D,2,1)

(Y,1,1)

(-,0,0)

YA

C

D

EB

X


42

{1,2}

(D,2,1)

(C,3,1)

(A,4,1)

(X, -, 1) (X,5,1)

(Y,1,1)

(-,0,0)

(D, -,1)

(E, -,1)

(B, -,1)

{2}

(C, -,1)

(A, -,1)

Graph capturing all possible policy-compliant paths through the topology

YA

C

D

EB

X


43

0 1 2 3 4 5Y D C A X

0Y

Σ

1

{1,2}

(D,2,1)

(C,3,1)

(A,4,1)

(X, -, 1) (X,5,1)

(Y,1,1)

(-,0,0)

(D, -,1)

(E, -,1)

(B, -,1)

{2}

(C, -,1)

(A, -,1)

Preferences

YA

C

D

EB

X


44

0 1 2 3 4 5Y D C A X

0Y

Σ

1

{1,2}

(D,2,1)

(C,3,1)

(A,4,1)

(X, -, 1) (X,5,1)

(Y,1,1)

(-,0,0)

(D, -,1)

(E, -,1)

(B, -,1)

{2}

(C, -,1)

(A, -,1)

Preferences

YA

C

D

EB

X


45

0 1 2 3 4 5Y D C A X

0Y

Σ

1

{1,2}

(D,2,1)

(C,3,1)

(A,4,1)

(X, -, 1) (X,5,1)

(Y,1,1)

(-,0,0)

(D, -,1)

(E, -,1)

(B, -,1)

{2}

(C, -,1)

(A, -,1)

Y D C A XAccept: Preferences

YA

C

D

EB

X


46

0 1 2 3 4 5Y D C A X

0Y

Σ

1

{1,2}

(D,2,1)

(C,3,1)

(A,4,1)

(X, -, 1) (X,5,1)

(Y,1,1)

(-,0,0)

(D, -,1)

(E, -,1)

(B, -,1)

{2}

(C, -,1)

(A, -,1)

YA

C

D

EB

X

Compilation to BGP:

47

Idea 1: Restrict advertisements to edges• Encode state in a BGP community tag• Incoming edges — import filters• Outgoing edges — export filters

Let BGP find some allowed path dynamically

(D,2,1)

(C,3,1)

(A,4,1)

(X, -, 1) (X,5,1)

(Y,1,1)

(-,0,0)

(D, -,1)

(E, -,1)

(B, -,1)

{2} {1,2}

(C, -,1)

(A, -,1)

Compilation to BGP:

48

D allows import matching regex(Y)

(D,2,1)

(C,3,1)

(A,4,1)

(X, -, 1) (X,5,1)

(Y,1,1)

(-,0,0)

(D, -,1)

(E, -,1)

(B, -,1)

{2} {1,2}

(C, -,1)

(A, -,1)

YA

C

D

EB

X

Compilation to BGP:

49

D exports to C with tag (2,1)

(D,2,1)

(C,3,1)

(A,4,1)

(X, -, 1) (X,5,1)

(Y,1,1)

(-,0,0)

(D, -,1)

(E, -,1)

(B, -,1)

{2} {1,2}

(C, -,1)

(A, -,1)

YA

C

D

EB

X

Compilation to BGP:

50

C allows import from D with tag (2,1)

(D,2,1)

(C,3,1)

(A,4,1)

(X, -, 1) (X,5,1)

(Y,1,1)

(-,0,0)

(D, -,1)

(E, -,1)

(B, -,1)

{2} {1,2}

(C, -,1)

(A, -,1)

YA

C

D

EB

X

Compilation to BGP:

51

C exports to A,B,D,E with tag (3,1)

(D,2,1)

(C,3,1)

(A,4,1)

(X, -, 1) (X,5,1)

(Y,1,1)

(-,0,0)

(D, -,1)

(E, -,1)

(B, -,1)

{2} {1,2}

(C, -,1)

(A, -,1)

YA

C

D

EB

X

Compilation to BGP:

52

Idea 2: Find preferences

Let BGP find the best path dynamically

• Direct BGP towards best path• Under all combinations of failures

(D,2,1)

(C,3,1)

(A,4,1)

(X, -, 1) (X,5,1)

(Y,1,1)

start

(D, -,1)

(E, -,1)

(B, -,1)

{2} {1,2}

(C, -,1)

(A, -,1)

Compilation to BGP:

53

(D,2,1)

(C,3,1)

(A,4,1)

(X, -, 1) (X,5,1)

(Y,1,1)

start

(D, -,1)

(E, -,1)

(B, -,1)

{2} {1,2}

(C, -,1)

(A, -,1)

Router Cmatch peer = D …

match peer = E …

Compilation to BGP:

54

(D,2,1)

(C,3,1)

(A,4,1)

(X, -, 1) (X,5,1)

(Y,1,1)

start

(D, -,1)

(E, -,1)

(B, -,1)

{2} {1,2}

(C, -,1)

(A, -,1)

??????


match peer = E …local-pref ← ???

local-pref ← ???

Compilation to BGP:

55

(D,2,1)

(C,3,1)

(A,4,1)

(X, -, 1) (X,5,1)

(Y,1,1)

start

(D, -,1)

(E, -,1)

(B, -,1)

{2} {1,2}

(C, -,1)

(A, -,1)

Efficient algorithm to assign preferences that forces BGP to find the best paths for all possible failures

See the paper for details!


match peer = E …local-pref ← ???

local-pref ← ???

Compilation to BGP:

56

Less preferredimport

More preferredimport

Implementation: Local preference(D,2,1)

(C,3,1)

(A,4,1)

(X, -, 1) (X,5,1)

(Y,1,1)

start

(D, -,1)

(E, -,1)

(B, -,1)

{2} {1,2}

(C, -,1)

(A, -,1)

Compilation to BGP:

57

Less preferredimport

More preferredimport

Implementation: MED/Prepending(D,2,1)

(C,3,1)

(A,4,1)

(X, -, 1) (X,5,1)

(Y,1,1)

start

(D, -,1)

(E, -,1)

(B, -,1)

{2} {1,2}

(C, -,1)

(A, -,1)


58


AS 100

Y

A

C

D

EB

X

lp←100

lp←101

MED←80

MED←81

filter(Y)

filter(Y)

Implementation

Written in 7000 lines of F#

Generates Quagga configurations

A number of other analyses & features

59

Benchmarks

60

Configurations from a large cloud provider

Policy described in English documents

Datacenter and Backbone policies

Policy Size

61

Without prefix/peer definitions

Datacenter policy: 31 lines of Propane

Backbone policy: 43 lines of Propane

Conventional BGP configurations are 1000s of lines

Compilation Time

62

Data center (< 9 min) Backbone (< 3 min)

Compile for each prefix equivalence class

Compile for each equivalence class in parallel

8 core, 3.6 GHz Intel Xeon processor

Configuration Size

63

Reuse community values across peers

Merge import/export behaviors across peers

Optimizations yield 50-100x decrease in config size

Avoid using community tags when unambiguous

Optimizations

Results

Configurations ~1000-10000 lines per router

64

Propane: Summary

Centralized network programmability

Distributed implementation via BGP

Uniform abstractions for Inter- and Intra-domain routing

High-level language

Compiler

Static analysis guarantees policy compliance for all failures

Scales to reasonably sized network topologies

Core policy in 30-50 lines of Propane vs. 1000s

Constraints specify preferred paths and backups in case of failure

Documents

Don’t Mind the Gapvarghese/NETWORK_VERIFICATION... · 2020-06-27 · Don’t Mind the Gap: Bridging Network-wide Objectives and Device-level Conﬁgurations Ryan Beckett (Princeton,