A Measurement-based Deployment Proposal for IP Anycasthitesh/talks/talk-imc06-anymeasure.pdf ·...

A Measurement-based Deployment

Proposal for IP Anycast

Hitesh Ballani (Cornell University)Paul Francis (Cornell University)

Sylvia Ratnasamy (Intel-Research)

IMC 2006

What is IP Anycast?

IP Address IP Address 2.1.1.12.1.1.1

AnycastServer

NetworkDomains

Same IP Address assigned to the servers

making the anycast group

One-to-Any communicationwith no changes to Internet routing and clients

What is IP Anycast?

IP Address IP Address 2.1.1.12.1.1.1

AnycastServer

NetworkDomains

to2.1.1.1

CC Client

One-to-Any communicationwith no changes to Internet routing and clients

What is IP Anycast?

IP Address IP Address 2.1.1.12.1.1.1

AnycastServer

NetworkDomains

to2.1.1.1

CC Client

C

to2.1.1.1

One-to-Any communicationwith no changes to Internet routing and clients

What is IP Anycast?

IP Address IP Address 2.1.1.12.1.1.1

AnycastServer

NetworkDomains

to2.1.1.1

CC Client

C

to2.1.1.1

IP Anycast as a Service Discovery Primitive

I Distributes client load across servers

I Reduces access latency for clientsI Offers network-level resilience to DDoS attacks

IP Anycast Usage

I Anycasting of six of the thirteen root-servers(C-Root, F-Root, I-Root, J-Root, K-Root, M-Root)

I IPv4-to-IPv6 transition [RFC 3068]

I Rendezvous discovery for IP multicast [RFC 3446]

I Other usage scenarios[]

I AS112 Project [http://as112.net]

I Commercial CDNs [http://cachefly.net]

I DDos sinkholes [Greene et. al., NANOG’03]

IP Anycast Usage

I Anycasting of six of the thirteen root-servers(C-Root, F-Root, I-Root, J-Root, K-Root, M-Root)

I IPv4-to-IPv6 transition [RFC 3068]

I Rendezvous discovery for IP multicast [RFC 3446]

I Other usage scenarios[]

I AS112 Project [http://as112.net]

I Commercial CDNs [http://cachefly.net]

I DDos sinkholes [Greene et. al., NANOG’03]

IP Anycast Usage

I Anycasting of six of the thirteen root-servers(C-Root, F-Root, I-Root, J-Root, K-Root, M-Root)

I IPv4-to-IPv6 transition [RFC 3068]

I Rendezvous discovery for IP multicast [RFC 3446]

I Other usage scenarios[]

I AS112 Project [http://as112.net]

I Commercial CDNs [http://cachefly.net]

I DDos sinkholes [Greene et. al., NANOG’03]

IP Anycast Usage

I Anycasting of six of the thirteen root-servers(C-Root, F-Root, I-Root, J-Root, K-Root, M-Root)

I IPv4-to-IPv6 transition [RFC 3068]

I Rendezvous discovery for IP multicast [RFC 3446]

I Other usage scenarios[]

I AS112 Project [http://as112.net]

I Commercial CDNs [http://cachefly.net]

I DDos sinkholes [Greene et. al., NANOG’03]

In spite of growing usage, IP Anycast and itsinteraction with IP Routing is not well understood!

IP Anycast is not well understood

Are clients routed to close-by anycast servers?

What is the impact of the failure of an anycastserver?

Is the client load across the anycast server sitesbalanced?

Are subsequent packets from a client routed to thesame anycast site?

IP Anycast is not well understood

Are clients routed to close-by anycast servers?

What is the impact of the failure of an anycastserver?

Is the client load across the anycast server sitesbalanced?

Are subsequent packets from a client routed to thesame anycast site?

I Affinity offered by IP Anycast

I Load-distribution across deployments

I Failover properties of IP Anycast

I Proximity offered by IP Anycast

A sneak peek at the study’s results

Property

AnycastIP

Proximity Failover Load Affinity

Ad-Hoc Deployment

PlannedDeployment

A sneak peek at the study’s results

Property

AnycastIP

Proximity Failover Load Affinity

Ad-Hoc Deployment

PlannedDeployment

A sneak peek at the study’s results

Property

AnycastIP

Proximity Failover Load Affinity

Ad-Hoc Deployment

PlannedDeployment

Take Home Message

Talk Outline

I Introduction[]y

I Terminology[]y

I Deployments Measured[]y

I Methodology[]yI Measurements[]y

I ProximityI FailoverI Load-distributionI Affinity

I Conclusions

Terminology

IP Address IP Address 2.1.1.12.1.1.1

AnycastServer

NetworkDomains

Anycast Serversbelong to different domains

Study focusses on Inter-domain IP Anycast

Terminology

IP Address IP Address 2.1.1.12.1.1.1

AnycastServer

NetworkDomains

2.1.1.0/24 2.1.1.0/24

BGPAdvert.

Anycast Prefix = 2.1.1.0/24

Anycast Servers advertise the Anycast Prefix intoBGP through their Upstream Provider

Deployments Measured

External Deployments

F-Root : 27 serversJ-Root : 13 serversAS112 : 20 servers

Deployments Measured

External Deployments

F-Root : 27 serversJ-Root : 13 serversAS112 : 20 servers

Internal Deployment

Internal : 5 servers

IR Cambridge

UK

Internet

BGP Advert.for Anycast Prefix 204.9.168.0/22

Internal Deployment

ATT ggWCG

IRSeattle

US

Cornell Ithaca

US

IRBerkeley

US

IR Pittsburgh

US

ATT-World

Probing Methodology

Internet

F-Root (Palo Alto)

F-Root (Chicago)

F-Root (New York)

F-Root (Madrid)

F-Root (Osaka)

C C

F-Root Deployment: Anycast Servers are DNS Servers

Probing Methodology

Internet

ATT

WCGATT-World

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

C C

Internal Deployment: Anycast Servers can run DNS Servers

Probing Methodology

Internet

ATT

WCGATT-World

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

C C

DNS Query

Anycast Servers can be probedusing DNS queries

Probing Methodology

Internet

ATT

WCGATT-World

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

C C

DNS Query DNS Response

Anycast Servers can be probedusing DNS queries

Probing Methodology

Internet

ATT

WCGATT-World

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

DNS Query DNS Response

Recursive DNS Servers can be used as Vantage Points [KING, IMW’02]

C

Recursive DNSServer used asCa Vantage Point

Probing Methodology

Internet

ATT

WCGATT-World

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

DNS Query DNS Response

C

Recursive DNSServer used asCa Vantage Point

StationMeasurement

such that it queries the Anycast Server Query the Recursive DNS Server

Probing Methodology

Internet

ATT

WCGATT-World

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

DNS Query DNS Response

C

Recursive DNSServer used asCa Vantage Point

StationMeasurement

Response from the Anycast Server isforwarded onto the Measurement Host

Probing Methodology

Internet

ATT

WCGATT-World

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

DNS Query DNS Response

C

Recursive DNSServer used asCa Vantage Point

StationMeasurement

Measurements: 1). Anycast Server being accessed by C 2). Latency from C to the Anycast Server

Probing Methodology

23,858 Recursive DNS Servers used as Vantage Points

Region No. of clients % of Total

North America 12931 54.827Central America 317 1.344South America 461 1.954

Europe 5585 23.680Asia 2402 10.184

S.E. Asia 566 2.400Oceania 1196 5.071Africa 187 0.792

Arctic Region 9 0.038Unknown 204 0.864

Total 23858 100.000

Talk Outline

I Introduction[]y

I Terminology[]y

I Deployments Measured[]y

I Methodology[]yI Measurements[]y

I ProximityI FailoverI Load-distributionI Affinity

I Conclusions

Proximity

Internet

ATT

WCGATT-World

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

StationMeasurement

C

Anycast Server is chosen by Inter-Domain Routing

Proximity

Internet

ATT

WCGATT-World

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

StationMeasurement

C

Probe toAnycast Address

Ideally, Clients would be routedclose-by Anycast Server to a

Proximity

Internet

ATT

WCGATT-World

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

StationMeasurement

C

Probe toAnycast Address

Poor choice of Anycast Server is possible!

Proximity

Internet

ATT

WCGATT-World

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

StationMeasurement

C

Probe toAnycast Address

Probe Latency =

Probe the Deployment’s Anycast Address from the client

Anycast Latency

Proximity

Internet

ATT

WCGATT-World

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

StationMeasurement

C

Probe toAnycast Address

Probe to Unicast Address

Latency to closest Anycast Server =Client probes individual Anycast Servers

Min. Unicast Latency

Proximity

Internet

ATT

WCGATT-World

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

StationMeasurement

C

Probe toAnycast Address

Probe to Unicast Address

STRETCH = (Anycast Latency - Minimum Unicast Latency)

Proximity

0

0.2

0.4

0.6

0.8

1

-50 0 50 100 150 200

CD

F

Stretch (msec)

F-RootJ-RootAS112

Internal

0

0.2

0.4

0.6

0.8

1

-50 0 50 100 150 200

CD

F

Stretch (msec)

F-RootJ-RootAS112

Internal

Proximity

0

0.2

0.4

0.6

0.8

1

-50 0 50 100 150 200

CD

F

Stretch (msec)

F-RootJ-RootAS112

Internal

0

0.2

0.4

0.6

0.8

1

-50 0 50 100 150 200

CD

F

Stretch (msec)

F-RootJ-RootAS112

Internal

Internal 31%

30 msec

AS112 61%

All four deployments offer poor Proximity

Investigating Poor Proximity

ATT

WCGATT-World

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

StationMeasurement

C

Probe toAnycast Address

Probe to Unicast Address

Level3

Client at UC Berkeley

Client probes Anycast address of Internal Deployment

Investigating Poor Proximity

ATT

WCGATT-World

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

StationMeasurement

C

Probe toAnycast Address

Probe to Unicast Address

Level3

Client at UC Berkeley

Best path toany Server

Client probes Anycast address of Internal Deployment Routed to Ithaca (NY) instead of Berkeley (CA)

Investigating Poor Proximity

ATT

WCGATT-World

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

StationMeasurement

C

Probe toAnycast Address

Probe to Unicast Address

Level3

Best path toany Server

http://pias.gforge.cis.cornell.edu/trace.htmlSee this example at

Investigating Poor Proximity

ATT

WCGATT-World

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

StationMeasurement

C

Probe toAnycast Address

Probe to Unicast Address

Level3

Best path toany Server

Level3 does not realize that these lead to different locationsLevel3 has two paths of 2 AS-hops: through ATT and WCG

Alleviating Poor Proximity

ISP X

US

US Europe

Asia Oceania (W. Coast)

(E. Coast)

Internet

Anycast Servers should have the same Upstream ISP

Alleviating Poor Proximity

ISP X

US

US Europe

Asia Oceania (W. Coast)

(E. Coast)

Internet

Anycast Servers should have the same Upstream ISP

Multiple Providers: Geographically cover all providers

US

USEurope

Asia

Oceania(W. Coast)

(E. Coast)

ISP X

US

USEurope

Asia

Oceania(W. Coast)

(E. Coast)

ISP Y

Internet

Alleviating Poor Proximity

ISP X

US

USEurope

Asia

Oceania(W. Coast)

(E. Coast)

Internet

C

Alleviating Poor Proximity

ISP X

US

USEurope

Asia

Oceania(W. Coast)

(E. Coast)

Internet

C

Anycast probes from client(s) routed to ISP X

Alleviating Poor Proximity

ISP X

US

USEurope

Asia

Oceania(W. Coast)

(E. Coast)

Internet

C

Server chosen is based on X’s intra-domain routing

Verifying our hypothesis

Internet

ATT

WCGATT-World

StationMeasurement

C

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

A subset of the Internal Deployment

Verifying our hypothesis

0

0.2

0.4

0.6

0.8

1

-50 0 50 100 150 200

CD

F

Stretch (msec)

InternalSubset

0

0.2

0.4

0.6

0.8

1

-50 0 50 100 150 200

CD

F

Stretch (msec)

InternalSubset

31%

30 msec

5%

Verifying our hypothesis

0

0.2

0.4

0.6

0.8

1

-50 0 50 100 150 200

CD

F

Stretch (msec)

InternalSubset

0

0.2

0.4

0.6

0.8

1

-50 0 50 100 150 200

CD

F

Stretch (msec)

InternalSubset

31%

30 msec

5%

Planned Anycast Deployment ⇒ good Proximity

Failover

Internet

ATT

WCGATT-World

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

StationMeasurement

C C

Probe to Anycast Address (pre-failure)

Failover

Internet

ATT

WCGATT-World

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

StationMeasurement

C C

Probe to Anycast Address (pre-failure)

Probe to AnycastAddress (post-failure)

Clients are re-routed to a different Anycast Server

Failover

Internet

ATT

WCGATT-World

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

StationMeasurement

C C

Probe to Anycast Address (pre-failure)

Probe to AnycastAddress (post-failure)

Clients are re-routed to a different Anycast ServerWhat is the failover time?

Failover

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

20 40 60 80 100 120

CD

F

Failover Time (sec)

IthacaPittsburgh

CambridgeSeattle

Berkeley

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

20 40 60 80 100 120

CD

F

Failover Time (sec)

IthacaPittsburgh

CambridgeSeattle

Berkeley

Failover

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

20 40 60 80 100 120

CD

F

Failover Time (sec)

IthacaPittsburgh

CambridgeSeattle

Berkeley

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

20 40 60 80 100 120

CD

F

Failover Time (sec)

IthacaPittsburgh

CambridgeSeattle

Berkeley

Ithaca and Cambridge servers have slow failover

Failover

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

20 40 60 80 100 120

CD

F

Failover Time (sec)

IthacaPittsburgh

CambridgeSeattle

Berkeley

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

20 40 60 80 100 120

CD

F

Failover Time (sec)

IthacaPittsburgh

CambridgeSeattle

Berkeley

Other servers have faster failover

Failover

Internet

ATT

WCGATT-World

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

BGP Withdrawal

BGP Withdrawal propagated beyond WCGGlobal Routing Event Slow convergence

Failover

Internet

ATT

WCGATT-World

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

BGP Withdrawal

BGP Withdrawal restricted to ATTLocal Routing Event Faster convergence

Failover

Internet

ATT

WCGATT-World

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

Planned Deployment Fast Failover

Load Distribution

Distribution of client-load across Anycast Servers

I Can operators control this load?I Used AS-Path Prepending for controlling load

AS-Path Prepending a BGP Advertisement

I Changing the advertisement’s AS-Path lengthI n-hop Prepending: Add n ASs to the AS-Pathg

Load Distribution

0 0.1 0.2 0.3 0.4 0.5 0.6

43210

Load

Fra

ctio

n

Prepending at the Ithaca server (AS-hops)

IthacaPittsburgh

SeattleBerkeley

Cam.

0 0.1 0.2 0.3 0.4 0.5 0.6

43210

Load

Fra

ctio

n

Prepending at the Ithaca server (AS-hops)

IthacaPittsburgh

SeattleBerkeley

Cam.

Load Distribution

0 0.1 0.2 0.3 0.4 0.5 0.6

43210

Load

Fra

ctio

n

Prepending at the Ithaca server (AS-hops)

IthacaPittsburgh

SeattleBerkeley

Cam.

0 0.1 0.2 0.3 0.4 0.5 0.6

43210

Load

Fra

ctio

n

Prepending at the Ithaca server (AS-hops)

IthacaPittsburgh

SeattleBerkeley

Cam.

Skewed distribution of clients

Load Distribution

0 0.1 0.2 0.3 0.4 0.5 0.6

43210

Load

Fra

ctio

n

Prepending at the Ithaca server (AS-hops)

IthacaPittsburgh

SeattleBerkeley

Cam.

0 0.1 0.2 0.3 0.4 0.5 0.6

43210

Load

Fra

ctio

n

Prepending at the Ithaca server (AS-hops)

IthacaPittsburgh

SeattleBerkeley

Cam.0.34 0.23 0.18

Load Distribution

0 0.1 0.2 0.3 0.4 0.5 0.6

43210

Load

Fra

ctio

n

Prepending at the Ithaca server (AS-hops)

IthacaPittsburgh

SeattleBerkeley

Cam.

0 0.1 0.2 0.3 0.4 0.5 0.6

43210

Load

Fra

ctio

n

Prepending at the Ithaca server (AS-hops)

IthacaPittsburgh

SeattleBerkeley

Cam.0.34 0.23 0.18

AS-Path Prepending provides coarse-grainedcontrol over client load

Affinity

Internet

ATT

WCGATT-World

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

C

IP Anycast is a network-layer service

Affinity

Internet

ATT

WCGATT-World

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

C

Client can to a different Anycast Serverflap

Affinity

Internet

ATT

WCGATT-World

Seattle, WA

Berkeley, CA Pittsburgh, PA

Ithaca, NY Cambridge, UK

C

Client can to a different Anycast ServerflapWhat is the offered by IP Anycast? Affinity

Affinity

0

0.2

0.4

0.6

0.8

1

1001010

CD

F

Average Flaps per day

Internal

0

0.2

0.4

0.6

0.8

1

1001010

CD

F

Average Flaps per day

Internal

Affinity

0

0.2

0.4

0.6

0.8

1

1001010

CD

F

Average Flaps per day

Internal

0

0.2

0.4

0.6

0.8

1

1001010

CD

F

Average Flaps per day

Internal

95% of clientsobserve less than 1 Flap per day

Affinity

0

0.2

0.4

0.6

0.8

1

1001010

CD

F

Average Flaps per day

Internal

0

0.2

0.4

0.6

0.8

1

1001010

CD

F

Average Flaps per day

Internal

Less than 1% of clients experience frequent flaps

Affinity

0

0.2

0.4

0.6

0.8

1

1001010

CD

F

Average Flaps per day

Internal

0

0.2

0.4

0.6

0.8

1

1001010

CD

F

Average Flaps per day

Internal

Less than 1% of clients experience frequent flaps

Frequent flaps can be attributed toload-balancing at the client site

Conclusions

Property

AnycastIP

Proximity Failover Load Affinity

Ad-Hoc Deployment Poor Slow

Conclusions

Property

AnycastIP

Proximity Failover Load Affinity

Ad-Hoc Deployment Poor Slow

PlannedDeployment Good Fast

Due to the planned deployment

Conclusions

Property

AnycastIP

Proximity Failover Load Affinity

Ad-Hoc Deployment Poor Slow

PlannedDeployment Good Fast

Skewed

Manipulatable

BGP Traffic Engineering techniques

Conclusions

Property

AnycastIP

Proximity Failover Load Affinity

Ad-Hoc Deployment Poor Slow

PlannedDeployment Good Fast

Skewed

Manipulatable

Good*

Good*

Conclusions

Property

AnycastIP

Proximity Failover Load Affinity

Ad-Hoc Deployment Poor Slow

PlannedDeployment Good Fast

Skewed

Manipulatable

Good*

Good*

Traceshttp://pias.gforge.cis.cornell.edu/measure.php

Alleviating Poor Proximity

ISP X

US

USEurope

Asia

Oceania(W. Coast)

(E. Coast)

Internet

C

Hot-Potato Routing

Least-CostRouting

Only Proximity measurements for the

External Deployments

Internet

F-Root (Palo Alto)

F-Root (Chicago)

F-Root (New York)

F-Root (Madrid)

F-Root (Osaka)

C

Probes to External DeploymentsCannot determine the identity of the responding server