Archipelago - AFRINICmeeting.afrinic.net/afrinic-11/slides/aaf/ark_aaf_af11.pdf · group monitors into teams and dynamically divide ... 3 teams active 11. IPv4 Routed /24 Topology

kc claffyCAIDA

AAF WorkshopNov 23, 2009

ArchipelagoMeasurement Infrastructure

Outline

Focus and Architecture

Monitor Deployment

Measurements

Future Work

2

Introduction

Archipelago (Ark) is CAIDA’s next-generation active measurement infrastructure

evolution of the skitter infrastructure

in production since Sep 12, 2007

3

Focus

easy development and rapid prototyping

lower barriers => implement better measurements faster with lower cost• measurement infrastructures notoriously lack funding

raise level of abstraction with high-level API and scripting language• inspiration from Scriptroute, Metasploit, Scapy, Racket

4

Focus

dynamic and coordinated measurements

take advantage of multiple distributed measurement nodes in sophisticated ways• one measurement triggers another measurement• use multiple nodes to divide and conquer• synchronize measurements

for example: Doubletree; tomography; Rocketfuel-like targeted discovery of a single network’s topology

5

Focus

measurement services

build upon the work of others; share services between measurement activities• for example, on-demand traceroute/ping service; IP-to-AS

mapping service

similiar in goal to service-oriented architecture (SOA) but at finer granularity and without the complexity

6

Architecture

Ark is composed of measurement nodes (machines) located in various networks worldwide

many thanks to the organizations hosting Ark boxes

please contact us if you want to host an Ark box

Ark employs a tuple space to enable communication and coordination

a tuple space is a distributed shared memory combined with a small number of easy-to-use operations

a tuple space stores tuples, which are arrays of simple values (strings and numbers), and clients retrieve tuples by pattern matching

7

Architectureuse tuple space for decentralized (that is, peer-to-peer) communication, interaction, and coordination

8

monitor1

central server

monitor2

monitor3

monitor4 monitor5

Monitor Deployment

38 monitors in 24 countries

9

13 North America2 South America

12 Europe1 Africa5 Asia3 Oceania

Continent20 academic10 research network4 network infrastructure2 commercial network1 community network1 military research

Organization

Measurements

IPv4 Routed /24 Topology

IPv4 Routed /24 AS Links

IPv6 Topology

DNS Names

DNS Query/Response Traffic

Spoofer Project Collaboration

10


ongoing large-scale topology measurements

ICMP Paris traceroute to every routed /24 (7.4 million)

running scamper• written by Matthew Luckie of WAND, University of Waikato

group monitors into teams and dynamically divide up the measurement work among team members

13-member team probes every /24 in 48 hrs at 100pps

only one monitor probes each /24 per cycle

3 teams active

11


12

0

5

10

15

20

25

30

Sep07

Nov07

Jan08

Mar08

May08

Jul08

Sep08

Nov08

Jan09

Mar09

amw-us (1)cjj-kr (1*)dub-ie (1)hel-fi (1*)mnl-ph (1*)nrt-jp (1*)san-us (1*)syd-au (1*)laf-us (1*)lej-de (1)hlz-nz (1)bcn-es (1*)yto-ca (1*)iad-us (2*)vie-at (2)cbg-uk (2*)lax-us (2)hnl-us (2)cmn-ma (2)gig-br (2)sjc-us (2)zrh-ch (2)bwi-us (2)ams-nl (2)tpe-tw (2)yow-ca (2)she-cn (3)scl-cl (3)her-gr (3)pna-es (3)dfw-us (3)eug-us (3)nap-it (3)

Sep 2007 to Jan 2009 (17 months): 2.5 billion traceroutes; 1.0TB data


13

0

5

10

15

20

25

30

Sep07

Nov07

Jan08

Mar08

May08

Jul08

Sep08

Nov08

Jan09

Mar09



software failure


14

0

5

10

15

20

25

30

Sep07

Nov07

Jan08

Mar08

May08

Jul08

Sep08

Nov08

Jan09

Mar09



power supply died


14

0

5

10

15

20

25

30

Sep07

Nov07

Jan08

Mar08

May08

Jul08

Sep08

Nov08

Jan09

Mar09



power supply died

replacementpower supply


14

0

5

10

15

20

25

30

Sep07

Nov07

Jan08

Mar08

May08

Jul08

Sep08

Nov08

Jan09

Mar09



power supply died



14

0

5

10

15

20

25

30

Sep07

Nov07

Jan08

Mar08

May08

Jul08

Sep08

Nov08

Jan09

Mar09



power supply died



15

0

5

10

15

20

25

30

Sep07

Nov07

Jan08

Mar08

May08

Jul08

Sep08

Nov08

Jan09

Mar09



power supply died


died

IPv4 Routed /24 AS Links

AS links from Routed /24 Topology traces

map IP addresses to ASes with RouteViews BGP table

16

IPv4 Routed /24 AS Linksstatistics for 1 month of AS links from three sources (Dec 2008):

17

“avg neighbor deg” = avg neighbor degree of the avg k-degree node averaged over all k

“mean clustering” = (avg number of links between neighbors of k-deg nodes) / (max possible such links for k) averaged over all k

nodes linksmax

degreeaveragedegree

averageneighbordegree

meanclustering

Ark

DIMES

RouteViews (rv2)

23,425 56,760 2,509 4.85 467.3 0.354

22,995 74,140 3,590 6.45 705.4 0.446

30,760 65,775 2,328 4.28 487.2 0.241

3 AS Links Sources: 1 Month

18

10-5

10-4

10-3

10-2

10-1

100

100 101 102 103 104

CCDF

Node degree

DIMES AS links (2008-12)Ark AS links (2008-12)

RouteViews (rv2) AS links (2008-12)


19

100

101

102

103

100 101 102 103 104

aver

age

neig

hbor

deg

ree

Node degree




20

10-4

10-3

10-2

10-1

100

100 101 102 103 104

clust

erin

g

Node degree



AS Links Growth

AS links seem to accumulate linearly without bound

in skitter, Ark, DIMES; possibly in BGP

even with fixed traceroute sources and destination list (which happened with skitter for 4 years)

AS graph densification: average degree increases

for example:

1 year of Ark (2008): 104k AS links, 28k ASes

2 years of DIMES: 356k AS links, 29k ASes

7.5 years of skitter: 209k AS links, 27k ASes

21

AS Links Growth

hard to determine the “natural” time period to aggregate AS links

1 month? 6 months? years?

when do we get a representative AS graph?

22

Ark AS Links Growth

23

23000

24000

25000

26000

27000

28000

29000

1 2 3 4 5 6 7 8 9 10 11 12 55000

60000

65000

70000

75000

80000

85000

90000

95000

100000

105000#

node

s

# lin

ks

Months of accumulation

# nodes# links

Ark AS Links Growth

24

2250

2500

2750

3000

3250

3500

3750

4000

4250

4500

1 2 3 4 5 6 7 8 9 10 11 12 4.5

5

5.5

6

6.5

7

7.5m

ax d

egre

e

aver

age

degr

ee


max degreeaverage degree

Ark AS Links Growth

25

450

500

550

600

650

700

750

800

850

1 2 3 4 5 6 7 8 9 10 11 12 0.34

0.36

0.38

0.4

0.42

0.44

0.46

0.48

0.5

0.52

0.54av

erag

e ne

ighb

or d

egre

e

clust

erin

g


average neighbor degreeclustering

Ark AS Links: 1, 6, 12 Months

26

10-5

10-4

10-3

10-2

10-1

100

100 101 102 103 104

CCDF

Node degree

Ark AS links (2008, 1 to 12)Ark AS links (2008, 7 to 12)

Ark AS links (2008, 12 to 12)


27

101

102

103

100 101 102 103 104

aver

age

neig

hbor

deg

ree

Node degree




28

10-3

10-2

10-1

100

100 101 102 103 104

clust

erin

g

Node degree



Ark IPv6 Topology

ongoing “large-scale” IPv6 measurements since Dec 12, 2008

10 monitors: 3 in US, 7 International

another IPv6 box coming Real Soon Now

ICMP Paris traceroute to every routed prefix

each monitor probes a random destination in every routed prefix in every cycle; 1,553 prefixes <= /48

reduced probing rate to take 2 days per cycle

running scamper

29

Ark IPv6 Topology

statistics for 8 weeks of AS links from six sources:

Dec 12, 2008 to Feb 7, 2009

30

nodes linksmax

degreeaveragedegree

averageneighbordegree

meanclustering

IPv68 weeks

IPv44 weeks

520 1,181 94 4.54 36.3 0.265

23,425 56,760 2,509 4.85 467.3 0.354

Ark IPv6 AS Links

31

10-5

10-4

10-3

10-2

10-1

100

100 101 102 103 104

CCDF

Node degree

Ark IPv4 AS links (2008-12, 4 weeks)Ark IPv6 AS links (2008-12, 8 weeks)

Ark IPv6 AS Links

32

101

102

103

100 101 102 103 104

aver

age

neig

hbor

deg

ree

Node degree


Ark IPv6 AS Links

33

10-3

10-2

10-1

100

100 101 102 103 104

clust

erin

g

Node degree


DNS Names

automated ongoing DNS lookup of IP addresses seen in the Routed /24 Topology traces

all intermediate addresses and responding destinations

using our in-house bulk DNS lookup service (HostDB)• can look up millions of addresses per day

257M lookups since March 2008

34

DNS Traffic

tcpdump capture of DNS query/response traffic

only for lookups of Routed /24 Topology addresses

continuous collection of 3-5M packets per day

can download most recent 30 days of pcap files

a broad sampling of the nameservers on the Internet due to the broad coverage of the routed space in traces

how many nameservers have IPv6 glue records? DNSSEC records? support EDNS? typical TTLs?

35

Alias Resolution

Goal: collapse interfaces observed in traceroute paths into routers

toward a router-level map of the Internet

alias resolution work led by Ken Keys

36

Spoofer Project

collaboration with Rob Beverly on MIT Spoofer Project

how many networks allow packets with spoofed IP addresses to leave their network?

Ark monitors act as targets for spoofed probes sent by willing participants

forwards received probe data to MIT server

37

Spoofer Project

38

monitor monitor

monitor

MIT

CAIDA

UDP port 53

tuple space

Ark Statistics Pagesper-monitor analysis of IPv4 topology data

RTT, path length, RTT vs. distance

39

www.caida.org/projects/ark/statistics

http://www.caida.org/data/active/ipv4_routed_24_topology_dataset.xml


Future Work

release Marinda tuple space under GPL

implement large-scale RadarGun measurements

more in-depth analysis of data for stats pages

investigate AS link densification

DNS open resolver surveys?

high-level packet generation, capture, and analysis API

allow semi-trusted 3rd parties to conduct measurements

40

Thanks!

www.caida.org/projects/ark

41

For more information and to request data:



Documents

Archipelago - AFRINICmeeting.afrinic.net/afrinic-11/slides/aaf/ark_aaf_af11.pdf · group monitors into teams and dynamically divide ... 3 teams active 11. IPv4 Routed /24 Topology