BGP Communities: A measurement study @RIPE77, Amsterdam · 2018. 10. 16. · RTBH: how it works AS...

BGP Communities:

A measurement study

@RIPE77, Amsterdam

Florian Streibelt1, Franziska Lichtblau1, Robert Beverly2, Cristel Pelsser3,

Georgios Smaragdakis4, Randy Bush5, Anja Feldmann1

Oct. 2018

1 Max Planck Institute for Informatics (MPII), 2 Naval Postgraduate School (NPS),3 University of Strasbourg, 4 TU Berlin (TUB), 5 Internet Initiative Japan (IIJ)

BGB-Communities:A weapon for the Internet!

RIPE 77 / Amsterdam2018.10.16

Introduction

BGP Community usage is increasing

●●

2010 2012 2014 2016 2018

●●

● ● ● ● ●

# Unique ASes in Communities# Unique Communities

# Absolute Communities# BGP table entries

Increasing usage warrants a closer look.

BGP Community usage is increasing

●●

2010 2012 2014 2016 2018

0.2B0.3B

# Unique ASes in Communities# Unique Communities

# Absolute Communities# BGP table entries

Increasing usage warrants a closer look.

BGP Communities

• Optional Attribute in BGP message (32 bit field)

• Defined in RFC 1997

• By convention written ASN:VALUE

• ASN can be both sender or intended ’recipient’

• It’s up to the peers to agree upon ’values’ used

BGP Large Communities

• Defined by RFC 8092 (usage recommendations ins RFC 8195)

• 12 byte attribute

• Enable networks with 4-byte ASNs to use communities

• The first 4 byte contain the ASN of the ”global administrator”

BGP Large Communities

Sorry. . . as we only found a very small number of

occurrences1 we could not conduct any meaningful

measurements, yet.1283 individual large communities by 51 global administrators over the whole month of April 2018 at all available

route collectors at RIPE/RIS, Routeviews, Isolario and PCH

BGP Communities: Usage

Informational Communities

(Passive Semantics)

• Location tagging

• RTT tagging

Action Communities

(Active Semantics)

• Remote triggered blackholing

• Path prepending

• Local pref/MED

• Selective announcements

Without documentation, you can not tell

if a community is active or passive!

What This Talk Is About

Given the increasing popularity of BGP communities and the

ability to trigger actions as well as relay information,

the first question that comes to the mind of an

Internet measurement researcher is. . .

What This Talk Is About

What could possibly go wrong?

Propagation behavior

• 14% of transit providers propagate received communities

(2.2k of 15.5k)

• Ratio seems small, but AS graph is highly connected

• RFC 1997: Communities as a transitive optional attribute

• RFC 7454: Scrub own, forward foreign communities

Still many people do not expect communities

to propagate that widely.

Potential (for) misuse

• Propagated communities might trigger actions multiple

AS-hops away

• No way of knowing if intended or not, e.g., for traffic

management

• But are there also unintended consequences?

Our assessment is that there is a high risk for attacks!

Observations

Dataset

BGP updates and table dumps of April 2018 from publicly available

BGP Collector Projects: RIPE RIS, Routeviews, Isolario, PCH.

BGP messages 38.98 bn

IPv4 prefixes 967,499

IPv6 prefixes 84,953

Collectors 194

AS peers 2,133

Communities 63,797

More than 75% of all BGP announcements have at least one

BGP community set, 5,659 ASes are using communities.

BGP Community Propagation Observations

●● ● ● ●

0 2 4 6 8 10

AS hop count

• 10% of communities have a AS hop count of more than six

• More than 50% of communities traverse more than four ASes

• Longest community propagation observed: 11 AS hops

0 2 4 6 8 10

AS hop count

0 2 4 6 8 10

AS hop count

BGP Community Propagation Behavior

• AS1 announces prefix p

, tagged with 3:123

• Community is intended for signaling towards AS3

• AS4 also receives this announcement

Off-path:

ASN from community is not on the observed AS-path at AS4.

• AS1 announces prefix p

, tagged with 3:123

Off-path:

3:1233:123

• AS1 announces prefix p, tagged with 3:123

Off-path:

3:1233:123

Off-path:

3:1233:123

Off-path:

3:1233:123

p: 4, 2, 1 3:123

p: 3, 2, 1 3:123

Off-path:

3:1233:123

p: 4, 2, 1 3:123

"on−path"

p: 3, 2, 1 3:123

Off-path:

3:1233:123

"on−path"

p: 3, 2, 1 3:123

p: 4, 2, 1

"off−path"

Off-path:

On-path versus off-path

3000 2

9498 200

1000 100 1

2000 10 2

3000 0

off-path on-path

• Blackholing communities (e.g., :666) ’leaking’ off path

• But AS implementing RTBH

SHOULD add NO ADVERTISE or NO EXPORT (RFC7999)

Suggests ASes not implementing RTBH do not filter.

Experiments

Experimental setup

• Experiments conducted in a lab environment

• Validated on the Internet

Scenarios

• Remote Triggered Blackholing (RTBH)

• Traffic redirection attack

...for others see our paper.

RTBH: how it works

• AS announces BH-prefix

to upstream

→ Provider blackholes prefix

AS3 AS4

Safeguards:

• Provider should check customer prefix before accepting RTBH

• Customer may only blackhole own prefixes

• Different policies for Customers/Peers

• On receiving RTBH, add NO ADVERTISE or NO EXPORT

(RFC7999)

RTBH: how it works

to upstream

AS3 AS4

BGP announcements

Safeguards:

(RFC7999)

RTBH: how it works

to upstream

AS3 AS4

BGP announcements

Traffic flow

Safeguards:

(RFC7999)

RTBH: how it works

to upstream

AS1 sends p, tagged 2:666

AS3 AS4

BGP announcements

Traffic flow

p2:666

Safeguards:

(RFC7999)

RTBH: how it works

to upstream

AS2 continues announcing p

AS3 AS4

BGP announcements

Traffic flow

p2:666

Safeguards:

(RFC7999)

RTBH: how it works

to upstream

AS3 AS4

BGP announcements

Traffic flow

p2:666

Safeguards:

(RFC7999)

RTBH: how it works

to upstream

→ Provider blackholes prefix X

AS3 AS4

BGP announcements

Traffic flow

p2:666

Safeguards:

(RFC7999)

RTBH: how it works

to upstream

Traffic to p is dropped at AS2

AS3 AS4

BGP announcements

Traffic flow

p2:666

Safeguards:

(RFC7999)

RTBH: how it works

to upstream

AS3 AS4

BGP announcements

Traffic flow

p2:666

Safeguards:

(RFC7999)14

RTBH: how it should not work

BGP announcementsAS2

• AS on ’backup’ path adds RTBH-community

• Provider blackholes prefix

• Not only traffic traversing AS2 is dropped

AS1 announces p

BGP announcements

Traffic flow

Community

Target

Attackee

Attacker

AS1 announces p

BGP announcements

Traffic flow

Community

Target

Attackee

Attacker

AS2 tags p with AS3:666

AS1 announces p

BGP announcements

Traffic flow

AS3:666

Community

Target

Attackee

Attacker

AS1 announces p

BGP announcements

Traffic flow

AS3:666

Community

Target

Attackee

Attacker

AS1 announces p

BGP announcements

Traffic flow

AS3:666

RTBH: how it should not work (with hijack)

Community

Target

Attackee

Attacker

AS1 announces p

BGP announcements

Traffic flow

AS2 hijacks p, with AS3:666

AS3:666

• Hijacker announces RTBH

• Prefix filters circumvented due to misconfiguration

RTBH: Attack confirmed

Attack confirmed to work on the Internet, works multi hop

and is hard to spot

Triggering RTBH is possible for attackers because, e.g.,:

• BH prefix is more specific, accepted via exception

• Providers check BH community before prefix filters2

• NO ADVERTISE or NO EXPORT often is ignored / not set

• Problem: No validation for origin of community

2we found configuration guides with that bug

Traffic redirection attack

AS2AS1

• Attacker AS2 uses community to add path-prepending in AS3

• AS6 routes traffic towards prefix p via AS5, AS4

• Network tap?

• Slow/Congested link?

• ...

BGP−Announcements

AS2AS1

• Network tap?

• ...

Trafficflow

BGP−Announcements

AS−Paths at AS6:

AS2AS1

p: 5, 4, 2, 1

3, 2, 1

• Network tap?

• ...

Attackee Attacker

Community TargetTrafficflow

BGP−Announcements

AS−Paths at AS6:

AS2AS1

p: 5, 4, 2, 1

3, 2, 1

• Network tap?

• ...

AS3:3x

pAttackee Attacker

BGP−Announcements

AS−Paths at AS6:

AS2AS1

p: 3, 3, 3,

5, 4, 2, 1

• Network tap?

• ...

AS3:3x

pAttackee Attacker

BGP−Announcements

AS−Paths at AS6:

AS2AS1

p: 3, 3, 3,

5, 4, 2, 1

• Network tap?

• ...

AS3:3x

pAttackee Attacker

BGP−Announcements

AS−Paths at AS6:

AS2AS1

p: 3, 3, 3,

5, 4, 2, 1

• Network tap?

• ...

AS3:3x

pAttackee Attacker

BGP−Announcements

AS−Paths at AS6:

AS2AS1

p: 3, 3, 3,

5, 4, 2, 1

• Network tap?

• ...

AS3:3x

pAttackee Attacker

BGP−Announcements

AS−Paths at AS6:

AS2AS1

p: 3, 3, 3,

5, 4, 2, 1

• Network tap?

• ...

Discussion: What now?

BGP Communities Shortcomings Summarized

• Notation of ”ASN:value” is just convention

• No defined semantics: values can mean anything

• Used both for signaling and triggering of actions

• No cryptographic protection

• Attribution is impossible

• Large Communities have, in principle, similar limitations

BGP Communities: The Problem

• BGP Communities as they are used are not necessarily broken

• Secure usage requires good operational knowledge and

diligence

• While people in this room probably know what they are doing:

Based on experience we do not rely on that globally. . .

Do we need less fragile protocols and mechanisms?

BGP Communities: The Problem

• BGP Communities as they are used are not necessarily broken

• Secure usage requires good operational knowledge and

diligence

• While people in this room probably know what they are doing:

Based on experience we do not rely on that globally. . .

Do we need less fragile protocols and mechanisms?

Recommendations

• Filter incoming Informational Communities for your ASN

• Publish community documentation, to enable others to filter

• Monitor and log received communities to track abuse

• Talk to your Downstreams, so they filter

Action Communities for your ASN on ingress if neccessary

• Provide a looking glass (that shows communties!)

Discussion: Authenticity

• Communities can be modified, added, removed by every AS

• No attribution is possible

• No cryptographic protection

• Still operators rely on their ’correctness’

• Large communities partially improve the situation

How can we achieve authenticity, or at least attribution?

Discussion: Transitivity

• Communities can help in debugging

• Easy, low overhead communication channel

• Widely in use, but often only 1-2 hops

• But: High risk of being abused!

Are fully transitive communities still worth the clear risk?

Discussion: Monitoring

• There is no global state in BGP

• Route collectors only see the ’end-result’

• Inferring modifications between origin-AS and collector:

almost impossible

• The meaning of a particular community can not be known

• No universal way for attribution of changes

Monitoring communities to detect abuse is extremely

difficult.

Discussion: Standards

• There are limited standardized communities

• Many AS do not implement these

• Is the lack of standardized communities a problem?

• Are standards doing harm, by helping attackers?

• Security by obscurity never works

Standardization is necessary.

Discussion: Documentation

There is no easy way to find meaning of a community:

• Some ASes document in the whois

• Some ASes document on their website

• Some ASes provide documentation only to customers

• Some ASes do not provide any documentation

Documentation is limited and fragmented.

Summary

• Communities are widely in use

• Foundation of many policies

• Relies heavily on mutual trust in capabilities

• No authenticity/security in place

• Attribution is impossible

• Hard to detect attacks

• While our prefix hijacks were reported,

no one reported our community attacks

It’s unknown if there are other unnoticed attacks.

Get the preprint at:

https://people.mpi-inf.mpg.de/~fstreibelt/preprint/

communities-imc2018.pdf

Published at ACM IMC 2018

https://conferences.sigcomm.org/imc/2018/

Contact:

Florian Streibelt <fstreibelt@mpi-inf.mpg.de>

Images:

Unicorn illustrations: Telegram stickers by Darya Ogneva:

https://tlgrm.eu/stickers/BornToBeAUnicorn

The Spanish Inquisition: by Miki Montllo

http://miquelmontllo.blogspot.com/2013/10/

the-spanish-inquisition-wallpaper.html

BGP Communities: A measurement study @RIPE77, Amsterdam · 2018. 10. 16. · RTBH: how it works AS...

Documents

Blackholes (Main)i

Abstract( BlackHoles,(BlackSwans,BlackMagic ... Ho… · ThirteenthVecellioDistinguishedLecture! 1!! Abstract(BlackHoles,(BlackSwans,BlackMagic(…..theemergingArtsofprojectexecution

tinglado-mediafiles.s3.amazonaws.com€¦ · Web viewA detailed analysis of the waveform has revealed that this gravitational wave is from the merger of 2 blackholes of unequal

Ob i Bl k H lObserving Black Holes - Science in the Newssitn.hms.harvard.edu/.../uploads/2011/05/blackholes-2.pdf · 2013. 5. 8. · Observing Black Holes: Outline eh t•Btu y're

LossRadar: Fast Detection of Lost Packets in Data Center ...types (e.g., congestion, blackholes). With fast loss detection, operators have enough time to diagnose their root causes

Quantum Simulators of Fundamental Physics · 2018. 11. 23. · Quantum Field Theory Quantum Field Theory In Curved-Spacetimes e.g. Superradiance BlackHoles Cosmological spacetimes

Black Holes - Thornekip/scripts/PubScans/BlackHoles-Thorne-Starmus.pdfcarrying a detailed, but encoded, picture of their source, the orbiting black holes. The waves also carry energy

Blackholes,gravitationalwavesandfundamental physics: aroadmap · Blackholes,gravitationalwavesandfundamentalphysics: aroadmap 3 Abstract. The grand challenges of contemporary fundamental

DNS over anything but UDP: Implications - ripe77.ripe.net · DoH: DNS over HTTPS UDP or Json blob in HTTP on port 443 DoH[1] pending RFC publication Envisioned as Application to Resolver

PROTECT NETWORK EDGE WITH BGP, URPF AND S/RTBH · PROTECT NETWORK EDGE WITH BGP, URPF AND S/RTBH by John Brown, CityLink Telecommunications, LLC . About Me ... ¨ We can protect the

Black Holes - Thornekip/PubScans/BlackHoles-Thorne...Kip S. Thorne, California Institute of Technology When a massive star exhausts the nuclear fuel whose burning keeps it hot, the

L23 Blackholes

RIPE Database - ripe77.ripe.net · region ASNs, confusing contact information, policy, etc. • This facilitates hijacking for out-of-region resources and RIPE NCC is neither mandated

CNU-AST-96 - arXiv · CNU-AST-96 Numerical Simulations of Standing Shocks in AccretionFlows around BlackHoles: A Comparative Study1 Diego Molteni Istituto di Fisica , Via Archiraﬁ

Non-Abelian Magnetized Blackholes and Unstable Attractors

10 BlackHoles

Methods of Measuring Black Hole Masses: Reverberation Mappingmctp/SciPrgPgs/events/2011/Blackholes/Talks/ben… · What systematic errors are inherent in AGN black hole masses? 3

The Early GrowthoftheFirst BlackHoles · With detections of quasars powered by increasingly massive black holes (BHs) at increasingly early times in cosmic history over the past decade,

Activity Guide for Classroom Educatorsastc.org/exhibitions/blackholes/BH_TeacherEdGuide.pdf · Black Holes: Space Warps & Time ... Activity Guide for Classroom Educators ... facilitators

Newcomers Axel RIPE77 wide · I’m Axel Pawlik RIPE NCC Managing Director German Friendly. Formed in 1992 Secretariat for the RIPE community Regional Internet Registry (RIR) for