DDoS Resiliency with Amazon Web Services (SEC305) | AWS re:Invent 2013

DDoS Resilience with Amazon Web Services

nated@amazon.com

November 14, 2013

Agenda

• Anatomy of DDoS

• Things We Do So You Don’t Have To

• Designing for Availability

• Attack Response

DDoS Facts

• Yes, DDoS attacks are on the rise and the big

ones are getting bigger

• …although those attacks average out to

~14Gbps* and target services owners ~1 per

*source: Arbor Networks

DDoS Facts

*source: Arbor Networks

Percentile Max Gbps Duration

(minutes) 10 2.39 5.87

20 4.28 7.68

30 6.55 9.00

40 8.27 10.53

50 10.49 13.23

60 11.85 16.80

70 13.97 23.12

80 17.38 35.87

90 25.45 66.13

95 35.74 141.74

99 84.90 906.80

Max 299.43

Average 13.81

DDoS Anatomy

Application Exhaustion

/search.php?expensive-params

attacker service

DDoS Anatomy

Host Exhaustion

attacker

service

DDoS Anatomy

Traditional Datacenter Exhaustion

attacker

traditional

datacenter transit

attacker

DDoS Anatomy

Intermediary Exhaustion

attacke

traditional

datacenter transit

transit

attacke

DDoS Anatomy

• Large enough attacks consume the capacity of

application layer, host, datacenter connectivity,

Internet connectivity, or intermediary networks

How can we help you?

• Scale and Diversity of AWS

• Resilient Service Designs

• Business or Enterprise Support

Things We Do So You Don’t Have To

model credit:

traditional

datacenter transit

region

transit

More Bandwidth

transit AWS

region

transit

More Compute

transit AWS

region

edge transit

transit

More Points of Presence

Scale Attack Absorbed

transit

attacker

region

edge transit

transit

Diversity

Internet Transit and Peering

region peer

transit

Diversity

Amazon Route 53 Example - Anycast Striping

• Leverages Resolver Behavior

• Edge Location Diversity

• Network Path Diversity

Delegation Set [nated@xyz ~]$ dig NS internetkitties.com

;; QUESTION SECTION:

;internetkitties.com. IN NS

;; ANSWER SECTION:

internetkitties.com. 172800 IN NS ns-1131.awsdns-13.org.

internetkitties.com. 172800 IN NS ns-1751.awsdns-26.co.uk.

internetkitties.com. 172800 IN NS ns-340.awsdns-42.com.

internetkitties.com. 172800 IN NS ns-952.awsdns-55.net.

Delegation Set [nated@xyz ~]$ dig NS internetkitties.com

;internetkitties.com. IN NS

;; ANSWER SECTION:

internetkitties.com. 172800 IN NS ns-1131.awsdns-13.org.

internetkitties.com. 172800 IN NS ns-1751.awsdns-26.co.uk.

internetkitties.com. 172800 IN NS ns-340.awsdns-42.com.

internetkitties.com. 172800 IN NS ns-952.awsdns-55.net.

awsdns-13.org.

awsdns-26.co.uk.

awsdns-42.com.

awsdns-55.net.

Edge Location Diversity

awsdns-13.org.

awsdns-26.co.uk.

awsdns-42.com.

awsdns-55.net.

awsdns-13.org.

awsdns-26.co.uk.

awsdns-42.com.

awsdns-55.net.

awsdns-13.org.

awsdns-26.co.uk.

awsdns-42.com.

awsdns-55.net.

awsdns-13.org.

awsdns-26.co.uk.

awsdns-42.com.

awsdns-55.net.

Network Path Diversity

awsdns-13.org.

awsdns-26.co.uk.

awsdns-42.com.

awsdns-55.net.

[nated@xyz ~]$ traceroute ns-1131.awsdns-13.org.

traceroute to ns-1131.awsdns-13.org (205.251.196.107), 64 hops max, 52 byte packets

1 (192.168.1.1) 1.748 ms 0.830 ms 0.750 ms

2 * * *

3 cat.seattle.wa.seattle.comcast.net (68.85.255.255) 14.634 ms 12.822 ms 10.774 ms

4 ae-20-0-ar03.burien.wa.seattle.comcast.net (69.139.164.125) 31.766 ms 13.898 ms

5 ae-20-0-ar03.seattle.wa.seattle.comcast.net (69.139.164.129) 20.108 ms

6 he-1-7-0-0-11-cr01.seattle.wa.ibone.comcast.net (68.86.93.5) 18.781 ms

7 ae12.edge2.seattle3.level3.net (4.68.63.65) 34.371 ms 36.504 ms 27.301 ms

8 ae-31-51.ebr1.seattle1.level3.net (4.69.147.150) 48.557 ms 60.610 ms 56.751 ms

9 ae-7-7.ebr2.sanjose1.level3.net (4.69.132.49) 58.662 ms 46.830 ms 62.458 ms

11 ae-6-6.ebr2.losangeles1.level3.net (4.69.148.201) 55.190 ms 58.829 ms 55.751 ms

12 ae-92-92.csw4.losangeles1.level3.net (4.69.137.30) 49.261 ms

13 ae-3-80.edge5.losangeles1.level3.net (4.69.144.139) 58.707 ms 53.091 ms

14 amazon.com.edge5.losangeles1.level3.net (205.129.4.26) 46.477 ms 36.525 ms 42.110 ms

15 LAX3

[nated@xyz ~]$ traceroute ns-1751.awsdns-26.co.uk.

traceroute to ns-1751.awsdns-26.co.uk (205.251.198.215), 64 hops max, 52 byte packets

1 (192.168.1.1) 1.298 ms 0.755 ms 0.694 ms

2 * * *

4 ae-20-0-ar03.seattle.wa.seattle.comcast.net (69.139.164.129) 17.281 ms 18.580 ms 17.906

9 ae-62-62.csw1.sanjose1.level3.net (4.69.153.18) 44.181 ms

10 ae-3-80.edge1.sanjose3.level3.net (4.69.152.144) 46.817 ms

11 4.53.208.22 (4.53.208.22) 54.634 ms 60.111 ms 44.187 ms

12 205.251.229.155 (205.251.229.155) 47.758 ms

13 205.251.230.91 (205.251.230.91) 52.714 ms 43.560 ms

14 SFO5

[nated@xyz ~]$ traceroute ns-340.awsdns-42.com.

traceroute to ns-340.awsdns-42.com (205.251.193.84), 64 hops max, 52 byte packets

1 (192.168.1.1) 2.444 ms 1.676 ms 1.028 ms

2 * * *

4 ae-20-0-ar03.seattle.wa.seattle.comcast.net (69.139.164.129) 24.366 ms 20.753 ms 29.955 ms

8 ae-2-2.ebr2.denver1.level3.net (4.69.132.54) 163.723 ms 159.037 ms 174.670 ms

9 ae-3-3.ebr1.chicago2.level3.net (4.69.132.62) 169.379 ms 167.307 ms 168.454 ms

10 ae-6-6.ebr1.chicago1.level3.net (4.69.140.189) 166.002 ms 168.125 ms 164.232 ms

11 ae-2-2.ebr2.newyork2.level3.net (4.69.132.66) 167.861 ms 167.893 ms 160.681 ms

12 ae-1-100.ebr1.newyork2.level3.net (4.69.135.253) 163.919 ms 166.782 ms 161.686 ms

13 4.69.201.45 (4.69.201.45) 164.023 ms

14 ae-42-42.ebr2.london1.level3.net (4.69.137.69) 165.560 ms 160.461 ms

15 ae-46-46.ebr2.amsterdam1.level3.net (4.69.143.73) 165.627 ms

16 ae-59-224.csw2.amsterdam1.level3.ne (t4.69.153.214) 172.909 ms 166.052 ms

17 4.69.162.154 (4.69.162.154) 166.353 ms

18 212.72.41.162 (212.72.41.162) 171.714 ms 174.033 ms 179.219 ms

19 AMS50

[nated@xyz ~]$ traceroute ns-952.awsdns-55.net.

traceroute to ns-952.awsdns-55.net (205.251.195.184), 64 hops max, 52 byte packets

1 (192.168.1.1) 1.352 ms 0.642 ms 0.630 ms

2 * * *

4 be-1-ur08.seattle.wa.seattle.comcast.net (69.139.164.134) 13.561 ms

5 ae-1-0-ar03.seattle.wa.seattle.comcast.net (68.85.240.94) 21.009 ms

6 he-1-12-0-0-11-cr01.seattle.wa.ibone.comcast.net (68.86.93.177) 17.366 ms 19.162 ms

7 be-12-pe03.seattle.wa.ibone.comcast.net (68.86.84.106) 19.949 ms 22.968 ms 24.976 ms

8 * * *

9 * * *

10 * 65-122-235-178.dia.static.qwest.net (65.122.235.178) 40.707 ms 30.916 ms

11 205.251.225.22 (205.251.225.22) 85.275 ms

12 205.251.225.122 (205.251.225.122) 35.017 ms 38.568 ms

13 205.251.226.136 (205.251.226.136) 36.560 ms

14 SEA50

Striping in Action

awsdns-13.org.

awsdns-26.co.uk.

awsdns-42.com.

awsdns-55.net.

Striping in Action

awsdns-13.org.

awsdns-26.co.uk.

awsdns-42.com.

awsdns-55.net.

Striping in Action

awsdns-13.org.

awsdns-26.co.uk.

awsdns-42.com.

awsdns-55.net.

Striping in Action

Diversity

transit

attacker

region

client

Diversity

• Amazon Route 53 - Anycast Striping

• Amazon CloudFront Edge Locations

• AWS Regions

• Scale and Diversity of AWS

• Amazon Route 53 and Amazon CloudFront

Designing for Resilience

• N+1 Failover

• Resilient Clients

• Capped Workloads

• Process Isolation

• Shuffle Sharding

• N+1 Failover

N+1 Failover

• Scale Out, Plus Redundancy

N+1 Failover

• Failure of 1/100 < Failure of 1/10

N+1 Failover

• Failure of 1/100 < Failure of 1/10

• Automatic Failover with Health Checked DNS

N+1 Failover

client

attacker

N+1 Failover

client

attacker

N+1 Failover

Check out Amazon Route 53

Health Checks

• N+1 Failover

Resilient Clients

• Use multi-record RRSets

• Randomize the record on connect retry

• Popular HTTP clients already do this!

Resilient Clients [nated@xyz ~]$ dig www.internetkitties.com

;www.internetkitties.com. IN A

;; ANSWER SECTION:

www.internetkitties.com. 32 IN CNAME d3g5kqnbrlf3fg.cloudfront.net.

d3g5kqnbrlf3fg.cloudfront.net. 30 IN A 54.230.69.190

Resilient Clients

Num Time Source Destination

4 2.535515 10.61.60.17 54.230.69.141 [SYN]

5 2.736659 10.61.60.17 54.230.69.190 [SYN]

6 2.93782 10.61.60.17 54.230.71.141 [SYN]

7 3.138996 10.61.60.17 54.230.71.172 [SYN]

8 3.339767 10.61.60.17 54.230.71.233 [SYN]

9 3.540963 10.61.60.17 54.240.188.66 [SYN]

11 3.541123 10.61.60.17 54.230.68.41 [SYN]

12 3.742296 10.61.60.17 54.230.68.212 [SYN]

13 3.824502 10.61.60.17 54.230.69.190 [SYN]

14 3.824515 10.61.60.17 54.230.69.141 [SYN]

15 4.024809 10.61.60.17 54.230.71.141 [SYN]

16 4.225094 10.61.60.17 54.230.71.172 [SYN]

Browser Packet Capture

Client Retry Behavior, SYN Timeout Browser OS Rotates

Time to

Rotation

Chrome 30.0.1599 Windows 7 Yes 12

Internet Explorer 8 Windows 7 Yes 12

Firefox 25 Windows 7 Yes 20

Safari 5.0.5 Windows 7 Yes 20

Safari 6.0.5 OSX 10.7.5 Yes <1

Firefox 25 OSX 10.7.5

Yes (2) <1

Chrome 32.0.1678 OSX 10.7.5

Yes (2) DNS TTL, or

Refresh

Resilient Clients attacker

service

client

Resilient Clients

• N+1 Failover

Capped Workloads

• Protect Application Layer Capacity

• Strive for Sameness

• Throttle or Sample Request Workloads

Strive for Sameness

Application Exhaustion

attacker service

Strive for Sameness

attacker service

Search_Result_Page_1

Capped Workloads Host/OS

~500K to 5M pps

AppLayer

~1K to ~10K rps

~500K to 5M pps

AppLayer

~1K to ~10K rps

Logging

~500K to 5M pps

AppLayer

~1K to ~10K rps

Logging

Throttle

~10 to ~100K rps

~500K to 5M pps

AppLayer

~1K to ~10K rps

Logging

Throttle

~10 to ~100K rps

1,000 samples /

• N+1 Failover

Process Isolation

• Isolate application components across

processes

• Let the OS protect critical resources

Process Isolation

Logging

Process Isolation

Logging

• N+1 Failover

Evolution of Resilience

client

N Choose M Isolation

• 2 endpoints 2 AZs = 4 permutations

• 8 endpoints 2 AZs = 64

Shuffle Sharding – Amazon Route 53

• Define Availability Lattice • Stripes – Edge Location

• Braids – Host Isolation

• Assign Endpoints to the Lattice • Virtual Name Servers

• Allocate Endpoints to Resources • Hosted Zone Delegate Set

Non-Overlapping Delegation Sets

;gray.internetkitties.com. IN NS

;; ANSWER SECTION:

ns-1131.awsdns-13.org.

ns-1751.awsdns-26.co.uk.

ns-340.awsdns-42.com.

ns-952.awsdns-55.net.

;orange.internetkitties.org. IN NS

;; ANSWER SECTION:

Shuffle Sharding

.co.uk

Shuffle Sharding

.co.uk

Shuffle Sharding

.co.uk

A B C D

Shuffle Sharding

.co.uk

A B C D

gray.internetkitties.com

orange.internetkitties.org

Shuffle Sharding

.co.uk

A B C D

Non-Overlapping Delegation Sets

;gray.internetkitties.com. IN NS

;; ANSWER SECTION:

;orange.internetkitties.org. IN NS

;; ANSWER SECTION:

Shuffle Sharding Resilience

.co.uk

client

attacke

Shuffle Sharding Resilience

.co.uk

client

attacke

Shuffle Sharding Toolkit

• Define a Lattice of Availability

• Allocate Service Resources to the Lattice

• Assign Customers Isolated Resources

• https://github.com/awslabs/route53-infima

Lattice Configuration // Create a 1-D lattice with "AvailabilityZone” as the dimension

OneDimensionalLattice<HealthCheckedRecordSet> myServiceLayout =

new OneDimensionalLattice<HealthCheckedRecordSet>("AvailabilityZone”);

Lattice Configuration // Add endpoints in the us-west-1a Availability zone

myServiceLayout.addEndpoint("us-west-1a”,

new HealthCheckedRecordSet("192.0.2.1"));

// Add endpoints in the us-west-1b Availability zone

myServiceLayout.addEndpoint("us-west-1b”

Lattice Configuration // Add endpoints in the us-west-1a Availability zone

// Add endpoints in the us-west-1b Availability zone

myServiceLayout.addEndpoint("us-west-1b”

Shuffle Shard // Create a shuffle sharder

SimpleSignatureShuffleSharder shuffleSharder = new SimpleSignatureShuffleSharder(5353L);

Lattice shard = shuffleSharder.shuffleShard(myServiceLayout, "v123543234", 1);

Vulcanized Lattice // Create a shuffle sharder

// Create a RubberTree of DNS records

Route53RubberTree rubberTree =

new Route53RubberTree(”v123543234.video.internetkitties.com", shard);

List rrsets = rubberTree.vulcanize();

Lattice Shard RRSet

[nated@xyz ~]$ dig v123543234.video.internetkitties.com

; v123543234.video.internetkitties.com. IN A

;; ANSWER SECTION:

v123543234.video.internetkitties.com. 60 IN A 192.0.2.12

us-west-1b

us-west-1a

us-west-1c

• N+1 Failover

Attack Response

• Detection

• Src-IP Blocking

• Engaging Customer Support

Attack Response

• Detection

• Src-IP Blocking

Detect

• Traffic Spikes, Drops

• CPU Utilization

• Network Stats

Detect

• Use Resilience Patterns to Access Logs

• X-Forwarded-For

• Sort and Sum

X-Forwarded-For

• Use a trusted load balancer or proxy

X-Forwarded-For

• Enable logging

X-Forwarded-For

• Enable logging – IIS7

• Install ‘IIS Advanced Logging’

• Configure X-Forwarded-For field

X-Forwarded-For

Enable Logging

nginx:

if($http_x_forwarded_for !='-’) {

log_format main '$http_x_forwarded_for - $remote_user

[$time_local] $status '

'"$request" $body_bytes_sent "$http_referer" '

'"$http_user_agent" "$remote_addr"';

else {

log_format main '$remote_addr - $remote_user [$time_local]

$status '

'"$request" $body_bytes_sent "$http_referer" '

'"$http_user_agent" "$http_x_forwarded_for"';

X-Forwarded-For

• Enable X-Forwarded-For logging

Sort & Sum

• Used to identify “top talkers”

[nated@xyz.com ~]$ grep 'expensive-param' ./access.log | awk '{print $1}' |

sort | uniq -c | tail

2 10.54.4.1

3 10.63.34.1

5 10.23.97.212

1182 10.54.0.183

Sort & Sum

• Used to identify “top talkers”

[nated@xyz.com ~]$ grep 'expensive-param' ./access.log | awk '{print $1}' |

sort | uniq -c | tail

2 10.54.4.1

3 10.63.34.1

5 10.23.97.212

1182 10.54.0.183

Src-IP Blacklisting

• Host-Level Firewalling

• Web-Server Configuration

• VPC Network ACLs

• Web Application Firewall

Src-IP Blacklisting

• Host-Level Firewalling (IPTables)

• Web-Server Configuration (Nginx / Apache, IIS)

Src-IP Blacklisting

• Web-Server Configuration

VPC Network ACLs

• Apply to a VPC subnet

• Supports DENY rules

VPC Network ACLs

• Enter each source IP

• Set DENY

Src-IP Blacklisting

Web Application Firewall

• Src-IP Blacklist

• HTTP Headers (X-Forwarded-For)

• URI-Based Filtering

• Advanced Throttling

Attack Response

• Detection

• Src-IP Blocking

Engaging Customer Support

http://aws.amazon.com/premiumsupport/

Summary

How can we help? • Scale and Diversity

• Route 53 and CloudFront

• Business and Enterprise

Support

Resilient Design • Availability Lattice

• N+1 Failover

• Process Isolation Attack Response • Enable X-Forwarded-For Logging

• Detect, Sum and Sort

• Engage Customer Support

Summary

How can we help? • Scale and Diversity

• Route 53 and CloudFront

• Business and Enterprise

Support

Resilient Design • Availability Lattice

• N+1 Failover

• Process Isolation Attack Response • Enable X-Forwarded-For Logging

• Detect, Sum and Sort

• Engage Customer Support

Please give us your feedback on this

presentation

As a thank you, we will select prize

winners daily for completed surveys!

SEC305

DDoS Resiliency with Amazon Web Services (SEC305) | AWS re:Invent 2013

Technology

DDoS Testing Services - zencurity.com · DDoS Testing Services DOES YOUR BUSINESS HAVE AN OPTIMISED DDOS PROTECTION STRATEGY Our DDoS testing service exercises your infrastructure

DDoS hybrid protection - rsnog.rs · DDoS protection system –Hybrid solution •Telenor hybrid DDoS protection is based • Local netflow based DDoS protection segment • Cloud

Continuous Deployment @ AWS Re:Invent

DDoS Protection

AWS re:Invent 2017 Recap

A Taxonomy of DDoS Attack and DDoS Defense Mechanisms

AWS re:Invent 2016: Scaling Security Resources for Your First 10 Million Customers (SEC305)

A10 DDoS Protection Cloud · A10 DDOS PROTECTION CLOUD DATA SHEET FULL SPECTRUM ENTERPRISE DDOS PROTECTION With the increase in sophistication, intensity and frequency of DDoS attacks,

Re:Invent announcements 2014

Datasheet: DDoS Protection - Grayhats Datasheet DDoS...or OpenBSD vulnerabilities ... Datasheet: DDoS Protection DNS DDoS Protection ... DNS server. This protects your

AWS re:Invent - Accelerating Research

(SEC307) Building a DDoS-Resilient Architecture with Amazon Web Services | AWS re:Invent 2014

DDoS Threat Landscape - CHI-NOGchinog.org/.../2016/05/10.-DDoS-Threat-Landscape.pdf · DDoS Threat Landscape . DDoS Handbook • A history and overview of DDoS • Review of attack

DDoS Protection Protecting Against The DDoS Attacks Since 2007

DDOS REPORT | 2019 Global DDoS Threat Landscape · Global DDoS Threat Landscape DDOS REPORT | 2019. 2 2019 oa DDoS Trea acape Repor Contents 01 Highlights 02 Overview 03 Network layer

Prolexic DDoS Analytics & DDoS Mitigation In Real Time

(SEC305) IAM Best Practices | AWS re:Invent 2014

(SEC305) How to Become an IAM Policy Ninja in 60 Minutes or Less

DDoS Protection Place For DDoS Attacks

Evolving DDoS Threat and Standards Based Mitigation Techniques Evolving DDoS Thre… · Considerations For Determining Best DDoS Strategy DDoS protection strategy appropriate for