HeartbleedNick Sullivan (@grittygrease)

Friday, August 8, 2014

Overview• What is Heartbleed?

• The Heartbleed Challenge

• How certificate revocation is broken and endangered our network

2

Application Layer - CloudFlare• DNS (TCP & UDP port 53)

• HTTP (TCP port 80)

• HTTPS (TCP port 443) - powered by OpenSSL

3

Heartbleed• A bug so bad it has its own website

and branding

4

What is it?• A bug in OpenSSL 1.0.1

!

• Changelog

• Major changes between OpenSSL 1.0.0h and OpenSSL 1.0.1 [14 Mar 2012]:

• TLS/DTLS heartbeat support.

5

What is a heartbeat?• Heartbeat: a keepalive extension to TLS

!

• Client sends (length, challenge)

• Server sends (length, challenge)

6

What was the bug?• Classic buffer over-read

• Attacker sends length valuethat is too long

• Server returns attacker suppliedamount of memory (up to 64kB)

7

8

Why was it so dangerous?• One request gets attacker server data

• Typically not logged — doesn’t leave a trace

• Valuable information

• Random decrypted data

• Login session cookies

• SSL/TLS private keys (???)

9

Who was vulnerable?

10

Who was vulnerable?• Any server running OpenSSL

• Apache and nginx use OpenSSL by default: 65% of all active sites

!

!

• 0.8% of the top 200,000 still vulnerable (May 2, 2014)

11

Who was impacted?• Almost everybody

12

Who discovered it?• Neel Mehta at Google

• Codenomicon

!

• Sometime in March 2014 or earlier

13

Disclosure — keeping it secret• CloudFlare, Google, Akamai, Facebook, others were notified early

• Why: large web-facing networks with the largest impact(?)

!

• Encrypted communication

• Source code visibility restricted to need-to-know

• Secure software upgrade

14

Big Questions• Were private keys at risk?

• Do I have to revoke all my certificates?

15

The CloudFlare Heartbleed challenge• Can someone really steal private keys from nginx?

• Code said probably not

• Temporary variables cleaned up

• Private key allocated earlier

!

• We set up a challenge on https://www.cloudflarechallenge.com/heartbleed

16

The CloudFlare Heartbleed challenge!

!

!

• Results: solved in under 10 hours

• Private keys are vulnerable

• Server had 200Mbps of “mystery” outbound traffic

17

18

RSA• Two prime numbers P & Q

• Public key, including P x Q

• Finding P or Q can get you the private key

19

How it was solved• Take every 128byte block

• Attempt to divide into public RSA key

!

• Coppersmith’s attack (only requires partial prime factor)

20

How it was solved• Why was the private key on the heap?

• There was a second bug in OpenSSL

!

• The prime factor was used in the computation in a temporary variable

• Temporary variables were not cleaned during a resize

21

Challenge aftermath• Undeniable key compromise potential

• All certificates need to be revoked and re-keyed

!

• CloudFlare revoked over 100,000 certificates

22

How revocation works• Certificate Revocation Lists (CRLs)

• Online Certificate Status Protocol (OCSP)

• CRLSets (Google Chrome proprietary)

23

Revoking 100,000 SSL certificates in 24 hours

24

Revoking 100,000 SSL certificates in 24 hours

25

Revoking 100,000 SSL certificates in 24 hours• CRL for GlobalSign grew from 22KB to 4.7MB

• CloudFlare provides caching for these CRLs

• We started seeing 30Gbps extra baseline traffic

• Repeated waves of 100Gbps every three hours (24 hours below)

26

Revoking 100,000 SSL certificates in 24 hours• The issue: CRL was being downloaded by web browsers

• New CRL was being published on a 1-3 hour basis

• Internet Explorer 7/8 downloads CRLs, 9/10 OCSP with CRL fallback

• OS X: OCSP with CRL fallback

• No delta updates, we downloaded the whole thing

27

Revoking 100,000 SSL certificates in 24 hours• Intra-machine links were being congested

• Had to modify cache strategy

• Moved from one cache box per rack to caching on all boxes

• Update cache headers to increase browser cache time

• Asked CA to give CloudFlare their own intermediate certificate

28

Revocation is broken• None of 100,000+ certificates were in Chrome’s CRLSets

• CRL growth can’t scale

• Too many cases when OCSP hard fail

29

Revocation is broken• Most efficient revocation code ever:

30

Revocation solutions?• Shorter certificate expiration periods

• CRL lists not necessary after expiration

• OCSP Must-staple

• Server performs OCSP check and sends to client when connecting

31

Conclusion• Bug in pervasive server software

• Huge unexpected impact on Internet security

• Crowdsourcing works

• Revocation shown to be problematic

32

HeartbleedNick Sullivan (@grittygrease)

Friday, August 8, 2014