PowerPoint Presentation

Managing Certificates and TLS Endpoints

Charles Neill & Henry Yamauchi Security EngineeringRackspace Hosting

1

Charles Neill & Henry Yamauchi of the Security Engineering team at Rackspace

Tasked with managing a large number of TLS/SSL endpoints

Since Heartbleed, everyone's eyes are on TLS/SSL

Many headaches later, we now have a tool called TLSManager to help us out! (More on that later)www.rackspace.com2What's this all about?

Wire encryption protocol

Provides authentication, integrity and confidentiality

Based on Public Key Infrastructure (PKI)

SSL originally designed at Netscape for eCommerce in the early-mid 1990sSSLv1 was never released, leaving us with SSLv2 and v3 + TLSv1.0, 1.1, and 1.2 today

Widely deployed on a variety of devices (smart phones, laptops, servers, appliances, etc.)

Deceptively simple to get started with (but can be much harder to maintain)www.rackspace.com3What is TLS/SSL?

Time is fast approaching where TLS/SSL will be mandatory

Browsers beginning to warn users about sites not using TLS/SSL

Firefox ChromeEncryption overhead, cost are no longer valid excusesInitiatives like Let's Encrypt (letsencrypt.org) are aiming to provide a free SSL certificate authority very soon to provide free certificates for all

Despite becoming easier/cheaper to set up, TLS/SSL can't just be turned on and left alone in most cases, as we'll seewww.rackspace.com4TLS adoption is increasing

4

EmailIRCAPIs and web servicesVPNAppliances (load balancers, proxies, etc)Smart phone apps

www.rackspace.com5TLS/SSL not just for web servers

5

Certificate Authorities (CAs)ProtocolsCiphersKey exchangeKey sizesHashesRandom Number Generators (RNGs)www.rackspace.com6TLS/SSL moving parts

Multiple protocol versions (SSLv2, v3, TLSv1, v1.1, v1.2)

Certificates expire at different times

Server misconfiguration, multiple servers for single domain

Outdated systems

Large number of endpoints to keep track of

www.rackspace.com7Issues managing TLS/SSL:

Intermediate certs expire

Server certificates can accidentally get checked into version control and have to be revoked

Lax or compromised CAs may have to reissue their certs (e.g. Comodo attack)

Use of self-signed certificateswww.rackspace.com8Also:

New SSL vulnerabilities are discovered (e.g. POODLE)

Bugs in encryption software like OpenSSL or LibreSSL (e.g. Heartbleed)

Regulation compliance (e.g. PCI-DSS 3.1 will disallow TLSv1 in 2016)

SHA1 in certs will be flagged insecure in Jan 1st, 2017 (browser warnings now)

www.rackspace.com9Oh, and don't forget:

10www.rackspace.com

11www.rackspace.comCiphers

When first connecting, client and server engage in negotiation

Client initially sends the cipher suites it supports

Server then (ideally) selects the strongest cipher that both supports

Server should be configured with only strong ciphers

There's a catch: the client essentially gets to decide what cipher is used by controlling the ciphers that they send to the server. In some man-in-the-middle attacks, the attacker can potentially downgrade ciphers in this way (e.g. LOGJAM).

www.rackspace.com12TLS cipher negotiation

Combination of encryption, MAC and key exchange algorithms

Available suites depend on the SSL/TLS version

TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 ^ ^ ^ ^ | | | | | | | MAC hash algo | | Symmetric key attrs (algo, key size, mode) | Authentication algo Key exchange algowww.rackspace.com13Cipher suites

RecommendedAvoidKey ExchangeECDHE_ECDSADHE_RSARSANULL*_anonMAC HashSHA384SHA256NULLMD5SHA1CiphersAES_256_GCMAES_128_GCMAES_256_CBC AES_128_CBCNULLRC2_*RC4_*DES_*IDEA_*3DES_**_EXPORT_*

www.rackspace.com14Cipher suite suggestions

15www.rackspace.comOther configuration options

Avoid DES, RC4, MD5, 3DES key length < 128, export ciphers

Prefer AES, GCM, SHA2, TLSv1.2, OpenSSL >= 1.0.1p / 1.0.2d

Not all permutations of these are supported (see: cipher suites)

Sometimes you are dependent on vendor support (e.g. hardware load balancers)

Some servers deliberately configured with weak ciphers to support old browsers (IE6 still in use)

Cert expiration is just one reason to replace them (other reasons: use of SHA1, weak key sizes, cert revocation, CA compromises, etc)www.rackspace.com16Server configuration

Example nginx configurationssl_protocols TLSv1.2 TLSv1.1 TLSv1;ssl_ciphers ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA; ssl_prefer_server_ciphers on;

Example Apache configurationSSLProtocol All -SSLv2 -SSLv3SSLCipherSuite HIGH:!aNULL:!MD5SSLHonorCipherOrder on

More in-depth examples can be found on https://cipherli.st/www.rackspace.com17Configuration examples

Certificate ordered/installed has too short a key (< 1024 bits)

Full certificate chain (CA bundle) not installed

Preferred cipher not set correctly

Honor cipher order feature not turned on

SSL engine/library outdated

Server not sufficiently protected (so private key is not protected)

Servers in load balance group configured inconsistentlywww.rackspace.com18Other server configuration issues

Protects past conversations even if server private key is compromised

Uses ephemeral keys in key agreement protocols

ECDHE_ECDSA, DHE_RSA

www.rackspace.com19PFS (Perfect Forward Secrecy)

Server configuration to only accept secure connections for this site

Server initially sends Strict-Transport-Security header to browser

Browser makes a note that site should only be accessed with HTTPS

If an attempt is made to connect to port 80, force user to reconnect via HTTPS (port 443 by default)

Particularly useful if sites mix secure and unsecure resources

www.rackspace.com20STS / HSTS

21www.rackspace.comVulnerabilities

Heartbleed (leaks random memory in plain text. Fixed in OpenSSL 1.0.1g)

Logjam (force export ciphers) / WeakDH (weak/reused Diffie-Hellman params)

BEAST (browser issue but can be mitigated server side. TLSv1 and earlier)

CRIME (TLSv1 compression)

FREAK (export ciphers)

POODLE (affects SSLv3)

www.rackspace.com22Some recent vulnerabilities

Problem: MITM can force client and server to agree to use a lower protocol version than what they actually support

Note: The above is possible because SSL/TLS handshake is in cleartext (session key has not been agreed to yet)

Solution: Browser include a dummy cipher to tell the server to check whether its highest protocol version is greater than ClientHellos. If so, abort communication.

Fake ciphers (x00FF)EMPTY-RENEGOTIATION-INFO-SCSV TLS_FALLBACK_SCSV

* Signaling Cipher Suite Valuewww.rackspace.com23TLS downgrade attack protection (SCSV*)

Ciphers allowed by the US Government for the export market

Devised in the 1990s. Laws restricting encryption export are no longer in force.

TLS 1.2 no longer includes them

Export ciphers use short keys (40-56 bit), making them insecure

Identified by the presence of EXP or EXPORT in their name

www.rackspace.com24Export ciphers

25www.rackspace.com

Just because an endpoint is acceptable today does not mean it will remain so tomorrow since:

Certificates expireServers are reconfiguredNew vulnerabilities are uncoveredIncreasing computing power can make old standards/assumptions obsolete (e.g. SHA1)

This means you need to be constantly monitoring your endpoints

www.rackspace.com26In summary

www.rackspace.com27Constantly?!

28www.rackspace.comHow to keep up?!

OpenSSL's "s_client" is useful for debugging SSL. On most Linux/UNIX systems, you can call it like so:openssl s_client [-debug] host host.com port 443

The "-debug" flag will give you more information about the actual connection, showing a hex dump of the handshake and negotiation process

Doesn't tell you much about vulnerabilities, but gives you the information you need if you already know what you're looking forwww.rackspace.com29OpenSSL's s_client

From the testssl.sh site:

"testssl.sh is a free command line tool which checks a server's service on any port for the support of TLS/SSL ciphers, protocols as well as recent cryptographic flaws and more."www.rackspace.com30testssl.sh

Reports SSL/TLS issues such as:SSL/TLS versionWeak protocols, ciphers and hashesKnown vulnerabilities

Uses an easy to understand letter rating system

Scoring is calculated by assigning weights to:Protocol support (30%)Key exchange support (30%)Cipher strength (40%)

Can be accessed through API Official API client here: https://github.com/ssllabs/ssllabs-scanwww.rackspace.com31Qualys SSL Labs

ScoreMeaningA+Domains with no warnings and use HSTS and TLS_FALLBACK_SCSVADomains with 2048 bit keys, at least TLS 1.2 support and no warningsA-Domains with 2048 bit keys, at least TLS 1.2 support but with some warningsBUses SSL 3.0, RC4, missing intermediate certificates, vulnerable to BEAST or CRIME, NS >= 65CDomain is vulnerable to POODLE, NS >= 50DNumeric Score >= 35 ENumeric Score >= 20 MDomain and cert host mismatchTCertificate is not trustedFUses MD5, keys with length < 1024, or SSLv3 is the strongest protocol supported

www.rackspace.com32Qualys letter rating system

33www.rackspace.comYou want an A+, right?

https://cipherli.st

Collection of good configurations for TLS/SSL, covering several different servers (Apache, nginx, HAProxy, Postfix, OpenSSH )

Useful for making recommendations to product teams for software you might not have configured before

Simple copy-and-paste format takes headaches out of implementers making decisions about certain ciphers or other settings (though you should vet these configurations based on what clients you want/need to support!)

www.rackspace.com34Cipherli.st

Hosts several endpoints with broken SSL configurations

Great resource for vetting SSL verification code / test scripts

Examples:https://expired.badssl.com/https://self-signed.badssl.com/

Many, many more!www.rackspace.com35badssl.com

www.rackspace.com36Wanna manage all those endpoints manually?

www.rackspace.com37One possible solution: TLSManager

Automatically scans daily all endpoints in its database

Uses Qualys Labs SSL tool (https://www.ssllabs.com/ssltest)

Reports SSL vulnerabilities

Proactively warns of any certificates about to expire

Emails reports to interested parties

Keeps a history of the resultswww.rackspace.com38Features

38

39www.rackspace.comDEMO TIME

40www.rackspace.com

On Github: https://github.com/rackerlabs/TLSManager OR http://bit.ly/1MUQiss

License: Apache 2.0

Pull requests most welcome!www.rackspace.com41Go get it!

42www.rackspace.comCurrent Events

www.rackspace.com43Google + BoringSSL

www.rackspace.com44Let's Encrypt

www.rackspace.com45SHA-1 IS TOTES BROKEN! SAVE YOURSELF!

www.rackspace.com46SHA-1 IS TOTES BROKEN! SAVE YOURSELF!

www.rackspace.com47Diffie-Hellman APOCALYPSE!1!!

SSL/TLS management is more involved than it looks at first

Lots of moving parts

A constantly changing landscape: a server rated A+ today can be downgraded tomorrow if:Certs expireNew cipher vulnerabilities are uncoveredBugs in SSL/TLS libraries are reportedServer is misconfiguredCert or CA compromise

It's helpful to monitor SSL endpoints in an automated way to keep from missing any issueswww.rackspace.com48Closing thoughts

www.rackspace.com49Questions?

Charles NeillTwitter: @ccneillGithub: cneillEmail: cneill@inventropy.usHenry YamauchiEmail: henry.yamauchi@rackspace.com

Group that focuses on the security of one of the largest open source projects in the world!

IRC channel: #openstack-security on Freenode

Weekly meetings: #openstack-meeting-alt @ 1700 UTC Thursdays

Mailing list: openstack-dev ML with [Security] tag

www.rackspace.com50OpenStack Security Project

THANK YOURACKSPACE | 1 FANATICAL PLACE, CITY OF WINDCREST | SAN ANTONIO, TX 78218US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM RACKSPACE LTD. | RACKSPACE AND FANATICAL SUPPORT ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM

51