Upload
tim-hilliard
View
57
Download
0
Embed Size (px)
Citation preview
This session
1. Hearbleed case study2. Q & A with:
• Tim Hilliard (Cloud Eng)• Adam Malone (Support)• Chris O’Neill (Support)• Phil Ingrim (Ops)
Risk assessment
Lucid:[00:35:27] [email protected]:~# openssl versionOpenSSL 0.9.8k 25 Mar 2009!Precise:[00:34:37] [email protected]:~# openssl versionOpenSSL 1.0.1 14 Mar 2012
Where’s Wally OpenSSL8000 EC2 Machines:- 99.9% of them puppetized- Candidates:
- Balancers- SVN Servers- Appliances
- ELBs- 3rd party AMIs
- Unique little snowflakes(Jira, Crucible,…)
Stack
Web tier
Other services DB, shared filesystem, memcache
BalancersVarnish
Port 80 Port 443Nginx
ELBsInternet
Here!
Stack
Web tier
Other services DB, shared filesystem, memcache
BalancersVarnish
Port 80 Port 443Nginx
ELBsInternet
Here!
and here!
Support
11:52:32 Adam Malone: hi QQ opes, I here ther is a heartbloom security issue with ssh. Is this being treated with high urgencies (p1) we need to escalade this if possible
RolloutWe did not fail over EIPs to passive balancers when
upgrading Nginx. !
Failing over an EIP leaves the IP disassociated for up to about 3 minutes. Upgrading Nginx in place takes as long as it takes
to restart Nginx. So a matter of seconds. !
Linux package management ++
Internal
• Pre-determined chat rooms
• Dial-in conference bridges
• A communication plan
Thanks SSAE-16, PCI and FedRAMP… I guess :)
Documentationhttps://docs.acquia.com/articles/heartbleed-acquia-cloud
Since then:Incident Commander
(shamelessly stolen from Heroku)http://en.wikipedia.org/wiki/Incident_command_system